Taming Randomness: Neural Networks Predict Stochastic Systems

by

Author: Denis Avetisyan

A new deep learning framework offers a powerful approach to modeling and predicting the behavior of complex systems governed by stochastic differential equations.

Following 5000 epochs of training, the model achieved performance metrics with $n=4096$, demonstrating a stabilized state indicative of its capacity within the defined parameters.
Following 5000 epochs of training, the model achieved performance metrics with $n=4096$, demonstrating a stabilized state indicative of its capacity within the defined parameters.

This work introduces Stochastic Physics-Informed Neural Networks (SPINNs) for approximating solutions to stochastic differential equations driven by Lévy processes using a path space representation.

Accurately simulating stochastic dynamics governed by Lévy processes remains a persistent challenge in many scientific and engineering domains. This paper introduces SPINNs-a deep learning framework for approximation of stochastic differential equations-that addresses this limitation by representing solutions as deterministic functions of the driving noise. Leveraging neural networks, SPINNs learns this functional mapping, offering a novel approach to solving stochastic differential equations. Could this methodology unlock more efficient and accurate simulations across fields reliant on modeling random phenomena?

The Inevitable Dance of Randomness

The behavior of countless natural and engineered systems – from the fluctuations of financial markets and the diffusion of pollutants to the movements of microscopic particles and the growth of populations – is fundamentally governed by randomness, necessitating the use of Stochastic Differential Equations, or SDEs, to model them. These equations account for unpredictable forces and their cumulative effect on system evolution, but their very nature introduces significant computational challenges. Obtaining accurate and efficient solutions to SDEs is therefore crucial not only for theoretical understanding, but also for practical applications like prediction, control, and optimization. The complexity arises because SDEs don’t yield simple, deterministic answers; rather, they describe a probability distribution of possible outcomes, demanding methods capable of faithfully representing this inherent uncertainty. Consequently, research into robust and scalable numerical schemes for solving SDEs remains a vibrant area of scientific inquiry, impacting fields ranging from physics and engineering to finance and biology.

The practical application of Stochastic Differential Equations (SDEs) is frequently hampered by the computational demands of their solution. Conventional numerical schemes, while effective in lower dimensions or with simple noise, rapidly become intractable as the number of variables increases – a phenomenon known as the ‘curse of dimensionality’. Furthermore, many real-world processes are driven by complex noise patterns – such as those exhibiting non-Gaussian characteristics or correlated fluctuations – that render standard methods inaccurate or unstable. This limitation restricts the ability to model intricate systems accurately, from financial markets and turbulent fluids to biological processes and climate dynamics, necessitating the development of more robust and scalable solution techniques to unlock the full potential of SDE-based modeling.

The accurate modeling of stochastic processes demands a nuanced understanding of the space in which their solutions reside. Unlike deterministic systems with solutions existing within Euclidean space, solutions to Stochastic Differential Equations (SDEs) are typically functions evolving over time, and these functions aren’t necessarily continuous. Consequently, the appropriate mathematical framework isn’t simply a vector space, but a more general function space. The Skorokhod Space, comprised of càdlàg functions – those that are right-continuous with left limits – provides this necessary structure. This space accounts for the jumps and discontinuities frequently present in solutions of SDEs driven by noise, ensuring mathematical rigor and preventing invalid operations. Ignoring this crucial aspect can lead to incorrect analysis and flawed simulations, particularly when dealing with systems exhibiting abrupt changes or discontinuities, highlighting the importance of the Skorokhod Space in the reliable study of stochastic dynamics.

Neural Networks as Stochastic Cartographers

Stochastic Physics-Informed Neural Networks (SPINNs) represent a computational technique for approximating solutions to Stochastic Differential Equations (SDEs) by leveraging the function approximation capabilities of Artificial Neural Networks. Traditional numerical methods for SDEs can be computationally expensive or struggle with high-dimensional problems. SPINNs address these limitations by training a neural network to learn the solution manifold of the SDE, effectively mapping initial conditions to the probabilistic distribution of future states. This is achieved by formulating the SDE problem as an optimization task where the network’s parameters are adjusted to minimize a loss function that quantifies the discrepancy between the network’s predicted solution and the true, but often unknown, solution of the SDE. The approach allows for the estimation of solutions to $SDEs$ without requiring explicit discretization schemes, potentially offering significant computational advantages.

SPINNs build upon the foundation of Physics-Informed Neural Networks (PINNs) by specifically addressing limitations in modeling stochastic differential equations (SDEs). Traditional PINNs often struggle with the inherent randomness present in SDEs, leading to inaccurate or unstable solutions. SPINNs introduce modifications to the network architecture and loss function to effectively represent and learn from the probabilistic nature of these equations. This is achieved through techniques that allow the network to approximate the stochasticity, enabling more accurate predictions of system behavior when subjected to random forces or noise. Consequently, SPINNs offer improved performance and reliability in applications involving systems governed by stochastic dynamics, such as financial modeling, weather forecasting, and molecular dynamics.

SPINNs utilize a process of $NeuralNetworkTraining$ to iteratively refine the weights and biases of an artificial neural network. This training is guided by a specifically designed $LossFunction$ which quantifies the discrepancy between the network’s predicted solution to a stochastic differential equation (SDE) and the known, true solution. The $LossFunction$ typically incorporates multiple terms, including a measure of the residual of the SDE, a penalty for deviations from initial and boundary conditions, and potentially terms related to the stochasticity inherent in the SDE. Minimization of this loss, usually via gradient-based optimization algorithms, effectively adjusts the network’s parameters to produce increasingly accurate approximations of the SDE’s solution.

After 5000 epochs of training on example 45, the model achieves performance with n=4096.
After 5000 epochs of training on example 45, the model achieves performance with n=4096.

Decoding Stochasticity: Methods and Theoretical Underpinnings

Stochastic Neural Networks (SPINNs) offer a solution to stochastic differential equations (SDEs) applicable to both AdditiveNoise and MultiplicativeNoise scenarios, representing an advancement over conventional SDE solution methods. Traditional techniques often struggle with the complexities introduced by state-dependent noise, inherent in MultiplicativeNoise. SPINNs, however, are capable of approximating solutions for SDEs of the form $dX_t = \mu(X_t, t)dt + \sigma(X_t, t)dW_t$, where $\sigma$ can be state-dependent, without requiring simplifying assumptions or transformations in certain cases. This capability extends the applicability of neural networks to a broader class of stochastic problems, enabling simulation and analysis of systems affected by more realistic noise structures.

Stochastic Partial Differential Equations (SPDEs) involving multiplicative noise require specific treatment due to the non-linearity of the noise term. SPINNs addresses this by employing the Doss-Sussman transformation, a technique that effectively linearizes the SPDE, allowing for accurate numerical approximation. This transformation involves a change of variable that converts the multiplicative noise into an additive noise term, enabling the use of standard solution techniques applicable to additive noise SPDEs. By applying the Doss-Sussman transformation prior to numerical discretization, SPINNs mitigates the challenges associated with directly approximating SPDEs with multiplicative noise and maintains solution accuracy.

The reliability of the Stochastic Parameter Identification Neural Network (SPINN) method is substantiated by theoretical error bounds derived using the Grönwall Lemma. This lemma provides a framework for establishing the stability and convergence of the approximated solutions to Stochastic Differential Equations (SDEs). Specifically, it allows for the derivation of an upper bound on the error between the true solution and the solution obtained through the SPINN approximation, given certain conditions on the noise terms and the network’s parameters. These bounds are crucial for quantifying the accuracy of the method and validating its performance, ensuring that the error remains within acceptable limits for a given time horizon and level of noise. The established bounds contribute to the confidence in SPINN’s ability to accurately approximate solutions to SDEs, particularly in scenarios where analytical solutions are unavailable.

The training of SPINNs utilizes the Robbins-Monro algorithm, a stochastic approximation technique designed for optimizing functions defined by expectations. This algorithm iteratively updates parameters based on noisy observations, enabling efficient optimization in scenarios where exact gradient calculations are intractable. Specifically, the algorithm employs a decreasing step size, denoted as $\alpha_n$, to balance exploration and exploitation during the learning process. The convergence of the Robbins-Monro algorithm, under certain regularity conditions, guarantees that the estimated parameters converge to the optimal solution as the number of iterations, $n$, approaches infinity, making it suitable for training neural networks to approximate solutions to stochastic differential equations.

In a specific implementation with $n=4096$, the Stochastic Parameter Identification Neural Network (SPINNs) achieved an error value of approximately 1.5e-04 at the final time, T. This level of accuracy was obtained following 5000 training epochs, demonstrating the network’s convergence and performance in approximating solutions to stochastic differential equations under the specified parameters. The reported error represents the difference between the SPINNs-approximated solution and the true solution at time T, providing a quantitative measure of the method’s efficacy.

Following 5000 training epochs, Spiking Neural Networks (SPINNs) exhibit an accumulated error of approximately $2.5 \times 10^{-3}$ when simulating stochastic differential equations over the time interval $[0, T]$, using a network size of n=4096. This error value represents the total deviation between the SPINNs-approximated solution and the true solution of the SDE across the specified time interval. The metric provides a quantifiable measure of the method’s accuracy and serves as a benchmark for performance evaluation with varying network sizes and training durations.

Training loss decreased alongside approximated errors during example 1, indicating successful model convergence.
Training loss decreased alongside approximated errors during example 1, indicating successful model convergence.

The Evolving Landscape: Implications and Future Trajectories

Stochastic differential equations ($SDE$s) model systems evolving randomly over time, appearing ubiquitously across diverse scientific fields, yet their solutions often demand substantial computational resources. Spiking neural networks, or SPINNs, present a fundamentally different and increasingly viable approach. By leveraging the inherent parallelism and energy efficiency of brain-inspired computation, SPINNs offer a robust alternative to traditional numerical methods for solving $SDE$s. This bio-inspired architecture doesn’t rely on iterative calculations, but rather maps the stochastic process directly onto the network’s spiking dynamics, resulting in faster and more energy-efficient solutions. Consequently, SPINNs are poised to impact fields ranging from financial modeling – where accurate option pricing and risk assessment depend on solving $SDE$s – to physics simulations of Brownian motion and engineering applications involving noisy control systems, offering a pathway to tackle previously intractable problems with greater efficiency and scalability.

Stochastic differential equations (SDEs), ubiquitous in modeling real-world phenomena, often present significant challenges due to the inherent noise within the systems they describe. Traditional numerical methods struggle when confronted with complex or non-Gaussian noise, limiting their applicability. However, Spiking Neural Networks (SPINNs), through their inherent ability to process asynchronous, noisy signals, offer a powerful alternative. Recent advancements demonstrate SPINNs can effectively handle a wider spectrum of noise types-including those with non-standard distributions-than conventional techniques. Moreover, the integration of machine learning algorithms allows these networks to learn and adapt to the specific characteristics of the SDE, effectively bypassing limitations previously encountered. This adaptability unlocks the potential to model and solve previously intractable problems in fields ranging from financial modeling-where asset prices exhibit complex volatility-to physics and engineering, opening new avenues for simulation and prediction.

The continued development of Stochastic Parameterized Neural Networks (SPINNs) promises to unlock solutions to increasingly complex scientific challenges. Current research focuses on extending SPINNs beyond their present capabilities by tackling higher-dimensional Stochastic Differential Equations ($SDEs$). These higher-dimensional problems, prevalent in fields like multi-asset finance and complex fluid dynamics, require significantly more computational power and sophisticated modeling techniques. Simultaneously, investigations are underway to incorporate more nuanced and realistic noise processes – moving beyond simple Wiener processes to include phenomena like Lévy jumps or colored noise. Successfully integrating these advancements will not only broaden the scope of problems addressable by SPINNs but also enhance the accuracy and reliability of their solutions, potentially revolutionizing simulations and predictions across diverse scientific disciplines.

The pursuit of approximating stochastic differential equations, as detailed in this work on Stochastic Physics-Informed Neural Networks, reveals an inherent tension between modeling complexity and computational stability. The article demonstrates an attempt to map the unpredictable fluctuations of Lévy processes onto deterministic functions, acknowledging that all representations are, ultimately, transient. This echoes Perelman’s sentiment: “I don’t think I’ve made any significant contribution to mathematics.” Though focused on vastly different domains-mathematics versus machine learning-the statement underscores a shared understanding: any system built to model reality contains an inherent degree of approximation, and the illusion of permanence is merely a function of the time scale considered. The method presented attempts to cache this instability within the network’s learned function, accepting latency as the unavoidable tax on every request for a solution.

What Lies Ahead?

The introduction of Stochastic Physics-Informed Neural Networks represents, predictably, not an arrival, but a relocation of difficulty. The field has shifted from seeking solutions to stochastic differential equations to seeking accurate representations of the mapping between driving Lévy processes and solution states. This is a subtle, yet critical, change; it acknowledges the inherent incompleteness of any finite approximation. Technical debt accumulates not in the code, but in the necessary simplifications of the underlying stochasticity.

Future work will likely focus on the limits of this path-space representation. Can the deterministic function learned by the network truly capture the fractal complexity inherent in many Lévy processes, or will it inevitably smooth over critical features? Uptime, in this context, is not sustained operation, but a rare phase of temporal harmony between model fidelity and computational cost. The challenge lies not in achieving a solution, but in quantifying the nature of the inevitable decay from that ideal.

Ultimately, the success of SPINNs, and similar approaches, will depend on its ability to gracefully age. The system will not avoid entropy; it will merely redistribute it. The pertinent question, then, is whether the resulting errors are benign-manifesting as gradual drift-or catastrophic, revealing fundamental limitations in the representational power of the chosen architecture.

Original article: https://arxiv.org/pdf/2512.14258.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2025-12-17 20:28