Richard Bellman's 1952 work on dynamic programming laid the mathematical foundation for today's reinforcement learning and diffusion models. His continuous-time formulation produced a partial differential equation that mathematicians later recognized as identical to the 19th-century Hamilton-Jacobi equation from physics.
This mathematical bridge connects several modern ML topics: continuous-time reinforcement learning, stochastic control, diffusion models, and optimal transport. The key insight is that Bellman's value function satisfies the Hamilton-Jacobi-Bellman equation, which governs optimal control in both deterministic and stochastic settings.
The HJB equation takes the form $
ho V(x) = max_a { r(x,a) + mathcal{L}^a V(x) }$ for controlled diffusions, where $mathcal{L}^a$ is the infinitesimal generator. This structure underlies policy iteration algorithms in continuous-time reinforcement learning, where neural networks represent both value functions and policies.
Source: Hacker News · Summarized by HeadlinesBriefing