ocramz

From HJB to Stochastic Programming and Reinforcement Learning

The Hamilton–Jacobi–Bellman (HJB) equation is a continuous-time object, while multistage stochastic programming (SP) and reinforcement learning (RL) are both discrete-time. This note derives both from HJB through a common ancestor: the Bellman recursion obtained by discretizing time. From there the two fields diverge, because each approximates the three hard objects in that recursion (an expectation, a minimization, and a value function) in a different way and under different assumptions.

Costs are minimized throughout, following the SP convention. RL's reward is simply $r = -c$.

---

0. Setup: the HJB equation

ocramz

/-
Marco Zocca (@ocramz) 2026
-/
import Mathlib.Analysis.LocallyConvex.Separation
import Mathlib.Data.EReal.Operations
import Mathlib.Topology.Semicontinuity.Basic

/-!
# The Fenchel-Moreau theorem in a locally convex space

ocramz

{-# language DeriveDataTypeable #-}
{-# language FlexibleContexts #-}
{-# language StandaloneDeriving #-}
{-# language TypeOperators #-}
{-# LANGUAGE UndecidableInstances #-}
{-# language TypeFamilies #-}
module Data.Recursion where

import Data.Data (Data)
import           Data.Functor.Classes (Read1(..), Show1(..), Eq1(..), Ord1(..), Show1(..), compare1, eq1, showsPrec1, readsPrec1)

ocramz

#!/bin/bash

apt update
apt -y install build-essential  # install GCC
apt -y install --reinstall software-properties-common # enable add-apt-repository
apt-get -y install lshw
apt-get clean

export DISTRO_UBUNTU=ubuntu2204/x86_64

ocramz

import Data.Bits ((.|.),(.&.),shiftL)
import Data.Char (chr, isHexDigit, digitToInt)
import Data.Text (Text, pack, unpack)

{-
sources : 

uri-encode (https://hackage.haskell.org/package/uri-encode-1.5.0.7/docs/src/Network.URI.Encode.html#decodeText)

network-uri (https://hackage.haskell.org/package/network-uri-2.6.4.2/docs/src/Network.URI.html#unEscapeString)

ocramz

<!DOCTYPE html>
<html>
<body>

 Select some text below and then click the button:<br/>
<textarea id="myText" rows="5" cols="60" onselect="alert(getTextSelection())">
Lorem ipsum dolor sit amet, 
consectetur adipiscing elit.
</textarea>

ocramz

<!DOCTYPE html>
<html>
<head>
{{>head}}
    {{#user}}
    <title>Back {{name}} checkout | satsbacker</title>
    {{/user}}
</head>
<body>
    {{>header}}

ocramz

{-# LANGUAGE RecordWildCards #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE BangPatterns #-}

module Invoicing
    ( invoiceRoutes
    , newInvoice
    , waitInvoices
    ) where

ocramz

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "width": 200,
  "height": 200,
  "padding": 5,
  "data": [
    {
      "name": "d1",
      "values": [
         {"a": 0, "c": "2.5", "z": 0},

ocramz

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "width": 200,
  "height": 200,
  "padding": 5,
  "data": [
    {
      "name": "d1",
      "values": [
        {"a": 0, "b": 0, "c": "36.5"},