Drag reduction or reward hacking? Recurrent multi-agent reinforcement learning that earns its reward G. M. Cavallazzi1,∗ , M. Pérez Cuadrado1 , A. Pinelli1
arXiv:2606.06227v1 [physics.flu-dyn] 4 Jun 2026
1
School of Science and Technology, Department of Engineering, City St. George’s, University of London, London, UK ∗ Corresponding author: [email protected]
Abstract A reinforcement-learning agent maximises its reward, which can diverge from the outcome its designer intended. In physical control the reward rarely closes that gap, and drag reduction in wall turbulence makes it concrete. A mass-conservation projection couples agents’ outputs and erases the per-agent credit the policy gradient needs; a memoryless policy cannot resolve the slow near-wall cycle it acts on; and a pressure-gradient reward pays for nominal drag reduction by pumping power through the wall. Two degenerate controllers achieve large drag reductions while total dissipation rises, so the reported figure can mask a more wasteful flow. We trace each fault to its cause and fix it: a differentiable projection that restores credit, a recurrent policy with a widened sensing stencil, and a reward scored on the true wall power. The corrected controller acts on the flow within a closed energy budget, earning a conservative 17% under honest accounting.
A reinforcement-learning agent optimises the reward it is given rather than the outcome its designer intended. Where the reward measures that outcome faithfully the two stay aligned, but control problems seldom supply such a measure: the reward is built from whatever can be sensed cheaply and continuously, and stands in for a target that is more expensive to evaluate. An agent trained against such a proxy will raise it by whatever route the dynamics permit, including routes that leave the intended target unmoved (Amodei et al., 2016). As reinforcement learning has passed from games into the control of physical systems (Rabault et al., 2019; Garnier et al., 2021; Vignon et al., 2023b), this gap has turned from a theoretical curiosity into a practical one, because the physics offers ways to a high reported score that a reader of the score alone never sees. Two features of physical control sharpen the difficulty. The actuation is rarely free to take any value: conservation laws of mass or of charge tie the admissible actions together, so that what one part of a distributed actuator may do depends on what the rest of it is doing. When the control is shared among many local agents, a scalable and now common arrangement, this coupling cuts against the separable per-agent contributions that the learning signal is built to credit (Foerster et al., 2018). Physical systems also carry their own time scales. A policy that maps an instantaneous measurement to an action follows a fast process but loses the phase of a slow one (Kaelbling et al., 1998; Hausknecht and Stone, 2015), and the quantity worth controlling is often the slow one. The skin-friction drag a turbulent boundary layer exerts on a surface accounts for much of the energy spent moving vehicles and pumping fluids through pipes, so cutting it even by a few percent carries a large economic and environmental payoff (Spalart and McLean, 2011). Wallturbulence drag reduction brings these reinforcement learning-related difficulties together in a setting where each can be measured against an established baseline. The near-wall region of a channel sustains a regeneration cycle of streaks and quasi-streamwise vortices (Kline et al., 1967; Hamilton et al., 1995; Waleffe, 1997) that runs autonomously within about a hundred wall units of the surface (Jiménez and Pinelli, 1999) and fixes the skin friction a controller sets out to lower. Lengths here are quoted in wall units, physical distances non-dimensionalised by the 1
near-wall viscous scale ν/uτ formed from the skin-friction velocity uτ , so that the channel height in wall units is the friction Reynolds number Re τ built on the wall shear stress. Model-based wall control has a long record against this cycle, from opposition control (Choi et al., 1994) through suboptimal and predictive laws (Lee et al., 1998; Bewley et al., 2001), with the net energy budget rather than the nominal drag used to rank schemes once the actuation cost is counted (Fukagata et al., 2009; Marusic et al., 2021). Reinforcement learning re-entered through the channel in a multi-agent form, an identical policy on every wall patch with a centralised critic during training (centralised training, decentralised execution, CTDE) (Vignon et al., 2023a; Guastoni et al., 2023), and the template that has spread since shares a parameter-shared CTDE policy, the dragreduction percentage as the objective, and the reported figure as the headline (Sonoda et al., 2023; Cavallazzi et al., 2025; Wälchli et al., 2024). That template inherits three problems that the headline figure does not expose. The first is structural to the multi-agent formulation. Blowing and suction must inject no net mass: a net flux through the wall is incompatible with incompressibility in the closed channel, and even held numerically stable it would add momentum to the flow, lowering the drag by simply moving around terms in the momentum balance. The joint action is therefore projected onto its zero-mean subspace before it reaches the flow; under that projection an agent’s applied action depends on what every other agent did, and the reward an agent should be credited for can no longer be separated from its neighbours’. The second is a question of observability. The buffer-layer cycle evolves over roughly a hundred viscous time units, while the policy acts on an instantaneous slice; a memoryless map from a single snapshot cannot represent the phase of a cycle that is slow compared with its own sampling. The third concerns the objective. In constant-flow-rate operation the drag-reduction percentage measures only the saved pumping power and ignores the power the actuation delivers to the fluid. The standard accounting in the learning literature charges that actuation a kinetic-energy-flux cost that scales with the cube of the actuation amplitude, which is then found negligible (Kametani et al., 2015; Guastoni et al., 2023); the thermodynamic work the wall actually does on the flow is a different quantity, and one that the classical control literature has long insisted on when ranking schemes by net energy budget (Fukagata et al., 2009; Marusic et al., 2021). We treat these as design faults with concrete fixes and report the controller that results (Fig. 1). A differentiable projection layer puts the zero-mean constraint inside the actor, so the policy gradient sees the constraint rather than fighting it after the fact. A recurrent core and a widened sensing stencil give the policy the memory the cycle demands. An energy-aware reward, scored against the physically correct wall power, removes the incentive to pump the wall. Two deliberately degenerate controllers, an open-loop stripe pattern and a memoryless learnt policy with none of the fixes, show what the unguarded objective rewards: both post sizeable dragreduction percentages while raising the total dissipation above the uncontrolled value, and the memoryless learnt policy reaches that state by collapsing into a fixed standing wave of its own, a reward-hacking artefact rather than control of the flow. The corrected controller, a recurrent multi-agent policy we refer to throughout as GRU-MARL, instead reduces the drag it is supposed to, at an energy budget that matches opposition and at an amplitude well below it, and it transfers from its small training domain to a much larger evaluation channel without retraining.
Results Drag-reduction percentage is not a legitimate objective All controllers are evaluated in the same constant-flow-rate half-channel at Re τ ≃ 180, on a box + + of (L+ x , Ly , H ) ≃ (1922, 576, 180) wall units actuated through the wall-normal velocity alone, with the zero-net-mass constraint Z ww (x, y, t) dx dy = 0 ∀t (1) Γw
2
{oi } oi 1
z
2
shared πθ
Lx
ai
x y
Ub
CNN → GRU → tanh + proj
{ai }
central critic Qϕ (training only) 3
Ly same πθ at every patch
∇θ
1 differentiable zero-mean projection, Jacobian δij − 1/N carried through autograd
2 recurrent GRU memory and a widened 3 × 3 sensing stencil
3 energy-aware reward scoring the true wall power Ww
Figure 1: Setup and the corrected control loop. A turbulent channel at Re τ ≃ 180 is driven at constant flow rate, and the lower wall is actuated by zero-net-mass blowing and suction (orange out, teal in) on a grid of patches; the field above the wall is the streamwise velocity on the detection plane the policy reads. Every patch runs the same actor πθ , a convolutional encoder feeding a GRU and a tanh head with the zero-mean projection as its last layer, reading a 3 × 3 ring of patches oi and emitting the wall-normal action ai ; a central critic Qϕ sees the joint state and action during training only and returns the policy gradient ∇θ . The three numbered safeguards correct, in turn, the credit assignment destroyed by the mass-conservation projection, the partial observability of the slow near-wall cycle, and a reward that would otherwise pay for drag reduction by pumping the wall. enforced at every step. In this setting the bulk kinetic-energy budget closes on two external power terms, the pumping power supplied by the flow-rate controller and the work the actuation does on the fluid, Z 1 Pp = |⟨∂x p⟩| Ub , Ww = − ⟨ww p⟩ dx dy. (2) Lx Ly Γw The viscous wall-traction work vanishes identically here: continuity at a wall that is no-slip in the wall-parallel components forces ∂z w|wall = 0 pointwise, so the only surviving wall-work term is the pressure covariance in (2). Stationarity then fixes the volume-averaged dissipation, ε = Pp +
Ww , H
(3)
with the factor 1/H converting surface work to a volume-distributed rate. The quantity a useful controller must lower is ε. The conventional benchmark instead reports DR = 1 −
|⟨∂x p⟩| , |⟨∂x p⟩0 |
(4)
the relative saving in pumping power, which says nothing about Ww . Subtracting (3) from its uncontrolled value separates the two contributions, Ww ∆ε = ε0 − ε = |⟨∂x p⟩0 | Ub − Pp − , H so a controller can raise DR by pumping the wall rather than by quieting the flow.
3
(5)
This loophole is usually argued away with the wrong cost. The standard accounting in the blowing/suction learning literature charges the actuation a kinetic-energy-flux input power Z cf,0 − (cf + Win ) 1 1 Win = |ww |3 dx dy, S= , (6) 2 Lx Ly Γw cf,0 and reports the net-energy saving S alongside the drag reduction (Kametani et al., 2015; Guastoni et al., 2023). Because Win scales as the cube of the actuation amplitude, at the O(10−2 ) amplitudes these controllers use it evaluates to O(10−6 ), two to three orders below the friction coefficient, and the net saving collapses onto the drag reduction. The proxy is the problem. It is a function of the actuation amplitude alone, so penalising it merely caps the output; it cannot register the covariance ⟨ww p⟩ through which a bounded-amplitude actuation still pumps real power into the flow. The true wall power in (2) is that covariance, and it is what enters the dissipation balance. Table 1 measures every term of (3) directly for five controllers spanning the design space, and the amplitude proxy and the true wall power part company exactly where it matters. The peak amplitude |ww |max is essentially the same, between 0.118 and 0.131, for opposition, the stripes and the memoryless vanilla-DRL policy, so the proxy (6) would rate their actuation cost as identical and negligible. Their true wall power Ww instead spans 0.007 to 2.91 × 10−3 , a factor of roughly four hundred, and it is that spread, invisible to the proxy, that decides whether a controller saves or wastes energy. The open-loop stripe pattern records the highest drag-reduction percentage in the table, 33.2%, while driving ε fourteen percent above the uncontrolled value; the memoryless vanilla-DRL policy reports 15.5% drag reduction while its wall-work lifts the total dissipation by more than half. Both are nominal successes and physical failures, and neither DR nor the amplitude proxy can separate them from a genuine reduction. Opposition and GRUMARL sit on the other side of (5): their wall-work is two to three orders of magnitude smaller, so the wall is a near-passive observer in the sense of net energy injection and their ordering by ε follows their ordering by DR. DR [%]
Pp (×103 )
uncontrolled
0.0
4.10
0.00
opposition
21.4
3.22
0.007
stripes (open-loop)
33.2
2.74
1.93
4.67
vanilla DRL
15.5
3.46
2.91
6.38
GRU-MARL
17.3
3.39
0.007
3.39
Controller
Ww (×103 )
ε (×103 )
∆ε [%]
|ww |max
4.10
0.0
0.000
3.22
+21.5
0.131
−13.9
0.128
+17.3
0.052
−55.5
0.118
Table 1: Energy budget of the five controllers, half-channel constant-flow-rate units with Ub = H = 1. ∆ε is (ε0 − ε)/ε0 × 100; positive marks a fall in total dissipation, negative a net rise. The peak amplitude |ww |max is nearly equal across opposition, stripes and vanilla DRL, so the amplitude proxy (6) cannot distinguish them; the wall power Ww separates them by a factor of ∼ 400. Bold marks the two controllers that raise ε while reporting positive DR, and the smallest peak amplitude. Three significant figures. The stripe pattern earns its place in the table as a control: it is the simplest policy that games the metric. It imposes a fixed square wave of blowing and suction that alternates along the streamwise direction and is uniform across the span, πx ⋆ , independent of y and t, (7) ww (x, y) = w sgn sin ∆s with stripe half-period ∆s at the smallest admissible patch size and amplitude fixed at the saturation level w⋆ . There is no sensor and no feedback. It is a deliberately unintelligent openloop forcing, and it reaches the highest nominal drag reduction in the table by injecting a large, 4
steady wall power. The reason it belongs next to the learnt controllers is empirical, and is the subject of the next sections: the memoryless vanilla-DRL policy converges to behaviour of the same kind. A note on the evaluation domain. The drag-reduction and energy figures reported here are all measured on the large box. A trained policy evaluated in a minimal flow unit can drive the controlled flow towards relaminarisation, at which point the pressure gradient drops to its laminar value and the drag-reduction percentage saturates at an artificially large number that reflects the box rather than the controller. The large box keeps the flow turbulent throughout, so the numbers in Table 1 are representative and are compared across controllers at identical domain size and resolution.
The mass-conservation constraint erases per-agent credit The wall is tiled into patches, each running an identical policy πθ in the parameter-shared multiagent arrangement, with a single centralised critic during training. Zero net mass forces the scalar outputs ai onto their mean, a′i = ai −
1 X aj , N
(8)
j
before they are lifted to the wall. The flow receives a′i , not ai , so the action that earns reward at patch i is a function of every other patch’s output: a small ai can be turned negative by the mean of its neighbours, and an action uniform across patches is sent to zero. Applied as a post-processing step on the emitted actions, the projection leaves the actor receiving a gradient computed for ai while the environment responded to a′i , and the per-agent credit the deterministic policy gradient relies on is corrupted by the very constraint that makes the actuation admissible (Fig. 2a). The fix is to make (8) the last layer of the actor. The projection is linear, with constant Jacobian ∂a′i 1 (9) = δij − , ∂aj N so automatic differentiation propagates the coupling back through the network at no modelling cost. The actor learns to emit actions already close to zero-mean, and the policy gradient is taken with respect to the field the flow actually sees (Fig. 2b). The mechanism is not specific to the flow. In a fluid-dynamics-free model, a shared policy controlling a set of agents under the same zero-mean projection learns down to the reachable optimum when the Jacobian (9) is differentiated through, but stalls with most of its agents pinned against their output bound when the projection is applied after the actor, the same saturation the wall policy shows. The shortfall grows with the net demand the conserved resource cannot supply and closes when that demand is zero, which fixes the constraint itself, not any dynamics, as the cause (Supplementary Fig. S2).
Without memory the policy saturates into a standing wave Reinforcement learning casts control as a Markov decision process: the action is chosen from the current state on the assumption that the state and the action together determine what comes next. A memoryless policy takes this literally, treating the instantaneous sensed slice as the full state. The near-wall flow does not comply. The buffer-layer cycle turns over on a scale of order one hundred viscous units, far longer than the snapshot the policy reads at each step, so a single plane is a partial observation and the current slice is not a sufficient statistic for the cycle the controller acts on. A fixed map from that slice cannot recover the phase of a process slow relative to its sampling, and the optimum it settles on is degenerate. What that looks like is visible in the
5
∇ on ai , not a′i a projection after the actor
πθ → ai
a′i = ai − ā
flow, reward
∇ carries δij − 1/N b projection inside the actor
a′i = ai − ā (last layer)
πθ
flow, reward
Figure 2: Credit assignment under the zero-mean constraint. (a) With the projection (8) after the actor, the gradient is computed for the emitted ai while the flow responded to the mixed a′i , so neighbouring agents contaminate each agent’s credit. (b) Made the actor’s last layer, the projection’s Jacobian (9) propagates during training and the gradient is taken on the applied field. vanilla-DRL actor, whose input is the pointwise pair of streamwise and wall-normal fluctuations (u′ , w′ ) at the detection plane and whose output is a scalar action a = tanh fθ (u′ , w′ ). Its entire policy is a surface on that plane, and the surface collapses onto an asymmetric one-dimensional switch: a saturating tanh on a single linear combination a1 u′ + a2 w′ reproduces the trained network to R2 ≃ 0.999 (Fig. 3). The policy has degenerated into a hard switch pinned against its amplitude bound, the saturated two-level regime that a continuous-control return is known to favour when nothing penalises it (Seyde et al., 2021), and the behaviour the wall-work column of Table 1 charges for. That the collapse follows from the timescale gap, and not from anything in the flow, can be shown in a system with separated scales but no turbulence. In a two-scale Lorenz–96 model, a control forced under the same zero-mean projection and rewarded for lowering the slow-variable energy reproduces both degeneracies: an actuation interval at the fast decorrelation time leaves successive observations uncorrelated and drives the policy to the same saturated two-level switching, while an interval longer than the slow turnover acts on stale information and settles to near-zero output. Only an intermediate cadence, short against the slow scale yet long against the fast one, yields a structured low-amplitude control, the surrogate of the interval ∆t+ a ≃ 5 chosen at the wall (Supplementary Fig. S3). In the channel this saturated forcing takes a particular spatial form. Figure 4 shows the instantaneous wall actuation for the three actuated controllers on the large box; vanilla DRL, although a closed-loop policy that reads the flow at every step, settles into a near-stationary banded field, a standing wave that persists in the same arrangement whatever box it is deployed in. We read this as a problem-specific outcome rather than a universal signature of the timescale failure. The high-frequency switching injects an outsized amount of power into the domain, the wall-work column of Table 1 measures it, and the particular standing wave is the flow state this solver settles into under a forcing that large, plausibly a reward-hacking interaction with the discrete pressure solve that lowers the sensed pressure gradient while the wall does the work. A banded, near-stationary wall actuation of just this kind is in fact discernible in the developed-flow snapshots reported for memoryless learnt channel controllers elsewhere, so the configuration is not peculiar to the present solver, even if its precise coupling to the discrete solver would need reproducing case by case before being read as universal. The transferable statement is the one the Lorenz–96 surrogate makes, that a memoryless policy off the right cadence either saturates or fades. Stripes is fixed by construction, an imposed open-loop forcing with no sensing, and is shown only as the reference for what a static pattern costs energetically; the contrast here is 6
that a closed-loop learnt policy arrives at a frozen pattern of its own. GRU-MARL, carrying a per-patch hidden state, instead places its actuation in register with the moving near-wall streaks of the detection-plane field. a(u0 , w0 ) (network)
3
asymmetric-tanh fit
3 1.00
2
0.50
−1.0 −1.5 −2
−1
0
1
u0 /u0rms
2
−0.75
0.00 −0.25 −0.50 −0.75
−2
−3
−2
−1
0
1
2
−1.00 −3
3
u0 /u0rms
0.2
1
0.1
0
0.0
−0.75
−1
−0.1
−1.00
−2
−0.2
a vanilla DRL: a(u′ , w′ )
−2
−1
−3
−2
−1
0
0
1
2
3
u0 /u0rms
diagonal slice
2
−3
−3
residual 0.75
−0.25 −0.50
0.25
0
−1
−0.50
3
0 w0 /wrms
0 w0 /wrms
−3
0.00
−0.5
0.50
−1.00
0.25
0.0
−0.25
−2
0.50 0.5
0.00
−1
0.75
1.0
0
0 w0 /wrms
1.00
1.5
0.25
0.75
1
1
2
3
u0 /u0rms
MSE= 4.99e − 04 MAE= 7.53e − 03 R2 = 0.9993
0.50 0.25
a
a(u , w ) 0
0 w0 /wrms
1
0
1.00 2
0.75
0.00 −0.25 −0.50 network fit
−0.75 −1.00 −1.0
−0.5
0.0
0.5
1.0
t (along u0 = t, w0 = −t)
b asymmetric 1-D tanh fit, R2 ≃ 0.999
Figure 3: The memoryless vanilla-DRL policy saturates. (a) Its action surface over the two sensed scalars at the detection plane, overlaid with the contours of the observed (u′ , w′ ) density (the white lines enclose the bulk of the sampled states), so that the saturated region the policy actually visits is visible. (b) The surface is reproduced by a saturating tanh on a single linear combination a1 u′ + a2 w′ , so the policy is effectively a one-dimensional switch pinned against its amplitude bound.
The recurrent policy is expressive and energy-honest Adding the recurrent core changes the character of the policy. GRU-MARL reads a three-channel stencil over a 3 × 3 ring of patches together with its own previous action, and carries a per-patch hidden state through time. Its input is no longer the two scalars of the memoryless policy, so the action surface of Fig. 3 has no direct analogue here; what can be drawn instead is the action conditionally averaged over the rollout at fixed patch-mean (u′ , w′ ) (Fig. 5). At each point of that conditional map the action keeps a spread the patch-mean coordinates cannot set, and that residual variance is what the recurrence, the spatial ring and the hidden state supply beyond the local instantaneous state. The effect on the near-wall flow is read off the detection-plane joint statistics (Fig. 6): against the broad, switch-driven distribution left by vanilla DRL, GRUMARL reshapes the (u′ , w′ ) density and its Reynolds-shear-weighted form in the second and fourth quadrants where the sweep–ejection cycle carries the momentum that sets the friction. The near-wall Reynolds shear stress confirms that the recurrent controller acts on the flow rather than on the bookkeeping (Fig. 7). Against the non-actuated reference, GRU-MARL suppresses the −⟨u′ w′ ⟩+ that carries momentum to the wall, close to what opposition achieves, while the saturating vanilla-DRL switch leaves it near the uncontrolled level and reorganises the field at the energetic cost the budget has already exposed. The recurrent policy reaches its operating point by a route that opposition does not take: it correlates with the streamwise velocity at the detection plane rather than opposing the wall-normal component, and it does so with no opposition prior in the architecture or the reward (the sensing correlations are reported in
7
u0 (x, y; z + ≃ 15)
400
vanilla DRL
0
0.4
0.05
0.2
400 0.00
0.0
200 0
GRU-MARL
0.10
−0.05
−0.2
−0.10
−0.4
u0 (x, y; z + ≃ 15)
200
ww
stripes
ww (x, y; z + ≃ 0.2)
400 200 0 0
250
500
750
1000
1250
1500
1750
0
x+
250
500
750
1000
1250
1500
1750
x+
Figure 4: Instantaneous wall actuation ww (x, y) (left column of each row) and detection-plane streamwise fluctuation u′ (right column) on the large box, in viscous units, for stripes, vanilla DRL and GRU-MARL. Stripes is the imposed open-loop forcing, fixed by construction. Vanilla DRL, though closed-loop, has settled into a near-stationary standing wave of its own, a learnt reward-hacking artefact rather than control of the flow, while GRU-MARL places its actuation in register with the instantaneous streaks. Shared colour scale per column. Supplementary Table S1, and the field correlation maps in Supplementary Fig. S1). A practical property follows from the parameter sharing. The same weights define the policy at every patch, so a network trained on a minimal flow unit of 64 × 64 patches transfers without change to the 256 × 256 evaluation grid of a box sixteen times larger in wall-parallel area; every GRU-MARL number reported here is from that transferred policy.
The credit-assignment and timescale failures are architectural Two of the three failures owe nothing to the fluid. The credit-assignment collapse and the timescale collapse each reappear in the fluid-dynamics-free surrogates introduced alongside them, which carry no flow yet reproduce the saturation and the dead actuation seen at the wall, marking them as properties of the control architecture rather than of turbulence (Supplementary Figs. S2 and S3). The energy budget is different in kind. The wall power the drag proxy omits is a physical quantity with no analogue in a system that carries no actuation cost, so the third fault is a property of the flow, one the channel exposes rather than abstracts away. Construction, training and numerical checks for both surrogates are given in the Supplementary Information.
Discussion Discussion Reducing the drag exerted by a turbulent flow on a surface has a clear physical interpretation: a reduction in the total rate of energy dissipation in the system. In much of the recent reinforcement-learning literature, however, this objective has been replaced by a more readily measurable proxy, namely the reduction in pumping power inferred from the streamwise pressure gradient. While convenient, this quantity captures only one component of the overall energy balance and does not account for the work performed by the actuation itself on the flow (Guastoni 8
σa (u0 , w0 )/uτ
ha | u0 , w0 i/uτ 0.3
4
4
0.25
0.2 2
0.1
0
0 w0 /wrms
0 w0 /wrms
2
0.0 −0.1
−2
0.20 0.15
0
0.10 −2
0.05
−0.2 −4
−4
−0.3 −2
0
2
0.00 −2
u0 /u0rms
0
2
u0 /u0rms
Figure 5: GRU-MARL conditional policy response. Unlike the memoryless vanilla-DRL actor of Fig. 3, whose entire policy is a surface on the two sensed scalars, GRU-MARL has no twodimensional input; the panels show the action conditionally averaged over the rollout at fixed patch-mean (u′ , w′ ) (left) and its spread (right). The spread is the share of the action the local two scalars cannot account for, supplied by the recurrence, the spatial ring and the hidden state. et al., 2023; Wälchli et al., 2024). The distinction is not new. Classical flow-control studies have long emphasised that any meaningful assessment of control performance must consider the net energy balance. For example, Fukagata et al. (2009) derived bounds on the net power savings achievable by actuated wall-bounded flows, while Marusic et al. (2021) argued that energy efficiency, rather than nominal drag reduction alone, provides the appropriate criterion for evaluating the practical value of a control strategy. A scheme that reduces the pressure gradient may therefore still be energetically unfavourable if the control action requires a comparable or larger energy input. The mismatch between the physical objective and the quantity used to evaluate it lies at the heart of the three shortcomings discussed in this paper. The first fault is structural. The mass-conservation constraint that makes blowing and suction admissible in a closed channel couples the actions of all agents through a mean-subtraction projection. When this projection is applied after the actor network rather than embedded within it, the policy-gradient update is computed for a control field different from the one actually imposed on the flow. This distorts the credit assignment mechanism on which learning relies. The remedy is straightforward and incurs no additional modelling cost: by implementing the projection as the final layer of the actor, its constant Jacobian becomes part of the computational graph, ensuring that gradients are taken with respect to the physically admissible actuation field. The second fault concerns observability. The near-wall regeneration cycle evolves over time scales of the order of one hundred viscous units, substantially longer than the instantaneous flow snapshots presented to a memoryless policy. A static mapping from a single observation cannot reliably infer the phase of such a slowly evolving process. As a consequence, the optimisation tends to converge towards a saturated two-level actuation strategy, the familiar bang-bang regime that often emerges in continuous-control problems when rapid switching is left unpenalised (Seyde et al., 2021). To address this limitation, we introduce two complementary modifications. First, the actuation interval is selected to lie between the fast decorrelation time of the near-wall turbulence and the slower turnover time of the streak-regeneration cycle. Successive observations therefore con9
Q1: 22%
1
3
1.035
2
0.828
0 0.621 −1
−4
Q3: 22%
−4
Q4: 29%
−2
0
2
4
0.2637 0.2198
1
0.1758
0 0.1319 0.0879
−2
0.207
−3
0.0440
−3
0.000
−4
u0 /urms
Q3: 20%
−4
a vanilla DRL: f (u′ , w′ ) ×10−1
Q1: 22%
4.670
2
3.114
1
1.557
0
0.000
−1
−1.557
−2
−3.114
−4
Q3: 22%
−4
0
2
2
0.0000
4
u0 /urms
Q2: 30%
×10−1
Q1: 16%
1.085
2
0.723
1
0.362
0
0.000
−1
−0.362
−2
−0.723
−3
−4.670
Q4: 29%
−2
0
3
w0 /wrms
w0 /wrms
3
−3
−2
Pre-mult. joint PDF at z + ≃ 13.6
4
u0 w0 f (u0 , w0 )
Q2: 27%
Q4: 34%
b GRU-MARL: f (u′ , w′ )
Pre-mult. joint PDF at z + ≃ 13.8
4
Q1: 16%
−1
0.414
−2
Q2: 30%
−4
4
u0 /urms
c vanilla DRL: u′ w′ f (u′ , w′ )
Q3: 20%
−4
−1.085
Q4: 34%
−2
0
2
u0 w0 f (u0 , w0 )
w0 /wrms
2
1.242
w0 /wrms
3
joint PDF
Q2: 27%
Joint PDF (u0 , w0 ) at z + ≃ 13.6
4
joint PDF
Joint PDF (u0 , w0 ) at z + ≃ 13.8
4
4
u0 /urms
d GRU-MARL: u′ w′ f (u′ , w′ )
Figure 6: Detection-plane joint statistics of the streamwise and wall-normal fluctuations: the raw density f (u′ , w′ ) (top row) and its Reynolds-shear-weighted form u′ w′ f (u′ , w′ ) (bottom row), for vanilla DRL (left) and GRU-MARL (right). All four are computed over the statistically steady window, with the initial transient discarded so that no start-up effect enters the statistics. GRUMARL reshapes the second- and fourth-quadrant sweep–ejection events that the memoryless switch leaves broad.
10
0.0030
non-actuated opposition vanilla DRL GRU-MARL
0.0025
−hu0 w0 i+
0.0020
0.0015
0.0010
0.0005
0.0000 0
25
50
75
100
125
150
175
z+
Figure 7: Near-wall Reynolds shear stress −⟨u′ w′ ⟩+ in viscous units referenced to the uncontrolled friction velocity, pooled over the snapshot ensemble, for the non-actuated channel, opposition, vanilla DRL and GRU-MARL. GRU-MARL suppresses the momentum-carrying shear stress close to opposition, while the memoryless switch leaves it near the non-actuated level. tain meaningful dynamical information while still allowing multiple control updates within the lifetime of a coherent structure. Second, the policy is augmented with a recurrent architecture and an enlarged sensing stencil, providing the temporal memory and spatial context required to track streak-scale dynamics rather than reacting to the fastest fluctuations. The third fault concerns the way energetic performance is evaluated and reported. The standard accounting framework assigns an actuation cost through a kinetic-energy-flux term that scales with the cube of the wall-normal velocity amplitude. At the amplitudes typically used in turbulent-flow control, this contribution is often two to three orders of magnitude smaller than the friction coefficient itself, causing the reported net-energy saving to become numerically indistinguishable from the drag reduction. More recent formulations replace this cubic estimate with a quadratic kinematic cost, sometimes applied only to blowing while effectively treating suction as energetically free (Hasegawa and Kasagi, 2011). Although different in form, both approaches share the same limitation: neither accounts for the pressure-covariance term through which a bounded-amplitude actuation exchanges mechanical power with the flow. This omission becomes particularly important for reinforcement-learning controllers. Neither metric imposes a meaningful energetic penalty on bang-bang saturation beyond a largely amplitudeindependent cost. As a result, a controller that spends most of its time at the actuation bounds can incur essentially the same reported energy expenditure as a smoother policy, despite performing substantially more work on the fluid. In the extreme case, a policy may converge to a persistent standing-wave actuation pattern that continuously injects energy into the flow while still appearing favourable under conventional drag-reduction metrics. For this reason, the physically relevant quantities are the true wall-power input, represented by the pressure-covariance term in (2), and the corresponding net-energy balance. These are the measures advocated in the classical flow-control literature (Fukagata et al., 2009; Marusic et al., 2021) and should likewise form the basis for evaluating learned controllers. Without them, a controller that actively pumps energy into the flow can appear indistinguishable from one that genuinely suppresses turbulent activity. The stripe and vanilla-DRL results reported in Table 1 provide a clear illustration of this ambiguity. These considerations also suggest caution when comparing controllers trained and evalu-
11
ated under different assumptions regarding domain size and energetic accounting. A controller evaluated in a minimal flow unit, where relaminarisation can substantially inflate the reported drag reduction, and whose actuation cost is estimated using a metric insensitive to bang-bang saturation, will naturally produce a larger headline figure than a controller assessed in a large computational domain using the true wall-power expenditure. The difference between these figures does not necessarily reflect a difference in control effectiveness, but may instead arise from the underlying evaluation methodology. Viewed in this context, the 17% drag reduction achieved by GRU-MARL should be interpreted as a conservative estimate obtained under more stringent evaluation conditions. The controller is assessed in a large domain, its energetic performance is measured using the physically relevant wall-power contribution, and the reward function is designed to avoid known degeneracies associated with purely drag-based objectives. We therefore view this result not as a final benchmark, but as a reference point for future comparisons performed under the same physical and energetic criteria. Several questions remain open. How the corrected policy reorganises the near-wall flow, and what aspects of the regeneration cycle it exploits beyond the instantaneous local state, is the subject of a companion study aimed at directly analysing the information content of the learned policy. Such an analysis may also provide new insight into the dynamics of the wall cycle itself. The present architecture is not fully optimised: the sensing stencil, recurrent memory and reward weighting all offer opportunities for further improvement while maintaining the same energetic standard. A natural next step is to determine whether the same corrections remain effective at higher Reynolds numbers, where the effectiveness of opposition-based control is known to diminish (Zhou et al., 2025).
Methods Flow configuration The flow is an incompressible turbulent channel at Re τ ≃ 180, the lowest of the standard channelflow reference simulations (Moser et al., 1999) that reach Re τ ≃ 2000 (Hoyas and Jiménez, 2006), in a half-channel of height H, periodic in the wall-parallel directions, with an actuated no-slip wall at the bottom and a symmetry condition at the top, driven in constant-flow-rate mode so that the streamwise pressure gradient ⟨∂x p⟩ (t) adjusts to hold the bulk velocity at Ub = 1. The uncontrolled mean gradient is ⟨∂x p⟩0 ≈ −4.10×10−3 in bulk-velocity and half-height units. Eval+ + uation runs use a box of (Lx , Ly , H) = (10.68, 3.2, 1), that is (L+ x , Ly , H ) ≃ (1922, 576, 180), + + on a 256 × 256 × 100 mesh with near-wall resolution (∆x , ∆y ) ≃ (7.5, 2.25) and a wall-normal grid clustered at the wall by a hyperbolic-tangent stretching that places the first point inside z + ≲ 1. Actuation is a Dirichlet condition on the wall-normal velocity, w(x, y, 0, t) = ww (x, y, t), with the wall-parallel components no-slip and the zero-net-mass constraint (1) imposed at every actuation step. The time step is set by a Courant limit of order one half and is of order 10−3 bulk time units; each run is advanced for a spin-up of 40 bulk time units before any statistics are taken, and the energy-budget terms of Table 1 are then averaged over a window of order one hundred bulk time units in the statistically stationary regime. The same uncontrolled initial condition seeds every controlled run, so the comparison is at matched flow history.
Flow solver The Navier–Stokes equations are integrated with CaNS (Costa, 2018), a second-order finitedifference solver on a staggered Cartesian grid that advances the momentum equations with a low-storage third-order Runge–Kutta scheme and enforces incompressibility by a pressurecorrection step. The pressure Poisson equation is solved with an FFT-based direct method in the two periodic directions and a tridiagonal solve in the wall-normal direction. Constant-flow-rate 12
operation is imposed by adjusting the bulk pressure gradient each substep. The simulations are run on GPUs using the accelerated CaNS implementation (Costa et al., 2021) with the adaptive pencil decomposition of cuDecomp (Romero et al., 2022) for multi-GPU communication. The wall actuation enters as a time-dependent Dirichlet boundary condition updated at the actuation cadence.
Reinforcement-learning framework Control is provided through a multi-agent reinforcement-learning framework that couples the flow solver to a PyTorch (Paszke et al., 2019) policy and a PettingZoo (Terry et al., 2021) parallel multi-agent environment (Fig. 1). The environment advances CaNS as a subprocess for one actuation interval, extracts the per-patch observations from the detection-plane fields, passes them to the shared policy, and writes the resulting wall-velocity field back as the boundary condition for the next interval; observations, actions and rewards are exchanged through the solver’s actuation interface. Training follows the centralised-training, decentralised-execution scheme established for multi-agent flow control by Vignon et al. (2023a) and Guastoni et al. (2023): a single critic with access to the joint state and action supplies the gradient, while at deployment each patch acts on its local observation alone. Reinforcement learning reached flow control through wake stabilisation (Rabault et al., 2019) and was scaled by training across parallel environments (Rabault and Kuhnle, 2019), then carried to wings (Vinuesa et al., 2022), thermal convection (Beintema et al., 2020) and turbulence closure (Novati et al., 2021; Bae and Koumoutsakos, 2022), a line of work reviewed in Brunton et al. (2020). In the channel it has been pursued with an early network trained on a suboptimal target (Lee et al., 1997), singleagent convolutional controllers (Han and Huang, 2020; Lee et al., 2023), and studies of how the Reynolds number changes the physics the policy exploits (Varela et al., 2022), before the multi-agent formulation adopted here. The framework is available at the repository in the Code availability statement. Every wall patch runs the same actor πθ , a convolutional encoder feeding a single-layer GRU of width dh = 64 and a tanh head, CNN GRU head t t ⊤ t oij −−−−→ zij −−−−→ (ht+1 (10) ij , gij ) −−−→ aij = tanh W LN(gij ) + b , where the per-patch hidden state htij carries memory across actuation steps and is what lets the policy follow the slow near-wall cycle. The zero-mean projection (8) is the actor’s last layer, so its Jacobian (9) is differentiated through during training; the resulting per-patch field is Gaussiansmoothed and re-centred before it reaches the wall, which holds the net mass flux at zero and prevents collapse to the discontinuous stripe pattern. The centralised critic Qϕ reads the joint state and action and is used during training only, while at deployment each patch runs the actor on its local observation ring alone. The policy is trained off-policy by multi-agent deterministic policy gradient with the centralised critic (Lillicrap et al., 2016), under an energy-aware reward that offsets a drag-reduction term with penalties on the action’s temporal and spatial variation, its zero-mean residual and its magnitude; the magnitude and smoothness penalties are the practical form of the energy-aware objective, holding the policy off the saturation bound that the wall power penalises. Training + is on a minimal flow unit (Jiménez and Moin, 1991) of (L+ x , Ly ) ≃ (481, 144) for up to 500 episodes of 1800 actuation steps each, after which the shared weights are deployed on the large evaluation box without retraining. The network sizes, the learning update and the full set of hyperparameters are given in the Supplementary Information (Supplementary Table S2).
Wall units, observation and patches Throughout, the superscript + denotes normalisation in viscous (wall) units: lengths by ν/uτ and times by the viscous time ν/u2τ , with friction velocity uτ ≃ 0.064 and ν = 1/2870 in bulk13
velocity and half-height units, so that Re τ = uτ H/ν ≃ 180. One viscous time unit is therefore ν/u2τ ≃ 0.085 bulk time units. The wall is partitioned into square patches of p = 4 grid cells, so the 64 × 64 training grid carries 16 × 16 = 256 patches and the 256 × 256 evaluation grid carries 64 × 64 = 4096; the same shared weights are used in both. Each patch observes a 3 × 3 ring of neighbouring patches with periodic closure, a three-channel 12 × 12 image of the plane-centred streamwise and wall-normal velocities at the detection plane zd+ ≃ 14 together with its own previous action. The two velocity channels are plane-mean-subtracted and scaled by ωmax = uτ so that the observation has an O(1) + range. The patch sides are ∆x+ p ≃ 30 and ∆yp ≃ 9, both below the spanwise streak spacing + λy ≃ 100. What the policy senses, and where it is sensed, shapes the law it learns (Paris et al., 2021; Vinuesa et al., 2024), which is the rationale for the widened detection stencil used here.
Actuation time scale The interval between two wall updates is a design parameter, and we fix it from the time scale of the structures the controller must act on rather than from the solver step. The near-wall streaks and the regeneration cycle that maintains them, fed by the sweep and ejection events that carry Reynolds shear stress across the buffer layer (Wallace et al., 1972), evolve over O(100) viscous units (Jiménez and Pinelli, 1999; Hamilton et al., 1995). An actuation interval close to a single viscous unit, ∆t+ a ≃ 1, forces the policy to act far below that scale, on near-uncorrelated successive snapshots; with no memory to integrate, a policy driven that fast collapses into the high-frequency switching that the vanilla-DRL controller exhibits. We therefore set ∆t+ a ≃ 5, equivalently one update every ≃ 0.43 bulk time units: short enough that several updates fall within one streak lifetime, yet coarse enough that, paired with the recurrent memory, the policy follows the streak-scale evolution instead of chasing the fastest fluctuations, which is what gives GRU-MARL its purchase on the near-wall cycle. The same effect can be isolated away from the turbulence in a controlled two-timescale model, detailed in the Supplementary Information (Supplementary Fig. S3): with a clean separation between a fast and a slow scale, an actuation interval at the fast scale collapses the policy to saturated two-level switching and one above the slow scale to near-zero actuation, leaving only an intermediate window admissible. The interval is held fixed across training and evaluation.
Energy budget and the definition of wall power The pumping power and wall power are defined in (2). The viscous wall-traction work vanishes because continuity at the no-slip wall forces ∂z w|wall = 0 pointwise, so the only surviving surfacework term is the pressure covariance. The closed balance is (3), the drag reduction is (4), and the net gain is (5). We report Ww rather than the kinetic-energy-flux proxy (6) because the latter is a function of the actuation amplitude only and therefore cannot detect power delivered through the correlation of the wall velocity with the wall pressure; a controller can keep Win small by bounding its output while Ww remains large. The streamwise momentum balance ⟨τw ⟩ = |⟨∂x p⟩|H holds independently of the actuation and is used as a per-run solver check; the same control volume underlies the identity that splits the skin friction into the contributions a controller can target (Fukagata et al., 2002). Each controller is run for of order one hundred bulk-convective times past a stationary transient, and every term is measured directly over that window.
Reference and degenerate controllers Opposition applies ww = −(wd − wd ), with wd the wall-normal velocity at the detection plane zd+ ≃ 15 and the plane-mean subtracted to meet (1); the rule is saturation-free, its amplitude set by the sensed signal. It suppresses drag by raising a virtual wall that screens the surface
14
from the near-wall vortices (Hammond et al., 1998) and carries across channel and boundarylayer geometries at a comparable rate (Stroh et al., 2015). The open-loop stripe controller imposes the fixed streamwise square wave (7) at the saturation amplitude w⋆ , a steady-pattern counterpart of the open-loop travelling-wave and blowing-and-suction schemes that reach large nominal drag reductions (Quadrio et al., 2009; Gatti and Quadrio, 2016; Kametani and Fukagata, 2011). The vanilla-DRL controller is a memoryless parameter-shared multi-agent policy that maps the pointwise detection-plane pair to a scalar action a = tanh fθ (u′ , w′ ) through a small network, trained on the drag-reduction reward without the projection layer, recurrence or energy term that define the corrected controller; its learnt surface is reproduced by an asymmetric tanh on a1 u′ + a2 w′ with (a1 , a2 ) ≃ (3.6, −1.6) to R2 ≃ 0.999.
Data and code availability The modified CaNS flow solver, the multi-agent reinforcement-learning training and evaluation framework, and the trained GRU-MARL policy are available at https://github.com/ gmcavallazzi/CaNS_GRU-MARL, so the controller reported here can be re-run and evaluated.
15
Supplementary Information This supplement reports the actuation–velocity coupling that supports the main-text claim that GRU-MARL operates in a streamwise-sensing regime distinct from the wall-normal opposition law, together with two fluid-dynamics-free models that reproduce the credit-assignment and actuation-interval pathologies from first principles, and it collects the network architecture and the full training hyperparameters of the controller. The flow quantities are evaluated over the statistically stationary window of the per-step wall and detection-plane fields, with the spin-up transient discarded.
Actuation–velocity coupling Table S1 reports the zero-lag Pearson correlations between the wall-normal wall velocity ww and the detection-plane fluctuations at zd+ ≃ 15. Opposition is the near-ideal wall-normal opposer (ρww ,w ≃ −0.95); vanilla DRL locks onto w with the opposite sign; GRU-MARL couples instead to the streamwise velocity, with essentially no linear coupling to w. controller
ρww ,u
ρww ,w
opposition
+0.46
vanilla DRL
−0.08
−0.95
GRU-MARL
+0.47
+0.96 +0.05
Table S1: Zero-lag Pearson correlations between the wall actuation and the detection-plane velocity fluctuations at zd+ ≃ 15. Bold marks the dominant linear coupling of each controller. Opposition opposes w; vanilla DRL follows w with the opposite sign; GRU-MARL is a u-sensor. The full shift dependence of the coupling is shown in Fig. S1: the spatial cross-correlation maps Cww ,u (∆x+ , ∆y + ) and Cww ,w (∆x+ , ∆y + ) between the actuation and the detection-plane velocities. The GRU-MARL maps peak on the streamwise channel, the opposition maps on the wall-normal channel, and the vanilla-DRL maps carry the near-stationary imprint of its standing wave.
A minimal model of the credit-assignment pathology The credit-assignment pathology can be reproduced in a system with no fluid dynamics. We set up N = 32 agents, each emitting a bounded scalar ai = tanh(·) from a single shared policy, a small multilayer perceptron that reads a noisy local observation of a per-agent demand ti . A 1 P ′ conserved resource forces the joint action onto its zero-mean subspace, a = a − N j aj , before the quadratic reward r = −⟨(a′ − t)2 ⟩ is returned. The demand is redrawn every step from a random latent context and carries a persistent mean m0 , which stands in for a net wall actuation the conserved resource cannot supply. Training is by stochastic gradient descent (Adam) on these random demands, with noisy observations, so the learning curves fluctuate and the optimum is approached rather than imposed. The two runs differ in one respect only, the backward pass through the projection: with the projection inside the actor the gradient carries its Jacobian δij − 1/N , while with the projection applied after the actor the Jacobian is dropped, as it is when the zero-mean constraint is enforced outside the network. Figure S2 reports the outcome. With the Jacobian retained, the policy learns down to the reward floor −⟨t̄2 ⟩ and no agent saturates (panel a, blue; panel b). That floor is set by the conserved resource itself: a zero-mean action cannot supply the mean of the demand, so a residual always remains. With the Jacobian dropped, the same network and data settle well above the
16
floor, most agents pinned against their output bound (panel a, orange; panel b). The dropped common-mode term acts as a steady forcing that the bounded actor turns into saturation, the same failure the wall controller shows but with no fluid present. Panel (c) sweeps the mismatch m0 : the inside policy stays on the floor across the range, while the penalty from the dropped Jacobian grows with m0 and closes as m0 → 0. The collapse is thus a property of the shared conserved resource and not of any dynamics, and putting the projection inside the differentiated policy removes it.
A minimal two-timescale benchmark The actuation interval can be studied away from the turbulence in a controlled multiscale setting. We use a two-scale Lorenz–96 system (Lorenz, 1996), a standard surrogate for a flow with separated time scales: a ring of slow variables Xk , each coupled to a bath of fast variables Yj,k that relax on a time scale 1/c with c = 10, so the fast field decorrelates roughly every 0.1 time units while the slow variables keep memory for order one. Control enters the fast equation as a per-site forcing under the same zero-mean projection across sites that the wall actuation obeys, and the objective is to lower the slow-variable energy. The only quantity varied between the runs of Fig. S3 is the actuation interval ∆ta . The two failure modes and the working window appear cleanly. When ∆ta is set near the fast decorrelation time, successive observations are essentially uncorrelated and the policy, with nothing coherent to integrate, collapses to a saturated two-level forcing that switches sign at every step (Fig. S3, top), the same degeneracy the vanilla-DRL controller shows at the wall. When ∆ta is set far above the fast time, the observation is stale by the time the action is applied and the policy collapses to near-zero actuation (bottom). Only an intermediate cadence, short relative to the slow turnover yet long relative to the fast decorrelation, produces a structured low-amplitude forcing that lowers the slow-variable energy (middle). The channel controller’s choice ∆t+ a ≃ 5, between the O(1) viscous fast scale and the O(100) streak lifetime, is the wall-turbulence counterpart of that intermediate band.
GRU-MARL architecture and training details The shared actor reads each patch’s three-channel 3×3 ring as an image oij ∈ R3×12×12 and maps it to a scalar action. A two-stage convolutional encoder, each stage Conv(3 × 3)/GroupNorm(4 groups)/ReLU/MaxPool(2) with [16, 32] channels, reduces the ring to a 32 × 3 × 3 map; a linear projection to dimension 64 feeds a single-layer GRU of width 64 that carries the per-patch hidden state; and a layer-norm with a tanh head, initialised small so the policy begins near zero actuation, produces the bounded scalar atij ∈ [−1, 1]. The shared actor holds about 4.9 × 104 parameters. The zero-mean projection is the actor’s last layer; the constant-per-patch field At it returns is smoothed and re-centred, Ãt = Gσ ∗ At ,
Ãt ← Ãt − Ãt ,
ww = w⋆ Ãt ,
(S1)
with Gσ a periodic Gaussian of width σ = 1.5 grid points, so the field sent to the solver carries exactly zero net mass and does not collapse to the discontinuous stripe pattern. The centralised critic Qϕ acts on the joint field. The per-patch rings are reassembled into the three-channel detection-plane state and concatenated with the current action lifted to a constant-per-patch field; the resulting four-channel stack passes through a three-stage convolutional encoder ([32, 64, 32] channels, Conv/GroupNorm/ReLU/MaxPool) and a two-layer perceptron ([256, 128] with layer-norm and ReLU) to the scalar value Qϕ (s, a). On the training grid it holds about 6 × 105 parameters and is discarded at deployment, where each patch runs the actor on its local ring alone. 17
Learning is by multi-agent deterministic policy gradient (Lillicrap et al., 2016), ∇θ J = E(s,a)∼D ∇a Qϕ (s, a) a=π (s) ∇θ πθ (s) , θ
(S2)
with the differentiable projection’s constant Jacobian carried in ∇θ πθ , and the centralised critic trained on the temporal-difference residual LQ = E (Qϕ (s, a) − r − γ Qϕ′ (s′ , πθ′ (s′ )))2 . (S3) The reward rt = g rDR − λ1 Rtemp − λ2 Rspat − λ3 Rzero − λ4 Rmag
(S4)
combines a piecewise-linear drag term with penalties on the action time-variation ∥at − at−1 ∥2 , the in-plane total variation of the wall field, the residual of the zero-mean constraint before renormalisation, and the action magnitude ∥at ∥2 . Both networks are trained off-policy from a replay buffer with target copies updated by a slow Polyak average; exploration adds bounded Gaussian noise to the actions ahead of the projection layer, and because the recurrent core carries state the transitions are replayed as short sequences so the hidden state is rolled rather than reset at each update. Optimisation uses Adam (Kingma and Ba, 2014). Table S2 lists the full set.
References D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. H. J. Bae and P. Koumoutsakos. Scientific multi-agent reinforcement learning for wall-modelled turbulent flows. Nature Communications, 13(1):1–9, 2022. G. Beintema, A. Corbetta, L. Biferale, and F. Toschi. Controlling rayleigh–bénard convection via reinforcement learning. Journal of Turbulence, 21(9-10):585–605, 2020. T. R. Bewley, P. Moin, and R. Temam. DNS-based predictive control of turbulence: an optimal benchmark for feedback algorithms. Journal of Fluid Mechanics, 447:179–225, 2001. S. L. Brunton, B. R. Noack, and P. Koumoutsakos. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52:477–508, 2020. G. M. Cavallazzi, L. Guastoni, R. Vinuesa, and A. Pinelli. Deep reinforcement learning for the management of the wall regeneration cycle in wall-bounded turbulent flows. Flow, Turbulence and Combustion, 115:1291–1317, 2025. doi: 10.1007/s10494-024-00609-4. H. Choi, P. Moin, and J. Kim. Active turbulence control for drag reduction in wall-bounded flows. Journal of Fluid Mechanics, 262:75–110, 1994. P. Costa. A FFT-based finite-difference solver for massively-parallel direct numerical simulations of turbulent flows. Computers & Mathematics with Applications, 76(8):1853–1862, 2018. P. Costa, E. Phillips, L. Brandt, and M. Fatica. GPU acceleration of CaNS for massivelyparallel direct numerical simulations of canonical fluid flows. Computers & Mathematics with Applications, 81:502–511, 2021. J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
18
K. Fukagata, K. Iwamoto, and N. Kasagi. Contribution of Reynolds stress distribution to the skin friction in wall-bounded flows. Physics of Fluids, 14(11):L73–L76, 2002. K. Fukagata, K. Sugiyama, and N. Kasagi. On the lower bound of net driving power in controlled duct flows. Physica D, 238(13):1082–1086, 2009. P. Garnier, J. Viquerat, J. Rabault, A. Larcher, A. Kuhnle, and E. Hachem. A review on deep reinforcement learning for fluid mechanics. Computers & Fluids, 225:104973, 2021. doi: 10.1016/j.compfluid.2021.104973. D. Gatti and M. Quadrio. Reynolds-number dependence of turbulent skin-friction drag reduction induced by spanwise forcing. Journal of Fluid Mechanics, 802:553–582, 2016. L. Guastoni, J. Rabault, P. Schlatter, H. Azizpour, and R. Vinuesa. Deep reinforcement learning for turbulent drag reduction in channel flows. The European Physical Journal E, 46(4):27, 2023. J. M. Hamilton, J. Kim, and F. Waleffe. Regeneration mechanisms of near-wall turbulence structures. Journal of Fluid Mechanics, 287:317–348, 1995. E. P. Hammond, T. R. Bewley, and P. Moin. Observed mechanisms for turbulence attenuation and enhancement in opposition-controlled wall-bounded flows. Physics of Fluids, 10(9):2421– 2423, 1998. B.-Z. Han and W.-X. Huang. Active control for drag reduction of turbulent channel flow based on convolutional neural networks. Physics of Fluids, 32(9):095108, 2020. Y. Hasegawa and N. Kasagi. Dissimilar control of momentum and heat transfer in a fully developed turbulent channel flow. Journal of Fluid Mechanics, 683:57–93, 2011. M. Hausknecht and P. Stone. Deep recurrent Q-learning for partially observable MDPs. In AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents, 2015. S. Hoyas and J. Jiménez. Scaling of the velocity fluctuations in turbulent channels up to Reτ = 2003. Physics of Fluids, 18:011702, 2006. J. Jiménez and P. Moin. The minimal flow unit in near-wall turbulence. Journal of Fluid Mechanics, 225:213–240, 1991. J. Jiménez and A. Pinelli. The autonomous cycle of near-wall turbulence. Journal of Fluid Mechanics, 389:335–359, 1999. L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1–2):99–134, 1998. Y. Kametani and K. Fukagata. Direct numerical simulation of spatially developing turbulent boundary layers with uniform blowing or suction. Journal of Fluid Mechanics, 681:154–172, 2011. Y. Kametani, K. Fukagata, R. Örlü, and P. Schlatter. Effect of uniform blowing/suction in a turbulent boundary layer at moderate Reynolds number. International Journal of Heat and Fluid Flow, 55:132–142, 2015. D. P. Kingma and J. Ba. arXiv:1412.6980, 2014.
Adam: a method for stochastic optimization.
arXiv preprint
S. J. Kline, W. C. Reynolds, F. A. Schraub, and P. W. Runstadler. The structure of turbulent boundary layers. Journal of Fluid Mechanics, 30(4):741–773, 1967. 19
C. Lee, J. Kim, D. Babcock, and R. Goodman. Application of neural networks to turbulence control for drag reduction. Physics of Fluids, 9(6):1740–1747, 1997. C. Lee, J. Kim, and H. Choi. Suboptimal control of turbulent channel flow for drag reduction. Journal of Fluid Mechanics, 358:245–258, 1998. T. Lee, J. Kim, and C. Lee. Turbulence control for drag reduction through deep reinforcement learning. Physical Review Fluids, 8(2):024604, 2023. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016. E. N. Lorenz. Predictability: a problem partly solved. In Proc. Seminar on Predictability, volume 1, pages 1–18. ECMWF, Reading, UK, 1996. I. Marusic, D. Chandran, A. Rouhi, M. K. Fu, D. Wine, B. Holloway, D. Chung, and A. J. Smits. An energy-efficient pathway to turbulent drag reduction. Nature Communications, 12:5805, 2021. R. D. Moser, J. Kim, and N. N. Mansour. Direct numerical simulation of turbulent channel flow up to Reτ = 590. Physics of Fluids, 11(4):943–945, 1999. G. Novati, H. L. de Laroussilhe, and P. Koumoutsakos. Automating turbulence modeling by multi-agent reinforcement learning. Nature Machine Intelligence, 3:87–96, 2021. R. Paris, S. Beneddine, and J. Dandois. Robust flow control and optimal sensor placement using deep reinforcement learning. Journal of Fluid Mechanics, 913, 2021. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, 2019. M. Quadrio, P. Ricco, and C. Viotti. Streamwise-travelling waves of spanwise wall velocity for turbulent drag reduction. Journal of Fluid Mechanics, 627:161–178, 2009. J. Rabault and A. Kuhnle. Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Physics of Fluids, 31(9):094105, 2019. J. Rabault, M. Kuchta, A. Jensen, U. Réglade, and N. Cerardi. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. Journal of Fluid Mechanics, 865:281–302, 2019. J. Romero, P. Costa, and M. Fatica. Distributed-memory simulations of turbulent flows on modern GPU systems using an adaptive pencil decomposition library. In Proceedings of the Platform for Advanced Scientific Computing Conference (PASC ’22), 2022. T. Seyde, I. Gilitschenski, W. Schwarting, B. Stellato, M. Riedmiller, M. Wulfmeier, and D. Rus. Is bang-bang control all you need? Solving continuous control with Bernoulli policies. In Advances in Neural Information Processing Systems, volume 34, pages 27209–27221, 2021. T. Sonoda, Z. Liu, T. Itoh, and Y. Hasegawa. Reinforcement learning of control strategies for reducing skin friction drag in a fully developed turbulent channel flow. Journal of Fluid Mechanics, 960:A30, 2023. doi: 10.1017/jfm.2023.147. P. R. Spalart and J. D. McLean. Drag reduction: enticing turbulence, and then an industry. Philosophical Transactions of the Royal Society A, 369(1940):1556–1569, 2011. 20
A. Stroh, B. Frohnapfel, P. Schlatter, and Y. Hasegawa. A comparison of opposition control in turbulent boundary layer and turbulent channel flow. Physics of Fluids, 27(7):075101, 2015. J. K. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. S. Santos, C. Dieffendahl, C. Horsch, R. Perez-Vicente, N. L. Williams, Y. Lokesh, and P. Ravi. PettingZoo: Gym for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pages 15032–15043, 2021. P. Varela, P. Suárez, F. Alcántara-Ávila, A. Miró, J. Rabault, B. Font, L. M. García-Cuevas, O. Lehmkuhl, and R. Vinuesa. Deep reinforcement learning for flow control exploits different physics for increasing Reynolds number regimes. Actuators, 11(12), 2022. doi: 10.3390/ act11120359. C. Vignon, J. Rabault, J. Vasanth, F. Alcántara-Ávila, M. Mortensen, and R. Vinuesa. Effective control of two-dimensional Rayleigh–Bénard convection: invariant multi-agent reinforcement learning is all you need. Physics of Fluids, 35(6):065146, 2023a. C. Vignon, J. Rabault, and R. Vinuesa. Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions. Physics of Fluids, 35(3):031301, 2023b. R. Vinuesa, O. Lehmkuhl, A. Lozano-Durán, and J. Rabault. Flow control in wings and discovery of novel approaches via deep reinforcement learning. Fluids, 2022. doi: 10.3390/fluids7020062. R. Vinuesa, J. Rabault, H. Azizpour, and L. Guastoni. Influence of the state observation on deep-reinforcement-learning drag-reduction policies in wall-bounded flows. In Proceedings of the 13th International Symposium on Turbulence and Shear Flow Phenomena (TSFP-13), 2024. D. Wälchli, L. Guastoni, R. Vinuesa, and P. Koumoutsakos. Drag reduction in a minimal channel flow with scientific multi-agent reinforcement learning. Journal of Physics: Conference Series, 2753:012024, 2024. F. Waleffe. On a self-sustaining process in shear flows. Physics of Fluids, 9(4):883–900, 1997. J. M. Wallace, H. Eckelmann, and R. S. Brodkey. The wall region in turbulent shear flow. Journal of Fluid Mechanics, 54(1):39–48, 1972. Z. Zhou, M. Zhang, and X. Zhu. Reinforcement-learning-based control of turbulent channel flows at high Reynolds numbers. Journal of Fluid Mechanics, 1006:A12, 2025. doi: 10.1017/jfm. 2025.27.
21
Cww ,u (∆x+ , ∆y + )
Cww ,w (∆x+ , ∆y + )
C ww , u
C ww , w 0.414
200
0.276
0
0.000 −0.138
−100
0.298
0
0.000 −0.298
−100
−0.276
−200
0.595
100
0.138
∆y +
∆y +
100
opposition
0.893 200
−0.595
−200
−0.414 −500
0
∆x+
−0.893
500
−500
C ww , u
0
∆x+
500
C ww , w 0.705
200
0.470
0
0.000 −0.235
−100
0.288
0
0.000 −0.288
−100
−0.470
−200
0.575
100
0.235
∆y +
∆y +
100
vanilla DRL
0.863 200
−0.575
−200
−0.705 −500
0
∆x+
−0.863
500
−500
C ww , u
0
∆x+
500
C ww , w 0.4913
200
0.3275 0.1638
0
0.0000 −0.1638
−100
−0.3275
−200
0.1509
100
∆y +
∆y +
100
GRU-MARL
0.2264 200
0.0755
0
0.0000 −0.0755
−100
−0.1509
−200
−0.4913 −500
0
∆x+
500
−0.2264 −500
0
∆x+
500
Figure S1: Spatial cross-correlation maps between the wall actuation ww and the detection-plane streamwise (left) and wall-normal (right) velocity, in viscous shifts, for opposition, vanilla DRL and GRU-MARL. GRU-MARL couples to u and not to w, the reverse of the wall-normal opposer; vanilla DRL shows the fixed imprint of its standing wave.
22
(a) learning curves
(b) action distribution
−0.20
(c) penalty grows with m0 0.00
inside after
30
floor inside after
eval reward −h(a0 − t)2 i
−0.05
count
projection inside actor projection after actor
−0.24
floor −ht̄2 i
final eval reward
25
−0.22
20 15 10
−0.26
0
500
1000
1500
2000
2500
0
3000
training step
−0.15 −0.20 −0.25 −0.30
5 −0.28
−0.10
−1.0
−0.5
0.0
0.5
−0.35
1.0
0.0
0.1
agent output ai
0.2
0.3
0.4
0.5
conserved-resource mismatch m0
Figure S2: Credit-assignment pathology in a fluid-dynamics-free model. (a) Evaluation reward during training for the projection inside the actor (blue) and applied after it (orange); the dashed line is the irreducible floor −⟨t̄2 ⟩ set by the conserved resource. (b) Distribution of agent outputs ai at convergence: spread for the inside gradient, piled against ±1 for the omitted-Jacobian gradient. (c) Final reward against the conserved-resource mismatch m0 (mean and spread over seeds); the inside policy tracks the floor, the after policy falls increasingly below it as m0 grows.
actuation u(t)
slow-variable energy
6
0.5
4
0.0
2
−0.5
0 0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
−1.0
0.5
4
0.0
2
−0.5
0 5.0
7.5
10.0
12.5
15.0
17.5
20.0
17.5
20.0
40 30 2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
0.0
2
−0.5
0 125
150
175
200
uncontrolled controlled
50 1 kXk2 2
k (action uk )
0.5
100
15.0
t
1.0
75
12.5
uncontrolled controlled
0.0
4
50
10.0
−1.0
6
25
7.5
50
t
0
5.0
t
6
2.5
2.5
1 kXk2 2
k (action uk )
good ∆ta
30 0.0
1.0
slow ∆ta
40
t
0.0
uncontrolled controlled
50 1 kXk2 2
k (action uk )
fast ∆ta
1.0
40
30
−1.0
0
t
25
50
75
100
125
150
175
200
t
Figure S3: Two-scale Lorenz–96 benchmark: actuation u(t) (left) and slow-variable energy (right) for three actuation intervals. A fast-scale cadence collapses to saturated two-level switching (top); a slow-scale cadence collapses to near-zero actuation on a stale observation (bottom); only the intermediate cadence yields a structured, energy-reducing control (middle).
23
Hyperparameter
Value
Observation and action Patch size Observation ring Detection plane Action interval ∆t+ a Saturation amplitude w⋆ Smoothing width σ
4 × 4 grid cells 3 × 3 patches, 3 × 12 × 12 zd+ ≃ 14 5 (≃ 0.43 bulk t.u.) 0.0512 (= 0.8 uτ ) 1.5 grid points
Networks Actor encoder channels GRU hidden size dh Actor parameters Critic conv channels Critic MLP widths Critic parameters
[16, 32] 64 ≈ 4.9 × 104 [32, 64, 32] [256, 128] ≈ 6 × 105
Optimisation (MADDPG) Optimiser Actor / critic learning rate Discount γ Target smoothing τ Mini-batch Sequence length / burn-in Gradient steps / interval Gradient-norm clip Replay buffer
Adam 3 × 10−5 / 5 × 10−4 0.98 2 × 10−3 128 sequences 16 / 4 steps 2 every 20 steps 0.5 15 episodes
Reward and schedule Weights (λ1 , λ2 , λ3 , λ4 ) Reward gain g Exploration noise σ Episodes × length
(0.3, 0.25, 2.0, 0.3) 10 0.15 → 0.01 over 150 ep. ≤ 500 × 1800 steps
Table S2: GRU-MARL architecture and training hyperparameters; quantities in wall units where marked, with uτ ≃ 0.064. Training is on the minimal flow unit (16 × 16 patches) and the same shared weights are deployed on the 64 × 64 evaluation grid.
24