Deep reinforcement learning with spatial and temporal awareness for active boundary control of buoyancy-driven convection

Abstract

Deep reinforcement learning (DRL) applied to thermal convection control consistently produces \textit{degenerate actuation}: wall-temperature policies whose outputs are saturated, pseudo-random, or spatially incoherent. Two compounding deficiencies are responsible: multilayer-perceptron policies that discard spatial flow structure, and memoryless policies that cannot distinguish self-induced flow changes from background evolution. Together they prevent the discovery of physically meaningful control laws even when cell coalescence (the merging of convection rolls into fewer, larger structures), which would reduce $\mathrm{Nu}$, is accessible to boundary actuation. The present framework addresses both causes through four targeted design choices: convolutional policy networks, Gated Recurrent Unit (GRU) memory, off-policy training (TD3/MADDPG), and action-smoothness constraints. A systematic $2\times2$ factorial design isolates the contribution of each component. On Rayleigh--Bénard convection at $\mathrm{Ra}=10{,}000$, all four configurations achieve cell coalescence and reduce $\mathrm{Nu}$ to as low as $1.83$ ($26\%$ below the uncontrolled baseline) in 350 episodes, without the full-field data augmentation required by prior work. Crucially, coalescence is achieved even by the single-agent configuration, demonstrating that the multi-agent formulation is not a prerequisite once the policy architecture is sufficiently expressive. Applied to double-diffusive convection in the salt-finger regime, the framework spontaneously discovers a travelling-wave actuation whose phase speed adapts to the evolving mixing state of the flow, enhancing heat transfer by $19.1\%$ and reducing salinity variance by $21.0\%$.