Machine-learning (ML) parameterizations of subgrid processes (here of turbulence, convection, and radiation) may one day replace conventional parameterizations by emulating high-resolution physics without the cost of explicit simulation. However, their development has been stymied by uncertainty surrounding whether or not improved offline performance translates to improved online performance (i.e., when coupled to a large-scale general circulation model (GCM)). A key barrier has been the limited sampling of the online effects of the ML design decisions and tuning due to the complexity of performing large ensembles of hybrid physics-ML climate simulations. Our work examines the coupled behavior of full-physics ML parameterizations using large ensembles of hybrid simulations, totalling 2,970 in our case. With extensive sampling, we statistically confirm that lowering offline error lowers online error (given certain constraints). However, we also reveal that decisions decreasing online error, like removing dropout, can trade off against hybrid model stability and vice versa. Nevertheless, we are able to identify design decisions that yield unambiguous improvements to offline and online performance, namely incorporating memory and training on multiple climates. We also find that converting moisture input from specific to relative humidity enhances online stability and that using a Mean Absolute Error (MAE) loss breaks the aforementioned offline/online error relationship. By enabling rapid online experimentation at scale, we empirically answer previously unresolved questions regarding subgrid ML parameterization design.

Key Points: 10 • We improve a long-standing stratocumulus (Sc) dim bias in a high-resolution Mul-11 tiscale Modeling Framework. 12 • Incorporating intra-CRM hypervisocity hedges against the numerics of its momen-13 tum solver, reducing entrainment vicinity. 14 • Further adding sedimentation boosts Sc brightness close to observed, opening path 15 to more faithful low cloud feedback analysis. Abstract 17 High-Resolution Multi-scale Modeling Frameworks (HR)-global climate models that 18 embed separate, convection-resolving models with high enough resolution to resolve bound-19 ary layer eddies-have exciting potential for investigating low cloud feedback dynam-20 ics due to reduced parameterization and ability for multidecadal throughput on mod-21 ern computing hardware. However low clouds in past HR have suffered a stubborn prob-22 lem of over-entrainment due to an uncontrolled source of mixing across the marine sub-23 tropical inversion manifesting as stratocumulus dim biases in present-day climate, lim-24 iting their scientific utility. We report new results showing that this over-entrainment 25 can be partly offset by using hyperviscosity and cloud droplet sedimentation. Hypervis-26 cosity damps small-scale momentum fluctuations associated with the formulation of the 27 momentum solver of the embedded LES. By considering the sedimentation process ad-28 jacent to default one-moment microphysics in HR, condensed phase particles can be re-29 moved from the entrainment zone, which further reduces entrainment efficiency. The re-30 sult is an HR that is able to produce more low clouds with a higher liquid water path 31 and a reduced stratocumulus dim bias. Associated improvements in the explicitly sim-32 ulated sub-cloud eddy spectrum are observed. We report these sensitivities in multi-week 33 tests and then explore their operational potential alongside microphysical retuning in 34 decadal simulations at operational 1.5 degree exterior resolution. The result is a new HR 35 having desired improvements in the baseline present-day low cloud climatology, and a 36 reduced global mean bias and root mean squared error of absorbed shortwave radiation. 37 We suggest it should be promising for examining low cloud feedbacks with minimal ap-38 proximation. 39 Plain Language Summary 40 Stratocumulus clouds cover a large fraction of the globe but are very challenging 41 to reproduce in computer simulations of Earth's atmosphere because of their unique com-42 plexity. Previous studies find the model produces too few Stratocumulus clouds as we 43 increase the model resolution, which, in theory, should improve the simulation of impor-44 tant motions for the clouds. This is because the clouds are exposed to more conditions 45 that make them evaporate away. On Earth, stratocumulus clouds reflect a lot of sun-46 light. In the computer model of Earth, too much sunlight reaches the surface because 47 of too few stratocumulus clouds, which makes it warmer. This study tests two methods 48 to thicken Stratocumulus clouds in the computer model Earth. The first method smooths 49 out some winds, which helps reduce the exposure of clouds to the conditions that make 50 them evaporate. The second method moves water droplets in the cloud away from the 51 conditions that would otherwise make them evaporate. In long simulations, combining 52 these methods helps the model produce thicker stratocumulus clouds with more water. 53

We design a new strategy to load-balance high-intensity sub-grid atmospheric physics calculations restricted to a small fraction of a global climate simulation’s domain. We show why the current parallel load balancing infrastructure of CESM and E3SM cannot efficiently handle this scenario at large core counts. As an example, we study an unusual configuration of the E3SM Multiscale Modeling Framework (MMF) that embeds a binary mixture of two separate cloud-resolving model grid structures that is attractive for low cloud feedback studies. Less than a third of the planet uses high-resolution (MMF-HR; sub-km horizontal grid spacing) relative to standard low-resolution (MMF-LR) cloud superparameterization elsewhere. To enable MMF runs with Multi-Domain CRMs, our load balancing theory predicts the most efficient computational scale as a function of the high-intensity work’s relative overhead and its fractional coverage. The scheme successfully maximizes model throughput and minimizes model cost relative to precursor infrastructure, effectively by devoting the vast majority of the processor pool to operate on the few high-intensity (and rate-limiting) HR grid columns. Two examples prove the concept, showing that minor artifacts can be introduced near the HR/LR CRM grid transition boundary on idealized aquaplanets, but are minimal in operationally relevant real-geography settings. As intended, within the high (low) resolution area, our Multi-Domain CRM simulations exhibit cloud fraction and shortwave reflection convergent to standard baseline tests that use globally homogenous MMF-LR and MMF-HR. We suggest this approach can open up a range of creative multi-resolution climate experiments without requiring unduly large allocations of computational resources.