Conclusion
HPC centers now look to Jupyter as a way to make interacting with supercomputing easier, attract new users, and help them solve new science problems. But exposing the unique capabilities and features of supercomputers through Jupyter is a work in progress. HPC stakeholders have engaged the Jupyter community to help put Jupyter in the position of rich supercomputer user interface.
Because of design, mission, and technological trends, supercomputers and HPC centers that run them are still less homogeneous as a group than cloud providers. This means "one size fits all" solutions are sometimes harder to come by. And while providers of supercomputing power want to increase ease of use, they are not interested in homogenizing or concealing specialized capabilities from expert users. Developers working on Jupyter projects that may intersect with HPC especially should avoid making assumptions about HPC center policy (e.g. queue configuration, submit and run limits, privileged access) and seek input from HPC developers on how to generalize those assumptions. As long as Jupyter developers remain committed to extensibility, abstraction, and remaining agnostic about deployment options, developers at HPC centers and their research partners can help fill the gaps.
Likewise, center management should understand the benefits of enabling their staff to contribute to open-source projects when possible, in this case Jupyter. HPC centers can engage with university researchers and application developers as leverage (e.g. the Pangeo project). Users should recognize that the shift to data science in supercomputing places new demands on HPC centers and that change takes time. They can probably best help by engaging the research apparatus at the agency level, explaining their needs for expanded and easier interfaces to supercomputing to policymakers and program managers. This mechanism is what releases resources at the HPC center level to effect change strategically.
At NERSC our next step is to renew focus on supporting Jupyter at new scales and on new hardware. While we do an adequate job of meeting the needs of several hundred users per month, we need to do more to ease the barriers to entry to analytics cluster software like Dask, Spark, and parallelized visualization tools like yt. We look forward to this work on Perlmutter beginning this year.
...
Scaling Compute: We support the Dask distributed computing framework as the recommended way to scale tasks through Jupyter. We do this through a combination of worked example scripts and notebooks, as well as Dask JupyterLab integrations. Dask worker processes run on compute nodes and enable us to farm out notebook code to a set of remote workers enabling us to scale up. Acknowledgments
This work was supported by Lawrence Berkeley National Laboratory, through the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. This work used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory.