- File system navigation: HPC centers have large shared file systems with complex, sprawling folder hierarchies. JupyterLab users at NERSC often "got lost" while navigating NGF via UI. To effectively manage points of entry relevant to each user, we developed the jupyterlab-favorites and jupyterlab-recents labextensions to bookmark or highlight commonly used files and directories (e.g., scratch directories, shared project directories).
- Batch queue: NERSC uses the Slurm Workload Manager to manage batch job allocations on Cori's compute nodes. We have created jupyterlab-slurm, a labextension that lets users monitor and submit batch jobs directly from JupyterLab, creating a common interface to the interactive and batch components of a supercomputing workflow. We are also developing integrations for the Dask distributed framework to scale out notebook code on a set of worker nodes.
- Reusability and reproducibility: In team science with Jupyter it is common for a user to develop a recipe or template notebook that captures a scientific workflow, along with its software dependencies in a Jupyter kernel, and then share this with collaborators. We have developed the clonenotebooks serverextension to allow users to clone and run such notebooks and kernels from an nbviewer gallery. Users in a collaboration can then reproduce a given workflow and extend it to handle new data sets and parameters. Creating a general pattern of reproducible Jupyter HPC workflows with containers similar to Binder \cite{ramakrishnan2020} is a topic of current work.
The early phases were organic with lessons learned and re-learned as we engaged the Jupyter ecosystem and the ecosystem itself matured. From this experience, we have come to appreciate the strategic value of Project Jupyter's commitments to abstraction, extensibility, and deployment-agnostic design.