Recommendation
Accept If Certain Minor Revisions Are Made
Confidential Comments
The contribution and differentiation, via related work, with previous publications needs to highlighted in the introduction, abstract and title. The reproducible workflow is singled out in the title and abstract, but a majority of the paper is a tutorial, update to previous work, and a description of experience in the software engineering endeavor.
Public Comments
The paper demonstrates a proposed reproducibility workflow based on Docker in Microsoft's Azure public cloud and documents their experiences. Section one defines why reproducibility is essential for science and describes the related work but fails to differentiate their work from past contributions. The reader is left to speculate that it is merely an update to M. Schwab et al., C. Boettiger, C. Freniere et al. works. Section two walks the reader through the authors' reproducibility workflow outlined in figure 1. Note that the delete jobs and pool step is missing in the figure, but presented (including a listing) in the text. Do we need to see the content of listing 1 - 4? Why not add a script to the repository and point to it like the YAML config files in examples/snake2d2k35/config_shipyard? The docker build fails on the COPY ../ssh_config stage. Moreover, do the authors want someone to push their image to the authors' docker hub repository? Besides, the repository in footnote 1 contains a typo which highlights the far more significant issues in reproducibility of well-written error-free publications. This paper is a well-written paper, but errors happen. The authors generalize that "same container ... can now run at scale." What about the MPI drivers? Does this effect where it can run and what hardware it can use? At its' core, this paper is describing the use of cloud computing as an alternative to onsite high-performance computing (HPC). At 'scale' is a loaded term in HPC. This paper examines performance improvements in public cloud offerings and compares small runs in the cloud versus a six-year-old onsite HPC system. However, an MPI performance test up to 8 nodes, a mini-application performance testy up to 8 nodes, and computational fluid dynamics domain science examination of 2 nodes do not adequately address scaling. Section three provides the results in the form of MPI and Poisson mini-application performance analysis and analysis associated with a standard set of computational science experiments involving flow around a flying snake. Both use FDR Infiniband. Is Azure superior performance tuning, configuration, or age? The analysis provided by this section is that Azure interconnect is improving. Moving from 10 Gbps Ethernet to FDR Infiniband in the past four years makes this result visible without testing. Did we need the OSU benchmarks for verification? For the Poisson mini-application, the results again verify Azure has newer hardware than the authors onsite HPC system. However, what was the cause of the variance? Job/Node placement? The detailed description of the flying snake workflow is appreciated. Section four provides the cost analysis for a typical set of computational science experiments and insights on the user experience. For the user experience, how much did free computing and augmented support from Microsoft influence the authors' determination to utilize this environment. If it cost 20K, would the authors have completed the project? How does this cost compare to buying a 2 - 8 node equivalent cluster? Singularity containers appear out of the blue in this section with no description. Table 5 will be outdated by the time this article is published. It does not add value. The reference for Table 5 in the text is just (5). Is this a cite, a figure, or a listing.
Files attached