\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage{natbib}




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[english]{babel}



\usepackage{float}








\begin{document}

\title{Review: ``Reproducible Workflow on a Public Cloud for Computational
Fluid Dynamics''}



\author[1]{Patrick O'Leary}%
\affil[1]{Kitware}%


\vspace{-1em}



  
  \date{May 27, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup








\sloppy


This is a review of manuscript CiSESI-2019-04-004 submitted to Computing
in Science \& Engineering:

Reproducible Workflow on a Public Cloud for Computational Fluid Dynamics
(Lorena Barba and~ Olivier Mesnard).

\textbf{Keywords}:~cisemag, reproducibility, review~

\section*{Content Summary}

{\label{255998}}

A demonstration of a proposed reproducibility workflow based on Docker
for Microsoft's Azure public cloud and documents their experiences.

\textbf{Keywords} -~ reproducibility, cloud computing, computational
fluid dynamics, scientific workflows, high-performance computing.

\par\null

\section*{Contributions}

{\label{619877}}

Neither the introduction or the paper highlight the thesis or novel
contributions. The reader is left to decide for themselves what aspect
they believe are important. This needs to be improved. I found the
following list to be reasonable possibilities.~

\begin{enumerate}
\tightlist
\item
  Use of cloud computing as an alternative to onsite high-performance
  computing (HPC).
\item
  Improvement in public cloud offerings in last 2 to 4 years.
\item
  A proposed reproducible workflow.
\item
  The author's experiences.
\end{enumerate}

The paper is readable, but requires some effort to digest. In the
reviewer's opinion, this effort is due to the author's beliefs on what
is required for reproducible research where research papers begin to
look like laboratory notebooks. Also, the code snippets breakup the
prose with generic, easy to find information such as how to instantiate
or run a docker image (not highlighting their workflow in any manner).

\textbf{References} -~References are sufficient and appropriate

\section*{Overview}

{\label{468802}}

The contribution and differentiation, via related work, with previous
publications needs to highlighted in the introduction, abstract and
title. The reproducible workflow is singled out in the title and
abstract, but a majority of the paper is a docker/Microsoft Azure
tutorial, an update to previous work, and a description of experience in
the software engineering endeavor, but this is not clearly described by
the authors.~

\par\null

The paper demonstrates a proposed reproducibility workflow based on
Docker for Microsoft's Azure public cloud and documents their
experiences.

\section*{Detailed Review}

{\label{991958}}

\textbf{Section One} - Defines why reproducibility is essential for
science and describes the related work but fails to differentiate their
work from past contributions. The reader is left to speculate that it is
merely an update to M. Schwab et al., C. Boettiger, C. Freniere et al.
works.~

\textbf{Section Two} - Walks the reader through the authors'
reproducibility workflow outlined in figure 1.

\textbf{Notes}:

\begin{itemize}
\tightlist
\item
  The delete jobs and pool step is missing in figure 1, but presented
  (including a listing) in the text.
\item
  Do we need to see the content of listing 1 - 4? Why not add a script
  to the repository and point to it like the YAML config files in
  examples/snake2d2k35/config\_shipyard?~
\item
  The docker build fails on the COPY ../ssh\_config stage. Moreover, do
  the authors want someone to push their image to the authors' docker
  hub repository? Besides, the repository in footnote 1 contains a typo
  which highlights the far more significant issues in reproducibility of
  well-written error-free publications. This paper is a well-written
  paper, but errors happen.
\item
  The authors generalize that ``same container \ldots{} can now run at
  scale.'' What about the MPI drivers? Does this effect where it can run
  and what hardware it can use?
\end{itemize}

At its' core, this paper is describing the use of cloud computing as an
alternative to onsite high-performance computing (HPC). At `scale' is a
loaded term in HPC. This paper examines performance improvements in
public cloud offerings and compares small runs in the cloud versus a
six-year-old onsite HPC system. However, an MPI performance test up to 8
nodes, a mini-application performance tests up to 8 nodes, and a
computational fluid dynamics domain science examination of 2 nodes does
not adequately explore scaling.

~\textbf{Section Three} - Provides the results in the form of an MPI
benchmark and a Poisson mini-application performance analysis, and
analysis associated with a standard set of computational science
experiments involving flow around a flying snake.

\textbf{Notes}:

\begin{itemize}
\tightlist
\item
  Both the cloud and the six-year old HPC system use FDR Infiniband. Is
  Azure superior performance a result of tuning, configuration, or age?
  The analysis provided by this section is that Azure interconnect is
  improving. Cloud offerings moving from 10 Gbps Ethernet to FDR
  Infiniband in the past four years makes this result visible without
  testing. Did we need the OSU benchmarks for verification?
\item
  For the Poisson mini-application, the results again verify Azure has
  newer hardware than the authors onsite HPC system. However, what was
  the cause of the variance? Job/Node placement?
\item
  The detailed description of the flying snake workflow is appreciated.
\end{itemize}

\textbf{Section Four} - Explores the cost analysis for a typical set of
computational science experiments and insights on the user experience.

\textbf{Notes}:

\begin{itemize}
\tightlist
\item
  For the user experience, how much did free computing and augmented
  support from Microsoft influence the authors' determination to utilize
  this environment. If it cost 20K, would the authors have completed the
  project?
\item
  Pricing changes every month, every week, maybe every day. Will any of
  this analysis be relevant by publication?
\item
  Table 5 will be outdated by the time this article is published. It
  does not add value.
\item
  How does this cost compare to buying a 2 - 8 node equivalent cluster?~
\item
  Singularity containers appear out of the blue in this section with no
  description.
\item
  The reference for Table 5 in the text is just (5). Is this a cite, a
  figure, or a listing.
\end{itemize}

\selectlanguage{english}
\FloatBarrier
\end{document}

