\section{Results}
We evaluated the effect of imputation by drawing subsets of the BD2009 training set for the well-characterized allele HLA-A*02:01. Predictors were trained on a range of simulated training set sizes and tested on the remaining data (Figure~\ref{fig:imputecomparison}). We find that imputation gives a modest improvement up to approximately 100 training samples. With more training data there is no benefit to imputation.

% The figure doesn't show up in the preview but does show up if you export to pdf.
\begin{figure}[h]
\centering
\includegraphics[scale=0.75]{figures/HLA-A0201-vs-nsamples-hidden-64-activation-tanh-impute-mice-epochs-250-embedding-32-pretrain-quadratic.pdf}
\caption{MHCflurry performance on down-sampled training data for HLA-A*02:01 with and without imputation}
\label{fig:imputecomparison} 
\end{figure}

We then compared the performance of MHCflurry against NetMHC, NetMHCpan, and SMM on the blind test data. The MHCflurry ensemble model contains 32 predictors initialized with different random weights. The MHCflurry ensemble is competitive with NetMHC and NetMHCpan.

\begin{table}[h]
\centering
\begin{tabular}{llll}
\toprule
{} &               AUC &       $F_1$ score &  Kendall's $\tau$ \\
\midrule
MHCflurry (ensemble)        &  0.93260 &           0.78459 &   \textbf{0.58686} \\
MHCflurry (single predictor)    &           0.93225 &           0.78106 &           0.58572 \\
NetMHC                          &           0.93234 &  \textbf{0.80722} &   0.58633 \\
NetMHCpan                       &  \textbf{0.93264} &           0.79957 &           0.58138 \\
SMM-PMBEC                       &           0.92134 &           0.79026 &           0.56488 \\
\bottomrule
\end{tabular}

\caption{Performance on BLIND dataset}
\label{tab:measurementweighted}
\end{table}

\section{Discussion}
Imputing training data shows promise in cross-validation as a way to improve performance on alleles with few observations, but only seems to help for very small training sizes ($\leq 100$). Unfortunately, none of the alleles included in the BLIND dataset had fewer than 100 samples in BD2009, and only one had fewer than 200. Thus, additional work is required to assess the accuracy of MHCflurry and other predictors on alleles with scarce training data. Additionally, we need to further investigate the interaction between imputation parameters, the  decay schedule for the weights of imputed samples, and stopping criteria for training individual allele-specific predictors.

% These are generated in the 'paper plots' notebook; do not edit by hand.
