tcomm.tex

% -*- mode: latex; mode: visual-line; fill-column: 9999; coding: utf-8 -*-

\subsection{Effect of \Rcompcomm on Scaling Performance}
\label{sec:tcomm}

In Section \ref{sec:I/O}, we improved scaling limitations due to read I/O by splitting the trajectory, but scaling remained far from ideal because of increased communication costs.
These results motivated an analysis in terms of the communication costs.
In addition to the compute to I/O ratio \RcompIO discussed in Section \ref{sec:increasedworkload} we defined another performance parameter called the compute to communication ratio $\Rcompcomm$ (Eq.~\ref{eq:Compute-comm}).

We analyzed the data for variable workloads (see Section \ref{sec:increasedworkload}) in terms of the $\Rcompcomm$ ratio.
The performance clearly depended on the $\Rcompcomm$ ratio (Figure \ref{fig:tcom_tcomm_effect}).
Performance improved with increasing $\Rcompcomm$ ratios (Figures \ref{fig:tcomp_tcomm_ratio} and \ref{fig:S1_tcomp_tIO_effect}) even if the communication time was larger (Figure \ref{fig:Comm_time_tcomp_tcomm_effect}).
Although we still observed stragglers due to communication at larger $\Rcompcomm$ ratios ($70\times$ RMSD and $100\times$ RMSD), their effect on performance remained modest because the overall performance was dominated by the compute load. 
Evidently, as long as overall performance is dominated by a component such as compute that scales well, then performance problems with components such as communication will be masked and overall acceptable performance can still be achieved (Figures \ref{fig:S1_tcomp_tIO_effect} and \ref{fig:tcomp_tcomm_ratio}).

Communication was usually not problematic within one node because of the shared memory environment.
For less than 24 processes, i.e., a single compute node on \emph{SDSC Comet}, the scaling was good and $\Rcompcomm \gg 1$ for all RMSD loads (Figures \ref{fig:S1_tcomp_tIO_effect} and \ref{fig:tcomp_tcomm_ratio}).
However, beyond a single compute node ($>$ 24 cores), scaling appeared to improve with increasing $\Rcompcomm$ ratio while the communication overhead decreased in importance (Figures \ref{fig:S1_tcomp_tIO_effect} and \ref{fig:tcomp_tcomm_ratio}).

\begin{figure}[!htb]
  \centering
  \begin{subfigure}{.3\textwidth}
    \includegraphics[width=\linewidth]{figures/Compute_to_comm_ratio_on_performance_v17.pdf}
    \captionsetup{format=hang}
    \caption{Compute to communication ratio \Rcompcomm}
    \label{fig:tcomp_tcomm_ratio}
  \end{subfigure}
  \hspace{1em}
  \begin{subfigure}{.33\textwidth}
    \includegraphics[width=\linewidth]{figures/comm_comparison_different_RMSD_overload.pdf}
    \caption{Communication time \tcomm}
    \label{fig:Comm_time_tcomp_tcomm_effect}
  \end{subfigure}
  \caption{Effect of the ratio of compute to communication time \Rcompcomm on scaling performance on \emph{SDSC Comet}.
    These are the same data as shown in Figure~\protect\ref{fig:S1_tcomp_tIO_effect} but analyzed with respect to the communication load.
    (a) Change in \Rcompcomm with the number of processes $N$ for different workloads. 
    (b) Comparison of communication time for different RMSD workloads.
    Five repeats were performed to collect statistics and error bars show standard deviation with respect to mean.}
  \label{fig:tcom_tcomm_effect}
\end{figure}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End: