Fix remaining swarm D->H->D copies #5399

Job	Run time
CI-Cuda-Host (g++, Release, cuda, serial)	42m 53s
CI-Cuda-Host (g++, Release, cuda, mpi)	44m 40s
CI-Cuda-Host (g++, Release, host, serial)	12m 27s
CI-Cuda-Host (g++, Release, host, mpi)	12m 27s
CI-Cuda-Host (g++, Debug, cuda, serial)	1h 15m 16s
CI-Cuda-Host (g++, Debug, cuda, mpi)	1h 15m 46s
CI-Cuda-Host (g++, Debug, host, serial)	11m 33s
CI-Cuda-Host (g++, Debug, host, mpi)	11m 20s
CI-Cuda-Host (clang++-15, Release, cuda, serial)	27m 44s
CI-Cuda-Host (clang++-15, Release, cuda, mpi)	28m 17s
CI-Cuda-Host (clang++-15, Release, host, serial)	10m 12s
CI-Cuda-Host (clang++-15, Release, host, mpi)	10m 29s
CI-Cuda-Host (clang++-15, Debug, host, serial)	8m 12s
CI-Cuda-Host (clang++-15, Debug, host, mpi)	7m 55s
CI-Rocm-HIP (hipcc, Release, hip, serial)	34m 41s
CI-Rocm-HIP (hipcc, Release, hip, mpi)	34m 28s
CI-Rocm-HIP (hipcc, Debug, hip, serial)	57m 47s
CI-Rocm-HIP (hipcc, Debug, hip, mpi)	57m 26s
	9h 23m 33s

Provide feedback