Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Example] Distributed nccl test with OpenMPI #3693

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[Example] Distributed nccl test with OpenMPI #3693

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
904d0aa
[Example] Add example for distributed nccl test
Michaelvll Jun 27, 2024
2636564
Avoid export
Michaelvll Jun 27, 2024
5dbabd3
use 2 GPUs instead
Michaelvll Jun 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions examples/distributed_nccl_test_with_mpi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
resources:
accelerators: T4:2
cloud: gcp

file_mounts:
~/.ssh/id_rsa: ~/.ssh/sky-key

num_nodes: 2

setup: |
sudo mv /etc/apt/sources.list.d/kubernetes.list /etc/apt/sources.list.d/kubernetes.list.save || true
sudo apt update
sudo apt -y install git
# install MPI (optional, if you intend to use multiple GPUs)
sudo apt install -y openmpi-bin openmpi-doc libopenmpi-dev
# install nccl
pip install nvidia-nccl-cu12
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/nccl2/lib
echo $LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/
echo $LD_LIBRARY_PATH
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/usr/local/nccl2/include
git clone https://github.com/NVIDIA/nccl-tests.git
cd ./nccl-tests
make MPI=1 NCCL_HOME=/usr/local/nccl2 MPI_HOME=/usr/lib/openmpi

run: |
cd ./nccl-tests
NCCL_TEST_PATH=$(pwd)/build/all_reduce_perf
HOST_IPS=$(echo "$SKYPILOT_NODE_IPS" | tr '\n' ",")
HOST_IPS=${HOST_IPS%,}
echo $HOST_IPS
MASTER_IP=$(echo "$SKYPILOT_NODE_IPS" | head -n1)

if [ "$SKYPILOT_NODE_RANK" == "0" ]; then
mpirun -np $SKYPILOT_NUM_NODES \
--allow-run-as-root \
-H $HOST_IPS \
-x MASTER_ADDR=$MASTER_IP \
-x MASTER_PORT=12354 \
-mca orte_base_help_aggregate 0 \
-map-by node \
--mca btl_tcp_if_exclude lo,docker0 \
-mca plm_base_verbose 5 \
-x PATH \
$NCCL_TEST_PATH -b 8 -e 16 -f 2 -g $SKYPILOT_NUM_GPUS_PER_NODE
fi

Loading