From 352a1524c2d349fcf2a9b04d0bae0c60a8824ab8 Mon Sep 17 00:00:00 2001 From: JimmyZhang12 <67203904+JimmyZhang12@users.noreply.github.com> Date: Mon, 16 Sep 2024 22:53:16 -0700 Subject: [PATCH] Update cudagraphs.md Signed-off-by: JimmyZhang12 <67203904+JimmyZhang12@users.noreply.github.com> --- docs/source/performance/cudagraphs.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/performance/cudagraphs.md b/docs/source/performance/cudagraphs.md index e7d69cc5c447..8e2ce8d1b92f 100644 --- a/docs/source/performance/cudagraphs.md +++ b/docs/source/performance/cudagraphs.md @@ -21,6 +21,7 @@ As a demonstration we show that with a GPT3 20B model, cudagraphs can improve pe ### Usage As of the NeMo 24.09 release, Cudagraphs is currently supported only for the pretraining of dense models. To enable please add the following configs: + `model.enable_cuda_graph=True` `model.use_te_rng_tracker=True`