Successful Builds

[zhi-wang]

I'll use this thread to record the successful builds on the new machines or with the new softwares (drivers, compilers, etc.).

You are welcome to paste your tinker9 info output as well.

Template (feel free to modify):

Device: GPU; Computer
| MM/DD/YYYY | Nvidia    | Commit   | GCC    | CUDA     | OpenACC       | OS         |
|------------|-----------|----------|--------|----------|---------------|------------|
|            |           |          |        |          |               |            |

Useful commands:

nvidia-smi
g++ --version
cat /etc/os-release
hostnamectl
./info9 # in Tinker9

[zhi-wang]

Device: GeForce GTX 970; Mac Pro 1,1

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
01/11/2021	460.32.03	ec9b2dd	10.2.0	11.1.105	nvc++ 20.11-0	Arch Linux

[zhi-wang]

Device: GeForce RTX 2070 with Max-Q Design; Razer Blade 15 RZ09-03017EM8

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
11/10/2020	460.20	N/A	9.3.0	11.0.221	nvc++ 20.9-0	WSL2 Ubuntu 20.04.1

[roseane-reis]

Device: GeForce RTX 2070 with Max-Q Design; Razer Blade 15 RZ09-03287EM2

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
01/15/2021	460.32.03	dc1b5af	7.5.0	11.1.105	nvc++ 20.11-0	Ubuntu 20.04.1

[asedova]

Device: NVIDIA V100; Computer: Summit at OLCF

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
02/02-05/2021	418.116.00	c4b7d56 3b0791c	4.8.5	10.1.243	pgi 19.9	RHEL 7.6 on IBM 4.14.0-115.21.2.el7a.ppc64le

mkdir build && cd build
module load pgi cuda
export FC=/usr/bin/gfortran
module load cmake
cmake -DCUDA_DIR=$CUDA_DIR/ -DCOMPUTE_CAPABILITY=70 ..
make -j
make test

jsrun -r1 -g1 -a1 -c1 --smpiargs="-disable_gpu_hooks" ../tinker9 dynamic dhfr2.xyz 20000 2 10 2 298

[leucinw]

• Device: GeForce RTX 3070

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
02/11/2021	460.27.04	53e04c8	8.3.1	11.2.67	nvc++ 21.1.0	CentOS 8.2

• compile file

#!/bin/bash

rm -fr !("compile")

export CUDAHOME=/usr/local/cuda-11.2
export CUDACXX=$CUDAHOME/bin/nvcc
export FC=/usr/bin/gfortran
export CXX=/usr/bin/g++
export ACC=/home/liuchw/shared/nvidia/hpc_sdk/Linux_x86_64/21.1/compilers/bin/nvc++
export opt=release
export host=0
export prec=m
export compute_capability=80
export cuda_dir=$CUDAHOME
export CMAKEHOME=/home/liuchw/shared/cmake3.12/bin/

$CMAKEHOME/cmake ../
make -j

• tinker9 info output

   Version:                             1.0.0 GIT 53e04c8c
   Synchronized with Tinker commit:     6a1c6104
   C++ compiler:                        g++ 8.3.1
   Size of real (bytes):                4
   Size of mixed (bytes):               8
   Using deterministic force:           true
   Debug mode:                          off
   Platform:                            CUDA and OpenACC
   Primary GPU package:                 CUDA
   Latest CUDA supported by driver:     11.2
   CUDA runtime version:                11.2
   Thrust version:                      1.10.0 patch 0
   CUDA compiler:                       nvcc 11.2.67
   OpenACC compiler:                    nvc++ 21.1.0
   GPU detected:                        1
   GPU 0:
      PCI:                              0000:02:00.0
      Name:                             GeForce RTX 3070
      Maximum compute capability:       8.6
      Single double perf. ratio:        32
      Compute mode:                     default
      Error-correcting code (ECC):      off
      Clock rate (kHz):                 1770000
      Number of Multiprocessors:        46
      Number of CUDA cores:             5888
      Used/Total GPU memory:            3.33 % / 7.79 GB

[zhi-wang]

This commit "6a1c6104" was not found in the Git history. This must have been due to the local pull-merge operation.

[zhi-wang]

Oh, I see. This commit is the version of libtinker.a. I've corrected it with the Tinker9 commit hash.

[batatasfritas]

• Device: GeForce GTX 2070; custom built computer (dual CPU, dual GPU)

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
05/12/2021	418.181.07	fded8a3	8.3.0	10.1.105	nvc++ 20.7.0	Debian 10.8

• To compile:

export FC=$(which gfortran)
cmake . -B build \
   -DCUDA_DIR=$CUDA_ROOT \
   -DCMAKE_BUILD_TYPE=Release \
   -DHOST=0 \
   -DPREC=m \
   -DCMAKE_INSTALL_PREFIX=/path/to/tinker
make -C build/

• Output of CUDA_VISIBLE_DEVICES=0 tinker9 info

    Version:                             1.0.0 GIT fded8a3c
    Synchronized with Tinker commit:     080e8f1d
    C++ compiler:                        nvc++ 20.7.0
    Size of real (bytes):                4
    Size of mixed (bytes):               8
    Using deterministic force:           true
    Debug mode:                          off
    Platform:                            CUDA and OpenACC
    Primary GPU package:                 CUDA
    Latest CUDA supported by driver:     10.1
    CUDA runtime version:                10.1
    Thrust version:                      1.9.4 patch 0
    CUDA compiler:                       nvcc 10.1.105
    OpenACC compiler:                    nvc++ 20.7.0
    GPU detected:                        1
    GPU 0:                              
       PCI:                              0000:3B:00.0
       Name:                             GeForce RTX 2070
       Maximum compute capability:       7.5
       Single double perf. ratio:        32
       Compute mode:                     default
       Error-correcting code (ECC):      off
       Clock rate (kHz):                 1620000
       Number of Multiprocessors:        36
       Number of CUDA cores:             4608
       Used/Total GPU memory:            1.52 % / 7.77 GB

• Setting CUDA_VISIBLE_DEVICES:
  Many of our machines have two GPUs.tinker9 info does not work if I don't specifically select one with CUDA_VISIBLE_DEVICES. If I don't do that (either inline with the command show above or with by running export CUDA_VISIBLE_DEVICES=0 beforehand it fails with:
  Terminating with uncaught exception : Errno 708 (cannot set while device is active in this process) at /path/to/sources/tinker9/src/cudart/gpu_card.cpp:308
  Leaving it here, in case it may help others, but since there is an easy work around I don't think it's worth it to create an issue report.

[zhi-wang]

Fantastic and thanks! This is an issue I'd like to follow up with a bit more because it looks familiar to me and I still don't fully understand or replicate it on our multi-GPU machines. Is there any switch to set up the number of processes when you submit jobs to your nodes? This error message seemed to complain about cards being accessed/initialized by another process.

[batatasfritas]

This isn't a node, this is a standalone server. There is nothing to isolate any of the resources. However, I checked and I'm currently the only user on the machine (standard account, not root). There is no X running and nvidia-smi shows no processes running on either GPU on the machine.

I can try comparing to some other machines I have access to that use Slurm and isolate resources, but this will take a while since I'll need to install the Nvidia pre-requisites.

[zhi-wang]

We think this is an issue related to the undocumented changes in CUDA 11. As the code was mostly tested with CUDA 11, not 10, we didn't find out the cause of this issue until recently. We believe that this issue is fixed now.

[zhi-wang]

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
05/27/2022		29ac343		9.1.128	N/A

OS: macOS High Sierra 10.13.6 (17G4015)
machine: iMac (27-inch, Late 2012)
device: GeForce GTX 675MX
gfortan: 8.1.0
clang++: clang++ 9.0.0 (xcode)
quadro & geforce macos driver release (web driver): 387.10.10.10.40.113 https://www.nvidia.com/download/driverResults.aspx/141130/ On the same page, the download link for the recommended version of cuda driver (418.105) is included.
cuda driver: 418.105 https://www.nvidia.com/en-us/drivers/cuda/macosx-cuda-418-105-driver/
nvcc: V9.1.128

# configure cmake
$ FC=gfortran CC=clang CXX=clang++ gpu_lang=cuda compute_capability=30 cmake ..
$ make

# run
$ ./info9 

 GPU Device :  Setting Device ID to 0 from GPU utilization


 Program Information

    Version:                             1.0.0 GIT 29ac343e
    Synchronized with Tinker commit:     3dc966e2
    C++ compiler:                        clang++ 9.0.0 (xcode)
    Size of real (bytes):                4
    Size of mixed (bytes):               8
    Using deterministic force:           true
    Debug mode:                          off
    Platform:                            CUDA
    Primary GPU package:                 CUDA
    Latest CUDA supported by driver:     10.1
    CUDA runtime version:                9.1
    Thrust version:                      1.9.1 patch 2
    CUDA compiler:                       nvcc 9.1.128
    OpenACC compiler:                    Unused
    GPU detected:                        1
    GPU 0:                              
       PCI:                              0000:01:00.0
       Name:                             GeForce GTX 675MX
       Maximum compute capability:       3.0
       Single double perf. ratio:        24
       Compute mode:                     default
       Error-correcting code (ECC):      off
       Clock rate (kHz):                 719000
       Number of Multiprocessors:        7
       Number of CUDA cores:             1344
       Used/Total GPU memory:            91.14 % / 1.00 GB

[jayponder]

Great! Tinker9 works on macOS, in addition to LInux. Now we just have to get Apple and NVIDIA to work together again, and support NVIDIA GPUs on macOS :)

[zhi-wang]

Reference: #199
Provided by @gwiecz1

CPU: 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz
GPU: NVIDIA RTX A3000 (max cap. 8.6)
Debian sid 5.17.11-1 (2022-05-26) x86_64 GNU/Linux
Compilers:
gcc (Debian 11.3.0-3) 11.3.0 - for gcc compilation
icpc and ifort 2021.6.0 20220226 - for intel compilation
nvc++ (nvhpc) 22.5.0
nvcc 11.7.64

Tinker9 1.0.0 GIT 18f90f1 (+1 line :))
Tinker GIT 3dc966e2
nvidia driver 470.129.06
CUDA 11.7.0
cmake 3.23.2

[vojtechkostal]

GPU: NVIDIA GeForce RTX 3080
CPU: Intel(R) Core(TM) i7-7800X CPU @ 3.50GHz

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
07/18/2022	515.48.07	3dc966e2	10.4.0	11.7.0	N/A	Arch Linux

create build directory

cd tinker9
mkdir build && cd build

load spack packages

source ~/build/spack/20220704-git/share/spack/setup-env.sh
spack load gcc@10.4.0
spack load cmake@3.23.2
spack load cuda@11.7.0

export paths

export CUDAHOME=~/build/spack/20220704-git/opt/spack/linux-arch-skylake_avx512/gcc-10.4.0/cuda-11.7.0-g7gmej6jrqclzpptqdnq4jwt27iztyed
export cuda_dir=$CUDAHOME

compilation

FC=gfortran compute_capability=86 gpu_lang=cuda cmake -DCMAKE_CUDA_ARCHITECTURES=86 ..
make

• output of ./tinker9 info:

    Version:                             1.0.2 GIT cae970e5
    Synchronized with Tinker commit:     3dc966e2
    C++ compiler:                        g++ 10.4.0
    Size of real (bytes):                4
    Size of mixed (bytes):               8
    Using deterministic force:           true
    Debug mode:                          off
    Platform:                            CUDA
    Primary GPU package:                 CUDA
    Latest CUDA supported by driver:     11.7
    CUDA runtime version:                11.7
    Thrust version:                      1.15.0 patch 0
    CUDA compiler:                       nvcc 11.7.64
    OpenACC compiler:                    Unused
    GPU detected:                        1
    GPU 0:                              
       PCI:                              0000:65:00.0
       Name:                             NVIDIA GeForce RTX 3080
       Maximum compute capability:       8.6
       Single double perf. ratio:        32
       Compute mode:                     default
       Error-correcting code (ECC):      off
       Clock rate (kHz):                 1905000
       Number of Multiprocessors:        68
       Number of CUDA cores:             8704
       Used/Total GPU memory:            7.74 % / 9.78 GB

[zhi-wang]

#207 (comment)

[aurbn]

Device: GPU; Computer

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
04/24/2023	530.30.02	d4dfb2d	12.1.0	12.1.0	nvc++ 23.3.0	Ubuntu 22.04

Had to manually edit CMakeLists.txt and comment those lines:

@@ -432,9 +432,9 @@ elseif (GPU_LANG STREQUAL "OPENACC")
    string (APPEND CMAKE_CXX_FLAGS " --diag_suppress=11,177")
    # Compute Capability 60,70 -> ,cc60,cc70
    set (__T9_ACC_CCLST4) # ""
-   foreach (var ${T9_CUCCLIST})
-      string (APPEND __T9_ACC_CCLST4 ",cc${var}")
-   endforeach () # ,cc60,cc70
+    #foreach (var ${T9_CUCCLIST})
+    #   string (APPEND __T9_ACC_CCLST4 ",cc${var}")
+    #endforeach () # ,cc60,cc70
 endif ()

mkdir build && cd build
cat >> ../envs.sh

#!/bin/bash
export CUDAHOME=/usr/local/cuda-12.1
export CUDACXX=$CUDAHOME/bin/nvcc
export FC=/usr/bin/gfortran
export CXX=/usr/bin/g++
export ACC=/opt/nvidia/hpc_sdk/Linux_x86_64/23.3/compilers/bin/nvc++
export opt=release
export host=0
export prec=m
export compute_capability=89
export cuda_dir=$CUDAHOME
cmake -DCMAKE_INSTALL_PREFIX=/opt/tinker9 -DCOMPUTE_CAPABILITY=89 .. 

../envs.sh
make -j20

./tinker9 info

Primary GPU package :  CUDA

GPU Device :  Setting Device ID to 0 from GPU utilization


Program Information

   Version:                             1.4.0 GIT d4dfb2d6
   Synchronized with Tinker commit:     b92eacc8
   C++ compiler:                        nvc++ 23.3.0
   Size of real (bytes):                4
   Size of mixed (bytes):               8
   Using deterministic force:           true
   Debug mode:                          off
   Platform:                            OpenACC and CUDA
   Primary GPU package:                 CUDA
   Latest CUDA supported by driver:     12.1
   CUDA runtime version:                12.1
   Thrust version:                      2.0.1 patch 0
   CUDA compiler:                       nvcc 12.1.105
   OpenACC compiler:                    nvc++ 23.3.0
   GPU detected:                        1
   GPU 0:                              
      PCI:                              0000:01:00.0
      Name:                             NVIDIA GeForce RTX 4090
      Maximum compute capability:       8.9
      Single double perf. ratio:        64
      Compute mode:                     default
      Error-correcting code (ECC):      off
      Clock rate (kHz):                 2565000
      Number of Multiprocessors:        128
      Number of CUDA cores:             8192
      Used/Total GPU memory:            1.72 % / 23.65 GB

[aurbn]

Actually noticed that Number of CUDA cores is 8192, but nvidia-settings -q CUDACores -t shows 16384 so there is some problem.

[aurbn]

Looks like just a mistake in number of cores calculation #223.

[aurbn]

info9
 Primary GPU package :  CUDA

 GPU Device :  Setting Device ID to 0 from GPU utilization


 Program Information

    Version:                             1.4.0 GIT d4dfb2d6
    Synchronized with Tinker commit:     b92eacc8
    C++ compiler:                        nvc++ 23.3.0
    Size of real (bytes):                4
    Size of mixed (bytes):               8
    Using deterministic force:           true
    Debug mode:                          off
    Platform:                            OpenACC and CUDA
    Primary GPU package:                 CUDA
    Latest CUDA supported by driver:     12.1
    CUDA runtime version:                12.1
    Thrust version:                      2.0.1 patch 0
    CUDA compiler:                       nvcc 12.1.105
    OpenACC compiler:                    nvc++ 23.3.0
    GPU detected:                        1
    GPU 0:                              
       PCI:                              0000:01:00.0
       Name:                             NVIDIA GeForce RTX 4090
       Maximum compute capability:       8.9
       Single double perf. ratio:        64
       Compute mode:                     default
       Error-correcting code (ECC):      off
       Clock rate (kHz):                 2565000
       Number of Multiprocessors:        128
       Number of CUDA cores:             16384
       Used/Total GPU memory:            1.72 % / 23.65 GB

[zhi-wang]

Again, thank you for the pull request. But I still don't understand why commenting these lines in CMake script is necessary. The intension is to pass 'cc80' to nvc++. In your case, this would be 'cc86'. Would you help me understand the necessity? Is it because cc86 is not supported by nvc++ and it will result in compilation failure?

[aurbn]

Yes. Looks like nvc++ doesn't support ccXX in -ta argument. At least version 23.3. It gives this error in leave it as is:

nvc++-Error-Switch -ta with unknown keyword cc89
-ta=host|multicore|tesla:{[no]autocollapse|ptxinfo}
                    Choose target accelerator (supported only for OpenACC, DEPRECATED please refer to -acc and -gpu)
    host            Compile for serial execution on the host CPU
    multicore       Compile for parallel execution on the host CPU
    tesla           Compile for parallel execution on a Tesla GPU
     [no]autocollapse
                    Automatically collapse tightly nested parallel loops
     ptxinfo        Print ptxas information

Its the same with 86.

[gwiecz1]

Hi,
All tests passed.

./info9
Primary GPU package :  CUDA

GPU Device :  Setting Device ID to 0 from GPU utilization

Program Information
Version:                             1.4.0 GIT d4dfb2d6
Synchronized with Tinker commit:     b92eacc8
C++ compiler:                        nvc++ 23.3.0
Size of real (bytes):                4
Size of mixed (bytes):               8
Using deterministic force:           true
Debug mode:                          off
Platform:                            OpenACC and CUDA
Primary GPU package:                 CUDA
Latest CUDA supported by driver:     12.0
CUDA runtime version:                12.0
Thrust version:                      2.0.1 patch 0
CUDA compiler:                       nvcc 12.0.140
OpenACC compiler:                    nvc++ 23.3.0
GPU detected:                        1
GPU 0:                              
   PCI:                              0000:91:00.0
   Name:                             Quadro RTX 5000
   Maximum compute capability:       7.5
   Single double perf. ratio:        32
   Compute mode:                     default
   Error-correcting code (ECC):      off
   Clock rate (kHz):                 1815000
   Number of Multiprocessors:        48
   Number of CUDA cores:             6144
   Used/Total GPU memory:            4.70 % / 15.73 GB

[AlainWil]

August 2023
OS=Ubuntu 22.04
All tests passed.

GPU Device : Setting Device ID to 0 from GPU utilization

Program Information

Version:                             1.4.0 GIT d4dfb2d6
Synchronized with Tinker commit:     b92eacc8
C++ compiler:                        g++ 11.4.0
Size of real (bytes):                4
Size of mixed (bytes):               8
Using deterministic force:           true
Debug mode:                          off
Platform:                            CUDA
Primary GPU package:                 CUDA
Latest CUDA supported by driver:     12.2
CUDA runtime version:                12.2
Thrust version:                      2.1.1 patch 0
CUDA compiler:                       nvcc 12.2.128
OpenACC compiler:                    Unused
GPU detected:                        1
GPU 0:                              
   PCI:                              0000:26:00.0
   Name:                             NVIDIA GeForce RTX 2070 SUPER
   Maximum compute capability:       7.5
   Single double perf. ratio:        32
   Compute mode:                     default
   Error-correcting code (ECC):      off
   Clock rate (kHz):                 1815000
   Number of Multiprocessors:        40
   Number of CUDA cores:             5120
   Used/Total GPU memory:            7.75 % / 7.78 GB

[AlainWil]

September 2023
OS=Ubuntu 22.04
All tests passed.

GPU Device : Setting Device ID to 0 from GPU utilization

Program Information
Version: 1.4.0 GIT d4dfb2d
Synchronized with Tinker commit: b92eacc8
C++ compiler: g++ 11.4.0
Size of real (bytes): 4
Size of mixed (bytes): 8
Using deterministic force: true
Debug mode: off
Platform: CUDA
Primary GPU package: CUDA
Latest CUDA supported by driver: 12.2
CUDA runtime version: 12.2
Thrust version: 2.2.0 patch 0
CUDA compiler: nvcc 12.2.140
OpenACC compiler: Unused
GPU detected: 1
GPU 0:
PCI: 0000:01:00.0
Name: NVIDIA GeForce RTX 3070
Maximum compute capability: 8.6
Single double perf. ratio: 32
Compute mode: default
Error-correcting code (ECC): off
Clock rate (kHz): 1845000
Number of Multiprocessors: 46
Number of CUDA cores: 2944
Used/Total GPU memory: 7.60 % / 7.78 GB

[blakemertz]

Used native nvidia utilities, no ACC calls.

MM/DD/YYYY	Nvidia	Commit	GCC	CUDA	OpenACC	OS
03/28/2024	525.147.05	b92eacc8	12.2.0	12.0	no	Debian 12

    Version:                             1.4.0 GIT 70bd0529
    Synchronized with Tinker commit:     b92eacc8
    C++ compiler:                        g++ 12.2.0
    Size of real (bytes):                4
    Size of mixed (bytes):               8
    Using deterministic force:           true
    Debug mode:                          off
    Platform:                            CUDA
    Primary GPU package:                 CUDA
    Latest CUDA supported by driver:     12.0
    CUDA runtime version:                11.8
    Thrust version:                      1.17.2 patch 0
    CUDA compiler:                       nvcc 11.8.89
    OpenACC compiler:                    Unused
    GPU detected:                        1
    GPU 0:                              
       PCI:                              0000:01:00.0
       Name:                             NVIDIA GeForce RTX 3060 Laptop GPU
       Maximum compute capability:       8.6
       Single double perf. ratio:        32
       Compute mode:                     default
       Error-correcting code (ECC):      off
       Clock rate (kHz):                 1425000
       Number of Multiprocessors:        30
       Number of CUDA cores:             1920
       Used/Total GPU memory:            2.23 % / 5.80 GB

cat ../envs.sh
#!/bin/bash

export CUDAHOME=/usr
export CUDACXX=/usr/bin/nvcc
export FC=/usr/bin/gfortran
export CXX=/usr/bin/g++
export opt=release
export host=0
export prec=m
export compute_capability=86
export cuda_dir=$CUDAHOME

cmake -DCMAKE_INSTALL_PREFIX=/media/Data/binaries/github/tinker9 -DGPU_LANG=CUDA -DCOMPUTE_CAPABILITY=86 .. 

Also had to install catch2 utility on one of my Debian systems to get cmake to do 100% build.