Skip to content

Compiling OpenMP Code at NERSC

OpenMP Flags for Different PrgEnv Modules

Depending on the compiler suite used, a different flag is needed to use OpenMP:

Vendor PrgEnv Language(s) OpenMP flag
GNU PrgEnv-gnu C/C++/Fortran -fopenmp
NVIDIA PrgEnv-nvidia C/C++/Fortran -mp
HPE PrgEnv-cray C/C++ -fopenmp
HPE PrgEnv-cray Fortran -homp

GPU Offload

GPU target must be set

Either load the cudatoolkit and craype-accel-nvidia80 modules or use the "target" flags below.

PrgEnv-nvidia

Use the -mp=gpu option.

Details of OpenMP support and programming recommendation are available at in the OpenMP section of the Perlmutter Readiness Guide.

PrgEnv-cray

Language(s) OpenMP flag target
C/C++ -fopenmp -fopenmp-targets=nvptx64 -Xopenmp-target=nvptx64 -march=sm_80
Fortran -homp -hnoacc -haccel=nvidia80

linker warning

The following linker warning can be safely ignored:

warning: linking module '@': linking module flags 'SDK Version': IDs have conflicting values ('[2 x i32] [i32 11, i32 1]' from /opt/cray/pe/cce/14.0.1/cce-clang/x86_64/lib/libcrayomptarget-nvptx-sm_80.bc with '[2 x i32] [i32 11, i32 5]' from ../benchmarks/babelStream/main.cpp) [-Wlinker-warnings]

Target selection error with Fortran

Target selection error With PrgEnv-cray Fortran applications with OpenMP offload can be avoided by

  1. unset the CRAYPE_ACCEL_TARGET variable
  2. manually specify the target architecture -homp -hnoacc -haccel=nvidia80

mixed C/C++ and Fortran applications

Use -fopenmp -fopenmp-targets=nvptx64 -Xopenmp-target=nvptx64 -march=sm_80 to manually specify the target architecture for C/C++ code.

PrgEnv-gnu

PrgEnv-gnu is not recommended for OpenMP offloading.

To access it: 

ml use /global/cfs/cdirs/m1759/wwei/Modules/perlmutter/modulefiles/
ml gcc/13.1.0

An example to compile your code:

gcc -Ofast -fopenmp -foffload=nvptx-none="-Ofast -lm -latomic -misa=sm_80"  main.c -o main