Skip to content

GPU Power Capping on Perlmutter

As HPC enters exascale, power has become a critical limiting factor in HPC. Power capping as one of the commonly used power management approaches can effectively keep the system and jobs within a preset power limit. To prepare more for power-constrained future systems, we encourage users to explore the power capping option with your production workloads. NERSC encourages users to explore this option with their production workloads and see if their workloads can adopt the power capping without hurting performance too much.

Perlmutter allows end users to cap GPU power through a SLURM directive. This feature makes the nvidia-smi -pl <power limit> command which requires root privileges, available to users via a SLURM plugin developed at NERSC.

How to Apply a GPU Power Cap

The supported power limits on Perlmutter GPU nodes are:

  • A100 40 GB GPUs: Power cap range is 100 W - 400 W.
  • A100 80 GB GPUs: Power cap range is 100 W - 500 W.

To request a specific power cap value for your job, use the following SLURM directive in your job script:

```bash

SBATCH –gpu-power=200

```

or

```bash

SBATCH –gpu-power=200W

```

This will apply a 200 W GPU power cap to all allocated nodes for your job.

Sample job script:

```slurm

!/bin/bash

SBATCH -J pc200w

SBATCH -q regular

SBATCH -C gpu

SBATCH -N 2

SBATCH -G 8

SBATCH -t 4:00:00

SBATCH -A mxyz

SBATCH –gpu-power=200

SBATCH -o %x-%j.out

srun -n 8 -c 32 –cpu-bind=cores -G 8 –gpu-bind=none ./a.out ```

Note

  • GPU power capping also works for the shared QOS (#SBATCH -q=shared) on Perlmutter, where it sets the power limit for individual GPUs on the shared node.

How to Track Power Cap Usage

You can track the GPU power cap usage via the sacct command, which reports the power cap in the AdminComment field on Perlmutter.

Example Workflow

  1. Request an interactive job with a 200 W power cap:

bash elvis@perlmutter:login34:~> salloc -C gpu -q interactive --gpu-power=200 -A mxyz ... salloc: Nodes nid001124 are ready for job

  1. Check the GPU power cap and usage using nvidia-smi:

bash elvis@nid001124:~> nvidia-smi -q -i 0 -d POWER

Sample output:

```bash ============== NVSMI LOG ============== Timestamp : Mon Dec 16 11:34:42 2024 Driver Version : 535.216.01 CUDA Version : 12.2

Attached GPUs : 4 GPU 00000000:03:00.0 GPU Power Readings Power Draw : 53.87 W Current Power Limit : 200.00 W Requested Power Limit : 200.00 W Default Power Limit : 400.00 W Min Power Limit : 100.00 W Max Power Limit : 400.00 W Power Samples Duration : 2.39 sec Number of Samples : 119 Max : 53.94 W Min : 53.87 W Avg : 53.88 W Module Power Readings Power Draw : N/A Current Power Limit : N/A Requested Power Limit : N/A Default Power Limit : N/A Min Power Limit : N/A Max Power Limit : N/A ```

  1. Exit the interactive session:

bash elvis@nid001124:~> exit exit salloc: Relinquishing job allocation 33994454

  1. Check the power cap usage with sacct:

bash elvis@perlmutter:login34:~> sacct -j 33994454 -XPno admincomment | jq . | fgrep gpuPower "gpuPower": "200", "gpuPowerRaw": "200",

The gpuPower field displays the applied power cap (e.g., 200 W in this case). If the #SBATCH --gpu-power=200W directive is used (note the “W” for watts), the gpuPowerRaw field will report “200W”, while the gpuPower field will display “200” (without the unit).

Notes

  • The power cap applies to all GPUs allocated to your job throughout the job duration (hence to all job steps).
  • Currently, power capping capability is not available via srun (per job step).

  • Ensure you specify the correct power limit based on the GPU model (e.g., A100 40 GB vs. A100 80 GB). If the requested power exceeds or falls short of the allowed range, the power limit will be automatically adjusted to the nearest valid value. Your job will proceed with a message similar to the following:

    bash slurmstepd: error: gpu-power: nid001204: requested power 80W less than 100W, setting to 100W

    or

    bash slurmstepd: error: gpu-power: nid001021: requested power 600W greater than maximum rating of 400W, setting to 400W

  • The nvidia-smi tool can provide detailed power usage metrics, which can be useful for debugging or monitoring power consumption.

For more information about the options of nvidia-smi and sacct used in this document, please refer to the nvidia-smi man page (nvidia-smi -h) and the sacct command man page (man sacct).