SF API Examples
Here, we show some example uses of the SF API as inspiration for your
workflow. Please note that some examples may require a client with
certain security level. The security level
of all endpoints is stored in the x-nersc-scope field which
you can find by clicking on an API endpoint at
https://api.nersc.gov/api/v1.2.
Most of the following Python examples assume that a session has been created in the manner described below. However, general informational queries like system status can be made without authenticating.
```python from authlib.integrations.requests_client import OAuth2Session from authlib.oauth2.rfc7523 import PrivateKeyJWT
token_url = “https://oidc.nersc.gov/c2id/token”
client_id = “
session = OAuth2Session( client_id, private_key, PrivateKeyJWT(token_url), grant_type=”client_credentials”, token_endpoint=token_url ) session.fetch_token() ```
Checking System Status
To check the status of a given NERSC system, one can use the “status” method and specify the name of the NERSC system of interest. For example, to get the status of Perlmutter:
python
system = "perlmutter"
r = session.get("https://api.nersc.gov/api/v1.2/status/"+system)
perlmutter_status = r.json()
print(perlmutter_status)
This will print out something like:
json
{'name': 'perlmutter', 'full_name': 'Perlmutter', 'description': 'System is active', 'system_type': 'compute', 'notes': [], 'status': 'active', 'updated_at': '2021-07-22T04:55:00-07:00'}
To see the status of all NERSC systems, you can use the “status” method with no field, like so:
python
r = session.get("https://api.nersc.gov/api/v1.2/status")
systems_status = r.json()
print(systems_status)
Checking System Status without Logging In
You can also use the requests module to see system status without first creating a session.
python
import requests
system = "archive"
url = "https://api.nersc.gov/api/v1.2/status/"+system
resp = requests.get(url)
perlmutter_status = resp.json()
print(perlmutter_status)
print(perlmutter_status["status"])
Listing Directory Contents and File Information
Listing Directory Contents
To list the contents of a directory on a NERSC file system, the ls utility
method can be used in combination with the directory of interest. For example,
to list the contents of one’s home directory on Perlmutter, we specify both the
system and the full directory path to the request:
python
system = "perlmutter"
home = "<your home dir>" # (e.g., /global/homes/u/username)
r = session.get("https://api.nersc.gov/api/v1.2/utilities/ls/"+system+home)
home_ls = r.json()
print(json.dumps(home_ls, indent=2))
If the user elvis ran this command on their home directory which only
contained a file called test.sh, an example of what this printout would
look like is:
json
{
"status": "ok",
"entries": [
{
"perms": "drwx--x--x",
"hardlinks": 75,
"user": "elvis",
"group": "elvis",
"size": 12800.0,
"date": "2020-01-01T00:00:00",
"name": "."
},
{
"perms": "drwxr-xr-x",
"hardlinks": 594,
"user": "root",
"group": "root",
"size": 64000.0,
"date": "2020-01-01T00:00:00",
"name": ".."
},
{
"perms": "-rwxrwx---",
"hardlinks": 1,
"user": "elvis",
"group": "elvis",
"size": 30.0,
"date": "2021-01-01T12:00:00",
"name": "test.sh"
},
],
"error": null,
"file": null,
"is_binary": null
}
We can see here that the output contains the request status and the list of
entries returned by the ls utility.
Retrieving File Information
The ls utility can also be used to see information about specific files. For
example, if the user elvis wanted to know about the file test.sh in their
home directory, they could use the command
python
system = "perlmutter"
filename = "/global/homes/e/elvis/test.sh"
r = session.get("https://api.nersc.gov/api/v1.2/utilities/ls/"+system+filename)
file_ls = r.json()
print(json.dumps(file_ls, indent=2))
which would print
json
{
"status": "ok",
"entries": [
{
"perms": "-rwxrwx---",
"hardlinks": 1,
"user": "elvis",
"group": "elvis",
"size": 30.0,
"date": "2021-01-01T12:00:00",
"name": "/global/homes/e/elvis/test.sh"
}
],
"error": null,
"file": null,
"is_binary": null
}
Running Commands and Scripts on Login Nodes
Running a Command on a Login Node
Running commands on a login node is done via POST requests to the
/utilities/command path. This request takes two parameters:
- A path parameter of the system name
- A form data parameter
executable, which is a string of the command you wish to run on the system
For example, to print a file’s contents with the cat command, we would use a
request like the following:
python
system = "perlmutter"
cmd = "cat <path to file>" # (e.g., "cat /global/homes/u/username/script.sh")
r = session.post("https://api.nersc.gov/api/v1.2/utilities/command/"+system,
data = {"executable": cmd})
print(r.json())
This will print the SF API task ID corresponding to the POST request and the
status of the associated HTTP request:
json
{'task_id': '123', 'status': 'ok', 'error': None}
!!! danger “This feature requires elevated client permissions” See our API client policies
Getting Task Information and Output
To see the information associated with an SF API task (e.g., the output of the
cat command run above), one can use a GET request on the tasks URL using
the task_id like so:
python
r = session.get("https://api.nersc.gov/api/v1.2/tasks/123")
print(r.json())
This will print the task ID, the task status, and the task result:
json
{
"id": "123",
"status": "completed",
"result": "{\"status\": \"ok\", \"output\": \"#!/bin/bash\\n#\\n#SBATCH -N 1\\n#SBATCH -C cpu\\n#SBATCH -q regular\\n#SBATCH -t 5\\n\\nsrun hostname\\n\\n\", \"error\": null}"
}
To look at the output of the cat command itself, we can extract the result
of the task and then print the output of that result (here, we simply use the
most recent task again):
python
result = json.loads(r['result'])
print(result["output"])
This will print:
```bash
!/bin/bash
SBATCH -N 1
SBATCH -C cpu
SBATCH -q regular
SBATCH -t 5
srun hostname ```
which is the file contents that the cat command wrote to standard output.
Job Submission
Job submissions are done via POST requests to the /compute/jobs path. This
request takes several parameters:
- A path parameter of the system name
- Form data parameters of the job script path on the file system and a boolean value indicating whether specified file is a job to be run via Slurm or a shell script to be run directly on a login node
Batch Job Submission
Here, we demonstrate submitting a job to the
Slurm scheduler via the SF API. The system name, perlmutter, is
placed in the URL path, following the API base path and the /compute/jobs
path. Additionally, a data dictionary of key-value pairs is included in the
POST request. The data dictionary specifies the job submission script path
as the value to the job key and the value of True to the isPath key to
indicate that the file should be sent to the Slurm scheduler.
python
system = "perlmutter"
submit_script = "<path to script>" # (e.g., /global/homes/u/username/script.sub)
r = session.post("https://api.nersc.gov/api/v1.2/compute/jobs/"+system,
data = {"job": submit_script, "isPath": True})
print(r.json())
The result output will include the SF API task ID, the status of the
POST request, and any error text:
json
{'task_id': '124', 'status': 'ok', 'error': None}
!!! warning “POST request status does not indicate job submission status”
The result output status is that of the POST request only and does
not indicate whether or not the job submission was accepted by the Slurm
scheduler.
!!! danger “This feature requires elevated client permissions” See our API client policies
Getting Job Status and Information
To see the information associated with jobs you have submitted via the SF API,
one can use a GET request on the tasks URL (here task_id is 124):
python
r = session.get("https://api.nersc.gov/api/v1.2/tasks/124")
print(r.json())
This will print the SF API task ID, the status of the associated HTTP request, and the result of the request. For successful batch job submission, the request result will include the job ID of the submitted Slurm job.
json
{'id': '124',
'status': 'completed',
'result': '{"status": "ok", "jobid": "12345678", "error": null}'}