Support Slurm 24.11 and REST API v0.0.40

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [unreleased]
 
 ### Added
+- Support Slurm 24.11 and Slurm REST API v0.0.40 (#366 → #400).
 - agent:
   - Return RacksDB infrastructure name and a boolean to indicate if metrics
     feature is enabled in `/info` endpoint, in addition to the cluster name.
@@ -64,14 +65,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Add requirement on markdown external library for `gateway` extra package.
 
 ### Changed
+- agent: Bump minimal required Slurm version from 23.02.0 to 23.11.0.
 - gateway: Change error message when unable to parse agent info fields.
 - docs:
   - Update configuration reference documentation.
   - Update dashboard screenshot in overview page with example of resource chart.
+  - Replace mention of Slurm REST API version v0.0.39 by v0.0.40.
+  - Mention requirement of Slurm >= 23.11 and dropped support of Slurm 23.02.
 - conf:
   - Convert `[cache]` > `password` agent parameter from string to password type.
   - Convert `[ldap]` > `bind_password` gateway parameter from string to password
     type.
+  - Bump `[slurmrestd]` > `version` default value from `0.0.39` to `0.0.40` in
+    agent configuration for compatibility with Slurm 24.11.
 - pkgs:
   - Add requirement on RFL.core >= 1.1.0.
   - Add requirement on RFL.settings >= 1.1.1.
@@ -96,6 +102,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Update dependencies to fix CVE-2024-45812 and CVE-2024-45811 (vite),
     CVE-2024-47068 (rollup), CVE-2024-21538 (cross-spawn).
 
+### Removed
+- Support of Slurm 23.02 and Slurm REST API v0.0.39.
+- conf:
+  - Remove unused `required` from default selected jobs field on `slurmrestd`
+    `/slurm/*/jobs` endpoint.
+  - Remove unused `state_reason` from default selected job field on `slurmrestd`
+    `/slurm/*/job/<id>` endpoint.
+
 ## [3.2.0] - 2024-09-05
 
 ### Added

conf/vendor/agent.yml

@@ -59,7 +59,7 @@ slurmrestd:
     doc: Path to slurmrestd UNIX socket
   version:
     type: str
-    default: '0.0.39'
+    default: '0.0.40'
     doc: |
       Slurm REST API version.
 
@@ -101,7 +101,6 @@ filters:
     - partition
     - priority
     - qos
-    - required
     - script
     - state
     - steps
@@ -131,7 +130,6 @@ filters:
     - standard_error
     - standard_input
     - standard_output
-    - state_reason
     - tasks
     - tres_req_str
     doc: |

dev/crawl-tests-assets

@@ -14,6 +14,7 @@ import getpass
 import socket
 import shlex
 import random
+import os
 import logging
 
 import requests
@@ -39,6 +40,7 @@ USER = getpass.getuser()
 METRICS_PREFERRED_CLUSTER = "emulator"
 # Map between infrastructure names and cluster names that are visible in Slurm-web.
 MAP_CLUSTER_NAMES = {"emulator": "atlas"}
+ADMIN_PASSWORD_ENV_VAR = "SLURMWEB_DEV_ADMIN_PASSWORD"
 
 
 def slurmweb_cluster_name(infrastructure: str):
@@ -172,13 +174,13 @@ def crawl_slurmrestd(socket: Path) -> None:
 
     session = requests.Session()
     prefix = "http+unix://slurmrestd/"
-    api = "0.0.39"
+    api = "0.0.40"
     session.mount(prefix, SlurmrestdUnixAdapter(socket))
 
     # Get Slurm version
     text, _, _ = query_slurmrestd(session, prefix, f"/slurm/v{api}/ping")
     ping = json.loads(text)
-    release = ping["meta"]["Slurm"]["release"]
+    release = ping["meta"]["slurm"]["release"]
     version = release.rsplit(".", 1)[0]
     logger.info("Slurm version: %s release: %s", version, release)
 
@@ -238,7 +240,7 @@ def crawl_slurmrestd(socket: Path) -> None:
     )
 
     def dump_job_state(state: str):
-        if _job["job_state"] == state:
+        if state in _job["job_state"]:
             dump_slurmrestd_query(
                 session,
                 requests_statuses,
@@ -317,6 +319,8 @@ def crawl_slurmrestd(socket: Path) -> None:
         assets_path,
         "slurm-nodes",
         skip_exist=False,
+        limit_dump=100,
+        limit_key="nodes",
     )
 
     def dump_node_state():
@@ -387,7 +391,7 @@ def crawl_slurmrestd(socket: Path) -> None:
 
     # Save resulting status file
     with open(status_file, "w+") as fh:
-        json.dump(requests_statuses, fh, indent=2)
+        json.dump(requests_statuses, fh, indent=2, sort_keys=True)
         fh.write("\n")
 
 
@@ -438,7 +442,15 @@ def gateway_url(dev_tmp_dir):
 def user_token(url: str, user: str):
     """Ask user password interactively, authenticate on gateway and return
     authentication JWT."""
-    password = getpass.getpass(prompt=f"Password for {user} on gateway: ")
+
+    try:
+        password = os.environ[ADMIN_PASSWORD_ENV_VAR]
+    except KeyError:
+        logger.info(
+            "Unable to read admin password from environment, opening interactive "
+            "prompt."
+        )
+        password = getpass.getpass(prompt=f"Password for {user} on gateway: ")
 
     response = requests.post(
         f"{url}/api/login", json={"user": user, "password": password}
@@ -461,6 +473,7 @@ def dump_component_query(
     asset_name: dict[int, str] | str,
     skip_exist: bool = True,
     prettify: bool = True,
+    limit_dump=0,
 ) -> Any:
     """Send GET HTTP request to Slurm-web component pointed by URL and save JSON result
     in assets directory."""
@@ -499,7 +512,10 @@ def dump_component_query(
     else:
         with open(asset, "w+") as fh:
             if asset.suffix == ".json":
-                fh.write(json.dumps(data, indent=2 if prettify else None))
+                _data = data
+                if limit_dump:
+                    _data = _data[:limit_dump]
+                fh.write(json.dumps(_data, indent=2 if prettify else None))
             else:
                 fh.write(data)
     return data
@@ -568,6 +584,7 @@ def crawl_gateway(cluster: str, infrastructure: str, dev_tmp_dir: Path) -> str:
         assets_path,
         "jobs",
         skip_exist=False,
+        limit_dump=100,
     )
 
     if not (len(jobs)):
@@ -578,7 +595,7 @@ def crawl_gateway(cluster: str, infrastructure: str, dev_tmp_dir: Path) -> str:
         min_job_id = jobs[0]["job_id"]
 
         def dump_job_state() -> None:
-            if _job["job_state"] == state:
+            if state in _job["job_state"]:
                 dump_component_query(
                     requests_statuses,
                     url,
@@ -696,7 +713,7 @@ def crawl_gateway(cluster: str, infrastructure: str, dev_tmp_dir: Path) -> str:
 
     # Save resulting status file
     with open(status_file, "w+") as fh:
-        json.dump(requests_statuses, fh, indent=2)
+        json.dump(requests_statuses, fh, indent=2, sort_keys=True)
         fh.write("\n")
 
     return token
@@ -742,7 +759,13 @@ def crawl_agent(port: int, token: str, metrics: bool) -> None:
         "stats",
     )
     dump_component_query(
-        requests_statuses, url, f"/v{get_version()}/jobs", headers, assets_path, "jobs"
+        requests_statuses,
+        url,
+        f"/v{get_version()}/jobs",
+        headers,
+        assets_path,
+        "jobs",
+        limit_dump=100,
     )
     nodes = dump_component_query(
         requests_statuses,
@@ -813,7 +836,7 @@ def crawl_agent(port: int, token: str, metrics: bool) -> None:
     # FIXME: Download unknown job/node
     # Save resulting status file
     with open(status_file, "w+") as fh:
-        json.dump(requests_statuses, fh, indent=2)
+        json.dump(requests_statuses, fh, indent=2, sort_keys=True)
         fh.write("\n")
 
 

dev/firehpc/conf/tiny/group_vars/all.yml

@@ -1,7 +1,7 @@
 common_with_devs_repos: true
 common_hpckit_derivatives:
 - main
-- slurm24.05
+- slurm24.11
 slurm_with_jwt: false
 slurm_params:
   PriorityType: priority/multifactor

docs/antora.yml

@@ -8,6 +8,7 @@ asciidoc:
     source-language: asciidoc@
     table-caption: false
     version: 4.0.0
+    api_version: 0.0.40
 nav:
 - modules/overview/nav.adoc
 - modules/install/nav.adoc

docs/modules/conf/examples/agent.ini

@@ -81,8 +81,8 @@ socket=/run/slurmrestd/slurmrestd.socket
 # rather than end users. Slurm-web is officially tested and validated with
 # the default value only.
 #
-# Default value: 0.0.39
-version=0.0.39
+# Default value: 0.0.40
+version=0.0.40
 
 [filters]
 
@@ -100,6 +100,7 @@ version=0.0.39
 # - qos
 # - cpus
 # - node_count
+# - nodes
 jobs=
   job_id
   user_name
@@ -111,6 +112,7 @@ jobs=
   qos
   cpus
   node_count
+  nodes
 
 # List of slurmdbd job fields selected in slurmrestd API when retrieving a
 # unique job, all other fields are filtered out.
@@ -126,7 +128,6 @@ jobs=
 # - partition
 # - priority
 # - qos
-# - required
 # - script
 # - state
 # - steps
@@ -148,7 +149,6 @@ acctjob=
   partition
   priority
   qos
-  required
   script
   state
   steps
@@ -176,7 +176,6 @@ acctjob=
 # - standard_error
 # - standard_input
 # - standard_output
-# - state_reason
 # - tasks
 # - tres_req_str
 ctldjob=
@@ -192,7 +191,6 @@ ctldjob=
   standard_error
   standard_input
   standard_output
-  state_reason
   tasks
   tres_req_str
 

docs/modules/conf/partials/conf-agent.adoc

@@ -159,7 +159,7 @@ the default value only.
 
 
 
-*Default:* `0.0.39`
+*Default:* `0.0.40`
 
 |-
 
@@ -207,6 +207,8 @@ jobs, all other fields arefiltered out.
 
 * `node_count`
 
+* `nodes`
+
 
 |-
 
@@ -242,8 +244,6 @@ unique job, all other fields are filtered out.
 
 * `qos`
 
-* `required`
-
 * `script`
 
 * `state`
@@ -303,8 +303,6 @@ unique job, all other fields are filtered out.
 
 * `standard_output`
 
-* `state_reason`
-
 * `tasks`
 
 * `tres_req_str`

docs/modules/install/pages/quickstart.adoc

@@ -2,9 +2,9 @@
 
 == Requirements
 
-:fn-slurm-version: footnote:slurm-version[Slurm-web {version} actually requires Slurm REST API v0.0.39 available in Slurm 23.02 and above. Please refer to xref:overview:architecture.adoc#slurm-versions[Slurm REST API versions section] for more details.]
+:fn-slurm-version: footnote:slurm-version[Slurm-web {version} actually requires Slurm REST API v{api_version} available in Slurm 23.11 and above. Please refer to xref:overview:architecture.adoc#slurm-versions[Slurm REST API versions section] for more details.]
 
-* Cluster with Slurm >= 23.02 {fn-slurm-version} and
+* Cluster with Slurm >= 23.11 {fn-slurm-version} and
 https://slurm.schedmd.com/accounting.html[accounting enabled]
 * Host installed with a supported GNU/Linux distributions among:
 ** CentOS
@@ -132,16 +132,18 @@ Enable and start `slurmrestd` service:
 
 To check `slurmrestd` daemon is properly running, run this command:
 
-[source,console]
+[source,console,subs=attributes]
 ----
-# curl --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurm/v0.0.39/diag
+# curl --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurm/v{api_version}/diag
 {
    "meta": {
      "plugin": {
-       "type": "openapi\/v0.0.39",
-       "name": "Slurm OpenAPI v0.0.39",
-       "data_parser": "v0.0.39"
-     },
+      "type": "openapi\/slurmctld",
+      "name": "Slurm OpenAPI slurmctld",
+      "data_parser": "data_parser\/v{api_version}",
+      "accounting_storage": "accounting_storage\/slurmdbd"
+    },
+   }
   …
 }
 ----

docs/modules/misc/pages/troubleshooting.adoc

@@ -9,17 +9,17 @@ This page contains troubleshooting tips to help find out the reason of issues.
 Test Slurm `slurmrestd` API is properly responding on Unix socket with this
 command:
 
-[source,console]
+[source,console,subs=attributes]
 ----
-$ curl --silent --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurm/v0.0.39/diag | \
+$ curl --silent --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurm/v{api_version}/diag | \
   jq '.statistics | with_entries(select(.key | startswith("jobs")))'
 {
-  "jobs_submitted": 0,
-  "jobs_started": 0,
-  "jobs_completed": 0,
+  "jobs_submitted": 385,
+  "jobs_started": 407,
+  "jobs_completed": 411,
   "jobs_canceled": 0,
   "jobs_failed": 0,
-  "jobs_pending": 40,
+  "jobs_pending": 0,
   "jobs_running": 0
 }
 ----
@@ -29,9 +29,9 @@ cluster.
 
 Test Slurm accounting on in REST API with this command:
 
-[source,console]
+[source,console,subs=attributes]
 ----
-$ curl --silent --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurmdb/v0.0.39/config | \
+$ curl --silent --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurmdb/v{api_version}/config | \
   jq .clusters[].nodes
 "cn[1-4]"
 ----