Job specification & wire contract¶

A job is a YAML/JSON document (the wire contract is the Pydantic models in nirs4all_cluster.schemas). Provide exactly one of pipeline/pipelines and one of dataset/datasets; lists decompose into the cartesian product (Jobs, tasks and decomposition).

The Python/CLI adapter adds an optional parity field (DistributedRunParity) to jobs it creates. This is traceability metadata, not a scheduler input: it records that the distributed beta runs one whole nirs4all.run per task, expects metric parity for atomic and explicit matrix jobs, and defers fine-grained DAG/variant/fold/subtree parity until core/dag-ml execution-unit and data-provider contracts exist.

The server also persists additive scheduler/rights metadata:

scheduler (DagSchedulerContract) records whether the request is atomic, an explicit pipeline × dataset matrix, or a DAG-shaped whole-run job. V1 still leases whole nirs4all.run tasks; it does not claim fine-grained graph execution.
submission (JobSubmissionMetadata) is overwritten by the server from the authenticated submitter credential (submit right).
leased TaskPayload.assignment records the server-authoritative executor assignment returned to a worker/client holding execute.
stored TaskResult.provenance records the authenticated executor principal, worker id, job id, task id, attempt, and execute rights used to report the result.

Pipeline references (`kind`)¶

path — a pipeline YAML readable on the worker (shared/worker-local filesystem).
inline_json — the pipeline embedded in the job. The client pins a content fingerprint so the server can confirm the worker ran exactly what was submitted (Version compatibility & pipeline fingerprints).
artifact — an uploaded artifact id (see ClusterClient.upload_artifact).
python_entrypoint — a module:builder callable; gated behind --allow-python-jobs (server) and --allow-python (worker).

Dataset references (`kind`)¶

shared_path, artifact, worker_local (today behaves like shared_path), and catalog (a nirs4all-datasets id/DOI — not implemented in the beta worker).

Runtime requirement & mixed fleets¶

Every task runs a whole nirs4all.run() in a worker subprocess, so a task may only land on a worker that can prove the runtime. On submission the server attests a package-availability requirement for nirs4all.run jobs:

If you pin nothing, the server injects a presence-only requirement (requirements.packages.nirs4all == ""): any declared version qualifies, and a worker that never declared nirs4all is never leased the task (fail-closed).
Pin a range in requirements.packages.nirs4all (e.g. ">=0.9,<0.10") to constrain routing to a compatible library — the server preserves your pin, it does not overwrite it.
Pinning other packages (extra fleet capabilities such as torch) composes with the mandatory nirs4all presence rather than replacing it; a worker must declare every pinned package to be eligible.

Because eligibility is decided from what a worker declares at registration, a fleet can mix plain nirs4all workers with richer runtimes: routing stays correct, and a worker that cannot prove the required packages is simply passed over.

Examples¶

examples/job.shared-path.yaml — atomic job on a shared filesystem¶

# Atomic job (design Level 0): one nirs4all.run() on one worker.
#
# Both the pipeline and the dataset are referenced by paths that every worker
# can see (a shared/mounted filesystem) — the simplest, most realistic case for
# a lab cluster. Adjust the absolute paths to your environment.
#
#   n4cluster submit examples/job.shared-path.yaml --wait --out ./results
type: nirs4all.run
name: pls-grapevine
pipeline:
  kind: path
  path: /home/delete/nirs4all/nirs4all-cluster/examples/pipelines/pls.yaml
dataset:
  kind: shared_path
  path: /home/delete/nirs4all/nirs4all-data/regression/GRAPEVINE_LeafTraits/PSI_spxyG70_30_byCultivar_MicroNIR_NeoSpectra
params:
  random_state: 42
  refit: true
requirements:
  labels: {}
  # Route only to workers that declared a compatible nirs4all (availability +
  # version). The server adds a presence requirement for nirs4all automatically;
  # pin a range here when you need one.
  packages:
    nirs4all: ">=0.9,<0.10"
outputs:
  export_best_model: true
  keep_task_workspace: false
rank_metric: best_rmse
rank_mode: min
retry:
  max_attempts: 2

examples/job.matrix.yaml — one pipeline × three datasets¶

# Matrix job (design Level 1): pipelines x datasets -> one task per combination.
#
# The server decomposes this into N tasks (here 1 pipeline x 3 datasets = 3),
# distributes them across workers, then aggregates a ranking and selects the
# best model. Each task is an independent nirs4all.run() with its own workspace.
#
#   n4cluster submit examples/job.matrix.yaml --wait
type: nirs4all.run
name: pls-corn-targets
pipelines:
  - kind: path
    path: /home/delete/nirs4all/nirs4all-cluster/examples/pipelines/pls.yaml
datasets:
  - kind: shared_path
    name: corn-moisture
    path: /home/delete/nirs4all/nirs4all-data/regression/CORN/Corn_Moisture_80_WangStyle_m5spec
  - kind: shared_path
    name: corn-oil
    path: /home/delete/nirs4all/nirs4all-data/regression/CORN/Corn_Oil_80_WangStyle_m5spec
  - kind: shared_path
    name: corn-protein
    path: /home/delete/nirs4all/nirs4all-data/regression/CORN/Corn_Protein_80_WangStyle_m5spec
params:
  random_state: 42
rank_metric: best_rmse
rank_mode: min
outputs:
  export_best_model: true
retry:
  max_attempts: 2

examples/job.uploaded-bundle.yaml — inline pipeline + uploaded dataset¶

# Uploaded-input job: the pipeline travels inline (no shared filesystem needed)
# and the dataset is an artifact the client uploaded to the server first.
#
# 'inline_json' carries the pipeline config in the job itself; the worker writes
# it to a file and feeds it to nirs4all.run(). For the dataset 'artifact' kind,
# upload a zip first and paste the returned id:
#
#   ID=$(python -c "from nirs4all_cluster import ClusterClient as C; \
#       print(C('http://127.0.0.1:8765').upload_artifact('corn.zip'))")
#   # then set dataset.artifact_id below to $ID and submit.
type: nirs4all.run
name: pls-inline
pipeline:
  kind: inline_json
  inline:
    pipeline:
      - class: sklearn.preprocessing.StandardScaler
      - class: sklearn.cross_decomposition.PLSRegression
        params:
          n_components: 10
dataset:
  kind: artifact
  artifact_id: REPLACE_WITH_UPLOADED_ARTIFACT_ID
params:
  random_state: 42
outputs:
  export_best_model: true

Schema reference¶

class nirs4all_cluster.schemas.JobRequest[source]¶

Bases: BaseModel

A logical job submitted by a client.

Provide exactly one of pipeline/pipelines and one of dataset/datasets. When a list is given the server decomposes the job into one task per (pipeline, dataset) combination (design Level 1).

inferred_scheduler_contract()[source]¶

Infer the additive scheduler contract from the validated request shape.

Return type:: DagSchedulerContract

class nirs4all_cluster.schemas.PipelineRef[source]¶

Bases: BaseModel

How a worker should obtain the pipeline to run.

kind ordering mirrors the design’s preference list. python_entrypoint is only honoured when the server is started with --allow-python-jobs.

class nirs4all_cluster.schemas.DatasetRef[source]¶

Bases: BaseModel

How a worker should obtain the dataset to run on.

class nirs4all_cluster.schemas.Requirements[source]¶: Bases: BaseModel

class nirs4all_cluster.schemas.Outputs[source]¶: Bases: BaseModel

class nirs4all_cluster.schemas.RetryPolicy[source]¶: Bases: BaseModel

class nirs4all_cluster.schemas.DistributedRunParity[source]¶

Bases: BaseModel

Client-declared parity contract for a distributed nirs4all.run job.

The cluster beta distributes only whole nirs4all.run calls: one isolated task workspace per explicit pipeline x dataset pair. Fine-grained DAG grains such as variants, folds, or reusable subtrees are intentionally out-of-scope until the core / dag-ml contracts provide those execution units.

class nirs4all_cluster.schemas.DagSchedulerContract[source]¶

Bases: BaseModel

Additive scheduler contract for DAG-shaped jobs.

Cluster V1 still schedules whole nirs4all.run calls. This metadata makes the rights/result contract explicit for DAG-looking payloads without claiming fine-grained graph execution.

class nirs4all_cluster.schemas.JobSubmissionMetadata[source]¶

Bases: BaseModel

Server-attested provenance for a job accepted through POST /v1/jobs.

The client may send this field for traceability, but the server overwrites it from the authenticated principal before persisting the request. Rights are therefore credential-derived, not self-declared.

class nirs4all_cluster.schemas.TaskAssignmentMetadata[source]¶

Bases: BaseModel

Server-requested execution metadata returned on a worker lease.

class nirs4all_cluster.schemas.ResultProvenance[source]¶

Bases: BaseModel

Server-attested provenance attached to a stored task result.

class nirs4all_cluster.schemas.JobView[source]¶: Bases: BaseModel

class nirs4all_cluster.schemas.JobAggregate[source]¶: Bases: BaseModel

class nirs4all_cluster.schemas.TaskView[source]¶: Bases: BaseModel

class nirs4all_cluster.schemas.TaskResult[source]¶

Bases: BaseModel

Summary a worker reports on task completion. Mirrors design’s JSON.

class nirs4all_cluster.schemas.RunMetrics[source]¶: Bases: BaseModel

class nirs4all_cluster.schemas.ClusterStats[source]¶

Bases: BaseModel

Server-wide counters for the dashboard header and n4cluster tooling.

Job specification & wire contract¶

Pipeline references (kind)¶

Dataset references (kind)¶

Runtime requirement & mixed fleets¶

Examples¶

Schema reference¶

Pipeline references (`kind`)¶

Dataset references (`kind`)¶