nirs4all-cluster

Note

Beta · trusted-LAN. nirs4all-cluster is a small distributed job queue for nirs4all.run() — a coordinator that receives jobs and dispatches work to polling workers. It is built for a trusted local network, not the open internet or untrusted multi-tenant use. See Security & scope for the documented limits.

The server and client run without nirs4all installed; only the worker’s subprocess runner imports it. That subprocess boundary buys crash isolation and real cancellation, and keeps the control plane a thin, library-free orchestration layer.

60-second tour

n4cluster server --state ./cluster-state          # coordinator (+ dashboard at /ui)
n4cluster worker --server http://HOST:8765        # one or more workers
n4cluster submit examples/job.shared-path.yaml --wait --out ./results

Getting started

• Installation
  • Server / client hosts
  • Worker hosts
  • Documentation toolchain (optional)
• Quickstart
  • 1. Start the coordinator
  • 2. Start one or more workers
  • 3. Submit a job and wait
  • Python SDK

Concepts

• Architecture
  • The load-bearing invariant
  • A task’s life
• Jobs, tasks and decomposition
  • Level 0 — atomic job
  • Level 1 — pipelines × datasets
• Version compatibility & pipeline fingerprints
  • The handshake
  • Pipeline fingerprints

Reference

• CLI reference
  • n4cluster server
  • n4cluster worker
  • Client commands
• Configuration
  • Environment variables
  • Server settings (ServerConfig)
  • State directory layout
  • Logging
• Job specification & wire contract
  • Pipeline references (kind)
  • Dataset references (kind)
  • Runtime requirement & mixed fleets
  • Examples
  • Schema reference
• Python SDK
  • API
• REST & WebSocket API
  • Authorization & rights
  • Health & dashboard (no auth)
  • Client API
  • Worker API

Running it

• Operations
  • Running the system
  • Recovery model
  • Task state machine
  • Routing
  • Logging & observability
• Web dashboard
• Security & scope
  • Non-goals (this beta does not do these)
  • Where this is headed

Design & internals

• Design & internals