Spark-backed CI and development reservations

Butter and Iron can use a shared two-node NVIDIA Spark fleet for heavyweight CUDA correctness, model integration, and performance testing. The same machines also host interactive development and a long-running inference service, so they are not ordinary always-online GitHub runners.

This guide explains the admission model and how a repository opts into it.

Rollout status: the control plane must report the fleet as managed before enabling a Spark workflow. Do not register a persistent runner or change a normal PR job to runs-on: waffuru-spark while the controller reports either host as unknown or unmanaged.

Mental model

GitHub queues source work. The Spark controller owns physical admission.

PR label / scheduled test / development request
                     |
                     v
             Spark reservation queue
                     |
       +-------------+-------------+
       |                           |
   one Spark                  Spark pair
  (prefer Spark 2)       (atomic Spark 1 + 2)
       |                           |
       +-------------+-------------+
                     |
          one-job JIT runner or dstack task
                     |
          result/check attached to exact SHA

Development has the highest priority for the next allocation. It does not cancel an active CI or benchmark job. When development is requested, the target enters draining state: current work finishes and no subsequent CI starts.

A two-Spark request is one atomic reservation. If one host becomes idle first, it is held for the pair instead of accepting another job. This prevents a steady stream of one-node work from starving distributed tests.

Spark 1 normally belongs to the Deckard inference service. The controller drains Deckard, stops it completely, verifies its GPU allocation is gone, and then lends Spark 1. When borrowed work releases, Deckard is recreated and must pass its model-ready health check before it is advertised as online. Container or process pause is not considered a GPU release.

Workload classes

Work	Default shape	Admission order	Trigger
Interactive development	one or two nodes	100	Agent/CLI reservation
Explicit Deckard start	Spark 1	90	User or service reconciliation
Manual PR Spark test	one or two nodes	80	Maintainer PR label/comment
Merge correctness CI	one node	70	Controller-approved workflow
Scheduled benchmark	one or two nodes	50	Controller schedule
Background CI	one node	20	Opportunistic queue

Priority is intentionally non-preemptive. A separately authorized emergency operation may terminate work, but no label, PR, scheduled workflow, or ordinary development request can do so.

Maintainer experience

After repository integration is enabled, a maintainer adds one of these labels to a pull request:

spark-test — run the short CUDA correctness lane on the exact PR SHA on one Spark, preferring Spark 2;
spark-test-long — run correctness followed by the bounded performance lane on one Spark;
spark-pair-test — reserve both Sparks atomically for a test that actually exercises both nodes. Butter and Iron intentionally reject this today until a distributed test exists.

The GitHub check moves through states such as:

Spark test queued — position 2
Spark 2 draining — active job finishing
Spark 2 allocated — environment preparing
Spark test running — exact SHA 7fc2a91
Spark test passed — evidence receipt recorded

Removing the label cancels queued work only. It does not kill an active job. Pushing a commit supersedes an older request that has not started. An active run continues to report the SHA it actually tested.

The optional comment interface uses the same authorization and queue:

/spark test
/spark test --pair
/spark status
/spark cancel

Repository enablement

Enable repositories individually. Complete these steps only after the Spark controller itself is deployed, dstack host enrollment is verified, Deckard restore has passed, and both resources have been explicitly adopted.

1. Install and scope the GitHub App

Install the Spark controller GitHub App on only the approved repositories. Minimum repository permissions are:

Actions: read and write, to dispatch the admitted workflow;
Checks: read and write, to publish queue and result state;
Contents: read, to bind and check out the requested commit;
Metadata: read;
Pull requests: read, for PR identity and head SHA;
Issues: read, if label/comment triggers are enabled.

Subscribe to pull_request and, when comment commands are enabled, issue_comment. Validate every webhook signature. Do not put host credentials, dstack credentials, runner registration tokens, or controller secrets in the repository workflow.

2. Scope the runner group

The organization runner group must allow only the approved repositories. JIT runners receive cumulative labels describing the allocation, for example:

self-hosted
Linux
ARM64
waffuru-spark
spark-single       # or spark-pair

The controller creates the runner only after granting capacity. It processes one job and deregisters. Do not leave a persistent Spark runner online: GitHub could assign work while the host is being returned to development or Deckard.

3. Create the opt-in labels

Create spark-test, spark-pair-test, and spark-test-long. Treat label application as an authorization decision: only organization maintainers should be able to request private hardware for a PR. Never automatically admit an untrusted fork or expose repository secrets to its code.

4. Add a controller-dispatched workflow

The workflow lives on the default branch and is invoked only after the controller grants a reservation. It does not poll the controller or sleep on a runner while waiting for capacity.

name: Spark PR test

on:
  workflow_dispatch:
    inputs:
      reservation_id:
        description: Controller reservation ID
        required: true
        type: string
      head_sha:
        description: Exact reviewed PR commit
        required: true
        type: string
      topology:
        description: single or pair
        required: true
        type: choice
        options: [single, pair]
      suite:
        description: smoke or performance
        required: true
        type: choice
        options: [smoke, performance]

permissions:
  contents: read

concurrency:
  group: spark-${{ inputs.reservation_id }}
  cancel-in-progress: false

jobs:
  spark-test:
    # The controller creates a matching one-job runner only after admission.
    runs-on: >-
      ${{ inputs.topology == 'pair'
          && fromJSON('["self-hosted","Linux","ARM64","waffuru-spark","spark-pair"]')
          || fromJSON('["self-hosted","Linux","ARM64","waffuru-spark","spark-single"]') }}
    timeout-minutes: 120
    steps:
      - name: Check out the admitted commit
        uses: actions/checkout@v6
        with:
          ref: ${{ inputs.head_sha }}
          persist-credentials: false

      - name: Verify commit binding
        shell: bash
        run: test "$(git rev-parse HEAD)" = "${{ inputs.head_sha }}"

      - name: Verify controller lease and machine preflight
        shell: bash
        run: spark-runner-preflight "${{ inputs.reservation_id }}"

      - name: Run the bounded Spark test entry point
        shell: bash
        env:
          SPARK_TEST_SUITE: ${{ inputs.suite }}
        run: ./scripts/ci/spark-test.sh "${{ inputs.topology }}"

Keep repository commands in a versioned scripts/ci/spark-test.sh. The host preflight and lease verification belong to the managed runner image, not code from the PR. The host-side preflight must reject a stale fencing token, unknown resource state, competing GPU process, insufficient disk, failed interconnect for a pair run, or a Deckard process that has not released Spark 1.

The expression above documents the two runner-label shapes. If GitHub rejects a dynamic runs-on expression in the repository’s Actions version, use two jobs with static label arrays and mutually exclusive if conditions instead; do not weaken the labels to a generic self-hosted runner.

5. Add the repository test entry point

The entry point must be deterministic, bounded, and topology-aware. It should:

print toolchain and source revisions;
use the controller-provided work and cache directories;
preserve correctness coverage before performance measurement;
serialize writes to shared build targets;
avoid downloading a model already present in the managed model cache;
emit a compact receipt and store large logs/artifacts outside MongoDB;
clean its disposable work directory on successful release;
never stop services or processes it does not own.

For Butter, a first one-node PR lane should target the Rust CUDA workspace and one explicit model integration filter rather than the entire model matrix. A pair lane should be reserved for a test that genuinely exercises distributed placement or collectives. Do not request both machines merely to run two independent test commands; submit two ordinary one-node reservations instead.

For Iron, use the same workflow contract with the CUDA correctness backend. Keep CPU/codegen/unit work on ordinary Linux CI and Metal correctness on the existing macOS lane. Spark capacity is for tests that require the NVIDIA backend, model residency, or the interconnect.

Initial bounded test definitions

The label-to-suite mapping is deliberately small:

Repository	`spark-test` (target: 30 minutes)	`spark-test-long` (target: 60 minutes)
Butter	Release-mode `wh-butter-cuda` `cuda_smoke`: allocation/upload, generated CUDA dispatch, elementwise ops, RMSNorm, and GEMV against CPU references	The same CUDA smoke, then Qwen3.8 27B NVFP4 serving at a 512-token prompt, 32 decode tokens, one warmup, and two measured runs
Iron	Filtered CUDA correctness for binary add/multiply, GEMV, and RMSNorm against CPU oracles	The same correctness subset, then only GEMV and RMSNorm benches with two warmups and five measured runs

These are signal-producing PR checks, not the daily benchmark matrix. They do not download models, enumerate every dtype/model/concurrency combination, or claim performance stability while another workload owns the GPU. Full prefill/decode/concurrency sweeps remain scheduled work after admission proves the machine is uncontended.

Each repository keeps the executable definition at scripts/ci/spark-test.sh. SPARK_TEST_SUITE=smoke is the short lane and SPARK_TEST_SUITE=performance is the longer lane. The performance lane always runs correctness first. A pair topology exits with an error until a real distributed assertion is implemented; it never silently consumes two nodes to run independent commands.

6. Decide whether the check blocks merging

Start with the Spark check optional and label-triggered. Make it required only for PRs whose paths or labels indicate Spark-only behavior and only after queue latency and restoration reliability are understood. A globally required manual check would leave unrelated PRs permanently pending.

Agent-driven development

Local agents use the same queue as CI. They request a lease through the shared sparkctl/MCP interface, wait for active, and attach to the dstack development environment assigned to that lease. Agents must not bypass a queued reservation with direct automation SSH.

Common requests map as follows:

User request	Reservation
“Test this model and work on integration”	one-node development, Spark 2 preferred
“Reserve both Sparks for development”	one atomic two-node development lease
“Start Deckard”	Spark 1 service request, next after active work
“I’m done with the Sparks”	release request followed by host cleanup acknowledgement

If the controller, host heartbeat, or fencing check is unavailable, automation fails closed. A human break-glass account is separate from the agent path and does not convert an unknown machine into schedulable capacity.

Failure and cancellation behavior

Controller unavailable: no new work starts; an active job may finish.
Host heartbeat stale: mark the resource offline; do not infer idle.
Lease expires with processes present: enter release-required; do not kill unknown development work or admit CI.
One node of a pair fails: fail the pair as a unit and preserve evidence; do not continue with silently degraded topology.
Deckard restore fails: keep Spark 1 out of CI, retry within a bounded policy, and alert with the health-check failure.
PR closes or label is removed: cancel only if queued; active work finishes.
New PR SHA: supersede a queued old-SHA request; never rewrite an active run’s identity.

Observability

The control-room dashboard may display sanitized resource state, reservation kind, topology, queue position, Deckard health, and GitHub linkage. SQLite on the controller remains authoritative for leases. MongoDB is a history and dashboard mirror, not a lock.

Never mirror host addresses, usernames, commands, environment values, model paths, credentials, raw logs, or fencing tokens into the dashboard database.