Skip to content
The Waffuru waffle markWaffuru Docs

Spark CUDA CI

Iron’s Spark lane is an opt-in hardware check driven by the shared Waffuru Spark capacity controller. The machines are also used for development, so a GitHub label requests queued capacity; it does not make a persistent runner claim the host.

Pull request labels

Section titled “Pull request labels”
  • spark-test: one Spark, short CUDA correctness suite, 30-minute target;
  • spark-test-long: one Spark, the same correctness suite followed by a small performance sample, 60-minute target;
  • spark-pair-test: atomic two-Spark reservation. The entry point currently rejects this because Iron has no distributed test yet.

The labels are inert until the controller grants a lease and creates a one-job JIT runner. Removing a label cancels queued work only; active work drains. A development reservation takes priority for the next allocation but does not cancel a running CI job.

Test definition

Section titled “Test definition”

The controller-dispatched workflow sets SPARK_TEST_SUITE and runs:

Terminal window
SPARK_TEST_SUITE=smoke ./scripts/ci/spark-test.sh single
SPARK_TEST_SUITE=performance ./scripts/ci/spark-test.sh single

The smoke lane selects binary add/multiply, GEMV, and RMSNorm CUDA correctness tests. Every selected kernel dispatches on the GPU and compares with a CPU oracle. The performance lane runs correctness first, then measures only GEMV and RMSNorm with two warmups and five samples. This provides a useful PR signal without running the full kernel inventory.

The full matrix remains a scheduled/manual workload. It should include broader kernel families, device metadata, and stable baselines, and must run only after the host-side admission preflight reports no competing GPU owner.

Workflow contract

Section titled “Workflow contract”

The default-branch workflow must accept controller-provided reservation_id, head_sha, topology, and suite inputs. It must check out and verify the exact SHA, call the host-owned spark-runner-preflight, then invoke the script above. The workflow must not poll for capacity or stop Deckard, development sessions, agents, containers, or other processes itself.

Keep CPU/codegen tests on ordinary Linux CI and Metal correctness on its native macOS lane. Never expose repository secrets to untrusted fork code on a self-hosted runner.