Spark language specification

Programming how-to (start here): Program with Spark (CLI primary) · IDE status: IDE status (verified ide new|open|save|run|buffer|ask|show + ide keys / ide key; paint = PPM wire, not a language op; show = real spark-engine-show)

Spark is a small, diffable language where AI is a first-class primitive. Programs are .spark files. They are interpreted by the Spark VM, which is written in x86_64 assembly and shipped as machine code (ELF), not as a Python/Rust/C interpreter.

Implementation tiers (owner hierarchy)

1. Machine language — CPU executes the spark binary directly
2. Assembly — asm/spark.s (GAS) → as → ld → ELF
3. HDL — hdl/classify_score.v models parallel label scoring (FPGA optional)

Lexical rules

• Comments: # to end of line
• Strings: "..." with \", \n, \t
• Keywords are ASCII identifiers at statement start
• -> binds a result name
• | may prefix a pipeline step

Statements

model <alias>

Set default model/alias for following calls (fast, code, best, …). Prefer gateway aliases, not vendor strings.

model analyze / compare / improve / build (first-class)

Analyze reachable models, compare on a suite, propose a better config, and write a blueprint. Dry-run uses fixtures only — offline leaderboard numbers. Build does not start train@* (plan + config).

Full methodology: MODEL_ANALYSIS.md (MODEL_ANALYSIS.md in repo).

model analyze "alias-code" -> report
        model analyze all -> catalog_report

        model compare ["fast", "code", "best"] on suite "examples/eval_suite.json" -> comparison

        model improve from report prefer quality -> blueprint
        # prefer: quality | speed | cost | local

        model build blueprint into "out/better-model.md"
        

“All models” = all reachable configured aliases + discovered local vLLM endpoints (read-only) — not every model in existence. Catalog: data/model-catalog.jsonl. Bifrost public probes need Infisical VK_PROBE; 401 → credential unavailable.

Why shape (mandatory): metrics (latency, tokens/s, quality proxy, tool JSON %), failure_modes, concrete reasons[], risks on improve/build.

Example: examples/model_improve.spark.

ask / generate

ask "Explain {topic} in one sentence" {
          topic: "gravity"
        } -> text

        ask stream "Say hi" -> out
        

Dry-run (--dry-run): offline heuristic replies (no network).

Live (--live): OpenAI-compatible POST /v1/chat/completions via companion ./spark-ask-http → AI_GATEWAY_URL (Bifrost). See ASK_LIVE.md (ASK_LIVE.md in repo). Alias from prior model fast|code|best. Public tunnel uses Infisical VK_PROBE; HTTP 401 → credential unavailable. make test never hits the network.

ask probe / gateway probe

Dry Infisical credential check (no network under --dry-run):

ask probe -> info
        gateway probe -> info
        

Companion ./spark-ask-probe --dry|--live. On miss/401 → exit 4 and print credential unavailable.

extract

Inline schema; language expects JSON matching fields.

extract Person {
          name: string
          age: int
        } from "Ada Lovelace was born in 1815" -> person
        

classify (first-class)

Single-label (default) or multi:

classify Intent { support, sales, spam }
          from "My washer is broken"
          min_confidence 0.7
          -> intent

        classify multi Tags { support, sales, spam }
          from message -> tags
        

Dry-run returns { label, confidence, reasons }. Invalid labels fail loud.

listen / speak / voice (first-class)

Provider-agnostic STT/TTS primitives. Extended voice surface (review / code / copy / model / pstn): VOICE.md (VOICE.md in repo).

listen "input.wav" -> transcript
        speak "Hello" -> "out.wav"
        speak with model jessica_like

        voice {
          listen -> user
          classify Intent { support, sales } from user -> intent
          ask "Reply helpfully to: {user}" -> reply
          speak reply
        }

        voice review "clip.wav" -> report
        voice code request "listen classify speak" -> artifact
        voice copy from "src.wav" to my_voice -> model_path
        voice model write my_voice spec { ... } -> path
        voice model load my_voice -> model
        voice pstn status
        voice pstn dial "+15555550100" -> call   # OFF by default
        

Dry-run: listen/speak use stub transcript + minimal WAV write (speak … -> "path" honored). --live forks ./spark-stt-tts (real WAV/mic/synth/local whisper; vendor HTTP OFF unless SPARK_STT_NET / SPARK_TTS_NET / SPARK_SPEECH_NET=1 — see VOICE.md (VOICE.md in repo)). voice review byte-parses real RIFF/WAVE; voice code|copy|model write real artifacts under out/. PSTN never places in --dry-run / make test (companion gated OFF).

pipeline

pipeline {
          ask "Summarize: {doc}" -> summary
          | ask "Translate to Spanish: {summary}" -> es
        }
        

Compose with classify/listen/speak the same way.

tool / with tools

tool weather(city: string) -> string { "stub:local" }

        with tools [weather] {
          ask "Weather in DSM?" -> answer
        }
        

tool registers a name. with tools [name,…] activates that registration for the following block (} clears scope). Dry-run ask inside an active scope returns [tool:<name>] stub:local (not a generic model string). Fail loud if with tools lacks […] or no prior tool registration matching the list.

let / print / set

Bindings and output for reviewable scripts.

review / builder / implement (first-class)

Review AI-written or web-context code without executing it. Builder picks a target level and codegen; implement writes artifacts.

review path "examples/fixtures/sample.js" -> report
        review url "file://examples/fixtures/sample.js" -> report
        review url "https://example.com/app.js" -> report   # needs --allow-net
        review text "function x(){ eval(y); }" -> report

        builder prefer lower request "add classify Intent and wire voice turn" -> patch
        # prefer: lower | higher | auto

        implement patch into "out/program.spark"
        

Report shape: { issues[], complexity, suggested_level: lower|mid|higher, rationale } — review url adds op, source, url, bytes, fetched, eval:false.

Levels - lower — emit/extend Spark that maps to asm VM ops (or .s later) - mid — stay on Spark / C-like surface - higher — Python/JS-ish suggestion layer (out/program.py.txt) — does not replace the asm VM

Safety: review url never evals. Policy A+B (owner 2026-08-31): B default — file:// or bare path → open/read + static scan (no network). A opt-in — remote http(s):// + --allow-net → curl fetch + same static scan. Without --allow-net, remote URLs exit non-zero with a clear error instructing --allow-net — never dial, never a silent stub, never a permanent QUESTION menu.

implement writes codegen into .spark / suggestion files the asm VM can re-run or humans can read.

IDE core (ide)

Asm: asm/ide_ops.s. Examples: examples/ide_hello.spark, examples/ide_ask.spark, examples/ide_show.spark, examples/ide_run_show.spark, examples/ide_ask_show.spark, examples/ide_save_reopen.spark. Verified under --dry-run / make test.

ide new
        ide open "examples/hello.spark" -> opened
        ide buffer -> shown
        ide save "out/ide/saved.spark" -> saved
        ide run -> ran
        ide ask -> reply
        ide show -> shown
        

Parent --dry-run → child --dry-run. Parent --live → child --live. Unknown op / empty run → non-zero exit. Exports ide_buf / ide_buf_len / ide_dirty; calls ide_paint_bind after mutate. Buffer edit (ide new) sets dirty → status strip appends * (out/ide/status_dirty.txt); open/save clear dirty. Keymap n→new is not proven. See IDE status.

IDE keymap (ide keys / ide key)

Asm: asm/ide_keys.s. Example: examples/ide_keys.spark. Script fixture: examples/fixtures/ide/cmds.txt. Verified under --dry-run (rc=0). Open/save use shared ide_buf. Open paints out/ide/editor.ppm; show/w reuses engine window.

ide keys "examples/fixtures/ide/cmds.txt"
        ide key open "examples/hello.spark"
        ide key show
        ide key save
        ide key run
        ide key quit
        

Script tokens (proven only): q/quit, s/save, r/run, o path/open path, w/show. Save token proof: examples/ide_keys_save.spark + examples/fixtures/ide/cmds_save.txt (s → "cmd":"save" under dry-run; no write). Run token proof: examples/ide_keys_run.spark + examples/fixtures/ide/cmds_run.txt (r → "cmd":"run" under dry-run; no fork). Dry-run: JSON traces (+ out/ide/keys_trace.jsonl); open still reads + paints; show dry-validates PPM; save/run do not write/fork. Dispatched before core buffer ops. No mouse GUI. Paint wire (make test-ide-paint → out/ide/editor.ppm) is not a .spark statement — IDE status.

Binary (any machine-level file)

First-class ops to open and analyze binaries (ELF shared objects, kernel modules, firmware blobs). Default ./spark stays syscalls-only. ELF headers are parsed in assembly via open/read/fstat/lseek. Dynsym filters and objdump -d windows are orchestrated by make spark-binary → ./spark-binary-probe (documented shell-out — not a fake in-asm x86 decoder).

binary open "path/to/lib.so" -> bin
        binary elf bin -> elf
        binary disasm bin at offset 0x1000 length 64 -> ops
        binary understand bin -> report
        binary understand bin focus cuda|memory|uvm -> report

        binary kernelmod "module.ko" -> report
        binary firmware "blob.bin" -> report
        

What “understand” / “disasm” write: under out/decompile/<basename>/ — every ELF section, full contents (no .text-only / no sampling / no “relevant sections” shortcut):

• ALL_SECTIONS.contents — objdump -s -w (full hex of every section)
• ALL_SECTIONS.disasm — objdump -D -w (disassemble every section; non-code still listed)
• <section>.raw — full raw bytes via ./spark-section-dump (any size; checkpoint/resume with --resume; NOBITS → empty file)
• lifted/ — C-like lift from ALL_SECTIONS.disasm via ./spark-lift (lifted_common.h, per-section .c, lifted.c index)

Files >32MiB print a progress note and continue the full dump (no gate, no silent partial). Asm forks objdump, spark-section-dump, and spark-lift. Re-running understand when ALL_SECTIONS.disasm + lifted/lifted.c already exist skips objdump/lift and only runs spark-section-dump --resume (checkpoint). Dynsym export names for the understand JSON come from ./spark-binary-probe --exports (mmap) — never the 64KiB asm file peek (that SIGSEGV'd when e_shoff sat past 64KiB, e.g. ~96MiB libcuda.so.1).

kernelmod: ELF header when uncompressed; .ko.zst is noted as compressed (not ELF until decompress). Never rmmod / rewrite.

firmware: size + magic + coarse limits — opaque ISA → no fake instruction semantics.

Example: examples/binary_any.spark (fixtures). CUDA drivers: examples/binary_cuda_drivers.spark.

./spark --dry-run examples/binary_any.spark
        ls out/decompile/tiny_cuda_stub.so/lifted/
        make spark-binary && ./spark-binary-probe --understand \
          /lib/x86_64-linux-gnu/libnvidia-ml.so.1 --focus memory
        

Network

Local pcap analysis. Dry-run never claims a live sniff unless --allow-net-capture is set and CAP_NET_RAW is available.

network capture probe -> info
        network capture interface "lo" duration 5s -> pcap
        network open "examples/fixtures/sample.pcap" -> pcap
        network analyze pcap -> traffic_report
        network explain traffic_report -> text
        

• capture probe: forks ./spark-net-capture --probe — AF_PACKET socket attempt only; always claimed:false; JSON cap_net_raw / ready. No --allow-net-capture needed.
• capture without --allow-net-capture: claimed:false, loads fixture pcap so analyze still works.
• capture with --allow-net-capture: forks ./spark-net-capture (AF_PACKET SOCK_RAW → classic pcap). Needs CAP_NET_RAW (sudo setcap cap_net_raw,cap_net_admin+ep ./spark-net-capture). CAP miss → exit 4, claimed:false.
• open: real open/read of magic 0xa1b2c3d4.
• analyze: DNS QNAME from pcap bytes after open. Without open → error.
• explain: plain-language summary of parsed bytes.

./spark --dry-run examples/network_analyze.spark
        ./spark --dry-run examples/network_capture.spark
        ./spark --dry-run examples/network_capture_probe.spark
        ./spark-net-capture --probe
        ./spark --dry-run --allow-net-capture examples/network_capture.spark
        

Browser / MITM (language SoT)

.spark programs are the browser driver. Asm dispatch in asm/browser_ops.s — not a Python stub.

Canonical entrypoints (only these)

Do not document python3 -m spark_browser run as the product entry (make run-host = Qt debug only).

browser run "examples/browser_main.spark"
        browser flags
        browser goto "https://example.com/"
        browser cdp status
        browser cdp navigate "https://example.com/"
        browser cdp evaluate "document.title"
        browser cdp screenshot "out/browser/cdp-shot.png"
        mitm ca-init
        mitm ca-status
        mitm ca-install
        mitm enable
        mitm smoke
        mitm disable_quic on
        mitm quic status
        mitm quic smoke
        mitm filter "example\\.com"
        mitm har export "out/browser/session.har"
        browser show "examples/fixtures/browser/engine_show.ppm"
        browser engine render
        engine fetch parse "file://examples/fixtures/engine/sample.html"
        engine parse "engine/fixtures/hello.html"
        engine paint fixture
        js eval "1+1"
        

Engine B (Spark asm — product render path)

Not Chromium / not Qt. Real ops only:

Dry layout→pixels: examples/engine_layout_render.spark → out/engine/layout.ppm. Full pipeline: examples/engine_pipeline.spark → out/engine/pipeline.ppm. Table: examples/engine_pipeline_table.spark; fetch→parse→layout: examples/engine_fetch_parse_layout.spark.

HTTPS uses the OpenSSL BIO companion ./spark-engine-fetch-tls under --allow-net. JS phase-1 covers numbers, strings, +, unary -, var num, and console.log.

Live X11 (real window, not dry):

make spark-engine-show
        ./spark --dry-run examples/engine_pipeline.spark   # paints pipeline.ppm
        ./spark --live examples/engine_pipeline.spark      # opens X11 from that PPM
        ./spark-engine-show --ppm out/engine/pipeline.ppm --hold 3000
        

Same dry/--live split: examples/browser_show.spark, examples/browser_engine_render.spark. Dry tests never open X11.

• browser run|open|start — session under out/browser/ (session.json). Dry-run never launches a GUI. Session JSON includes disable_quic:false by default (QUIC ON).
• browser flags — reports disable_quic default / override.
• browser goto — records navigation URL (requires session).
• browser cdp — CDP client to Qt :9222 (status|navigate|evaluate|screenshot). Dry-run returns mocks (claimed:false, never dials). --live forks ./spark-browser-cdp (Python in spark-browser). Screenshot optional path; default out/browser/cdp-shot.png.
• mitm ca-init — forks ./spark-mitm-ca --init (RSA CA under out/browser/ca/ + product spark-browser/data/ca/). Language SoT for MITM trust. Separate from the encrypt-to-model gateway (crypto / encrypt / gateway — see below).
• mitm ca-status — present/missing via helper --status.
• mitm ca-install — dry-run plans only (never auto-trust); --live forks install-ca.sh (NSS / optional --system).
• mitm enable|disable|filter — owner-local intercept (mitm.json). Dry-run session markers only. --live enable forks ./spark-mitm-h2 serve --daemon (CONNECT h2/h1 + capture
• HAR). Qt must attach, not start the forge.
• mitm smoke — forks ./spark-mitm-h2 --smoke (HTTPS forge proof; Spark-owned, no GUI).
• mitm disable_quic on|off — optional Chromium --disable-quic (product default is QUIC ON; use on for TCP h2/h1-only MITM).
• mitm quic status|listen|smoke|divert — HTTP/3 lane; smoke forks ./spark-mitm-quic (aioquic forge + SNI leaves). UDP MITM via divert→listen (CONNECT-UDP returns 501 on Qt TCP proxy).
• mitm har export — writes a real HAR 1.2 file (byte-written). Requires mitm enable. Dry HAR is a synthetic single-entry from goto URL (valid 1.2); live multi-flow HAR comes from ./spark-mitm-h2 session dir (not from Qt).
• browser gui — only with --live; forks ./spark-browser-host (Qt attach-only to Spark MITM on :8877). Dry-run → error. Shim defaults to Chromium --enable-quic. Optional --disable-quic. Debug: --own-mitm (not product).

./spark --dry-run examples/browser_ca.spark
        ./spark --dry-run examples/browser_h2.spark
        ./spark --dry-run examples/browser_quic.spark
        ./spark --dry-run examples/browser_main.spark
        ls out/browser/ca/ca.pem out/browser/session.har
        make test-e2e-browser
        

OS blueprint scaffold (os generate kind browser) remains for docs layout; language ops above are the runtime SoT.

Encrypt gateway (encrypt-to-model)

Off by default. When enabled, ask seals the prompt and sends a GCM envelope to ./spark-enc-gateway, which decrypts inside the gateway process and only then calls Bifrost. Full docs: ENCRYPT_GATEWAY.md (ENCRYPT_GATEWAY.md in repo).

crypto probe -> info
        crypto backend openssl
        crypto keygen -> key
        encrypt gateway enable key
        gateway encrypt on
        encrypt seal text "secret" -> blob
        encrypt open blob -> text
        ask "..." -> reply
        gateway encrypt off
        

crypto backend af_alg fails loud when algif_aead is blacklisted (CVE-2026-31431). Default remains OpenSSL. See ENCRYPT_GATEWAY.md (ENCRYPT_GATEWAY.md in repo).

./spark --dry-run examples/encrypt_gateway.spark
        ./spark --dry-run examples/crypto_probe.spark
        ./spark-enc-gateway self-test
        ./spark-enc-gateway probe
        

CUDA

GPU ops are pure x86_64 asm syscalls in asm/cuda_ops.s — not NVML, not a QUESTION. Runtime path:

1. open /dev/nvidiactl, /dev/nvidia0, /dev/nvidia-uvm
2. ioctl NV_ESC_CHECK_VERSION_STR (0xc04846d2) and NV_ESC_CARD_INFO (0xc90046c8) — numbers from /usr/src/nvidia-*/common/inc/nv-ioctl-numbers.h as data
3. Optional anonymous mmap/munmap path proof
4. Prefer compute on minor 0; refuse reserved minors for prefer/compute when policy marks them off-limits

fb_bytes comes from NV_ESC_CARD_INFO. Prefer minor 0. Refuse prefer gpu 2 when that device is reserved.

cuda probe -> info
        cuda memstat -> stats
        cuda prefer gpu 0
        

cuda pcie / pcie probe (live sysfs)

First-class hardware capability in asm/pcie_ops.s (additive; does not touch voice/browser asm). Reads live PCI link attrs only (measured Gen/width from sysfs):

• /sys/bus/pci/devices/<bdf>/current_link_speed
• current_link_width, max_link_speed, max_link_width

Report fields per GPU: bus_id, gen_current / gen_max, width_current / width_max, measured speed_*_gts, downgraded (true when width or gen < max), role (spark-prefer for minor 0; reserved roles for off-limits minors). Prefer-minor 0; refuse reserved GPU indices as Spark targets. No reboot. No claim that x16 is fixed.

cuda pcie -> report
        pcie probe gpu 0 -> report
        cuda pcie explain -> text
        

explain prints measured causes for a PCIe downgrade (riser / bifurcation / seating / shared lanes) from live /sys data. Fixture snapshot under examples/fixtures/pcie/ documents sample sysfs; runtime always reads live /sys. Example: examples/cuda_pcie.spark.

Optional companion make spark-cuda → ./spark-cuda-probe remains for NVML cross-checks; it is not required for cuda language ops.

memory pin

Real mmap + mlock syscalls in asm/cuda_ops.s. Fail loud with errno= on denial.

memory pin "buffer" size 1M -> buf
        

os (OS blueprints for AI / agents)

Spark generates OS blueprints + educational bootable stubs, not a host OS install. Output: out/os/<name>/. Bounds: OS_DESIGN.md (OS_DESIGN.md in repo).

os design name "agentos" kind ai_agent -> blueprint
        os design name "aikitchen" kind ai_runtime features [scheduler, model_router, sandbox, net] -> blueprint

        os specify blueprint {
          target: x86_64
          memory_model: flat
          ai: { agent_runtime: true, model_slots: 4, tool_bus: true }
          drivers: [serial, framebuffer_stub, virtio_net_stub]
        } -> spec

        os generate spec into "out/os/agentos" -> tree
        os build tree -> image
        os explain blueprint -> text
        

Dry-run os generate copies templates/os/ai_agent/ into the tree. os build prints a dry-run assemble note (never dd, never reboot). Example: examples/os_agentos.spark.

Runtime flags

./spark --dry-run examples/hello.spark
        ./spark --dry-run examples/cuda_mem.spark
        ./spark --dry-run examples/cuda_pcie.spark
        ./spark --dry-run examples/binary_any.spark
        ./spark --dry-run examples/network_analyze.spark
        ./spark --dry-run --allow-net-capture examples/network_capture.spark
        ./spark --dry-run examples/browser_main.spark
        ./spark --live examples/ask_live.spark          # needs gateway
        ./spark --version
        

--dry-run is assembly (syscalls + forked helpers). Live packet capture requires --allow-net-capture and CAP_NET_RAW on ./spark-net-capture (CAP miss → exit 4). Probe: ./spark-net-capture --probe. Live ask requires --live + ./spark-ask-http + gateway env. Optional model endpoint discovery: make model-probe (not in make test). NVML companion make spark-cuda is optional cross-check only. Browser dry E2E (no display): make test-e2e-browser.

Errors

Human-readable: file open failures, usage, and (when live lands) schema / classify mismatch messages. Dry-run uses heuristic stubs so CI needs no keys.