For AI agents: the complete documentation index is available at https://a3s-lab.github.io/ash/en/llms.txt, the full documentation bundle is available at https://a3s-lab.github.io/ash/en/llms-full.txt, and this page is available as Markdown at https://a3s-lab.github.io/ash/en/guide/benchmarks.md.
ash
  • English
  • next

      • Benchmarks and quality gates

      ash optimizes total token cost, latency, and recoverability for completed Coding Agent tasks. Every number must come from reproducible corpora, pinned tools, and reports committed with the source.

      Format baseline

      cargo run -p a3s-ash-bench --locked -- \
  --check benches/reports/v0.1.0/format.json

      The report pins corpus SHA-256, tokenizer, source bytes, tokens, ratio, and gate. Current ASON uses about 62% of compact row JSON tokens under both tokenizers; the regression ceiling is 65%.

      The same report compares all six retained-result formulas. Canonical / # ? - | > uses 80 tokens in both tokenizers, matching the direct ASCII-letter floor; direct Greek uses 86/86 and the former wrapper uses 97/98. The canonical form is 84% of the wrapper bytes and 83%/82% of its tokens.

      The report also calls the production ×N repeated-line reducer directly. It projects 8,192 deterministic diagnostic lines into 128 lines at about 1% of the source bytes and tokens under both tokenizers; the checked ceiling is 5%. The projection reduces context while a result reference preserves the exact source.

      An independent ×N#K block gate repeats 32 six-line blocks 64 times each. It projects 12,288 retained lines into 224 lines at about 2% of the source bytes and tokens under both tokenizers, below the same 5% ceiling.

      The ⋯N failure-diagnostic gate places 32 error anchors into 8,192 unique lines. Fixed edge and [-2, +6] windows project them into 325 lines at about 4% of source bytes and tokens under both tokenizers, below the same 5% ceiling. Every omitted line remains available through the exact stream reference.

      Multicore determinism

      cargo build -p a3s-ash --release --locked
cargo run -p a3s-ash-bench --release --locked -- --runtime

      Schema 14 contains twenty-two scenarios. Fifteen end-to-end paths cover recursive directory traversal, literal search, regular-expression search, BLAKE3 snapshot, disk spill/fetch, paired disk I/O with idle and fully occupied compute, fresh ash run startup, empty child spawn, three dual-pipe capture profiles, process-tree cancellation, warm framed RPC dispatch, and | column projection over retained ASON. Structured projection reads and decodes a 16,384-row source table, then selects six columns in stable order on the configured Rayon pool; every worker count must emit identical rows and canonical bytes. Four isolated primitives measure 4,096 hot lookups in a 1,024-entry path dictionary plus DAG validation and scheduling at 64, 256, and 1,024 nodes. Three multicore reducer scenarios arrange separate 131,072-line fixtures as consecutive 512-line runs, eight-line blocks repeated 64 times, and sparse failure diagnostics. Every worker setting must produce identical ×N, ×N#K, and ⋯N text and counts. Block search verifies exact bytes before omission; diagnostic classification keeps source-ordered fixed windows. List and search split the fixture into disjoint roots so traversal and scanning enter the bounded worker pool. Listing reports item throughput without inventing content-byte throughput. Matrix paths run at 1, 2, 4, 8, and host-available worker counts. Dictionary and DAG primitives use one caller and have null scaling fields. Each cold observation starts and reaps the real same-profile binary. Warm dispatch keeps one production RPC session alive and excludes only its handshake. Changed canonical evidence fails the report; the intentionally equivalent literal and regex queries must also emit the same result. Timing remains host-local.

      All three capture profiles write 8 MiB to each of stdout and stderr and verify the final 64 KiB of each disk-backed retained value. steady writes consecutive 16 KiB chunks. fragmented cycles through [1, 7, 31, 257, 4093, 16384, 65521] and flushes each write. bursty cycles through [512, 4096, 16384, 65536], then flushes and pauses for 2,000 microseconds at every 256 KiB boundary. The report carries these parameters and a profile-descriptor digest; the compiled fixture must describe the same configuration before timing begins.

      io-spill-idle-compute and io-spill-saturated-compute consume the same 8 MiB input and time only zero-memory-ceiling asynchronous writes plus the final flush. The saturated side keeps every Rayon worker inside an integer workload and proves all workers remain active when I/O completes. Only then does it stop the load, compute BLAKE3, and verify the same 64 KiB tail. Sample order alternates; the report emits a saturated/idle p50 ratio per worker configuration without a cross-host threshold.

      ASH/native-shell task corpus

      cargo run -p a3s-ash-bench --locked -- \
  --check-task-lock benches/tasks/v1/lock.json
cargo run -p a3s-ash-bench --locked -- --tasks

      Seven small contracts cover search, diagnostic aggregation, exact patching, recursive listing, ordered multi-file reading, a guarded file transaction, and an independent batch graph. Each starts twice from the same locked visible tree: once in the current platform's native shell and once as typed requests through the production ASH session.

      The schema-2 report verifies semantic output and final state. Native cost is the objective plus command, stdout, and stderr; ASH cost is the same objective plus every canonical ASON request and response. Requests and responses carry stable per-step and transcript hashes. The plans are hand-authored, so the report marks itself deterministic-tool-plan and agent_results: false. It is protocol accounting, not a model score or a Token-reduction claim; on these tiny tasks, structured metadata and digest safety are visible overhead. The embedded session handshake is not tokenized; a future model report must add amortized primer and format-instruction cost.

      Real Agent traces

      The repository now defines a strict paired trace format. Every repetition must run ASH and the native shell with the same model, seed, and task order while recording the driver digest, model revision, primers, sampling settings, provider usage, and SHA-256 of every tool result.

      cargo run -p a3s-ash-bench --locked -- \
  --validate-agent-trace ./trace.json
cargo run -p a3s-ash-bench --release --locked -- \
  --agent-trace ./trace.json --allow-native-agent-exec > report.json

      Replay starts from isolated copies of the locked fixtures. Model-selected ASON requests enter the production ExecutionSession; native scripts execute only after explicit operator authorization. Every result digest is checked before the semantic answer, expected files, and complete visible tree are verified. The primary total is provider input plus visible output: cached input remains charged, hidden reasoning is reported separately and excluded. Two pinned tokenizers also count primers, objectives, requests, tool results, and final answers. Failed tasks remain in the report.

      Replay proves internal consistency, not provider identity. Reports therefore carry provenance: external-self-attested-trace and provider_attestation_verified: false; published results must also include the adapter digest and any available provider-side audit evidence.

      This is trace-validation and replay infrastructure. The repository does not yet publish a real-model score and never relabels a deterministic plan as one.

      Current automation

      • Rust unit and integration suites across every workspace crate.
      • cargo check and tests on Linux, macOS, and Windows.
      • Real release builds exercised by all three platform installers.
      • Fifteen end-to-end runtime paths, three multicore output reducers, hot path-dictionary lookup, and three-size DAG scheduling evidence.
      • Seven locked tasks with production ASH traces and Linux, macOS, and Windows native-shell baselines.
      • Strict paired model-trace schema, provider Token accounting, result-digest checks, and final-tree replay.
      • Twice-weekly AddressSanitizer targets for ASON, bounded ASH/1 frames and typed requests, arbitrary update metadata, and validly signed update decisions. Every run continues a bounded corpus and retains a source-bound summary, raw log, final corpus, findings, and artifact digest for 90 days.
      • Thirty real file-transaction forward crash cutpoints, twelve recovery re-entry cutpoints, exhaustive four-node DAG propagation, update activation/recovery, and cancellation race tests.
      • A locked third-party license inventory.

      Evidence still required for release

      The first supported version still needs captured and published real Agent-task results, sustained fuzz-duration records, hardware-labelled latency and throughput reports, and a six-target install/update/rollback run with production credentials.