For AI agents: the complete documentation index is available at https://a3s-lab.github.io/ash/v0.1.0/en/llms.txt, the full documentation bundle is available at https://a3s-lab.github.io/ash/v0.1.0/en/llms-full.txt, and this page is available as Markdown at https://a3s-lab.github.io/ash/v0.1.0/en/guide/benchmarks.md.
ash
  • English
  • v0.1.0

      • Benchmarks and quality gates

      ash optimizes total token cost, latency, and recoverability for completed Coding Agent tasks. Every number must come from reproducible corpora, pinned tools, and reports committed with the source.

      Format baseline

      cargo run -p a3s-ash-bench --locked -- \
  --check benches/reports/v0.1.0/format.json

      The report pins corpus SHA-256, tokenizer, source bytes, tokens, ratio, and gate. Current ASON uses about 62% of compact row JSON tokens under both tokenizers; the regression ceiling is 65%.

      Multicore determinism

      cargo build -p a3s-ash --release --locked
cargo run -p a3s-ash-bench --release --locked -- --runtime

      Schema 6 measures eight real paths: literal search, BLAKE3 snapshot, disk spill/fetch, fresh ash run startup, empty child spawn, dual-pipe pressure, process-tree cancellation, and warm framed RPC dispatch. Matrix paths run at 1, 2, 4, 8, and host-available worker counts. Each cold observation starts and reaps the real same-profile binary. Warm dispatch keeps one production RPC session alive and excludes only its handshake. A changed canonical result fails the report. Cold startup has no scaling curve, so its speedup and parallel efficiency are null. Timing remains host-local.

      Native-shell task seed

      cargo run -p a3s-ash-bench --locked -- \
  --check-task-lock benches/tasks/v1/lock.json
cargo run -p a3s-ash-bench --locked -- --tasks

      The seed covers source discovery, diagnostic aggregation, and one exact config mutation. Its lock binds the manifest plus every initial and expected final workspace tree. The runner checks output and final state, then counts objective, command, stdout, and stderr with both tokenizers. It runs the current platform's native shell; it does not run a model or claim an ash task result.

      Current automation

      • Rust unit and integration suites across every workspace crate.
      • cargo check and tests on Linux, macOS, and Windows.
      • Real release builds exercised by all three platform installers.
      • Eight real runtime paths with exact-output validation and bounded process/service cleanup.
      • Three locked tasks with Linux, macOS, and Windows native-shell baselines.
      • Fuzz targets for ASON, bounded ASH/1 frames, typed requests, and update metadata.
      • File-transaction crash cutpoints, update activation/recovery, and cancellation race tests.
      • A locked third-party license inventory.

      Evidence still required for release

      The first supported version still needs real agent-task benchmarks, sustained fuzz-duration records, hardware-labelled latency and throughput reports, and a six-target install/update/rollback run with production credentials.