You define the model.Power runs it.

Run language, vision, OCR, embedding, and audio models inside a Rust process or behind an API. Power handles devices, queues, weights, and execution records without rewriting model logic.

Get started See measured performance

$cargo add a3s-power --no-default-features -F embedded-inference

3: CPU, CUDA, and Metal devices
4: backend and embedded paths
Q6_K: active target acceptance

Verify more tokens per forward passMTP · K7 / S7

One decode roundspec_mode = "mtp"

SNAPSHOT
DRAFT
CHECK
ROLLBACK
COMMIT

SNAPSHOTSave rollback state

Keep target, draft, sampler, and decoder state for seven candidate positions.

Draft proposals7 proposals

#0841P1candidate
#0298P2candidate
#1973P3candidate
#0456P4candidate
#0062P5candidate
#1401P6candidate
#0077P7candidate

Target verificationstop on the first mismatch

#0841T1match
#0298T2match
#1973T3match
#0456T4match
#0062T5match
#0981T6corrected
#0077T7skipped

Final output5 matches + 1 correction

#0841O1committed
#0298O2committed
#1973O3committed
#0456O4committed
#0062O5committed
#0981O6committed

State snapshotskeep S1-S7

Resources stay bounded

Power checks memory, devices, and queues before execution. It rejects work that cannot fit instead of hiding a fallback.

Results stay traceable

Model, configuration, device, input, and output are bound into one execution receipt that can be checked later.

Callers set the bar

Clients verify against their own hashes and policy. The server cannot quietly relax the acceptance rules.

Graphs and kernels

Owned byModel + backend

Fix the common shapes, reuse CUDA Graphs, then choose Flash Attention and fused kernels for the batch that actually runs.

Finite shapes
CUDA Graph
Flash Attention
Kernel fusion

Tensor path

Owned byPower

Merge small calls, pass device-resident results between adjacent graphs, and copy once at the final output.

Microbatching
Tensor batches
Device residency
One copy

Speculative decoding

Owned byPower + backend

The draft guesses and the target model checks. Only matching tokens are committed; a mismatch returns to the last correct position.

Target verification
K / S shapes
MTP / FR
Exact rollback

Scheduling and replicas

Owned byPower + host

Queues, continuous batches, and session replicas share one device budget. The host adds GPU priority and CPU affinity.

Device admission
Continuous batching
Session replicas
Host scheduling

Weights and residency

Owned byPower

Read on demand, prefetch, and keep hot weights resident. When cache space runs out, fall back to the original artifact without changing tensor identity.

LFRU
Async prefetch
Residency plans
Partial mirrors

Validation before rollout

Owned byPower + client

Run old and new settings in both orders on the same input. Replace the old path only when speed improves, output matches, and quality passes.

Two-order A/B
Output parity
Quality gates
Hardware record

174.413: median Q6_K steady decode token/s
41.035: request-wide token/s on the MTP quality workload
73.57%: request-wide gain over autoregressive
6 / 6: quality runs passed GPU exclusivity checks

Mode	Quality evidence	Request-wide t/s	Steady t/s
Q6_K / AR · 3×100	67 / 60	23.642	—
Q6_K + MTP / full · 3×100CURRENT	67 / 58	41.035	—
Q6_K + MTP / FR · K7/S6/B11	Matching digest	—	174.413

174.413 token/s is the median of nine 1,024-token steady-decode runs; the 172.723 minimum does not establish a 175 token/s service floor. MTP was 1.736× faster on the 3×100 quality workload, but strict score moved from 60/100 to 58/100, so it is not a lossless default.

Rust library

Embed it in your process

No listener and no child process. Model code uses devices, queues, state, and execution receipts directly.

embedded-inference

Inference service

Keep your existing clients

Use an OpenAI-compatible API for chat, completions, embeddings, and model management, with RA-TLS or vsock when needed.

server + backend

Artifact install

Put models on disk safely

Stream the model, verify size and SHA-256, then commit it once under a cross-process lock.

artifact-provisioning

Power runs the model. It does not redefine it.

Model crates own topology, tokenization, preprocessing, and quality policy. Power supplies devices, scheduling, integrity, privacy, and verification, so a new model does not require a new runtime core.

See the architecture boundary

→
ADMITDo memory and queue capacity fit?
→
EXECUTEWhich device path runs this request?
→
COMMITWhat does this output bind?
ACCEPTDoes the caller accept it?

Start with one model.

Embed the runtime or launch the OpenAI-compatible service. Performance tuning and execution verification use the same interfaces when you need them.

Get started View source