A3S GATEWAYAI NATIVE TRAFFIC LAYERAI NATIVE 流量层

AI traffic, governed and measured locally.

在本地治理并测量 AI 流量

Enforce model policy and stream lifecycles, recover across backends, then inspect published TTFT, ITL, and token-goodput evidence.

执行模型策略与流生命周期，在后端间恢复，并查验公开的 TTFT、ITL 与 Token goodput 证据。

See token evidence查看 Token 证据 Explore capabilities查看核心能力
LIVE REQUEST TOPOLOGY实时请求拓扑Policy and health select a bounded stream route策略与健康状态共同选择有界流路径 2 / 3 healthy2 / 3 健康

Capabilities that change how the gateway operates

这些能力改变的不只是转发方式

A3S brings model, stream, backend, and configuration decisions into one validated local request path.

A3S 将模型、流、后端和配置决策收敛到一条经过校验的本地请求路径。

MODEL API

Shared model access

多租户模型 API

Gateway action网关动作

Grant and admit by model

按模型授权与准入

Authenticate locally, enforce grants and capacity limits, then rewrite approved model names.

本地鉴权，执行模型授权与容量限制，再重写已批准的模型名称。

Operational effect运行效果

One request path decides who may use which model and how much capacity they may consume.

同一条请求路径决定谁能使用哪个模型，以及可以占用多少容量。

BACKENDS

Provider instability

模型后端波动

Gateway action网关动作

Select from live runtime state

基于实时状态选择后端

Combine health, circuit state, sticky selection, balancing, failover, and mirroring.

组合健康检查、熔断状态、粘性选择、均衡、故障转移和镜像。

Operational effect运行效果

Failed targets stop receiving work, while fallback ends once a response has started.

故障目标停止接收请求，响应开始后不再执行 fallback。

LONG-LIVED

SSE, WebSocket, gRPC

Gateway action网关动作

Bound the stream, not only the socket

约束整条流，而不只是连接

Separate first-response, idle-stream, and total-operation limits with backpressure and bounded drain.

分别设置首响应、流空闲和总操作时限，并处理背压与有界排空。

Operational effect运行效果

Long sessions remain controlled during upstream stalls, client disconnects, and shutdown.

上游停滞、客户端断连和进程关闭时，长会话仍然可控。

DESIRED STATE

Fleet configuration change

网关集群配置变更

Gateway action网关动作

Compile, then atomically activate

先编译，再原子激活

Validate listeners, references, policy, and health as one complete ACL or Cloud snapshot.

将监听器、引用、策略和健康规则作为完整 ACL 或 Cloud 快照统一校验。

Operational effect运行效果

Rejected state leaves the prior runtime active. In-flight requests keep their starting snapshot.

候选状态被拒绝时继续运行上一版本，在途请求保留启动时快照。

A3S Gateway and NGINX start from different problems

A3S Gateway 与 NGINX 的设计起点不同

NGINX is a mature general-purpose proxy. A3S makes AI workload semantics part of the gateway contract.

NGINX 是成熟的通用代理。A3S 则把 AI 工作负载语义纳入网关契约。

NGINX

Breadth and ecosystem

广泛场景与成熟生态

Strong for web serving, general reverse proxying, transport routing, and module-based deployments.

适合 Web 服务、通用反向代理、传输层路由和模块化部署。

A3S GATEWAY

Workload-aware control

工作负载感知控制

Strong when model policy, stream lifecycle, provider recovery, and desired state must act as one runtime.

适合需要将模型策略、流生命周期、提供方恢复和期望状态统一为一个运行时的场景。

WORKLOAD

Traffic meaning

流量语义

NGINX

Routes HTTP and transport traffic through URI, header, upstream, and proxy directives. Model identity remains application data by default.

通过 URI、header、upstream 和代理指令路由流量。默认情况下，模型身份仍属于应用数据。

A3S DESIGN

Model grants, rewriting, RPM, burst, concurrency, and request identities are first-class runtime policy.

模型授权、重写、RPM、突发、并发和请求身份都是运行时一等策略。

STREAMS

Lifecycle control

生命周期控制

NGINX

Generic buffering plus connect, read, and send timeouts control proxied connections and responses.

通过通用缓冲，以及连接、读取和发送超时控制代理连接与响应。

A3S DESIGN

First-response, idle-stream, and total-operation bounds model distinct stream failure modes.

首响应、流空闲和总操作边界分别对应不同的流故障模式。

STATE

Change activation

变更激活

NGINX

Reload validates configuration, starts new workers, and gracefully retires old workers. Failure keeps the old configuration.

reload 校验配置、启动新 worker 并平滑退出旧 worker。失败时保留旧配置。

A3S DESIGN

Domain references, listeners, policy, and health compile as one snapshot and stay pinned per request.

领域引用、监听器、策略和健康规则被编译为一个快照，并固定到每个请求。

RECOVERY

Backend decisions

后端决策

NGINX

Passive health is built in. Active health and live upstream reconfiguration are NGINX Plus capabilities.

开源版内置被动健康检查。主动健康和 upstream 在线重配置属于 NGINX Plus 能力。

A3S DESIGN

Active and passive health, circuit state, balancing, failover, mirroring, and pre-response fallback share one selection model.

主动与被动健康、熔断、均衡、故障转移、镜像和响应前 fallback 共用一套选择模型。

POLICY

Composition

策略组合

NGINX

Modules, authorization subrequests, and njs scripting compose deployment-specific behavior.

通过模块、鉴权子请求和 njs 脚本组合部署所需行为。

A3S DESIGN

Fifteen built-in policies cover common controls, with typed Rust middleware registered by stable names.

15 种内置策略覆盖常见控制，并可按稳定名称注册类型化 Rust 中间件。

One data plane, six capability areas

一个数据平面，六类核心能力

The current delivery spans protocol handling, model policy, backend recovery, desired state, middleware, and operations.

当前交付覆盖协议处理、模型策略、后端恢复、期望状态、中间件和运维能力。

PROTOCOL PLANEAVAILABLE

Streaming is a lifecycle

把流式传输作为完整生命周期

Full-duplex bodies, trailers, backpressure, verified upstreams, independent stream bounds, and bounded drain.

支持全双工 body、trailer、背压、上游校验、独立流式边界和有界排空。

HTTP/1.1HTTP/2SSEWebSocketgRPCTCPUDP
MODEL POLICYFOUNDATION

OpenAI-compatible request control

OpenAI 兼容请求控制

Models, chat, completions, embeddings, grants, admission, rewriting, identities, and ordered healthy targets.

模型列表、Chat、Completions、Embeddings、授权、准入、重写、身份和有序健康目标。

ROUTING AND BACKENDSAVAILABLE

Health-aware selection and recovery

健康感知的选择与恢复

Host, path, method, header, and SNI rules with priority, four balancing strategies, circuits, failover, and mirroring.

Host、路径、方法、header 和 SNI 规则，配合优先级、四种均衡策略、熔断、故障转移和镜像。

RUNTIME STATEAVAILABLE

Validated configuration lifecycle

经过校验的配置生命周期

Standalone ACL or Cloud snapshots, fail-closed validation, serialized listener reconciliation, atomic activation, and durable recovery.

支持 standalone ACL 或 Cloud 快照、失败关闭校验、串行监听器协调、原子激活和持久恢复。

POLICY PIPELINEAVAILABLE

Built-in policy, typed extension

内置策略与类型化扩展

Ordered policy composition stays visible in ACL. Embedded deployments can register typed Rust middleware under stable names.

有序策略组合在 ACL 中保持可见。嵌入式部署可以按稳定名称注册类型化 Rust 中间件。

Access
访问
API key, Basic auth, JWT, forward auth, IP allowlist
Resilience
韧性
Rate limit, Redis limit, retry, circuit breaker
HTTP control
HTTP 控制
CORS, headers, strip prefix, body limit, compression
Network
网络
TCP filter and typed Rust extensions
OPERATIONSAVAILABLE

Evidence and delivery paths

运行证据与交付方式

Prometheus metrics, traces, JSON logs, machine APIs, discovery providers, signed installers, Docker, Helm, and Cargo.

Prometheus 指标、trace、JSON 日志、机器 API、发现提供方、校验安装器、Docker、Helm 和 Cargo。

Production Candidate, with explicit GA gatesProduction Candidate，GA 门槛清晰可查

The core data plane is ready for controlled production. Managed recovery, long-duration reliability, security review, and adoption evidence remain promotion gates.

核心数据平面可用于受控生产环境；受管恢复、长期可靠性、安全审查和真实采用证据仍是晋级 GA 的门槛。

Check status查看状态

Measure the model stream, not a request-rate proxy

测量模型流，而不是用请求吞吐代替

The primary A3S-versus-NGINX lane decodes every token and publishes TTFT, ITL, TPOT, end-to-end latency, and completed-token goodput.

A3S 与 NGINX 的主要对比会解码每个 Token，并发布 TTFT、ITL、TPOT、端到端延迟和完整 Token 吞吐。

ZERO-DELAY / C1

Single-stream gateway overhead

单流网关开销

32 exact tokens / 1 KiB prompt / no upstream pacing

32 个精确 Token / 1 KiB Prompt / 上游不限速

A3S GATEWAY
TTFT 0.343 ms
ITL P99 40.309 ms
NGINX
TTFT 0.349 ms
ITL P99 40.618 ms
A3S / NGINX

TTFT 0.98× · ITL 0.99×

Token goodput 1.00×
ZERO-DELAY / C64

Current concurrency constraint

当前并发约束

64 concurrent streams isolate scheduler and proxy overhead

64 条并发流用于隔离调度器与代理开销

A3S GATEWAY
TTFT 2.438 ms
ITL P99 40.089 ms
NGINX
TTFT 1.184 ms
ITL P99 40.652 ms
CURRENT BOTTLENECK

TTFT 2.06× · ITL 0.99×

Token goodput 0.92×
PACED STREAM / C64

Model-like first-token pacing

接近模型输出的首 Token 节奏

50 ms first token / 10 ms token cadence / 32 tokens

首 Token 50 ms / Token 间隔 10 ms / 32 个 Token

A3S GATEWAY
TTFT 52.274 ms
ITL P99 12.137 ms
NGINX
TTFT 52.152 ms
ITL P99 12.192 ms
WITHIN THRESHOLD

TTFT 1.00× · ITL 1.00×

Token goodput 1.00×
LONG PROMPT / 256 KIB

Bounded long-context upload

有界长上下文上传

8 concurrent streams / 32 exact paced tokens

8 条并发流 / 32 个精确节奏 Token

A3S GATEWAY
TTFT 52.928 ms
ITL P99 11.627 ms
NGINX
TTFT 52.660 ms
ITL P99 11.695 ms
WITHIN THRESHOLD

TTFT 1.01× · ITL 0.99×

Token goodput 1.00×

Commit fc8aa258 / 4 vCPU EPYC 9V74 / 80 successful raw trials / synthetic regression evidence, not a capacity forecast.提交 fc8aa258 / 4 vCPU EPYC 9V74 / 80 次原始试验全部成功 / 合成回归证据，不代表容量预测。

PROTOCOL REGRESSION MATRIX

Transport throughput stays visible, but it is not a model-latency proxy

协议吞吐仍然公开，但不能代替模型延迟

The separate ten-profile matrix keeps HTTP, streaming, bidirectional, and layer-4 regressions reproducible.

独立的十类流量矩阵继续复现 HTTP、流式、双向和四层协议回归。

HTTPS / HTTP/2

Parallel model API calls

并行模型 API 调用

TLS termination / 4 connections / 16 streams each

TLS 终止 / 4 个连接 / 每连接 16 条流

A3S GATEWAY
47.7k requests/s
P99 2.70 ms
NGINX
26.1k requests/s
P99 3.02 ms
A3S RATE + P99

83% higher throughput / 11% lower P99吞吐高 83% / P99 低 11%

WEBSOCKET

Persistent bidirectional traffic

持久双向流量

64 persistent connections / 32-byte binary echo

64 个持久连接 / 32 字节二进制回显

A3S GATEWAY
70.7k messages/s
P99 1.66 ms
NGINX
84.7k messages/s
P99 3.13 ms
TAIL LATENCY TRADEOFF

17% lower throughput / 47% lower P99吞吐低 17% / P99 低 47%

OPENAI STREAM

Feature-bearing AI path

启用策略能力的 AI 路径

Bounded JSON validation / stream detection / SSE relay

有界 JSON 校验 / 流检测 / SSE 转发

A3S GATEWAY
43.8k streams/s
P99 3.32 ms
NGINX
56.2k streams/s
P99 3.01 ms
FEATURE COST

A3S validation: 22% lower throughput / 10% higher P99A3S 校验成本：吞吐低 22% / P99 高 10%

Commit fbb8ae0e / shared infrastructure / regression evidence, not a capacity forecast.提交 fbb8ae0e / 共享基础设施 / 用于回归判断，不代表容量预测。

Configuration maps directly to runtime behavior

配置直接映射到运行时行为

This complete ACL binds a listener, matches a route, applies admission, and selects a health-checked backend.

这份完整 ACL 绑定监听器、匹配路由、执行准入，并选择经过健康检查的后端。

gateway.aclVALID ACL
mode { kind = "standalone" }
entrypoints "web" {
  address = "127.0.0.1:8080"
}
routers "models" {
  rule        = "PathPrefix(`/v1`)"
  service     = "models"
  entrypoints = ["web"]
  middlewares = ["rate-limit"]
}
middlewares "rate-limit" {
  type  = "rate-limit"
  rate  = 60
  burst = 10
}
services "models" {
  load_balancer {
    strategy             = "least-connections"
    request_timeout      = "30s"
    stream_idle_timeout  = "5m"
    stream_total_timeout = "60m"
    servers = [{ url = "http://127.0.0.1:8000" }]
    health_check {
      path = "/health"; interval = "10s"; timeout = "5s"
      unhealthy_threshold = 3; healthy_threshold = 1
    }
  }
}
MODE

Choose the state source

选择状态来源

standalone reads local ACL. cloud-managed accepts complete snapshots from A3S Cloud.

standalone 读取本地 ACL，cloud-managed 接收 A3S Cloud 的完整快照。

Open the complete example查看完整示例

Cloud-managed or standalone, requests stay local

无论 Cloud 管理还是独立运行，请求始终在本地处理

A3S Cloud distributes complete desired state. Gateway validates, activates, and serves from the local snapshot.

A3S Cloud 下发完整期望状态，Gateway 在本地校验、激活并基于快照处理流量。

STATE SOURCEStandalone ACLLocal file and watch provider本地文件与监听提供方
OR
CONTROL PLANEA3S CloudComplete revisioned snapshots完整且带版本的快照
PREPAREValidate and compileListeners, references, policy, health监听器、引用、策略、健康
ACTIVATEAtomic snapshotPrior runtime retained on rejection拒绝变更时保留上一运行时
SERVELocal request pathRoute, policy, healthy target, stream路由、策略、健康目标、流
Node APIhealth / readiness / metrics / snapshot apply / usage acknowledgement

Put A3S Gateway on the request path

把 A3S Gateway 放到请求路径上

Verified installers select the platform archive, check its SHA-256 and binary version, then install for the current user.

校验安装器会选择对应平台归档，检查 SHA-256 和二进制版本，再安装到当前用户目录。

Read quick start阅读快速开始View source查看源码
$curl --proto '=https' --tlsv1.2 -LsSf https://a3s-lab.github.io/Gateway/install.sh | sh
Platform detection / exact SHA-256 / version check平台检测 / 精确 SHA-256 / 版本检查