Proxy pool with rotation strategies, dynamic providers, and per-request proxy management

Proxy

A3S Search includes a proxy pool for rotating proxies across requests. Useful for avoiding rate limits and IP bans when scraping search engines.

PooledHttpFetcher

PooledHttpFetcher implements PageFetcher and picks a proxy from the pool for each request. Falls back to direct connection when the pool is disabled or empty.

use std::sync::Arc;
use a3s_search::{PooledHttpFetcher, PageFetcher};
use a3s_search::proxy::{ProxyPool, ProxyConfig, ProxyStrategy};

let pool = Arc::new(ProxyPool::with_proxies(vec![
    ProxyConfig::new("proxy1.example.com", 8080),
    ProxyConfig::new("proxy2.example.com", 8080),
]).with_strategy(ProxyStrategy::RoundRobin));

let fetcher: Arc<dyn PageFetcher> = Arc::new(
    PooledHttpFetcher::new(Arc::clone(&pool))
        .with_timeout(std::time::Duration::from_secs(15))
);

Prop

Type

Static Proxy List

use a3s_search::proxy::{ProxyPool, ProxyConfig, ProxyProtocol, ProxyStrategy};

let pool = Arc::new(ProxyPool::with_proxies(vec![
    ProxyConfig::new("proxy1.example.com", 8080),
    ProxyConfig::new("proxy2.example.com", 8080)
        .with_protocol(ProxyProtocol::Socks5),
    ProxyConfig::new("proxy3.example.com", 3128)
        .with_auth("username", "password"),
]).with_strategy(ProxyStrategy::RoundRobin));

let fetcher: Arc<dyn PageFetcher> = Arc::new(PooledHttpFetcher::new(Arc::clone(&pool)));
search.add_engine(DuckDuckGo::with_fetcher(DuckDuckGoParser, fetcher));

from a3s_search import A3SSearch

search = A3SSearch()

# Set instance-level proxy pool — persists across all searches
await search.set_proxy_pool([
    "http://proxy1.example.com:8080",
    "socks5://proxy2.example.com:1080",
    "http://username:password@proxy3.example.com:3128",
])

response = await search.search("rust programming")

import { A3SSearch } from '@a3s-lab/search';

const search = new A3SSearch();

// Set instance-level proxy pool — persists across all searches
await search.setProxyPool([
  'http://proxy1.example.com:8080',
  'socks5://proxy2.example.com:1080',
  'http://username:password@proxy3.example.com:3128',
]);

const response = await search.search('rust programming');

Dynamic Provider

Implement ProxyProvider to fetch proxies from an external source (API, Redis, database). This is a Rust-only API — SDK users can call set_proxy_pool() to update proxies at any time.

use a3s_search::proxy::{ProxyProvider, ProxyPool, ProxyConfig, spawn_auto_refresh};
use async_trait::async_trait;
use std::{sync::Arc, time::Duration};

struct MyProxyApi {
    url: String,
}

#[async_trait]
impl ProxyProvider for MyProxyApi {
    async fn fetch_proxies(&self) -> a3s_search::Result<Vec<ProxyConfig>> {
        // Fetch from your API — format is up to you
        Ok(vec![
            ProxyConfig::new("dynamic-proxy.example.com", 8080),
        ])
    }

    fn refresh_interval(&self) -> Duration {
        Duration::from_secs(60)
    }
}

let pool = Arc::new(ProxyPool::with_provider(
    MyProxyApi { url: "https://api.example.com/proxies".into() }
));

// Background task refreshes proxies at the provider's interval
let _handle = spawn_auto_refresh(Arc::clone(&pool));

spawn_auto_refresh returns a JoinHandle that can be aborted to stop refreshing.

from a3s_search import A3SSearch

search = A3SSearch()

# Fetch from your API and call set_proxy_pool() to update at any time
import httpx

async def refresh_proxies():
    async with httpx.AsyncClient() as client:
        resp = await client.get("https://api.example.com/proxies")
        proxies = [f"http://{p['host']}:{p['port']}" for p in resp.json()]
        await search.set_proxy_pool(proxies)

# Call on startup and periodically
await refresh_proxies()
response = await search.search("rust programming")

import { A3SSearch } from '@a3s-lab/search';

const search = new A3SSearch();

// Fetch from your API and call setProxyPool() to update at any time
async function refreshProxies() {
  const resp = await fetch('https://api.example.com/proxies');
  const data = await resp.json();
  await search.setProxyPool(data.map((p: any) => `http://${p.host}:${p.port}`));
}

// Call on startup and periodically
await refreshProxies();
setInterval(refreshProxies, 60_000);

const response = await search.search('rust programming');

Runtime Toggle

Enable or disable the proxy pool at runtime without recreating it.

Thread-safe via AtomicBool — works through Arc<ProxyPool>:

let pool = Arc::new(ProxyPool::with_proxies(vec![...]));

pool.set_enabled(false);  // direct connection
pool.set_enabled(true);   // re-enable rotation
pool.is_enabled();        // check status

search.set_proxy_pool_enabled(False)  # direct connection
search.set_proxy_pool_enabled(True)   # re-enable rotation
print(search.is_proxy_pool_enabled())
print(await search.proxy_pool_size())

search.setProxyPoolEnabled(false);  // direct connection
search.setProxyPoolEnabled(true);   // re-enable rotation
console.log(search.isProxyPoolEnabled());
console.log(await search.proxyPoolSize());

Per-Request Proxy Pool

Override the instance proxy pool for a single search call.

Pass a PooledHttpFetcher or HttpFetcher directly to the engine constructor:

let fetcher: Arc<dyn PageFetcher> = Arc::new(PooledHttpFetcher::new(pool.clone()));
search.add_engine(DuckDuckGo::with_fetcher(DuckDuckGoParser, fetcher));

response = await search.search("rust", proxy_pool=[
    "http://per-request-proxy:8080",
])

const response = await search.search('rust', {
  proxyPool: ['http://per-request-proxy:8080'],
});

Rotation Strategies

Prop

Type

use a3s_search::proxy::ProxyStrategy;

let pool = ProxyPool::with_proxies(proxies)
    .with_strategy(ProxyStrategy::Random);

ProxyConfig

let proxy = ProxyConfig::new("proxy.example.com", 8080)
    .with_protocol(ProxyProtocol::Socks5)
    .with_auth("user", "pass");

proxy.url()  // -> "socks5://user:pass@proxy.example.com:8080"

Prop

Type

ProxyPool API

Prop

Type

With Browser Engines

Pass a proxy URL to BrowserPoolConfig:

use a3s_search::browser::BrowserPoolConfig;

let config = BrowserPoolConfig {
    max_tabs: 4,
    headless: true,
    chrome_path: None,
    proxy_url: Some("http://proxy.example.com:8080".into()),
    launch_args: vec![],
};

All browser traffic (including headless engines) routes through the proxy.

Proxy

On this page