Headless Browser

Engines like Google, Baidu, and Bing China require JavaScript rendering. A3S Search provides a BrowserPool that manages a shared headless browser instance (Lightpanda or Chrome) with tab concurrency control.

Browser Backends

A3S Search supports two headless browser backends:

Chrome/Chromium — Available with the headless feature; auto-detected or downloaded as Chrome for Testing.
Lightpanda — Available with the lightpanda feature on Linux/macOS; this feature implies headless.

BrowserBackend::default() selects Lightpanda when the crate is compiled with lightpanda; otherwise it selects Chrome.

Feature Gate

Headless browser support is opt-in in the Rust crate:

# Rust library (default: HTTP engines only)
[dependencies]
a3s-search = "1.3"

# With Chrome/Chromium headless backend
[dependencies]
a3s-search = { version = "1.3", features = ["headless"] }

# With Lightpanda backend (Linux/macOS only; implies headless)
[dependencies]
a3s-search = { version = "1.3", features = ["lightpanda"] }

The Rust crate exposes only two relevant feature flags: headless and lightpanda.

BrowserPool

BrowserPool manages a single browser process (Lightpanda or Chrome) with a tab semaphore for concurrency control:

use a3s_search::browser::{BrowserPool, BrowserPoolConfig, BrowserBackend};
use std::sync::Arc;

let pool = Arc::new(BrowserPool::new(BrowserPoolConfig {
    backend: BrowserBackend::Chrome,
    max_tabs: 4,
    headless: true,
    chrome_path: None,
    proxy_url: None,
    launch_args: vec![],
}));

When compiled with features = ["lightpanda"], BrowserBackend::Lightpanda and lightpanda_path are also available.

Configuration

Prop

Type

Lifecycle

The browser is lazily initialized on the first acquire_browser() call:

let browser = pool.acquire_browser().await?;
pool.shutdown().await;

BrowserFetcher

BrowserFetcher implements the PageFetcher trait using BrowserPool:

use a3s_search::browser::BrowserFetcher;
use a3s_search::WaitStrategy;

let fetcher = Arc::new(
    BrowserFetcher::new(pool.clone())
        .with_wait(WaitStrategy::Selector {
            css: "div.g".into(),
            timeout_ms: 5000,
        })
        .with_user_agent("Mozilla/5.0 ...")
);

Wait Strategies

Control when a page is considered "loaded":

Prop

Type

Each built-in headless engine uses an appropriate strategy:

Prop

Type

Browser Auto-Detection

When chrome_path or lightpanda_path is None, A3S Search automatically detects or downloads the browser:

Lightpanda (Default)

LIGHTPANDA environment variable
System PATH (lightpanda command)
Cache: ~/.a3s/lightpanda/<tag>/lightpanda
Auto-download from GitHub releases (59MB)

Supported platforms: macOS (arm64, x64), Linux (x64, arm64). First run downloads ~59MB, startup <100ms.

Chrome (Fallback)

CHROME environment variable
System PATH (google-chrome, chromium, chrome, etc.)
Well-known install paths (macOS /Applications/..., Linux /usr/bin/..., Windows C:\Program Files\...)
Cache: ~/.a3s/chromium/<version>/
Auto-download Chrome for Testing from Google CDN (200MB)

Supported platforms: macOS (arm64, x64), Linux (x64), Windows (x64, x86). First run downloads ~150MB, startup 1-2s.

Using Headless Engines

use a3s_search::{Search, SearchQuery};
use a3s_search::browser::{BrowserPool, BrowserPoolConfig, BrowserFetcher};
use a3s_search::engines::{Google, Baidu, BingChina, DuckDuckGo};
use a3s_search::WaitStrategy;
use std::sync::Arc;

let pool = Arc::new(BrowserPool::new(BrowserPoolConfig {
    max_tabs: 4,
    ..Default::default()
}));

let google_fetcher = Arc::new(
    BrowserFetcher::new(pool.clone())
        .with_wait(WaitStrategy::Selector {
            css: "div.g".into(),
            timeout_ms: 5000,
        })
);

let baidu_fetcher = Arc::new(
    BrowserFetcher::new(pool.clone())
        .with_wait(WaitStrategy::Selector {
            css: "div.c-container".into(),
            timeout_ms: 5000,
        })
);

let mut search = Search::new();
search.add_engine(DuckDuckGo::new());
search.add_engine(Google::new(google_fetcher));
search.add_engine(Baidu::new(baidu_fetcher));

let results = search.search(SearchQuery::new("rust programming")).await?;
pool.shutdown().await;

PageFetcher Trait

All fetchers implement this trait:

#[async_trait]
pub trait PageFetcher: Send + Sync {
    async fn fetch(&self, url: &str) -> Result<String>;
}

Prop

Type

Headless Browser

On this page