Skip to content

[Feature] [router] Add WebAssembly (Wasm) Extensibility #10902

@tonyluj

Description

@tonyluj

Checklist

Motivation

Currently, extensibility is limited to static Rust code or config. This makes it harder for production users to:

  • Experiment with custom routing/worker policies (e.g. SLO-aware load balance, upper-gw-decision load balance, multi-tier offload strategies to upper layer GW).
  • Insertable middleware (e.g. authentication, logging, rate limiting, request/response filter).
  • Perform safe, hot-swappable upgrades of routing logic in production.

WebAssembly (Wasm) provides a strong foundation for safe, portable, and dynamic extensibility:

  • Sandboxing & safety: Policies and middleware run in a restricted VM, preventing crashes or unsafe hosts accesses.
  • Portability: Extensions can be written in multiple languages (Rust, Go, etc.), compiled to Wasm, and run inside sgl-router.
  • Dynamic Deployment: New modules can be loaded, updated, or swapped without recompiling or restarting the router.

Proposed Design

1. Wasm runtime integration

  • Use wasmtime as the primary runtime.
    • Mature, high-performance JIT/AOT support.
    • WASI + async-friendly.
  • [OPTIONAL] Keep an option for a lighter runtime (wasmi) if startup size or pure-Rust dependency footprint is a concern.
pub mod wasm {
  pub struct WasmRegistry {
    runtime: wasmtime::Engine,
    instances: HashMap<String, WasmInstance>,
    policies: HashMap<String, Box<dyn WasmPolicy>>,
    middleware: HashMap<String, Box<dyn WasmMiddleware>>,
    routers: HashMap<String, Box<dyn WasmRouter>>,
  }

  pub trait WasmPolicy: Send + Sync {
    fn select_worker(&self, workers: &[Arc<dyn Worker>], request: &Request) -> Option<usize>;
    fn name(&self) -> &str;
    fn version(&self) -> &str;
  }

  pub trait WasmMiddleware: Send + Sync {
    fn process(&self, request: &mut Request, context: &mut Context) -> Result<(), MiddlewareError>;
    fn name(&self) -> &str;
  }

  /// TODO
}

2. Host <-> Wasm Bridge

  • Define a minimal ABI for communication (initially JSON serialization; later optimize with compact binary formats).
  • Provide host functions (retricted) for:
    • Accessing request metadata.
    • Querying replica health/load.
    • Return policy/router/middleware decisions.

Core Host Functions

// Host functions exposed to WASM modules (just an example — might change later)
pub mod host_functions {
    // Worker management
    pub fn get_worker_count() -> u32;
    pub fn get_worker_url(index: u32) -> String;
    pub fn get_worker_load(index: u32) -> u32;
    pub fn get_worker_health(index: u32) -> bool;
    
    // Request information
    pub fn get_request_text() -> String;
    pub fn get_request_headers() -> String; // JSON
    pub fn get_request_model() -> String;
    
    // Metrics and logging
    pub fn log_info(message: String);
    pub fn log_error(message: String);
    pub fn record_metric(name: String, value: f64);
    
    // External data access
    pub fn get_config(key: String) -> String;
    pub fn http_request(url: String, method: String, body: String) -> String;
}

3. Lifecycle & Management

  • Support hot reload of Wasm modules at runtime.
  • Provide clear error handling: fallback to default Rust logic if a Wasm module fails.
  • Add Optional sandbox limits (CPU time, memory usage).

Loading and Initialization

sequenceDiagram
    participant M as WasmManager
    participant R as Runtime
    participant W as WASM Module
    participant H as Host Functions
    
    M->>R: Load WASM binary
    R->>W: Initialize module
    W->>H: Register host functions
    H->>W: Return function pointers
    W->>M: Module ready
    M->>M: Register in registry
Loading

Execution Flow

sequenceDiagram
    participant R as Router
    participant M as WasmManager
    participant W as WASM Module
    participant H as Host Functions
    
    R->>M: Execute policy/middleware
    M->>W: Call WASM function
    W->>H: Access host functions
    H->>W: Return data
    W->>M: Return result
    M->>R: Return processed result
Loading

4. Security and Resource Management

  • Sandboxing: WASM modules run in isolated environments
  • Resource Limits: Memory and execution time limits
    • max_memory: MB
    • max_execution_time: second
  • Host Function Restrictions: Limited access to system resources

Example Use Cases

Custom Load Balancing Policy

  1. Random policy written in Javascript (just for example)
// A REALLY simple JS random policy
export class WasmRandomPolicy {
    constructor() {
        this.requestCount = 0;
        console.log('WASM Random Policy initialized');
    }

    // main entry point of policy
    selectWorker(workers) {
        this.requestCount++;
        
        console.log(`Random policy selecting worker for request #${this.requestCount}`);

        if (!workers || workers.length === 0) {
            console.log('No workers available');
            return -1;
        }

        // Get healthy workers
        const healthyWorkers = workers
            .map((worker, index) => ({ worker, index }))
            .filter(({ worker }) => worker.health);

        if (healthyWorkers.length === 0) {
            console.log('No healthy workers available');
            return -1;
        }

        // Select a worker
        const randomIndex = Math.floor(Math.random() * healthyWorkers.length);
        const selectedWorker = healthyWorkers[randomIndex];
        
        console.log(`Selected worker index: ${selectedWorker.index} (URL: ${selectedWorker.worker.url})`);

        return selectedWorker.index;
    }

    getPolicyName() {
        return 'wasm_random_js';
    }

    getPolicyVersion() {
        return '1.0.0';
    }

    getStats() {
        return {
            requestCount: this.requestCount,
            policyName: 'wasm_random_js'
        };
    }
}
  1. Compile to Wasm
# wasm_wrapper.c is a wrapper for calling function from Wasm
emcc wasm_wrapper.c \
    -std=c++17 \
    -O3 \
    -s WASM=1 \
    -s EXPORTED_FUNCTIONS="['_malloc', '_free']" \
    -s EXPORTED_RUNTIME_METHODS="['ccall', 'cwrap']" \
    -s MODULARIZE=1 \
    -s EXPORT_NAME="WasmRandomPolicyModule" \
    -o wasm_random_policy.js
  1. Load Wasm file with WasmPolicy and WasmPolicyAdapter in sgl-router
use crate::policies::wasm::{WasmManager, WasmPolicyAdapter};

impl PolicyRegistry {
    pub async fn register_wasm_random_policy(&mut self) -> Result<(), WasmError> {
        let mut wasm_manager = WasmManager::new();
        
        // Load Wasm module from local fs
        wasm_manager.load_module(WasmModuleConfig {
            name: "wasm_random".to_string(),
            path: "/path/to/wasm-random-policy.wasm".to_string(),
            module_type: WasmModuleType::Policy,
            config: serde_json::json!({}),
        }).await?;
        
        let wasm_policy = WasmPolicyAdapter::new(wasm_manager, "wasm_random");
        self.policies.insert("wasm_random".to_string(), Arc::new(wasm_policy));
        
        Ok(())
    }
}

struct WasmPolicyAdapter {
    wasm_manager: WasmManager,
    policy_name: String,
}

impl LoadBalancingPolicy for WasmPolicyAdapter {
    fn select_worker(&self, workers: &[Arc<dyn Worker>], _request_text: Option<&str>) -> Option<usize> {
        // Worker info in JSON format, just for example now
        let worker_infos: Vec<WorkerInfo> = workers
            .iter()
            .map(|worker| WorkerInfo {
                url: worker.url().to_string(),
                health: worker.is_healthy(),
            })
            .collect();
        
        let workers_json = serde_json::to_string(&worker_infos).ok()?;
        
        // Execute
        let selected_index = self.wasm_manager.execute_policy(
            &self.policy_name,
            &workers_json,
            None,
        )?;
        
        if selected_index >= 0 && (selected_index as usize) < workers.len() {
            Some(selected_index as usize)
        } else {
            None
        }
    }
    
    fn name(&self) -> &'static str {
        "wasm_random"
    }
}

Implemation Plan

Phase 1: Core Wasm Infrastructure

  • Integrate Wasm runtime (wasmtime)
  • Implement basic Wasm manager
  • Add basic security and resource management
  • Basic metrics
  • Create host function interface

Phase 2: Middleware (Auth/Billing/RateLimit/PII/Content Mod) Support

  • Implement Wasm middleware trait and adapter
  • Integrate with middleware chain
  • Added adaptation of auth, billing, ratelimit, PII, and content mod

Phase 3: Docs, tests and examples

  • Example Wasm middleware (auth/billing/ratelimit/PII/content mod)
  • Documentation for setting up a Wasm module
  • Integration tests

Phase 4: Policy Support

  • Implement Wasm policy trait and adapter
  • Add configuration support for Wasm policies
  •  Integrate with existing PolicyRegistry
  • Policy docs and examples

Phase 5: Advanced Features

  • Hot reloading support
  •  Performance optimization

Related resources

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions