Motivation

I identified a significant per-request overhead in the current WASM middleware implementation within sgl-model-gateway, which acts as a bottleneck for high-throughput serving.

The two primary performance issues addressed in this PR are:

1. Memory Allocation Overhead: The runtime currently allocates a new wasmtime::Store and linear memory (via mmap) for every single request.
2. Compilation Overhead: The WASM component is re-compiled from raw bytes (JIT) on every request inside the worker loop.

These operations add milliseconds of latency to every request. This PR introduces Instance Pooling to reuse memory slots and LRU Component Caching to skip redundant compilation, ensuring middleware execution remains near-zero cost.

Modifications

I updated sgl-model-gateway/src/wasm/runtime.rs to implement the following optimizations:

1. Instance Pooling:

   • Integrated wasmtime::PoolingAllocationConfig into the worker loop.
   • The system now pre-allocates memory slots (configured to 20 per worker thread) to avoid expensive OS memory allocation calls during request processing.
   • Aligned memory limits (max_memory_size, max_component_instance_size) with the new pooling strategy.

2. Smart Component Caching (LRU):

   • Replaced the naive HashMap strategy with a Least Recently Used (LRU) Cache (using the lru crate).
   • This prevents "cache stampedes" (where clearing a full cache causes a sudden latency spike) by gracefully evicting only the oldest unused modules when the limit is reached.
   • Optimized memory ownership to avoid unnecessary cloning of large WASM binaries during cache insertion.

Benchmarking and Profiling

I performed a local micro-benchmark simulating 1000 sequential requests to measure the full impact of the Instance Pooling + Caching strategy.

Benchmark Configuration:

• Scenario: Full request pipeline simulation (Compilation check + Instantiation).
• Module: Simple WASM module requiring 1 Memory Page.
• Iterations: 1000.

Local Results:

The baseline demonstrates the severe cost of re-compiling modules and re-allocating memory on every request (~370µs/op). The optimized pipeline reduces this to a negligible ~6µs/op by leveraging the pre-warmed cache and pre-allocated memory pool.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process