Infer-Sim: An open-source simulator for routing algorithms and cache policies for inference workloads

Motivation

During my time at Morph, one recurring problem was how hard it was to test ideas for inference optimization. Trying a new routing or caching theory often meant pushing code to production, then waiting days to see whether latency improved. Raw logs also made bottlenecks difficult to understand; it was easy to miss a growing queue or struggle to pinpoint why TTFT or latency spiked.

Many inference behaviors only show up under realistic live traffic patterns. That makes them hard to backtest, because inference engines have many tunable parameters and those parameters interact in complex ways.

The goal of this lightweight simulator is to emulate and quickly test different configurations. You can tune any of the following:

1. Mooncake compatible trace dataset and query arrival rate
2. LLM model settings such as quantization parameters and layers
3. Batch size
4. Router policy such as cache aware custom and round robin
5. GPU cluster configuration including custom bandwidth and FLOP specs

You can visualize any of the following:

1. Mean and p95 latency TTFT
2. Cache hits
3. Node utilization
4. Replay the GPU routing decisions
5. Backlog queue
6. Peak queue size

How we approximated each variable

Requests

Requests are replayed from a Mooncake-compatible trace format. Each row includes arrival time, input length, output length, and prefix block hashes.

The arrival gaps can be scaled to stress the system:

arrival_time = recorded_arrival_time * ARRIVAL_SCALE

Lower arrival scale means a hotter replay. Higher arrival scale means a calmer replay.

Prefix Cache

Prompts are represented as blocks. Two requests share a prefix when their leading block hashes match.

For each request, the simulator finds the longest cached prefix available from:

• local HBM
• local host RAM
• peer node over RDMA
• disk

It uses the cached prefix only when loading it is faster than recomputing it.

Prefill

Prefill is modeled as compute-bound:

prefill_time = 2 * active_params * tokens / (flops * MFU)

This captures the intuition that long prompts are expensive because the model has to process every input token.

Decode

Decode is modeled as memory-bound across the active batch:

decode_step_time =
  (active_weight_bytes + batch_kv_bytes) / (hbm_bandwidth * MBU)

The active weights are read once for the batch, while each sequence contributes KV traffic.

Cache Movement

Cache movement is modeled as bandwidth-bound:

cache_load_time = kv_bytes / tier_bandwidth

Different tiers use different bandwidths:

• HBM is effectively local
• host RAM uses PCIe bandwidth
• peer cache uses RDMA bandwidth
• disk uses local disk bandwidth

GPU Cluster

Each node is a group of GPUs serving together with tensor parallelism. Compute, HBM bandwidth, HBM capacity, RAM bandwidth, RDMA bandwidth, and disk bandwidth are aggregated across the GPUs in the node.

Nodes are independent serving replicas. Each node must fit the model in its combined HBM.

Queueing And Batching

Each node has a queue. Requests wait until they can be admitted. Decode runs as a continuous batch up to MAX_BATCH.

Prefill pauses decode in the current model, which approximates prefill-prioritizing schedulers and makes prefill/queue interactions visible.

Try it now

Please try it out and share your feedback with us. We would also love extensions to the open-source repository.

Live demo: inference-sim.vercel.app

GitHub: jwlaboratory/inference-sim

Quickstart:

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python server.py

Open:

http://localhost:8000

Or run the CLI:

python3 simulate.py