Build a CPU-only embedding pipeline with ONNX Runtime: batching, pooling, INT8 quantization, a quality check, and a benchmark script for your hardware.
By FELFAL Bouabid · October 17, 2026
Text embeddings power semantic search, caching, clustering and retrieval, and most tutorials assume you have a GPU. You often do not need one. A small encoder such as all-MiniLM-L6-v2, exported to ONNX and run on an ordinary CPU, can turn text into vectors fast enough for query-time embedding, modest corpora and privacy-sensitive or offline setups.
This article builds a compact, correct CPU embedding pipeline: tokenizer, batching, ONNX Runtime, mask-aware pooling and normalization. It then covers quantization, a quality check that catches silent damage, and a benchmark script that measures your own hardware. The code works with any BERT-style encoder exported to ONNX, and the examples use all-MiniLM-L6-v2.
A word about 'fast'. It is relative to your hardware, your text lengths and your model. I give you no throughput figures here, because a number from someone else's machine is not a number for yours. Section 7 shows how to produce your own.
1. When CPU embeddings make sense
A good fit:
- Embedding queries one at a time, where a small model answers in milliseconds and a network round trip to a GPU service would cost more.
- Corpora from thousands to a few million short documents, where a one-off or nightly job is acceptable.
- Offline, on-device or air-gapped deployments, and data that must not leave your servers.
- Teams with no GPU budget or no appetite for operating GPU machines.
A poor fit:
- Very large models. Cost grows with model size, and a CPU falls behind quickly once you leave the small-encoder class.
- Huge corpora with tight deadlines, where batch throughput on a GPU or a hosted embedding API may be cheaper per vector. Compare cost per million vectors on your own setup before deciding.
- Anything that needs fine-tuning. Training stays a GPU job, even if serving does not.
2. Anatomy of the pipeline
- Tokenize. Split text into word pieces and map them to integer ids. Use the Rust tokenizers library, which is fast and reads the model's own tokenizer file.
- Pad per batch. Models take rectangular input, so shorter texts are padded to the longest in their batch. An attention mask marks which positions are real.
- Run the encoder. ONNX Runtime executes the graph and returns one vector per token.
- Pool. Average the token vectors, using the mask so padding does not count. For this model family that is mean pooling.
- Normalize. Scale each vector to unit length. Then a plain dot product equals cosine similarity.
Most mistakes live in steps 2 and 4. Forgetting the mask makes the embedding of a text depend on what else happens to be in its batch.
3. Get the model
The official all-MiniLM-L6-v2 repository on Hugging Face ships ONNX files in an onnx folder: a full-precision model.onnx of about 90 MB, graph-optimized variants (O1 to O4), and quantized variants of about 23 MB built for specific CPU families, named for arm64, avx2, avx512 and avx512_vnni. Quantized graphs are tied to a CPU family, so a wrong choice can mean a load failure or a silent slowdown. To see what your x86 machine supports:
grep -o -E -w 'avx2|avx512f|avx512_vnni' /proc/cpuinfo | sort -uInstall the dependencies and fetch the full-precision model plus its tokenizer:
pip install onnxruntime tokenizers numpy huggingface_hubfrom huggingface_hub import hf_hub_download
repo = 'sentence-transformers/all-MiniLM-L6-v2'
model_path = hf_hub_download(repo, 'onnx/model.onnx')
tokenizer_path = hf_hub_download(repo, 'tokenizer.json')
print(model_path)
print(tokenizer_path)This model produces 384-dimensional vectors, and its graph returns one vector per token, so pooling and normalization are your job. If you use a different export whose graph already outputs a pooled sentence embedding, skip the pooling step and keep the normalization.
4. The embedder
Save this as embedder.py:
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer
def mean_pool_normalize(hidden, mask):
# hidden: [batch, tokens, dim]; mask: [batch, tokens] with 1 for real tokens, 0 for padding
m = mask[:, :, None].astype(hidden.dtype)
mean = (hidden * m).sum(axis=1) / np.clip(m.sum(axis=1), 1e-9, None)
norm = np.linalg.norm(mean, axis=1, keepdims=True)
return (mean / np.clip(norm, 1e-12, None)).astype(np.float32)
class OnnxEmbedder:
def __init__(self, model_path, tokenizer_path, *, max_length=256, threads=0, batch_size=32):
opts = ort.SessionOptions()
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
opts.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
opts.intra_op_num_threads = threads # 0 lets onnxruntime choose
self.session = ort.InferenceSession(
model_path, opts, providers=['CPUExecutionProvider'])
self.input_names = {i.name for i in self.session.get_inputs()}
self.tokenizer = Tokenizer.from_file(tokenizer_path)
self.tokenizer.enable_truncation(max_length=max_length)
self.tokenizer.no_padding() # we pad by hand, per batch
pad_id = self.tokenizer.token_to_id('[PAD]')
self.pad_id = 0 if pad_id is None else pad_id
self.batch_size = batch_size
self.dim = self._run_batch([[self.pad_id + 1]]).shape[1] # warm-up and dimension probe
def _run_batch(self, id_lists):
longest = max(len(ids) for ids in id_lists)
ids = np.full((len(id_lists), longest), self.pad_id, dtype=np.int64)
mask = np.zeros((len(id_lists), longest), dtype=np.int64)
for row, seq in enumerate(id_lists):
ids[row, :len(seq)] = seq
mask[row, :len(seq)] = 1
feeds = {'input_ids': ids, 'attention_mask': mask}
if 'token_type_ids' in self.input_names:
feeds['token_type_ids'] = np.zeros_like(ids)
hidden = self.session.run(None, feeds)[0]
return mean_pool_normalize(hidden, mask)
def embed(self, texts):
if not texts:
return np.zeros((0, self.dim), dtype=np.float32)
encodings = self.tokenizer.encode_batch(list(texts))
# Sort by token count so each batch holds similar lengths: less padding, less wasted compute.
order = sorted(range(len(encodings)), key=lambda i: len(encodings[i].ids))
chunks = []
for start in range(0, len(order), self.batch_size):
batch = order[start:start + self.batch_size]
chunks.append(self._run_batch([encodings[i].ids for i in batch]))
sorted_vectors = np.concatenate(chunks)
vectors = np.empty_like(sorted_vectors)
vectors[order] = sorted_vectors # restore the caller's order
return vectorsUse it like this:
from embedder import OnnxEmbedder
emb = OnnxEmbedder(model_path, tokenizer_path, max_length=256, batch_size=32)
vecs = emb.embed(['How do I reset my password?', 'I forgot my password, how can I change it?'])
print(vecs.shape) # (2, 384)
print(float(vecs[0] @ vecs[1])) # cosine similarity, because rows have unit lengthWhy it is written this way:
- Tokenization happens once, up front, with no padding. The Rust tokenizer handles the whole list quickly, and the code pads each batch itself.
- Texts are sorted by token count before batching. Padding a 5-token text to match a 200-token neighbor wastes most of the compute for that row. Sorted batches hold similar lengths.
- The order is restored at the end. Callers get vectors in the order they passed texts, regardless of the internal sorting.
- Pooling uses the mask. Padding positions never contribute, so a text's vector does not depend on its batch.
- token_type_ids is filled with zeros only if the graph asks for it. BERT-style exports usually declare three inputs, and a missing required input is an error.
- One warm-up run happens in the constructor. The first call to a session is slower than later ones, so do it before any timing or serving.
- max_length truncates. Match it to the model's configured limit, which is 256 word pieces for this model in the Sentence Transformers setup. Text beyond it is silently dropped, so split long documents into chunks.
For millions of documents, call embed on chunks of a few thousand texts, so the encodings do not all sit in memory at once.
5. Speed levers, and how to treat them
Each lever below is a hypothesis about your machine, not a guarantee. Change one at a time and measure it with the script in section 7.
- Batch size. Larger batches usually improve throughput and worsen the latency of any single request. For interactive queries, try batches of one to eight. For offline jobs, try 32 or more.
- Threads. The threads parameter sets intra-op parallelism. Zero lets ONNX Runtime pick. In a container or on a shared host, set it explicitly to the number of cores you are actually allowed to use, because the runtime may see more cores than your quota, and oversubscribed threads can make things slower.
- Several processes instead of many threads. For small batches, multiple worker processes each with few threads can use a machine better than one process with many threads. Whether it does is an empirical question.
- Maximum length. The cost of a text grows with its token count, and attention grows faster than linearly. Do not set max_length higher than your texts need.
- Quantization. Converting weights to 8-bit integers shrinks the model to roughly a quarter of its size, and is often faster on CPUs with good integer instructions. It also changes the numbers, so see section 6 before relying on it.
- Caching. The cheapest embedding is the one you do not compute. Key a cache on a hash of the normalized text, and you will skip repeated texts entirely. The same idea scales up to semantic caching for LLM answers.
To quantize the full-precision model yourself:
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic('model.onnx', 'model_int8.onnx', weight_type=QuantType.QInt8)Onnx Runtime prints a suggestion to run its pre-processing step before quantizing. For transformer models it is worth following the linked example. The alternative is to download one of the ready-made quantized graphs that match your CPU family, as described in section 3.
6. Verify quality before you trust any speedup
A faster pipeline that returns slightly wrong vectors is worse than a slow one, because nothing crashes and the damage shows up later as worse search results. Two checks catch most problems.
Parity with the original. Embeddings from your ONNX pipeline should match the original PyTorch model, row by row, to several decimal places. A low cosine here means a bug in tokenization, pooling or truncation.
Neighbor overlap, not just cosine. When you quantize, the average cosine between the old and new vector can look excellent while the nearest neighbors still change, and neighbors are what search returns. Measure both.
import sys
import numpy as np
from embedder import OnnxEmbedder
def topk(m, k=10):
s = m @ m.T
np.fill_diagonal(s, -1)
return np.argsort(-s, axis=1)[:, :k]
def compare(a, b, label):
cos = (a * b).sum(axis=1)
overlap = np.mean([len(set(x) & set(y)) / len(x) for x, y in zip(topk(a), topk(b))])
print(f'{label}: cosine mean {cos.mean():.4f}, min {cos.min():.4f} | top-10 overlap {overlap:.3f}')
# usage: python check_quality.py fp32.onnx int8.onnx tokenizer.json docs.txt
fp32_path, int8_path, tok_path, docs_path = sys.argv[1:5]
texts = [t for t in open(docs_path, encoding='utf-8').read().split('\n') if t.strip()][:2000]
fp32 = OnnxEmbedder(fp32_path, tok_path).embed(texts)
int8 = OnnxEmbedder(int8_path, tok_path).embed(texts)
compare(fp32, int8, 'ONNX fp32 vs ONNX int8')
try:
from sentence_transformers import SentenceTransformer
ref = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2', device='cpu')
ref_vecs = ref.encode(texts, normalize_embeddings=True)
compare(ref_vecs, fp32, 'PyTorch vs ONNX fp32')
except ImportError:
print('sentence-transformers not installed: skipped the PyTorch parity check')Run it on a few thousand of your real documents, not on sample sentences. Read the results this way:
- PyTorch vs ONNX fp32 should be close to identical, with cosine near 1.0000 on every row. Anything visibly lower is a bug to fix before going further.
- fp32 vs int8 will not be identical. Decide in advance how much neighbor overlap you can lose, and then test the thing you actually care about: with your own queries and your own relevance judgments, compare recall at 10 for fp32 and int8. If you have no judgments yet, write fifty by hand. That is a small price for knowing.
[Editor's note: run this script against the real model on your own documents, and state the numbers you get, before publishing any claim about quantization quality.]
7. Measure it on your own hardware
Save this as bench.py next to embedder.py. It reports documents per second for each thread and batch setting, plus the latency of a single batch at the median and the 95th percentile:
import argparse
import statistics
import time
from embedder import OnnxEmbedder
def main():
ap = argparse.ArgumentParser()
ap.add_argument('--model', required=True)
ap.add_argument('--tokenizer', required=True)
ap.add_argument('--texts', required=True, help='one document per line')
ap.add_argument('--threads', type=int, nargs='+', default=[0])
ap.add_argument('--batch-sizes', type=int, nargs='+', default=[1, 8, 32])
ap.add_argument('--max-length', type=int, default=256)
ap.add_argument('--repeats', type=int, default=5)
args = ap.parse_args()
texts = [t for t in open(args.texts, encoding='utf-8').read().split('\n') if t.strip()]
print(f'{len(texts)} documents')
for threads in args.threads:
for bs in args.batch_sizes:
emb = OnnxEmbedder(args.model, args.tokenizer, max_length=args.max_length,
threads=threads, batch_size=bs)
emb.embed(texts[:bs]) # extra warm-up
rates = []
for _ in range(args.repeats):
t0 = time.perf_counter()
emb.embed(texts)
rates.append(len(texts) / (time.perf_counter() - t0))
sample, lat = texts[:bs], []
for _ in range(30):
t0 = time.perf_counter()
emb.embed(sample)
lat.append((time.perf_counter() - t0) * 1000)
lat.sort()
label = threads or 'auto'
print(f'threads={label!s:>4} batch={bs:>3} docs/s median={statistics.median(rates):9.1f} '
f'one-batch latency p50={lat[len(lat) // 2]:6.1f} ms '
f'p95={lat[min(len(lat) - 1, int(len(lat) * 0.95))]:6.1f} ms')
if __name__ == '__main__':
main()Run it with a sample of real documents, one per line, whose length distribution matches production:
python bench.py --model onnx/model.onnx --tokenizer tokenizer.json \
--texts docs.txt --threads 1 2 4 8 --batch-sizes 1 8 32 --max-length 256
python bench.py --model onnx/model_qint8_avx512_vnni.onnx --tokenizer tokenizer.json \
--texts docs.txt --threads 4 --batch-sizes 1 8 32 --max-length 256How to keep the numbers honest:
- Run on an otherwise idle machine, and repeat the whole run at a different time of day.
- Use the same hardware as production, since cloud CPU models vary widely.
- Report latency percentiles as well as throughput. An average hides the slow tail that users feel.
- Record the versions of onnxruntime, tokenizers and Python, and the exact model file.
- Use documents with realistic lengths. Benchmarking on ten-word sentences when your production text is paragraphs measures a different workload.
Record your own results:
- CPU model and number of cores available to the process: ___
- Best throughput for the full-precision model, with its thread and batch settings: ___ docs/s
- Best throughput for the quantized model, with its settings: ___ docs/s
- Median and 95th percentile latency for a batch of one, full precision: ___ ms and ___ ms
- Top-10 neighbor overlap, full precision vs quantized: ___
- PyTorch vs ONNX full-precision cosine, minimum over your documents: ___
[Editor's note: fill in the blanks with measurements from your own server and documents, and state the library versions, before publishing. Do not publish estimates as measurements.]
8. Serving notes
- Load once per process. Creating a session and tokenizer is expensive. Do it at start-up, warm it, then serve.
- Budget threads across workers. Workers multiplied by threads per worker should not exceed the cores you have.
- Micro-batch interactive requests. If queries arrive concurrently, collect them for a few milliseconds and embed them together. You trade a little latency for better throughput. Measure whether that is a win for your traffic.
- Store vectors compactly. Float32 is the safe default. Half precision halves storage and is usually fine for similarity search, but check recall first, as in section 6.
- Pin the model and the tokenizer together. Vectors from different model versions are not comparable. Record the model file name and version next to the stored vectors, and re-embed everything when you change either.
9. Checklist
- Choose a small encoder and confirm its input names and output shape.
- Tokenize in bulk, pad per batch, and pool with the attention mask.
- Sort by token length before batching, and restore the order afterwards.
- Check parity against the original model on real documents.
- If you quantize, compare neighbor overlap and recall, not only cosine.
- Set the thread count to the cores you actually have.
- Benchmark on production hardware with realistic text, and report percentiles.
- Cache embeddings by text hash, and version them with the model.
Summary
A CPU embedding pipeline is mostly bookkeeping: correct padding and pooling, sorted batches, sensible threads, and an honest quality check. Get those right and a small ONNX model can serve many workloads without a GPU. Whether it is fast enough for yours depends on your hardware and your text, so measure it before you decide, and measure quality before you trust a speedup.