When frameworks lose on tiny models

The received wisdom for getting a PyTorch model to run fast is roughly: export it, hand it to TensorRT, and measure the win. That advice is good, and it is derived almost entirely from models that are large, dense, and run at meaningful batch sizes.

My model is a sub-1M-parameter CNN-BiLSTM at batch size one. None of it held.

Here are full-model latencies on a laptop RTX 3050 under WSL2, as ranges across three independent measurement sessions, because one session is not a measurement:

Backend	Mean latency (multi-session range)
ONNX Runtime, CPU	487–699 µs
torch.compile	1,519–1,777 µs
Eager PyTorch	2,050–2,247 µs
TensorRT FP16	2,427–2,966 µs
ONNX Runtime, GPU	3,862–4,652 µs

Read the first and last rows together. The CPU path is the fastest thing on the list, and the GPU path through ONNX Runtime is the slowest — roughly eight times slower than running the same model on the CPU. TensorRT FP16, the thing you are supposed to reach for, is slower than eager PyTorch.

Why this happens

At this size, essentially nothing in the measurement is arithmetic.

A model with a few hundred thousand parameters, run one sample at a time, does not have enough work per kernel launch to hide launch overhead, synchronisation, or the host-side dispatch cost of walking the graph. What the framework is actually spending its time on is getting to the arithmetic. That is a fixed cost per operator, and it is the same fixed cost whether the operator has ten thousand FLOPs or ten billion.

TensorRT’s wins come from operator fusion, precision selection, and picking good tactics for large GEMMs and convolutions. Those are all optimisations on the arithmetic, and the arithmetic was never the problem. Meanwhile FP16 conversion adds its own cost at the boundaries, and a bidirectional LSTM is a shape it has comparatively little to do with.

The CPU wins because there is no launch and no transfer. For a tensor this small, the round trip to the device costs more than the computation saves.

Recurrence is a compiler problem

The other thing that broke: torch.compile with CUDA graph capture crashes on the BiLSTM. Dynamic recurrent control flow is genuinely hostile to graph capture — the whole premise of a captured graph is a fixed sequence of operations with fixed shapes, and a loop whose trip count is a property of the input does not offer that.

This is not a bug so much as a boundary. It is worth knowing where the boundary is before you build a deployment plan on top of a compiler that will hit it.

It does not generalise, and that is the point

On V100S and A100 nodes, the same full model at batch size one gives a different ordering: TensorRT native FP16 becomes the fastest GPU path (528 µs on V100S, 588 µs on A100), and torch.compile beats eager comfortably. The laptop result is a laptop result. The server result is a server result. The one thing I will not do is compute a ratio across the two.

Which is the actual takeaway. “TensorRT is faster” is not a property of TensorRT. It is a property of a model, a batch size, a driver stack, and a specific GPU, and every one of those was doing work in the sentence you thought was about the framework.

The honest caveat about custom kernels

Writing the forward path in CUDA by hand does help on this workload, and per-operator against the matching PyTorch operator the custom kernels win by a wide margin on three of the four blocks.

But that is an operator-against-operator comparison, and it is the only comparison I am entitled to make, because the hand-written path covers four fused blocks and the full model has attention, LayerNorm, residual connections, pooling and a classifier outside of it. Dividing a partial pipeline sum by a full-model framework latency would produce a large, quotable number and it would be meaningless. Different scopes do not go in the same ratio, however much you want them to.