Optimize your model
TensorRT builds a fast engine out of the kernels it already has. When a region of your model has no good native kernel — a fused attention block, a depthwise convolution chain, an op TensorRT cannot convert at all — that region sets the floor on your latency, and no amount of recompiling moves it.
This flow removes that floor by joining the two products end to end: Kronos measures the engine on the target and carves the slow regions you choose out of the ONNX as custom-op nodes, Gaia generates a CUDA kernel for each one, and Kronos cross-compiles those kernels into TensorRT plugins and links them into the next build.
Nothing is installed on the target machine. The plugins travel with the compile as .so files.
graph LR
A[1. Baseline<br/>prepare + compile + verify] --> B[2. Choose regions<br/>profile + fx_nodes.json]
B --> C[3. prepare --plugin<br/>carve the regions]
C --> D[4. kernelgen compile<br/>generate kernels]
D --> E[5. build-plugin<br/>cross-compile .so]
E --> F[6. compile --custom-plugin<br/>rebuild the engine]
F --> G[7. verify<br/>measure the win]
Prerequisites
- A Kronos worker on the target hardware — see Kronos workers. It builds the engines and profiles them, which is where the measurements come from.
- A Gaia eval worker on a GPU of the same architecture as the target — see Gaia workers. Generated kernels are scored on real hardware, so this GPU decides whether a kernel is good.
- Your API key (get one).
Plugin building needs nothing from you: build-plugin cross-compiles in yasp's cloud for the arch spec you name.
The worked example
The walkthrough uses EfficientViT-B1, a computer-vision backbone, at batch 4 and 1088×1920, compiled fp16 for a DRIVE AGX Orin:
export ARCH=arm64-sm87-dos6010-cu114-trt86
It is an ordinary model.py — a Model class and a get_fwd_args(), exactly as in Compile your first model. Plugin selection never touches the model file; it all happens on the command line.
Step 1 — Baseline the engine
Prepare and compile as usual. The baseline is not optional: every later decision is made from measured time on the target, not from reading the graph.
ytk embedded prepare model.py \
--onnx-out base/model.onnx \
--npz-out base/model.npz \
--reference-out base/model.reference.npz
ytk embedded compile -a "$ARCH" -p fp16 \
--onnx base/model.onnx --npz base/model.npz -o base/base.tgz
Alongside the usual outputs, prepare writes base/fx_nodes.json — the traced FX graph's nodes, each carrying the torch module that owns it. The frontend exports a flat ONNX graph with no module scopes in the layer names, so this file is what later lets a TensorRT layer name be mapped back to the nn.Module it came from. Keep it next to the ONNX it was produced with.
The compile bundle already contains a per-layer profile captured on the target (Profile the model), so verify gives you the baseline immediately:
ytk embedded verify base/model.reference.npz base/base.tgz
│ base │ 10 │ 70.283 │ 70.3… │ 70.343 │ 70.4… │ 70.466 │ 70.4… │
70.3 ms. The per-layer table is in the same output, but TensorRT layer names (/conv_7/Conv + PWN(...)) are rarely enough on their own to tell you which part of your model to attack. That is the next step.
Step 2 — Choose the regions
You pick the regions; no command does it for you. Two files from step 1 are the inputs:
| File | What it gives you |
|---|---|
The per-layer profile in the compile bundle (profile_layers.csv) |
GPU time per layer on the target, with the median as the steady-state cost; ytk embedded verify <reference> <bundle> --verbose prints it. See Profile the model. |
base/fx_nodes.json |
The traced FX graph's nodes, each carrying the torch module that owns it, so a TensorRT layer name maps back to an nn.Module. |
- Read the profile top-down by median time and take the heavy layers.
- Map each layer to its owning module through
fx_nodes.json, and group the layers that belong to the same block. - Write one selector per region (grammar below); a plugin is worth a generation job only if the region's time is large enough to repay it.
Two kinds of region are not worth a kernel on EfficientViT-B1:
- Dense 1×1 convolutions and GEMMs: TensorRT already runs them at the memory roofline, and generated kernels lost every attempt here.
- Strided depthwise convolutions: the bundle grammar cannot express strided output extents.
The regions chosen for this walkthrough are the two LiteMLA attention families and the depthwise convolution families, six plugins in all, which together held about 27 ms of the 70 ms.
The name on the left of = in each selector is the ONNX node prepare inserts and the name the built plugin reports: TensorRT resolves a plugin by (namespace, name, version).
Step 3 — Carve the regions out
Run prepare again with one --plugin per region, and the TensorRT version of the target (…-trt86 is 8.6; prepare gates its ONNX passes on it, so a graph exported without it differs from the one you measured):
ytk embedded prepare model.py --trt-version 8.6 --bench-iters 0 \
--onnx-out gaia/model.onnx \
--npz-out gaia/model.npz \
--reference-out gaia/model.reference.npz \
--plugin 'yasp::LiteMLAC384=LiteMLA:around@where=*m.stages.2.blocks.*.context_module.main' \
--plugin 'yasp::LiteMLAC768=LiteMLA:around@where=*m.stages.3.blocks.*.context_module.main' \
--plugin 'yasp::DwConv3x3s1Dyn=ConvNormAct@where=*.0.blocks.1.main.depth_conv,*.local_module.main.depth_conv,*.res0.main.depth_conv' \
--plugin 'yasp::DwConv5x5s1C384=Conv2d@where=*m.stages.2.blocks.*.context_module.main.aggreg.0.0' \
--plugin 'yasp::DwConv5x5s1C768=Conv2d@where=*m.stages.3.blocks.*.context_module.main.aggreg.0.0' \
--plugin 'yasp::DwConv3x3s1C256=ConvNormAct@where=*.1.blocks.1.main.depth_conv,*.2.main.depth_conv'
Each matched region is replaced in the ONNX by a single custom-op node, and the export gains two things:
| Artifact | What it is |
|---|---|
plugins.json |
The machine-readable carve — one entry per plugin with its placeholder op, site count, selector, and the path to its Gaia input. |
plugin_candidates/<Name>.pt2 |
The kernel-generation reference: the carved region exported as a torch.export program with weights bundled. This is what Gaia optimizes against. |
Also written per candidate: <Name>.io.npz and <Name>.inputs.pt (the captured I/O, so the region's numerics can be checked), <Name>.reference.py (the torch source of the region), and candidates.json.
Re-running prepare refreshes only its own artifacts in that directory — generated kernels, reports and built .so files are left alone.
prepare also prints the carve as a table when it finishes — which regions matched, how many sites each covers, and which ones carry a kernel-generation input — and candidates.json records the same thing per candidate: the per-site input/output shapes, the region's op inventory, and which axes are constant across sites. ytk embedded inspect gaia summarizes them and checks that each .pt2 reproduces its captured output. A selector that matches nothing is silent: prepare exits 0 with a complete ONNX and simply no .pt2 for it,. Check that plugins.json exists after the run.
See embedded prepare for every flag.
Selector grammar
A selector reads left to right as "insert a plugin named PluginName in namespace ns, at the region called OpsName":
<namespace>::<PluginName>=<OpsName>[:around][@modifier=...]
| Part | Meaning |
|---|---|
<namespace> |
The TensorRT plugin namespace the .so's creator registers under. yasp is the builder default; abc::X targets a creator under abc; ::X targets the empty namespace (a hand-built .so that never called setPluginNamespace). |
<PluginName> |
The op emitted into the ONNX, and the name the plugin must report. Decoupled from OpsName, so you can name it for the target. |
<OpsName> |
The region — a torch class name. Bare, it replaces every instance of that block wholesale. |
:around |
Target the heavy op(s) inside the block and grow the region to absorb the plumbing around them (transposes, slices, casts, pads, pointwise, and the joining concat). The emitted .pt2 then carries those reformats, so the generated kernel can replace them too. |
Modifiers, appended in any order:
| Modifier | Effect |
|---|---|
@where=<globs> |
Scope a class match to module-path globs, comma-separated (OR); a !-prefixed glob excludes. Essential when one class is the right operator but the wrong scope — every depthwise conv here is the same ConvNormAct, but the stride-2 one needs a different kernel. |
@lift=<how> |
Pass the swapped module's weights to the plugin as real inputs. Automatic for a swapped module containing a conv (fold_conv_bn); @lift=none opts out. A region with more than one conv needs @lift=fold_conv_bn_seq, which emits one folded (weight, bias) pair per conv in call order — fold_conv_bn refuses such a region, since one folded pair cannot describe two convolutions. |
@nojoin |
Stop an :around region from swallowing the concat that joins its inputs. Absorbing it is the default and is worth ~0.6 ms/site here. |
Escape hatches for regions no class name describes: call:<dotted.path> (intercept an op during export), A...B (a dataflow span between two fx_nodes.json nodes), a comma-separated node list, or mod:<suffix> (a module qualified-name suffix).
Step 4 — Generate the kernels (Gaia)
One kernelgen compile per candidate:
ytk kernelgen compile \
-r gaia/plugin_candidates/DwConv3x3s1C256.pt2 \
-g "NVIDIA Jetson AGX Orin" \
--config deployment_target=tensorrt \
-o gaia/sources/DwConv3x3s1C256.module_source.py \
--report gaia/sources/DwConv3x3s1C256.report.json
deployment_target=tensorrt is required
Every kernelgen compile in this flow must pass --config deployment_target=tensorrt. It
tells the generator the kernel is destined for a TensorRT plugin rather than a standalone
torch extension, which is what makes the emitted bundle something
build-plugin can turn into a .so. Omit it and the run
produces a kernel you cannot plug in — a wasted job, since generation is not cheap.
-g names the eval worker's GPU, exactly as ytk eval workers lists it, and it should match the target's architecture — the DRIVE AGX Orin here is sm87, so kernels are scored on an Orin. Gaia optimizes against the torch reference in the .pt2 and keeps the candidate that measured fastest on that hardware.
Torch speedup is not TensorRT speedup
Gaia scores candidates against the torch reference, and torch is typically several times slower than TensorRT for the same region. A large speedup over torch does not by itself mean the kernel beats the TensorRT layer it replaces. The real bar is the measured TensorRT time of the region in the baseline profile, so read report.json against that number, not against the torch speedup.
Two properties of the generated kernel are hard requirements rather than preferences, because getting either wrong produces a kernel TensorRT cannot insert at all — check them on the result before spending a build-plugin on it:
- Input arity — the node's inputs, weights included. A region whose weights were lifted carries them as real inputs, not baked constants. A mismatch surfaces as TensorRT Error 9, "could not find any supported formats consistent with input/output data types" — a format error for what is really an arity problem.
- Layout and precision — e.g.
kHWC8(NHWC, channels padded to a multiple of 8) and fp16 in/out with fp32 accumulation. An NCHW kernel needs a reformat on every edge that costs more than the kernel saves, and an fp32-only kernel cannot go into an fp16 engine.
The output is a single Python file bundling everything a plugin needs: the CUDA header, the CUDA source with the generated kernel, and a plugin_meta.yaml block declaring the TensorRT contract (input/output dtypes and formats, output shape expressions, extents, launcher symbol, workspace function, and constraints).
See kernelgen compile for routing and --config options.
Step 5 — Build the plugins
build-plugin cross-compiles the generated source for your target architecture in yasp's cloud — no CUDA toolchain on your machine, and nothing installed on the target:
ytk embedded build-plugin -a "$ARCH" \
--plugin-name DwConv3x3s1C256 \
-s gaia/sources/DwConv3x3s1C256.module_source.py \
-o gaia/plugins/DwConv3x3s1C256.so
It prints the --custom-plugin line to paste into the next step when it finishes. A build takes a minute or so for a small kernel and a few for a fused block.
Always pass --plugin-name
TensorRT resolves a plugin by (namespace, name, version), so the .so has to report exactly
the <PluginName> your prepare --plugin selector put into the ONNX — DwConv3x3s1C256 here.
Set it yourself on every build-plugin call rather than relying on the name the source arrives
with: a plugin whose name does not match its node is one TensorRT cannot resolve, and you find
out at step 6 after paying for a compile.
--plugin-name rewrites plugin_name: in a copy of the source before uploading, so the
generated file — which cost a job to produce — is never modified in place. Use it instead of
hand-editing generated source.
Step 6 — Compile with the plugins
Pass every .so to the compile. They are registered with TensorRT before the build starts, so the builder resolves the custom-op nodes prepare inserted:
ytk embedded compile -a "$ARCH" -p fp16 \
--onnx gaia/model.onnx --npz base/model.npz -o gaia/gaia.tgz \
--custom-plugin gaia/plugins/DwConv3x3s1C256.so \
--custom-plugin gaia/plugins/DwConv3x3s1Dyn.so \
--custom-plugin gaia/plugins/DwConv5x5s1C384.so \
--custom-plugin gaia/plugins/DwConv5x5s1C768.so \
--custom-plugin gaia/plugins/LiteMLAC384.so \
--custom-plugin gaia/plugins/LiteMLAC768.so
Note --npz base/model.npz: the carved ONNX is compiled on the baseline's sample inputs, not on the ones the second prepare drew. verify only compares bundles that saw the same bytes, and the carved graph is the same model, so the baseline's inputs and reference apply to it unchanged. The plugins ship with the compile request. The target machine needs no plugin installation, no toolchain, and no change to how it runs the engine. A compile missing even one plugin fails, and names only the first unresolved op.
See embedded compile.
Step 7 — Measure
verify takes several bundles and lines their rows up, so the before/after is one command — both engines were compiled on the baseline's inputs, against the baseline's reference:
ytk embedded verify base/model.reference.npz base/base.tgz gaia/gaia.tgz
┃ Implementation ┃ Iter… ┃ Min ┃ Mean ┃ p50 ┃ p95 ┃ p99 ┃ Max ┃
│ torch host (cuda) │ 100 │ 93.618 │ 109.… │ 107.8… │ 136.… │ 147.9… │ 150.… │
│ base │ 10 │ 70.283 │ 70.3… │ 70.343 │ 70.4… │ 70.466 │ 70.4… │
│ gaia │ 10 │ 42.931 │ 42.9… │ 42.951 │ 42.9… │ 42.976 │ 42.9… │
70.3 ms → 42.9 ms, a 1.64× end-to-end speedup from six generated kernels. The torch host row is your reference machine's GPU, not the target, so ignore it when the two are different hardware.
verify also scores numerics against the torch reference for every output; a plugin changes how a region computes, so read that table too — see embedded verify.
Where to next?
- Profile the model — the per-layer profile that tells you which regions to carve, and how to collect one yourself on the device.
- Generate your first kernel — the Gaia flow on its own, for kernels that are not part of an engine.
- Debug taps — when a plugged engine drifts from torch, expose every yasp plugin boundary as a comparison point and let
verifypinpoint the region. embeddedreference — every flag forprepare,inspect,compile,build-pluginandverify.kernelgenreference —export,compile, andget.