Skip to content

Unsupported operators

Most models with ops that stock TensorRT cannot convert compile with yasp as-is — the frontend rewrites what it can and routes the rest to a plugin it already ships. Compile your first model does exactly that with spconv sparse convolutions.

Some ops have no plugin yet. When prepare meets one, it does not fail: it carves the op out of the graph as a custom-op node, exports the region as a kernel-generation reference, and tells you in the plugin table that the op has no plugin behind it. Generating that kernel is then a prerequisite, not an optimization — until the plugin exists, there is no engine.

Every command below is yasp-toolkit, spelled with its short alias ytk.

graph LR
    A[1. prepare<br/>plugin table] --> B[2. kernelgen compile<br/>generate the kernel]
    B --> C[3. build-plugin<br/>cross-compile .so]
    C --> D[4. compile --custom-plugin<br/>build the engine]
    D --> E[5. verify]

Prerequisites

A Kronos worker on the target (Kronos workers), a Gaia eval worker on a GPU of the same architecture (Gaia workers), and your API key. build-plugin cross-compiles in yasp's cloud, so you need no local toolchain.

Step 1 — Prepare, and read the plugin table

Nothing about the command changes. You do not have to know in advance that the model has an unsupported op, and you do not pass any plugin flags:

ytk embedded prepare model.py \
  --onnx-out      out/model.onnx \
  --npz-out       out/model.npz \
  --reference-out out/model.reference.npz

The tell is the table prepare prints at the end:

  wrote      out/model.onnx
  wrote      out/model.npz
  wrote      out/model.reference.npz
  wrote      out/plugins.json
  wrote      out/fx_nodes.json
  wrote      out/plugin_candidates
PLUGIN                       HAS_PLUGIN  PLACEHOLDER                GAIA_INPUT
------------------------------------------------------------------------------
Col2im                       false       yasp::Col2im               plugin_candidates/Col2im.pt2
╭─ Unsupported operators — this model will not compile yet ─────────────────────────────╮
│ 1 operator in this model has no plugin behind it — the HAS_PLUGIN false row(s) above. │
│ ...                                                                                   │
│   yasp::Col2im    1 node(s)                                                           │
│                                                                                       │
│ For each row above, two commands (spelled out for Col2im):                            │
│   1. ytk kernelgen compile -g <gpu> \                                                 │
│          -r out/plugin_candidates/Col2im.pt2 \                                        │
│   ...                                                                                 │
╰───────────────────────────────────────────────────────────────────────────────────────╯

Any false row raises that warning, and it repeats the steps below with the paths already filled in — the rest of this page is the same walkthrough with the reasoning attached.

Column Meaning
PLUGIN The region's name — the frontend derives it from the op it could not convert.
HAS_PLUGIN true — a plugin already handles this op and there is nothing to do. false — no plugin exists.
PLACEHOLDER The custom-op node now standing in for the region in the ONNX, as <namespace>::<Name>.
GAIA_INPUT The .pt2 reference to hand to kernel generation.

The table lists every plugin the model touches, shipped ones included, so a mixed model gives you true rows to ignore and false rows to act on.

false rows mean the engine will not compile yet

prepare exits 0 and writes a complete ONNX, so nothing looks wrong. But that ONNX now contains a yasp::Col2im node, and TensorRT resolves plugins by (namespace, name, version) — with no .so registering that name, the builder has nothing to build the layer from and the compile fails on the worker. Every HAS_PLUGIN false row is a blocker. Read the table before you spend a compile.

Getting candidates you did not ask for is itself the signal: with no --plugin flags on the command line, a non-empty plugin table means the frontend had to invent a placeholder to finish the export.

What prepare left you

Beside the ONNX, in out/:

Artifact What it is
plugins.json The same table, machine-readable — name, has_plugin, placeholder, gaia_input per row.
plugin_candidates/<Name>.pt2 The kernel-generation reference: the carved region as a torch.export program with weights bundled.
plugin_candidates/candidates.json The diagnosis — the failing aten op, the exporter's own reason, the owning module and its parameters, and the per-site input/output shapes and dtypes.
plugin_candidates/<Name>.io.npz, <Name>.inputs.pt The region's captured inputs and outputs, so its numerics can be checked independently.
plugin_candidates/<Name>.reference.py The torch source of the region, when it came from a module.

Find out why the op was unsupported

candidates.json records the exporter's own explanation under unsupported_reason, with reason set to onnx_export_unsupported_op:

python -c "
import json; c = json.load(open('out/plugin_candidates/candidates.json'))
for e in c: print(e['proposed_op'], '|', e.get('aten_op'), '|', e.get('unsupported_reason'))"

Two real examples, both of which reach the same flow from different causes:

An op the opset does not cover yet — a model ending in nn.Fold:

yasp::Col2im | aten::col2im | UnsupportedOperatorError: Exporting the operator
'aten::col2im' to ONNX opset version 17 is not supported. Support for this
operator was added in version 18, try exporting with this version

An op the exporter covers only in lower rank — a volume lifter whose F.grid_sample is 5-D:

yasp::GridSampler | aten::grid_sampler | OnnxExporterError: Unsupported: ONNX
export of operator GridSample with 5D volumetric input.

The second is the more common shape of this problem in practice: the op looks supported, and only the rank, dtype, or a particular attribute combination is not.

Step 2 — Generate the kernel

One kernelgen compile per false row, pointed at that row's .pt2:

export GPU=Orin        # your eval worker's GPU — see `ytk eval workers`

ytk kernelgen compile \
  -r out/plugin_candidates/Col2im.pt2 \
  -g "$GPU" \
  --config deployment_target=tensorrt \
  -o out/sources/Col2im.module_source.py \
  --report out/sources/Col2im.report.json

Pick an eval worker whose GPU matches the target's architecture — kernels are scored on real hardware, so that measurement is only meaningful on the architecture you will deploy to.

--config deployment_target=tensorrt is required: it makes generation emit a bundle build-plugin can package, rather than a standalone torch extension.

Two properties of the result are hard requirements, because getting either wrong produces a kernel TensorRT cannot insert at all:

  • Input arity — exactly the placeholder node's inputs, in order. Col2im takes one input; the 5-D GridSampler takes two, the volume and the sampling grid. A mismatch surfaces as TensorRT Error 9, "could not find any supported formats consistent with input/output data types" — a format error for what is really an arity problem.
  • Precision — match the region's dtypes as captured. A kernel advertising only fp32 cannot be inserted into an fp16 engine.

Both are recorded per candidate in candidates.json (node_inputs, call_site_shapes), so you can check the generated bundle against them before paying for a build.

Step 3 — Build the plugin

export ARCH=<advertised arch_spec from `ytk embedded workers`>

ytk embedded build-plugin -a "$ARCH" \
  --plugin-name Col2im \
  -s out/sources/Col2im.module_source.py \
  -o out/plugins/Col2im.so

--plugin-name must be the placeholder's name

Here the name is not a choice: prepare already put yasp::Col2im in the ONNX, so the .so has to report exactly Col2im. Pass it on every build-plugin call rather than relying on the name the source arrives with — a plugin whose name does not match its node is one TensorRT cannot resolve, and you find out at step 4 after paying for a compile.

--plugin-name rewrites plugin_name: in a copy of the source, so the generated file is never modified in place.

Step 4 — Compile

Same compile as any other, with one --custom-plugin per plugin you built:

ytk embedded compile -a "$ARCH" -p fp16 \
  --onnx out/model.onnx --npz out/model.npz -o out/model.tgz \
  --custom-plugin out/plugins/Col2im.so

The plugins are registered before the build starts, so the builder resolves the placeholder nodes. They travel with the compile request — the target machine needs no plugin installation and no toolchain.

Step 5 — Verify

ytk embedded verify out/model.reference.npz out/model.tgz

This is the step that closes the loop. The reference came from the original torch model, unsupported op included, so the accuracy rows tell you whether the generated kernel actually computes what nn.Fold computed. A plugin that builds and runs can still be wrong; verify is where that shows up. See embedded verify.

Not the same flow as optimizing

The commands overlap almost completely with Optimize your model, but the two are answering different questions:

Unsupported operators Optimize your model
Why The engine will not build at all The engine builds, but a region is slow
What picks the region The frontend, when the export fails You, from measured profile data via propose
Starting point The first prepare A compiled, profiled baseline engine
Plugin name Fixed — the placeholder prepare emitted Yours, chosen in the --plugin selector
If you skip it No engine A working, slower engine

Once the model compiles, the optimization flow applies to it like any other — and the unsupported region shows up there as an already-plugged row with a per-site time to beat.

Where to next?