Skip to content

yasp-toolkit embedded

Build a TensorRT engine for your target GPU from a PyTorch model: turn the model into compile inputs, run the compile, verify the engine's numerics against torch, and check on tasks and the workers that ran them.

Provided by yasp-kronos-edge-toolkit, which the kronos-edge extra installs: yasp-toolkit install kronos-edge.

The router host defaults to https://compile.yasp.ai/embedded and can be overridden with yasp-toolkit --embedded-api-base; build-plugin uses the plugin builder, --plugin-builder-api-base or $YASP_PLUGIN_BUILDER_HOST.

wrap

Inspect a HuggingFace repo and write the model.py that embedded prepare takes.

Usage:

$ yasp-toolkit embedded wrap [OPTIONS]
Option Default Description
--hf <str> required HuggingFace repo id to wrap (a local directory of a saved model also works).
--output <file> model.py Where to write the generated model.py.
--seq-len <int> 128 Text sequence length to inspect the model with.
--image-size <int> — Square image edge; default is the processor's own.
--audio-seconds <float> 1.0 Seconds of audio to inspect with.
--trust-remote-code — Run modelling code the repo ships. Off by default.
--task-heads / --no-task-heads task-heads Prefer a task head (logits out) on a bare checkpoint. --no-task-heads exports the pretrained model itself, with no randomly initialised head.
--force — Overwrite an existing file.
yasp-toolkit embedded wrap --hf <HF_REPO_ID>

wrap loads the repo, works out its inputs, and writes the two-symbol model.py (Model plus get_fwd_args) that prepare takes, so a HuggingFace model needs no hand-written wrapper. Pass --trust-remote-code only for a repo whose modelling code you trust.

prepare

Generate the ONNX + NPZ inputs that embedded compile expects from a torch model.

Also captures the torch-eager outputs into a reference npz so embedded verify can compare compiled artifacts against ground truth without re-loading the torch model. Unless --bench-iters 0, the same npz additionally records per-iteration host-torch timings (on CUDA when available) that verify renders as the reference row of its benchmark table.

The model file must define a Model class (nn.Module subclass) and a get_fwd_args() -> tuple returning example positional inputs.

Ends with the plugin table. A HAS_PLUGIN false row there is a blocker, not a note: the export left a placeholder node no .so implements, so embedded compile cannot build an engine until that kernel is generated with Gaia. prepare prints the commands for it.

Usage:

$ yasp-toolkit embedded prepare [OPTIONS] {model_file}
Argument Default Description
model_file required Torch model .py defining Model and get_fwd_args.
Option Default Description
--onnx-out <path> required Destination path for the ONNX model.
--npz-out <path> required Destination path for the .npz sample inputs.
--reference-out <path> — Destination path for the torch-eager reference outputs npz. Default: .reference.npz alongside the ONNX file.
--bench-iters <int> 100 Timed host-torch iterations stored as benchmark_timings_ms in the reference npz, so embedded verify can show host latency next to each worker's. 0 disables.
--bench-warmup <int> 10 Warmup iterations before the timed host-torch benchmark.
--trt-version <str> — TensorRT version of the compile target (e.g. '8.5'). Enables the target-specific ONNX passes; without it TRT-8 targets silently miss rewrites worth ~10%+ latency.
--capture-safe / --no-capture-safe — Export a CUDA-graph-capturable engine graph: swaps capture-unsafe patchers and externalizes the sparse spatial-shape derivation as a coors_max engine input (I/O contract gains one INT32[4] input; the sample npz includes it automatically).
--plugin <str> — Bring-your-own plugin selection (repeatable): '::=[:around]'. Inserts a plugin named (ONNX domain ) at every occurrence of the region named (its BLOCK / torch-class name), and emits its kernel-gen reference (.pt2). '' replaces the whole block (e.g. 'yasp::MBConvPlugin=MBConv'); ':around' targets the heavy op(s) inside the block and absorbs the plumbing around them so the .pt2 carries the reformats (e.g. 'yasp::LiteMLA_DRIVE_AGX_Plugin=LiteMLA:around'). is the emitted op / the .so creator name, decoupled from ; the you write is the TRT plugin namespace the .so must register ('yasp' = builder default; 'abc::X' targets a creator under 'abc'; '::X' one under the EMPTY namespace — a hand-built .so that never called setPluginNamespace). Escape hatches: 'call:', an 'A...B' dataflow span, a comma node list, or 'mod:'. Append '@lift=fold_conv_bn' to pass the swapped module's weights to the plugin, '@where=' to scope a class match to module-path globs (comma-OR; a '!'-prefixed glob excludes).
--debug-taps <str> — Expose tap() markers in the model as extra reference outputs so verify compares intermediate tensors too. Repeat once per tap name (e.g. --debug-taps alpha --debug-taps beta), or --debug-taps all for every tap. Omit to expose none.
--debug-plugin-taps — Enables debug plugin taps.
--stage <str> — Staged model files only: export just these stages (repeatable; default: every stage in chain order). A stage whose export is known to fail can then be left out.
--chain-min-cosine <float> 0.999 Staged model files only: the cosine every result of the chain's own torch run must reach against the model file's entry() for the prepare to pass (integer results must match exactly).
--dynamic-dim <str> — Let one input axis vary at runtime, so a single engine accepts a range of sizes instead of only the example's (repeatable): ':=[..]..[@]', e.g. '0:0=1..8' or '0:0=1000..4000..20000@N'. Indices are positions in the tuple get_fwd_args() returns; unlisted inputs keep the fixed size of their example. min/max bound what the engine will accept. The middle value, when you give one, is the size it is tuned to run fastest at; omit it and that comes from your example input instead (with a list of calibration frames, the FIRST frame). Either way the example's size must fall inside min..max, and verify both scores accuracy and measures latency at THAT size -- not at the tuned one, so a pinned opt makes the reported milliseconds describe a size the engine was not optimized for. Repeat the flag for further axes or inputs. names the axis: give two inputs the same symbol to declare their axes always equal, which you must then honour at runtime; otherwise each axis varies independently. On a sparse (spconv) model, layer capacities are sized from this range rather than measured from the example frames.

See Debug taps for more information on taps.

--plugin selects the regions to carve out as custom-op nodes; the export then also writes plugins.json and one kernel-generation candidate per plugin under plugin_candidates/. See Optimize your model for the selector grammar and the artifacts.

inspect

Summarize the plugin candidates emitted by embedded prepare.

Usage:

$ yasp-toolkit embedded inspect [OPTIONS] {prepare_dir}
Argument Default Description
prepare_dir required A prepare --onnx-out dir, its plugin_candidates dir, or candidates.json.
Option Default Description
--run / --no-run run Load each .pt2 and check it reproduces the captured output (needs torch).
--guidance — Write a .guidance.draft.txt beside each .pt2, prefilled with the parts that can be derived (region ops, per-site shapes, which axes are constant) and TODO markers for the rest. Edit it, then pass with kernelgen compile --config guidance=@.
--emit-kernelgen <file> — Write a runnable batch-submit script at this path (one ytk kernelgen compile per candidate, run in turn). No flag, no script.
yasp-toolkit embedded inspect <PREPARE_OUT_DIR>

inspect summarizes the kernel-generation candidates a prepare --plugin run wrote under plugin_candidates/, and by default loads each .pt2 to check it reproduces the captured output. --guidance drafts a guidance file per candidate for kernelgen compile --config guidance=@<file>, and --emit-kernelgen <path> writes a script that submits one kernelgen compile per candidate.

compile

Build a TensorRT engine from a prepared model on your target hardware.

Phases: create the task, upload model.onnx + sample.npz, start, poll until the task reaches a terminal state, then optionally download the result.

The task id is printed as soon as it's obtained, so a failure mid-flow (upload, wait, download) is recoverable via the granular commands (embedded get <task-id>, download …).

Usage:

$ yasp-toolkit embedded compile [OPTIONS]
Option Default Description
-a, --arch-spec <str> required Worker arch_spec tag the task should route to (e.g. amd64-sm120-u2404-cu128-trt00).
--routing <str> — Pin the task to workers registered with this routing name (e.g. a dev/test pool registered with --tag routing=<name> / worker.tags.routing). Omit to use the shared stable pool.
-p, --precision <str> native Precision tokens joined with _, in any order (e.g. native, fp16, fp16_int8, fp32_tf32). native builds strongly typed: the ONNX decides every layer's precision. Each worker advertises the tokens its target can build as its precision_tokens tag (see embedded workers). Ignored with --model-dir: each stage declares its own.
--preflight / --no-preflight preflight Before creating the task, check that a worker for --arch-spec (and --routing) advertises every token of --precision (with --model-dir, every token any stage declares). Skip it for a worker registered since, or a router that cannot list workers.
--onnx, --onnx-model <file> — Local ONNX model to upload as model.onnx.
--npz, --samples-npz <file> — Local NPZ sample input to upload as sample.npz.
--model-dir <directory> — A staged model as embedded prepare left it (the --onnx-out directory: chain.json, one export per stage, the chain's gate). ONE task builds every stage with the precision and plugins its declaration names and packages ONE artifact whose runner executes the chain. Exclusive with --onnx/--npz/--custom-plugin/--calibration-cache/--dataset; --precision is ignored (each stage declares its own).
-o, --output <file> — Where to write the resulting output.tgz (the demo log lives inside it as share/.../generated/demo_run.log — extract with transfer extract). Omit to skip the download step.
--dataset <str> — Name of a registered dataset (see dataset add/list) to attach to the task.
--refresh-manifest — Force a fresh dataset manifest even if a still-valid one is cached.
--calibration-cache <file> — Path to a pre-built INT8 calibration cache (TRT-XYZ-EntropyCalibration2 format, ~1 kB), produced by embedded calibrate with the same --precision. Skips live calibration on the worker. When omitted and the precision permits int8, the worker calibrates against the dataset attached via --dataset (regular manifest path).
--custom-plugin <file> — Path to an extra TensorRT plugin shared library to register for this compile (repeatable).
--tag KEY=VALUE — Attach a user-facing key=value tag to the task (repeatable, e.g. --tag env=prod --tag region=ca). Tags are for identification only; they do not affect routing.

The demo log lives inside output.tgz as share/.../generated/demo_run.log — pull it out with transfer extract.

extract-engine

Write the engine a compiled bundle embeds as a standalone package for your own TensorRT runtime.

The package holds model.engine, the plugin libraries it must load (plugins/), and a manifest.json listing them in load order with how each registers: dlopen for a library that registers its creators statically, IPluginRegistry::loadLibrary for one exporting the dynamic plugin interface (trtexec --plugins= vs --dynamicPlugins=). Load TensorRT's own plugin library too, as trtexec does. The engine runs only on the TensorRT version and GPU it was built for. Nothing is recompiled, and the bundle is left to verify unless --delete-binary.

Usage:

$ yasp-toolkit embedded extract-engine [OPTIONS] {binary}
Argument Default Description
binary required A compiled bundle (compile --output).
Option Default Description
-o, --output <file> — Where to write the engine package. Defaults to <binary>.engine.tgz beside the input.
--delete-binary — Remove the input bundle once the engine package is written.

For a host that runs its own TensorRT instead of libyasp_binary. The package is cut out of the compiled bundle, which already embeds everything a standalone load needs, so nothing is recompiled:

  • model.engine -- the serialized engine. It runs only on the TensorRT version and GPU it was built for.
  • plugins/custom_plugin_<i>.so -- the libraries the binary embeds: everything the engine must load before it deserializes. Libraries the engine serialized (TensorRT >= 8.6) are inside the engine itself and need nothing.
  • manifest.json -- plugins in load order, each with its loader: dlopen for a library that registers its creators statically, loadLibrary (IPluginRegistry::loadLibrary) for one that exports the dynamic plugin interface; with trtexec that is --plugins=<lib> or --dynamicPlugins=<lib>. Also the build's target, precision and io table.
  • build.json, sample_inputs.npz and sample_outputs.npz from the bundle.

Load TensorRT's own plugin library as well (initLibNvInferPlugins, which trtexec always does): the binary links it, and the manifest cannot tell whether the engine uses a built-in plugin.

The engine package is a delivery artifact, not something verify scores: verify the bundle first, then extract. The bundle is kept unless --delete-binary. A chained (multi-stage) binary has no single engine and is refused.

calibrate

Produce an INT8 calibration cache from a dataset on your target hardware.

Phases: create the task, upload model.onnx + the dataset manifest, start, poll until the task reaches a terminal state, then optionally download the resulting calibration.cache.

Feed that cache to compile --calibration-cache, with the same --precision, to skip live calibration on subsequent builds.

The task id is printed as soon as it's obtained, so a failure mid-flow (upload, wait, download) is recoverable via the granular commands.

Usage:

$ yasp-toolkit embedded calibrate [OPTIONS]
Option Default Description
-a, --arch-spec <str> required Worker arch_spec tag the task should route to (e.g. amd64-sm120-u2404-cu128-trt00).
--routing <str> — Pin the task to workers registered with this routing name (e.g. a dev/test pool registered with --tag routing=<name> / worker.tags.routing). Omit to use the shared stable pool.
-p, --precision <str> required The precision the cache is for, which must permit int8 (e.g. fp32_int8, fp16_int8). Required, because a cache only fits the precision it was calibrated for: build the engine with compile --precision set to the same value.
--preflight / --no-preflight preflight Before creating the task, check that a worker for --arch-spec (and --routing) advertises every token of --precision. Skip it for a worker registered since, or a router that cannot list workers.
--onnx, --onnx-model <file> required Local ONNX model to upload as model.onnx.
--dataset <str> required Name of a registered dataset (see dataset add/list) the worker calibrates against.
--input-name <str> — Name of the model input the dataset feeds; disambiguates models with multiple inputs.
-o, --output <file> — Where to write the resulting binary calibration.cache. Omit to skip the download step.
--refresh-manifest — Force a fresh dataset manifest even if a still-valid one is cached.
--custom-plugin <file> — Path to an extra TensorRT plugin shared library to register for this calibration (repeatable).

build-plugin

Cross-compile a TensorRT plugin from source for your target hardware.

Phases: create the build, upload the source file, start, poll until the build reaches a terminal state, then download the resulting shared object.

The build id is printed as soon as it's obtained, so a failure mid-flow (upload, wait, download) is recoverable via download.

Usage:

$ yasp-toolkit embedded build-plugin [OPTIONS]
Option Default Description
-a, --arch-spec <str> required Build spec to cross-compile for; routes to plugin builders advertising this tag (e.g. jp61-cu126-trt103-py311).
-s, --source <file> required Plugin source file to build.
--plugin-name <str> — Override plugin_name in the bundle's plugin_meta.yaml before building. Kernel generation names a plugin after the operation it implements, not after your selector, and TensorRT resolves plugins by (namespace, name, version) — so a generated bundle usually needs renaming to match the node prepare emitted. Use this instead of hand-editing generated source.
--input-format <str> cuda Source language of --source. The build backend validates the values it supports.
-o, --output <file> required Where to write the compiled shared object (plugin.so).

The source is the module_source.py a Gaia kernel generation produced — CUDA header, CUDA source, and the plugin_meta.yaml declaring the TensorRT contract. Build one .so per plugin, then pass them all to compile with --custom-plugin.

Pass --plugin-name on every call, set to the <PluginName> the matching prepare --plugin selector inserted into the ONNX: TensorRT resolves a plugin by (namespace, name, version), so a .so whose name does not match its node cannot be resolved at build time. It rewrites plugin_name: in a copy of the source, never in the generated file itself.

cancel

Usage:

$ yasp-toolkit embedded cancel [OPTIONS] COMMAND [ARGS]...

Options: * --help: Show this message and exit.

Commands:

  • compilation: Cancel a running compilation task.
  • calibration: Cancel a running calibration task.

yasp-toolkit embedded cancel compilation

Cancel a running compilation task.

Usage:

$ yasp-toolkit embedded cancel compilation [OPTIONS] {task_id}
Argument Default Description
task_id required Task ID.

Options: * --help: Show this message and exit.

yasp-toolkit embedded cancel calibration

Cancel a running calibration task.

Usage:

$ yasp-toolkit embedded cancel calibration [OPTIONS] {task_id}
Argument Default Description
task_id required Task ID.

Options: * --help: Show this message and exit. Cancellation only applies while the task is still pending or started; the router answers with a 409 once it reached a terminal state (completed / failed / cancelled).

get

Fetch a task's (compile, calibration or plugin build) current state.

With --wait, poll until terminal.

Usage:

$ yasp-toolkit embedded get [OPTIONS] {task_id}
Argument Default Description
task_id required Task ID (a compile, calibration or plugin-build id).
Option Default Description
--wait — Poll every 15s until the task is completed or failed.

key

Print a scoped key a Kronos worker can register with.

Use the JWT wherever a Kronos worker asks for YASP_API_KEY:

export WORKER_KEY=$(yasp-toolkit embedded key)
kubectl create secret generic yasp-api-token --from-literal=token="$WORKER_KEY"

Usage:

$ yasp-toolkit embedded key [OPTIONS]
Option Default Description
--audience <str> embedded-api Token audience claim.
--scope <str> — Token scope (repeatable). Defaults to embedded-register only.

workers

List embedded workers (optionally filtered by tag).

Usage:

$ yasp-toolkit embedded workers [OPTIONS]
Option Default Description
-t, --tag <str> — KEY=VALUE filter on worker tags. Repeatable.
--detailed — Include worker ID and user columns.

verify

Compare each compiled .tgz's outputs against the torch reference.

Reads the reference npz produced by embedded prepare, then extracts share/.../sample_inputs.npz, share/.../generated/sample_outputs.npz and, when the worker shipped one, share/yasp_binary/build.json from each tgz. Sanity-checks that the worker saw the same bytes across every tgz (otherwise comparing outputs is meaningless), then pairs every output tensor BY NAME with the reference's and scores it — MAPE / MSE / max-diff / cosine (the same definitions the worker's build-time check uses), and top-K agreement when the output shape looks classification-y. Every dtype the engine emits is compared at its true shape and dtype (float16, int64 and bool included).

When a tgz's outputs npz carries the worker's benchmark_timings_ms telemetry array, a per-tgz benchmark table (min/mean/p50/p95/p99/max ms) is appended after the accuracy table. A reference npz produced by embedded prepare with --bench-iters > 0 carries host-torch timings under the same key; those render as the table's first row, so each target's latency reads against the host baseline.

Usage:

$ yasp-toolkit embedded verify [OPTIONS] {reference} {tgzs}...
Argument Default Description
reference required Reference outputs npz produced by embedded prepare -- or, for a staged model, the prepare directory itself (its chain_reference.npz: the chain's results by ref, plus every loop's step count).
tgzs... required One or more compiled output .tgz files to score against the reference.
Option Default Description
--max-mape <float> — Exit 1 if any output tensor's MAPE (100 * sum|ref-out| / sum|ref|) exceeds this percentage.
--min-cosine <float> — Exit 1 if any output tensor's cosine similarity to the reference falls below this.
--fx-nodes <file> — fx_nodes.json from embedded prepare: with it, the per-output cosines (debug taps included) are attributed to modules and the regions that INTRODUCE the drift are ranked.
--remove-outliers — Drop MAPE outliers via 1.5x IQR before averaging (mirrors the inference notebook).
--topk <int> — Report top-K classification agreement vs reference. Auto-on for 2D outputs with last-dim > 100.
--json — Emit machine-readable JSON instead of a Rich table.
--verbose — Output all missing outputs and profiling layers.

verify compares every array in the reference against the corresponding array in each build's outputs — the model outputs plus any debug taps exposed at prepare time — skipping only the benchmark telemetry. Each becomes a row scored with MAPE / MSE / max-diff / cosine (and top-K when a shape looks classification-y).

After the accuracy table, verify also prints a benchmark table (min/mean/p50/p95/p99/max ms). Each .tgz contributes a row from the worker's benchmark_timings_ms telemetry, and — when the reference npz was produced by prepare with --bench-iters > 0 (the default) — a torch host row from the host-torch timings, so you can read the compiled-vs-eager speedup directly. This replaces pulling demo_run.log by hand for latency.