Skip to content

Compile your first model

With your reference machine set up, a compile is four commands: start a runner on the target machine, then prepare, compile, and verify from the reference machine.

graph LR
    A[1. Start a runner] --> B[2. Prepare] --> C[3. Compile] --> D[4. Verify]

Run them yourself below, or skip to the agent and describe the job in plain English.

We'll use this tiny model.py throughout — a model file needs only a Model class and a get_fwd_args() returning the example inputs to trace. It uses spconv sparse 3D convolutions, which stock TensorRT can't convert (there's no native op) — so it's the perfect first model: yasp compiles it as-is.

# model.py  (needs `spconv` — e.g. pip install spconv-cu120, matching your CUDA)
import os

os.environ["CUDA_VISIBLE_DEVICES"] = ""  # trace on CPU — set before importing torch

import torch
from torch import nn
import spconv.pytorch as spconv


class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = spconv.SparseSequential(
            spconv.SubMConv3d(4, 16, kernel_size=3, padding=1, bias=False, indice_key="s0"),
            nn.ReLU(),
            spconv.SubMConv3d(16, 16, kernel_size=3, padding=1, bias=False, indice_key="s0"),
            nn.ReLU(),
        )
        self.head = nn.Linear(16, 10)

    def forward(self, features, coords):
        # features: (N, 4) per-voxel features; coords: (N, 4) as [batch, z, y, x]
        x = spconv.SparseConvTensor(features, coords, spatial_shape=[32, 32, 32], batch_size=1)
        x = self.net(x)
        pooled = x.features.max(dim=0, keepdim=True).values  # (1, 16)
        return self.head(pooled)


def get_fwd_args():
    # ~100 random occupied voxels in a 32³ grid, 4 features each.
    # spconv requires unique [batch, z, y, x] coordinates, so we dedupe
    # the random draws before building the (features, coords) pair.
    n = 100
    zyx = torch.randint(0, 32, (n, 3), dtype=torch.int32)
    zyx = torch.unique(zyx, dim=0)  # drop any duplicate voxels
    n = zyx.shape[0]
    features = torch.randn(n, 4)
    coords = torch.cat([torch.zeros(n, 1, dtype=torch.int32), zyx], dim=1)
    return (features, coords)

Step 1 — Start a runner

A model is compiled on the target GPU, not in the cloud — so you first put a worker there: a container that detects the hardware, registers with the router, and then builds an engine for every compile you send. You deploy it once; it stays up and serves all your compiles.

Follow Kronos workers to pick the image for your hardware and deploy it. Then confirm it registered and note its advertised arch_spec — the value you pass to --arch-spec in step 3:

yasp-toolkit embedded workers

Step 2 — Prepare

On the reference machine, turn model.py into an ONNX graph, a sample-input .npz, and a torch reference output:

yasp-toolkit embedded prepare model.py \
  --onnx-out      build/model.onnx \
  --npz-out       build/model.npz \
  --reference-out build/model.reference.npz

See embedded prepare.

Model won't prepare?

If prepare can't trace or export your model, send the exact error (and your model.py if you can share it) to support@yasp.ai.

Step 3 — Compile

Submit the build with your worker's arch_spec and a precision (fp32, fp16, bf16, int8). One command uploads the inputs, runs the build, and downloads the engine:

export ARCH=<advertised arch_spec from step 1>

yasp-toolkit embedded compile \
  --arch-spec "$ARCH" \
  --precision fp16 \
  --onnx      build/model.onnx \
  --npz       build/model.npz \
  --output    "build/model.$ARCH.fp16.tgz"

See embedded compile for quantization and other options.

Compile failed?

Send the exact error and the compilation ID — printed by compile when the task starts, and listed against the run on compile.yasp.ai — to support@yasp.ai.

Step 4 — Verify

Score the engine against the torch reference from step 2:

yasp-toolkit embedded verify \
  build/model.reference.npz \
  "build/model.$ARCH.fp16.tgz"

Reports MAPE, MSE, max diff, and cosine per output (plus top-K for classifiers). Pass several .tgz files to compare builds, or --max-mape <pct> to fail on accuracy regressions. See embedded verify.

Beyond accuracy, verify also surfaces the latency and per-layer profile the worker already measured on the target GPU during the compile. Each .tgz ships a captured --benchmark and --profile-layers run (in share/yasp_binary/examples/generated/), and verify reads those back to print two more tables after the accuracy scores — a Benchmark table (min/mean/p50/p95/p99/max ms, with the host-torch timings as a baseline row) and a per-build Profiling table (each layer's share of total time and its mean latency). So a single verify gives you accuracy, end-to-end latency, and hotspots without ever touching the target yourself — and when you pass several .tgz files, the benchmark rows line up so you can compare precisions at a glance.

Accuracy worse than expected, or verify errors?

Share the verify output and the build's compilation ID with support@yasp.ai.

Prefer the agent?

The same four steps, by conversation. Launch the shell and describe the job:

yasp

A startup check verifies your environment first; then the bundled skills handle the rest — e.g. "Deploy an embedded worker to gpu-lab-03, then compile model.py at fp16 and int8 and show me the latency and accuracy for each." It deploys the worker, prepares the inputs, runs the compiles in parallel, and reports the numbers back.

Where to next?

Your own models follow the exact same four steps — point prepare at a different model.py and reuse the running worker for every compile. From here:

  • Run the demo on your hardware — unpack the build on the target GPU and run inference with the bundled yasp_inference_demo, then profile it layer by layer.
  • Debug taps — when verify shows accuracy drift, expose intermediate tensors so verify pinpoints the layer where the engine diverges from torch: tensors you mark yourself, or every yasp plugin boundary automatically, with no model changes.
  • embedded reference — every flag for prepare, compile, and verify, plus precision sweeps and quantization.
  • dataset reference — register a calibration dataset for INT8 compiles.
  • CLI reference — the full yasp and yasp-toolkit surface, command by command.