Skip to content

Run the demo on your hardware

Every compile produces a self-contained C++ package that runs the engine on the target GPU. Inside it is yasp_inference_demo — a ready-built command-line tool that loads your model, feeds it NPZ inputs, writes the outputs back, and can benchmark end-to-end latency.

graph LR
    A[1. Unpack the build] --> B[2. Copy to the target] --> C[3. Run the demo]

What's in the build

Compiling a model gives you a .tgz named for the arch and precision it was built for, e.g. model.amd64-sm120-u2404-cu129-trt1014.fp32.tgz. Unpack it:

tar xzf model.amd64-sm120-u2404-cu129-trt1014.fp32.tgz

You get a versioned package directory — yasp_binary-0.1.0-Linux-x86_64/:

yasp_binary-0.1.0-Linux-x86_64/
├── bin/
│   └── yasp_inference_demo          # the pre-built demo (run this)
├── lib/
│   └── libyasp_binary.so*           # the engine + runtime
├── include/yasp/                    # headers, to link the runtime into your own app
└── share/yasp_binary/
    ├── examples/
    │   ├── yasp_inference_demo.cpp  # the demo's source — read it to see the API
    │   ├── sample_inputs.npz        # a ready-to-run input for this model
    │   ├── CMakeLists.txt           # build the demo yourself against the headers
    │   ├── external/cnpy/           # tiny NPZ reader the demo uses
    │   └── generated/               # captured example runs from the worker (see below)
    │       ├── demo_run_benchmark.log  # console log of the worker's --benchmark run: command line + timings
    │       ├── demo_run_profile.log    # console log of the worker's --profile-layers run
    │       ├── profile_layers.csv      # the per-layer profile that profiling run produced
    │       └── sample_outputs.npz      # the outputs.npz produced, incl. benchmark_timings_ms
    └── doc/yasp_binary/             # Doxygen HTML API reference + BUILD_INFO

bin/yasp_inference_demo is already compiled — you don't need to build anything to run it. The share/…/examples/ copy of yasp_inference_demo.cpp is there so you can see how the runtime API is driven and adapt it into your own integration.

Copy the package to the target

The demo runs on the GPU it was compiled for, not on your reference machine. The engine is hardware-specific — a sm120 build only loads on that architecture. Copy the whole unpacked directory across so the layout stays intact:

scp -r yasp_binary-0.1.0-Linux-x86_64 user@target:~/

Keep the directory structure

yasp_inference_demo finds libyasp_binary.so through an rpath of $ORIGIN/../lib — it looks one level up from bin/. As long as bin/ and lib/ stay siblings you don't need to set LD_LIBRARY_PATH. The target only needs the CUDA runtime (libcudart.so.12) that the build was made against — the arch string (cu129) tells you which.

Everything below runs on the target, from inside the package directory:

cd ~/yasp_binary-0.1.0-Linux-x86_64

Inspect the engine

Confirm the engine loads and inspect its interface — inputs, outputs, dtypes, and shapes — without running inference:

./bin/yasp_inference_demo --info

Run a single inference

The build ships a matching sample input, so this works out of the box:

./bin/yasp_inference_demo \
  --input  share/yasp_binary/examples/sample_inputs.npz \
  --output outputs.npz

The input NPZ must contain one array per model input, named to match the input tensor (--info lists the names); results are written to outputs.npz under the output tensor names. If a name or dtype doesn't line up, the demo tells you which one.

Benchmark end-to-end latency

--benchmark runs untimed warmup iterations, then times each iteration end-to-end and reports the distribution:

./bin/yasp_inference_demo \
  --input  share/yasp_binary/examples/sample_inputs.npz \
  --output outputs.npz \
  --benchmark \
  --iterations 1000

===== Benchmark Results =====
Iterations: 1000
Min:    0.011241 ms
Max:    0.011799 ms
Mean:   0.0113966 ms
p50:    0.011338 ms
p95:    0.011669 ms
p99:    0.011773 ms
=============================

The per-iteration timings are also saved as a benchmark_timings_ms array in the output NPZ, so you can pull them into a notebook.

The worker already ran this for you

You don't have to run anything to see how this model performs — the worker benchmarks and profiles the model as part of the compile, and ships the captured results in share/yasp_binary/examples/generated/:

  • demo_run_benchmark.log — a full --benchmark run: the exact command line, the loaded input, and the resulting benchmark distribution.
  • demo_run_profile.log + profile_layers.csv — a full --profile-layers run and the per-layer GPU-time table it produced.
  • sample_outputs.npz — the outputs of those runs, including the benchmark_timings_ms array.

Open the logs to see what correct output looks like before running your own — or just run embedded verify, which reads these same files back and prints the benchmark and profile tables alongside the accuracy scores.

All options

Flag Purpose
--input <file> Input NPZ; arrays named to match the model's input tensors.
--output <file> Output NPZ for the results.
--info Print the engine's inputs/outputs/shapes and exit.
--benchmark Time each iteration end-to-end; report min/max/mean/p50/p95/p99.
--profile-layers Collect a per-layer GPU-time profile — see Profile the model.
--iterations <N> Iterations to benchmark or profile (default 100).
--profile-output <f> Where the layer profile CSV is appended (default profile_layers.csv).
--verbosity <level> Runtime log level: verbose, info, warning (default), error.
-h, --help Full usage.

The source

share/yasp_binary/examples/yasp_inference_demo.cpp is the whole thing — argument parsing, loading NPZ tensors onto the GPU, and the executeAsync / waitUntilExecutionComplete loop. It's the reference for embedding the YASP runtime in your own application: find_package(YaspBinary) against the bundled lib/cmake/YaspBinary, link YaspBinary::yasp_binary, and drive the same API. The full class reference is in share/yasp_binary/doc/yasp_binary/html/index.html.

Where to next?

  • Profile the model — get a per-layer GPU-time breakdown on real hardware to find your hotspots.
  • embedded reference — recompile at other precisions and verify accuracy against a torch reference.