Profile the model

Benchmarking tells you how long the model takes end-to-end. Profiling tells you which layers took it — a per-layer GPU-time breakdown collected on the real device, which is what you want when deciding what to optimize or which precision to recompile at.

This uses the same yasp_inference_demo binary from your build — see Run the demo on your hardware for unpacking the .tgz, copying it to the target, and the base command. Everything below runs on the target GPU, from inside the package directory.

Collect a per-layer profile

Add --profile-layers:

./bin/yasp_inference_demo \
  --input  share/yasp_binary/examples/sample_inputs.npz \
  --output outputs.npz \
  --profile-layers \
  --iterations 200 \
  --profile-output profile_layers.csv

What happens:

• Warmup — the demo first runs max(5, 5% of iterations) untimed iterations so one-time costs (allocation, autotuning warm-up, clocks spinning up) don't pollute the numbers.
• Profiled iterations — it then runs --iterations (default 100) passes with a profiler attached, recording a GPU time for every layer on every iteration.
• The table — those raw samples are reduced to a per-layer table and appended to --profile-output (default profile_layers.csv).

Read the table

The profile is written as CSV, one row per layer, with a header row:

layer,time_pct,average_ms,median_ms,std_dev,total_ms
/net_0/Conv_myl0_2,24.3,0.0052,0.0053,0.0001,0.0524
/net_2/GlobalAveragePool_myl0_3,25.5,0.0055,0.0053,0.0006,0.0550
...

Column	Meaning
layer	The layer name, as the engine names it.
time_pct	The layer's share of the whole model's cumulative time — read this column first to find your hotspots.
average_ms	Mean per-iteration time for the layer.
median_ms	Median per-iteration time — less swayed by the odd slow iteration than the mean.
std_dev	Standard deviation of the per-iteration times — how noisy this layer's timing is.
total_ms	Total GPU time in this layer across all profiled iterations.

The output file is appended, never overwritten — the header is re-emitted at the start of each run's block — so you can profile several builds (e.g. fp32 vs fp16) into one file and compare their tables. Give each run its own --profile-output if you'd rather keep them separate.

The worker already captured one

Every compile ships a captured --profile-layers run at share/yasp_binary/examples/generated/profile_layers.csv (in this exact CSV format), plus its console log in demo_run_profile.log. embedded verify reads that CSV back and prints a per-layer profiling table alongside the accuracy and benchmark results — so you can see hotspots without running the demo yourself.

Profiling perturbs latency

Don't profile in production

Attaching the per-layer profiler forces a synchronization after every layer to time it individually. That serializes work the runtime would otherwise overlap, so the profiled wall-clock latency is higher than production latency — the proportions between layers are what's meaningful, not the absolute total. For latency-sensitive measurement, run --benchmark instead.

The workflow: use --profile-layers to find what to speed up, then --benchmark (without the profiler attached) to confirm the speedup lands.

Under the hood

The profile comes from the runtime's profile_per_layer() call — you can see exactly how it's driven, and how the raw per-iteration samples are reduced into the table above, in share/yasp_binary/examples/yasp_inference_demo.cpp. The LayerProfileView struct it fills is documented in the bundled API reference at share/yasp_binary/doc/yasp_binary/html/index.html.