yasp-toolkit eval

Score candidate kernels on real hardware: submit implementations against a reference module, measure their speedup and correctness, and inspect past evals and the evaluators that ran them.

The router host defaults to https://compile.yasp.ai/eval and can be overridden with $YASP_EVAL_ROUTER_HOST or the advanced-auth --router-* flags shown in --help.

compare

Score candidate kernels against a reference for speedup and accuracy.

Phases: create the task with len(implementation) slots, upload reference.pt2 + each implementation, start, poll until terminal, then download the worker-produced results.json and write it to -o (default stdout) verbatim.

The task id is printed as soon as it's obtained, so a failure mid-flow (upload, wait, download) is recoverable via the granular commands (eval get <task-id>).

Usage:

$ yasp-toolkit eval compare [OPTIONS]

Option	Default	Description
`-r, --reference <file>`	required	Reference module (`reference.pt2`) to compare implementations against.
`-i, --implementation <file>`	required	Candidate kernel .py file. Repeat once per implementation; first becomes `kernel_1.py`, etc.
`-g, --gpu-name <str>`	required	Hardware target the eval routes to (e.g. `nvidia-rtx-6000-pro`).
`--routing <str>`	—	Pin to a specific development evaluator by its routing key.
`-o, --output <filename>`	`-`	Write the `results.json` here. Default `-` (stdout).
`--progress / --no-progress`	`progress`	Render transfer progress bars.

benchmark

Measure a reference model's baseline timings on target hardware.

Phases: create the task, upload reference.pt2, start, poll until terminal, then download the worker-produced results.json and write it to -o (default stdout) verbatim.

The task id is printed as soon as it's obtained, so a failure mid-flow (upload, wait, download) is recoverable via the granular commands (eval get <task-id>).

Usage:

$ yasp-toolkit eval benchmark [OPTIONS]

Option	Default	Description
`-r, --reference <file>`	required	Reference module (`reference.pt2`) to benchmark.
`-g, --gpu-name <str>`	required	Hardware target the benchmark routes to (e.g. `nvidia-rtx-6000-pro`).
`--routing <str>`	—	Pin to a specific development evaluator by its routing key.
`-o, --output <filename>`	`-`	Write the `results.json` here. Default `-` (stdout).
`--progress / --no-progress`	`progress`	Render transfer progress bars.

get

Fetch a task's (eval or benchmark) current state. With --wait, poll until terminal.

Usage:

$ yasp-toolkit eval get [OPTIONS] {task_id}

Argument	Default	Description
`task_id`	required	Eval or benchmark task ID.

Option	Default	Description
`--wait`	—	Poll every 15s until the task is completed or failed.

cancel

Cancel a pending or started task.

Usage:

$ yasp-toolkit eval cancel [OPTIONS] COMMAND [ARGS]...

Options: * --help: Show this message and exit.

Commands:

eval: Cancel a pending or started eval task.
benchmark: Cancel a pending or started benchmark task.

`yasp-toolkit eval cancel eval`

Cancel a pending or started eval task.

The router rejects the request with a 409 once the task reached a terminal state (completed / failed / cancelled) -- there is nothing left to recall.

Usage:

$ yasp-toolkit eval cancel eval [OPTIONS] {task_id}

Argument	Default	Description
`task_id`	required	Eval task ID.

Options: * --help: Show this message and exit.

`yasp-toolkit eval cancel benchmark`

Cancel a pending or started benchmark task.

The router rejects the request with a 409 once the task reached a terminal state (completed / failed / cancelled) -- there is nothing left to recall.

Usage:

$ yasp-toolkit eval cancel benchmark [OPTIONS] {task_id}

Argument	Default	Description
`task_id`	required	Benchmark task ID.

Options: * --help: Show this message and exit.

inspect

Fetch each eval's results.json report and tabulate speedup and correctness.

Rows keep the order the task ids were given in; an eval holding several implementations contributes one row per opt_<n> entry. Not-completed tasks render their status in place of a speedup, and ids that are not eval tasks (404) are skipped -- so the id list can freely mix eval, benchmark, and compile tasks. Local reports passed via --file follow the fetched ones.

Composes with activity list --task-ids:

yasp-toolkit eval inspect $(yasp-toolkit activity list -c <CORRELATION_ID> --task-ids)

Usage:

$ yasp-toolkit eval inspect [OPTIONS] [task_ids]...

Argument	Default	Description
`task_ids...`	—	Eval task IDs, tabulated in the order given.

Option	Default	Description
`-k, --kernelgen <str>`	—	Kernelgen task ID whose `report.json` to fetch; repeatable. Rendered after the eval task ids.
`-f, --file <file>`	—	Already-downloaded report; repeatable. Tabulated after the task ids. Accepts both an eval `results.json` and a kernelgen `report.json` (eval result nested under `eval_result`).
`--full`	—	Render each report in full (long value lists summarized) instead of the summary table. Automatic when a single report renders.
`--progress / --no-progress`	`no-progress`	Render transfer progress bars.

key

Print a scoped key a Gaia worker can register with.

Use the JWT wherever a Gaia worker asks for YASP_API_KEY:

export WORKER_KEY=$(yasp-toolkit eval key)
kubectl create secret generic yasp-api-token --from-literal=token="$WORKER_KEY"

Usage:

$ yasp-toolkit eval key [OPTIONS]

Option	Default	Description
`--audience <str>`	`eval-api`	Token audience claim.
`--scope <str>`	—	Token scope (repeatable). Defaults to eval-register only.

workers

List registered evaluators (optionally filtered by tag).

Usage:

$ yasp-toolkit eval workers [OPTIONS]

Option	Default	Description
`-t, --tag <str>`	—	KEY=VALUE filter on evaluator tags. Repeatable.
`--detailed`	—	Include evaluator ID and user columns.