Skip to main content

Schema refresh (operations guide)

This page covers the practical operations for refreshing engine schemas. For the conceptual treatment of how schema discovery works (and how it parallels the invariant-mining pipeline), see engine introspection pipelines.

For the format spec of the schema.discovered.json artefact, see schema discovered format.

For the operations side of the parallel mining pipeline, see miner pipeline (debugging guide).

Manual refresh (workflow_dispatch)​

For ad-hoc refreshes outside the Renovate flow:

# vllm or tensorrt
gh workflow run engine-pipeline.yml \
  --field engine=vllm \
  --field pr_number=123

# transformers: trigger Build engine image. Publish engine image fires
# on its success (workflow_run); schemas-transformers and
# invariants-transformers then fire on the push's success (also
# workflow_run).
gh workflow run engine-pipeline.yml

For local refresh against an installed engine:

./scripts/refresh_discovered_schemas.sh <engine>

The script runs python -m scripts.engine_introspectors --engine <engine> inside the matching container and writes the result to src/llenergymeasure/engines/<engine>/schema.discovered.json.

Manual version-bump guard​

If a developer bumps an engine version ARG in a Dockerfile without running discovery, the schema-version-check job in ci.yml catches the mismatch (path-filtered to docker/Dockerfile.*, skips Renovate PRs). On failure, the developer can either:

  • Run locally: ./scripts/refresh_discovered_schemas.sh <engine>
  • Trigger remotely: gh workflow run engine-pipeline.yml --field engine=<engine> --field pr_number=<N> (for transformers, the chain re-fires automatically once the build completes).

Prerequisites​

  • Mend Renovate GitHub App installed on the repo (free for open-source).
  • Self-hosted GPU runner available for the introspection jobs.
  • Docker plus NVIDIA Container Toolkit on the runner.

Troubleshooting​

SymptomCauseFix
Renovate not detecting bumpsfileMatch pattern doesn't cover the DockerfileCheck Renovate dashboard, verify docker/Dockerfile\\..* matches
Renovate not detecting transformers bumpscustomManagers regex not matchingVerify ARG TRANSFORMERS_VERSION=X.Y.Z format in docker/Dockerfile.transformers
Schema discovery fails to import engineContainer missing --gpus allVerify GPU runner has NVIDIA drivers plus Container Toolkit
Version guard fails on non-version changeShould not happen - guard only compares version ARGsIf it does, check _parse_arg regex in scripts/check_discovered_schema_versions.py
NGC registry auth failurePrivate image or rate-limitedAdd hostRules to renovate.json
Schema unchanged after discoveryEngine version did not change paramsExpected: workflow commits nothing and exits cleanly
Discovery commits a 2-line diff on every CI runLLENERGY_DISCOVERY_FROZEN_AT not setThe introspector writes a fresh wallclock discovered_at on every invocation; set the env var to a stable anchor (typically the author date of the most recent commit touching any input path)

Adding a new engine​

For step-by-step instructions on adding a new engine to llem (which covers both the introspection-pipeline registration and the broader plugin / Pydantic / Dockerfile work), see engine extensibility and extending miners.

The introspection-pipeline-specific surface is small:

  1. Add a per-engine module under scripts/engine_introspectors/, mirroring an existing *_introspector.py, and register it in scripts/engine_introspectors/__init__.py.
  2. Add a case to scripts/refresh_discovered_schemas.sh.
  3. Run discovery once: ./scripts/refresh_discovered_schemas.sh <engine>.
  4. Add a Renovate packageRule in renovate.json.
  5. If the Dockerfile ARG maps directly to the engine version, add an entry to _ENGINE_SPECS in scripts/check_discovered_schema_versions.py.

See also​