Skip to main content

Docker Setup Guide

llenergymeasure uses Docker to run vLLM and TensorRT-LLM (and future SGLang) engines in isolated containers with GPU access. This guide walks through setting up Docker with GPU support from scratch on a Linux host.

Docker is the supported path. All published measurements run in containers - that is what gives you reproducibility, dependency isolation between engines, and access to vLLM and TensorRT-LLM. The Transformers engine still has a process runner mode, but it is a developer convenience for smoke-testing on a host with a working CUDA toolchain, not a recommended way to produce results.

Prerequisites​

Before starting, confirm you have:

  • A Linux host (GPU passthrough in Docker is Linux-only - no macOS or Windows Docker GPU support)
  • An NVIDIA GPU installed
  • Root or sudo access to install system packages

No prior Docker knowledge is assumed. Each step includes a verification command.

Step 1: Install Docker​

If Docker is already installed, skip to the verification step.

Follow the official Docker Engine install guide for your distribution. For Ubuntu/Debian, Docker's own script is the easiest path:

# Install Docker Engine (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verify Docker Compose and Buildx versions (v2.32+ and v0.17+ recommended for fast rebuilds):

docker compose version   # need v2.32+
docker buildx version    # need v0.17+

If your versions are below these, you can upgrade the plugins directly:

# Upgrade Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64" \
  -o /usr/libexec/docker/cli-plugins/docker-compose
sudo chmod 755 /usr/libexec/docker/cli-plugins/docker-compose

# Upgrade Docker Buildx (find latest version at https://github.com/docker/buildx/releases)
BUILDX_VERSION=v0.32.1
sudo curl -L "https://github.com/docker/buildx/releases/download/${BUILDX_VERSION}/buildx-${BUILDX_VERSION}.linux-amd64" \
  -o /usr/libexec/docker/cli-plugins/docker-buildx
sudo chmod 755 /usr/libexec/docker/cli-plugins/docker-buildx

Post-install step: Add your user to the docker group so you can run Docker without sudo:

sudo usermod -aG docker $USER
newgrp docker

Verify Docker is working:

docker run hello-world

Expected output includes Hello from Docker!.

Rootless Docker and Podman​

The tested and supported runtime is rootful Docker Engine (the standard install above). llem resolves whatever docker binary is on PATH, so a Docker-compatible CLI - rootless Docker, or Podman exposed as docker - may work, and llem records the detected runtime in its environment snapshot. These paths are not tested and GPU passthrough on them is not validated: rootless Docker and Podman each have their own NVIDIA Container Toolkit / CDI setup that differs from rootful Docker. If you must use one, verify GPU passthrough with the container check in Step 4 first, and configure the toolkit per your runtime's own documentation. For a supported setup, use rootful Docker.

Docker image builds use BuildKit under the hood. The default builder has a conservative garbage-collection limit (~10% of disk) that is too small when building all three engine images (Transformers, vLLM, TensorRT). This causes build cache eviction and expensive recompilation (FA3 takes ~1 hour from scratch).

Create a dedicated builder with a 200 GiB cache limit:

make docker-builder-setup

This creates a docker-container driver builder called llem-builder with tuned GC limits (configured in docker/buildkitd.toml). To use it, set the BUILDX_BUILDER environment variable:

export BUILDX_BUILDER=llem-builder
docker compose build

Or add BUILDX_BUILDER=llem-builder to your .env file for project-scoped use.

The command is idempotent - running it again is a no-op if the builder already exists.

To recreate the builder (e.g. after changing buildkitd.toml):

make docker-builder-rm
make docker-builder-setup

Step 2: Install NVIDIA Drivers​

If nvidia-smi already works on your host, skip this step.

nvidia-smi

If nvidia-smi is not found or returns an error, install NVIDIA drivers for your distribution. Follow the NVIDIA driver installation guide for your OS and GPU model.

Expected nvidia-smi output (your GPU name and driver version will differ):

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|=========================================================================================|
|   0  NVIDIA A100-SXM4-80GB          On  |   00000000:00:04.0 Off |                    0 |
| N/A   34C    P0             54W /  400W |       0MiB /  81920MiB |      0%      Default |
+-----------------------------------------------------------------------------------------+

Step 3: Install NVIDIA Container Toolkit​

The NVIDIA Container Toolkit enables Docker containers to access the host GPU. This is the critical step that makes docker run --gpus all work.

Ubuntu/Debian:

# Add NVIDIA Container Toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure Docker to use the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker

# Restart Docker for the configuration to take effect
sudo systemctl restart docker

Other distributions: Follow the NVIDIA Container Toolkit install guide for RHEL/CentOS, Fedora, SUSE, or Arch Linux.

Step 4: Verify GPU Access in Docker​

Run a container with GPU access and check that nvidia-smi works inside it:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

This command:

  • Pulls a small CUDA base image (~100 MB)
  • Launches a container with all host GPUs passed through (--gpus all)
  • Runs nvidia-smi inside the container
  • Exits and removes the container (--rm)

Expected output: the same nvidia-smi table you saw in Step 2, but printed from inside the container.

This is exactly the check that llem's Docker pre-flight performs automatically before launching any engine container. If this command works, llem pre-flight will pass.

If this command fails, see Troubleshooting below.

Step 5: Verify with LLenergyMeasure​

With Docker and NVIDIA CT installed, verify that llem sees the GPU:

llem doctor

Expected output shows your GPU under the GPU / driver section and the Docker runner availability under Docker and Engines. See Installation for the full llem doctor output format.

Run a vLLM experiment by creating a YAML file:

# experiment.yaml
serving_mode: offline
engine: vllm
task:
  model: gpt2
  dataset:
    source: aienergyscore
    n_prompts: 50
runners:
  vllm: container

Then run it:

llem run experiment.yaml

llem will automatically pull the vLLM Docker image on first use, launch a container, run the experiment inside it, and return the results. See Getting Started for an annotated walkthrough.

Run a TensorRT-LLM experiment by creating a YAML file:

# experiment.yaml
serving_mode: offline
engine: tensorrt
task:
  model: meta-llama/Llama-2-7b-hf
  dataset:
    source: aienergyscore
    n_prompts: 50
runners:
  tensorrt: container

Then run it:

llem run experiment.yaml

llem will pull the TensorRT-LLM Docker image and run inference. The default pytorch backend loads the model directly; the compiled trt backend instead compiles a TensorRT engine (first run of a config only - several minutes) and caches it on disk. See Getting Started for the full TensorRT-LLM walkthrough.

Image Management​

Image sources​

each engine has two possible image sources:

SourceTag patternBuilt byUse case
Local buildllenergymeasure:{engine}make docker-build-{engine}Development - reflects current source tree
Registry (transformers)ghcr.io/henrycgbaker/llenergymeasure/transformers:v{package_version}CI on release tagsProduction, CI, pip-install users
Upstream (vllm, tensorrt)vllm/vllm-openai:v{engine_version}, nvcr.io/nvidia/tensorrt-llm/release:{engine_version}vLLM / NVIDIA (NGC)Same - the canonical engine image, project source bind-mounted

Image resolution​

When you run llem run, the tool resolves which image to use for each engine. Resolution follows a precedence chain (highest wins):

  1. Environment variable LLEM_IMAGE_{ENGINE} (e.g. LLEM_IMAGE_VLLM=my/custom:tag)
  2. Study YAML images: section
  3. Runner spec shorthand (container:my/custom:tag in runners:)
  4. User config images: section (~/.config/llenergymeasure/config.yaml)
  5. Smart default: local build image if present, otherwise registry image

In practice, most users rely on the smart default (level 5). If you have built images locally with make docker-build, those are used. Otherwise, llem resolves the per-engine default: the first-party GHCR image for transformers (at the package version), and the canonical upstream image for vLLM / TensorRT-LLM (vllm/vllm-openai, nvcr.io/nvidia/tensorrt-llm/release, at the pinned engine version).

Heads-up: a stale local tag silently wins. Because the local build takes precedence, a months-old llenergymeasure:{engine} tag keeps winning level 5 long after the pinned default has moved on, which can surface as a schema-handshake mismatch. When a local tag wins, llem logs a warning naming the pinned default it bypassed, and llem doctor flags it on the engine's row. To restore the pinned default, remove the local tag with docker rmi llenergymeasure:{engine}; to keep a specific image, pin it explicitly at level 1-4 (for example LLEM_IMAGE_{ENGINE}).

Auto-pull on first use​

When llem needs a registry image that is not cached locally, it pulls it automatically before the experiment runs. The preflight panel shows which images will be used and whether they are cached or need pulling:

Runners
  vllm              docker
    ↳ llenergymeasure:vllm                              ← local build
  transformers      docker
    ↳ ghcr.io/henrycgbaker/llenergymeasure/transformers:v0.6.0  ← registry

Study-level image preparation​

For multi-experiment studies, llem checks and pulls all required Docker images once before the first experiment runs, not per-experiment. This avoids redundant pulls when multiple experiments share the same engine image. The CLI shows this as a "Preparing Docker images" section with per-image status (cached vs pulled) and metadata (image ID, size, age, layers).

Pre-fetch images manually​

For offline environments or to avoid pull latency during experiments:

# Pull the first-party transformers image for your installed version
make docker-pull

# Or pull each engine's default image individually. transformers is a
# first-party GHCR image tagged with the llenergymeasure version; vLLM and
# TensorRT-LLM are upstream images tagged with the pinned engine version.
docker pull ghcr.io/henrycgbaker/llenergymeasure/transformers:v0.6.0
docker pull vllm/vllm-openai:v0.19.1
docker pull nvcr.io/nvidia/tensorrt-llm/release:1.2.1

Replace the transformers tag with your installed version (llem --version); the vLLM and TensorRT-LLM tags track the pins in engine_versions/<engine>/current.yaml. llem doctor prints the exact image each engine resolves to.

Check current image resolution​

To see which images llem will use for each engine:

make docker-images

Output shows local vs registry source for each engine:

=== Image resolution ===
  tensorrt   -> nvcr.io/nvidia/tensorrt-llm/release:1.2.1  (registry)
  transformers -> ghcr.io/henrycgbaker/llenergymeasure/transformers:v0.6.0  (registry)
  vllm       -> vllm/vllm-openai:v0.19.1  (registry)

Building or pulling images locally​

Only the Transformers engine is built from a project Dockerfile. vLLM and TensorRT-LLM use canonical upstream images directly and bind-mount the project source at run time.

# Transformers - build from project source
make docker-build

# vLLM - pull upstream
docker pull vllm/vllm-openai:v0.19.1

# TensorRT-LLM - pull upstream (NGC)
docker pull nvcr.io/nvidia/tensorrt-llm/release:1.2.1

make docker-build builds the project's first-party engine images (currently just transformers - the only project-built image). It uses docker compose build under the hood and pulls cached layers from GHCR on first build (see Fast rebuilds and first-pull cost for the full mechanism).

Advanced. Setting COMPOSE_BAKE=true routes builds through buildx bake for parallel multi-engine builds. With the current cache architecture this is rarely worth enabling - vLLM/TRT cold builds are already 4-13 min and warm rebuilds are seconds, so the parallelism gain is small. Left out of .env.example to avoid noise; opt in only if you frequently run make docker-build from cold.

When to rebuild. Images bundle the llenergymeasure source at build time. If you modify config models, engines, or the container entrypoint, rebuild for changes to take effect inside containers. Process-runner experiments (Transformers without a container) use the installed source directly and do not need a rebuild.

Override images in YAML​

runners:
  transformers: process                    # host execution, no container
  vllm: container                          # default resolution (local → registry)
  tensorrt: "container:my/custom:tag"      # explicit image override

Future engines​

SGLang images (when SGLang ships) will follow the same naming convention, resolution logic, and auto-pull behaviour. No additional setup is needed when SGLang ships.

Layer cache sharing via GHCR registry​

See installation.md - Fast rebuilds and first-pull cost for the user-facing walkthrough (mechanism, sizes, authentication, offline fallback) and the ref breakdown of what the local seed and the promotion publish (transformers-cache:transformers-<VER> promotion source, its -buildcache companion, and the canonical transformers:transformers-<VER>/transformers:latest).

Operator notes:

  • make docker-seed-transformers writes the :transformers-<VER>-buildcache ref (separate from the runnable promotion-source image), so storage growth is bounded by version drift.
  • Inspect what's cached on the active builder: docker buildx du --builder llem-builder.
  • If the cache is corrupt, recreate it with make docker-builder-rm && make docker-builder-setup. The remote buildcache ref repopulates on the next local seed.

TensorRT-LLM engine build cache​

Distinct from the Docker layer cache above, and specific to the compiled trt backend: the TensorRT engine build cache holds compiled .engine artefacts so re-running the same experiment config skips the multi-minute compile. It ships on out of the box - .env.example sets LLEM_TRT_BUILD_CACHE_ENABLED=1 (delete the line to fall back to TensorRT-LLM's disabled default). It lives on the host at ~/.cache/trt-llm and is bind-mounted into every tensorrt container at /root/.cache/trt-llm; llem defaults the cache location to that mount out of the box (override with LLEM_TRT_BUILD_CACHE_PATH). TensorRT-LLM keys each entry by the full build config (model, dtype, tensor-parallel size, max-shape, quantisation) plus the engine version: an identical config hits even across separate containers, while a TP / quantisation / max-shape change or a version bump keys to a distinct engine and misses. Each result records engine_build_cache_hit so you can tell a reused engine from a fresh compile.

The lifecycle is manual and visible - llem never auto-evicts. Inspect it with:

llem doctor

which reports the cache location, engine-entry count, and total size. Entries are large (often 1-15 GB each); clean the cache manually when it grows. Entries are written by the container's root process, so removing them from the host needs sudo:

sudo rm -rf ~/.cache/trt-llm/engine-*

Image labels​

docker/Dockerfile.transformers stamps a single OCI label:

LabelPurpose
org.opencontainers.image.sourcePoints at the GitHub repository

Inspect the labels on a local image:

docker image inspect llenergymeasure:transformers \
    --format '{{json .Config.Labels}}' | python3 -m json.tool

Schema-fingerprint label is legacy; engine-version probe is live. Earlier versions stamped org.opencontainers.image.version and llem.expconf.schema.fingerprint labels at build time so StudyRunner._prepare_images could detect host/container schema skew. Once the image stopped baking the project source (it is bind-mounted at runtime - see the development guide), the schema-fingerprint check became structurally redundant: the in-container source always equals the host source. That label is no longer set on first-party images and never existed on upstream-direct images (vllm, tensorrt). What version_handshake.py does today is a different check: it probes each image's engine library version (vllm.__version__, tensorrt_llm.__version__, transformers.__version__) and compares it against the engine_version envelope on the wheel-bundled rules

  • schema artefacts. A real library/artefact mismatch is a hard error; set LLEM_SKIP_IMAGE_CHECK=1 to bypass if you know the skew is harmless.

See troubleshooting.md for the remediation flow when a mismatch is reported.

Troubleshooting​

"docker: Error response from daemon: could not select device driver"​

NVIDIA Container Toolkit is not installed, or Docker was not restarted after configuration.

Fix:

  1. Confirm toolkit is installed: which nvidia-ctk should return a path.
  2. Re-run: sudo nvidia-ctk runtime configure --runtime=docker
  3. Restart Docker: sudo systemctl restart docker
  4. Retry the docker run --gpus all ... command.

"nvidia-container-cli: initialization error"​

A driver version mismatch between the host NVIDIA driver and the CUDA version in the container.

Fix: Check your host driver version with nvidia-smi. The container image requires a minimum driver version for its CUDA release. See the CUDA compatibility matrix to confirm your driver supports the container's CUDA version.

For example, CUDA 12.4 requires driver >= 525.60.13.

"Permission denied" when running docker commands​

Your user is not in the docker group.

Fix:

sudo usermod -aG docker $USER
newgrp docker   # apply group change in current shell

Or log out and log back in. Verify with: groups | grep docker.

GPU not visible inside container​

The --gpus all flag is missing from the docker run command.

llem adds --gpus all automatically when launching engine containers. If you are running Docker commands manually, ensure you include --gpus all (or --gpus device=0 for a specific GPU).

To make llem itself target specific GPUs on a shared host, there are two levers:

  • LLEM_DOCKER_GPUS (env var, host-wide). Set it to the docker run --gpus value (empty means every visible GPU). Quote multi-device values so the shell keeps the comma, e.g. LLEM_DOCKER_GPUS="device=2,3".
  • study_execution.gpu_indices (study YAML, per study). A list of host GPU indices, e.g. gpu_indices: [2, 3], translated to --gpus device=2,3. This lets a study YAML declare its own GPU placement without an env var.

Both are host device indices as the NVIDIA driver / NVML enumerate them (what nvidia-smi shows). Restricting at the docker level (rather than setting CUDA_VISIBLE_DEVICES inside the container) keeps CUDA and NVML indices consistent inside the container - both re-enumerate from 0 - so energy attribution addresses the correct physical device without any index translation.

Precedence: LLEM_DOCKER_GPUS (env) overrides study_execution.gpu_indices (config). When both are set the env wins and llem logs a one-line warning; the config indices are ignored. Because the env fully overrides the config, the two never compose - there is no "config indices index into the env-restricted set" case to reason about. Pick one lever per run.

For tensor-parallel runs, llem also forwards every NCCL_* host variable into the container; see multi-GPU with TensorRT-LLM.

LLEM_DOCKER_GPUS also accepts a Multi-Instance GPU (MIG) instance UUID (device=MIG-<uuid>) to pin llem to a single MIG slice on a partitioned A100 or H100. See Running on a cloud GPU VM - MIG and partitioned GPUs for the operational steps and the power-telemetry caveat.

Shared memory errors with vLLM​

vLLM requires more than the default 64 MB of shared memory (/dev/shm). llem automatically sets --shm-size 8g when launching engine containers. Override the size with the LLEM_DOCKER_SHM_SIZE env var (e.g. LLEM_DOCKER_SHM_SIZE=16g for very large tensor-parallel runs, or a smaller value on memory-constrained hosts); empty means the 8g default. If you are running the vLLM container manually, add --shm-size 8g to your docker run command.

Model cache location​

llem bind-mounts the host HuggingFace cache into each engine container so model weights persist across runs instead of re-downloading. Override the host directory with the LLEM_DOCKER_HF_CACHE env var (default $HOME/.cache/huggingface) to point at shared storage or a larger disk.

Pre-flight check failures​

llem runs pre-flight checks before launching any Docker container. The checks and their failure modes:

CheckFailure messageFix
Docker CLI on PATHDocker not found on PATHInstall Docker Engine (Step 1)
NVIDIA Container Toolkit binaryNVIDIA Container Toolkit not foundInstall NVIDIA CT (Step 3)
Host nvidia-smiWarning (non-blocking)Expected if using remote Docker daemon
GPU visibility in containerGPU not accessible inside Docker containerRe-run Steps 3-4
CUDA/driver compatibilityCUDA/driver compatibility error inside containerUpdate host driver

To bypass pre-flight checks temporarily (not recommended for production):

llem run experiment.yaml --skip-preflight

Keeping Engine Schemas Fresh​

When you update an engine version (by bumping the pin in engine_versions/<engine>/current.yaml), the discovered parameter schema must be regenerated. This is a local maintainer step - CI verifies the committed schema but never runs discovery itself. Run:

./scripts/refresh_discovered_schemas.sh <engine>     # equivalently: make discover-schema ENGINE=<engine>

The script runs discovery inside the engine's Docker image, writes src/llenergymeasure/engines/<engine>/schema.discovered.json, and prints the diff for you to review and commit. See Schema refresh (operations guide) for the full workflow and the schema-version-check CI guard.

Next Steps​