TensorRT-LLM Engine Schema

Engine version: 1.2.1
Discovered at: 2026-07-13T00:00:00+00:00
Discovery method: TorchLlmArgs+TrtLlmArgs.model_json_schema() union with per-field backends + dataclasses.fields(SamplingParams)
Schema version: 1.0.0

Summary: 95 engine parameters, 48 sampling parameters.

Discovery limitations

engine_params - BuildConfig is not a Pydantic model; appears as Optional[object] in the schema Affected fields: build_config

engine_params - shared field: TorchLlmArgs and TrtLlmArgs disagree on type/default/description; the trt (TrtLlmArgs) spec is recorded and the field is tagged applicable to both backends Affected fields: load_format

sampling_params - SamplingParams is a dataclass; no per-field descriptions

Engine Parameters

Field	Type	Default	Backends	Description
`model`	`string`	-	pytorch, trt	The path to the model checkpoint or the model name from the Hugging Face Hub.
`tokenizer`	`string	None`	-	pytorch, trt
`tokenizer_mode`	`Literal['auto', 'slow']`	`auto`	pytorch, trt	The mode to initialize the tokenizer.
`custom_tokenizer`	`string	None`	-	pytorch, trt
`skip_tokenizer_init`	`boolean`	`false`	pytorch, trt	Whether to skip the tokenizer initialization.
`trust_remote_code`	`boolean`	`false`	pytorch, trt	Whether to trust the remote code.
`tensor_parallel_size`	`integer`	`1`	pytorch, trt	The tensor parallel size.
`dtype`	`string`	`auto`	pytorch, trt	The data type to use for the model.
`revision`	`string	None`	-	pytorch, trt
`tokenizer_revision`	`string	None`	-	pytorch, trt
`pipeline_parallel_size`	`integer`	`1`	pytorch, trt	The pipeline parallel size.
`context_parallel_size`	`integer`	`1`	pytorch, trt	The context parallel size.
`gpus_per_node`	`integer	None`	-	pytorch, trt
`moe_cluster_parallel_size`	`integer	None`	-	pytorch, trt
`moe_tensor_parallel_size`	`integer	None`	-	pytorch, trt
`moe_expert_parallel_size`	`integer	None`	-	pytorch, trt
`enable_attention_dp`	`boolean`	`false`	pytorch, trt	Enable attention data parallel.
`enable_lm_head_tp_in_adp`	`boolean`	`false`	pytorch, trt	Enable LM head TP in attention dp.
`pp_partition`	`array	None`	-	pytorch, trt
`cp_config`	`object	None`	-	pytorch, trt
`load_format`	`Literal['auto', 'dummy']`	`auto`	pytorch, trt	The format to load the model.
`fail_fast_on_attention_window_too_large`	`boolean`	`false`	pytorch, trt	Fail fast when attention window is too large to fit even a single sequence in the KV cache.
`enable_lora`	`boolean`	`false`	pytorch, trt	Enable LoRA.
`lora_config`	`LoraConfig	None`	-	pytorch, trt
`kv_cache_config`	`KvCacheConfig`	-	pytorch, trt	KV cache config.
`enable_chunked_prefill`	`boolean`	`false`	pytorch, trt	Enable chunked prefill.
`guided_decoding_backend`	`string	None`	-	pytorch, trt
`batched_logits_processor`	`Optional[tensorrt_llm.sampling_params.BatchedLogitsProcessor]`	-	pytorch, trt	Batched logits processor.
`iter_stats_max_iterations`	`integer	None`	-	pytorch, trt
`request_stats_max_iterations`	`integer	None`	-	pytorch, trt
`peft_cache_config`	`PeftCacheConfig	None`	-	pytorch, trt
`scheduler_config`	`SchedulerConfig`	-	pytorch, trt	Scheduler config.
`cache_transceiver_config`	`CacheTransceiverConfig	None`	-	pytorch, trt
`sparse_attention_config`	`RocketSparseAttentionConfig	DeepSeekSparseAttentionConfig	SkipSoftmaxAttentionConfig	None`
`speculative_config`	`DraftTargetDecodingConfig	EagleDecodingConfig	LookaheadDecodingConfig	MedusaDecodingConfig
`max_batch_size`	`integer	None`	-	pytorch, trt
`max_input_len`	`integer	None`	-	pytorch, trt
`max_seq_len`	`integer	None`	-	pytorch, trt
`max_beam_width`	`integer	None`	-	pytorch, trt
`max_num_tokens`	`integer	None`	`8192`	pytorch, trt
`gather_generation_logits`	`boolean`	`false`	pytorch, trt	Gather generation logits.
`num_postprocess_workers`	`integer`	`0`	pytorch, trt	The number of processes used for postprocessing the generated tokens, including detokenization.
`postprocess_tokenizer_dir`	`string	None`	-	pytorch, trt
`reasoning_parser`	`string	None`	-	pytorch, trt
⚠️ `decoding_config`	`Optional[tensorrt_llm.llmapi.llm_args.DecodingConfig]`	-	pytorch, trt	The decoding config.
`otlp_traces_endpoint`	`string	None`	-	pytorch, trt
`backend`	`string	None`	-	pytorch, trt
`return_perf_metrics`	`boolean`	`false`	pytorch, trt	Return perf metrics.
`orchestrator_type`	`string	None`	-	pytorch, trt
`env_overrides`	`object	None`	-	pytorch, trt
`enable_tqdm`	`boolean`	`false`	trt	Enable tqdm for progress bar.
`workspace`	`string	None`	-	trt
`enable_build_cache`	`Union[tensorrt_llm.llmapi.build_cache.BuildCacheConfig, bool]`	`false`	trt	Enable build cache.
`extended_runtime_perf_knob_config`	`ExtendedRuntimePerfKnobConfig	None`	-	trt
`calib_config`	`CalibConfig	None`	-	trt
`quant_config`	`QuantConfig	None`	-	trt
`embedding_parallel_mode`	`string`	`SHARDING_ALONG_VOCAB`	trt	The embedding parallel mode.
`fast_build`	`boolean`	`false`	trt	Enable fast build.
`build_config`	`BuildConfig	None`	-	trt
`enable_prompt_adapter`	`boolean`	`false`	trt	Enable prompt adapter.
`max_prompt_adapter_token`	`integer`	`0`	trt	The maximum number of prompt adapter tokens.
`batching_type`	`BatchingType	None`	-	trt
`normalize_log_probs`	`boolean`	`false`	trt	Normalize log probabilities.
`garbage_collection_gen0_threshold`	`integer`	`20000`	pytorch	Threshold for Python garbage collection of generation 0 objects.Lower values trigger more frequent garbage collection.
`cuda_graph_config`	`CudaGraphConfig	None`	-	pytorch
`attention_dp_config`	`AttentionDpConfig	None`	-	pytorch
`disable_overlap_scheduler`	`boolean`	`false`	pytorch	Disable the overlap scheduler.
`moe_config`	`MoeConfig`	-	pytorch	MoE config.
`nvfp4_gemm_config`	`Nvfp4GemmConfig`	-	pytorch	NVFP4 GEMM backend config.
`attn_backend`	`string`	`TRTLLM`	pytorch	Attention backend to use.
`sampler_type`	`string	SamplerType`	`auto`	pytorch
`sampler_force_async_worker`	`boolean`	`false`	pytorch	Force usage of the async worker in the sampler for D2H copies, even if confidential compute is not active. Normally, the async worker should only be used when confidential compute is active. This argument is provided to enable it for testing purposes, irrespective of confidential compute state.
`enable_iter_perf_stats`	`boolean`	`false`	pytorch	Enable iteration performance statistics.
`enable_iter_req_stats`	`boolean`	`false`	pytorch	If true, enables per request stats per iteration. Must also set enable_iter_perf_stats to true to get request stats.
`print_iter_log`	`boolean`	`false`	pytorch	Print iteration logs.
`perf_metrics_max_requests`	`integer`	`0`	pytorch	The maximum number of requests for perf metrics. Must also set request_perf_metrics to true to get perf metrics.
`batch_wait_timeout_ms`	`number`	`0`	pytorch	If greater than 0, the request queue might wait up to batch_wait_timeout_ms to receive max_batch_size requests, if fewer than max_batch_size requests are currently available. If 0, no waiting occurs.
`batch_wait_timeout_iters`	`integer`	`0`	pytorch	Maximum number of iterations the scheduler will wait to accumulate new coming requests for improved GPU utilization efficiency. If greater than 0, the scheduler will delay batch processing to gather more requests up to the specified iteration limit. If 0, disables timeout-iters-based batching delays.
`batch_wait_max_tokens_ratio`	`number`	`0`	pytorch	Token accumulation threshold ratio for batch scheduling optimization. If greater than 0, the scheduler will accumulate requests locally until the total token count reaches batch_wait_max_tokens_ratio * max_num_tokens. This mechanism enhances GPU utilization efficiency by ensuring adequate batch sizes.If 0 disables token-based batching delays.
`torch_compile_config`	`TorchCompileConfig	None`	-	pytorch
`enable_autotuner`	`boolean`	`true`	pytorch	Enable autotuner for all tunable ops. This flag is for debugging purposes only, and the performance may significantly degrade if set to false.
`enable_layerwise_nvtx_marker`	`boolean`	`false`	pytorch	If true, enable layerwise nvtx marker.
`enable_min_latency`	`boolean`	`false`	pytorch	If true, enable min-latency mode. Currently only used for Llama4.
`stream_interval`	`integer`	`1`	pytorch	The iteration interval to create responses under the streaming mode. Set this to a larger value when the batch size is large, which helps reduce the streaming overhead.
`force_dynamic_quantization`	`boolean`	`false`	pytorch	If true, force dynamic quantization. Defaults to False.
`allreduce_strategy`	`string	None`	`AUTO`	pytorch
`checkpoint_loader`	`Optional[tensorrt_llm._torch.models.checkpoints.BaseCheckpointLoader]`	-	pytorch	The checkpoint loader to use for this LLM instance. You may use a custom checkpoint loader by subclassing `BaseCheckpointLoader` and providing an instance of the subclass here to load weights from a custom checkpoint format. If neither checkpoint_format nor checkpoint_loader are provided, checkpoint_format will be set to HF and the default HfCheckpointLoader will be used. If checkpoint_format and checkpoint_loader are both provided, checkpoint_loader will be ignored.
`checkpoint_format`	`string	None`	-	pytorch
`kv_connector_config`	`KvCacheConnectorConfig	None`	-	pytorch
`mm_encoder_only`	`boolean`	`false`	pytorch	Only load/execute the vision encoder part of the full model. Defaults to False.
`ray_worker_extension_cls`	`string	None`	-	pytorch
`ray_placement_config`	`RayPlacementConfig	None`	-	pytorch
`enable_sleep`	`boolean`	`false`	pytorch	Enable LLM sleep feature. Sleep feature requires extra setup that may slowdown model loading.Only enable it if you intend to use this feature.
`disable_flashinfer_sampling`	`boolean`	`false`	pytorch	Disable the use of FlashInfer.sampling. This option is likely to be removed in the future.
`max_stats_len`	`integer`	`1000`	pytorch	The max number of performance statistic entries.

Sampling Parameters

Field	Type	Default	Description
`end_id`	`int	None`	-
`pad_id`	`int	None`	-
`max_tokens`	`int`	`32`
`bad`	`str	list[str]	None`
`bad_token_ids`	`list[int]	None`	-
`stop`	`str	list[str]	None`
`stop_token_ids`	`list[int]	None`	-
`include_stop_str_in_output`	`bool`	`false`
`embedding_bias`	`Tensor	None`	-
`logits_processor`	`LogitsProcessor	list[LogitsProcessor]	None`
`apply_batched_logits_processor`	`bool`	`false`
`n`	`int`	`1`
`best_of`	`int	None`	-
`use_beam_search`	`bool`	`false`
`top_k`	`int	None`	-
`top_p`	`float	None`	-
`top_p_min`	`float	None`	-
`top_p_reset_ids`	`int	None`	-
`top_p_decay`	`float	None`	-
`seed`	`int	None`	-
`temperature`	`float	None`	-
`min_tokens`	`int	None`	-
`beam_search_diversity_rate`	`float	None`	-
`repetition_penalty`	`float	None`	-
`presence_penalty`	`float	None`	-
`frequency_penalty`	`float	None`	-
`prompt_ignore_length`	`int	None`	-
`length_penalty`	`float	None`	-
`early_stopping`	`int	None`	-
`no_repeat_ngram_size`	`int	None`	-
`min_p`	`float	None`	-
`beam_width_array`	`list[int]	None`	-
`logprobs`	`int	None`	-
`prompt_logprobs`	`int	None`	-
`return_context_logits`	`bool`	`false`
`return_generation_logits`	`bool`	`false`
`exclude_input_from_output`	`bool`	`true`
`return_encoder_output`	`bool`	`false`
`return_perf_metrics`	`bool`	`false`
`additional_model_outputs`	`list[str]	None`	-
`lookahead_config`	`LookaheadDecodingConfig	None`	-
`guided_decoding`	`GuidedDecodingParams	None`	-
`ignore_eos`	`bool`	`false`
`detokenize`	`bool`	`true`
`add_special_tokens`	`bool`	`true`
`truncate_prompt_tokens`	`int	None`	-
`skip_special_tokens`	`bool`	`true`
`spaces_between_special_tokens`	`bool`	`true`

Engine Parameters​

Sampling Parameters​

Engine Parameters

Sampling Parameters