H2O-Lightning-Bolt-74B v1.1

H2O-Lightning-Bolt-74B v1.1 is a specialized decision model developed by H2O.ai, built as a full fine-tune of nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB. Unlike generative conversational models that output conversational text, this model functions as an automated evaluation and classification engine. It operates on a Mixture of Experts (MoE) architecture with 74 billion total parameters and 9 billion active parameters per forward pass, earning a score of 65.51 on the Decision Index benchmark.

The model is engineered to resolve typed decision questions against structured records, text transcripts, and policies. Rather than generating multi-token prose explanations, it outputs exact probabilities across predefined categories: selecting from up to 255 discrete options, evaluating the truth value of a statement, or returning an ordinal score alongside its full probability distribution. Because it outputs calibrated logprobs rather than descriptive text, inference bypasses standard autoregressive decoding bottlenecks.

The model weights are gated on Hugging Face under the H2O Model Non-Commercial Licence v1.1. This restricts deployment strictly to non-commercial research, academic evaluation, and local experimentation. For engineers looking to run H2O-Lightning-Bolt-74B v1.1 locally, it provides high-capacity routing and decision-making without the compute penalty typical of dense 70B-class models.

The architecture of H2O-Lightning-Bolt-74B v1.1 balances deep parameter capacity with low runtime compute via sparse expert routing. Built directly upon NVIDIA's Nemotron-3.5 MoE architecture, the network maintains 74B total parameters distributed across specialized feed-forward layers, but routes each token to just 9B active parameters during execution.

This split delivers notable H2O-Lightning-Bolt-74B v1.1 MoE efficiency. While the model retains the broad representational capacity of a 74B network, the memory bandwidth and compute required during the forward pass match that of a compact 9B dense model.

Key technical characteristics include:

• Parameter Configuration: 74B total parameters, 9B active parameters per forward pass.
• Quantization Scheme: The official release integrates NVIDIA's IQ2_XXS 2-bit quantization on the routed experts, while preserving BF16 precision on the dense self-attention layers and activations.
• On-Disk Footprint: The merged model weights occupy approximately 32.2 GiB.
• Execution Stack: Optimized for vLLM paired with custom b12x kernels designed specifically to accelerate low-bit expert selection and matrix multiplication.
• Context Length: Unspecified in the official card, but structured to ingest substantial multi-turn conversations, audit logs, and complex case files.
• Generation Profile: Exactly 1 output token per decision query, eliminating output token generation costs entirely.

H2O-Lightning-Bolt-74B v1.1 is not a conversational chatbot or a code generator. It is designed specifically for automated document triage, policy enforcement, and record classification.

The model handles three primary query types:

• Multi-Class Choice: Evaluates a record and selects the correct classification among up to 255 distinct candidate options, returning calibrated logprobs for each option.
• Truth Verification (Binary Decisions): Assesses whether a factual assertion about a record is true or false, providing a raw probability score.
• Ordinal Scoring: Assigns an ordinal rating on a predefined scale (such as a 1 to 5 risk tier) along with the probability distribution across all rating steps.

Practical local use cases include:

• Customer Ticket Routing: Classifying unstructured enterprise support logs into hundreds of granular, pre-existing categories without brittle regex or external API calls.
• Automated Policy Compliance: Ingesting financial filings, medical records, or legal contracts to verify adherence to specific regulatory rules with verifiable confidence scores.
• Fraud and Risk Triage: Scoring loan applications or claim summaries on custom ordinal risk scales locally, ensuring that sensitive financial records never leave an air-gapped machine.

Deploying this model locally requires an understanding of its unique runtime dependencies. Because the experts are quantized to 2-bit using custom layouts, standard GGUF loaders like unpatched llama.cpp or default Ollama builds cannot run the native weights without prior conversion.

The raw model weights occupy roughly 32.2 GiB of memory. However, operational VRAM requirements depend heavily on context length and concurrency:

• Minimum VRAM (Batch size 1, short context): 40 GB to 48 GB. A workstation with two 24GB GPUs (such as 2x RTX 3090 or 2x RTX 4090) can split the weights across PCIe using tensor parallelism in vLLM.
• Recommended VRAM (High concurrency / large context): 80 GB to 96 GB. The official reference environment runs on a single NVIDIA RTX PRO 6000 (96GB Blackwell workstation GPU), which provides sufficient headroom for large context windows, deep KV caches, and high concurrent batching.
• Best GPU for H2O-Lightning-Bolt-74B v1.1: An enterprise-grade 80GB+ GPU (NVIDIA A100 80GB, H100 NVL, or RTX PRO 6000 96GB) allows the model to run without tensor parallel communication overhead across PCIe slots.

If you are wondering how to run 74B model on consumer GPU setups, running the native IQ2_XXS checkpoint on a single 24GB RTX 4090 is impossible due to the 32.2 GiB weight size. You must pool at least two 24GB cards or run an offloaded quantization on high-RAM Apple Silicon (such as an M3 Max or M4 Max with 64GB or 128GB of unified memory).

The native release uses the b12x library with specialized vLLM kernels. The upstream release includes an execution script (serve.sh) and a lightweight Python shim (lightning_bolt_shim.py) that exposes two local endpoints:

• POST /v1/systemone: Internal raw decision endpoint.
• POST /v1/decisions: An OpenAI-compatible endpoint accepting decision schemas.

For users seeking the best quantization for H2O-Lightning-Bolt-74B v1.1 in non-vLLM stacks, community GGUF conversions (typically Q4_K_M or Q3_K_L) are standard. A standard Q4_K_M quant would push the total model size closer to 42-45 GiB, which requires at least 48 GB to 64 GB of unified system memory or dual enterprise GPUs. If community GGUF quantizations become accessible in your environment, Ollama offers the quickest route for local testing, provided your machine has the memory to handle a 74B-class model.

Because the model outputs only 1 token per question, traditional H2O-Lightning-Bolt-74B v1.1 tokens per second metrics do not reflect how the system behaves. Instead, performance is measured in requests per second (RPS) and latency:

• Concurrency: Under a 64-request concurrent load on an RTX PRO 6000 96GB card, the model processes roughly 5.62 requests per second.
• Inference Latency: Median single-query latency sits at approximately 314 milliseconds.
• Compute Cost: All generation cost is concentrated strictly in input processing (prefill), making batch classification workloads predictable and compute-efficient.

Evaluating H2O-Lightning-Bolt-74B v1.1 against alternative models requires separating generative text engines from structured decision systems.

Meta's Llama 3.1 70B is a dense conversational model. Running Llama 3.1 70B requires computing all 70 billion parameters for every single token generated. If you prompt Llama 3.1 70B to evaluate a record and output JSON, it generates dozens of tokens explaining its answer, introducing latency, high VRAM consumption, and potential parsing errors.

H2O-Lightning-Bolt-74B v1.1 activates only 9 billion parameters per token. It outputs the classification logprobs directly in a single step, yielding lower latency and eliminating parsing failures. However, Llama 3.1 70B is licensed for commercial use and can write open-ended text, whereas Bolt-74B cannot generate open-ended answers and is strictly non-commercial.

Mixtral 8x22B is an open-weights MoE model containing roughly 141B total parameters and activating approximately 39B per token. Mixtral provides strong reasoning, multilingual capabilities, and standard chat functionality, but requires roughly 80 GiB of VRAM even at 4-bit quantization, with much higher FLOP requirements during generation.

Bolt-74B v1.1 is significantly smaller in memory footprint (32.2 GiB on disk) and vastly cheaper to compute (9B active parameters). As a local AI model 74B parameters 2026 release, Bolt-74B is the superior choice for deterministic triage and automated categorization pipelines on workstation hardware, provided the commercial license restrictions fit your research requirements. Choose Mixtral or dense Llama variants if you require narrative summaries, coding, or commercial deployment.