Systematic Inference Audit and Runtime Telemetry of Aleph Alpha Kolibri-1: Serving Forensics, Token Economics, and Parser Remediation on NVIDIA H200 SXM

Abstract

Systematic Inference Audit and Runtime Telemetry of Aleph Alpha Kolibri-1: Serving Forensics, Token Economics, and Parser Remediation on NVIDIA H200 SXM Available in English and German. Both technical reports (PDF), Markdown sources, raw JSONL telemetry, and replication scripts are included in this record. Abstract (English) We present an independent, systematic inference audit of Aleph Alpha's Kolibri-1 model deployed under vLLM on a single NVIDIA H200 SXM (141 GB HBM3e). Comprising 5,372 live requests across four benchmark stages (Probe, Ablation, Stress/Concurrency, and Needle-in-a-Haystack), this audit resolves a critical failure class in LLM production pipelines: the silent failure of naive JSON parsers on reasoning models. Key Empirical Findings: Parser Remediation & Semantic Compliance: An initial 0.0% usable response rate in standard toolchains was isolated as a harness parsing artifact caused by unhandled delimiters. Upon delimiter-aware extraction and Pydantic validation, Kolibri-1 achieved a 99.98% semantic accuracy (5,371 / 5,372 requests correct; 95% Wilson CI lower bound: 99.89%). Token Economics & Fast-Lane Architecture: For routine information extraction, disabling reasoning (enable_thinking: false) reduces generated tokens by ~90% (median 492 to 50 tokens), cutting compute resource allocation by 71% and increasing request throughput by 3.4x on 4k-token unique prompts (up to 8.8x with cached prefixes). Hardware Saturation & The Latency Wall: Prefill compute saturates at ~35,100 prompt tokens/s. Under high concurrency (C=64, 31k context), P95 TTFT is 38.88 s across the full cell due to the synchronized start-burst prefill queue (steady-state P95: 5.0 s). Peak KV-cache utilization reached 42.66% with zero preemptions, demonstrating that H200 capacity is prefill-compute bound rather than memory bound. Enterprise On-Premises Dimensioning: Sizing extrapolations indicate that typical enterprise and legal workloads (e.g., 500 documents/day) utilize < 1% of an H200's prefill capacity. Dual 48 GB PCIe setups (e.g., 2x NVIDIA L40S or RTX 6000 Ada) paired with an intelligent API gateway (Fast Lane vs. Deep Lane) provide a cost-effective on-premises alternative. Kurzfassung (Deutsch) Dieser technische Bericht dokumentiert ein unabhängiges Inferenz-Audit von Aleph Alphas Kolibri-1 auf NVIDIA H200 SXM unter vLLM über 5.372 Live-Requests. Das Audit beweist, dass frühere 0,0-%-Ausfälle in Standard-Toolchains auf unmaskierte Reasoning-Delimiter ( ) im Test-Harness zurückzuführen sind. Nach Implementierung eines strikten Delimiter-Parsings liefert Kolibri-1 eine inhaltliche Genauigkeit von 99,98 %. Durch die Entkopplung von Routine-Extraktion („Fast Lane", Reasoning aus, 50 Tokens) gegenüber komplexer Urteilsfindung („Deep Lane", Reasoning an, 492 Tokens) lässt sich der Durchsatz um das 3,4- bis 8,8-Fache steigern und der Rechenaufwand pro Request um 71 % senken. Archive Contents / Enthaltene Dateien kolibri_audit_report_en.pdf: Complete Technical Report in English. kolibri_audit_report_de.pdf: Vollständiger technischer Bericht auf Deutsch. kolibri_audit_complete_package.zip: Complete replication bundle including raw request telemetry (data/runs/*/requests.jsonl), summary tables, and audit harness code.