Hallucination Risk Calculator

Summary

Hallucination Risk Calculator & Prompt Re-engineering Toolkit (OpenAI-only) Post-hoc calibration without retraining for large language models. This toolkit turns a raw prompt into: a bounded hallucination risk using the Expectation-level Decompression Law (EDFL), and a decision to ANSWER or REFUSE under a target SLA, with transparent math (nats). It supports two deployment modes: Evidence-based: prompts include evidence/context; rolling priors are built by erasing that evidence. prompts include evidence/context; rolling priors are built by erasing that evidence. Closed-book: prompts have no evidence; rolling priors are built by semantic masking of entities/numbers/titles. All scoring relies only on the OpenAI Chat Completions API. No retraining required. Table of Contents Install & Setup pip install --upgrade openai export OPENAI_API_KEY=sk-... The module uses openai>=1.0.0 and the Chat Completions API (e.g., gpt-4o , gpt-4o-mini ). Core Mathematical Framework The EDFL Principle Let the binary event $\mathcal{A}$ be the thing you want to guarantee (e.g., Answer in decision mode, or Correct for factual accuracy). Build an ensemble of content-weakened prompts (the rolling priors) ${S_k}_{k=1}^m$ . For the realized label $y$ , estimate: Information budget: $$\bar{\Delta} = \tfrac{1}{m}\sum_k \mathrm{clip}_+(\log P(y) - \log S_k(y), B)$$ (one-sided clipping; default $B=12$ nats to prevent outliers while maintaining conservative bounds). Prior masses: $q_k = S_k(\mathcal{A})$ , with: $\bar{q}=\tfrac{1}{m}\sum_k q_k$ (average prior for EDFL bound) $q_{\text{lo}}=\min_k q_k$ (worst-case prior for SLA gating) By EDFL, the achievable reliability is bounded by: $$\bar{\Delta} \ge \mathrm{KL}(\mathrm{Ber}(p) | \mathrm{Ber}(\bar{q})) \Rightarrow p\le p_{\max}(\bar{\Delta},\bar{q})$$ Thus the hallucination risk (error) is bounded by $\overline{\mathrm{RoH}} \le 1 - p_{\max}$ . Decision Rule (SLA Gating) For target hallucination rate $h^*$ : Bits-to-Trust: $\mathrm{B2T} = \mathrm...