Semantic Uncertainty

Kuhn and colleagues estimate uncertainty across meanings of generated answers. Grouping equivalent wordings aims to avoid counting paraphrases as entirely different answers. Their implementation approximates equivalence through bidirectional entailment and depends on sampling.^[1]

Imagine several generated answers: “Delivery is Tuesday” and “Tuesday is delivery day.” In this original teaching example, wording varies while meaning stays fixed. “Delivery is Wednesday” introduces a different answer.

Now suppose the actual tracking record says Wednesday but every generated sample says Tuesday. Agreement among samples does not authenticate the answer. This hypothetical case shows why variation and grounding are different questions; it is not a measured failure rate of the method.

Compare Hallucination and the fabricated-citation case. Yarin Gal offers a researcher reading entry, while Scalable Oversight asks how difficult work can be evaluated more broadly.