Can AI Know What It Doesn’t Know?

The study dissects the confidence scaling of large language models-specifically GPT-5, DeepSeek-V3.2-Exp, and Mistral-Medium-2508-across three distinct tasks, revealing disparities in their ability to align reported confidence levels with task accuracy, as evidenced by metrics like [latex]d'\relax[/latex] and [latex]\text{Mrati}\relax[/latex], and further refined by the exclusion of outlier data points-approximately 0.1% for Mistral-Medium-2508 in task B-to ensure a robust assessment of confidence calibration across a trial count of [latex]2 \times \Gamma_{3}0^{\relax}[/latex] for task A and [latex]\Gamma_{3}0^{\relax}[/latex] for tasks B and C.

New research explores whether large language models possess the capacity for metacognition – the ability to assess their own confidence and uncertainty.