The paper argues that calibration— the agreement between a language model’s expressed confidence and its empirical correctness—should be treated as a core evaluation criterion alongside traditional performance metrics. It notes that standard calibration measures need only a confidence score and a correctness label per example, data that most existing NLP benchmarks already provide, allowing immediate reporting of calibration scores without extra annotation. The authors identify two concrete harms of miscalibration: at deployment, overconfident predictions can cause real‑world harm, and within the research pipeline, methods that rely on model confidence—such as LLM‑as‑a‑judge, synthetic data generation, and active learning—may produce biased results when confidence is unchecked. They observe that despite the availability of calibration metrics, the NLP community routinely introduces new models and benchmarks without reporting calibration, creating an adoption gap. To close this gap, the authors recommend that each subfield pair its primary accuracy or fluency metric with a calibration score and regard calibration as an essential model property rather than a niche topic. For open‑ended generation, they acknowledge that defining suitable confidence and correctness inputs remains an open challenge.