TODO: literature review for evaluation measures of confidence scores.

(search acl anthology for “confidence scores”)

• 2004 - Confidence Estimation for Machine Translation
• 2008 - Computing Confidence Scores for All Sub Parse Trees
• Nguyen Bach 2011 - Goodness: A Method for Measuring Machine Translation Confidence Has a good explanation of MT confidence
• 2018 - Confidence Modeling for Neural Semantic Parsing Measures “the relationship between confidence scores and F1 using Spearman’s ρ correlation coefficient which varies between −1 and 1

(0 implies there is no correlation).”

• 2019 - Modeling Confidence in Sequence-to-Sequence Models
• 2020 - Calibrating Structured Output Predictors for Natural Language Processing