Colloquium / Seminars

Topic：Calibration and continuity
Speaker：Dr. Colin McSwiggen
　　　　(Academia Sinica)
Date time：Dec. 10, 2024 13:30-14:20
Venue：SA213
Abstract：
Abstract.
A statistical model is said to be calibrated if it has the appropriate level of confidence in its own predictions: that is, the confidence that it assigns to a predicted outcome should accurately reflect that outcome's likelihood. For example, if a weather model is calibrated, then out of all of the days when the model predicts a 30% chance of rain, we should expect that it actually will rain on 30% of them. Calibration is crucial for managing the risks associated with incorrect predictions, but modern deep learning models are systematically miscalibrated: they are overconfident when they are incorrect. To make matters worse, theorists can't agree about how miscalibration should be quantified! The prevailing miscalibration metric in engineering applications is the expected calibration error (ECE), which has been widely criticized because it is discontinuous: a tiny change in the model can lead to a large change in the error. In this talk, I'll try to convince you that this problem isn't really a problem, that ECE was fine all along, and that engineers should feel free to keep using it the way they always have (at least for binary classification tasks). The argument will require us to answer a strange but fundamental question about the topological properties of the conditional expectation operator.

$返回$ go back