Just rotate it! Uncertainty estimation in closed-source models via multiple queries
CoRR(2024)
Abstract
We propose a simple and effective method to estimate the uncertainty of
closed-source deep neural network image classification models. Given a base
image, our method creates multiple transformed versions and uses them to query
the top-1 prediction of the closed-source model. We demonstrate significant
improvements in the calibration of uncertainty estimates compared to the naive
baseline of assigning 100% confidence to all predictions. While we initially
explore Gaussian perturbations, our empirical findings indicate that natural
transformations, such as rotations and elastic deformations, yield even
better-calibrated predictions. Furthermore, through empirical results and a
straightforward theoretical analysis, we elucidate the reasons behind the
superior performance of natural transformations over Gaussian noise. Leveraging
these insights, we propose a transfer learning approach that further improves
our calibration results.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined