babylonpolice.com

arXiv:2402.15163v2 Announce Type: replace
Abstract: This paper presents the first systematic study of Deep Neural Network (DNN) evaluation under stochastic assumptions, focusing on wildfire prediction. We note that current evaluation strategies emphasize a DNN's replication of observed ground truths rather than its ability to learn the underlying stochastic processes, crucial for capturing wildfire evolution's complexity. To bridge this gap, we propose a novel evaluation criterion: Has the DNN learned the stochastic process? Using a synthetic dataset, we introduce a framework to characterize the stochastic process (generated by randomness in fire evolution rules). Through this framework, we assess an evaluation metric's capability to test if the DNN has learned the stochastic process. Our findings show that conventional metrics, including classification-based metrics and proper scoring rules, are inadequate. We identify the Expected Calibration Error (ECE) as a robust metric that tests the proposed evaluation criteria, offering asymptotic guarantees of proper scoring rules and improved interpretability through calibration curves. We extend our analysis to real-world wildfire data, highlighting the limitations of traditional evaluation methods and demonstrating the utility of ECE as a stochasticity-compatible metric alongside existing ones.