Adaptive Data Analysis for Growing Data
CoRR(2024)
Abstract
Reuse of data in adaptive workflows poses challenges regarding overfitting
and the statistical validity of results. Previous work has demonstrated that
interacting with data via differentially private algorithms can mitigate
overfitting, achieving worst-case generalization guarantees with asymptotically
optimal data requirements. However, such past work assumes data is static and
cannot accommodate situations where data grows over time. In this paper we
address this gap, presenting the first generalization bounds for adaptive
analysis in the dynamic data setting. We allow the analyst to adaptively
schedule their queries conditioned on the current size of the data, in addition
to previous queries and responses. We also incorporate time-varying empirical
accuracy bounds and mechanisms, allowing for tighter guarantees as data
accumulates. In a batched query setting, the asymptotic data requirements of
our bound grows with the square-root of the number of adaptive queries,
matching prior works' improvement over data splitting for the static setting.
We instantiate our bound for statistical queries with the clipped Gaussian
mechanism, where it empirically outperforms baselines composed from static
bounds.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined