Universal Lossless Coding For Sources With Repeating Statistics

A lower bound is derived on the achievable redundancy for universal lossless coding of parametric sources with piecewise stationary, abruptly changing, occasionally repeating statistics. In particular, it is shown that if the number of repeating statistical parameter vectors (or states) is not too large, for any uniquely decipherable code, for almost every set of states that govern all the different segments in the data sequence, for almost every arrangement of these states in the different segments, and for almost every vector of transition times, the minimum achievable redundancy is composed of 0.5 log d extra code bits for each unknown component of each state, log m extra code bits for each unknown transition time, and log s extra code bits for each repetition of a state, where d is the average duration of each state in the input string, m is the average length of a segment, and s is the total number of states. If s is essentially large compared to m, it is shown that the minimum redundancy is composed of 0.5 log m bits for each unknown component in each segment and log m bits for each unknown transition time, which is the same lower bound as that of general piecewise stationary sources (PSSs). These results are true also in the minimax and maximin senses. The lower bound is shown to be achievable through construction of mixture and estimation based codes. Different special cases are reviewed, and it is shown that unless s is essentially large compared to m, optimal codes that are designed particularly for sources with repeating statistics outperform codes designed for PSSs when coding sources with repeating statistics. In particular, the bound for general PSSs is shown to be a special case of the new bound.