On The E↵ect of Auxiliary Tasks on Representation Dynamics: Appendices

Proof. Without loss of generality, we may take the vectors U1, . . . , U|X | to be the canonical basis vectors. Under the assumptions of the theorem, we exclude initial conditions for which the matrix A with (i, j)th element aij is not full rank. Note that under this condition, the matrix At with (k, i)th element akie it is also full rank for all but finitely many t. By performing row reduction operations and scaling rows, for all such t we may pass from (Wk(t) | k 2 [K]) to an alternative spanning set (f Wk(t) | k 2 [K]) of the same subspace such that f Wk(t) Uk 2 hUK+1:|X |i, and kf Wk(t) Ukk = O(e t( K+1 k)) = o(1). We therefore obtain an orthonormal basis for this subspace of the form U1 + o(1), . . . , UK + o(1).