It is well known that the optimal transform for scalar quantization is defined as the one that generates completely decorrelated transform coefficients. Under this definition, the Karhunen-Lo&ve transform is optimal and the discrete cosine transform is close to optimal. However, if vector quantization (VQ) is used in the transform domain, such a definition is no longer appropriate because compression performance of VQ is actually better when the components within the vector are more correlated. Therefore, the optimal transform for the purpose of VQ should reduce the correlation between the transform domain vectors and preserve the correlation among the components within each vector. In this paper, these two attributes are discussed, an intrinsic coupling factor is defined to quantitatively measure how closely the components are coupled within a vector, and a bitallocation mechanism is derived to assign bits to the transform domain vectors. Following the guidance of such a general study, a new scheme is proposed for image and video compression.