Progress in computing architectures is approaching a paradigm shift: traditional computing based on digital complementary metal-oxide semiconductor technology is nearing physical limits in terms of miniaturization, speed, and, especially, power consumption. Consequently, alternative approaches are under investigation. One of the most promising is based on a “brain-like” or neuromorphic computation scheme. Another approach is quantum computing using photons. Both of these approaches can be realized using silicon photonics, and at the heart of both technologies is an efficient, ultra-low power broad band optical modulator. As silicon modulators suffer from relatively high power consumption, materials other than silicon itself have to be considered for the modulator. In this Perspective, we present our view on such materials. We focus on oxides showing a strong linear electro-optic effect that can also be integrated with Si, thus capitalizing on new materials to enable the devices and circuit architectures that exploit shifting computational machine learning paradigms, while leveraging current manufacturing infrastructure. This is expected to result in a new generation of computers that consume less power and possess a larger bandwidth.