Recent years have witnessed increased attention towards vehicular communications as a part of an overall modernization trend towards the emergence of a reliable, less human-dependent, and more efficient Intelligent Transportation System (ITS) conjugated with the rapid growth of smart cities. ITS imposes better safety and security through the employment of Autonomous Vehicles (AV) to reduce the possibility of accidents caused due to human intervention. The application of autonomous vehicles to the traditional Vehicular Ad-hoc Networks (VANET) has paved the way for the development of a newer networking paradigm called the Internet of Autonomous Vehicles (IoAV). IoAV enjoys several advantages over VANET in terms of robustness, security, and scalability. However, due to the gradual transition from existing vehicles to autonomous ones, both types may be going to coexist together in the same environment. Therefore, a reliable, fast responsive, and flexible infrastructure is necessary to serve both kinds in such a hybrid setting until the transition to all AV is completed. In this context, this paper represents a concise review of the architecture of IoAV infrastructure, its communication modules, message dissemination, protocols and services that comprise the main body of the IoAV framework, in addition to further remarks and research challenges.