Cellular vehicle-to-everything (C-V2X) communications amass research interest in recent days because of its ability to schedule multiple access more efficiently as compared to its predecessor technology, i.e., dedicated short-range communications (DSRC). However, the foremost issue still remains: a vehicle needs to keep the V2X performance in a highly dynamic environment. This paper proposes a way to exploit the dynamicity. That is, we propose a resource allocation mechanism adaptive to the environment, which can be an efficient solution for air interface congestion that a V2X network often suffers from. Specifically, the proposed mechanism aims at granting a higher chance of transmission to a vehicle with a higher crash risk. As such, the channel access is prioritized to those with urgent needs. The adaptation is implemented based on reinforcement learning (RL). We model the RL framework as a contextual multi-armed bandit (MAB), which provides efficiency as well as accuracy. We highlight the most superb aspect of the proposed mechanism: it is designed to be operated at a vehicle autonomously without need for any assistance from a central entity. Henceforth, the proposed framework is expected to make a particular fit to distributed V2X network such as C-V2X mode 4.