Voluntary movement is driven by the primary motor cortex (M1), and individuals can learn to modulate even single neurons at will. Yet M1 also receives pronounced sensory inputs and contributes to sensory-driven motor responses. To what extent do these non-volitional signals restrict voluntary modulation of M1? Using a task in which the firing rate of a single neuron directly determines the position of a computer cursor along a visual axis, we assessed the ability of monkeys to modulate individual neurons under different sensory contexts. We found that sensory context persistently affected volitional control of single neurons in M1. For instance, visually rotating the biofeedback axis could render the same neural task effortless or problematic. Notably, extended training within or across days did not resolve this disparity. Our findings suggest that sensory context can limit the degree to which M1 activity is under volitional control.