Characterized by complicated backgrounds, various types, large size variations, and arbitrary orientations, the detection and recognition of arbitrary-oriented objects in remote sensing images is challenging. To address the above problem, an anchor-free arbitrary-oriented object detector using box boundary-aware vectors is proposed. With the idea of CenterNet to detect objects as points, oriented object detection is achieved by predicting the center, the box boundary-aware vectors, the size, and the type of the bounding box. In the feature extraction stage of the designed architecture, Res2Net, a multi-scale convolutional neural network, is used to extract feature maps of different scales and adaptively spatial feature fusion is adopted to improve the detector's adaptability to objects of different sizes. In the detector, a context enhancement module with a multi-branch network is designed to enhance the contextual information of the objects and improve the detector's robustness to the complicated backgrounds. Experiments are carried on three challenging benchmarks (i.e., HRSC2016, UCAS-AOD, and DOTA) and our method achieves state-of-the-art performance with 90.30%, 89.70%, and 77.18% mAP, respectively.