Detection of small sea vessels in synthetic aperture radar (SAR) images has received much attention in recent years because the small vessels have weak scattering intensity and few image pixels. The existing detection network structures are not well adapted to small-scale targets, the polarimetric data are not properly utilized and the sea-land segmentation process to remove land false alarms is time-consuming. Regarding these problems, firstly, a single low-level path aggregation network (S-LPAN) is designed specifically for small target. The structure reduces false alarms at the feature level by finding suitable single-scale feature maps for detection and adding a semantic enhancement module. Secondly, adaptive dual-polarimetric feature fusion (ADPFF) is proposed to filter the multi-channel features acquired by dual-polarimetric decomposition to reduce feature redundancy. Thirdly, a segmentation layer is added to the network to shield the land from false alarms. The detection and segmentation layers share the feature extraction and feature fusion modules and are jointly trained by a joint loss. Finally, polarimetric SAR detection and segmentation dataset (P-DSD) containing small vessel detection and sea-land segmentation labels is created with reference to the LS-SSDD v1.0 dataset, and experimental results on this dataset verify the improvement of this proposed method over other typical methods.