For both economic and research purposes, accurate information on forest composition, and the amount of dead wood is of paramount importance. It is not only important to know the biomass and the distribution of tree species but also to detect vermin and diseases to assess the health of a forest. Performing such inventory accurately with conventional methods of surveying (e.g., terrestrial laser scanners) is very labor and time-consuming as forests can be very dense, thus requiring many setups with a terrestrial laser scanner. We present an innovative approach for forest inventory combining UAV-borne light detection and ranging (LiDAR), multispectral aerial imagery, and terrestrial point cloud data measured with a handheld laser scanner. For such multi-sensor measurement campaigns, however, reliable extraction of forest parameters such as canopy height, the diameter at breast height (DBH), deadwood volumes, etc., strongly relies on the quality of sensor data fusion. Especially in dense forests, the GNSS signal, which is necessary for georeferencing the point clouds, can be very weak. However, terrestrial laser scans can capture much more information underneath the forest canopy which is partly obscured to the airborne data. To circumvent this mismatch, we propose an easily adaptable two-step workflow for fusing the directly georeferenced airborne LiDAR point clouds with the corresponding multi-spectral photogrammetric data and their unreferenced terrestrial counterparts. In the first step of the processing chain, individual terrestrial scans of the forest are coregistered using spherical laser targets located above exactly measured reference points. Secondly, these coarsely coregistered scans are then combined with the georeferenced airborne point clouds using control points. This is especially challenging as forests are very unstructured environments, and in addition, typical SLAM-features such as leaves, and branches tend to move. The result of this procedure are radiometrically enhanced 3D point clouds containing detailed information about the forest structure above and underneath its canopy. This provides a versatile basis for further processing such as tree species recognition, e.g., using convolutional neural networks. We demonstrate the applicability of the workflow on recent data sets measured in Landshut (Bavaria, Germany) under leaf-on and leaf-off condition.