A study on the effect of the physical properties and moisture sorption isotherm of palm kernels constitutes the critical criteria in evaluating the drying performance. The drying was evaluated as a function of moisture content (MC) in the range of 0.31–0.02 kg/kg (d.b.). Whereas, the equilibrium moisture content (EMC) of palm kernels (whole kernel and ground kernel) was determined experimentally using the standard gravimetric method at different temperatures (50 °C to 80 °C), over a range of relative humidity (RH) from 10% to 81%. Palm kernel length, width, and thickness decrease from 16.08 ± 2.09 mm to 14.17 ± 2.30 mm, 12.06 ± 1.40 mm to 11.24 ± 1.08 mm, and 10.01 ± 1.27 mm to 9.18 ± 1.04 mm, respectively, when MC decreased. Bulk density, surface area, and specific surface area decreased as the MC decreased, while porosity and true density were increased. EMC of palm kernels (whole kernel and ground kernel) decreased with an increase in temperature at constant RH. Modified Oswin and modified Halsey models were found to be the best for predicting desorption moisture isotherms for whole and ground palm kernel, respectively. Therefore, the study of the effect of drying on physical aspects as well as moisture sorption isotherms is important to further analyze the drying performance of Tenera palm kernel (e.g., equipment design and energy requirement).