In the LTE system, the traditional detection of number of antenna ports uses blind detection to decode the 1, 2, and 4 port numbers respectively until the system message in physical broadcast channel successfully passes the cyclic redundancy check. This method generates a large amount of computational redundancy and delay. In response to this problem, this paper proposes an improved Kalman autoregressive antenna port number detection algorithm. This algorithm obtains channel state information by extracting the cell reference signals corresponding to different antenna ports, performs Kalman autoregression on the phase information of channel states, and consequently determine the number of antenna ports. Theoretical analysis and simulation results show that the algorithm has low complexity, small delay and a high accuracy rate even when the residual frequency offset is relatively large.