The prediction of pedestrian motion is challenging, especially in crowded roads and intersections. Most of the current approaches apply offline methods to learn motion behaviors, but as a result, they are not able to learn continuously and typically do not generalize well to new environments. This paper presents Similarity-based Incremental Learning Algorithm (SILA) for pedestrian motion prediction with the ability of improving the learned model over the time as data is obtained incrementally. To keep the model size efficient, the motion primitives learned from the new data are compared with the previously known ones, and similar motion primitives are fused while novel motion primitives are added to the model. Results show that the SILA model growth rate is about 1/3 that of an incremental approach that does not fuse motion primitives. SILA is evaluated on different datasets and scenarios including intersections and busy streets. The results show that, even though SILA learns incrementally, it performs comparably to (and sometimes outperforms) state-of-the-art algorithms in pedestrian prediction. Additionally, SILA learning time only depends on the size of the data added incrementally, which makes SILA more efficient in terms of time and space compared to batch learning.