Small molecule modulators of protein-protein interactions (PPIs) have emerged as promising drug candidates with potential applications in anticancer, antiviral, and antimicrobial therapies. While recent virtual screening methods have demonstrated promising results in identifying modulators for specific PPI targets, challenges remain in predicting modulators for novel PPI targets and vice versa. For the first time, we directly predict PPI-modulator interaction through a binary classification task. We construct a benchmark dataset comprising PPIs information and active/inactive modulators data. Moreover, we develop a novel deep learning framework MultiPPIMI that leverages multimodal representations of PPI targets and modulators, and incorporates a bilinear attention network to capture inter-molecule interactions between them. Experiments conducted under both in-domain and cross-domain settings demonstrate the impressive performance and generalization capabilities of MultiPPIMI. The hit rate for identifying modulators of PPIs is greatly improved by combining deep learning with molecular docking. We believe that this work represents a significant step forward in the development of PPI-targeted therapeutics, offering new insights into predicting and understanding the interactions between small molecules and PPIs.