Optimizing Local Repository Search with Bidirectional Encoder Representations from Transformer Models

Abstract

—In this research, we investigate the relationship between sentence transformer models and the accuracy and recall of asymmetric semantic search engines. Our results show that the chosen sentence transformer model has a significant impact on the performance of the search engine. Using a pre-trained model specific to the domain resulted in a high cosine similarity score of 0.64. Fine-tuning a pre-trained sentence transformer for asymmetric semantic search improved the score to 0.4, but a larger corpus was needed for optimal results. Our findings highlight the importance of careful model selection and training data in the development of asymmetric semantic search engines.