Content-based Medical Image Annotation and Retrieval using Perceptual Hashing Algorithm

Nagarajan, Suresh Kumar

doi:10.9790/3021-0204814818

“…Content-Based Image Retrieval System (CBIR) is a method of retrieving an image based on the input image. In CBIR, the content of an image is analyzed in terms of attributes such as colors, shapes, and textures of the [3]. There are many drawbacks to only relying on image-based similarity analysis, such as computational pressure on the pixel-based comparison and significant feature engineering prior to training.…”

Section: Image-based Similarity Analysismentioning

confidence: 99%

DLI-IT: a deep learning approach to drug label identification through image and text embedding

Liu

¹

,

Meehan

²

,

Tong

³

et al. 2020

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

Background: Drug label, or packaging insert play a significant role in all the operations from production through drug distribution channels to the end consumer. Image of the label also called Display Panel or label could be used to identify illegal, illicit, unapproved and potentially dangerous drugs. Due to the time-consuming process and high labor cost of investigation, an artificial intelligence-based deep learning model is necessary for fast and accurate identification of the drugs. Methods: In addition to image-based identification technology, we take advantages of rich text information on the pharmaceutical package insert of drug label images. In this study, we developed the Drug Label Identification through Image and Text embedding model (DLI-IT) to model text-based patterns of historical data for detection of suspicious drugs. In DLI-IT, we first trained a Connectionist Text Proposal Network (CTPN) to crop the raw image into sub-images based on the text. The texts from the cropped sub-images are recognized independently through the Tesseract OCR Engine and combined as one document for each raw image. Finally, we applied universal sentence embedding to transform these documents into vectors and find the most similar reference images to the test image through the cosine similarity. Results:We trained the DLI-IT model on 1749 opioid and 2365 non-opioid drug label images. The model was then tested on 300 external opioid drug label images, the result demonstrated our model achieves up-to 88% of the precision in drug label identification, which outperforms previous image-based or text-based identification method by up-to 35% improvement. Conclusion: To conclude, by combining Image and Text embedding analysis under deep learning framework, our DLI-IT approach achieved a competitive performance in advancing drug label identification.

show abstract

“…These questions will be answered in the methodology and result sections of our paper. [22]. There are many drawbacks to only relying on image-based similarity analysis, such as computational pressure on the pixel-based comparison and significant feature engineering prior to training.…”

Section: Motivationmentioning

confidence: 99%

DLI-IT: A Deep Learning Approach to Drug Label Identification Through Image and Text Embedding

Liu

¹

,

Meehan

²

,

Tong

³

et al. 2020

Preprint

View full text Add to dashboard Cite

[Background] Drug label, or packaging insert play a significant role in all the operations from production through drug distribution channels to the end consumer. Image of the label also called Display Panel or label could be used to identify illegal, illicit, unapproved and potentially dangerous drugs. Due to the time-consuming process and high labor cost of investigation, an artificial intelligence-based deep learning model is necessary for fast and accurate identification of the drugs. [Methods] In addition to image-based identification technology, we take advantages of rich text information on the pharmaceutical package insert of drug label images. In this study, we developed the Drug Label Identification through Image and Text embedding model (DLI-IT) to model text-based patterns of historical data for detection of suspicious drugs. In DLI-IT, we first trained a Connectionist Text Proposal Network (CTPN) to crop the raw image into sub-images based on the text. The texts from the cropped sub-images are recognized independently through the Tesseract OCR Engine and combined as one document for each raw image. Finally, we applied universal sentence embedding to transform these documents into vectors and find the most similar reference images to the test image through the cosine similarity. [Results] We trained the DLI-IT model on 1749 opioid and 2365 non-opioid drug label images. The model was then tested on 300 external opioid drug label images, the result demonstrated our model achieves up-to 88% of the precision in drug label identification, which outperforms previous image-based or text-based identification method by up-to 35% improvement. [Conclusion] To conclude, by combining Image and Text embedding analysis under deep learning framework, our DLI-IT approach achieved a competitive performance in advancing drug label identification.

show abstract

“…Perceptual Hash (Phash) is mostly used for similarity identification. Phash is a hashing function which can be used in Crypto-hashing, [22]. There are many drawbacks to only relying on image-based similarity analysis, such as computational pressure on the pixel-based comparison and significant feature engineering prior to training.…”

Section: Related Workmentioning

confidence: 99%

DLI-IT: A Deep Learning Approach to Drug Label Identification Through Image and Text Embedding

Liu

¹

,

Meehan

²

,

Tong

³

et al. 2020

Preprint

0

View full text Add to dashboard Cite

[Background] Drug label, or packaging insert play a significant role in all the operations from production through drug distribution channels to the end consumer. Image of the label also called Display Panel or label could be used to identify illegal, illicit, unapproved and potentially dangerous drugs. Due to the time-consuming process and high labor cost of investigation, an artificial intelligence-based deep learning model is necessary for fast and accurate identification of the drugs. [Methods] In addition to image-based identification technology, we take advantages of rich text information on the pharmaceutical package insert of drug label images. In this study, we developed the Drug Label Identification through Image and Text embedding model (DLI-IT) to model text-based patterns of historical data for detection of suspicious drugs. In DLI-IT, we first trained a Connectionist Text Proposal Network (CTPN) to crop the raw image into sub-images based on the text. The texts from the cropped sub-images are recognized independently through the Tesseract OCR Engine and combined as one document for each raw image. Finally, we applied universal sentence embedding to transform these documents into vectors and find the most similar reference images to the test image through the cosine similarity. [Results] We trained the DLI-IT model on 1749 opioid and 2365 non-opioid drug label images. The model was then tested on 300 external opioid drug label images, the result demonstrated our model achieves up-to 88% of the precision in drug label identification, which outperforms previous image-based or text-based identification method by up-to 35% improvement. [Conclusion] To conclude, by combining Image and Text embedding analysis under deep learning framework, our DLI-IT approach achieved a competitive performance in advancing drug label identification.

show abstract

Content-based Medical Image Annotation and Retrieval using Perceptual Hashing Algorithm

Cited by 12 publications

References 6 publications

DLI-IT: a deep learning approach to drug label identification through image and text embedding

DLI-IT: a deep learning approach to drug label identification through image and text embedding

DLI-IT: A Deep Learning Approach to Drug Label Identification Through Image and Text Embedding

DLI-IT: A Deep Learning Approach to Drug Label Identification Through Image and Text Embedding

Contact Info

Product

Resources

About