Combined semantic and similarity search in medical image databases

Abstract

The current diagnostic process at hospitals is mainly based on reviewing and comparing images coming from multiple time points and modalities in order to monitor disease progression over a period of time. However, for ambiguous cases the radiologist deeply relies on reference literature or second opinion. Although there is a vast amount of acquired images stored in PACS systems which could be reused for decision support, these data sets suffer from weak search capabilities. Thus, we present a search methodology which enables the physician to fulfill intelligent search scenarios on medical image databases combining ontology-based semantic and appearance-based similarity search. It enabled the elimination of 12 % of the top ten hits which would arise without taking the semantic context into account.

Efficient similarity search on 3D bounding box annotations

“…A recent summary over retrieval-based classification systems is listed in Ref. 2. However, the classical idea of image retrieval has as well advanced into the field of liver lesions: Napels et al 4 and Seifert et al 7 describe similarity retrieval on liver lesions. In Ref.…”

“…7, a first solution to this problem is introduced. The most simple image characteristic of a 3D annotation is its actual size in millimeters.…”

“…Current CBIR approaches are therefore either restricted to small datasets of exact 2D annotations 4,5 or they approximate a larger set of 3D annotations as minimum bounding boxes or rectangles (MBRs), 6,7 instead of using more precise exact mesh or mask annotations. An advantage of MBRs is that they are very convenient for visual examination because any 3D region of interest (ROI) is usually loaded and displayed in a bounding box.…”

“…In Ref. 7, this problem is dealt with by representing a box annotation as an unordered sequence of 2D image descriptors and by using a flexible but expensive distance measure. Though this approach yielded effective similarity search, the descriptors and the similarity measures are rather complex.…”

A Discriminative Distance Learning–Based CBIR Framework for Characterization of Indeterminate Liver Lesions

“…Seifert et al [7] propose a learning-based "search by scribble" system and apply it to the retrieval of similar liver lesions in 3D CT scans. In this case, the search context is fixed and based on pairwise similarity scores annotated for 160 hepatic lesions, with the similarity criterion left to the discretion of the annotator.…”