Automatic hidden-web table interpretation, conceptualization, and semantic annotation

Abstract

The longstanding problem of automatic table interpretation still illudes us. Its solution would not only be an aid to table processing applications such as large volume table conversion, but would also be an aid in solving related problems such as information extraction, semantic annotation, and semi-structured data management. In this paper, we offer a solution for the common special case in which so-called sibling pages are available. The sibling pages we consider are pages on the hidden web, commonly generated from underlying databases. Our system compares them to identify and connect nonvarying components (category labels) and varying components (data values). We tested our solution using more than 2,000 tables in source pages from three different domains -car advertisements, molecular biology, and geopolitical information. Experimental results show that the system can successfully identify sibling tables, generate structure patterns, interpret tables using the generated patterns, and automatically adjust the structure patterns as it processes a sequence of hidden-web pages. For these activities, the system was able to achieve an overall F-measure of 94.5%. Further, given that we can automatically interpret tables, we next show that this leads immediately to a conceptualization of the data in these interpreted tables and thus also to a way to semantically annotate these interpreted tables with respect to the ontological conceptualization. Labels in nested table structures yield ontological concepts and interrelationships among these concepts, and associated data values become annotated information. We further show that semantically annotated data leads immediately to queriable data. Thus, the entire process, which is fully automatic, transform facts embedded within tables into facts accessible by standard query engines.

Theoretical Foundations for Enabling a Web of Knowledge

“…We have also done some work on automated extractionontology construction [32,24,33,34] and some work on free-form query processing [36,2]. We nevertheless still have much work to do, even on fundamental WoK components such as creating a sharable data-frame library, constructing data frames for relationship sets, finding ways to more easily produce instance recognizers, reverse-engineering of many genres of semi-structured sources to extraction ontologies, enhancing query processing, incorporating reasoning, and addressing performance scalability.…”

“…We have shown elsewhere that we can automatically construct an ontology for the site (and any other site with sibling tables) and extract the information in all the tables to populate the ontology [32]. Third we can create extraction ontologies automatically (although they likely need some enhancement) [32]. Fourth we can turn the process around and let users specify ontologies via nested forms [33].…”

“…Second, nested tables from the WormBase site are all sibling tables-they have identical, or nearly identical structure. We have shown elsewhere that we can automatically construct an ontology for the site (and any other site with sibling tables) and extract the information in all the tables to populate the ontology [32]. Third we can create extraction ontologies automatically (although they likely need some enhancement) [32].…”

Converting heterogeneous statistical tables on the web to searchable databases

“…Input tables were matched with known conceptualizations in an attempt to interpret them in [56]. Information extraction from sibling tables with identical headers was demonstrated in [57]. A taxonomy of tables based on the geometric relationship of tabular structures to isothetic tessellations and to X-Y trees was proposed in [58], a machine learning approach to segmentation of grid tables in [59], and algorithms for turning web tables into relational tables by recovering and factoring header paths in [60].…”

FOCIH: Form-Based Ontology Creation and Information Harvesting

“…A user can then modify the form, 9 if desired, and use it to harvest information. We have implemented this reverse-engineering of tables into FOCIH forms based on a system called TISP (Table Interpretation for Sibling Pages) [29,30]. TISP converts tables from sites like hidden-web sites that have machinegenerated sibling pages into FOCIH forms and thus into FOCIH-generated ontologies.…”