Handbook for Evaluating Knowledge-Based Systems

Adelman, Leonard; Riedel, Sharon L.

doi:10.1007/978-1-4615-6171-2

Cited by 29 publications

(18 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In other words, we do not first acquire knowledge and then build a system; knowledge acquisition is an ongoing process which is closely coupled with the general software development effort. Of course, this is hardly a novel observation, and there are many development methodologies for knowledgebased systems that stress iterative development and continual ka (Adelman & Riedel, 1997).…”

Section: Development Methodology: Using Multiple Ka Techniquesmentioning

confidence: 99%

“…Research has also been done on verifying and validating knowledge to check that it is correct (Adelman & Riedel, 1997). Verification techniques focus on detecting logical anomalies and inconsistencies that often reflect mistakes in the elicitation or coding process; we will not further discuss these, as such errors are not our primary concern in this paper.…”

Section: Knowledge Acquisition and Validationmentioning

confidence: 99%

See 1 more Smart Citation

Acquiring Correct Knowledge for Natural Language Generation

Reiter¹,

Sripada²,

Robertson³

2003

jair

View full text Add to dashboard Cite

Natural language generation (NLG) systems are computer software systems that produce texts in English and other human languages, often from non-linguistic input data. NLG systems, like most AI systems, need substantial amounts of knowledge. However, our experience in two NLG projects suggests that it is difficult to acquire correct knowledge for NLG systems; indeed, every knowledge acquisition (KA) technique we tried had significant problems. In general terms, these problems were due to the complexity, novelty, and poorly understood nature of the tasks our systems attempted, and were worsened by the fact that people write so differently. This meant in particular that corpus-based KA approaches suffered because it was impossible to assemble a sizable corpus of high-quality consistent manually written texts in our domains; and structured expert-oriented KA techniques suffered because experts disagreed and because we could not get enough information about special and unusual cases to build robust systems. We believe that such problems are likely to affect many other NLG systems as well. In the long term, we hope that new KA techniques may emerge to help NLG system builders. In the shorter term, we believe that understanding how individual KA techniques can fail, and using a mixture of different KA techniques with different strengths and weaknesses, can help developers acquire NLG knowledge that is mostly correct

show abstract

Section: Development Methodology: Using Multiple Ka Techniquesmentioning

confidence: 99%

Section: Knowledge Acquisition and Validationmentioning

confidence: 99%

Acquiring Correct Knowledge for Natural Language Generation

Reiter¹,

Sripada²,

Robertson³

2003

jair

View full text Add to dashboard Cite

show abstract

“…These criteria build upon an earlier set found in Adelman and Riedel (8). For in-depth specific HF design standards one can also refer to MIL Standard 1472-G (9).…”

Section: Step 2: Measurement Frameworkmentioning

confidence: 99%

“…This branch of the measurement framework builds directly on the HD research performed in step no. 1 of our systems engineering approach and utilizes some of the performance criteria found in Adelman and Riedel (8). In particular, the Performance branch measures the system's effect on the ability of the operator, staff and leader, and larger organization to perform their work in a timely way, and on the quality of their processes and resulting products.…”

Section: Step 2: Measurement Frameworkmentioning

confidence: 99%

Systems Engineering Approach and Metrics for Evaluating Network-Centric Operations for U.S. Army Battle Command

Grynovicki¹,

Branscome²

2013

View full text Add to dashboard Cite

Approved for public release; distribution is unlimited.ii REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. REPORT DATE (DD-MM-YYYY)July 2013 ARL-TN-551 SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) SPONSOR/MONITOR'S REPORT NUMBER(S) DISTRIBUTION/AVAILABILITY STATEMENTApproved for public release; distribution is unlimited. SUPPLEMENTARY NOTES ABSTRACTWhile many studies have taken a traditional task-analysis approach to assessing the effects of technology on battle command performance, this report describes the implementation of a global systems approach for examining usability, functionality, and performance from operator, commander, staff, and organizational perspectives. The approach had three steps: the first step was identifying tasks and behavioral characteristics that were associated with effective battle command performance at the operator, commander and staff, and organizational levels; the second step was developing a framework for measuring how well technology provides the usability, functionality, and performance required at each of the three levels; the third step was developing metrics and tools to implement the measurement framework. This approach has been used successfully to assess technology ("digitization") effectiveness by the U.S. Army Training and Doctrine Command and the Operational Test Command during network integration evaluations, limited user tests, and initial operational test and evaluation. This report describes the three-step approach and presents data demonstrating the reliability of the questionnaire as it was used to evaluate the Advanced Field Artillery Tactical Data System version 6.5.0. iii

show abstract

“…The BAMS is an extensible text-based framework that can be used to navigate the brain and access various forms of textual information (Adelman & Riedel, 1997;Bota & Arbib, 2000Bota et al 2003). It can be used as a source of information pertaining to brain structures collated from the literature, and as a system for organizing the neuroscientific data characteristic of a given structure.…”

Section: Bamsmentioning

confidence: 99%

A multimodal, multidimensional atlas of the C57BL/6J mouse brain

et al. 2004

View full text Add to dashboard Cite

Strains of mice, through breeding or the disruption of normal genetic pathways, are widely used to model human diseases. Atlases are an invaluable aid in understanding the impact of such manipulations by providing a standard for comparison. We have developed a digital atlas of the adult C57BL/6J mouse brain as a comprehensive framework for storing and accessing the myriad types of information about the mouse brain. Our implementation was constructed using several different imaging techniques: magnetic resonance microscopy, blockface imaging, classical histology and immunohistochemistry. Along with raw and annotated images, it contains database management systems and a set of tools for comparing information from different techniques. The framework allows facile correlation of results from different animals, investigators or laboratories by establishing a canonical representation of the mouse brain and providing the tools for the insertion of independent data into the same space as the atlas. This tool will aid in managing the increasingly complex and voluminous amounts of information about the mammalian brain. It provides a framework that encompasses genetic information in the context of anatomical imaging and holds tremendous promise for producing new insights into the relationship between genotype and phenotype. We describe a suite of tools that enables the independent entry of other types of data, facile retrieval of information and straightforward display of images. Thus, the atlas becomes a framework for managing complex genetic and epigenetic information about the mouse brain. The atlas and associated tools may be accessed at

show abstract

Handbook for Evaluating Knowledge-Based Systems

Cited by 29 publications

References 0 publications

Acquiring Correct Knowledge for Natural Language Generation

Acquiring Correct Knowledge for Natural Language Generation

Systems Engineering Approach and Metrics for Evaluating Network-Centric Operations for U.S. Army Battle Command

A multimodal, multidimensional atlas of the C57BL/6J mouse brain

Contact Info

Product

Resources

About