Evaluation Design: Issues Confronting Evaluators of Gifted Programs

Abstract

How should one evaluate a program for the gifted? Unique design concerns need to be taken into account when selecting a method for assessing gifted program outcomes. The author reviews design issues and proposes creative solutions to omnipresent dilemmas facing the would-be evaluator.Educational evaluation can take on different forms depending upon the questions asked. For example, sponsors of evaluations of gifted programs might ask one or more of the following questions: &dquo;Who should be served by the program?&dquo;, &dquo;Is the curriculum consistent with the goals and objectives of gifted education?&dquo;, &dquo;Do students become more creative as a result of the program?&dquo;, and &dquo;Is the program worth the cost?&dquo; These questions represent different types of evaluations. Respectively, they are needs assessment, process evaluation, outcome evaluation and evaluation of cost effectiveness.Evaluation design refers to the procedures and conditions employed by the evaluator to collect data. Since data collection is often different across the four major types of evaluations, design issues depend upon the type of evaluation conducted. In this paper, the discussion is limited to outcome evaluations. I have selected this type of evaluation because of the current emphasis on educational outcomes (for example, see &dquo;A Nation at Risk,&dquo; 1983) and because I am an advocate of that approach (Carter and Hamilton, 1985).There are three reasons why evaluation designs must be judged against criteria different from the experimental designs in the laboratory. First, field based evaluations must be designed around administrative constraints such as time, money and logistics. These constraints restrict the evaluator's ability to manipulate and control variables. Consequently, evaluations in the school setting are not as &dquo;tight&dquo; as more traditional laboratory studies. Second, Popham (1975), points out that program evaluations are not concerned with external validity, i.e., the ability to generalize beyond the program setting. Sponsors of program :evaluations desire to know how effective &dquo;their&dquo; programs is in &dquo;their&dquo; setting. Third, it is often impossible to determine whether participant outcomes are due to the program or some other variable(s)-a matter of internal validity (Campbell and Stanley, 1963) -unless controls are imposed which ironically, create a setting that is atypical of the program. In fact, Snow (1974) argues that true program effects can only be determined when a &dquo;representative design&dquo; is employed. Snow recommends designs that are representative of the natural environment because he assumes that experimental manipulation causes the learners to act differently than they would under normal conditions. These reasons illustrate the dilemma of program evaluators. They seek designs that control extraneous variables but do not disturb the natural setting nor violate the school's administrative structure. I have yet to find a design to accomplish this goal. Howev...

Documenting gifted program results for key decision‐makers

“…Student outcome data may also relate to these components, but more importantly to what it is that studentsin exercising their autonomy -have accomplished that could not have been done in a regular classroom setting. While there are several issues related to measuring student outcomes (e.g., their idiosyncratic and long-term nature), these are not insurmountable (Callahan, 1983;Carter, 1986). Overcoming these issues and providing sound data on student outcomes to decision-makers is ultimately what must be done if gifted programs are to survive the recurring threats to their existence.…”

A Model for Evaluating Programs for the Gifted under Non-experimental Conditions

“…Of those that have been published, most use attitudinal data that are drawn from surveys, case histories or testimonials. Very few published evaluations are based upon student outcomes (Carter, 1986a. Most likely, the sparsity of outcome evaluations is due to the difficulty in conducting these evaluations.…”

“…Most likely, the sparsity of outcome evaluations is due to the difficulty in conducting these evaluations. As discussed elsewhere (Carter, 1986a(Carter, , 1986bRenzulli & Smith, 1979), outcome evaluations present unique problems to the evaluator of programs for the gifted. For example, appropriate comparison groups are difficult to find and suitable measuring instruments may not exist to access the curriculum and/or the nature of the sample.…”

“…If the literature of a discipline is intended to offer models for program implementation and to stimulate new ideas and approaches for the field (Carter, 1986a), then the literature for the gifted fails to meet this function with regard to evaluation. Although some examples and models exist (Callahan, 1983(Callahan, , 1986Carter 1986a, 1 9 8 6~;Carter & Hamilton, 1985;Coleman & Fults, 1983;Tremaine, 1979), more pragmatic methodologies are needed to distinguish between a qualitatively different curriculum and the random assortment of fun and games mentioned by Renzulli. …”

REDSIL: A forth generation evaluation model for gifted education programs

“…As the call for pro-gram evaluation becomes more resonant, those involved with gifted education are exploring and writing about the many issues associated with this endeavor. For example, Gowie (1982) presents a framework for what to look for in evaluating a gifted program; Callahan (1986) encourages us to ask the right questions; Renzulli (1984) tries to remind us of the "larger issues", while Carter (1986) discusses general issues confronting evaluators of gifted programs. Despite this recent attention, in most instances, evaluation of gifted education programs is still infrequent and a haphazard affair at best.…”