Reprinted from The Program Evaluation Standards, 2nd Edition (Sage, 1994) with permission from the publisher and The Joint Committee on Standards for Educational Evaluation , Arlen R. Gullickson, Chair.

"Terms are defined in this glossary as they are used in this volume, in the context of evaluation. In other contexts, a number of the terms may have broader or different definitions." (p. 203)

Accuracy The extent to which an evaluation is truthful or valid in what it says about a program, project , or material.

Adversarial/advocacy group A group of people who enter into cross-examination of counter plans, strategies, or outcomes .

Advocacy teams Groups of people who are brought together to develop competing strategies for achieving a given set of objectives.

Affective dimension The psychological concept that refers to a person's feelings, emotions, or degree of acceptance or rejection of some object.

Anonymity (provision for) Evaluator action to ensure that the identity of subjects cannot be ascertained during the course of a study, in study reports, or in any other way.

Archival search An examination of existing records, reports, and documents pertaining to the object of the evaluation.

Assessment The act of determining the standing of an object on some variable of interest, for example, testing students, and reporting scores.

Attrition Loss of subjects from the defined sample during the course of a study.

Audiences Those persons who will be guided by the evaluation in making decisions and all others who have a stake in the evaluation (see stakeholders).

Audit (of an evaluation) An independent examination and verification of the quality of an evaluation plan, the adequacy with which it was implemented, the accuracy of results, and the validity of conclusions .

Bias A consistent alignment with one point of view.

Case study An intensive, detailed description and analysis of a single project , program, or instructional material in the context of its environment.

Client The individual, group, or organization that hires the evaluator.

Code (information) To translate a given set of data or items into a set of quantitative or qualitative symbols.

Coefficient A value expressing the degree to which some characteristic or relation is to be found in specified instances; e.g., the coefficient of correlation is a value expressing the degree to which two variables vary concomitantly.

Cognitive ability The psychological concept that refers to such processes as perceiving, knowing, recognizing, conceptualizing, judging, and reasoning.

Comparative experimental studies Studies that assign a program, project, or instructional material to one group of persons and compare their subsequent performance on some structured task to that of another group that was not exposed to the program, project, or instructional material.

Comparison group (in experimentation) A group that provides a basis for contrast with an experimental group (i.e., the group of people participating in the program or project being evaluated). The comparison group is not subjected to the treatment (independent variable ), thus creating a means for comparison with the experimental group that does receive the treatment.

Conclusions (of an evaluation) Final judgments and recommendations .

Conditioning Associating a response with a previously unrelated stimulus.

Content analysis The process of identifying and listing--in accordance with a parsimonious classification system--categories of expression contained in a variety of information sources.

Context (of an evaluation) The combination of the factors accompanying the study that may have influenced its results. These factors include the geographic location of the study, its timing, the political and social climate in the region at that time, the other relevant professional activities that were in progress, and any existing pertinent economic conditions.

Contract A written or oral agreement between the evaluator and the client that is enforceable by law. It is a mutual understanding of expectations and responsibilities for both parties.

Control group A group as closely as possible equivalent to an experimental group (one that is exposed to a program, project, or instructional material), and exposed to all the conditions of the investigation except the program, project, or instructional material being studied.

Convergence group A group that is responsible for incorporating the important features of alternative strategies proposed by advocacy teams into a compromise strategy.

Correlation A statistical measure of the degree of relationship between or among variables. It is expressed in the form of an index that may vary from -1.00 to .00.

Cost effectiveness The extent to which one program, project, or instructional material produces equal or better results than competitors that cost about the same amount of time, effort, and resources; or the extent to which an object produces the same results as competitors but is less costly.

Covariate A variate occurring concomitantly with the variate of primary interest and measured for the purpose of making informed adjustments on the variate of primary interest (e.g., measuring pretest performance of two groups in order to adjust their posttest scores so that they take account of differences between groups that existed prior to the treatment of one of the groups).

Criterion A standard by which something can be judged.

Criterion-referenced tests Tests whose scores are interpreted by referral to well defined domains of content or behaviors, rather than by referral to the performance of some comparable group of people.

Data Material gathered during the course of an evaluation which serves as the basis for information, discussion, and inference.

Data access The extent to which the evaluator will be permitted to obtain data during the course of an evaluation.

Decision rule A rule for choosing between optional interpretations or courses of action given certain evidence (e.g., a rule by which teachers pass or fail students in a course based on their test scores and other performances in the course; a rule by which a government agency ranks project proposals for funding based on their contents and the ratings assigned to them by judges; or a rule by which an evaluator decides that the difference between the test scores of students exposed to different programs is statistically significant).

Delphi Technique A method for obtaining group consensus involving the use of a series of mailed questionnaires and controlled feedback to respondents which continues until consensus is reached.

Dependent variable A measure (e.g., a student's performance on a test) that is assumed to vary as a result of some influence (often taken to be the independent variable ).

Design (evaluation) A plan for conducting an evaluation; e.g., data collection schedule, report schedules, questions to be addressed, analysis plan, management plan, etc. Designs may be either preordinate or emergent .

Dissemination The communication of the actions--by written, oral, and/or audio-visual reporting--of evaluators to foster knowledge of the evaluation findings among all right-to-know audiences.

Editorial authority The extent of the evaluator's authority to edit evaluation reports prior to dissemination.

Emergent design An implementation plan in which the specification of every step depends upon the results of previous steps, sometimes also known as a cascading or rolling design.

Escrow agent A third party who, by agreement, controls certain information, such as the names on tests, submitted by a first party, so that this information is not obtained by the second party.

Evaluation Systematic investigation of the worth or merit of an object; e.g., a program, project, or instructional material.

Evaluator Anyone who accepts and executes responsibility for planning, conducting, and reporting evaluations.

Executive report An abbreviated report that has been tailored specifically to address the concerns and questions of a person whose function is to administer an educational program or project.

Executive summary A nontechnical summary statement designed to provide a quick overview of the full-length report on which it is based.

Experimental design The plan of an experiment, including selection of subjects, order of administration of the experimental treatment, the kind of treatment, the procedures by which it is administered, and the recording of the data (with special reference to the particular statistical analyses to be performed).

Experimental group A group of subjects assigned to receive a treatment (independent variable), the effects of which are measured (dependent variable). Often comparisons are made between these effects and those observed for a comparison or control (nontreatment) group.

Experimental research Scientific investigation in which an investigator manipulates and controls one or more independent variables to determine their effects on the outcome (dependent) variable.

External evaluation Evaluation conducted by an evaluator from outside the organization within which the object of the study is housed.

Extrapolate To infer an unknown from something that is known. (Statistical definition--to estimate the value of a variable outside its observed range.

Feasibility The extent to which resources allow an evaluation to be conducted.

Field test The study of a program, project, or instructional material in settings like those where it is to be used. Field tests may range from preliminary primitive investigations to full-scale summative studies.

Focus group A group selected for its relevance to an evaluation that is engaged by a trained facilitator in a series of discussions designed for sharing insights, ideas, and observations on a topic of concern.

Formative evaluation Evaluation designed and used to improve an object, especially when it is still being developed.

Gain scores The difference between a student's performance on a test and his or her performance on a previous administration of the same or parallel test.

Generalizability The extent to which information about a program, project, or instructional material collected in one setting can be used to reach a valid judgment about how it will perform in other settings.

Generic rights Rights that are shared by all members of a group.

Goal-free evaluation Evaluation of outcomes in which the evaluator functions without knowledge of the purposes or goals .

Guideline A procedural suggestion intended to help evaluators and their audiences to meet the requirements of the evaluation standards; strategy to avoid mistakes in applying the standards.

Hardware (data processing) The physical components, such as a computer and keypunch machine, of a data processing system, as opposed to the instructional (content-related) components.

Illustrative case An illustration of how a standard might be applied, which includes the description of a certain setting, a situation in which the standard is not met, and a discussion of corrective actions that would result in the standard being met.

Information needs Information requirements of the evaluator, clients, and other pertinent audiences to be met by the evaluation.

Information sources The persons, groups, and documents from which data are obtained.

Informed consent Agreement by the participants in an evaluation of the use of their names and/or confidential information supplied by them in specified ways, for stated purposes, and in light of possible consequences prior to the collection and/or release of this information in evaluation reports.

Instrument An assessment device adopted, adapted, or constructed for the purpose of the evaluation.

Internal evaluation Evaluation conducted by a staff member or unit from within the organization being studied.

Jury trial for projects Project evaluation patterned after jury trials in their procedures for clarifying issues, introducing and assessing evidence, and reaching conclusions. Sometimes known as Adversary Model of Evaluation.

Level of significance The probability that observed or greater differences occurred by chance.

Matching An experimental procedure in which the subjects are so divided, by means other than lottery, that the groups are regarded for the purposes at hand to be of equal merit or ability. (Often matched groups are created by ensuring that they are the same or nearly so on such variables as sex, age, grade point averages, and past test scores.

Materials evaluation Evaluations that assess the merit or worth of content-related physical items, including books, curricular guides, films, tapes, and other tangible instructional products.

Mean (arithmetic) A measure of central tendency calculated by dividing the sum of all the values by the number of the values.

Merit The excellence of an object as assessed by its intrinsic qualities or performance.

Metaevaluation Evaluation of an evaluation.

Modus operandi analysis Deducing the cause of effects based upon analysis of events, process, or properties associated with the effects; analogous to procedures used in detective work.

Nonreactive measures Assessments done without the awareness of those being assessed.

"No significant difference" A decision that an observed difference between two statistics occurred by chance.

Norm A single value, or a distribution of values, constituting the typical performance of a given group.

Null hypothesis The hypothesis of no difference or no differential effects.

Object of the evaluation What one is evaluating; e.g., a program, a project, or instructional material.

Objectives-referenced test A test whose scores are referenced to the attainment of the objectives the test was designed to measure, rather than to the performance on the test by some comparison group of people.

Operational definition A definition of a term or object achieved by stating the operations or procedures employed to distinguish it from others.

Overview A conceptual/introductory statement that gives essential definitions; provides a general rationale; and presents summarized procedures, common problems, and special difficulties that are applicable.

Parallel forms Multiple forms of a test constructed to be as comparable and interchangeable as possible in their content, length, and procedures of administration, and in the scores and test properties (e.g., means, variances, and reliability indices).

Pilot test A brief and simplified preliminary study designed to try out methods to learn whether a proposed project or program seems likely to yield valuable results.

Pitfall A not easily recognized difficulty believed to be associated with a particular standard. These are mistakes that would be made out of ignorance of the import and intent of a standard.

Population All the persons in a particular group.

Post-test A test to determine performance after the administration of a program, project, or instructional material.

Pre-test A test to determine performance prior to the administration of a program, project, or instructional material.

Program evaluations Evaluations that assess ongoing activities that provide services. Educational examples include evaluation of a school district's reading program, a state's special education program, and a university's continuing education program in a specialized field such as ophthalmology.

Project evaluation Evaluations that assess activities that are funded for a defined period of time to perform a specified task. Some examples are a three-day workshop on behavioral objectives, a two-year development effort, or a three-year career education demonstration.

Propriety The extent to which the evaluation has been conducted in a manner that evidences uncompromising adherence to the highest principles and ideals (including professional ethics , civil law, moral code, and contractual agreements).

Purposes (of an evaluation study) The objectives of an evaluation (e.g., to judge the relative merits of competing textbooks, or to monitor and report on how well a project plan is implemented) and the intended use of its reports (e.g., to help teachers choose a textbook or to help a school district carry out a special project).

Qualitative information Facts and claims presented in narrative, not numerical, form.

Quantitative information Facts and claims that are represented by numbers.

Random Affected by chance.

Random sampling Drawing a number of items of any sort from a larger group or population so that every individual item has a specified probability to be chosen.

Regression effect The tendency of examinee scoring above or below the mean of a distribution on a pretest to score closer to the mean on the posttest .

Reinforce To strengthen a learned way of behaving by some response-contingent external or internal influence.

Reliable measure A measure that provides consistent indications of the characteristics being measured. (Also, see Generalizability.

Replicate To repeat an evaluation with all essentials unchanged.

Right-to-know audience A group of people entitled to be informed about the results of the evaluation.

Sample A part of a population.

School profile A description (graphic, numeric, or variable) of the status of a school with respect to a set of concepts or variables.

Secondary data analysis A reanalysis of data using the same or other appropriate procedures to verify the accuracy of the results of the initial analysis or for answering different questions.

Self-report instrument A device in which persons make and report judgments about the functioning of their project, program, or instructional material.

Significant difference (statistically) A decision that an observed difference between two statistics probably did not occur by chance.

Simulation study A study in which the probable effects of alternative solutions to a problem are identified by using symbolic representations of real activities, situations, or environments.

Sociodrama Dramatization and role playing used to teach an audience about the findings of an evaluation and to illustrate their potential applications.

Sponsor The individual, group, or organization which provides the funds for the evaluation.

Stakeholders Individuals or groups who may affect or be affected by program evaluation.

Standard A principle commonly agreed to by experts in the conduct and use of evaluation for the measure of the value or quality of an evaluation.

Standardized test A sample of items or situations with definite directions for administration and scoring most often accompanied by data on reliability and validity and sometimes by normative information.

Statistic A summary number that is typically used to described a characteristic of a sample.

Stratified random sample A grouping achieved by dividing the population to be surveyed into a number of nonoverlapping classes or categories which together include all cases, followed by taking cases at random from within the categories.

Summative evaluation Evaluation designed to present conclusions about the merit or worth of an object and recommendations about whether it should be retained, altered, or eliminated.

Table of specifications A two-way grid, sometimes called a test blueprint, that lists major areas of content to be covered by the test as row headings and major kinds of abilities to be developed and tested as column headings.

Test retest reliability The extent to which two administrations of the same test to the same group of subjects yield consistent results.

Time series study A study in which periodic measurements are obtained prior to, during, and following the introduction of an intervention or treatment in order to reach conclusions about the effect of the intervention.

Triangulation The use of multiple sources and methods to gather similar information.

Unit of analysis The least divisible element on which measures are taken and analyzed.

Utility The extent to which an evaluation produces and disseminates reports that inform relevant audiences and have beneficial impact on their work.

Validity The soundness of the use and interpretation of a measure.

Values clarification techniques Procedures used to help groups recognize the different values held in the groups, to discern conflicts among these values, and to consider how these conflicts might be resolved.

Variable A characteristic that can take on different values.

Variate The quantitative measure of a variable.

Word attack skills Means by which a person recognizes and perceives the meaning of words.

Worth The value of an object in relationship to a purpose.