The Validation Panel met at the Evaluation Center in Kalamazoo, Michigan on May 7 and 8, 2002. All members of the Panel were present at the meeting. Prior to the meeting, the Panel had received copies of the January 2002 Student Evaluation Standards, summaries of the National Hearings and of the Field Tests. The Panel had also received Power Point slides summarizing the procedures of the Joint Committee and of ANSI. The Panel began its deliberations by reviewing Operating Procedures for the Development and Approval of the Joint Committee on Standards for Educational Evaluation . The Panel also requested and received copies of Joint Committee Meeting Minutes from 1998 through 2001, lists of panels of writers, panels of reviewers, and field test sites. The Panel commends Arlen Gullickson and the staff of the Evaluation Center for going out of their way to assist the Panel and to respond to the Panel’s requests for information.

Summary Conclusions

1. The Validation Panel considered the assumptions underlying the Joint Committee’s work that are implicit in the introduction to the Standards, the categories of standards used, and in the Operating Procedures and finds those assumptions to be reasonable.

2. Based upon its review of materials the Validation Panel concluded that the Joint Committee has made a good-faith effort to follow the Operating Procedures.

3. The Joint Committee has been responsive to feedback received on drafts of the standards and produced a document that has reasonable content validity and that is more defensible and potentially useful than earlier drafts of the standards.

4. The Joint Committee has succeeded in producing a set of standards that can be read and understood by people who do not specialize in evaluation as well as those who do.

5. The Student Evaluation Standards provide sound guidance for judging procedures used for the evaluation of students in the classroom. They need to be considered in conjunction with the Standards for Educational and Psychological Tests for the wider range of uses of student evaluation information for purposes such as identification for special education, tracking, and retention.

Issues for future consideration of the Joint Committee

Although the Validation Panel’s overall evaluation is positive, there are a number of issues that we believe need to be addressed by the Joint Committee in the future. Some of these concern items in the Operating Procedures that should be either adhered to more closely or changed. Others have to do with steps that can be taken to increase the usefulness and likelihood that the Standards will be used.

1. Diversity. The Validation Panel recommends that the Joint Committee develop a clear plan to achieve greater diversity of the Joint Committee itself, as well as of all writing and review panels, and the field test sites. It is clear from statements in the Operating Procedures and from a review of the minutes of Joint Committee meetings that achieving racial/ethnic diversity as well as in terms language background and of people with disabilities for all groups involved in the development of the Standards is a goal of the Joint Committee. This is an important goal, in part, because of the implications it has for the ways in which issues of diversity are addressed in the Standards. The Panel’s review of lists of the membership of the Joint Committee, of panels of writers, and of reviewers suggests that additional work is needed to increase diversity throughout the work of the Joint Committee. The Panel notes that the By Laws allow for mechanisms that might be used to increase the diversity of the membership of the Joint Committee.

2. Enhancing the Value of the Standards. The Validation Panel recommends that efforts should go into putting the Standards on the Web and/or on CDs in a form that is layered and searchable. The Standards have great potential utility. However, they are long and may be difficult to use as a stand-alone document. Rather, the Standards document might better be thought of as an authoritative reference. It would be useful to produce a summary document that contains only the standards. Layering would allow a user to identify a standard and move from there to specific materials explaining and illustrating the use of the standard. Furthermore, with the use of metatags it should be possible to allow a user to enter a topic and have the relevant standards identified.

3. Improving the Standards. The Validation Panel recommends that additional context be provided for the Standards and greater emphasis be placed on the role of student evaluation in improvement of instruction and student learning . The fact that almost all states have adopted content standards and the requirements of the No Child Left Behind Act of 2001 are examples of context that would be useful to acknowledge and to point to how the standards might help teachers do a better job of evaluating students in ways that are consistent with the state content standards and that contribute to improved instruction and learning .

4. Procedures for the Evaluation, Interpretation, and Revision of the Standards. The Validation Panel recommends that the Joint Committee develop explicit plans for the evaluation, interpretation, and revision of the Standards as called for in the Operating Procedures.

5. Format. The Validation Panel recommends that multiple layout and display options be pursued that will increase the likelihood of effective usage of the standards by preservice and inservice teachers, parents, students, and other users. The Panel recommends that the Joint Committee consider these and other options for format. (1) Format 1 – Publication of the full set of standards as presented in the February 12, 2002 Draft. (2) Format 2 – Publication of the full set of standards and accompanying sections (Using the Standards and Glossary). (3) Format 3 – A series of four self-assessment tools (which could be checklists, rubrics, or other scoring guides), one each for Propriety, Utility, Feasibility, and Accuracy. These tools would include summary descriptions as, for example, on pages 25-26 of the February 12, 2002 Draft. (4) Format 4 – A web-based version, also duplicated on a CD, that is layered, as is the print version: Descriptor, Standard, Overview, Guidelines, Common Errors, and Cases. This version should be catalogued using metatags, such as those identified by the Dublin Core. This would permit searches by key words, and by standards areas, and could include electronic links reflecting the cross references in the text.

6. Operating Procedures. The Validation Panel recommends that revisions of some aspects of the Operating Procedures be made. Four specifics in this regard are as follows (1) The meaning of the phrase “legislative procedures” under section 4.2.2.8 should be clarified. (2) The statement in section 5.1 that Project Staff will attend and participate in the annual meeting of each Sponsoring Organization should be removed. (3) Section 5.2: the label “Criteria” seems misleading. What are listed might better be called “Guiding Principles”. (4) Section 5.4: the Joint Committee does not appear to be in compliance with the requirement of advising persons who provide comments on the actions taken in response to the comments and the reasons for those actions. The Validation Panel believes that complying with this requirement is not feasible and therefore recommends that the Operating Procedures be revised to remove this requirement. Persons who have submitted critiques have the opportunity to review the final copy of the Standards and should be responsible for submitting objections if their concerns have not been dealt with to their satisfaction .

7. Business/Marketing Plan. The Validation Panel recommends that the Joint Committee develop a business/marketing plan. The Joint Committee has had scarce resources to support the development and dissemination of the Standards. Such a plan might identify professional development opportunities and other services related to the Standards that could generate the funds that are needed to support the promotion of the standards and future revisions.

Respectfully submitted May 9, 2002 by:

Robert L. Linn, Chair _______________________

Kathleen B. Boundy _______________________

J. Placido Garcia, Jr. _______________________

Douglas E. Harris _______________________

Audrey M. Kleinsasser _______________________

Victor L. Willson _______________________