Validation Panel Report

 

Student Evaluation Standards Project

 

 

 

Prepared by

 

 

 

Robert L. Linn, Chair

 

 

 

Validation Panel

 

 

 

September 30, 2001

 

This report is based on

 

Field test results sent out prior to the September 27-29, 2001 meeting of the Joint Committee and observations made during the meeting on September 27 and 28.

 

I applaud the Joint Committee and project staff for the great progress that has been made in the past year.  The Field Test Hearings and the National Hearings that have been conducted since the September, 20000 meeting were carried out in a conscientious manner and produced results that were carefully considered by the Joint Committee at its September, 2001 meeting.

 

I also commend the Joint Committee the serious efforts made at the meeting to identify aspects of the standards that need additional revision.  The members of the Joint Committee displayed a high level of professionalism and willingness to put in the effort needed to make further improvements in the standards.

 

Annual Meeting

 

The first two days of the 2001 annual meeting of the Joint Committee focused on revisions of the Student Evaluation Standards.  The Joint Committee was handicapped in its work in that several of the members who had been expected to attend were unable to do so.  As a consequence, the Joint Committee did not have a quorum on September 27 or the afternoon of September 28.  Hence, except for the morning of September 28 when there is a quorum in attendance, the Joint Committee functioned as a committee of the whole.  Action items requiring votes by the Joint Committee were thus confined to the morning of September 28.

 

Despite the handicap of not having a quorum for most of the meeting, considerable progress was made.  The Field Test results were reviewed and accepted.  It was decided that the Field Tests that were completed were sufficient and consequently field testing was closed.  The results of the National Hearings that have been completed and plans for the ones yet to be conducted were discussed.  An ambitious schedule for revision of the standards and submission of them to ANSI was approved that would lead to the completion and approval of the Student Evaluation Standards at the next meeting of the Joint Committee scheduled for May, 2002.

 

Since it is the standards statements that are most critical for ANSI the main attention during the meeting was focused

on the standards statements rather than the associated explanations, rationales, guidelines, and case studies.  Considerable progress was made in restating some of the standards statements and identifying a couple of instances where standards could be combined.  The emphasis on the standards statements was sensible given the need to come up with a version of the standards that could go to ANSI.  Getting review and sign off by the Joint Committee via e-mail in time to meet a December target date will be critical to meeting the ambitious schedule that was established.  Special attention may need to be given to bringing the members of the Joint Committee who were unable to attend the 2001 Annual Meeting up to date and ensuring their attention and responses to the revised materials.

 

Although it was expedient to focus on the standards statements at this stage, the materials that accompany the standards statements are in most cases essential to the understanding of the intent of the standards and what is required to meet them.  Hence, it would seem prudent to plan for the review and approval of all of explanations, rationales, guidelines, and case studies that are associated with the standards.  In earlier standards documents, this approval has been done formally by voting not just on the standards statements, but on all of the material associated with each standard.  Alternative approaches were discussed at the September meeting.  Whatever approach is to be used, it would be wise to decide on it in advance of the May, 2002 meeting so that procedures to be used at that meeting will be known in advance and so that necessary review and/or polling of members of the Joint Committee can be completed prior to the meeting. 

 

Issues for the Joint Committee

 

There was some discussion at the September meeting about making a number of major changes in the Student Evaluation Standards.  The idea of having one “super standard” per major section (e.g. Validity) followed by several standards aimed at aspects of the super standard was discussed, for example.  An even more radical change that would move away from the four traditional labels of Propriety, Utility, Feasibility, and Accuracy was mentioned.  Although it was decided it was too late in the development process to consider such radical changes, the question of whether such changes should be considered in the future for new versions of any of the three sets of standards is one worthy of serious consideration.

 

The fact that suggestions for major changes in structure were even made at this late stage of the process , may reflect an underlying problem that the four major categories of standards may be more suitable for one set of standards than another.  Forcing the same mold on all standards has a big advantage of continuity for the trilogy of standards.  On the other hand, it also is a kind of straight-jacket that may be more uncomfortable for standards in one arena than in another.  “Validity”, for example, may be a better name than “Accuracy”.  Furthermore, the forcing of standards statements into the mold of a phrase followed by “so that” leads to awkward statements in some instances.  Careful consideration to the tradeoffs would be a worthwhile agenda item for the Joint Committee at some future date.

 

I thank the members of the Joint Committee for the assistance they provided me and for their openness with regard to the work of the Validation Panel.