Validation Panel Report
Student Evaluation Standards Project
Prepared by
Robert L. Linn, Chair
Validation Panel
October 1, 2000
This report is based on
1. The draft of Student Evaluation Standards sent out for review in January, 2000;
2. The draft of Student Evaluation Standards distributed to the Joint Committee at the September 28-30 meeting;
3. Discussions of the draft Student Evaluation Standards by the Joint Committee at their meeting on September 28 and 29.
I believe that the Joint Committee and project staff should be complimented for the great progress that has been made in the past year. The January 2000 draft was a good working document for purposes of obtaining input from the national review panel and the few international review panel members that provided comments. Effective use was made of the comments obtained from reviewers and as a result, the draft that was distributed to the Joint Committee on September 28 was a substantial improvement over the January draft.
The Joint Committee also deserves to be complemented for the serious efforts made at the meeting to identify aspects of the standards that need additional revision. The members of the Joint Committee displayed a high level of professionalism and willingness to put in the effort needed to make further improvements in the standards.
The Joint Committee had two major items on the agenda for the 2000 annual meeting corresponding to its two major projects: the development of the Student Evaluation Standards and the development of the second edition of the Personnel Evaluation Standards. A day and a half of the three-day annual meeting was devoted to each of these projects. This report is limited to the first of these projects.
In the day and a half the annual meeting devoted to the Student Evaluation Standards, the Joint Committee had to review the September draft standards, identify revisions that need to be made before the standards are distributed for the field tests, approve a plan for getting those revisions made, and refine and approve the plan for obtaining the field test sites. The Joint Committee also had to make plans for the national hearings. This clearly is a lot to accomplish in a day and a half, especially when it is recognized that the September draft of the standards had not been seen by committee members prior to the start of the annual meeting.
The Joint Committee decided to postpone the national hearings until after the field tests have been completed, thereby allowing time for further planning of the hearings by a subcommittee.
Although the progress made during the day and a half was impressive, it is clear that considerable work remains to be done to have a draft ready for the field tests in early January 2001. Since committee members had to spend the a substantial fraction of time reviewing and making suggested revisions in a small subset of the standards, there was not time for each member of the committee to review all of the standards or see revisions made by the subgroups working on other subsets of standards. Consequently, there is a need to complete that overall review in early November so that the draft for the field tests can be completed in time for the early January target date.
It seems highly desirable that future drafts of the Student Evaluation Standards that will need to be reviewed and eventually approved by the Joint Committee be distributed to committee members in advance of the annual meeting to allow review of the entire document prior to the meeting.
Focus. Previous reports of the Validation Panel urged the Joint Committee to clarify the definition of student evaluation and what constitutes a student evaluation. The Joint Committee was also encouraged to clarify the audience(s) for the Student Evaluation Standards. Considerable progress was made with regard to both of these issues. There still seems to be some confusion, however, about the limits of student evaluation that are intended. On pages 1 and 2 of the introduction, for example, references are made to “information gathered individually and in the aggregate” and to uses for “learning and accountability”. These pairings might lead to the expectation that the standards are intended to apply to uses of test data for purposes such as school accountability.
There is also some ambiguity about the degree to which the Student Evaluation Standards are intended to include uses of standardized tests. The introduction makes a distinction between the Student Evaluation Standards and the Test Standards published by AERA, APA and NCME. But the inclusion of illustrative cases involving uses of standardized tests may lead to some confusion.
Funding. Funding remains a major concern for the work of the Joint Committee. There is a great need to obtain funding that will support the major work that remains to be done, including adequate staff support, funds to provide incentives for people conducting the field tests, funds for conducting the national hearings, and for travel of at-large members of the Joint Committee and members of the Validation Panel. The Joint Committee needs to decide the level of priority that will be given to fund raising and then develop a strategy for seeking the needed support. At the same time, it would seem prudent to develop contingency plans for completing the Student Evaluation Standards in the event that the efforts to obtain funding are not successful.
I thank the members of the Joint Committee for the assistance they provided me and for their openness with regard to the work of the Validation Panel.