1. The standards for users and uses focuses on the core idea of who will use the standards and what the appropriate uses should be in developing good evaluation practice. The program evaluation standards explains the identification and involvement of the various stakeholder groups, while the student and personnel standards focus on the users and uses most appropriate for that emphasis. The student standards guide the evaluator by the identification of those affected by the evaluation and the intended use of the information obtained during the evaluation. The personnel standards include user groups and the intended use of the evaluation as a guide for who should be involved in the evaluation planning. The overlap or similarity of the three evaluation standards for this category lies in the identification and establishment of good evaluation practice for the various stakeholders or ways in which the evaluation will be used. The difference was in the particular audience the standard was written to address, changing the identified user groups. The standards each discussed the composition of stakeholder groups and potential uses of the evaluation. Another emphasized similarity noticed was the caution about unintended use of the evaluation findings and the risk of damage to the way in which the evaluation is perceived by the audience. The context of use was the only difference in the focus of the standards. The program standards addressed the stakeholders both internally and externally to the program, while the student standards identified the stakeholders unique to the classroom setting: students, teachers, administration, and parents (particularly students). The personnel standards included those prospective users of the evaluation (the evaluator) and those affected by the evaluation (particularly teachers). The student and personnel evaluation standards address the specific groups forming the relevant population of the developed standard, adding richness to the more general interpretation and use of the program standards. The relevance of the evaluation system, determination of the use of evaluation, specific recommendations for user involvement, use of evaluation information, the reporting of evaluation process and the monitoring of the evaluation system are all clarified for the specific targets of the student and personnel standards. This provides a follow-up of the relevant context for each within the parameters of the program standards with a clear picture of application to the evaluation of students or personnel.

1. The standards for evaluator credibility and qualifications share the core idea for the establishment and maintenance of evaluator credibility and integrity during the evaluation process. This universally important standard will ensure the responsiveness of the client and those affected by the evaluation in a positive manner. The Program Evaluation Standards give general guidance for the evaluator’s behavior and credentials to effectively maintain credibility. The Student Evaluation Standards emphasize the necessity of the authority and effectiveness of evaluation based on the perception of the qualifications of the evaluator by the students and the impact on the results of that evaluation. There is an emphasis on the acquisition of knowledge and skills of a teacher as an evaluator, in recognition of the lack of preparation in good evaluation practice for many teachers. In the Personnel Evaluation Standards the establishment of the pertinent expertise of the evaluator and the importance of exercising professionalism and effectiveness in the role of the evaluator are emphasized for evaluator credibility. The overlap in the Standards documents exists in the confidence in the evaluator to do a fair and objective assessment and judgment, affecting the confidence in the evaluator and the results. A clear delineation and description determining key needs will assist the development of credibility of the evaluator and the reception of the evaluation results. This central theme is carried out in the Student Standards where students’ perception of the evaluator’s expertise and confidence in impartial, useful results for learning can strengthen the evaluation process, contributing to the constructive use of evaluation for improvement. First the students have to trust the credibility of the teacher and evaluation tool before useful progress using the information can be made. The positive acceptance of evaluation by those evaluating personnel depends upon the perception of the evaluator as credible, with authority, knowledge, and free from conflict of interest. The evaluator must be credible, display integrity, sensitivity, and a commitment to improvement. The context is changed for each of the standards, but the focus of these standards is on the perception of the evaluator in each case. In the Program Standards the credibility must be established with the stakeholder groups, particularly the clients. Student Standards have a context in the classroom with the identified students, and Personnel Standards emphasize a context in the realm of the evaluatee affected by the evaluation. The use of the Student Standards adds richness in guiding the teacher toward ways to establish credibility with their students and others. The Personnel Standards assist the evaluator of personnel in developing accountability for competent and therefore non-threatening evaluation and fulfillment of evaluation responsibilities.

The standards appear to share a core idea in that the focus is on the collection of “sufficient ” and “appropriate” data/information to make fair and balanced “decisions” regarding the object of interest (i.e., program, student, employee). In addition, all information on which a decision or judgment is based should be constructive rather than punitive in order to help the object of interest (i.e., program, student, employee) improve; timely so that improvements can be made as quickly as possible before corrective action is taken or undo consequences are suffered; accurate so that judgments are fair and just; and sufficient to “ cover all the bases” in making meaningful and possibly significant decisions.

In the parallel cases of program, student and personnel standards (conflating the Program standards into one re: effective reporting), the standards highlight the need of clear, timely and accurate reporting of relevant information. There is substantial overlap in the explanation/overview sections as well as within the guidelines and common errors.

The main focus of the Evaluation and Evaluation Impact Standards for programs (U7), students (U7), and personnel (U5) is that the follow-up that occurs is important and that efforts should be made to ensure that results are presented to the stakeholders in an understandable manner so as to increase the likelihood that they will be used or appropriate action will be taken. Similarities among the three standards relate to the involvement of stakeholders, monitoring of follow-up, and providing for clear and open communication. Each of the standards speaks to the need to involve the stakeholders, students, or personnel in developing or identifying activities to incorporate in the evaluation and how such elements address the needs of the target population. In doing so the overall objective remains one of addressing areas of weakness while maintaining areas of strength. Each standard also calls for some type of monitoring the impact of the follow-up to ensure the utilization of evaluation results, but not at the expense of other components of the program or overall learning process. Stakeholders are also invited, when appropriate, to participate in developing evaluation question and carrying out the evaluation. This type of involvement can include getting their input on developing follow-up activities as a way to improve overall program, student, or personnel performance. Finally, stakeholder input is gathered in relation to identifying changes in the evaluation process that will result in better utilization of the evaluation results and improve the overall outcomes of the program.

Feasibility F1 Table | F2 Table | F3 Table Approach: The Feasibility standards and their organization was reviewed for the three evaluation areas?program, student, and personnel?with the Program standards serving as the base. While the organization was not identical, all had the following elements: definition, overview, guidelines, common errors, case descriptions and analyses, and documentation. A spreadsheet was developed which identified the guidelines and common errors for the 3 Standards and associated each of the 3 Standard areas with guidelines and errors. (See attached). In this way, commonalities and/or differences were identified. Three standards make up the Feasibility: Practical Procedures (Practical Orientation), Political Viability, and Cost Effectiveness (Evaluation Support, Fiscal Viability). Core ideas Definition is common with some difference in terminology Some overlap with emphasis/focus changing with context (Program, Student, Personnel). More commonality in emphasis/focus between Program and Student than with Personnel.

Yes, all three standards essentially contain the same ideas: the need for evaluation procedures to be practical, to produce needed information, and to do so in ways that are not disruptive of those impacted by the evaluation. I would argue that in essence, the standards do overlap and do mean the same thing within different contexts. However, with regards to the personnel and student standards, it seems as though the underlying concern that feeds the standard is a concern for people and the impact that the evaluation has on them. In particular, the student standards emphasize the necessity to give particular care in making sure that students are not harmed in anyway by the procedures and that their confidentiality must be protected.

Question 1 - The core ideas that are shared across the three standards are that the planning, development, and conducting of evaluations must be done while keeping in mind the various interest groups affected by or interested in the evaluation. Evaluations should be conducted so as to address interest groups’ questions and concerns and in a manner that allows for the obtaining of their cooperation, and collaboration, if necessary. For this particular standard, it cannot be argued that the meaning is the same with simple context changes. In particular, while both the program and student standard are meant to refer to single evaluations conducted at a specific point in time, the personnel standards focus is on actual evaluation “systems” already (or potential new systems) used to evaluate the performance and qualifications of individuals Because the student standards are similar to the program standards, I do not any particular ways they can “add richness” to the program standards. One might argue that to consolidate the program and student standards on political viability, it might only be necessary to point out that the key parties to keep in mind would be students and their parents, and that as such, it is necessary to keep in mind the well-being of the child and the implications of the evaluation for that child’s future.

The three standards share a common core of ideas regarding the necessity of being clear about the evaluations purposes, ensuring that the resources necessary to conduct the evaluation are sufficient, accurately estimating timing, personnel and financial costs, and generally being frugal (though not allowing the evaluation to become a slave to a budget and become detrimental) To some extent the meaning of the standard is the same across the three. However, as with the political viability standard, the personnel standard refers to the allocation of funding necessary to correctly and efficiently carry out existing evaluation systems, while the student standards refer to making sure that teachers and administrators are the recipients of the proper resources (whether those resources are actual dollars or training) to conduct good evaluations. Because of the similar purposes they share, I don’t believe that the other two can add any real richness to the program standard.

The overall focus of the Service Orientation Standards (P1) for programs, students, and personnel is to address and meet the needs of the targeted population. In comparing the program, student, and personnel standards for P1: Service Orientation, it is evident that each standard specifically addresses the need to identify a clear purpose of the evaluation that is communicated to the stakeholders, students, or personnel. Resources should be effectively utilized so they promote the goals of the organization or institution or the needs of the student or community. The standards also speak to the need for reporting of interim findings or results of formative evaluations as a mechanism to monitor the effectiveness of the targeted program. Each standard also speaks to the need to promote continual improvement and excellence in the services or learning that is delivered to the target audience. In addition, the evaluation should identify both intended and unintended program effects on the target population so that later action can be taken if necessary. Another common element between the three standards is the specification that the evaluation process should be explained to the stakeholders in relation to how the overall evaluation serves their respective needs. Finally, a connection should be clearly identified between the assessed needs of the target population and the evaluation itself. Examination of the three standards in greater detail indicates that the standards are closely related and in many respects are supportive of each other. In regards to the Service Orientation standard (P1) for program evaluations, the student and personnel standards address specific components within programs; specifically the students or community whose needs are being addressed and the personnel required to implement the program to meet those needs. Thus the student and personnel standards provide additional structure and guidance to program evaluations in relation to elements to consider and how they should be conducted in order to address the needs of the target population. Through this added support the effectiveness of the program is likely reinforced and the outcomes hopefully more assured.

These three standards share a common idea that policies and procedures governing the evaluation process, whether the evaluand is a program, student, or employee, should be in writing to ensure that all parties concerned understand what will occur in the evaluation process. Transparency, fairness, consistency, and buy-in are the rationale. Common features include: (1) meaningful involvement of the stakeholders (including students, where appropriate) in developing the evaluation guidelines and procedures, (2) periodic review and revisions by affected parties, (3) roles in the evaluation clearly defined, (4) protocols for the use and dissemination of results outlined in writing, and (5) conforms to relevant laws, rules, regulations and union contracts.

The basic thrust of these standards is to treat individuals with dignity and protect them from harm by respecting and honoring their legal and human rights.

Each of the three standards analyzed share the core ideas of minimizing harm to the subject, understanding the subject, generally respecting the subject, respecting the privacy of the subject, and the expectation that the subject will gain from the evaluation. These ideas pervade all three sets of standards; only the nature of the subject changes from program to student to personnel. The guidelines for the treatment of students and personnel can and do add richness to the program evaluation human interactions standard. The evaluation of students and personnel are subsets of the evaluation of a program. Thus, the program evaluation standard provides the overarching ideas, while the student and personnel evaluation standards give specificity to those subsets of the evaluation.

These standards are essentially identical. They both require the presentation of both strengths and weaknesses so that strengths can be built upon and problem areas addressed. Both sets of standards suggest the solicitation of comments on the thoroughness and fairness of the evaluation from parties representing diverse perspectives. Both standards also warn against manipulating the evaluation of strengths and weaknesses and discontinuing an evaluation that might prove embarrassing.

P6 and P7

A1 and A2

A3 and A4

“Validity is the single most important issue in the assessment of any evaluation process,” as directly stated in both the Student and Personnel Evaluation Standards (p. 128 and p. 99 respectively). Validity is not singled out as the most important issue in the Program Evaluation Standards but is clearly important. All three standards highly value the use of more than one method of assessment (p. 146 A5H, p. 129 A1B, p. 99 A4A and p. 105 A5G)

The three standards are quite similar across contexts concerning reliability. Each states the importance of using multiple means of assessing effects (A6A implied, A6D, A5G), emphasizes training for evaluators to assure consistency (A6G, A6G, A5D and A5E), and cautions of the importance of checking for inconsistencies in interpretations between evaluators (A6F, A6H, A5B).

The core idea of this standard is that information, including judgments based on that information, should be systematically and securely analyzed and stored. How this standard is manifested depends on information sources (e.g., students, employees, employers) and users (e.g., clients, employers, policymakers). Policies should be established and people handling data should be trained to ensure accurate and secure data entry, analysis, and storage.

The core idea of this standard is that quantitative information should be appropriately and systematically analyzed so that evaluation questions are answered. Analysis results should be useful and replicable. Any weaknesses in the research design and analysis should be divulged and their influence on interpretations and conclusions described. Reliability and replicability are essential elements of this standard. Reliability of both program and student evaluation requires that multiple measures be used for data collection and that data be analyzed in multiple ways, yielding similar patterns. Procedures for data collection and analysis should fit the evaluation questions, whether the focus is student learning or program impact. There is no equivalent personnel standard.

The Analysis of Qualitative Information (A9) of the Program Evaluation Standards and the Analysis of Information (A9) of the Student Evaluation Standards do share a core idea in that both are concerned with the development of appropriate methods for the collection and analysis of data. Both standards address the need for accurate data that is systematically and accurately analyzed. The underlying objective for both is to bring us closer to what we are trying to understand and to provide an evaluation that is effective, fair, and accurate. Each of the standards also addresses the need to establish measures that can be verified and have a useful purpose. The standards also address the need for communication and explanation to stakeholders. In addition, both standards are concerned with the use of multiple methods in order to provide objectivity and accuracy. However, the Program Standard specifically addresses multiple measures within qualitative analysis. The Student Evaluation Standard, on the other hand, addresses the use and combination of both qualitative and quantitative data. Qualitative data, within the Student Standards refers more specifically to a type of student assessment that might be open ended or consist of observations, portfolios and presentations. The Program Standards (A9) and (A8) separately address the treatment of qualitative and quantitative data. Today many evaluation studies combine these two data collection methodologies and certainly from a data collection and analysis perspective it makes sense to look at these two areas as separate entities. However, even though the context of use in the Student Standards is very different it might be useful within Program Standards to look at methods for integration of these two analytic techniques.

Justified Conclusions (A10) of the Program Evaluation Standards and the Student Evaluation Standards do share a core idea. In both cases the standards reflect the belief that judgments and recommendations should be defensible and justified. Evaluators in both cases should be prepared to share their plans, procedures and conclusions with appropriate stakeholders. The acceptability of results for both standards is highly dependent on the adequacy of the data and the persuasiveness of the rationales supporting conclusions. The Student Evaluation Standard (A10) explicitly advocates for the use of a written framework to guide student learning, content and performance, sources of information and interpretation guides. This evaluation plan ensures consistency with policies and is presented to stakeholders in an understandable format. The context of program evaluation is obviously quite different. However, an overall descriptive component that outlines the rationale for methods might be useful as a guideline within Justified Conclusions of the Program Standards. This would enable stakeholders to understand the choice of methods and would ensure that evaluators effectively address conclusions and limitations.

A11 Table All three sets of standards focus on applying guidelines that would encourage impartial reporting by removing bias from the evaluation. The core ideas for all three standards seem to be focused on 1) regularly collaborating with the client to ensure impartial reporting steps, 2) developing an evaluation free of bias, 3) seeking out prior conflicting reports and becoming aware of avaluatees potential special needs, and 4) engaging other independent evaluators promptly and as needed. The program evaluation standard (A11: Impartial Reporting) indicates that the evaluator “should guard against distortion caused by personal feelings and biases of any party to the evaluation so that evaluation reports fairly reflect the evaluation findings.” The guidelines for this program evaluation standard focus on the following: • Collaborate with client regarding impartial reporting steps • Clarify report editing roles • Ensure an independent perspective is presented in the report • Seek out prior reports (conflicting with your report) • Maintain independence (i.e., outside audits) • Describe steps taken to protect the integrity of the report The student evaluation standards (A7: Bias Identification and Management) indicates that the evaluation should be “free from bias, so that conclusions can be fair.” The guidelines for this program evaluation standard focus on the following: • Ensure evaluation goals are appropriate • Ensure student interactions are free of bias and appropriate • Use multiple methods to evaluate student’s performance • Develop written policies for evaluating students with special needs • Develop appropriate scoring methods for measuring performance and that the scoring process is not influenced by other factors • Consider the background and history of students • Remove names of the students • Engage other teachers in scoring (check-and-balance) • Summarize other accomplishments (participation, effort) separately from the grade – provide evidence to support each piece • Include both strengths and weaknesses for a balanced evaluation report • Remove any bias from the evaluation process • Provide evaluation process procedures to students and parents The personnel evaluation standard (A7: Bias Control) indicates that “the evaluation process should provide safeguards against bias, so that the evaluatee’s qualifications or performance are assessed fairly.” The guidelines for this program evaluation standard focus on: • Involve evaluatees (and relevant personnel) in evaluation design process • Train all involved in the evaluation process and allow for opportunities for all to criticize the process • Exclude any factors that would disadvantage a group • Use multiple sources to obtain data and judgments • Provide ample opportunities for the evaluates and other relevant personnel to review the report • Compare unscheduled versus scheduled observations • Remove bias in evaluation areas that are subjective than objective • Ensure prompt third-party review as needed The student and personnel evaluation standards include several useful guidelines not included in the program evaluations: 1) use of multiple methods to obtain data and judgments, 2) developing appropriate scoring methods for measuring performance, 3) blind-fold evaluations, 4) reporting and evaluating individual components of a program separate from the final overall evaluation component, and 5) comparing unscheduled versus scheduled observations. These standards could add richness to the program evaluation interpretation and use. For a table of categories on similarities and differences between the evaluation standards, please see Table A.

A12 Table All three sets of standards focus on monitoring evaluations. Although several core ideas are shared between student and personnel standards, not many standards are shared with program evaluation standards. Specifically, the program evaluation standard (A12: Metaevaluation) indicates that “the evaluation itself should be formatively and summatively evaluated against these and other pertinent standards, so that its conduct is appropriately guided and, or completion, stakeholders can closely examine its strengths and weaknesses.” The guidelines for this program evaluation standard focus on the following: • Sufficient budget for metaevaluation • Assign a person to document the evaluation process • Nominee a professional evaluator for metaevaluation • Identify rules for reaching consensus regarding the metaevlaution • Indicated that metaevaluators that do not meet contract obligations will be dismissed • Metaevaluators will have the final authority of editing the metaevlaution report • Identify which audiences will receive reports and how • Evaluate the data collection (evaluation process) how they were implemented • Record steps for metaevalution The student evaluation standards (A11: Metaevaluation) indicate that the evaluation should be “examined periodically using these and other pertinent standards, so that mistakes are prevented or detected and promptly corrected, and sound student evaluation practices are developed over time.” The guidelines for this program evaluation standard focus on the following: • Involve students and stakeholders in the metaevaluation • Determine if student evaluations had a positive impact on performance • Compare how the evaluation was conducted to how it was planned • Judge the evaluation process against these standards • Identify parts of evaluation that need more frequent review • Seek professional metaevaluators The personnel evaluation standard (A8: Monitoring Evaluation Systems) indicates that the evaluation system should be “reviewed periodically and systematically, so that appropriate revisions can be made.” The guidelines for this program evaluation standard focus on: • Determine if the evaluation is having a positive effect on outputs • Budget resources accordingly • Identify parts of the evaluation that need more frequent review • Compare how the evaluation was conducted to how it was planned • Judge the evaluation process against these standards • Survey staff and obtain criticism and recommendations regularly • Engage others to review evaluation policy and procedures annually • Train other evaluators and evaluatees in using these standards • Check for validity and reliability of information The program and student evaluation standards each have one category in common with the program evaluation standards: identifying a third-party metaevaluator and setting aside metaevluation budget, respectively. However, the student and personnel evaluation standards have several standards in common: 1) involving evaluates and stakeholders in the metaevaluation, 2) determining if the evaluation had a positive impact on performance of evaluatee, 3) comparing how the evaluation was conducted to how it was planned, 4) judging the evaluation process against the standards, and 5) identifying parts of the evaluation that need more frequent reviews. For a table of categories on similarities and differences between the evaluation standards, please see Table B. The standard guidelines for student and personnel evaluation standards can add richness to the program evaluation interpretation and use. For example, the involvement of all stakeholders in the metaevaluation process could enhance the program evaluation standards. The metaevaluators could potentially discover key details about the evaluation process from the stakeholders that otherwise would have been unknown.

2. The unique aspects of each of the evaluation standards stem from the audience and type of evaluation the document addresses. The Program Evaluation Standards are more general in nature with overall recommendations for the identification and inclusion of stakeholder groups at all levels of the evaluation plan. The Student Evaluation Standards list the specific potential user groups for classroom instruction, the relevance of the evaluation system, alternatives for use by each user group, and the reporting of the evaluation process to that user group (students, teachers, administration, parents and others). The caveats for use in decision making and student centered education are appropriate considerations in the identification and relationship with these user groups. Another unique aspect of the Student Evaluation Standards is that the evaluation is primarily based on the student and educator roles in the evaluation process, with less emphasis on other groups (as it should be). The most noticeable characteristic of the Personnel Evaluation Standards is the Suggestions for Improvement that should assist the reader in understanding the use of the standard beyond simple text recommendations. This particular standard is based on the supervisory roles in evaluation and relationship to evaluatee. Interestingly, the standard emphasizes supervisor as evaluator but not a reverse evaluation of a supervisor by the direct reports of the supervisor.

2. The unique features of each Standards set support development of the focus for that particular standard set. The Program Standards develop the key audience of the evaluation, emphasizing the work plan and composition of the evaluation team to establish evaluation qualifications. The use of teams is recommended to enhance the credibility to address the needs of all stakeholder groups. The Student Standards emphasize the evaluative development of teachers rather than the key audience selection, since the primary audience exists in the classroom situation naturally. The understanding of effective teaching techniques and principles of learning psychology – and the acquisition of these skills by teachers to establish credibility and proper judgment are included. This form of evaluation is generally done by one evaluator although it may be a piece of a larger whole. The Personnel Standards emphasize the training of evaluators in the skills and principles to be evaluated – establishing a knowledge base to build the evaluation on. This in turn establishes the evaluator’s expertise and authority, together with responsible, accountable evaluation will enhance the reception of the evaluation information. This standard recommends either individual or team approaches to evaluation.

It seems to me that the main way in which the program standard is unique is that the object of the evaluation is a program and not a human being.

The program standard regarding effective reporting is unique structurally as it is broken up into two whereas the parallel concepts in personnel and student standards are contained in a single standard. It is also unique insofar as it may include the concepts of personnel and student evaluation in practice. The formative evaluation of educational programs, for example, may include the formative evaluation of students and personnel within the program (e.g. implementing new curriculum in the classroom), necessitating the use of reporting standards for personnel and students. The reverse case (e.g. program evaluation as a part of student evaluation) is not immediately apparent.

The program evaluation standard of Evaluation Impact (U7) differs from the related standards for students and personnel in that neither of the latter two standards specifically advise providing interim reports of the results. In regards to the student standards providing interim reports via formative evaluations is implied and may be addressed in other standards, but it is not specifically stated in regards to the follow-up component. In comparison the personnel standard includes guidelines that speak to the need for continual communication where records are kept in relation to the development of a professional growth plan and monitoring it fulfillment. Through the various components of the personnel standard the use of interim reports is addressed in an appropriate context. Overall, there are very few differences in the guidelines related to evaluating the impact of a program in comparison to following-up on the results of either a student or personnel evaluation.

See Approach, #1 F1 Table | F2 Table | F3 Table Unique In general, there is much commonality/interplay between Program and Student

As I mentioned above, I think that all three standards are very similar and thus, I don’t feel that the program standard possesses any truly unique qualities. I will mention that the program standards are the only ones that specifically mention that procedures must take into account budget constraints and that procedures should be pilot tested ahead of time. However, though the don’t specifically say so, I’m certain that the personnel and student standards would certainly be amenable to these additions.

With regards to the personnel standards, it could be argued that the program standard could be enhanced by thinking about and conducting the program evaluation as if the evaluation’s success was completely dependent upon the program’s personnel understanding, supporting (and even possibly administering procedures for) the evaluation. Again, the program standard is similar to the student standard (with the emphasized stakeholders in the student evaluation being parents and students). It is unique from the personnel standard in its focus on individual evaluations rather than on the assessment of existing evaluation systems.

The program evaluation standard is not truly unique with regards to the other two.

In regards to unique elements of the program standard compared to the related student and personnel standard a key element that is evident is the need to minimize interruptions to the instructional process while also fulfilling the purpose of the evaluation. Neither the student or personnel evaluation standards speak to this point, but they do address issues that facilitate the balance between instruction and fulfilling the goals of the evaluation. For example both the student and personnel standards speak to the issue of fairness in evaluation in relation to addressing specific individual needs that call for alternate evaluation procedures due to language or disability related needs. The personnel standard speaks to the issue of fairness in regards to conducting a fair, consistent, and legal evaluation process that leads to appropriate results. By applying evaluation processes in a clear and respectful manner to individuals with unique needs or in a consistent manner the likelihood of obtaining reliable results is more assured. In effect the unique comments of the student and personnel standards address specific characteristics of those types of evaluations. In the differences between the three standards it is important to note that the unique needs of the respective populations are better met.

Neither the program nor student evaluation standards require that the evaluation plan address (and incorporate where appropriate) the standards themselves, whereas the personnel evaluation standard does. Guideline C. in P2 of the Personnel standards suggests that evaluations “Ensure that the guidelines address all of the elements for acceptable evaluations set forth in these standards.”

Student standard P4 and Personnel standard P5 differ from the Program evaluation standard in two ways: (1) Although all three standards are concerned about protecting participants in the evaluation process, the Student and Personnel standards go a step further by noting that individual growth and development are more likely to be fostered by fair and credible evaluations, and (2) the evaluator’s credibility will be strengthened in the eyes of the evaluatee (student or employee) if they are treated with dignity and respect. Student standard (P4), Guideline D also recommends that the purpose of the evaluation is made meaningful to the student, whereas this recommendation is not part of the program or personnel standards.

Differences in the three sets of standards are evident in the guidelines and common errors. In these guidelines there is a sense of the program as a distinct object of evaluation that is separate from rather than simply encompassing students and personnel. For example, the program guidelines suggest taking time to learn the particular concerns about the evaluation held by participants. The student and personnel standards do not advise adherence to such a standard. Rather, the evaluator of the student will recognize and attend to such matters only if a student feels an inability to be heard.

One obvious difference in these standards is the lack of a comparable standard for personnel. It is unclear why this component was omitted. The differences between the program and student standards are apparent in the explanation of the standards. In the program evaluation version, complete and fair assessment applies to both the assessment of the program and to the evaluation methodology. In the student version the standard applies only to the evaluation of the student not the assessment of the methodology. Moreover, the program version warns against the evaluator’s personal interests, bias, and speculation affecting the evaluation while no such warnings are made in the student version. For students, there are clear-cut and well established standpoints for judging achievement of this standard. The student standards suggest identifying strengths and weaknesses in relation to the following frames of reference: specified standards, peers, aptitude, amount of improvement, and amount learned. No frames of reference were provided for the program version of this standard.

P6 and P7

A1 and A2

A3 and A4

Generally, Program Evaluation Standards allows more creative flexibility in suggesting methods of assessments whereas Student and Personnel Evaluation Standards suggest more standardized methods exist and should be used.

The Program Evaluation Standard is not particularly unique from the others in the category of reliability.

Student and personnel evaluations yield information that is personal and individual, and is typically used as the basis for judgments such as who is allowed access to services, who is promoted, and who receives other benefits. Alternatively, the program evaluation standard is directed toward ensuring accurate and secure data storage. Program evaluation data may include sensitive information about individuals, but the standard itself focuses less on probity than on data storage and access (instead, these issues are addressed by the Program Evaluation Propriety Standards).

The student evaluation standard considers both quantitative and qualitative data in a single standard. A possible benefit of this approach is that the user can more easily weigh the benefits and challenges of using methods that generate quantitative vs. qualitative data than when the data types are treated separately. Also, the student evaluation standard points out that data analysis often includes a frame of reference (e.g., evaluating student performance in relation to peers, aptitude, pre-specified content standards, etc.).

The Program Evaluation Standard (A9, Analysis of Qualitative Information) is used in a broader context than the Student Evaluation Standard (A9, Analysis of Information). It is unique in a number of ways. First, the standard specifically focuses on program evaluation and on qualitative data. Qualitative data is defined within the standard and specific reference is made to the methodologies that are consistent within qualitative inquiry. Second, the context in which qualitative data is described is consistent with qualitative research which stresses the identification of themes and categories within a group or domain being studied. Third, the standard stresses the use of independent evaluators in order to address consistency and bias and allows for the emergent nature of the evaluation to develop. The Student Standard focuses more directly on student assessment and on both internal consistency of test measurement as well as relevance, purpose and balance of student evaluation.

The Program Evaluation Standard (A10) is unique in a couple of ways. First, the standard stresses that feedback from a variety of participants about credibility should be sought before finalizing the report. This differs from the Student Standards in which consistency with the school’s instructional policies and evaluation methods are communicated prior to the onset of assessment. Second, the Program Standard advocates communicating caution to stakeholders in the interpretation of equivocal findings and stresses the need for both awareness and reporting of limitations. While the Student Standard advocates an approach that takes into account the classroom setting and the need to maintain instructional flow and evaluation feedback, making it impossible for teachers to explicitly justify all evaluation judgments.

A11 Table The student and personnel evaluation standards are unique in that they focus on methodology approaches to the evaluation, development of scoring methods, summarizing other components of the evaluation from the final evaluation (other accomplishments versus grades), and comparing unscheduled and scheduled observations.

A12 Table The program evaluation standards are unique in that they mainly focus on requirements for the metaevaluator. For example, the program evaluation standards specify the requirements for the metaevaluator to meet the contract requirements.

3. The three standards sets are generally compatible, although the context and focus is a bit different. The Program Standards are more general in nature with an overall recommendation for the identification and inclusion of stakeholder groups, to include clients and others impacted by the evaluation and results. The evaluation is on the goods and services to address needs rather than the human interactions due to the evaluation. The guidelines are a formal list of checks and balances for good evaluation practice. For example, the standards recommend defining those in leadership roles first to assist in the identification of other stakeholder groups, to be addressed during the evaluation process. This could potentially limit the stakeholder groups without other methods to identify the potential stakeholders. The guidelines caution about implying all information needs will be addressed by the evaluation that could lead to a less than satisfactory review of the evaluation if a stakeholder group had expectations for receiving information and then did not. The Student Evaluation Standards process and information is directed toward an understanding of the use of information for improvement of learning. The use of formal and informal assessments and evaluation is recommended, rather than primarily a formal evaluation plan. As an example, for students of special needs, evaluation helps identify and develop appropriate programs designed specifically to help those students. The intended use targets the evaluation properly and produces useful information as a consequence. The Personnel Evaluation Standards are directed toward the knowledge and skills of employees and is therefore a more formal process of evaluation and reporting. The use of the information for positive improvement and the risk of revealing information is a critical emphasis of these standards. The standards recommend reaching formal agreements with all involved and authorized use of the evaluation information, for example in the case of training programs that targets skill improvement, yet are subsequently used to determine promotions, which was not the intended use.

3. The Standards sets are generally compatible although there are some features that are not in each set. The Program Standards exhibit a defense of the utility, practicality, integrity, and technical adequacy of the evaluation reports as a basis for the establishment of evaluator credibility. The consideration of work plan review or audit for establishment of qualifications, as well as a statement of qualifications in the evaluation report would only be done for a program evaluation. The Student Standards emphasize the unique nature of the relationship between the teacher and student. The lack of knowledge in good evaluation practice is often not recognized by the teacher, although it is often very obvious to the student. The teacher is in authority by their position yet must develop trust through consistent and fair practice of evaluation. The difficulty of sensitively relating criticism in a constructive way is appropriate for evaluation with a human element. The Personnel Standards suggest a lack of recognition of weak evaluation practice similar to yet subtly different from Student Standards for credibility. For example, an evaluator’s personality affects the nature of the communication so the evaluator must know the person in their role as well as response. Or, the evaluator could support a practice as sound due to length of use rather than reliable information.

I do not believe that the standards are incompatible.

I see no way in which they are not compatible given that evaluation reporting in any of these three cases may have evaluation purposes that are summative or formative.

Examination of the program, student, and personnel evaluation standards no inconsistencies were identified. All three standards were found to address the need for communicating the results of the evaluation in a clear and effective manner. As appropriate and as the availability of resources allow follow-up of the utilization of the evaluation results is strongly advised. In addition efforts to engage stakeholders in the implementation or utilization of the results of the evaluation is suggested when and where appropriate.

See Approach, #1 F1 Table | F2 Table | F3 Table Compatibility The Standards are compatible.

I think they are compatible.

I believe that are the standards are generally compatible. Again the concern is over the necessity to take into consideration all interest groups, to make sure that they possess a common understanding of the evaluation and its purpose to the extent that obtaining their support for and/or participation in the evaluation is not problematic.

They are compatible. One only need remember what or whom the evaluation funding is for, but the need to clear about its usage and to be frugal, yet effective is the same across the board.

Comparison of the three Service Orientation (P1) standards does not result in the identification of any inconsistencies among them. In fact the comparison indicates an effective fit among them in that they each focus on identifying, addressing, monitoring, and communicating the results of the evaluations to their respective constituents.

There seems to be considerable overlap and compatibility among these standards; only the evaluand (program, student, or employee) differs.

There is considerable overlap. However, the potential for the evaluation to foster personal development for the employee and academic achievement for the student and the notion of adding to the evaluator’s credibility sets these two apart from program evaluation.

One possible source of incompatibility in the standards is the notion that it would not be in error to embarrass individuals if it exposes weaknesses in a program. However, embarrassing students and personnel would always be in error.

These standards are very similar, and, consequently, very compatible.

P6 and P7

A1 and A2

A3 and A4

Inconsistent but with reason Program and Personnel Evaluation Standards recommend the inclusion of the affected parties as part of the validation steps (p. 146 A5A., p. 99 A4C and p. 86 A1B). Students who have deficient performance are to be provided with opportunities to explain why; otherwise their input is not sought (p. 144 A3F). Inconsistent but with reason Program and Student Evaluation Standards consider handicaps or special circumstances which affect the evaluation, but this is absent in the Personnel Standards. (A5J and A3F)

Inconsistent For learners described in the Student Evaluation Standards, the conditions of the assessment should be fair because each assessment opportunity is a snapshot in time of the learner’s progress (A6F). Personnel and program evaluations tend to be determinations over a longer time period so that the conditions on any given day of an evaluation are likely to create fewer inconsistencies. Inconsistent (and minor) Program Evaluation Standards recommends familiarity with the Standards for Educational and Psychological Testing and Responsibilities of Users of Standardized Tests when testing instruments are indicated. Student Evaluation Standards does not overtly recommend the later and Personnel Evaluation Standards does not reference these works. Inconsistent (and minor) Program and Personnel Evaluation Standards draw attention to newly developed information gathering procedures (A6B, A5F) and encourages reliance on research for consistency. The student standards (A6E) refer to “accepted assessment development procedures” without suggesting there can be innovation in this area. This is not a strong conflict between the areas, but rather a difference in semantic emphasis.

Conflict may arise because the program evaluation standard encourages double-checking of data entry by different people. Because the student and personnel evaluation standards place significant emphasis on security of private information about individuals, review of student and personnel data entry should be done by trained individuals who understand the importance of maintaining the security and confidentiality of the data. In addition, the personnel evaluation standard encourages providing personnel with opportunities to respond to evaluation information. It is intriguing to consider whether students and program personnel should be offered opportunities to fully examine and respond to evaluation findings. Collaborative program evaluation approaches allow for this, but student evaluation often discourages or prohibits regular student and/or parent responses or documentation thereof.

The standards do not appear to be incompatible. In fact, meeting the requirement of reliability and replicability in student evaluations themselves bolsters the validity of patterns observed in evaluation of student programs.

The Standards most notably lack compatibility in that the Program Evaluation Standard (A9) focuses strictly on qualitative data while the Student Evaluation Standard (A9) uses quantitative and qualitative data as student assessment tools. The Student Evaluation Standard (A9) looks at methods to quantify qualitative assessments. Within a strict qualitative study quantitative measures would be discouraged.

The Justified Conclusion standards appear to be very compatible. The primary difference is in the scope of what the evaluation is trying to address.

A11 Table Almost all guidelines included in student and personnel evaluations (that are not included in program evaluations) can also be applied to program evaluations. However, one of the guidelines in student evaluations focuses on summarizing individual accomplishments of a student separately from reporting the grade. In this manner, the student will receive merit for class participation and/or demonstrated efforts (not including grades). Since program evaluations are unique in that (in most cases) a complete picture of the evaluation findings must be presented to the audience, developing separate reports of different components of an evaluation may be misleading (not providing a complete picture). However, depending on the magnitude of the program, it might be useful to report on separate component of a program evaluation, and not the program as a whole.

A12 Table The program evaluation standards have few guidelines in common with student and personnel evaluation standards. However, the student and personnel evaluation standards have several standards in common: 1) involving evaluates and stakeholders in the metaevaluation, 2) determining if the evaluation had a positive impact on performance of evaluatee, 3) comparing how the evaluation was conducted to how it was planned, 4) judging the evaluation process against the standards, and 5) identifying parts of the evaluation that need more frequent reviews. The standard guidelines for student and personnel evaluation standards could add richness to the program evaluation interpretation and use. For example, involving all stakeholders in the metaevaluation process could enhance the program evaluation. In addition, depending on the program, it might be useful to identify parts of the evaluation that would need more frequent reviews. Furthermore, a program evaluation standard could also include determining if the evaluation had a positive impact on performance of evaluatees.

4. The Standards for Students and Personnel should be used as point of reference for the specific recommendations for how to evaluate the stakeholder groups who are affected by the evaluation. When designing the key components of the program evaluation that target specifically the students and personnel. The program plan should include a plan for the use of the information in the improvement of learning – by students, teachers, and evaluator personnel. The advantages of identifying and involving potentially important stakeholders who will likely attend to the results and give a positive response lend support to the evaluation results. The promotion of the inclusion of minority groups enhances the broad application of the information to discover and accommodate all needs possible. The Student Standards emphasize the student needs and improvement of learning as central to the role of evaluation. Limiting access to predetermined purposes and users assists in the development of trust and acknowledgement of the importance of evaluation to improvement efforts. The Personnel Standard for identification of specific uses guards against misuse to limit damage and withholding of important information. This in turn supports the validity of the evaluation information.

4. The Program Standards should be used for guidelines in the overall evaluation plan. Both the Program and Student Standards are beneficial to the teacher for good evaluation practice. The Student Standards outline practice for evaluating students in everyday evaluation and assists in knowledge of the questions to ask about school, district, and regional evaluation. It will also improve the importance of addressing the student as the center of learning, supporting credibility of the teacher as evaluator. The Personnel Standards are appropriate to stand alone for those not involved in the evaluation of the whole program but rather this part of evaluation. Personnel evaluators are not necessarily involved in program or student evaluation, while teachers are generally involved in both program and student evaluation practices. The maintenance of credibility and impartiality in assessment and evaluation of programs affects the use of the standards. Teachers should use both the Program Standards to outline evaluation plans for the programs or courses they teach/support and the Student Standards for student evaluation work. Teachers should apply knowledge and skills to conduct meaningful student evaluations, leading directly to improvement. Personnel evaluators should in turn use the teachers’ use of meaningful assessment and evaluation to tie the personnel evaluation to the programs the individual is responsible for/to in order to produce a fair evaluation of work.

In general, I don’t believe that including personnel and student standards changes the use of the program evaluation standards. I do, however, think it is important to extend program standards by keeping the student and personnel standards in mind during the process because they and others potentially can be affected (positively or negatively) throughout.

In the case of the effective reporting standard, including personnel and student standards does not appear to change the use of the program standards. As mentioned above, the application of those sets will depend upon the focus of the program being evaluated.

When applying the Program Evaluation Impact Standard (U7) both the student and personnel related standards assist in breaking down the components of the program to help ensure its success, however that may be defined, and to identify appropriate follow-up activities. In effect the student and personnel standards provide additional guidance on how to address needs of individuals involved in the program as either recipients or implementers respectively. By providing this additional structure in an effective manner one is likely ensuring the effective implementation of the program goals. For example, when evaluating an educational program the effective use of the student standards can assist in identifying elements or tasks that need to be carried. The same is true when applying the personnel standards in that by addressing the needs of personnel in a timely and respectful manner and addressing their needs identified in the evaluation process one is helping to ensure that the program’s goals will be effectively addressed and results appropriately incorporated.

See Approach, #1 F1 Table | F2 Table | F3 Table Points of Reference A full checklist along the lines of the checklists produced would be very useful especially for those interested in Program and Student standards as there in overlap across Program, Student, and Personnel Standards.

I think that the special attention given to protecting individuals in the process of conducting your procedures is something that could easily be taken up by the program evaluation standards without taking anything away from it.

Because all three of the standards share the same underlying goals, it does not appear that it would be necessary to take into account the personnel and student standards when applying the program standards. Nor does it seem that bringing the other two into the picture enhances the advantages of the program standards in any significant manner or addresses any of its disadvantages. However, as I mentioned above, it might be beneficial to keep a “personnel evaluation mindset” when conducting a program evaluation in that focusing on ALL program personnel may prevent you from missing out on organizational structures and dynamics you might not have otherwise seen.

The standard is so understandable, predictable, and self evident that it doesn't require the referencing of any of the other two to enhance its application.

When applying the program evaluation Service Orientation Standard (P1) the related student and personnel standards should be incorporated within their respective areas. In doing so it is likely that the overall needs of the evaluation process will be more effectively identified and addressed.

It may be worth considering whether program evaluation standard P2 should incorporate a guideline similar to Guideline C in personnel standard P2, which recommends that evaluators address and incorporate (to the extent possible) program standards in designing evaluation plans. The recommendation to address these or similar standards are referred to in conducting metaevaluations (A12 in Program and A11 in Student), but there is no recommendation to utilize these standards in designing an evaluation. Such a recommendation may be worthy of consideration.

The dual rationale of individual growth and development and strengthening evaluator credibility could possibly be incorporated into the program standard as well. Program evaluations frequently entail collecting information from a variety of individuals, such as program participants, program administrators, and representatives of the funding agency. When the rights and welfare of these individuals are protected and honored, there is potential for further growth and development of the program, in the sense that these individuals may become more favorably disposed to and supportive of the program. Furthermore, while the program standard (P3) does not mention strengthening the evaluator’s credibility, this could potentially be an outcome if individuals involved in the program evaluation are treated with respect and honor.

Although the program evaluation standard for human interactions could stand alone, the student and personnel standards provide guidelines specific to the population that could compromise a program evaluation if they were not referenced. For example, the guidelines for the treatment of students suggest using instruments in the student’s language for LEP students and establishing procedures for dealing with students with disabilities. If a program evaluation design involves student data aggregated to a program level, failing to follow these guidelines for the treatment of students would result in inaccurate student level data and inaccurate program level data.

11. The student standard should be a point of reference when applying this standard. The student version takes into account strengths and weakness of students that affect the appearance of the program’s strengths and weakness. For example, in outcome evaluation with various programs as treatment groups, student strengths and weakness fall into the category of within group variation while strengths and weakness of programs fall in the category of between group variation. Failing to recognize the affect of the student on the program and the program on the student would be incomplete evaluation.

P6 and P7

A1 and A2

A3 and A4

To the extent that program evaluations include the measurement of the performance of learners, the Student Evaluation Standards would be helpful regarding validity. If a program evaluator needs guidelines in developing job descriptions for hiring subcontractors the Personnel Evaluation Standards could be useful. Otherwise the Program Evaluation Standards appear to be comprehensive and stand alone concerning validity.

A program evaluator could use the Personnel and Student Evaluation Standards to amplify the message in the Program Evaluation Standards book but the standards for reliability are fairly consistent across contexts.

As recommended above, entry of student and personnel data should be done by trained individuals who understand the importance of maintaining the security and confidentiality of the information. Each standard would benefit from consideration of guidelines for the complementary standards. For example, student evaluation might become more service-oriented if students are given opportunities to respond to their evaluations (e.g., using tests in a formative rather than summative way). The propriety and accuracy of program evaluation are reinforced by the consideration of the complementary student and personnel evaluation standards regarding secure and confidential data handling.

It may be worthwhile to add guidelines or examples of common errors that encourage evaluators to consider whether quantitative or qualitative data collection methods are most appropriate given the specific evaluation questions. Student evaluations often include a frame a reference for judgments about a student’s achievement. It is of interest or even appropriate to offer frames of reference for judging programs? Should program performance be judged in relation to other programs, anticipated outcomes, pre-specified standards, or other benchmarks? Different models would offer different answers to these questions, but are there overarching standards for judging program merit or worth within a frame of reference?

In that both standards address some similarities in the core idea they should be considered points of reference. Consistency between the standards in terms of overriding values is particularly important. For example, both standards stress the common value of the importance of accurate data that is systematically and objectively analyzed. The use in both standards strengthens that overall belief. Similar categorization in each of the standards also serves in ease of use. However, one would need to be cautious of the different contexts and meanings with in each of the individual standards.

The Student Evaluation Standard should be considered a point of reference for three purposes. First, when the program evaluation relates to the assessment of students. In this case the Justified Conclusions could be cited as having or not having been met. Second, consistency and compatibility of the Standards serves to strengthen the overall value of the importance of justified conclusions. Third, similarity in classification serves in ease of use

A11 Table Including personnel and student standards expand the scope of program evaluation standards by ensuring use of unbiased methodologies. For example, the student and personnel evaluation standards indicated use of multiple methods for data and judgments. Including this standard in a program evaluation would allow the evaluator to ensure that the methodologies used for data collection (or scoring of qualitative data) was unbiased and reliable.

A12 Table Since there are differences in the standards used in student and personnel standards versus the program evaluation standards, including these additional standards (student and personnel) would change the scope and more completely lay out program evaluation standards. For example, involving all stakeholders in the metaevaluation process could enhance the program evaluation - the metaevaluator could potentially discover key details about the evaluation process that otherwise would have been unknown. Also, it could be useful for the metaevaluator to compare how the evaluation was conducted to how it was planned and determine any differences and explanations for these differences – collection of potential data to explain evaluation planning/design efficiency. In addition, depending on the program, it could be useful to identify parts of the evaluation that would need more frequent reviews. Furthermore, a program evaluation standard could also include determining if the evaluation had a positive impact on performance of evaluatees.

5. The use of all three standards, particularly the Program Standards with the use of the Student Standards by teachers, or Personnel Standards for institutional evaluators, define and address the general and more specific concerns of educational evaluation. The needs of students and personnel, underlying human interaction and development of trust relationships are outlined in these standards. While the central idea is the identification and appropriate involvement of appropriate stakeholder groups, much understanding of the subtleties of users and uses can be additionally obtained from these three standards. The Program Standards outline the definitions and general guidelines for stakeholders beyond clients effectively, yet the Student and Personnel Standards emphasize those whose work is being studied, those most likely to be affected by the results of evaluation.

5. The evaluators of the overall program should apply principles of sound evaluation from each of the three perspectives as appropriate to the stakeholder groups. The Student and Personnel Standards each assist in the evaluation of the critical stakeholder group for the evaluation situation, therefore serve to establish the credibility or qualifications of the evaluator with the particular critical group in question.

Overall, the important focus is on being extremely careful and cautious about making judgments and/or decisions that may carry significant consequences in order to meet the best interests or needs of the stakeholders concerned (i.e., program stakeholders, student, parent, employee, employer, organization/institution, other interested parties).

I see only strengths in using the parallel standards together in the evaluation of educational programs. Many of these involve introducing innovations to the classroom, where monitoring of teacher and student effects is crucial to understanding the how the innovation functions and might be improved.

See emailed figure.

See Approach, #1 F1 Table | F2 Table | F3 Table Comments To repeat, a checklist encompassing guidelines and common areas across the Standard areas would be very useful

I think that while it may not be the case for many of the other standards, this particular standard and in particular, its underlying concerns, really are identical to the other two.

No comment

It seems to me that in general, there is no harm in bringing to bear the "mindset" of a particular standard to the others. If it has any effect at all, it is only enhance their application.

See emailed figure

These standards raise awareness that regardless of what is being evaluated (program, student, or employee), similar considerations about transparency of procedures, involvement of evaluatees in developing and revising the procedures (to the extent possible), clarity of roles, written procedures for the dissemination and use of results, and compliance with existing laws and contracts are important in ensuring fairness, consistency, and buy-in.

Program evaluation could be strengthened by incorporating the student (P4) and personnel (P5) standards. The impact of doing so would be in terms of (1) fostering growth and development of the evaluand, and (2) strengthening the evaluator’s credibility.

Source: Joint Committee on Standards for Educational Evaluation, The Program Evaluation Standards The Student Evaluation Standards The Personnel Evaluation Standards

Source: The Joint Committee on Standards for Educational Evaluation The Program Evaluation Standards The Student Evaluation Standards The Personnel Evaluation Standards

P6 and P7

A1 and A2

A3 and A4

none

No additional comment

No other comments.

No other comments.

I was struck by the lack of comments and/or cautions about the use of the (A9) standard in different cultural or ethnic settings. There is also no reference to the use of computerized evaluation or assessment tools.

Justified Conclusions is a critical component of the Accuracy Standards. Final evaluation is highly dependent on the stakeholder’s belief that the data was valid and useful.

A11 Table Use of parallel standard would add to the evaluation’s reporting. For example, the student evaluation standards suggest reporting on the grade separate from other accomplishments of the student. Depending on a program’s structure and the political environment of the issues that are addressed in an evaluation, it might be useful to report on different components of a program separately instead of in its entirely.

A12 Table As described in the examples above, the use of student and personnel evaluation standards (for metaevaluation) could not only be useful, but also be necessary for program evaluation standards.