E

The University of the State of New York
THE STATE EDUCATION DEPARTMENT
Albany, New York 12234

Information Booklet for Scoring
the Regents Comprehensive Examination in English

GENERAL INFORMATION

The general procedures to be followed in administering Regents Examinations are provided in the publications Directions for Administering Regents Examinations (DET 541), and Regents Examinations, Regents Competency Tests, and Proficiency Examinations: School Administrator’s Manual, 2001 Edition. Copies of the Directions are shipped to schools prior to each Regents Examination period and
may also be accessed on the Department’s web site at: http://www.emsc.nysed.gov/osa/hsgen.html.
The School Administrator’s Manual may be accessed on the Department’s web site at: http://www.emsc.nysed.gov/osa/hsinfogen/hsinfogenarch/sam2001.pdf.

Questions about general administration procedures for Regents Examinations should be directed to the Office of State Assessment at 518-474-8220 or 518-474-5902. For information about the rating of the Regents Comprehensive Examination in English, contact Marguerite Pileggi in the Office of State Assessment at 518-474-5912.

School administrators should print or photocopy this information booklet and distribute copies to all school personnel who will be scoring this examination.

SCORING THE EXAMINATION

The Scoring Key and Rating Guide

The Scoring Key and Rating Guide contains:

· Correct answers to the multiple-choice questions

· Scoring rubrics

· Two to three prescored anchor papers at each score level, ranging from high to low, with commentary

· Five prescored practice papers

Rating the Examination

The reliability of the scores is a fundamental concern in the measurement of the student’s achievement. Therefore, each part of the examination must be scored by at least two teachers. Qualified raters include teachers of English, reading, English as a second language, and special education who know the English curriculum and have received training. These raters should have previously received training in rating tasks in the test sampler draft as part of the turnkey training process that began in August 1998.

In order to ensure reliable scoring, the principal of each high school administering the Regents Comprehensive Examination in English must appoint a scoring coordinator who will:

· Manage the training and logistics of the scoring process.

· Provide task-specific training immediately before scoring.

· Assign two teachers to rate each task independently, with a third teacher available to resolve discrepant scores.

Organizing the Rating and Recording

Before student responses can be read and rated, each school must set up a procedure for collecting, arranging, and processing the answer papers and for maintaining records of the examination results. The procedure used in a particular school should be designed to produce a reliable score for each student and to facilitate maintenance of the school’s records of each student’s score. A suggested procedure for managing the mechanics of the rating process is described on pages E-3 and E-4.

Scoring of Multiple-Choice Questions

Multiple-choice questions may be either hand scored or machine scored. When hand scoring, indicate by means of a check mark each incorrect or omitted answer to multiple-choice questions on the designated answer sheet for the appropriate session. Do not place a check mark beside a correct answer. Use only red ink or red pencil. In the box provided for each part, record the number of questions the student answered correctly for that part. Transfer the number of correct answers for each part of the multiple-choice questions to the appropriate spaces in the box in the upper right corner of each student’s Session One answer sheet.

Machine-scorable answer sheets must be provided and scored by the school. A separate sheet must be used for each session of the examination; students may not use the same answer sheet for both sessions. Answer sheets supplied by the school must provide the same number of response options as are given in the examination questions, and the choices must be labeled 1, 2, 3, 4, not A, B, C, D. Instructions for using the answer sheets must be developed locally and provided to the proctors administering the examination.

Before answer sheets can be machine scored, several samples must be both machine and manually scored to ensure the accuracy of the machine-scoring process. All discrepancies must be rectified before student answer sheets are machine scored. When machine scoring is completed, a sample of the scored answer sheets must be scored manually to verify the accuracy of the machine-scoring process.

Detailed Directions for Training Raters to Score Student Responses

In training raters to score student responses for each part of the examination, follow the procedures outlined below:

Introduction to the Task

The introduction to the task may take place once the administration of the session has begun. However, any use of the actual Scoring Key and Rating Guide for the session may not begin until after the Uniform Statewide Admission Deadline.

· Raters read the task and summarize its purpose, audience, and format.

· Raters read passage(s) and plan a response to the task.

· Raters share response plans and summarize expectations for student responses.

Introduction to the Rubric and Anchor Papers

· Trainer reviews rubric with reference to the task. At this point, raters should also be directed to the additional scoring considerations printed at the bottom of each rubric. For all parts of the examination, if a student writes only a personal response and makes no reference to the text(s), the response can be scored no higher than a 1. Responses totally unrelated to the topic, illegible, incoherent, or blank should be given a 0. A response totally copied from the text(s) with no original student writing should be scored as a 0. Additionally, for all parts except Session One, Part A (listening task), a student response that addresses only one document can be scored no higher than a 3.

· Trainer reviews procedures for assigning holistic scores (i.e., by matching evidence from the response to the language of the rubric and by weighing all qualities equally).

· Trainer leads review of each anchor paper and commentary. (Note: Anchor papers are ordered from high to low within each score level.)

Practice Scoring Individually

· Raters score a set of five practice papers individually. Raters should score the five papers independently without looking at the scores provided after the five papers.

· Trainer records scores and leads discussion until raters feel comfortable enough to move on to actual scoring.

Suggested Rating Procedure

The following procedure is recommended for managing the mechanics of the rating process. The Rating Sheet and the Record Sheet are included in the Appendix. You may photocopy as many copies as needed.

1. Designate one person as the coordinator of the rating process. The scoring coordinator will be responsible for coordinating the movement of papers, calculating a final score for each student’s essays, recording that information on the student’s answer paper for Session One, and determining the student’s final score for the examination.

2. Set aside one room as a central rating room for collecting, sorting, circulating, and storing answer sheets/essay booklets and for preparing and maintaining records.

3. Provide a suitable location for the rating of essays.

4. Allow time to provide training for scoring the specific task for all raters immediately before the rating of the students’ responses (about two hours per task).

5. Provide adequate time for rating (3-4 minutes per response for one rating).

6. Divide raters into two-person teams for each part. Designate one team member as Rater 1 and the other as Rater 2. Make every effort to avoid having a teacher rate his or her own students’ responses.

7. After the session has been administered, separate the essay booklets into Part A and Part B. Before separating the parts, be sure to verify that the student has entered his or her name and the school name and has circled the correct session on each page of the essay booklet. After separating the parts, staple together all of the pages of the student’s Part A response. Staple together all of the pages of the student’s Part B response.

8. Arrange the essay responses for each part according to a sequence, using whatever order is most convenient, e.g., class period, alphabetical, or local identification number. Beginning with the first paper in the sequence, enter each student’s name on a copy of the Record Sheet. (Master for duplicating appears in the Appendix.)

9. Divide each group of essays into bundles of 25-30 papers.

10. Prepare a rating sheet for each bundle. (See sample rating sheet in the Appendix.) After recording the students’ names on the Rating Sheet, photocopy the Rating Sheet. Each rater will need a separate rating sheet for each bundle of essay papers he or she rates.

11. Distribute the bundles of essay papers to the rating teams, making sure that each rating team receives two rating sheets for each bundle. Each rater on a team should rate one of the bundles and record his or her ratings on one of the rating sheets. The two raters should exchange bundles. The second rater should record his or her scores on the second rating sheet. No scores or corrections should be indicated on the essay papers.

12. After each team has completed rating a bundle, the team should return those answer papers to the central rating room. Remove the rating sheets completed by each rater from the bundles and enter the scores on the Record Sheet. Make sure each response has two independent ratings. Enter the resolved scores in the appropriate columns on the Record Sheet.

13. Review the two scores for each student to determine if the student’s scores for that part are discrepant, i.e., a difference of two or more points between the two scores. Separate the students’ responses with discrepant scores and make another bundle. Prepare a separate rating sheet for those discrepant papers. List the names of the students on a new rating sheet and attach the sheet to the corresponding bundle of student responses. Assign each of these bundles to a rater to obtain a third independent rating of the students’ responses. Make sure that the third rater is not one of the original two raters of that response.

14. After all the necessary third ratings have been obtained, remove the rating sheets from the bundles of student responses and determine the resolved scores by using the method for resolving discrepant scores described on page E-5. Enter the resolved scores in the appropriate columns on the Record Sheet.

15. Repeat this process for all parts of the examination.

16. Transfer the resolved scores to the appropriate spaces on the students’ Session One answer sheets.

Method for Determining the Score for Each Essay

Two Ratings:

1. Compare the two ratings.

2. If the two ratings agree, the student receives that score.

3. If the two ratings are contiguous, average the two scores.

4. If the two ratings are not contiguous, a third rating is necessary.

Three Ratings:

1. Compare the three ratings.

2. If two of the three ratings agree, the student receives that score.

3. If the three ratings are different, the student receives the middle score.

Examples:

Rater 1	Rater 2	Rater 3	Resolved Score*	Reason
2	2	—	2	Two ratings agree. Use that score.
2	3	—	2.5	Two ratings are contiguous. Average the two scores.
2	4	4	4	Two ratings are two or more points apart. Third rating is done. Two of the three ratings agree. Use that score.
2	5	4	4	Two ratings are two or more points apart. Third rating is done. Three ratings differ. Use the middle score.
0	1	—	0.5	Two ratings are contiguous. Average the two scores.

* If the final score ends in .5, do not round at this point.

Entering Essay Scores on the Record Sheet

The examples below show how students’ scores should be recorded on the Record Sheet.

Name	Session One Essay A Scores				Session One Essay B Scores				Session Two Essay A Scores				Session Two Essay B Scores
	Rater 1	Rater 2	Rater 3	Resolved Score	Rater 1	Rater 2	Rater 3	Resolved Score	Rater 1	Rater 2	Rater 3	Resolved Score	Rater 1	Rater 2	Rater 3	Resolved Score
Student A	4	4	—	4	4	6	5	5	4	3	—	3.5	4	2	5	4
Student B	0	1	—	.5	1	3	2	2	1	1	—	1	1	4	2	2

Determining the Student’s Final Examination Score