H&G
The University of the State of
THE STATE EDUCATION DEPARTMENT
The general procedures to be followed in administering Regents Examinations
are provided in the publications Directions for Administering Regents
Examinations (DET 541), and Regents
Examinations, Regents Competency Tests, and Proficiency Examinations: School
Administrator’s Manual, 2001 Edition. Copies of the Directions are
shipped to schools prior to each Regents Examination period and
may also be accessed on the Department’s web site at: http://www.emsc.nysed.gov/osa/hsgen.html.
The School Administrator’s Manual may be accessed on the Department’s
web site at: http://www.emsc.nysed.gov/osa/hsinfogen/hsinfogenarch/sam2001.pdf.
Questions about general
administration procedures for Regents Examinations should be directed to
the Office of State Assessment at 518-474-8220 or 518-474-5902. For information
about the rating of the Regents
Examinations in Global History and Geography and United States History and
Government, contact Gary Warren, Donna Merlau, Greg Wilsey, or Patricia Polan
of the Office of State Assessment
at 518-474-3860, or JoAnn Larson or Lawrence Paska of the Office of Curriculum,
Instruction and Instructional Technology at 518-474-5922.
School administrators should print or photocopy this information booklet and distribute copies to all school personnel who will be scoring these examinations.
The Scoring Key and Rating Guide
contains:
· Correct answers to the multiple-choice questions.
· Specific scoring rubrics for both the thematic
(Part II) and document-based question (DBQ) essays (Part III B) and the DBQ
scaffold (open-ended) questions (Part III A).
· Prescored anchor papers at each essay score
level, with commentary explaining why a particular student paper was awarded
that specific score.
· Prescored practice papers.
The reliability of the scores is a fundamental concern in the
measurement of the student’s
achievement. Therefore, each essay must be scored by at least two qualified
teachers. The short-answer document-based scaffold questions need only be
scored by one qualified teacher. Qualified raters include teachers of Grades
7-12 social studies and special education teachers who are knowledgeable about
the Global History and Geography or United States History and Government
curriculum. Raters should have previously received some school-level,
district-level, or regional training on scoring social studies essays or
scaffold (open-ended) questions as part of the turnkey training process.
It is recommended that schools with a small number of qualified social studies raters form a consortium of teachers to score as a group the answer papers from several schools.
In order to ensure reliable scoring, the principal of each high school administering the social studies Regents Examinations must appoint a scoring coordinator who will:
· Manage the training and logistics of the scoring
process.
· Provide task-specific training, including review
of the rating guide just prior to scoring.
·
Assign two
teachers to rate each essay response independently, with a third teacher
available to resolve discrepant scores. (A discrepant score is one that varies
by more than one credit on a 5‑credit rubric.) Only one rater is needed
for the scaffold questions. If staffing is sufficient, separate teams of
teachers should rate the Part II thematic essay, the Part III A scaffold
questions, and the Part III B essay.
Every effort should be made to avoid having a teacher rate his or her own students’ responses. When this is not feasible, a teacher should score no more than one part of his or her students’ paper (i.e., a thematic essay, the scaffold questions, or a DBQ essay).
Before student responses can be read and rated, each school must set up a procedure for collecting, arranging, and processing the answer papers and for maintaining records of the examination results. The procedure used in a particular school should be designed to produce a reliable score for each student and to facilitate maintenance of the school’s records of each student’s score. A suggested procedure for managing the mechanics of the rating process is described on pages HG-3 and HG-4.
Multiple-choice questions may be either hand scored or machine scored. When hand scoring, indicate by means of a check mark each incorrect or omitted answer to multiple-choice questions on the designated answer sheet. Do not place a check mark beside a correct answer. Use only red ink or red pencil. In the appropriate space on the student’s answer sheet, record the number of multiple-choice questions the student answered correctly.
Machine-scorable answer sheets must be provided and scored by the school. Answer sheets supplied by the school must provide the same number of response options as are given in the examination questions, and the choices must be labeled 1, 2, 3, 4, not A, B, C, D. Instructions for using the answer sheets must be developed locally and provided to the proctors administering the examinations.
Before answer sheets can be machine scored, several samples must be both machine and manually scored to ensure the accuracy of the machine-scoring process. All discrepancies must be rectified before student answer sheets are machine scored. When machine scoring is completed, a sample of the scored answer sheets must be scored manually to verify the accuracy of the machine-scoring process.
In training raters to score student responses for Part II and Part III of the examination, follow the procedures outlined below:
1. Introduction to the Task
The introduction to the task may take place once
the administration of the examination has begun. However, the actual Scoring Key and Rating Guide for this examination
may not be removed
from the shrink-wrapped package of scoring keys for use by raters until after
the Uniform Statewide Admission Deadline has passed.
a. Raters
read the task.
b. Raters
identify answers to the task.
c. Raters
discuss possible answers and summarize expectations for student responses.
2. Introduction to the Specific Rubric and
Anchor Papers
The introduction to the specific rubric and
anchor papers may take place once the Uniform Statewide Admission Deadline has
passed.
a. Trainer
leads review of specific rubric with reference to the task.
b. Trainer leads discussion of procedures for
assigning holistic scores (i.e., by matching evidence from the response to the
rubric).
c. Trainer
leads review of each anchor paper and commentary.
3. Practice Scoring Individually
a. Raters
score the practice papers independently without looking at the scores and
commentaries provided after the papers.
b. Trainer
records scores and leads discussion of scoring criteria until raters feel
confident enough to move on to actual scoring.
c. If
additional practice is required to reach scoring consensus, trainer may use a
sample of student answer papers from the current administration of the
examination.
The following procedure is recommended for managing the mechanics of the rating process. A copy of the rating sheet and the record sheet are included in the Appendix. You may make as many photocopies as are needed.
1.
The person
assigned as the coordinator of the rating process, or other designated
representative(s), will be responsible for coordinating the movement of papers,
calculating a final score for each student’s essays, recording that information
on the student’s Part I answer sheet, and determining the student’s final score
for the examination.
2.
Set aside
one room as a central rating room for collecting, sorting, circulating, and
storing answer sheets/essay booklets and for preparing and maintaining records
for these examinations.
3.
Provide a
suitable location for rating of essays.
4.
Allow time
to provide training for scoring the specific task for all raters immediately
before the rating of the students’ responses (about 2 hours per essay and about
30-45 minutes per document). It is strongly recommended that teachers be
trained on one document and score those responses, then be trained and score
the responses to the next document, etc.
5.
Provide
adequate time for rating (3-5 minutes per response for each essay, 1 minute per
response
for each scaffold question scored 0-2, and ½ minute per response for each
scaffold question
scored 0-1).
For
Part II and Part III B, continue with these procedures:
6.
Each essay must be scored by at least two
qualified teachers. For each essay question, divide raters into two-person
teams. Designate one team member as Rater 1 and the other as Rater 2. After the
examination has been administered, either keep the essay booklets together and
shift them between raters or separate the students’ essay booklets into Part II
and Part III B. If the essay answer
booklet is separated, be sure to verify that the student has entered his or her
name and the school name on the page where Part III (Document-Based Question)
begins before separating the parts. After separating the essays, staple
together all pages of the student’s Part II response and staple together all
pages of the student’s Part III B response.
7.
Arrange the
essay responses for each part according to a sequence, using whatever order is
most convenient for your school, e.g., class period, alphabetical, or local
identification number. Beginning with the first paper in the sequence, enter
each student’s name on a copy of the record sheet. (Master for duplicating
appears in the Appendix.)
8.
Divide each
group of essays into bundles of 25-30 papers.
9.
Prepare a
rating sheet for each bundle. (See sample rating sheet in the Appendix.) After
recording the students’ names on the rating sheet, photocopy the rating sheet.
Each rater will need a separate rating sheet for each bundle of 25 essay papers
he or she rates. The second rater must not be aware of the score
assigned by the previous rater.
10.
Distribute
the bundles of essay papers to the rating teams, making sure that each rating
team receives two rating sheets for each bundle of papers. Each rater on a team
should rate one of the bundles and record his or her ratings on one of the
rating sheets. The two raters should then exchange bundles. The second rater
should only record his or her scores on the second rating sheet. No scores or corrections should be indicated
on the essay papers.
11.
After each
team has completed rating a bundle, the team should return those answer papers
to the central rating room. Remove the rating sheets completed by each rater
from the bundles and enter the scores on the record sheet. Make sure there are
two independent ratings for each response. Enter the resolved scores in the
appropriate columns on the record sheet.
12.
Review the
two scores for each student to determine if the student’s scores for that essay
are discrepant, i.e., a difference of more than one credit between the two
scores. Separate the students’ responses with discrepant scores and make
another bundle. Prepare a separate rating sheet for those discrepant papers.
List the names of the students on a new rating sheet and attach the sheet to
the corresponding bundle of student responses. Assign each of these bundles to
a rater to obtain a third independent rating of the students’ responses. Make
sure that the third rater is not one of the original two raters of that task
and that the third rater has undergone the training for scoring that task.
13.
After the
necessary third ratings have been obtained, remove the rating sheets from the
bundles of student responses and determine the resolved scores by using the
method for resolving discrepant scores described on the next page. Enter the
resolved scores in the appropriate columns on the record sheet.
14.
Transfer the
resolved scores to the appropriate spaces on the students’ Part I answer
sheets.
15.
All rating
sheets and record sheets used in scoring the social studies Regents Examinations
must be kept for at least one year by the school where they were administered.
For
Part III A:
1.
Follow a
similar procedure for processing the papers.
2.
The
short-answer (open-ended) questions need only be scored by one qualified
teacher.
3.
The scores
for each scaffold question may be recorded in the student’s examination
booklet.
4.
Record the
total Part III A score in the space provided on the student’s Part I answer
sheet.
Method for
Determining the Score for Each Essay
Two
Ratings:
1. Compare the two ratings.
2. If the two ratings agree, the student
receives that score.
3. If the two ratings are contiguous, average
the two scores.
4. If the two ratings are not contiguous, a
third rating is necessary.
Three
Ratings:
1. Compare the three ratings.
2. If two of the three ratings agree, the
student receives that score.
3. If the three ratings are different, the
student receives the middle score.
Examples:
|
Rater
1 |
Rater
2 |
Rater 3
|
Resolved
Score*
|
Reason
|
|
2 |
2 |
— |
2 |
Two
ratings agree. Use that score. |
|
2 |
3 |
— |
2.5 |
Two
ratings are contiguous. Average the two scores. |
|
2 |
4 |
4 |
4 |
Two
ratings are more than one point apart. Third rating is done. Two of the three
ratings agree. Use that score. |
|
2 |
4 |
3 |
3 |
Two
ratings are more than one point apart. Third rating is done. Three ratings
differ. Use the middle score. |
|
0 |
1 |
— |
0.5 |
Two
ratings are contiguous. Average the two scores. |
* If the final score ends in .5, do not
round at this point.
Entering Essay Scores on the Record Sheet
The examples below show how students’ scores should be recorded on the record sheet.
|
Student’s Name |
Part II Essay Scores
|
Part III B Essay Scores |
||||||
|
Rater 1 |
Rater 2 |
Rater 3 |
Resolved Score |
Rater 1 |
Rater 2 |
Rater 3 |
Resolved Score |
|
|
Student A |
4 |
4 |
— |
4 |
4 |
2 |
3 |
3 |
Student B
|
0 |
1 |
— |
0.5 |
1 |
4 |
2 |
2 |
|
Student C |
3 |
5 |
4 |
4 |
4 |
3 |
— |
3.5 |
Determining the Student’s Final Examination Score
Part I Score ______ Part III A Score ______