A complete description of the examination, including test specifications and scoring rubrics, is given in a test sampler.

Data related to the performance standards for the test were obtained from a committee of experts. Judgments from committee members were quantified using standard practices employed by psychometricians who conduct standard setting studies. The committee made their judgments with respect to the difficulty scale resulting from the scaling and equating of field test items. In the field testing, each item, or score category if the item has multiple scores, is given a difficulty parameter obtained through item response methods. Test items corresponding to various points on the difficulty scale are presented as examples of test items at that difficulty level. The items used came from the anchor test form. The anchor test form is the test form upon which the cut-points are set and the form to which all later forms of the test will be equated.

Committee members were given definitions of three performance categories—not meeting standards, meeting standards, and meeting standards with distinction. The State Education Department has developed these category definitions and they are applied to all of the Regents tests that are being developed. In addition, committee members were given an exercise designed to help familiarize them with the examination and an exercise in which they were asked to categorize some of their students into the performance categories as defined by the State Education Department.

The committee met as a group on April 2, 2001 at the State Education Department.

The standard setting study test used the bookmarking approach because all the multiple choice items and constructed response item had been scaled using item response theory methods and because the bookmarking procedure enables committee members to consider these two item types together.

In the bookmarking procedure, multiple choice items and constructed response items are ordered in terms of their difficulty parameters. The purpose of the items is to illustrate the meaning of the difficulty scale at specific points. Committee members are asked to apply their judgments to these ordered items. The committee meeting is conducted in rounds. The rounds and the activities employed in each round are given below.

Committee members were also asked four overall questions about accomplishment of the learning standards and test performance. Answers to these questions might aid New York in setting appropriate performance standards on the test. These questions asked:

• Each committee member's estimate of the percentage of students in their classes who are currently meeting the learning standards.
• Each committee member's estimate of the percentage of students in their classes who are currently meeting the learning standards with distinction.
• Which was the more serious error--to categorize a student as having met the standards when, in fact, that student has not met the learning standards or to categorize a student as having not met the learning standards when, in fact, that student has met the standards?
• Which was the more serious error--to grant distinction a student who has not met the learning standards at that level or to fail to grand distinction to a student who had achieved that level of proficiency.

Committee members provided judgments relating to the performance test using the following procedure.

• Committee members reviewed the directions to the student and the scoring rubric for each item. All committee members were familiar with the rubric and most had used the rubric in scoring the performance test.
• Committee members then estimated the score that the borderline student (i.e., a student who meets the standards minimally) who meets standards would achieve on the test.
• Committee members then estimated the score that the borderline student who meets the standards with distinction would achieve on the test.
• Score distributions of the individual committee member judgments were obtained.

The New York State Education Department assembled a committee of 24 people to provide judgments for the study. Committee members were, with one exception, current or former classroom teachers. Some were supervisors. One committee member was a representative from the business community. All committee members were recognized as very knowledgeable of the learning standards pertaining to physical setting and earth science and of how students perform on standardized tests similar to the Physical Setting/Earth Science Examination. Some had worked on an aspect of either the standards or development of the tests.

Committee members, their schools, the number of years experience each has in teaching Earth Science and the number of students who are currently in their Earth Science classes are given in the table below.

Committee members were chosen so that they would represent a wide range of schools and different types of students. Each committee member was asked to complete a short background questionnaire that included questions about their sex, ethnic background, and the setting for their school. Results of the questionnaire tabulations are given in the table below.

In round 2 every committee member independently placed his or her own bookmarks for meeting standards. The results of the placements are given in the table below. The table gives the difficulty parameter, corresponding raw score, and percentage of students below that raw score based on the field test results. The cut-points include the committee average plus or minus one or two standard deviations (i.e., standard deviations of the committee estimates) and the median committee cut-point including the cut-points corresponding to the 75^th and 25^th percentile ranks of committee estimates.

In round 3 every committee member independently placed his or her own bookmarks for meeting standards with distinction. The results of the placements are given in the table below. The table gives the difficulty parameter, corresponding raw score, and the percentage above that raw score based on the field test results. The cut-points include the committee average plus or minus one or two standard deviations (i.e., standard deviations of the committee estimates) and the median committee cut-point including the cut-points corresponding to the 75^th and 25^th percentile ranks of committee estimates.

In round four, committee members received a report of their round two results. They also were placed in small groups where individual results were discussed. After the discussion, committee members were asked to place another bookmark for meeting standards based on the information and knowledge they had gained up to this point. The round four results, which generally show less variation than the round two results, are given in the table below.

In round five, committee members received a report of their round three results. They also were placed in small groups where individual results were discussed. After the discussion, committee members were asked to place another bookmark for meeting standards with distinction based on the information and knowledge they had gained up to this point. The round five results, which generally show less variation than the round three results, are given in the table below.

In round six, committee members received a report of their round four and five judgments. They also received a report of the impact of their estimates from that round. Impact was reported in terms of the frequency distributions of the field test scores. The committee was also advised that scores from field testing generally underestimate operational test performance, but that the amount of the underestimate was not known. Committee members then returned to their groups and discussed the report and their judgments. At the end of the discussion, committee members were asked to place new bookmarks for both meeting standards and meeting standards with distinction based on the information and knowledge they had at that time. Results of this final placement are given in the table below.