Back to EMSC Home SED Home Disclaimers and Notices

New York State Testing Program

Science Exam Development

Each Grade 4 Elementary-Level Science Test, Grade 8 Intermediate-Level Science Test, and Regents Examination in science is developed over a three-year period.  Each step in the process must be repeated annually because, unlike other states, New York State releases its written examinations to the public after they are administered.  Assessment and curriculum specialists of the New York State Education Department lead the examination development process.  Each step in the process involves New York State teachers who have experience in teaching the corresponding subject area.

First Year
The first step in the exam development process is item writing.  Teachers who are experienced item writers and teachers who are first time item writers are involved in this step of the process.  In order to become an item writer, and write items for the first time, teachers must attend a New York State Education Department sponsored item writer training session.  These sessions are usually held annually.  New item-writers are selected from the pool of teachers who complete and submit to the Education Department an on-line application titled Call for Expertise which may be found at http://www.emsc.nysed.gov/ciai/call.htm.  All interested New York State teachers are encouraged to submit an application.  Experienced and new item writers are given assignments to write multiple-choice and/or open-ended items mapped to specific standards and key ideas.  Once new item writers have successfully completed their first item writing assignment, they are eligible to take part in other test development activities.

After the new items are received at the Education Department, they go through exhaustive editing.  Separate committees consisting of New York State teachers edit the items for each content area.  Once the new items are edited, they are divided into pretest forms.  The number of pretest forms produced varies for each content area.  Each pretest form is read, edited, and checked to make sure there is no duplication of content, concepts, or process skills.  The secure pretests are shipped to a sample of New York State schools in the spring.  The pretest forms are completed by a diverse sample of students.  Schools then ship the pretests back to the Education Department for rating.

Second Year
During the summer of the second year, the multiple-choice item answer sheets are scanned electronically and committees of teachers rate the student responses to the constructed response items.  Further item editing and clarification take place at this time.  Data analysis occurs when pretest rating is completed.  Subsequently, the items are then sorted according to key ideas in the core curriculum.  Field tests are constructed from these pretested items using examination blueprints.  Each field test goes through a series of reviews involving teachers from New York State.  After these exhaustive reviews are completed, each secure field test is split into secure short forms that are distributed to a sample of New York State schools in the spring.  The field test short forms are completed by a diverse sample of students.  This sample of students is much larger than the sample of students completing the pretest forms.  Schools then ship the field tests back to the Education Department.

Third Year
The New York State Education Department contracts with a vendor who oversees the rating of the field tests for the high school examinations.  The field test rating for both the elementary and intermediate-levels is not contracted with a vendor, but the overall process and result is similar to the high school examinations.  Prior to rating the field tests, New York State teachers and Education Department personnel work with the vendor in a process known as Range Finding to review rating guides and select actual student responses that represent responses for each score level for each constructed response item.  These selected responses are used as exemplars to guide the vendor’s raters.  Using this information, the vendor rates each field test constructed response item.

When the statistical analysis of the field tests is completed, the field test results are reviewed and forms are designated as operational for one of the administrations of the State examination.  Prior to the initial administration of any content area examination, Standard Setting is completed.  During this process, New York State teachers determine the raw score points necessary to demonstrate meeting the standards and the raw score points necessary to demonstrate meeting the standards with distinction.  Prior to each administration, the examinations are vetted one final time by a team of teachers during the Final-Eyes Review.  In order to be chosen for the Final-Eyes Review team, a teacher must be new to the test development process.  This practice provides a fresh look at the exam. Teachers that form these committees are also selected from the Call for Expertise.  Each teacher on the Final-Eyes Review team completes the examination, paying particular attention to making sure that the language of each item is clear, that there is only one correct answer for each multiple-choice item, that the spelling and grammar for each item are correct, and that the layout of the exam will minimize student anxiety.

After the approximately three years of development described above, the operational form of the State examination is shipped to schools throughout the State, but the exam development process is not yet completed.  After administering the examination, teachers are asked to complete an evaluation form and return it to the Education Department.  Both the Education Department and New York State teachers who are involved in the development of future examinations review the responses on these evaluation forms.

At any given time new items, pretests, field tests, and operational tests are being developed for future examinations.  All stages of development involve New York State teachers.

Pretest and Field Test Strands
Sound procedures for pretesting and field testing are a fundamental component in building a high quality statewide testing program.  The New York State Education Department’s Office of State Assessment has developed a new pretest and field test sampling method that employs a multi-year sampling matrix.  This sampling method randomly assigns every public school in New York State, and those nonpublic schools that participate in the State’s testing program to strands. Each strand contains a representative sample of schools, based on needs-resource capacity definitions. Each examination, other than those in Languages Other Than English, is also included and the resulting matrix assigns each examination to a strand. This new sampling method will address the needs of schools in several ways: 

• It gives all schools advance notice of their pretest and field test responsibilities;

• The matrix system depends on all schools participating in accordance with the strand schedule, thus sharing the responsibility of pretesting and field testing while ensuring an adequate and representative sample; and

• To the greatest extent possible pretests and field tests have been designed for administration in one class period.  The exceptions to this are Comprehensive English, US History and Government, and Global History and Geography, which include extended response questions requiring more time.  No school will be assigned more than one of these longer field tests or pretests per year and the Education Department will continue to look at this issue.

The strand to which any high school has been assigned can be determined by using the alphabetized list that is posted at http://www.emsc.nysed.gov/osa/important-notices/importantnoticearch/sch-strand.htm(strand list).  The matrix for high schools that correlates the strand, the exam for which the pretest or field test will be administered, and the year for that pretest or field test administration is given at http://www.emsc.nysed.gov/osa/important-notices/importantnoticearch/matrix.htm (MATRIX).  The assigned strand number for each high school will remain in place through the 2009 field test period.

The elementary-level and intermediate-level pretest and field test information may be found at http://www.emsc.nysed.gov/3-8/fieldtest.htm.

Participation of all schools in all scheduled pretests and/or field tests each year is crucial. Full participation helps to guarantee that the data collected are truly representative of New York State’s student population. The State’s student population is carefully analyzed, by school building, to design pretest and field test samples that accurately reflect the diversity of students and needs in New York State.