SED Home
Disclaimers and Notices
|
|
Back to EMSC Home SED Home Disclaimers and Notices |
The following is a
REQUEST FOR INFORMATION
For
Printing, Scanning and Electronic Scoring
For the New York State Testing Program Grades 3 – 8
July 30, 2004
Vendors should submit questions regarding this
request via e-mail no later than the close of business August 23, 2004. All
questions and answers will be posted on the Office of Assessment website by
September 3, 2004. All responses to this RFI must be received by 5:00
p.m. EST on September 17, 2004. Please provide three copies of each
proposal.
The State Education Department
Office of Elementary, Middle, Secondary and Continuing Education
Office for Standards, Assessment and Reporting
89 Washington Avenue
Albany, New York 12234
REQUEST FOR INFORMATION
July 30, 2004
Introduction
The New York State Education Department (SED), Office of Standards, Assessment and Reporting, is requesting the following information from prospective vendors regarding the contractual services described below. The New York State Testing Program is in the process of expanding its Grades 4 and 8 English Language Arts (ELA) and Mathematics testing program. The federal No Child Left Behind (NCLB) Act requires testing in Grades 3 through 8 beginning with the 2005-06 school year. This addition will require a change in how SED and the schools in New York State operate regarding test printing, administration and scoring.
This new testing program would require all ELA and Mathematics test booklets administered in Grades 3 through 8 to New York State students to be printed on scanable paper. SED would send camera-ready tests to the vendor to be printed. The vendor would print the tests on scannable paper in the quantities supplied by SED and ship the tests to schools throughout New York State to locations supplied by SED. Schools would administer tests to the students. The students would write the answers to all multiple-choice and written constructed-response questions in the testing booklet. These booklets would then be shipped back to the vendor, or an alternate site, for scanning into a software program. New York State has Regional Information Centers (RICs) with scanning capabilities with which the vendor may coordinate, if desired. The vendor would then electronically distribute student test booklets to a regional scoring center at various locations throughout the State. These locations could be RICs or other locations established by the vendor. Individuals, many of whom would be New York State teachers, would then electronically score these tests. These individuals would have to be trained by the vendor to score these tests using the necessary hardware and software. The vendor would also have to develop a system to audit a sample of the scorers’ work to ensure accuracy. Lastly, the vendor would forward the data with each student’s scores to the designated regional center.
The vendor should provide information on their abilities to perform all the contractual services described above. However, the services described could possibly be to split into two separate RFPs, one for Printing/Scanning and one for the Distributive Scoring. Please comment on your capabilities on the following information:
· What are the technical specifications for printing and scanning hardware, software, and paper to be sure that we have accurate images of student responses;
· What are the advantages and disadvantages of distributed compared with centralized scanning;
· Is it feasible to put this project in place on the required scale in 2005-06;
· Since students will have multiple test books (one per day), what are the advantages and disadvantages of scanning booklets with multiple-choice items as well as booklets with constructed- and extended-response items;
· What kind of network connections and computer hardware will scorers need; is it feasible for scorers to sign in and score from home; and
· What other states or assessment organizations are doing distributed scoring that you have developed on their behalf.
Below is a more detailed description of each service required:
I. Printing and Shipping
The New York State Education Department is considering issuing an RFP for web-based distributed scoring of student responses to English Language Arts (ELA) and Mathematics assessments to be administered in Grades 3 through 8 beginning with the 2005-06 school year. The test booklets would be printed on scannable paper and students would record their responses in the test booklets. Each subject will administer six tests per year, one test for each grade in Grades 3 through 8 for a total of 12 tests per year. These tests are administered in schools throughout New York State at approximately 4,500 locations. The total number of test booklets per year will be 15 for ELA and 13 for Mathematics. ELA booklets are a combination of multiple-choice questions, short constructed-response questions, extended-response questions (in Grades 4, 6, and 8), and passage-editing questions (in Grades 3, 5, and 7). All six Mathematics assessments will include constructed-response items and extended-response items of approximately 40-60 words in length. Mathematics has six booklets that are solely multiple-choice questions (approximately 25) and seven booklets with constructed-response and extended-response questions. About 250,000 students in each grade 3 through 8 (a total of 1.5 million students) will take the ELA and the mathematics assessments. Given this information, the vendor would be printing approximately 3 million tests (7 million booklets) per year. The ELA assessments will be administered in January and the mathematics assessments in March of each school year starting in 2006 and continuing through 2009. The tests must be scanned and scored within eight to ten weeks from the date of administration.
Maintaining security of this test material is of paramount concern during printing, storage, shipping, and delivery. The vendor will have to meet SED test security standards and will be subject to periodic review to ensure that those standards are satisfied.
II. Scanning
After test administration, the test booklets will be sent to one or more designated sites for scanning. Scanning these booklets may be a vital component of this process. Scanning these booklets in a timely manner would require the vendor to process the job quickly and accurately. SED has the responsibility to report student scores in a timely manner to the field. As the schools ship their test booklets back to the vendor, the vendor would need to develop a process to scan these booklets and download this data into a software system that would begin to organize and distribute it to the scoring centers. New York State currently uses Regional Information Centers (RICs) to scan the multiple-choice answer sheets for the Grade 4 and Grade 8 tests. The RICs’ capabilities are limited for a job of this size. The vendor could propose a system to scan these booklets using the RICs as scanning sites. This may require additional manpower and equipment.
Distributive scoring is an electronic mechanism for delivering images of student responses to written test items to sites around the state for electronic scoring. This system requires the vendor to consider capacity, cost, hardware and software. SED is interested in this system due to increased security, speed of scoring and score reporting, and flexibility. This scoring process should provide a system that is more reliable and accurate as well as a system that monitors the scoring process. This process will also decrease the amount of paper trails and storage issues.
New York State has used New York teachers in the scoring process since the inception of the current Grades 4 and 8 ELA and Mathematics tests in 1998. This activity as also provided an opportunity for professional development throughout the State for teachers. We would like your submission to include the use of NYS teachers as scorers and/or auditors. Due to the volume of this scoring process, other scorers may need to be used due to the amount of time teachers would need to be out of their districts.
After test administration, the test booklets would be sent to one or more designated sites for scanning. This proposal should include the development of software to store and distribute the scanned student responses to scorers statewide. The software must perform the following functions:
· Store in a file layout approved by SED an accurate visual image of each student’s response to each test item associated with the following data: district of responsibility, school of attendance, unique ID, scanning center, test form and item, scorer, and scorer’s rating.
· Provide web-based interfaces for scoring administration, scorer registration, training, and scoring.
· Provide an authorization and authentication system for test scorers to read and score responses using a web browser. This authorization system will ensure that only authorized scorers and administrators can access the system.
· Provide a scoring administration interface allowing a scoring administrator in each public school district, charter school, and nonpublic school to establish scorer accounts with authorization codes, and provide parameters for those scorers to ensure they are assigned only appropriate test items.
· Include a system that tracks the number of papers submitted by each district and the items assigned to scorers in each district and ensures that each district scores a proportional share of student responses at each grade level.
· Provide scoring administrators at the State, regional, and district level access to a screen showing the number of responses by grade and item that the district was required to score at each grade level and the number of responses actually scored. The district scoring administrator would use this information to assign scorers and ensure that the district completed scoring by the deadline.
· Assign authorized scorers an identification number and a particular test item to score based on registration information, such as name, district, grade levels and subjects taught.
· Provide on-line training in scoring the assigned item to each scorer. The scorer would not receive actual student responses for scoring until they reached a designated level of proficiency in scoring training responses. Scorers would previously have received general training in scoring.
· Randomly assign to each scorer student responses from the statewide database for the assigned item. The scorer would see only the student’s response and would have no other information about the student. After scoring, the scorer’s rating and identification number would be added to the data associated with the student’s response to that item.
· Automatically and randomly select 10 percent of responses to each item to be scored by two scorers to provide a measure of scoring reliability.
· Return a data file with each student’s scores to the designated regional center.
In order to monitor the reliability of scoring among the teachers who score the student responses, approximately 10 percent of the student papers are selected and submitted to a second group of raters provided by a contractor for a second read. The 10 percent of the test papers are selected through a stratified random selection process using Needs/Resource category as the strata. Schools in all six Needs/Resource groups (New York City, Big Four, Urban/Suburban High Need, Rural High Need, Average Need, and Low Need) should be proportionally represented in the rescoring sample. Special attention should be taken to include schools in all Big Five Cities (New York City, Buffalo, Rochester, Syracuse, and Yonkers) in the sample for the inter-rater reliability analysis.
Contact for questions and/or requests for clarification:
Vendors should submit questions regarding this request via e-mail no later than the close of business August 23, 2004. All questions and answers will be posted on the Office of Assessment website by September 3, 2004. All responses to this RFI must be received by 5:00 p.m. EST on September 17, 2004. Please provide three copies of each proposal.
Send to: Mr. William McQuay
New York State Education Department
Office of Assessment, Room 767 EBA
89 Washington Avenue
Albany, New York 12234
Note: Responses to this RFI will be reviewed by the Department for informational purposes only, and will not be considered as an offer to be accepted by the Department to form a binding contract. Please do not provide cost information, as this could exclude you from consideration in an RFP. Advertisements of any subsequent competitive solicitation that may result from this RFI will result in a RFP. Failure to respond to this RFI will not exclude you from bidding on an RFP.