RFP #08-035: Development of State Assessments in Elementary and Intermediate (Middle) Level English Language Arts (ELA) and Mathematics
Questions & Answers (Received as of 6/15/09)
Question #1: Are the passages already developed, reviewed, and approved?
No, NYSED will not have passages selected or questions developed for the 2010 field tests. The contractor is expected to select passages and develop items for the 2010 field tests in a very compressed schedule. NYSED will work very cooperatively with the contractor to efficiently review and approve passages and items.
Question #2: Have the items been written, reviewed, and approved?
No. See response to question #1.
Question #3: Is the current contractor developing passages for review prior to November 1, 2009?
No. See response to question #1.
Question #4: Is the current contractor developing test items for review prior to November 1, 2009?
No, the current contractor is not developing test items for review prior to November 1, 2009.
Question #5: If the current contractor is not developing passages and/or items, is the NYSED developing or obtaining passages and items through some other means prior to November 1, 2009?
No. It is expected the new contractor will develop passages and items. NYSED will work very cooperatively with the contractor to efficiently review and approve passages and items.
Question #6: Will the current contractor have any role in the development of passages and items for the winter 2010 field test? If so, what will that role be?
No, the current contractor does not have any role in the development of passages and items for the winter 2010 field tests.
Question #7: Given that committee members must be contacted two months prior to the review sessions, have committee members been contacted regarding reviews of these passages, items, and/or forms, or will they be prior to the start of this contract? If these contacts have been or will be made prior to November 2009, what are the meeting dates?
Ordinarily, NYSED provides a list of teacher names and contact information to the contractor for review sessions and the contractor initiates the contacts. However, given the compressed schedule for development of the 2010 ELA field tests, NYSED will contact the teachers directly. Meeting dates will be negotiated with the contractor that is awarded the contract.
Question #8: Is it necessary to obtain copyright permission prior to development of items? If so, what is the NYSED’s suggested timeline for passage development and review and securing of copyright permissions?
Given the compressed timeline for development of the 2010 field tests, the contractor is expected to secure copyright permission in accordance with a mutually agreed upon timeline.
Question #9: If no passages or items are currently available and if the NYSED has no plans to have the current contractor or anyone else develop those passages and items, would the NYSED consider postponing the winter 2010 field test to 2011 and the 2011 operational test to 2012?
No, it is not feasible to postpone the winter 2010 field tests to 2011. However, it is feasible to postpone the 2010 field tests by at least one month. Bidders should plan for administering field tests within one month of the date specified in the RFP.
Question #10: Would it be possible to begin a contract in August or September 2009, rather than in November?
No. A contract with the successful bidder could not be fully executed prior to November 1, 2009.
Question #11: Do the items and passages to be included on the 2010 ELA (and Mathematics) Field Test already exist, and merely need to be shepherded through the committees and updated in accordance with state and committee directives? If so, can NYSED share with Vendors the file format and layout of the items within the item bank that the winning vendor will be expected to use to create the 2010 Field Tests?
No, items and passages do not already exist. See response to question #1.
Question #12: Or, is NYSED requiring the Vendor to perform full item development in the month of November 2009 for committee review and final inclusion on the 2010 Field Tests?
Yes, the vendor will be required to perform all item development for the 2010 field tests. NYSED will work very cooperatively with the contractor to efficiently review and approve passages and items.
Question #13: Although there is no provision in the RFP for delays in performance as a result of a force majeure event, does the NYSED acknowledge that the Contractor will not be held liable in any way for failure to perform any provision of the Contract if such failure is caused by any event that is beyond the control of the Contractor; and should such acts or events occur, the Contractor will use commercially reasonable efforts to overcome the difficulties and to resume work as soon as reasonably possible?
Yes.
Question #14: Will the awarded bidder be required to store test materials and data (e.g.: used and unused field test books, answer documents, manuals, student data files, etc) for some length of time? Please provide specific retention requirements so that bidders may price accordingly.
The vendor must retain student responses to field tested open-ended questions, (either on paper or as scanned images), all field test data, and all operational test data through the end of the contract.
Question #15: May multiple-choice items be used on the Listening test? If so, may multiple-choice items be used at all grade levels on the Listening test?
No. All items associated with the Listening selections must be open-ended.
Question #16: May commissioned passages be used on the Listening test?
Yes. The Listening passages may be commissioned. However, approximatel70% of the passages (listening and reading) on each operational test must be authentic literature.
Question #17: May a paired-passage grouping be comprised of a permissioned passage and a commissioned passage?
Yes.
Question #18: Does NYSED provide a student pre-id file to the vendor? If so, when will this be provided to the vendor in 2009?
In preparation for field testing, the vendor will be provided with pre-id files from RICs and Big 5 city school district scanning centers in accordance with a timeline to be mutually agreed upon by NYSED and the vendor. The vendor must provide a secure FTP site to allow for the transfer of student information. This transfer will occur in November or December of 2009 and again in 2010.
Question #19: Does NYSED require the vendor to pre-slug the Field Test documents, or will schools affix a pre-ID label provided by the vendor on the Field Test booklets?
The vendor may propose to either pre-slug the field test documents or have schools affix the labels provided by the vendor.
Question #20: What is the requirement for the production of Field Test booklets? Is it expected that the Field Test booklets will be consumable, or does NYSED require the Field Test to be administered with a separate scannable answer document? If a separate answer document is required, does the student answer the open-ended and multiple-choice items in this separate answer document?
Vendors may propose to use field test booklets that are consumable or use scannable answer documents or a combination of both. Regardless of the format used, the vendor must be able to collect both open-ended and multiple-choice responses.
Question #21: (RFP page 14 and page 81, Appendix G. Regarding voluntary/assigned field test participation) Can NYSED clarify whether the Field Test is mandatory or voluntary in nature. If voluntary, can NYSED share with the vendors the 2008/2009 participation rates?
All schools selected in the matrix (Attachment G of the RFP) are expected to participate in the field testing ofthe Grades 3-8 Tests. Participation rates tend to be about 70%.
Question #22: (RFP page 15, item 19) The RFP requires matching field test results to operational test results. What matching criteria are used to accomplish this—student identification number, name and date of birth within school, other? How is this matching information made available to the vendor for the field test?
The matching criteria are generally the BEDS Code and a local student ID number. For field testing, data files containing student IDs and other demographic information, including BEDS Code, will be provided to the vendor by RICs and Big 5 city school district scanning centers. For the operational tests, the same information is provided to the vendor by NYSED through its Student Information Repository System.
Question #23: (RFP page 15: “Field tests must be matched by student to their operational test results to permit common scaling of all field test items to the operational scale.” RFP page 18-19: “The field test forms can be equated using the equivalent group and the common item design. Bidders must develop one (or multiple) secure anchor form(s) that can be administered in the field test every year and the items would not be included for operational tests. The field test forms and item parameters must be placed onto the operational scale through the secure anchor form which is a miniature version of the operational test with regard to the test contents and statistical specifications.”) Question: Should the linking be conducted through matching between field test and operational test; or should the linking be conducted using the anchor items/forms? If the linking needs to be conducted through anchor items, do the anchor items need to stay in an intact anchor form? Is there a secure anchor form that is currently being used?
A two-step equating procedure can be used. First, all mini field test forms are placed on the same scale via anchor forms (pre-equating). Second, in post-equating, Test Characteristics Curve (TCC) methods can be used to put the two results (field test versus operational test) on the same scale. Anchor forms are currently being used.
Question 24a: Where -- in the process of administering Day 1 and Day 2 field testing in so-called Book A, Book B, or Book C schools -- has the anchor item set been administered, if at all?
24b. Is the anchor set Day 2 of each field test book? 24c. Is the anchor set Day 2 of one of the field test books? If so, which one? 24d. Is the anchor set a separate booklet that is anonymously spiralled into the mix of Books A, B, and C? For example, does the anchor set have a cover that says "Book A" or "Book B," but is otherwise different within? 24e. If the anchor set is not administered in Books A, B, or C, then how and where is it administered? 24f. How many items are in the anchor set administered during field testing? 24g. How many reading passages are associated with the anchor set in ELA? 24h. Are we to assume that all of the anchor set items are multiple-choice items?
24a. The anchor set has been administered in Part 2 of Books A, B, and C.
24b. Yes.
24c. No.
24d. The anchor set is not currently spiraled, but will be for the new contract.
24e. The anchor set will be placed in one of the books A, B or C each year. It will be placed in a different book each year.
24f. Approximately 40 items per grade.
24g. Approximately 6 or 7 passages per grade.
24h. Anchor items are currently all multiple-choice.
Question #25: (RFP Page 20, Local Scoring of Operational Tests, #3) Would NYSED be willing to provide samples of the audio/visual training for the ELA test items to bidders? Does the current offering of audio file for the mathematics accompanied by separate pdf-type document images satisfy NYSED’s desires for audio/visual presentation, or does NYSED desire a more integrated audio/visual approach?
Yes, vendors may request samples of the audio/visual training for ELA by submitting an e-mail to the RFP mailbox. The offering of an audio file accompanied by a separate pdf-type document satisfies NYSED’s requirements for audio/visual presentation.
Question #26: (RFP, Page 20, bullet 5:“The Spanish edition will be assembled from the items field tested in Spanish” for mathematics test.”) The 2007 NYS 3-8 mathematics technical report (Page 44, Table 10), indicates the n count for students taking the Spanish version of the mathematics test was around 3,000 per grade level. This compares to approximately 200,000 students taking the English version of the test. If the testing population taking the Spanish version is approximately 3,000 per grade across the entire New York State, the majority, if not all, of these students will need to participate in the mathematics field testing, even though their school may be participating in an ELA field test. The n count per item will likely be considerably lower than 2,000. Do all the items on the Spanish mathematics test have to be field tested in Spanish? If so, would NYSED clarify how mathematics field testing in Spanish is conducted?
All multiple-choice items must be field tested in Spanish. The vendor must translate the English language multiple-choice items into Spanish and provide a method for those schools selected for field testing in mathematics to order Spanish versions if necessary. Schools are not specifically selected for populations using the Spanish version. The vendor is not required to translate open-ended items for field testing. During the administration of open-ended mathematics field test forms, those students using a Spanish version will be administered multiple-choice items in Spanish. The n count listed in the technical report is for operational tests only. The n count for students using the Spanish version of the field tests tends to be low.
Question #27: (RFP page 24 regarding security of test materials) NYSED requires secure materials to be shipped via locked Regents boxes. Does NYSED require the use of locked Regents boxes to ship FT materials from vendor to schools for test administration or only secure material shipments from vendor to NYSED and from vendor to committee meeting sites?
No. NYSED does not require vendors to ship field test materials to schools in locked Regents boxes.
Question #28: (RFP pages 59-73 regarding field testing accommodation materials) There is no mention of Braille field test materials for ELA field testing, while Braille versions are indicated for the mathematics field tests. Is this omission purposeful or inadvertent? In light of the short time frame for initial field test development, would a photocopy-type expansion for the field tests, rather than rebuilding books using larger font sizes, be acceptable?
For the 2010 field tests, the vendor must provide a braille field test for mathematics only. For 2011, the vendor must provide a braille field test for both ELA and mathematics. For large-type 2010 field tests photocopy-type enlargement is acceptable except for those mathematics questions which require ruler measurements.
Question #29: (page 2, Mandatory Requirements #12: “Teachers must be reimbursed for their lodging, meals, and travel at the approved NY State rates.”) What does column “First and Last Day (75% of M and IE)” stand for on your rates Web Link? Specifically in regards to “First and Last Day”
It stands for, "75% of Meals and Incidental Expense." Vendors should not use this last column of the chart for purposes of this RFP. Vendors are only to cover the costs of any applicable breakfast or dinner.
Below is a breakdown of breakfast and dinner expenses for Albany based on the daily Meals & Incidental Expense Rate:
MEAL ALLOWANCE BREAKDOWN
Meal Allowance Breakfast Dinner $49.00 $10.00 $39.00
Question #30: (page 2, Mandatory Requirements #12: “Teachers must be reimbursed for their lodging, meals, and travel at the approved NY State rates.”) Is the vendor required to provide reimbursement for lodging and meals the night before the start of the meetings?
Only when the teacher is required to travel more than 35 miles to the meeting.
Question #31: (page 2, Mandatory Requirements #12: “Teachers must be reimbursed for their lodging, meals, and travel at the approved NY State rates.”) What is the minimum one-way mileage to meetings that will qualify a participant for staying in a hotel room?
Over 35 miles.
Question #32: (page 8, 10, Test Development: “Item Bank delivered on a yearly basis through some mutually agreed upon procedure.”) What are the specifications for the item bank and in what format does the NYSDE expect the Item Bank deliverable?
The specifications and format will be determined by mutual agreement after the awarding of the contract.
Question #33: (page 11, Reading Passages # 5: “For the second year …following the test administration.”) Please clarify the intent of this sentence
Prior to a passage being used in the 2011 field tests, the vendor must obtain the permission of the author or copyright holder for the passage to be posted on the NYSED web site following its inclusion on the operational tests.
Question #34: (page 15, FT and Research for ELA and Mathematics Tests # 12: “The vendor must…after the due date and again after the vendor has made multiple, concerted efforts…”) Please define “multiple”.
The vendor must make a minimum of three attempts.
Question #35: (page 19, Operational Forms Preparation #2, Materials to be produced shall include all print ready test booklets, including…”) Please define “print ready”.
The vendor must provide NYSED with final proof hard copy and high-resolution pdf format files with all fonts embedded in the pdf file. The pdf must be in black plate with no colors and no lower than 600dpi (dots per inch) and no art compression. In addition, the vendor must submit the final QuarkXpress file of the examination and all related art files (either in Freehand MX or Illustrator CS2 or higher) to NYSED. The vendor must use Mac version of QuarkXpress 6.5 or 7 (OS X) to typeset the examination. All pages must be single sided and numbered (except the cover).
Question #35: (page 19, Operational Forms Preparation #2, Same paragraph as above, last sentence.) Please clarify the term “any other examination-related materials necessary for valid administration.
At this time, we are not anticipating that there will be any examination-related materials other than those listed on page 19. However, in the event that by mutual agreement it is determined that other materials are needed, these must be provided to NYSED print ready.
Question #36: (page 25, Subcontracting Limit: “Subcontracting will be limited to thirty percent (30%) of the annual contract budget.”) Is the 10% MWBE requirement in addition to the 30% limit?
No. The 10% MWBE requirement is included in the 30% limit.
Question #37: (page 25, Subcontracting Limit: “Subcontracting is defined as non-employee direct personal services and related incidental expenses, including travel.”) If the prime contractor uses vendors in its day-to-day operations (such as FedEx, UPS, etc) are they considered a vendor or a subcontractor for the purposes of this RFP?
Only subcontractors that provide direct personal services are subject to the 30% subcontracting limit. FedEx and UPS would not be considered subcontractors.
Question #38: (25, Subcontracting Limit: “Subcontracting is defined as non-employee direct personal services and related incidental expenses, including travel.”) Are copyright permission fees paid to passage authors(such as FedEx, UPS, etc) considered a vendor or a subcontractor expense for the purposes of this RFP?
See response to question #26.
Question #39: (Test Construction/Research) Is the vendor required to produce a vertical scale in the FT test books?
No, the vendor does not have to continue the vertical scaling research study that is a component of the current contract for the Grades 3-8 Tests.
Question #40: Can Vendors use items that have been previously used in other assessments in or outside of New York? Or must they be developed specifically for NYSED.
Vendors may propose to use previously used items provided those items are not published on any web site or in any textbook or review book. However, those items used must be aligned to New York State standards and must become the property of NYSED.
Question #41: Do all items need to meet New York state standards prior to the first item review?
Yes. All items need to meet New York State standards prior to the first item review.
Question #42: Considering the constrained timelines, does NYSED guaranty that the passage reviews, teacher Item reviews and NYSED approvals will be completed on the vendors schedule even if they fall over the holidays?
NYSED will work very cooperatively with the vendor to facilitate the efficient review and approval of items and passages.
Question #43: (page 13, Question Formats for Math Tests # 1: “The mathematics Tests contain questions in the following formats: multiple choice, short response, and extended response.”) In the event that both multiple choice and open-ended questions cannot be developed as scheduled, will NYSED consider administering tests of only multiple choice questions?
No. Items must be developed according to the test formats described in Attachment J of the RFP.
Question #44: (page 22, Project Manager Duties, #3: “The vendor will not perform work on the project until NYSED has approved the vendor’s project plan.”) How long will it take NYSED to approve the vendor’s project plan after November 1st start date?
NYSED will work very cooperatively with the vendor to review and approve the vendor’s project plan. The vendor receiving the award letter for this RFP may submit a project plan to NYSED for review and approval prior to November 1. However, no work may begin until the contract has received final approval from the Office of the NYS Comptroller.
Question #45: (page 26, Consultant Staff Changes: “The Contractor will maintain continuity of the Consultant Team staff throughout the course of the contract. All changes in staff will be subject to NYSED approval. The replacement Consultant(s) with comparable skills will be provided at the same or lower rate.”) Please define Consultant Team staff.
Consultant Team Staff are any employees of the contractor or subcontractor that will be providing direct personal services under this contract.