Data Analysis and Improving Student Performance

Tools for Schools

Please check your public television stations in November for a special parent night with Education Commissioner Richard Mills. He'll be discussing the tragedy.

(Music)

>> There's just a lot of pieces that have to pull together... just a lot of pieces that have to pull together... just a lot of pieces that have to pull together...

>> It's a broad field.

>> And it's relatively new to the field of public education... public education... public education...

>> Can't learn the material....

>> They are militant about this stuff, threatened by the data...threatened by the data....

>> You'll walk out of here with your head spinning... with your head spinning... with your head spinning....

>> You see these numbers and don't know what to do with the numbers. Raise your hand if that's ever happened to you.

>> Threatened by the data... threatened by the data... threatened by the data...

>> Rochelle: Hello. I'm Rochelle Cassella. Welcome to a new season of "Tools for Schools." Our programs are produced by the New York State Education Department's Broadcast Network, and they are designed to help school administrators and classroom teachers understand initiatives in education in New York and to help improve the quality of education for all students.

Business guru Tom Peters once said, "What gets measured gets done," so if we're to make changes in our districts and our classroom instruction that really impact what our students learn and how they learn it, we need to take some measurements. And as our opening montage shows, the information we can use to make these measurements is out there. But with so much data out there, how do we get it, put it together, understand it, and then make decisions based on it? Data analysis for decision making is a topic far too complex to completely cover in a one-hour television program. There are as many ways to combine facts and figures and to interpret them as there are people reviewing the information, and no one person professes to have all the answers. So it might be better to think of today's program as an introduction to data and

its uses. During the next hour, we'll take a look at the important role that analyzing data plays in making good decisions, both at the administrator and classroom teacher levels. We'll talk about the data that can be helpful and we'll see some technology that can help you get your arms around disparate data sources, and to prove that all the effort is worth it, we'll visit a Southern Tier school district that says data-driven decision making has made a significant difference in educating its students.

Comprehensive District Education Planning, CDEP, plays an important role in data analysis, and comprehensive planning, it just so happens, is one of our six tools for schools, those elements common to all successful schools, no matter their size, location, student demographics or financial situation. For more information about those tools, please check your Facilitators' Guide.

Knowing what data to look at, finding it and making it user friendly is the first step in the data analysis process. For many school district administrators and building principals, this knowledge doesn't come easily. The Summer Institute in Albany introduced a number of them to the basics, including the power and the pitfalls that data reports can contain, where to find some data and how to link data analysis to long-term planning. There was even a review of computer software programs that can help put data into user-friendly formats. The Institute used scores from the fourth and eighth grade math and E.L.A. assessments as its starting point with detailed information about how those assessments work. But take note: The assessment score should not be the sole basis for making change.

>> It's a broad field of study and it is relatively new to the field of public education.

>> It's been my mission and others who do work like myself to determine what are the important indicators that we should be looking for that help schools to determine their successes, to determine areas of weaknesses, and then once those areas are known, then they can plan accordingly and make adjustments to their instructional programs.

>> So this is an effort to identify for you, the educational audience, the users of this information, some of the best practices. This is also a major key because you're going to walk out of here with your head spinning, and you may be able to do a few of the things we're going to present, but certainly you're not going to come out of here experts in anything. But you're going to have a raised awareness level, a heightened awareness, and that is our goal.

>> In past, graduation rates have been a very critical measure, suspension rates... students' success on various Regents. But the current data is much more refined and much more precise measures, so much of my focus and the immediate needs of the school have been related to the fourth grade English language arts and math assessment and the equivalent eighth grade assessment in math and English language arts.

>> Steve Abelson, who is our CDEP facilitator, is going to speak and give you a quick overview of what root cause means.

>> CDEP, Comprehensive District Educational Planning, is a data-driven plan. If you don't have the data, you can not make a decision as to where the students are right now – how would you know where they are? -- or where you want to go. It's all based in numbers. It's based in other kinds of data: Your feelings, your hunches. What's your hunch about something?

"Well, we think that the kids are really not doing too well in E.L.A."

"Well, how do you know that, E.L.A. grade four?"

"Well, we looked at the data."

"How come they're not doing so well in E.L.A.?"

"Well, it might be that the second grade teacher went on maternity leave, and they had six other subs during the year."

Well, that's a possibility. But when we dug a little deeper in one of our districts, we found out that in that particular district, what ended up happening was the students walked to school and the culture of that district is it's okay when you walk to school if you're a little bit late, and we further found out by digging even deeper for something called the root cause, that well, maybe they showed up an hour late or an hour-and-a-half late, and the attendance rates looked pretty good but the lateness rates looked pretty bad. And we found out even more. As an "everything" teacher -- meaning an elementary teacher -- knows, typically you're going to do E.L.A. work first thing in the morning. Well, folks, if you're not there, you can't learn the material.

>> Pulling the data out as it pertains to the school building and then bringing meaning to, as I say, their successes or their weaknesses in terms of instructional program.

>> So the test scores showed our kids in district "X" aren't doing real well in E.L.A. It might have had something to do with that teacher going out on a leave or a number of subs, but it really had everything to do with the students not being there. We dug a little bit deeper.

>> Most assessments that we're used to are in the form of the Regents, for instance, where what was the most important thing was the number of correct items that the students had determined their grades. These assessments that are in both math and English language arts involve something called item response theory. It's not simply looking at how many questions did a student get right. You can't look at a raw score and give any meaning to that. It's which items did they get right? And the even more complicated part of it is the items themselves. If you got item 5, 10 and 15 right, that's more significant than if you got other items right. It's called pattern scoring.

>> What if I told you that 100% of all kids home schooled failed the E.L.A. 4? If you were a district administrator, you would go nuts: 100% of the kids are failing. Well, what did I fail to figure in my reasoning? If you had one kid. Okay? What's the big deal? One kid. It's meaningless. So that's why you have to have a sense of not only percentages, so you can compare groups by percentages, but then let's take a look at what size, what the real value is before we jump the gun and say we've got a problem with that.

>> There's a way to look at the assessment that we have been very much involved in, and that is to do a building interpretation of all the students' successes on particular items. What you come up with is called P-value in measurement,and knowing that P-value is very important because if we know that building's P-value and then we do the same thing for a much larger group of students, such as at the Questar BOCES level of 3,000 students, calculate the same P-value for that larger reference group, then you are able to see how your building did relative to the larger reference group.

>> Now, what you're seeing, folks, is that the highest level in this instance is at the BOCES level, so click on the BOCES' name, whether it's Capital Region or Questar, and then you can see all the school districts in your BOCES' region displayed.

>> It allows a building's teachers and administrators to evaluate how are we doing against the New York State standard, which is what assessment is designed to measure.

>> In the first field above the word "or," enter one school district. Below the word "or" enter another school district. You'll get both returns. It's difficult to compare, you know, to eyeball it, but that's how you would do it. You could bring back as many schools as you want by using the Boolean "or" argument.

>> You cannot make a sole determination on what a student knows and how they're performing based upon one single assessment test. It's important, but you really want to look at other authentic pieces of information.

>> What you want to do is a disaggregation on gender. So you'll find one of the boxes, towards the middle of the screen on the left side underneath the name of your district, you should have the word "gender."

>> There are all kinds of questions that school districts should be looking at but it really is questions that are specific to their needs, variables such as student demographics, attendance, discipline records, test scores... The more data you have at your fingertips, the more able you are to answer your questions.

>> In that database are your assessment scores. You can form your school's accountability performance index; you can look at Regents scores. That database is a wealth of information of the big picture. More than one measure, over years.

>> One tool in data analysis is disaggregation, where you can break things down by different population groups: For instance, males or females or ethnicity. And by doing that, that's a standard analysis you can do to make sure that performance is even across groups and you're not having certain population groups that are doing much better than others.

>> I want you to know that just by visually displaying your data and putting it in the right format and understanding what is there, you can get a lot out of that, really valuable tools. And another note: Even statisticians, heavy-duty statisticians, can't do their work without charts. You've got to see what your data looks like.

>> How do you use your data to make a difference? Well, in New York State, really they're taking a plan called the CDEP, The Comprehensive District Education Plan, and they are really trying to pull it all together, using your data to make a comprehensive plan. And by pulling your data from different sources and using your districts and all your resources together, you can answer some of the questions that come up about why you're not meeting your goals.

>> What does this say about your school, your 100 or 200 students that took this assessment, to the larger 18,000 students in Narrick, and then what does that say about the curriculum inside your school building? And that's the real significance of what Tammy is presenting. It's a way to see relative strengths and weaknesses in your building-level program on a key idea in math or an S.P.I. in the English language arts.

>> One of the steps is really to search for root cause and that's one of the fundamental steps, not to just jump to conclusions based on assumptions or maybe just one piece of data, but to really go back and ask questions about why student performance is low or why you're having discipline problems or why you're having attendance problems, and you can really take that through a problem-solving cycle regardless of the issue.

>> I would like to talk a lot about parallelism because if you see that your lines are parallel, you're seeing that your students are performing above or below but fairly consistent with the Level 3 benchmark, which means if you have a parallel line that's underneath the benchmark, your goal is to bring it up, but not necessarily to focus on any particular point. Basically what you're looking for here are discrepancies. Perhaps you might see that line being paralleled and at one point drop way down...well, you would want to investigate why.

>> If you wondered and the teachers in the group said, "Well, yes, this is because they don't come to class; the students scored poorly on E.L.A. 4 because they just don't come to class," you would have that data on your fingertips and you would say, "How did this student do poorly on E.L.A. 4? Did they attend?" And was that the case? Is that a possible explanation? You would look at the data, and in fact oftentimes, you would find that there is no difference, and that's just as important because you have taken an explanation that people have had assumptions about and you have showed them that that's not actually the case.

>> For Level 2 cutoff, it means for key idea one, that is there....

>> Do you have the benchmark for...

>> You go to the values... (Continued background conversations)

>> The information that is contained in schools is already in a variety of locations. They're referred to as disparate data sources. That is, they can't speak to one another. The Data Warehouse has the ability to massage the data and it cleans the data so that all of the information, all of the fields, components of a student record, for instance, are uniform, precise; they follow strict rules of format that gets cleansed and then gets loaded into the warehouse. And then another data source, the same process goes in, the cleansing, massaging, and then equivalent identification is made.

They're called keys, and the keys relate to one another so that we can bring in the data about student attendance, which will speak to the data about student demographics which will speak to the data in the assessments which ultimately will speak to the data on the report card or data found in the student discipline database. But still there's a tremendous amount of interpretation involved.

The computer says it on that written report; it must be true... and that's not the case at all. You must always look and interpret because you know your students; you know your school. You know what's going on. You must question the results of what comes out of the computer.

>> It's people that you're going to meet that really know computers and really know data but really don't know education. So when you're working with that, you're going to have someone who knows education and someone who knows data, and it's kind of hard to get in the middle. But if you learn a couple of basic skills, you can do these things yourself and you know what you want. I have seen people come out and do work for people and do things that aren't so wonderful. And that would be because they're great with computers and they're great with statistics, but they are not out there with you, don't know what you're doing, and don't really know what you need.

>> Eventually, I really see hope, I suppose I should say, that districts really build their capacity. They have information technology; they have data warehouses; they're able to access their data, and not only able to access their data but they're able to use it.

>> For those schools that need significant change to bring their curriculum in line with the New York State standards, this is a great process and it makes perfect sense.

>> Rochelle: It's not enough that school administrators have access to data for planning purposes. Teachers also need to make informed decisions based on data. So there was another Summer Institute, this focusing on data analysis in the classroom. For four days, facilitators led staff development personnel from across the state through a series of workshops. Here's just a portion of what they learned.

>> Thanks very much for being here with us. What we're going to do first this afternoon is just a connection to get you to see who's here, who's at your table...

>> We asked the regional support centers to use the data to become more informed, use more informed decision making, and, you know, it would be a good thing for me to learn what we're asking them to do and learn.

>> Also, the grades do not show the whole picture of the student, the whole child...

>> My experience is that people are nervous about this stuff. They're threatened by the data. And so they're a little bit shy about it. So we want to create some strategies for being able to engage everyone so that it is not threatening to them.

>> When you give teachers the tools to use to determine which kinds of data should be used in which kinds of situations, then we have really empowered teachers to use data in ways that will influence the well being of students.

>> We can use data to inform our decision making, our decision making about instruction, our decision making about curriculum, our decision making around our budgeting, our decision making about professional development. When we get this data thing right, what it does is it influences everything.

>> It's important that teachers gather information from their classrooms so they can make changes in their instruction, make changes in their curriculum, make changes in their assessment.

>> What are the conditions where we break out of our isolated path and break into a place where we can have collective learning? And then how many of you -- may I borrow this for one second? How many of you have 20 or more cool white binders on your shelves? (Laughter) See, you can come here and give me three of your precious summer days, and if this goes back and finds its way on to your shelf, I will be really, really depressed.

We are no longer allowed to make decisions by hunches and hypotheses, by the seat of our pants. Boards of Education are requiring it. We are required to in any grant application create the need, present the data, present it in a compelling manner...

>> What we have done here is we've built some resources that hopefully are designed and organized in a way that will be so friendly that you'll be able to walk out of here with materials to go back and work with small teams.

>> So there's four different kinds of data that we try and encourage people to look at. The first is student achievement data, all the data about learning. What we want to do with the student learning information is start to take a look at it by demographic group.

The third type of data that we want to access is perceptive data. And the fourth kind of data that we're looking at are school processes.

>> You each have to put down the answer in your packet.

>> Achievement tests, whether they're norm-referenced or criterion-referenced, whether they're the state test for standards, aligned with the standards... We want grades, teacher reporting, end of the unit test, weekly tests, any of the kinds of information that comes about student learning. And it's not just about their tests; it's about their demonstration of performance. It's about their authentic assessments where they're doing the practical use of information that

they have learned.

>> How many of you think you have at least 80 or 90%? Yes. All right.

>> Taking a look at formal student tests and taking a look at the way students react to learning situations, the way in which they process information, the way that they deal with unfamiliar situations and can apply information in unfamiliar settings... the ways in which they can take on new challenges as learners... all of those kinds of observations can be recorded by teachers and used in comparison with testing data.

It can be written evidence; it can be oral evidence; it can be anything that is used where a student can actually demonstrate learning and not just performance on a single test but go across times; go across different levels of instruction, integrating content areas...

>> If you begin to look at the standardized test data and you look at an itemized analysis, for example, and you look at a student score in one of his assessment tasks, say perhaps a listening, and you see a student score of a 3, that score doesn't mean quite as much unless you have the student work in front of you. Not only did we look at the assessment data, but we also looked at other student work and student portfolios, and asked ourselves the question does the student's portfolio refute or support the data from the standardized tests? So those kinds of conversations can take place because sometimes a snapshot such as a standardized test can't give us the full picture.

So now we're going to look at gender, race, social class. We're going to look at that same student achievement data, but we're going to break it out by the students with disabilities. We're looking at any of the information that we can break out based on the different configurations of kids.

High school teachers want to know things like how many hours a week are the children working outside of the school day? They want to ask questions about demographics of non-native speakers. They want to look at the whole variation of children with disabilities, the whole range. How do the kids like what we're doing in this new arts class? Is this working? What's working about it? We want the child's perception.

We're also gathering more and more perceptive data of the adults in the system, administrators and teachers. We're assessing perceptive data of parents, and we're starting to open up and say how can the parent data inform the classroom teacher? Are we open to that?

>> The school process might be the schedule; it might be the A.I.S. program, the Academic Intervention Service program. We might look at the configuration of our school for our grade level. Are we by department? Are we by cluster? What's working? What's not? How is the building configured? Are we a middle school? Are we a junior high, a 7-12? What's the organization and is it helping or hindering learning?

>> When we look at student learning, we have to understand the expectations for the learner. What is it that every child must be able to do and to know to be a successful student? Then we can begin to look at where the strengths and weaknesses are and what we have to do to get them to the next step. Sometimes it means major program decisions and sometimes it only means tweaking the resources or altering the strategies that are presented to students.

>> We are responsible for helping the State achieve its goal. Each school is responsible for helping the district; each classroom, the school; each child, the classroom. So if you build this system of a nesting, you'll see that we're all connected.

Now, what we want to do is influence this from the bottom up while we also influence it from the top down. So we don't just want to take that test but we also want to look at databased student work as information that will inform our decision-making.

>> Another way the teachers are using it is the information on their shared decision making team or a department team where they're sitting down together to do planning for what they want, what they need, what professional development they're looking for, saying, "Holy mackerel! Our kids didn't do well in that. Let's get some material so we can do this better. Let's get the material so we can do that better."

>> We can use data with students differently so we sit down with a parent and a child and say, "Here is the math and the strands or key ideas and here's how your child is doing on each of the key ideas, and here's some things..." so that the child then owns and says, "I am really good at this part, but I need some extra work on this part." So they're owning their own data. It's in their hands.

See, if students are aggregated and you keep it out of their hands, it becomes less useful. So I want to get it into the hands of the people who can do the most with it, and that's why we're here.

>> What I envision really is a culture where teachers will take student work and look at it regularly. For example, having a group of teachers look at a Regents exam or having a group of teachers look at an essay that the group of students have done say in the eleventh grade. Look at that collectively, at a variety of students' work based on that essay and then talk about what are the strengths? What are the weaknesses? What do students do well? What don't they do well? How can we address what they don't do well? Is there something in our instruction? Is there something in our curriculum? Maybe we're not assessing it well enough. So there are a lot of different kinds of issues that can come out from a conversation like that.

>> I'm seeing a shift in the way professional development is happening from this expert model, of somebody standing in front and talking at, to teams of people in a professional learning community sitting down, messing with their information, their kids, their student work, their success ratios on all sorts of instruments and assessments, and they're talking about their data in a way that's starting to influence what I do tomorrow.

>> Rochelle: The Office of Vocational and Educational Services for Individuals with Disabilities began advocating data-based decision making for the past six years, both at the state and district levels. It recommends the use of key performance indicators and CDEP as the basis of data analysis for special ed. I spoke with VESID official Fred De May.

Fred, tell me about the role that data plays in the area of special education.

>> Data is a critical part of our strategy in special education. Starting in 1996, VESID published a strategic plan which is entirely based on data and the use of data for determining whether or not we are achieving our goals and our outcomes at the state level. That naturally flows down to the school district level as well.

The key performance indicators or the data indicators that we use within special education include items like test scores on the fourth and eighth grade E.L.A. and math test, participation and performance on the various Regents examinations, the numbers and types of diplomas that are used, that are earned by students with disabilities, dropout incidents, classification rates of students with disabilities within school districts, integration in general education settings, placements in separate settings -- those would be settings outside the typical school setting -- and of minority representation in special education.

The way that that information is used at the local level is that we also have planning processes that are called Comprehensive System of Personnel Development which very much ties in with the CDEP that the Commissioner referenced earlier, the Comprehensive District Education Plans, and the purpose of the Comprehensive System of Personnel Development is to actually take a look at those key performance indicators and then make some decisions about what type of professional development is needed at the local district level.

It's important to keep in mind that professional development is a huge, huge investment on the part of local school districts, the state, even at the federal level. We're putting millions of dollars into professional development. So it's critical to make sure that that investment is being directed to identified needs within the district and that the district is putting strategies in place and employing strategies at the local level that are addressing very specific identified needs which are again based on those key performance indicators. So in the area of special education, we are driving literally all of our decision-making processes and evaluating our results based on changes in those performance indicators.

>> Rochelle: Is there something that you have available to help districts make those evaluations?

>> Yes. Yes, we have literally redirected the various networks that we are involved in. One in particular is the Special Education Training and Resource Centers, SETRC, which are located in every BOCES and in the large city school districts, and they have been trained extensively on the use of data in planning processes for both the Comprehensive District Education Plan and the Comprehensive System of Personnel Development. So they represent a very useful resource to school districts and they are available to work with school districts as they go through those planning processes.

One of the interesting things that we have seen is that as we have changed our planning processes over the past several years, and by several I mean the last five to six years, we have seen a dramatic difference in the quality of the professional development plans and the expectations for professional development plans.

School districts are becoming involved in what we call data mining, for the lack of a better term. They're taking a look at the trends that are associated with their key performance indicators and starting to ask questions that we never would have suspected three or four or five years ago. And they are questions such as why are we seeing these types of trends? What's happening with our fourth grade students or eighth grade students in math or English language arts? Why are our classification rates so high? What's happening within our district and what are contributing? What are the factors contributing to these types of classification rates?

So the districts very much are questioning the contributing factors or the root causes associated with these outcomes, and those are the things that they're going after and they're starting to show up in the plans.

We're getting phone calls, or I'm getting phone calls, and I have districts say, "Fred, we have never done this before. We have never looked at this data before." Just by virtue of putting it out there and incorporating the data into the various planning processes, we're seeing that districts are making the right types of decisions about the types of activities that they need to have in place to address their own needs. It's fascinating.

>> Rochelle: And smart ways to spend their dollars.

>> Yes.

>> Rochelle: Thank you.

About ten years ago, the Binghamton City School District began using what it calls an unsophisticated data analysis program to drive decision making. From that simple start, the district is now involved in a multilevel program, including a data-warehousing project, a software program that compares Binghamton student performance against similar school districts across the state, and a partnership to give teachers laptop computers for data collection and access in the classroom. Here's a quick look at Binghamton.

>> Good morning, everyone. Today we're going to do a lesson having to do with math word problems. Now, you know next year, I think, when you get to fourth grade, you are going to be taking a big test -- you may have heard about it -- the New York State math test. We, your teacher and I, took a look at this year's math test, at what the fourth graders did in May, and what turned out is a lot of times fourth graders and third graders are confused when they get to a word problem.

>> Our decisions for a long time have been based on the best available data that we have collected. It was based on how we began to recognize the importance of gathering information to make decisions in a community that was hard-hit with the economic downturn of the early '90s.

>> Put your finger on that and read it to yourself. Read it carefully. See if you can understand what the problem is saying.

>> It's also in Binghamton based on the understanding that the most critical aspect of instruction is the professional decision that a teacher makes or a principal makes. They are center to the instructional process and good academic achievement.

>> Are there any numbers in our problem?

>> No.

>> This is your job right now. You're going to look for what information do you need to solve the problem?

>> When we make decisions at the central office level, we perceive that our primary responsibility is to support the decisions that are made in the classrooms in the schools, and the only way that we really could begin to reallocate resources was to look at information, look at the data that suggests how we should allocate those resources and use this community's resources effectively. So that led to, I believe, a change in how we perceive and use

information.

>> You can write that on this line.

>> Write what?

>> What did you just tell me?

>> How much is a loaf of bread?

>> So you need to know how much is a loaf of bread, and if you need help spelling...

>> We were looking to evaluate programs and how much they impacted a child's intellectual development. We had the great fortune of working with some people that understood test analysis, experimental design, analysis of information, and then how we could synthesize that to improve the quality of education that our students should be provided.

>> Did I get the same amount back?

>> No.

>> What do you need to know to know how much change to get back?

>> By counting.

>> Counting what?

>> The money.

>> What money?

>> The money that your mother gave you?

>> The education of teachers and administrators, the education in fact of people who work in schools, doesn't include a great deal of background about how we use information extensively to make decisions. So, consequently, there has been in this occupation a distrust of data-driven decisions, specifically decisions that may be based on statistics. We have to ensure that everyone understands that this kind of analysis of information can be used very effectively to promote education in our school district, that it won't be used punitively, and that is a concern, that it will be used so we can understand where our students are and how we can improve the quality of their education.

>> You buy a loaf of bread for $1 and could you end up with $4?

>> No.

>> What did you end up with?

>> $2.

>> What did you do? Did you add or subtract?

>> Part of our everyday activity in terms of instruction, we talk about assessment and the need to look at data and those data to make decisions. In addition, we do provide our instructional staff with information about how they can assess and use those assessments to make decisions. We are right now in the process of purchasing laptops for our teachers who will be involved in the first phase of this assessment system. That laptop will allow teachers, whether they're working in the classroom or some office space or at home, to call up information, get that information and then analyze it.

The next phase as we implement frequent assessments throughout the school district is to provide human support so that there's someone in each building who is a good source of information for teachers who want specific information or want specific analysis and can go to someone who can get them working with central office staff or one of our consultants or BOCES or some corporate partner so that they can retrieve that information immediately.

We believe that if we put this plan in place, and we know we will, that the history we've had and the foundation of using data to make yearly decisions will then be used not just by central office staff, not just the Board of Education or the superintendent of schools or program directors, but principals and teachers working together on a daily basis making daily decisions, weekly decisions, or yearly decisions.

>> If you don't have enough you can't buy the bread.

>> That's very important to know. How many people thought you also needed to know how much money you had with you?

>> So we collect data from our tests. They are tests that are administered within the school district and statewide tests and national tests. We gather information from other sources: student mobility, student academic condition, ethnicity, other demographic information. We're gathering demographic data; we're gathering attendance; we're gathering academic achievement data, and then we're putting it together so that we get a clearer picture about where our students are. If we find, as an example, that students are having difficulty with a particular topic in a certain area, in physics that students are having difficulty with mechanics, a teacher using frequent assessments which we know are reliable, are good, are tied to the standards, they test what eventually will be tested on state and national assessments, that teacher can make immediate decisions. If the entire group has mastered a particular concept in mechanics, that teacher can move on. If a sizeable portion of the class is having difficulty or, more importantly, if individual students are having difficulties or a particular segment of the class based on characteristics are having difficulties, the teacher can make adjustments in the instruction, can provide additional resources. That's true whether it's physics or reading or writing or mathematics.

>> Please look up here. Raise your hand if you can tell me how much money should Brian take to the grocery store?

>> Gathering the information, putting it into a system that's easily retrievable is part of it, and then using that to make professional judgments is the most important part of that. The key is how we integrate all the systems that exist today into one so that's achievable, and we believe that we're very close to that. We're using E-scholar. We also use Panomation for attendance data. And then of course, Microcheck is a system that's used for our cafeteria services. That provides us with information relative to students that we also access. The teachers right now have access to the information through printouts and through their work with the principals and program directors. Eventually we hope every teacher will have access via a computer, whether it's a laptop or P.C., so they can draw those data up, and they can make an analysis or synthesize information or when someone is assigned to a building that has expertise in getting those kinds of answers, ask the questions so that a decision can be made at the classroom level.

>> 78% more kids on a free lunch than you have...

>> We had in place something called academic targeting. We would collect information and then profile our students collectively. We then in each building would set academic targets to improve the achievement of our students based on New York State tests, and we have seen significant improvements in E.L.A. and math and other achievement scores. We have looked at other information and drawn conclusions about how we can use those data to make decisions that will improve students, particularly helping students that are mobile, making curriculum decisions on textbooks based on information that we have gathered. But that has led to a reduction in the dropout rate, improvement in our average daily attendance, high achievement on state tests and overall improvement in the academic achievement of our students.

>> So I end up with $10? Can I have more money than I started with?

>> Yes -- no.

>> At a store, can you buy things and then end up with more money?

>> No.

>> So I want you to think after you solve it, "Is my answer a reasonable answer?"

>> The pitfalls are primarily in the area of the statistical analysis. One cannot draw a conclusion based on correlations. Correlations just suggest a relationship that isn't necessarily causing it. If we're going to evaluate programs, we have to ensure that the variables that we want to measure and their effectiveness on some criterion, some outcome, is the correct one, and we better make sure we control for other variables that influence those outcomes. Those pitfalls, though, can be managed and they certainly should -- they can be avoided and they shouldn't be used as an excuse not to introduce something that can be very positive.

>> Rochelle: Binghamton also provides a good example of what Fred De May

discussed: Using data to improve special education programs. The database showed Binghamton that it had a disproportionate number of classified students, so officials are now studying why that's happening and what can be done about it. They also found a high number of students with disabilities being exempted from state assessments. That changed immediately. This past year no students were exempted. The district recognizes the special education program is not perfect but points out that the data analysis has helped determine the specific areas that needed attention. So that's Data Analysis 101. Since it was a lot of information to absorb, here are a few things you need to keep in mind in the form of our tool tips...

(Music)

>> My experience is that people are nervous about this stuff. They're threatened by the data.

>> It is critical that people stop denying the importance of data and that they begin to recognize that without data we're just people with opinions.

>> There has to be an acceptance and an understanding and a belief that the more

information we have the better off we'll all be. Secondly, what we have to do as an institution is we have to provide that information so that it can be used without a lot of difficulty, that it's clear, that there's an ease of use and that it will lead to decisions.

>> I don't believe that teachers need to be statisticians, but I do believe they need to know what certain elements of data offer and what they can produce and what they can't produce.

>> It's important that teachers do this because the point of contact of where learning occurs is in that classroom with the teacher and the child. So if we say our business is teaching and learning, the workers are the teachers and the learners. So yes, I want information to be accessed and used by administrators for planning, but if I don't have access for the kids and for the teachers to use that information, I am missing a huge dimension of what's possible.

>> I really think it needs to be learned at all levels: Administrators and teachers becoming computer literate, becoming advanced in some of these basic programs that can do so much. Find out where your data is. Talk to people. Find out what you can do with it. Figure out what your questions are, what kinds of questions you want answered. And follow through with that. It's not going to be something that you're going to be able to whip together; it's going to be a process.

>> We need to develop and refine our skills in both putting that into warehouses, or spreadsheets or databases, and then extracting it in a way that will make sense to us.

>> Looking at it collectively takes some of the subjectivity out of it. Looking at it collectively focuses you more on the student and less on how the teacher is teaching. But looking at and analyzing student work helps you to understand what this student knows, what he or she is able to do or what kinds of things that -- what kinds of things are gaps in their learning and how might we address the gaps that these students have?

>> If you were to ask teachers what they need most, it's time, but I would add to that time for collaboration. All of the things we have been talking about are only able to be accomplished when there are conversations.

>> That's the focus, the standardized tests, the assessments, the Metropolitans or the Tonies or the other district tests that many districts are putting into place.

>> You cannot make a sole determination on what a student knows and how they're

performing based upon one single assessment test. It's important, but you really want to look at other authentic pieces of information such as students' report cards. If we were looking at the larger issue of the impact of poverty or free lunch, we would look at that whole group and you would look at their report cards and say, "Does this say the same thing as the assessment measure?"

>> You cannot look at school data without using multiple measures. By looking at one source of data, there's not enough reliability there to use it.

>> When we are able to compare three sources of data, one of them being a more

formalized test, another being a performance and the third perhaps being related to some smaller piece of evidence of learning like a journal or a notebook or a narrative report that the student has done, then you can get a really strong impression of how the student is learning because you're using multiple measures. You are also giving students varied opportunities to demonstrate

learning that are far richer than the single performance test.

>> It's not the technology that's going to solve the problem, but it's access to the information through the technology that will give you better information so you can use data to drive your decision. If we're going to draw conclusions about the effectiveness of particular programs, you better make sure that the analysis that we use, the statistical analysis, is sophisticated enough to support the decisions.

>> You should have your questions defined by the data you were interested in, not by the data that's available.

>> You will not have the internal resources necessary to do it in your school district. You cannot do it all. That's why you come to a specialty individual in your BOCES who may be a data analyst, but if you know it can be done by looking at data, then you are halfway there in terms of knowing what to ask for.

>> We have a couple of consultants that helped us with it, not only in terms of evaluating programs, over a yearly basis or longitudinal decisions, but evaluating our financial performance.

>> It's not a magic bullet. It's not simple, and it's not something that's going to answer all your problems.

>> It's a process where you look at your data and you form your questions; you look at other measures to validate your hypotheses and so forth. A lot of people expected it to be easier I think than it is, but it's definitely a process.

>> All of this information, all of these tools, all of this opportunity to analyze will not substitute for the professional discretion and judgment that teachers or principals make but rather to help them so that they can make more informed decisions.

(Music)

>> Rochelle: If you still have some questions or you're looking for some specific information about anything you have seen in today's program, be sure to join the Broadcast Network's on-line discussion. Opening at 7 o'clock tonight Ted Smith of Questar and Anita Murphy, Data Analysis Coordinator from Binghamton, will be available on-line to answer your questions. To join the discussion go to www.imaginetomorrow.com.

Need some additional copies of the Facilitators' Guide? Download them from our web site, www.emsc.nysed.gov/ciai/satellite.html.

While you're there, complete our on-line survey about today's program. It will take only a few minutes, but it provides us with some important data about how we're performing.