Study Shows How Data can be Transformed into Improved Education Practices - February 11, 2008

A research team at The University of Texas at Austin’s College of Education may have found a workable way to translate school districts’ data into improved teaching and learning – and it could be as basic as not getting the cart before the horse.

Dr. Jeff Wayman

For the past 10 years, school districts around the U.S. have become ever more adept at collecting data on students, but relatively few have had success at implementing that same data to enhance education practices at all levels throughout the district. Dr. Jeff Wayman, an assistant professor in the Department of Educational Administration, along with graduate students Vincent Cho and Mary T. Johnston, recently conducted a comprehensive data-use evaluation of a Wyoming school district. The process as well as the results could serve as a model for others who aim to be “data-informed districts.”

“Several issues can thwart a data-rich school district that wants to use its data in an efficient, meaningful way,” says Wayman. “First, teachers, principals and other school staff may not be adequately trained in how to access the data and transform the numbers and information into better practices. That problem is exacerbated by the fact that, although the body of research on best practices and processes is growing, there’s still much that’s not understood.

“Second, the computer systems used by a school district may not be integrated or user-friendly – they may not be able to ‘talk’ to one another and share information, and there may be inaccuracies in the actual data itself. Third, and perhaps most important, is that many districts first buy a computer system and then ask, ‘What will this system let us do?’ rather than asking, ‘What do we want a system to do for us?’ before they purchase the system.”

According to Wayman, Natrona Consolidated School District (NCSD) in Wyoming was rather unique in being willing to enlist a third party to perform an initial, thorough evaluation of district-wide data use before any further data decisions or purchases were made. The district was willing to assess the big picture before moving forward, which meant asking fundamental questions such as “What does learning mean to our district?”, “How will we know learning when it’s happening?” and “How will we make changes based on what we observe about student learning?”. This desire to begin at the beginning, engage all stakeholders and build a very clear vision of how data will be used is key to a data-informed district’s ultimate success, states Wayman.

The University of Texas at Austin team was able to study NCSD’s school and district culture, types of data that were used at all levels by all staff, ways data were used, structures and supports for using data and the technology used for accessing the data.

In the course of their evaluation, Wayman and his colleagues interviewed 128 individuals district-wide, from principals and teachers to bus drivers and board members. They also conducted focus groups and administered a district-wide online survey to which they received 435 responses. The team worked with a very broad definition of data, one that went well beyond student achievement test scores and included valuable sources such as teachers’ reported observations and student demographic information.

Although the evaluation was conducted at one school district in Wyoming, Wayman is optimistic about many of his team’s recommendations being transferable, or scalable, to most any school district that wants to improve data use practices.

Like many schools, NCSD faces a dropout problem – around 32 percent of its students leave school before graduation - and would like to use student data to craft more effective interventions. Like many teachers, some NCSD instructors report that they can’t directly access student data as they need it, and often are not sure how to use it once they do have it. Like other parents and community leaders around the nation, those in Casper, WY, want to know what sorts of data are collected and if young learners are benefiting from its use.

“A data-informed district is one that uses data to inform decisions at every level,” says Wayman, “and NCSD was visionary in the way it handled this process. Unlike in business, for example, the district employees were not going to lose their jobs if they didn’t improve data collection or buy an appropriate computer system. They just were conscientious and forward-thinking and elected on their own to be thorough, ask the right questions at the right time and make sure everyone was on the same page as far as goals.”

After evaluating facilitators and roadblocks to effective data use, the research team gave NCSD a detailed, 20-page summary and set of recommendations. They suggested that:

the district establish a clear vision for teaching, learning and data use
the district obtain an integrated computer system that will facilitate data use
accurate student demographic data and learning outcomes be readily available to interested individuals outside the district and that education be provided in what the data mean
district staff receive substantial training in data use, with the goal of reaching a point where they are able to translate data into better practices

NCSD already has begun implementation of Wayman’s recommendations, and his survey findings received an enthusiastic reception when he recently presented them in a keynote address to leaders of the five largest school districts in New York.

Further analyses of the NCSD data are being conducted. In addition to being reviewed by Wayman, the survey findings will serve as material for four different dissertations on the data-informed district. Wayman also is beginning a new project, unrelated to the NCSD study, in which he will be examining how Title I schools use data.

“One uses the word ‘data’ and sometimes listeners’ eyes glaze over,” says Wayman, “but this all is simply about helping students and helping teachers. Teachers want to do their jobs well and they are eager to gain tools that make success possible. If they have accurate, thorough information about their students and know how to turn that data into solutions to problems and into better instruction, then a really basic, significant victory has been won.”

—Kay Randall, University Communications, 512-232-3910