Report offers framework for evaluating D.C. school reform efforts, along with first impressions

A new report from the National Research Council offers a framework for evaluating the effects of a 2007 reform law on the District of Columbia's public schools. The evaluation program must include systematic yearly public reporting of key data as well as in-depth studies of high-priority issues, such as teacher recruitment and retention, the report says. As part of the evaluation program, the Mayor's Office should produce an annual report to the city on the status of the public schools, including an analysis of trends and all the underlying data. The evaluation program should be structured to provide ongoing, independent monitoring and feedback on a number of key aspects of D.C.'s public education system, which are described by the committee that wrote the report. "Although the immediate goal for the District is to evaluate the changes enacted in 2007, we also see an opportunity to build a continuing program of analyses that will be useful regardless of future political or personnel changes," said Christopher Edley, committee co-chair and dean of the School of Law at the University of California, Berkeley.

In 2007 the Public Education Reform Amendment Act (PERAA) shifted control of D.C. Public Schools (DCPS) from an elected school board to the mayor and created the position of chancellor, among other changes. The act also mandated an independent five-year evaluation to determine whether enough progress has been achieved to warrant continuation of the new structure. The D.C. City Council asked the Research Council to provide guidance on how to structure the evaluation and to offer initial feedback about implementation of PERAA, to the extent possible.

Report Offers First Impressions, With Cautions

The report offers some first impressions of school reform under PERAA and notes that the city and DCPS have made a good-faith effort to implement the required changes, but the committee cautions that it is premature to draw general conclusions about the reforms' effectiveness at this time.

Data suggest that a modest improvement in student test scores has continued since PERAA was enacted. However, the report stresses that scores on their own cannot identify the reasons for that improvement. Further examination is necessary to determine whether they can be linked to the reforms or are the result of other factors, such as demographic changes in D.C.'s highly mobile student population. Also, because the rising scores are averages, further analysis and monitoring are needed to determine how various subgroups of students -- such as particular grade levels or students in particular areas -- are faring; some could be making gains while others are not.

The report emphasizes that student test scores should not be the only factor used to judge improvement in the school system. A comprehensive evaluation should address the full range of district responsibilities: quality of teachers, principals, and other personnel; quality of classroom teaching and learning; capacity to serve vulnerable children and youth; promotion of family and community engagement; and quality and equity of operations, management, and facilities.

Although the committee found that the administrative and governance changes required by PERAA have generally been carried out, the report notes that the Office of the Ombudsman was defunded three years after its creation. Since the office was intended to be the primary channel for parents to communicate with school officials and seek redress for complaints, its absence is significant, said the committee.

Evaluation Program Should Be Long-Term, Independent

The report offers a framework for evaluating key elements of reform under PERAA that affect student outcomes -- including management structures and roles and conditions for learning -- as well as the student outcomes themselves.

A set of specific, measurable indicators should be established as soon as possible, building on suitable indicators already collected in the District, the report says. These would track how well the city's public schools are doing, how well the key strategies the city has chosen to improve education are working, the conditions for student learning, and the capacity of the system to produce valued outcomes.

In addition, in-depth studies should be designed to provide deeper analysis of specific questions about high-priority issues, including both the targets of PERAA and long-standing challenges in District schools, such as special education. These studies can answer questions about the causes of trends seen in indicator data, explore potential reasons for disappointing outcomes, provide information to help alter existing strategies, or help explain why some strategies appear to be working better in some schools than in others. They are key, the report notes, because they are the means by which evaluators can answer policymakers' questions about the effects of particular policies and practices.

To develop and sustain an infrastructure for ongoing research and evaluation of its public schools, D.C. will need to engage local universities, philanthropic organizations, and other institutions, the report says. The program must be independent of school and city leaders and responsive to the needs of all stakeholders, and its research should meet the highest standards for technical quality. There is no one model for the city to follow, but examples of research and evaluation programs that have been created in other cities -- such as Chicago, New York City, and Baltimore -- could be useful as D.C. develops its own program, the report notes.

"An independent evaluation program that can be an ongoing source of objective information and analysis will be an invaluable resource for the city," said committee co-chair Robert Hauser, executive director of the National Research Council's Division of Behavioral and Social Sciences and Education. "There are no quick answers. Education reform itself is a long-term process, and the evaluation of its outcomes also has to be seen in the long term."

The committee will discuss the report at a public briefing to be held at 2 p.m. on Sunday, March 6, at the National Academies' Keck Center, 500 Fifth St., N.W., Washington, D.C. A live audio webcast of the briefing will be available at http://national-academies.org.

Source: National Academy of Sciences