Because of the improved estimation enabled by HLM, including the use of maximum likelihood and empirical Bayes estimates, interpretation of statistical results can be broadened to include a larger p-value associated with statistical tests. Furthermore, statistical results with p-values at or near 0. Students' fall and spring scores are presented in Table 7. We began by running a two-level HLM analysis investigating the relationship between students' spring fluency scores and school effectiveness factors See Table 8.
This analysis was based on data from students across all 11 schools including the students from the three schools that were in the study during both Years 1 and 2. Across all schools, the mean school fluency score was One way of gauging the influence of collaborative leadership is to note that for every additional point scored on the collaborative leadership scale, a school's mean fluency score showed an increase of If we note that students increased their scores by an average of 20 words correct per minute per year see Table 7 and that school scores on the collaborative leadership scale ranged from 1.
Across all schools, the mean school writing score was 1. After including fall scores, HLM analyses investigating the relationship between school effectiveness scores or subscores e. However, one of these schools had been with the project for 2 years, so 4 cohorts of grade 6 students were included in the data analyses.
As reported earlier, all teachers in each building had completed a two-part self-study questionnaire during Year 2. The first part of the questionnaire dealt with teacher's perceptions of various building- and classroom-level factors within their school; the second part dealt with their opinions of the school's needs in regards to its change efforts. In the first part of the questionnaire, teachers rated each of 38 items on a scale from 1 strongly disagree to 5 strongly agree. These items dealt with school change, climate, and leadership; professional development; schoolwide decisions about reading instruction; classroom reading instruction; reading interventions for struggling readers, and school-home-community connections.
The school leadership ratings from our interviews were positively related to 12 of the 38 items from this first part of the questionnaire. The following is a list of the attributes which teachers saw as salient in cases where they perceived their schools to have strong leadership.
The questionnaire items which were positively related to the interviews' school leadership ratings were:. We ran two-level HLM analyses investigating the relationship between students' spring comprehension, fluency, and writing scores and the school reform effort score for the total sample of students in the 11 project schools with data from three schools included separately across Years 1 and 2.
HLM analyses investigating the relationship between the reform effort score and students' spring reading comprehension, fluency, or writing scores after including the relevant fall scores were not significant. Using our coding of the 10 components of reform implementation, we were able to take a look at which schools were successfully implementing various factors.
On the whole, we found that schools were having an easier time holding weekly study group meetings than they were holding monthly large-group meetings to share information across study groups and deal with schoolwide reform issues. Generally, schools were having an easier time meeting in grade-level groups than in cross-grade groups.
Finally, schools were having an easier time reflecting on instruction and student work in study groups than they were focusing on research-based topics for periods of 3 months or longer. Most schools had not yet turned to the reform component of working with parents as partners see Table Percent of schools demonstrating this reform variable. Sticking with substantive topics for months or more. Meeting once a month as a whole faculty to share, etc.
In this section, we highlight results from our classroom observations as useful data in their own right, irrespective of school and independent of their relationship to student growth.
In a sense, the data capture the nature of classroom instruction in schools like the ones in which we spent a year observing teachers and testing children. We report results for grades K, 1, , and Descriptive data on classroom practices are presented in Table They also provide us with an opportunity to compare what was going on in these schools with the practices we observed two years earlier in our study of high-poverty, high-performing schools Taylor et al.
Across all grades, whole-group instruction was coded more often than was small-group instruction except for grade 1, in which small-group was coded as often as whole-group instruction. In contrast, in our earlier study of primary grade reading instruction in schools that were beating the odds Taylor et al. Not surprisingly, word-level activities during reading were observed more in grades K-1 than in or Comprehension skill instruction and comprehension strategy instruction were seldom observed.
These findings are similar to those from our study of primary grade reading instruction in schools beating the odds Taylor et al. The findings on word skill activities also suggest that teachers are focusing on phonics instruction in kindergarten and first grade, which is compatible with the recommendation of the National Reading Panel Report that "phonics instruction taught early proved much more effective than phonics instruction introduced after first grade" section 2, p.
A relatively small amount of higher-level questioning or writing related to stories read was observed across all grades. Lower-level questioning was coded more often. A similarly low incidence of higher-level questioning was found in our earlier study Taylor et al. However, in that study we did find that teachers in the most effective schools were more frequently observed asking higher-level questions than teachers in the moderately effective and least effective schools.
Informational text was seldom a part of the lessons we observed at any grade level. In contrast, narrative text was coded much more frequently. Teacher and student actions. Overall, teachers were observed using a teacher-directed stance i. In our earlier study Taylor et al. Across all grades, students in the present study were coded as more often engaged in passive responding than in active responding.
Passive responding, which included reading turn-taking e. In Year 2 of the study, teachers kept logs for two weeks one in the winter and one in the spring , in order to document all literacy instruction in their classrooms. Students in full-day kindergarten classrooms averaged approximately minutes per day of literacy instruction and activities. In grades the average increased to minutes per day. Grade 4 students averaged minutes per day and grade 5 students minutes per day.
Generally, students spent about twice as much time in whole- or large-group instruction as they did in small-group instruction. Students across grades spent about 25 minutes per day in independent reading. Students in kindergarten and grade 1 averaged between 21 and 26 minutes per day of phonics or phonemic awareness instruction; this tapered off to an average of 16 minutes per day in grade 2, and 9 minutes per day in grade 3.
In grades students only averaged from 10 to 15 minutes per day of comprehension skill or strategy instruction. The means for the various literacy activities are shown in Table Eight of 11 schools were in their first year of the reform during Year 2 of the project. We had made a decision at the beginning of this project not to look at changes in instruction during a school year with only three observations per teacher.
Therefore, before turning to an examination of changes in teaching in the three schools which were in their second year of the project, we decided to take a look at the relationships between teacher practices during literacy instruction and students' reading and writing growth at various grade levels, irrespective of school, to see what could be learned about effective reading instruction practices. As was the case for Year 1 data from schools which started their participation in Year 2, these analyses provided useful Year 2 data as the schools began Year 3.
HLM analyses were conducted on the relationships between teacher practices and each of the major outcome variables: Grade 1 was analyzed separately from grades since different fall scores e. Grades and were analyzed separately in order to look for patterns in teaching practices, which might affect differences in reading growth between the primary and intermediate grades.
Kindergarten was analyzed separately because different pre-test and post-test variables were used. Thirty-five percent of the between-teacher variance was accounted for by the variables of higher-level questioning and small-group instruction. Students' mean spring fluency score was In grades , students' mean spring fluency score was Students' mean fluency score was Twenty-six percent of the between-teacher variance was accounted for by the variable of higher-level questioning.
Students' mean spring comprehension NCE score was In grades students' mean spring comprehension NCE score was Students' mean comprehension NCE score was Eighty-three percent of the between-teacher variance was accounted for by the variables of comprehension strategies instruction and student-support stance negatively related.
Students' mean spring writing score was 2. In grades , students' mean spring writing score was 1. Students' mean writing score was 2.
Students' mean phonemic awareness score in spring was 5. Students' mean spring concepts of print score was 6. Students' mean spring word dictation score was Students' mean rhyme score in spring was 5. However, no classroom observations variables contributed significantly to this between-classroom variance.
Summary of 2-level HLM classroom results. A relatively high level of phonics instruction was not found to be helpful for students' growth in fluency in grades , or to their phonemic awareness development in kindergarten; but phonemic awareness instruction was found to be related to students' spring emergent literacy scores in kindergarten.
Coaching in word recognition strategies during reading was related to students' growth in reading fluency in grades Comprehension strategy instruction was related to spring writing scores in grade 1, and to emergent literacy scores in kindergarten. A highly teacher-directed stance towards instruction was not found to be beneficial to students' reading growth in grades , whereas active responding was found to be beneficial to growth in reading fluency in grades A high-level student support stance e.
Small-group instruction was found to be beneficial in kindergarten and grade 1, whereas whole- or large-group instruction was found to be beneficial in grades To better explain the findings related to classroom factors, we provide descriptions of teachers who aptly illustrate the practices identified by the quantitative analyses as positive.
We also provide examples from classrooms in which a heavily teacher-directed stance was apparent, in order to better describe a practice which was identified as less helpful to students' literacy growth. Below, we provide illustrations from the field notes, with direct quotes from teachers and students in italics.
The HLM analyses found that children grew more in comprehension and fluency when their teachers were coded as asking more higher-level questions than other teachers. Teachers who were more often observed teaching their students in small groups in first grade also had students who showed larger gains in fluency during the year.
The teaching of comprehension strategies was found to be related to greater growth in writing in first grade children. Hernandez 5 , a first-grade teacher in the study, exemplifies many of these relationships in her classroom. She uses small-group instruction extensively. While she is with one group, the other students are in centers, where activities include writing words and word families, math, computer, library, and reading the room.
Hernandez focuses heavily on comprehension strategies and higher-level thinking as she teacher her first-graders. On one day, her students are reading a story. She introduces a GO chart with columns that are labeled "prediction," "vocabulary," "understanding," "interpretation," "connections," and "retelling," and prompts students to complete information on the chart.
Prediction--"I think the story is about She asks them to check in their books to look for challenging vocabulary that they think they should add to the chart. Hernandez asks for a word that has the same meaning as "house. Another word for woods? Hernandez refers to the "Understanding--I noticed" column of the chart.
A student suggests, "The giant does interesting things. Hernandez refers to the "Interpretation--I wonder" column. She encourages the students to think about what happens next, to go beyond the story, to imagine what the characters could do together. What connections can you make, what is the main thing you learned in the story? Hernandez asks, "What maps can you use to help you retell the story? Hernandez asks how and why the class could use each one of these devices.
One student suggests that they could use a bubble map to describe the character. On another day, Ms. Hernandez is working with a small group studying informational texts. She refers to the GO Chart. She asks them to quickly review the steps without looking at the chart.
Gleason, also a first-grade teacher, uses whole-group instruction about one third of the time and small-group activities about two thirds of the time. Gleason often engages her first-grade students in higher-level thinking.
On one day she has her students working on dog reports. They have collected information from multiple sources and are putting similar notes together. Writing the report requires higher-level thinking, as the children write down ideas that go together and follow an organizational pattern.
On a different day, after the students have read George shrinks, the small group is making a list of things they do at school--math, drawing, journal writing, eating lunch, special activities, feeding the bunny. Gleason asks the students to think about how they would do these things if they were small. Kou says, "We could break the pencil and just write with the point.
Gleason asks, "How could we drink milk? Gleason has students write, "If I were small, I'd need a toolkit to help me. While the children are writing, Mrs. Gleason circulates through the class, helping individual students with their ideas and coaching them with spelling words. Metcalf primarily teaches through small group instruction, but unlike Ms. Gleason, she typically does a lot of the work for her children, illustrating a highly teacher-directed stance towards instruction.
When she reviews the "magic e" rule with a small group, she says, "Today we'll review the silent e. When she introduces new vocabulary words before reading a story, one of the words is "donkey.
Metcalf is reading a story to the children. As they discuss the story, she interjects her own ideas and summarizes for the group. Metcalf confirms this correct response, offering a comparison with the need to wrap fragile items when packing.
Metcalf also stops to repeat important points and clarify meanings, but she misses another opportunity to have the children do the talking, "So when you see a dinosaur skeleton in a museum, it probably has a few bones that are not originals, since some might have been missing, or broken.
Fiberglass or plastic is used now. In grades 2 and 3 the HLM analyses revealed that students had higher growth in reading comprehension when teachers were less often observed teaching with a highly teacher-directed stance e. Students showed greater fluency growth when their teachers were more often observed providing small-group instruction, and less often observed teaching phonics.
Children showed greater growth in writing when their teachers were more often observed asking higher-level questions after reading. Schneiter, who teaches second grade, focuses on higher-level questioning. She models word recognition strategies frequently as students are reading. She primarily teaches through small-group instruction, and she also has those students who are not with a teacher work with a partner or small group to foster active involvement at their seats.
On one day, Mrs. Schneiter is helping students review what they should do when they are reading with a partner and the partner gets stuck on a word. She also writes on the board the things that students should do if they finish their work before she is finished with her group: Later, she discusses characters' emotions with the group. She asks the children to take out their journals.
Students are to write two sentences in their journals. They are to write about one emotion and one characteristic. Students are encouraged to look back at the story if they need to. On a second day, Mrs. Schneiter talks to all groups. How would you solve the problem? Stone, a third grade teacher, focuses more on lower-level comprehension.
Her style is highly teacher-directed, and she makes frequent use of recitation. One day, prior to working with a group, she explains the seatwork activities. One group is to go to a listening center. Another is to think of five words that rhyme with "protect" and look up their definitions in the dictionary.
This group is also to complete a worksheet in which they circle rhyming words--all low-level phonics activities. In the reading group, Mrs. Stone asks a student to read aloud from Frog's journal, while other students follow along reading turn-taking, which is coded as passive responding. When a student gets stuck on a word she tells it to them. She then asks low-level questions about a character in the section: What did he need or want to tell Frog?
Why was he still? What did he hear? On another day, Mrs. Stone is working with a reading group. She then says that she will read the introduction, and begins to read while the students follow along. Overall this is a very teacher-directed lesson.
Asking students higher level questions after reading was related to their growth in fluency, comprehension, and writing. Coaching in word recognition strategies during reading was related to students' growth in fluency and writing. Having students engaged in active responding was also related to their growth in fluency. Teaching with a highly teacher-directed stance negatively related and providing whole-class or large-group instruction were additional classroom characteristics related to students' growth in reading comprehension.
May, a fifth-grade teacher, provides many good examples of the "best practices" identified in the HLM analyses for grades He provides more whole-group than small-group instruction, but students also work independently or in small groups for a fair amount of time in between whole-class segments of a reading lesson. In a typical lesson, Mr. May begins by listing objectives for the reading hour on the board.
Then he reviews the vocabulary words that students should be looking for as they read the story. Each word is introduced in the context of a sentence from the book; for example, "The Herdmans had music blaring in the background. May stresses higher-level questions and active pupil involvement. As students prepare to read the next chapter in the Best Christmas pageant ever, he challenges them to think about what is happening next in the story.
In their response journal, students are to answer the following questions: Why or why not? How do you think the audience will react?
Give evidence from the story. May provides small-group instruction to struggling readers while the rest of the class is reading independently. He coaches the small group in word recognition as they read the chapter, and helps prepare them for the questions he'll be asking in the whole-group setting.
May returns to the whole class and they have a discussion of what they wrote in their journals, he has students break into small groups. They are to work together in these groups to write the meanings for two vocabulary words and answer an assigned question. They put the answers on an overhead sheet in order to share their work with the rest of the class. Burns, a fourth-grade teacher, asks mostly lower-level questions, and his students are engaged in a considerable amount of passive responding.
During one lesson, students are in three groups: The teacher has them doing round-robin reading. When a student gets stuck on a word, Mr. Burns tells him the word. Questioning is a rapid fire of low-level questions: Did she use her cane when she had her guide dog with her? What did her daughter want her to do when she went on errands? What is the name of the school she is going to? On the second day, Mr. Burns works with the whole group.
The introduction to the story takes 30 minutes. Prior to reading, Mr. Burns asks students about the meaning of words which they will come across when they read: When the students don't know a word, he tells them what it means. Almost all of the talking is done by the teacher.
Half of the students seem to not be engaged or listening. They don't look at the board or raise their hands. However, they are all well-behaved. Comprehension strategies instruction, phonemic awareness instruction, and phonics instruction negatively related significantly contributed to students' growth on a variety of emergent literacy measures.
Small-group instruction contributed to growth in concepts of print. Jackson demonstrates all the characteristics of an effective kindergarten teacher, as determined from the HLM analyses. She teaches reading in small groups of four. She uses many word-level activities: Instead of telling the children information, Ms. Jackson gets students actively involved in their lessons. When introducing a new book to a small group of kindergartners, Ms.
Jackson says, "This story is going to take place somewhere different. Where do you think this one takes place? One student says, "The zoo! Jackson asks, "Do these animals live on a farm? The teacher asks, "Have you ever seen a real lion? What do they eat? Jackson responds, "Do you think they eat people?
The teacher asks, "What is the difference between a lion and a tiger? Following their discussion, Ms. Jackson has every child in the small group read the book aloud while she listens and coaches in word recognition strategies. For example, when a child is stuck on a word, Ms. Jackson says, "What does it start with? What could it be? Look at the picture. Jackson asks, "Is there more than one?
Jackson asks, "How would you read this word then? Lawson, another kindergarten teacher, also uses small groups for reading instruction, but she relies on lower-level questioning about text more than Ms.
For example, after reading The very hungry caterpillar by Eric Carle, she says, "I'm going to ask five questions on the story I just read to you.
What is the name of the story? Who was the main character in the story? What did he do in the story? Three schools had each been in the project for 2 years.
Separate analyses were conducted on student scores and teacher practices at these schools across the 2 years, in order to investigate the impact of the reform effort. An analysis of covariance on students' spring comprehension scores across grades , with fall scores used as the covariate, revealed a significant effect for the school: Bonferroni pairwise comparisons revealed that School 1 and School 2 had significantly higher adjusted spring comprehension scores than School 3.
Overall, schools had higher adjusted spring comprehension scores in Year 1 than in Year 2. However, School 1 had significantly higher adjusted spring comprehension scores in Year 2 than in Year 1: There was no difference between Year 1 and Year 2 scores in Schools 2 and 3. An analysis of the covariance on students' spring fluency scores across grades , using fall fluency scores as the covariate, revealed a significant effect for school: School 1 had significantly higher adjusted spring scores than School 3.
There was no difference between Year 1 and Year 2 scores in any of these schools. An analysis of the covariance on students' spring writing scores across grades , using fall writing scores as the covariate, revealed a significant effect for school: School 1 had significantly higher adjusted spring writing scores than Schools 2 and 3.
School 3 had significantly lower spring writing scores in Year 2 than in Year 1. There was no difference between Year 1 and Year 2 scores in Schools 1 and 2.
An analysis was done of teaching practices across Years 1 and 2 at School 1, which had higher spring comprehension and fluency scores than School 3 after adjusting for fall scores, and higher spring writing scores than Schools 2 or 3 after adjusting for fall scores. School 1 also showed increased growth in spring reading comprehension from Year 1 to Year 2 in an analysis of covariance.
Except for the increase in teacher-directed stance and decrease in student-support stance categories, all differences in classroom practices were in the direction which would be expected based on the research on effective classroom reading instruction, which had been shared with the teachers before Year 1 and between Years 1 and 2.
These findings suggest that teachers at School 1 were making shifts in their classroom reading instruction and were positively influenced by the research on best practices in reading. The reform effort rating had stayed at 4 out of a possible 10 for Years 1 and 2.
The school effectiveness rating increased from 6. Teachers had more positive perceptions about building collaboration, instructional reflection, professional development, and parent partnerships in Year 2 than they had had in Year 1.
The results were more mixed at School 2, with respect to changes in classroom practice. Some of these changes, such as increase in coaching in word recognition strategies and increase in asking of higher-level questions and the corresponding decrease in asking of lower-level questions moved in the direction that research would suggest was beneficial. Other changes moved in a direction that research would suggest was not beneficial.
Again, we shared research on effective reading instruction with teachers from School 2 both before Year 1 and between Years 1 and 2. The strongest positive change in classroom practices at School 2 was related to comprehension. Teachers had decided that improving students' reading comprehension should be one area of focus for Year 2.
During Year 2, teachers did appear to shift to asking students more higher-level questions about what they had read than they had in Year 1. However, observations also indicated an increase in comprehension skill instruction rather than comprehension strategy instruction, although the latter has been found to be the most effective approach for increasing reading achievement NRP, The reform effort rating had dropped from 4 to 3 between Years 1 and 2.
There had been a change in principals between Years 1 and 2, and as a result teachers were meeting in study groups less often in Year 2 than they had in Year 1.
However, on a positive note, the school effectiveness rating had increased from 8. As we had done at Schools 1 and 2, we shared the research on best practices with teachers at School 3 before Year 1, and again between Years 1 and 2.
As was the case at School 2, this research did not make much of an impact on the teaching of reading at School 3. However, School 3's reform rating did increase from 1 to 4 between Years 1 and 2, most likely because the new external facilitator was able to get study groups meeting more consistently in Year 2 than they had done in Year 1. In Year 2 teachers also showed more positive perceptions about building collaboration, professional development, leadership, and parent partnerships.
Summary of changes in classroom instruction at schools that participated in the project for two years. Based on the evidence from just these three schools, we have to conclude that the effect of the reform effort on classroom reading instruction was mixed at the end of Year 2.
It does appear that the reform effort is helping teachers at one school make changes in their classroom reading instruction according to research-based practices; increased growth in students' reading achievement was observed from Year 1 to Year 2. At the second school, which had established the improvement of students' reading comprehension as a schoolwide goal, we observed a shift towards higher-level questioning. In general, however, we did not see a shift toward more effective reading instruction practices at this school.
The third school showed little evidence of a shift towards more effective reading instruction practices across Years 1 and 2. We often hear that effective school reform in reading--reform that significantly raises student achievement--takes dedication, hard work, and time.
The results of our study confirm this assertion. Even though the vast majority of teachers at all schools in the project voted to engage in schoolwide reading reform, the schools clearly have a long way to go to raise their reading scores to "breaking the mold" levels, with current mean standardized reading comprehension scores across schools, for example, standing at 40 in grades 2 and 3, and at 37 in grades 4 and 5.
If we consider the factors of building collaboration, professional development, instructional reflection and change, collaborative leadership, and parent partnerships, then schools had a mean effective school rating of 8.
However, the schools that joined the project are aspiring to beat the odds. Each of the three schools that have been with the project for 2 years increased their school effectiveness rating from Year 1 to Year 2. Our shared hope, as we continue to follow these schools over the next year, is that we will see significant changes toward more effective school-level and classroom-level practices and more success in reform efforts--changes which will be accompanied by increases in student performance on a range of reading measures.
Even so, it is important to note that collaborative leadership did make a significant contribution to growth in students' reading fluency and writing. Schools where teachers perceived strong collaborative leadership also displayed more positive perceptions of school climate, and more collaboration in both professional development and the delivery of reading instruction. In terms of the reform effort, our results are mixed.
With an average score of 4. On a positive note, the one school that had made the most consistent gains in students' reading and writing growth was also the school in which teachers had made the most research-based shifts in the delivery of their classroom reading instruction.
We believe that the most interesting findings in this study come from the observational data on classroom reading instruction, irrespective of school. The HLM analyses consistently found that higher-level questioning mattered: The teachers who were coded as asking more higher-level questions appear to be teachers who understand the importance of challenging their students' thinking about and comprehension of what they have read.
There is little cause to celebrate, however: Furthermore, comprehension strategy instruction was seldom observed. Interestingly, the information in teachers' weekly logs corroborated this finding.
The findings on word skill work suggest that spending relatively large amounts of time on phonics instruction in grades may not facilitate students' growth in reading fluency. This finding is compatible with the National Reading Panel's recommendation that phonics instruction should be concentrated in the earliest stages of schooling, mainly K and 1. Among older students, the practice of coaching in word recognition strategies during reading was found to be useful for students in grade However, coaching in the application of phonics strategies is very different from explicit instruction focused on the letter-sound correspondences and rules, and is inherently more metacognitive and strategic in nature.
A negative relationship was also found between a highly teacher-directed stance towards reading instruction and reading growth for grades This does not mean that teachers should never tell students information or engage them in recitation; it would be impossible to teach without doing so.
However, it does appear that a heavy reliance on telling and recitation as teaching techniques is negatively related to children's reading growth. Excessive amounts of "telling," especially in situations where it would be possible to coach students to come up with their own responses, may rob children of the opportunity to take responsibility for their own skills and strategies. It may be useful to provide teachers with observational data on the frequency of telling and recitation in their literacy teaching, in order to help them shift away somewhat from a teacher-directed stance.
This shift would ideally lead to enhanced student performance. Over the coming years, as we provide teachers and schools with data on how teaching practices are tied to students' performance, we plan to investigate the degree to which classroom- and school-level teaching practices shift over time toward practices which have been identified as more effective for enhancing student achievement.
On the other hand, our classroom observations indicate that children in grade 1 showed less growth in writing when their teacher exhibited a strong student support stance e. This finding may be an anomaly, since it is contrary to other findings on the importance of scaffolding Pressley et al. Or it may be that children benefit from more teacher direction when learning to write in grade 1. Clearly, this is an area in need of further research.
The findings on grouping practices were mixed, and depended on the grade level in question. For reading comprehension in grade 1 and concepts of print in kindergarten, the coding of small-group instruction was positively related to students' reading growth. In grades , the coding of whole- or large-group instruction was positively related to students' reading growth.
Although our findings to this point are primarily limited to teachers, irrespective of school, we will continue to study schools to see if project participation might lead to building-level shifts. We will also continue to investigate the impact of teacher-level factors on students' reading and writing growth, as we have done in the current paper.
One limitation of the study is that we were only able to investigate school- and classroom-level practices in five project schools during the first year, and eight schools in the second year.
Only three of these eight schools have completed 2 full years in the project by this writing. Since change takes time, our ability to analyze the impact of the reform effort has thus far been limited by the small number of schools which have been in the project for at least 2 years.
In addition, classroom information was gathered from three one-hour observations per classroom, per year. This only gives us a snapshot of the reading instruction within these classrooms. The log information that was collected in Year 2 helps to provide a more complete picture of the reading activities occurring within certain classrooms, but unfortunately, log data were not available for the Year 1 schools.
We therefore concentrate on the years from to Over this period, we are able to observe 7, individual principals and make use of 28, annual principal observations. The fundamental challenge to measuring the impact of school leaders is separating their contributions from the many other factors that drive student achievement.
For example, a school that serves largely affluent families may create the illusion that it has a great principal, when family backgrounds are the key cause of high achievement.
Alternatively, a school that serves disadvantaged students may appear to be doing poorly but in fact have a great principal who is producing better outcomes than any other principal would. Our basic value-added model measures the effectiveness of a principal by examining the extent to which math achievement in a school is higher or lower than would be expected based on the characteristics of students in that school, including their achievement in the prior year.
Put another way, it examines whether some schools have higher achievement than other schools that serve similar students and attributes that achievement difference to the principal. This approach is very similar to that employed in studies that measure teacher quality using databases tracking the performance of individual students over time.
The main concern with this approach is that there may be unmeasured factors that affect school performance. Our data contain only basic information on student background characteristics, such as gender, race or ethnicity, and eligibility for subsidized lunch. As a result, we cannot control for more nuanced measures of students and their families, such as motivation or wealth. Moreover, there are also school factors not under the direct control of the school, including the quality of teachers inherited by the principal.
Below we describe alternative approaches to isolating the contributions of the current principal. In estimating principal effectiveness, we want to minimize the influence of specific circumstances and look at the underlying stable differences in impacts. To account for any differences in effectiveness that are related to tenure as a principal in a given school, we begin our analysis by focusing on data from the first three years a principal leads a school. This first analysis indicates that the standard deviation of principal effectiveness is 0.
This is a very large figure, perhaps unbelievably large, implying that a principal at the 75th percentile of this effectiveness measure shows average achievement gains of 0. These differences are even more pronounced in high-poverty schools, for which the gap between the 25th and 75th percentile principal is more than one-third of a standard deviation.
On average across all schools, the impact of having a principal 1 standard deviation more effective than the average principal is as much as seven additional months of learning in a single academic year. As a result, it may overestimate the amount of influence principals actually have. We begin to address this issue by measuring principal effectiveness based only on comparisons of within-school differences in student achievement growth over time.
In simplest terms, we compare average student achievement gains in the same school under different principals. This method eliminates the influence of any student, school, or neighborhood characteristics that do not change over time. Its main drawback is that it ignores all differences in principal effectiveness between schools, potentially underestimating the amount of variation in principal quality.
For example, if each school tends to attract principals who are similar in quality whenever it searches for a new principal, this approach will understate the true extent of variation in principal effectiveness. We conduct this second analysis using all of the principals in our data, not just those in their first three years leading a school, because the numbers of schools with two principals observed in their first three years is quite small.
Note that re-doing the prior analysis using data on all principals does not significantly alter the results presented above. Restricting the analysis to comparisons within schools, however, cuts our estimate of the variation in principal effectiveness in half. Even this reduced estimate is substantial, however, indicating that a 1-standard-deviation increase in principal effectiveness raises school average achievement by slightly more than 0.
This impact is roughly comparable to that observed for variations in teacher effectiveness in studies that use the same kinds of within-school comparisons. Our first two methods involved estimating effectiveness measures for individual principals and then calculating the standard deviation of those measures. Although any unmeasured school factors that are unrelated to principal quality would not bias these results, such factors would inflate our estimates of the variation in principal quality based on these approaches.
We therefore employ a third approach that gauges the amount of variation in principal effectiveness directly by measuring the additional fluctuation in school average achievement gains when a new principal assumes leadership, as compared to typical fluctuations from year to year. Focusing on the additional variation in school average achievement gains around principal transitions reduces the magnitude of the estimates.
Nonetheless, the results remain educationally significant: The data do not include any observations about what a principal actually does, or fails to do, to improve learning. We now turn to an analysis of the interactions of principals with teaching staff, which bears directly on a number of current policy debates.
Teacher turnover per se has received considerable policy attention, largely because of the well-documented difficulties that new teachers experience. The potential benefits of reducing turnover nonetheless hinge on the effectiveness of both entering and exiting teachers. We expect highly rated principals to be more successful both at retaining effective teachers and at moving out less-effective ones. Less highly rated principals may be less successful in raising the quality of their teaching staffs, either because they are less skilled in evaluating teacher quality, place less emphasis on teacher effectiveness in personnel decisions, or are less successful in creating an environment that attracts and retains better teachers.
Although better principals may also attract and hire more-effective teachers, the absence of reliable quality measures for new teachers and the fact that many principals have little control over new hires lead us to focus specifically on turnover. Unfortunately, our data do not contain direct information on personnel decisions that would enable us to separate voluntary and involuntary transitions, and existing evidence suggests that teachers rather than principals initiate the majority of transitions.
In addition, the Texas data do not match students to individual teachers, meaning that we must draw inferences about teacher effectiveness from average information across an entire grade. With detailed information on teacher effectiveness and transitions, we could investigate whether better principals are more likely to dismiss the least-effective teachers and reduce the likelihood that the more-effective teachers depart voluntarily.
In the absence of such information, however, we focus on the relationship within schools between the share of teachers that exits each grade and the average value-added to student achievement in the grade. We examine how this varies with our measures of principal quality based on student achievement gains. For example, in a school where 5th-grade students learn more than 4th-grade students, we would expect a good principal to make more changes to the 4th-grade teaching staff.
The results of this analysis confirm that the relationship between higher teacher turnover and lower average valueadded in a given grade is stronger as principal quality rises.
This pattern of results is consistent with the theory that management of teacher quality is an important pathway through which principals affect school quality. The fact that less-effective teachers are more likely to leave schools run by highly effective principals also validates our measure of principal quality.
If our measure was just capturing random noise in the data rather than information about true principal quality, we would not expect it to be related to teacher quality and turnover. Along with teacher turnover, instability of leadership is often cited as an impediment to improving high-poverty and low-performing schools.
Consistent with these concerns, we find that Texas schools with a high proportion of low-income students are more likely to have first-year principals and less likely to have principals who have been at the school at least six years than those serving a less-disadvantaged population.
Sorting schools by initial achievement rather than poverty level produces even larger differences see Figure 1. The proportion of principals in their first year leading a school is roughly 40 percent higher in schools in the bottom quartile of average prior achievement than in schools in the top quartile; the proportion of principals that have been at their current school at least six years is roughly 50 percent higher in schools with higherachieving students.
Yet the import of leadership turnover also depends on whether highor low-quality personnel are leaving, something prior research has been unable to address.
We therefore examine whether the likelihood that a principal leaves following the third year in a school varies with her effectiveness and with the share of low-income students in the school.
OVER 25 YEARS AGO a federal paper was written to discuss the effectiveness of American education. The paper was funded by the U.S. Office of Education and written by James Coleman, a prominent education researcher. Effective Schools Research emerged in response to this controversial paper.
The Effective Schools model of school reform is based on more than thirty years of research conducted nationally and internationally.
NINE CHARACTERISTICS OF HIGH-PERFORMING SCHOOLS Nine Characteristics of High-Performing Schools Prepared by G. Sue Shannon, Senior Researcher Pete Bylsma, former Director, Research, Assessment, Accountability research on effective schools is included, as well as about new references and rel-. Effective schools. "Effective Schools" is both an educational movement and body of research which examines school-based factors which positively influence learning outcomes in K schools. Effective schools research has been widely adopted by school districts worldwide.
CORRELATES OF EFFECTIVE SCHOOLS. THE CORRELATES ARE THE MEANS to achieving high and equitable levels of student learning. It is expected that all children (whether they be male or female, rich or poor, black or white) will learn at least the essential knowledge, concepts and skills needed so that they can be successful at the . Research studies that have focused on identifying the characteristics or correlates of elementary and secondary schools that are unusually effective are reviewed, concentrating on the.