KNOW THY IMPACT: Using Hattie's Math to Determine Your Impact

John Hattie is an invaluable researcher in the field of education. His way of looking at the impact of particular influences on student achievement helps educators around the world shift the conversation from what works to what works best.

http://www.education.vic.gov.au/Documents/about/research/ravisiblelearning.pdf

Despite this, Hattie's research can sometimes seem removed from life in the classroom as the effect sizes that he presents in his books are based on very large research studies. Teachers may lament that it is hard to know if the large effect sizes of some the most impactful influences can be replicated in their classrooms, with their unique students.

Until now.

In his book Visible Learning for Literacy (2016) that he co-authored with Fisher & Frey, Hattie encourages teachers to reflect on the impact of their own instruction and presents a formula for calculating effect size in their classrooms.

Being able to calculate impact in this way gives teachers the mathematical ability to quantitatively see if instruction is having an impact on their students' achievement and who is not being impacted too. Armed with this information, teachers are able to adapt their teaching as to maximize their effectiveness for the benefit of all students in their classrooms.

In order to calculate effect size, Hattie suggests that:

• Lessons have clear learning intentions.
• Lessons have clear success criteria.
• The success criteria indicate what quality looks like.
• Students know where they stand in relation to the criteria for success (p. 136).

With this in place, teachers only need a pre-assessment and a post-assessment score to be able to calculate effect size.

Below, I demonstrate how to calculate effect size by analyzing students' thinking from an inquiry lesson I recently taught with third graders. The lesson is fully described here: Inquiry into Moon Phases.

What did I want students to learn?

During the lesson, our goal was to answer this essential question:

What would success look like?

A successful response will ...

• Contain academic science vocabulary related to the lesson (1 pt awarded for inclusion of each of the following:
• observe, Earth, Moon, change, orbit, shadow, light, new moon, crescent moon, full moon, phases*, Sun*, reflecting*, darker*, lighter*
• *Not introduced during the lesson, but still important scientific terms that came out during the post-assessment.
• Contain different ideas (1 pt awarded for each complete idea)
• examples of complete ideas are: the moon orbits the earth, the moon reflects the sun's light, the shadow gets bigger as the light gets smaller, the full moon is when the moon is all it up).
• Be accurate
• 4 pts awarded for accurate statements with details/evidence,
• 2 pts awarded for semi-accurate statements with little details/evidence and some misconceptions
• 0 pts awarded for inaccurate statements with no details/evidence and many misconceptions

How do I know they've learned?

Assessment task: The task for the pre-assessment (measure of what students initially understood, knew and could do) was the same as the post-assessment (measure of what they learned). For both, I prompted:

• "Write what you think the answer to our essential question is. Make sure to include scientific vocabulary in your response."

To calculate effect size (p 138)

1. Analyze the pre- and post-assessments.

This step was a snap, thanks to the pre-established success criteria. I recorded these results in a Google Sheet:

	Total pre	Total post
Student A	9	13
Student B	8	12
Student C	5	11
Student D	3	6
Student E	6	14
Student F	2	12
Student G	3	13
Student H	7	13
Student I	7	10
Student J	6	10
Student K	3	16
Student L	3	12
Student M	2	13
Student N	3	13
Student O	2	11
Student P	6	4
Student Q	2	14
Student R	4	11

2. Find the average of the pre- and post-assessments

Using the average formula (=AVERAGE) this step was easy too!

• Average pre: 4.50
• Average post: 11.56

3. Calculate the standard deviation for the pre- and post-assessment and then find the average of the two standard deviations.

This step was super simple too, as the Standard Deviation formula is just (=STDEV).

• Standard Deviation pre: 2.28
• Standard Deviation post: 2.83
• Average Standard Deviation: 2.56

4. Determine effect size

Using Hattie's formula: (Average Post - Average Pre) / Average Standard Deviation

• Effect size: 2.76

This effect size is quite sizable and is most definitely off the scale of the Barometer of Influence that Hattie presents in his work. Some things to consider:

• Whereas this is a large effect size, it is just a number. With this quantitative data, a teacher should also reflect qualitatively:
• In what ways did the students grow the most?
• What about the lesson was successful that should be replicated?
• What wasn't successful that can be eliminated?
• This was just one lesson and the sample size is minute, compared to the studies Hattie typically meta-analyzes. Therefore, little relative importance should be placed on this effect size. After several weeks of working on making scientific observations using scientific language, another assessment could be administered to see if students have continued to show growth with this skill.

5. Determine individual effect sizes

Although the impact of this lesson was quite high on average, that is not necessarily the case for all students. By calculating individual effect sizes using individual assessment scores and the average Standard Deviation, you can see for whom this lesson was successful and for whom it was not.

	Individual Effect Sizes
Student A	1.56
Student B	1.56
Student C	2.35
Student D	1.17
Student E	3.13
Student F	3.91
Student G	3.91
Student H	2.35
Student I	1.17
Student J	1.56
Student K	5.08
Student L	3.52
Student M	4.30
Student N	3.91
Student O	3.52
Student P	-0.78
Student Q	4.69
Student R	2.74

The effect sizes of these individual students is also beyond the scale of Hattie's Barometer of Influence presented in his work. I don't believe that is important though. What is important is to look at the individual effect sizes in relation to one another along with looking at students' thinking and reflect:

• What causes one student (student F, for instance) to make sizable gains, whilst another student (like A or B) just grew marginally?
• What kinds of thinking are these students demonstrating?
• What about the teaching made such an impact on these students that could be replicated in the future?
• What about the teaching caused other students to not gain as much that should be avoided or adapted in the future?
• What do these students need next in their learning?

Below, I've included some samples of students' thinking:

Student F Pre:

Student F Post:

Student N Pre:

Student N Post:

Student O Pre:

Student O Post:

Although most students showed they learned a great deal during this lesson, Student P did not do as well on the post-assessment as he did on the pre. Using this quantitative data, a teacher must look more deeply at the student's response and reflect:

• What kinds of thinking is this student demonstrating?
• What about the teaching had a negative effect on this student's learning that should be avoided in the future?
• What does this student need next in their learning?

Student P Pre:

Student P Post:

Being able to calculate effect size to determine what works best for individual students is powerful and has the potential to transform the ways which we respond to students. How could you use Hattie's math to determine the impact of your instruction on individual students' achievement?

Assessment-Capable Learners: Understanding one of Hattie's most powerful influences on student achievement

John Hattie published Visible Learning (2009) to synthesize evidence-based research in order to tell a specific and compelling story: 

• Almost nothing we do harms kids
• Most of what we do leads kids to learning
• Therefore, instead of talking about "what works", we should be focused on "what works best"

Doug Fisher says that Hattie's work tells us what works, how it works and when it works. Deb Masters suggests the Mindframes for Teachers and Leaders represent a succinct story of the Visible Learning Research. 

My sketchnotes from Masters' session on the 10 Mindframes for Teachers & Leaders.

To clearly show what works best (has greater probability of having a high impact on student learning), Hattie used the research to calculate the effect size for each influence studied. He found that an effect size of d = 0.15 to d = 0.40 represented what teachers can accomplish in a typical year of schooling. He argued therefore, we must focus on those strategies that have an effect size of d = 0.40 and higher; those influences that work best.

http://www.education.vic.gov.au/Documents/about/research/ravisiblelearning.pdf

When Visible Learning Trainer Dave Nagel came to our district in February 2016, he mentioned an influence I hadn't heard of before in Hattie's research: Assessment-Capable Learners. Assessment-Capable Learners, Nagel explained, had an effect size of d =1.44, well above the majority of the other influences listed in Hattie's research. This influence is one of the most important things we can do to raise student achievement.

Although Nagel showed the following slide during his keynote, I was still left confused; what the heck was an Assessment-Capable Learner? How do we develop these types of students?

A slide explaining the attributes of an Assessment-Capable Learner from Dave Nagel, February 2016

As with many of the influences on Hattie's list, there is great danger at taking misguided action based on a superficial understanding of what an Assessment-Capable Learner is. On the surface, one might think that Hattie is suggesting that we develop learners who are capable of successfully taking assessments, particularly those high stakes tests which are pervasive in schools throughout the world; learners who have practiced and are skilled at various test-taking strategies: closely-reading the questions, eliminating answers obviously wrong, going back and checking work, etc. However there is evidence that teaching learners these test-taking strategies actually does not have a high impact on student learning (teaching test-taking: d = 0.27).

Those that have an initial misunderstanding of an Assessment-Capable Learner aren't alone. When asked, "Why are we striving for an 'assessment-capable learner' and not a 'self-regulated learner?'" Hattie admitted that he struggles with a lot of "those words". Rather, he suggests that we think about Assessment Capable Learners in this way: "When students see themselves as their own teachers."

Each keynote and breakout session at the Annual Visible Learning Conference in Washington, D.C. allowed me to further construct my own understanding of Assessment-Capable Learners. Below, I share a synthesis of my thinking:



Assessment-Capable Learners:

• can answer the following three questions, as long as teachers have clearly communicated learning intentions and success criteria (teacher clarity: d = 0.75)

1. What am I learning?
2. Why am I learning it?
3. How will I know when I've had success & have learned it?

• know where they are, where they're going (based on clear learning intentions & success criteria) and their next steps to move forward.
• know the language of learning (VOICE) and make decisions about their learning (CHOICE)
• are open to and expect feedback.
• give feedback to others, because they recognize the powerful impact they can have on their peers' learning.
• set challenging, yet realistic learning goals AND put forth the effort to reach them.
• are active, involved and engaged in their own learning.
• are radical change agents
• see errors as opportunities for learning
• exhibit the eight Mindframes for Learners:

1. I want to know what success looks like.
2. I like challenging goals.
3. I want to master and have deep learning.
4. I am confident I can learn.
5. I want to become my own teacher.
6. I engage in dialogue, not monologue, about my learning.
7. I like to plan to implement my learning goals.
8. I want to learn to be strategic in my learning goals.

As I continue to construct my own understanding of an Assessment-Capable Learner, I begin to ask: what do we teachers need to do to develop these dispositions and encourage these actions in the learners with whom we work?

THANK YOU to all the presenters at the conference. My thinking above represents a synthesis of keynotes and breakout sessions led by those directly quoted above and: Paul Bloomberg, Barb Pitchford, Peter DeWitt, Michael Fullan, Jen Mall, Michael McDowell.

Know Thy Impact

Those in education understand that new, innovative ways of doing things are popping up all the time. Especially now, as savvy educators connect on social media platforms like Pinterest, Reddit, Facebook, Blogger and Twitter, there is a superfluity of new things to try out with students.

"Try this! Now try this! Look at this new thing! How about this new graphic organizer?! What about this new thinking routine?!"

All promise to increase student achievement. But John Hattie, a researcher who has authored various publications on maximizing teachers' impact on students' learning, says in this video that, "there is hardly anything we do as teachers that harms kids." He continues that, "95%-98% of things we do to students enhances achievement. All you need to enhance achievement," he jokes, "is a pulse."

But we cannot settle for simply increasing student achievement. Since virtually everything we do to kids will undoubtedly increase their achievement, what we need to do is carefully select the interventions that have the greatest impact on students' learning. In Hattie's language, those treatments with an effect size of .4 or more are desirable. Hattie explains that an effect size of .4 is about a year's growth and he argues that in order to have a remarkable, systematic, positive impact on kids, we need to be doing things with students that will increase their achievement more than that.

So, how do we know what we're doing is working and working to the level that we need? How do we know the strategies, interventions, activities and learning engagements we're doing with kids are growing their achievement more than a year's growth? We must answer Hattie's invitation to "Know Thy Impact".

When we're working with kids, we must look for the evidence that what we're doing with our students is working. We must ask, "what are our kids doing, saying, and thinking that shows they are learning ... and learning a lot about big ideas that are timeless, abstract, universal, and transferable? How can we prove that what we're doing is actually improving their lives? 

If we can't prove that students are learning with evidence, it isn't their fault: it is ours. It is our responsibility to change what we are doing, so that they can be successful.

To learn more about Hattie and his research, start here: Hattie Ranking: Influences And Effect Sizes Related To Student Achievement & Hattie's 8 Mindframes.

The concept of impact has been on my mind a lot in the last 36 hours, ever since I heard that my dear Auntie Judy passed away unexpectedly. Judy, or Ms Higbea as she was known to literally thousands of students, was an educator for 50 years. I learned so much from my aunt; among them, the continual need to question and reflect on what is best for students.

The very ironic thing about a teacher like Judy who is continually asking, "What's my impact?" is that they will never truly understand the impact they have. It is those teachers who aren't just increasing students' achievement scores. And they're not even just increasing students' learning. They're improving the quality of the lives of students, their families, colleagues, and those in the larger community.

As news spreads of Judy's passing, those that were impacted by her are voicing their shock, heartbreak, and words of comfort and appreciation. Through Judy's formal work with the Freeport school system, Freeport Community Services, Wolfe's Neck Farm and her informal interactions, she impacted the lives of many different people in many different ways. Many are sad. Many are grateful.

Judy would probably think it obnoxious of me to be writing about her in such a way. She was quite humble - one of her many lovely qualities I didn't "inherit". But if by her example, we all can continually question, "what's my impact?" her impact on us and on our students will continue.

Should I always start with an "I can" statement?

I was originally going to title this post "A Case Against 'I can' Statements," but as you'll soon find out, I'm advocating for starting inquiries with provocative questions, so why should this post be any exception? 

So here it is, the fascinating question that we're exploring today: Should I always start with an "I can" statement?

Some history: For the last several years, teachers in our district have been busy converting rigorous academic standards into student-friendly language in the form of "I can" statements. Here are some examples from various grade levels and subject areas.

I can identify cause and effect in a nonfiction passage (based on ELA benchmark 3.2.3.3)

I can sort objects by shape, size, color, and thickness (based on Math benchmark K.3.1.2)

I can describe societies that existed in Mesoamerica and North America before 1500 (based on History benchmark 5.4.4.15.1)

I can recognize that plants need space, water, nutrients, and air and I can explain that plants fulfill these needs in different ways (based on Life Science benchmark 2.4.2.1.1)

Clearly communicating to students the intent of the lesson is of utmost importance. Hattie writes it is important "to communicate the intentions of the lessons and the notions of what success means for these intentions," (p. 126 of Visible learning: A synthesis of over 800 meta-analyses relating to achievement.)

Hattie found that teacher clarity - which Fendick (1990) defined as organization, explanation, examples and guided practice, and assessment of student learning - has an effect size of .75 and ranked it 8th on his list of most effective influences on student achievement. Read more about Hattie's list on his website.

We all can agree that when students clearly understand where the lesson is heading and what they need to do to be successful, they'll achieve at higher rates. But is an "I can" statement the only way to clearly communicate the intention of a lesson?

In the fall, our school's teaching staff read an article by Brookhart and Moss about learning targets. The authors echo Hattie when they write that "when students understand exactly what they're supposed to learn and what their work will look like when they learn it, they're better able to monitor and adjust their work, select effective strategies, and connect current work to prior learning," (p. 29). They continue by saying that a learning target can take on several different forms. They suggest that the learning target can be words, pictures, demonstrations, or other experiences. They warn that a learning target does not have to be an "I can" statement.

I wonder, can a learning target be in the form of a question?

In a Primary Years Programme teachers should consider communicating the intentions of the lesson and what success means for these intentions in the form of a thought-provoking question. This will provide our students with opportunities to build understanding through structured inquiry. But how is this done?

Once teachers identify what they want students to understand (concepts), know (knowledge), and be able to do (skills), they should craft questions that will drive student inquiries. These questions should serve to motivate students to construct their own understanding of concepts and engage them in acquiring the knowledge and skills they need to be successful. Yes, teachers should clearly communicate the intentions of the lesson but I would warn against "giving away the answer" in the form of an "I can" statement. Instead, teachers should ask questions that engage learners and motivates them to seek out the answer.

Communicating the intention of the lesson in the form of a question allows teachers to "model explicitly [for their students] the asking of open-ended, driving questions that will promote conceptual development," (p. 37 of Making the PYP Happen). Fun Fact: the word 'question' appears in Making the PYP Happen 140 times.

McTighe and Wiggins advocate in multiple ways for the use of questions in their book Essential Questions, but they really ramp up the rhetoric when they purport that "once we have learned to question - really question - then we are immunized from falling victim to people who want us not to think too hard about what they say, be it politicians, advertisers, or bullying associates," (p. 18).

So, can teachers tweak "I can" statements to make them into questions that clearly communicate the intention of the unit/lesson and what success means while not "giving away the answer"? My answer is a resounding, YES! WHY NOT? LET'S GIVE IT A TRY!

Here is my attempt at it, converting the "I can" statements previously shared above into questions.

What are cause-effect relationships? Where are the cause-effect relationships described in the nonfiction text we're reading? How does discovering cause-effect relationships help us understand what we read?

What are the different ways we can sort things? How does putting things into groups help us understand them better? How does putting things into groups hurt our understanding of them?

What were societies like in Mesoamerica and North America before 1500? What caused them to change?

What do plants need to survive? Do all plants fulfill their needs in the same way?

So, let's revisit the question again: Should I always start with an "I can" statement?

I certainly am not making a case for throwing "I can" statements completely out the window. There is certainly an appropriate time and place for explicitly telling students the intent of the lesson and what they'll need to do to have success. 

However, I am suggesting that teachers consider clearly communicating the learning target by asking challenging questions at the onset of a unit or a lesson that engage learners and draws them into the inquiry. These questions should capture their head and tug at their heart. As the lesson or unit continues, students' responses should be revisited; misconceptions should be corrected; new learning should be added.

Recently, a first grade teacher asked her students at the onset of a unit of inquiry, "Why is it important to make careful and close observations?" She recorded her students' initial answers, leaving out any repeats. Throughout the entire unit, as the students explored, discovered, and investigated making observations, classifying, and drawing conclusions, they continually revisited their original ideas, editing and revising as necessary. The poster below is their "finished" product.

As a thoughtful colleague recently reflected, the questions we ask - both by teachers and students - must really kindle a strong desire, passion, or need to follow through. If through asking thoughtful, relevant, and provocative questions, we can successfully light a fire within our students, imagine the learning and achievement that is possible!

Using Google Docs to provide feedback on students’ writing

Fourth graders in one classroom are writing personal narratives using Google Docs, which they have electronically shared with their classroom teacher and other support teachers with whom they work. Drafting in Docs meets 4th grade ELA benchmarks:

• 4.6.5.5 With guidance and support from peers and adults, use a writing process to develop and strengthen writing as needed by planning, drafting, revising, and editing.

• 4.6.6.6 With some guidance and support from adults, use technology, including the Internet, to produce and publish writing as well as to interact and collaborate with others; demonstrate sufficient command of keyboarding skills to type a minimum of one page in a single sitting.

More importantly though, it allows teachers to comment on student work through the shared document, which encourages and challenges learners to continually improve their writing. Students read these comments, reply to them, revise and edit their narratives accordingly, without ever having to meet one-on-one with the teacher.


Professor John Hattie researched different influences and measured their effectiveness on student achievement (for a summary of his findings, check out this infographic). He found that providing students with effective feedback has high influence over student achievement. In order for feedback to be effective and useful though, three conditions have to be met: “the learner needs the feedback; the learner receives the feedback and has time to use it; and the learner is willing and is able to use the feedback,” (Sadler in Hattie, p. 153, Visible Learning for Teachers; Maximizing impact on learning, 2012, New York: Routledge).


Here's another powerful example on the effectiveness of critiquing and feedback on student performance: Austin’s Butterfly.

Based on these different stories, ask:

• How do you use feedback with your students?

• Do you find it is effective?

• How could you reexamine your practice on giving students feedback to have a greater impact on their learning?

Share your thoughts and ideas by commenting on this post.