KNOW THY IMPACT: Using Hattie's Math to Determine Your Impact

John Hattie is an invaluable researcher in the field of education. His way of looking at the impact of particular influences on student achievement helps educators around the world shift the conversation from what works to what works best.

http://www.education.vic.gov.au/Documents/about/research/ravisiblelearning.pdf

Despite this, Hattie's research can sometimes seem removed from life in the classroom as the effect sizes that he presents in his books are based on very large research studies. Teachers may lament that it is hard to know if the large effect sizes of some the most impactful influences can be replicated in their classrooms, with their unique students.

Until now.

In his book Visible Learning for Literacy (2016) that he co-authored with Fisher & Frey, Hattie encourages teachers to reflect on the impact of their own instruction and presents a formula for calculating effect size in their classrooms.

Being able to calculate impact in this way gives teachers the mathematical ability to quantitatively see if instruction is having an impact on their students' achievement and who is not being impacted too. Armed with this information, teachers are able to adapt their teaching as to maximize their effectiveness for the benefit of all students in their classrooms.

In order to calculate effect size, Hattie suggests that:

• Lessons have clear learning intentions.
• Lessons have clear success criteria.
• The success criteria indicate what quality looks like.
• Students know where they stand in relation to the criteria for success (p. 136).

With this in place, teachers only need a pre-assessment and a post-assessment score to be able to calculate effect size.

Below, I demonstrate how to calculate effect size by analyzing students' thinking from an inquiry lesson I recently taught with third graders. The lesson is fully described here: Inquiry into Moon Phases.

What did I want students to learn?

During the lesson, our goal was to answer this essential question:

What would success look like?

A successful response will ...

• Contain academic science vocabulary related to the lesson (1 pt awarded for inclusion of each of the following:
• observe, Earth, Moon, change, orbit, shadow, light, new moon, crescent moon, full moon, phases*, Sun*, reflecting*, darker*, lighter*
• *Not introduced during the lesson, but still important scientific terms that came out during the post-assessment.
• Contain different ideas (1 pt awarded for each complete idea)
• examples of complete ideas are: the moon orbits the earth, the moon reflects the sun's light, the shadow gets bigger as the light gets smaller, the full moon is when the moon is all it up).
• Be accurate
• 4 pts awarded for accurate statements with details/evidence,
• 2 pts awarded for semi-accurate statements with little details/evidence and some misconceptions
• 0 pts awarded for inaccurate statements with no details/evidence and many misconceptions

How do I know they've learned?

Assessment task: The task for the pre-assessment (measure of what students initially understood, knew and could do) was the same as the post-assessment (measure of what they learned). For both, I prompted:

• "Write what you think the answer to our essential question is. Make sure to include scientific vocabulary in your response."

To calculate effect size (p 138)

1. Analyze the pre- and post-assessments.

This step was a snap, thanks to the pre-established success criteria. I recorded these results in a Google Sheet:

	Total pre	Total post
Student A	9	13
Student B	8	12
Student C	5	11
Student D	3	6
Student E	6	14
Student F	2	12
Student G	3	13
Student H	7	13
Student I	7	10
Student J	6	10
Student K	3	16
Student L	3	12
Student M	2	13
Student N	3	13
Student O	2	11
Student P	6	4
Student Q	2	14
Student R	4	11

2. Find the average of the pre- and post-assessments

Using the average formula (=AVERAGE) this step was easy too!

• Average pre: 4.50
• Average post: 11.56

3. Calculate the standard deviation for the pre- and post-assessment and then find the average of the two standard deviations.

This step was super simple too, as the Standard Deviation formula is just (=STDEV).

• Standard Deviation pre: 2.28
• Standard Deviation post: 2.83
• Average Standard Deviation: 2.56

4. Determine effect size

Using Hattie's formula: (Average Post - Average Pre) / Average Standard Deviation

• Effect size: 2.76

This effect size is quite sizable and is most definitely off the scale of the Barometer of Influence that Hattie presents in his work. Some things to consider:

• Whereas this is a large effect size, it is just a number. With this quantitative data, a teacher should also reflect qualitatively:
• In what ways did the students grow the most?
• What about the lesson was successful that should be replicated?
• What wasn't successful that can be eliminated?
• This was just one lesson and the sample size is minute, compared to the studies Hattie typically meta-analyzes. Therefore, little relative importance should be placed on this effect size. After several weeks of working on making scientific observations using scientific language, another assessment could be administered to see if students have continued to show growth with this skill.

5. Determine individual effect sizes

Although the impact of this lesson was quite high on average, that is not necessarily the case for all students. By calculating individual effect sizes using individual assessment scores and the average Standard Deviation, you can see for whom this lesson was successful and for whom it was not.

	Individual Effect Sizes
Student A	1.56
Student B	1.56
Student C	2.35
Student D	1.17
Student E	3.13
Student F	3.91
Student G	3.91
Student H	2.35
Student I	1.17
Student J	1.56
Student K	5.08
Student L	3.52
Student M	4.30
Student N	3.91
Student O	3.52
Student P	-0.78
Student Q	4.69
Student R	2.74

The effect sizes of these individual students is also beyond the scale of Hattie's Barometer of Influence presented in his work. I don't believe that is important though. What is important is to look at the individual effect sizes in relation to one another along with looking at students' thinking and reflect:

• What causes one student (student F, for instance) to make sizable gains, whilst another student (like A or B) just grew marginally?
• What kinds of thinking are these students demonstrating?
• What about the teaching made such an impact on these students that could be replicated in the future?
• What about the teaching caused other students to not gain as much that should be avoided or adapted in the future?
• What do these students need next in their learning?

Below, I've included some samples of students' thinking:

Student F Pre:

Student F Post:

Student N Pre:

Student N Post:

Student O Pre:

Student O Post:

Although most students showed they learned a great deal during this lesson, Student P did not do as well on the post-assessment as he did on the pre. Using this quantitative data, a teacher must look more deeply at the student's response and reflect:

• What kinds of thinking is this student demonstrating?
• What about the teaching had a negative effect on this student's learning that should be avoided in the future?
• What does this student need next in their learning?

Student P Pre:

Student P Post:

Being able to calculate effect size to determine what works best for individual students is powerful and has the potential to transform the ways which we respond to students. How could you use Hattie's math to determine the impact of your instruction on individual students' achievement?