KNOW THY IMPACT: Using Hattie's Math to Determine Your Impact

John Hattie is an invaluable researcher in the field of education. His way of looking at the impact of particular influences on student achievement helps educators around the world shift the conversation from what works to what works best.

http://www.education.vic.gov.au/Documents/about/research/ravisiblelearning.pdf

Despite this, Hattie's research can sometimes seem removed from life in the classroom as the effect sizes that he presents in his books are based on very large research studies. Teachers may lament that it is hard to know if the large effect sizes of some the most impactful influences can be replicated in their classrooms, with their unique students.

Until now.

In his book Visible Learning for Literacy (2016) that he co-authored with Fisher & Frey, Hattie encourages teachers to reflect on the impact of their own instruction and presents a formula for calculating effect size in their classrooms.

Being able to calculate impact in this way gives teachers the mathematical ability to quantitatively see if instruction is having an impact on their students' achievement and who is not being impacted too. Armed with this information, teachers are able to adapt their teaching as to maximize their effectiveness for the benefit of all students in their classrooms.

In order to calculate effect size, Hattie suggests that:

• Lessons have clear learning intentions.
• Lessons have clear success criteria.
• The success criteria indicate what quality looks like.
• Students know where they stand in relation to the criteria for success (p. 136).

With this in place, teachers only need a pre-assessment and a post-assessment score to be able to calculate effect size.

Below, I demonstrate how to calculate effect size by analyzing students' thinking from an inquiry lesson I recently taught with third graders. The lesson is fully described here: Inquiry into Moon Phases.

What did I want students to learn?

During the lesson, our goal was to answer this essential question:

What would success look like?

A successful response will ...

• Contain academic science vocabulary related to the lesson (1 pt awarded for inclusion of each of the following:
• observe, Earth, Moon, change, orbit, shadow, light, new moon, crescent moon, full moon, phases*, Sun*, reflecting*, darker*, lighter*
• *Not introduced during the lesson, but still important scientific terms that came out during the post-assessment.
• Contain different ideas (1 pt awarded for each complete idea)
• examples of complete ideas are: the moon orbits the earth, the moon reflects the sun's light, the shadow gets bigger as the light gets smaller, the full moon is when the moon is all it up).
• Be accurate
• 4 pts awarded for accurate statements with details/evidence,
• 2 pts awarded for semi-accurate statements with little details/evidence and some misconceptions
• 0 pts awarded for inaccurate statements with no details/evidence and many misconceptions

How do I know they've learned?

Assessment task: The task for the pre-assessment (measure of what students initially understood, knew and could do) was the same as the post-assessment (measure of what they learned). For both, I prompted:

• "Write what you think the answer to our essential question is. Make sure to include scientific vocabulary in your response."

To calculate effect size (p 138)

1. Analyze the pre- and post-assessments.

This step was a snap, thanks to the pre-established success criteria. I recorded these results in a Google Sheet:

	Total pre	Total post
Student A	9	13
Student B	8	12
Student C	5	11
Student D	3	6
Student E	6	14
Student F	2	12
Student G	3	13
Student H	7	13
Student I	7	10
Student J	6	10
Student K	3	16
Student L	3	12
Student M	2	13
Student N	3	13
Student O	2	11
Student P	6	4
Student Q	2	14
Student R	4	11

2. Find the average of the pre- and post-assessments

Using the average formula (=AVERAGE) this step was easy too!

• Average pre: 4.50
• Average post: 11.56

3. Calculate the standard deviation for the pre- and post-assessment and then find the average of the two standard deviations.

This step was super simple too, as the Standard Deviation formula is just (=STDEV).

• Standard Deviation pre: 2.28
• Standard Deviation post: 2.83
• Average Standard Deviation: 2.56

4. Determine effect size

Using Hattie's formula: (Average Post - Average Pre) / Average Standard Deviation

• Effect size: 2.76

This effect size is quite sizable and is most definitely off the scale of the Barometer of Influence that Hattie presents in his work. Some things to consider:

• Whereas this is a large effect size, it is just a number. With this quantitative data, a teacher should also reflect qualitatively:
• In what ways did the students grow the most?
• What about the lesson was successful that should be replicated?
• What wasn't successful that can be eliminated?
• This was just one lesson and the sample size is minute, compared to the studies Hattie typically meta-analyzes. Therefore, little relative importance should be placed on this effect size. After several weeks of working on making scientific observations using scientific language, another assessment could be administered to see if students have continued to show growth with this skill.

5. Determine individual effect sizes

Although the impact of this lesson was quite high on average, that is not necessarily the case for all students. By calculating individual effect sizes using individual assessment scores and the average Standard Deviation, you can see for whom this lesson was successful and for whom it was not.

	Individual Effect Sizes
Student A	1.56
Student B	1.56
Student C	2.35
Student D	1.17
Student E	3.13
Student F	3.91
Student G	3.91
Student H	2.35
Student I	1.17
Student J	1.56
Student K	5.08
Student L	3.52
Student M	4.30
Student N	3.91
Student O	3.52
Student P	-0.78
Student Q	4.69
Student R	2.74

The effect sizes of these individual students is also beyond the scale of Hattie's Barometer of Influence presented in his work. I don't believe that is important though. What is important is to look at the individual effect sizes in relation to one another along with looking at students' thinking and reflect:

• What causes one student (student F, for instance) to make sizable gains, whilst another student (like A or B) just grew marginally?
• What kinds of thinking are these students demonstrating?
• What about the teaching made such an impact on these students that could be replicated in the future?
• What about the teaching caused other students to not gain as much that should be avoided or adapted in the future?
• What do these students need next in their learning?

Below, I've included some samples of students' thinking:

Student F Pre:

Student F Post:

Student N Pre:

Student N Post:

Student O Pre:

Student O Post:

Although most students showed they learned a great deal during this lesson, Student P did not do as well on the post-assessment as he did on the pre. Using this quantitative data, a teacher must look more deeply at the student's response and reflect:

• What kinds of thinking is this student demonstrating?
• What about the teaching had a negative effect on this student's learning that should be avoided in the future?
• What does this student need next in their learning?

Student P Pre:

Student P Post:

Being able to calculate effect size to determine what works best for individual students is powerful and has the potential to transform the ways which we respond to students. How could you use Hattie's math to determine the impact of your instruction on individual students' achievement?

Helping Students be Successful by using the Gradual Release of Responsibility Model

I recently had the opportunity to teach an inquiry lesson to third graders, during which we explored how the moon appears to change during the month. (The lesson is fully described here: Inquiry into Moon Phases). The essential question that we were seeking to answer during the lesson was:

One of the goals of the lesson was that students would be able to accurately describe how the moon looks like it changes during the month and provide evidence/details about what is really happening. In order to get them to meet this goal, I used the Gradual Release of Responsibility model, as described by Doug Fisher in the article Effective Use of the Gradual Release of Responsibility Model.

Focus Lesson/Modeling (I DO IT)
After walking around a model of the moon in the dark (with a flashlight pointed at it) and observing how the moon looks like it changes, we returned to the classroom and began to draw our observations out on a Moon Calendar (we used pictures to help us remember). After week one, we paused so I could model how to make a scientific statement.

On the board, I wrote the sentence stem, "I observe ..." and modeled how I would describe how the moon looked like it changed during week #1.

I said, "I observe that the moon looks like it is changing because the shadow on the moon is getting bigger."

I asked students to notice which scientific words they heard and added those to a bank of science words on the board, by the sentence stem. I added moon, changing and shadow.

Guided Instruction (WE DO IT)

We continued to draw our observations out on the Moon Calendar. After week two, we paused to make a scientific statement together.

I asked for volunteers to describe how the moon looked like it changed during week #2. I guided the volunteers to use the sentence stem, "I observe ..." and as many scientific words as possible. When students said scientific words that weren't yet in our word bank, we added them (eventually that list grew to 9 words).

As I guided different volunteers to describe how the moon looked like it changed during week two, my responsibility as the teacher was to:

• encourage them
• celebrate their effort and successes
• give feedback on how they could improve by
• adding more scientific words
• including additional new ideas
• addressing misconceptions

Collaborative Learning (YOU DO IT TOGETHER)

We continued to draw our observations out on the Moon Calendar. After week three, we paused to so that pairs of students could make a scientific statement together.

I asked the students to think about how the moon looked like it changed during week #3. I reminded them to use the sentence stem, "I observe ..." and as many scientific words as possible. After students had a chance to think, I invited them to pair up and share their scientific sentence with a partner.

After 1-2 minutes, I signaled all the students back together and randomly chose 3 different pairs. As one partner said their scientific sentence, the other partner was in charge of counting how many scientific words they used. 

As the different pairs described how the moon looked like it changed during week three, my responsibility as the teacher was to:

• encourage them
• celebrate their effort and successes
• give feedback on how they could improve by
• adding more scientific words
• including additional new ideas
• addressing misconceptions

Independent Learning (YOU DO IT ALONE)

We continued to draw our observations out on the Moon Calendar. After week four, we paused to so that individual students could make a scientific statement.

I asked the students to think about how the moon looked like it changed during week #4. I reminded them to use the sentence stem, "I observe ..." and as many scientific words as possible. After students had a chance to think, I randomly chose 3 different students to share their scientific sentences with the class. As each student shared, the rest of the class counted how many scientific words they used.

As the different students described how the moon looked like it changed during week four, my responsibility as the teacher was to:

• encourage them
• celebrate their effort and successes
• give feedback on how they could improve by
• adding more scientific words
• including additional new ideas
• addressing misconceptions

Post-Assessment

After gradually releasing responsibility to the students to describe how the moon looked like it changed during the month, I asked them to respond to our essential question (just as I had at the onset of the lesson for a pre-assessment):

The difference between students' response to this question at the beginning of the lesson versus the ending was astonishing. Below is a sample of students' responses. It is clearly evident how much they grew during this lesson in their ability to accurately describe their scientific observations.

Student #1 PRE:

Student #1 POST:

Student #2 PRE:


Student #2 POST:


Student #3 PRE: 


Student #3 POST: 

By slowing and gradually releasing responsibility to students, they were ultimately able to independently describe how the moon changes during the month. How do you successfully use the gradual release of responsibility model in your own classroom?

Word Wallets

At a recent professional development, teachers were given the opportunity to REFLECT on what they had learned, brainstorm a list of actions they could CHOOSE based on those reflections and then pick one and ACT on it.

One kindergarten teacher collaborated with a colleague to use Word Wallets as a way to track student progress and to celebrate the successful learning of sight words. Using these wallets also builds confidence and encourages students to continue to push themselves.

Being able to read sight words is an important reading skill. In the PYP, reading is categorized as a communication skill, one of the five sets of approaches to learning necessary for students to become life-long learners.

The teacher took a folder, cut about two thirds the way down the fold, rounded off the tops of both the sides and folded the tops down. She labeled one side "working on" and the other side "done ☺"

Students are working on learning sight words, five at a time. When they've mastered a word, they celebrate by coloring in the box.

When all five words are colored, they celebrate and move the words over to the "done ☺" side. The students continue to practice the sight words even when they're on the "done ☺" side.

Once students have mastered several strips of words, the teacher plans to make a Sight Word Crown so that students can prominently and proudly display the words they've worked so hard to learn.

When students are reading books appropriate for their developmental and skill level, they'll have an easier time identifying these sight words in context, because they have build up their knowledge of these words and their confidence in their ability to read them.

Word Wallets can, of course, be used to track progress and celebrate successes with other kinds of learning. How might you use these wallets in your classroom? How do you track students' progress and celebrate students' efforts in other ways?