KNOW THY IMPACT: Using Hattie's Math to Determine Your Impact

John Hattie is an invaluable researcher in the field of education. His way of looking at the impact of particular influences on student achievement helps educators around the world shift the conversation from what works to what works best.

http://www.education.vic.gov.au/Documents/about/research/ravisiblelearning.pdf

Despite this, Hattie's research can sometimes seem removed from life in the classroom as the effect sizes that he presents in his books are based on very large research studies. Teachers may lament that it is hard to know if the large effect sizes of some the most impactful influences can be replicated in their classrooms, with their unique students.

Until now.

In his book Visible Learning for Literacy (2016) that he co-authored with Fisher & Frey, Hattie encourages teachers to reflect on the impact of their own instruction and presents a formula for calculating effect size in their classrooms.

Being able to calculate impact in this way gives teachers the mathematical ability to quantitatively see if instruction is having an impact on their students' achievement and who is not being impacted too. Armed with this information, teachers are able to adapt their teaching as to maximize their effectiveness for the benefit of all students in their classrooms.

In order to calculate effect size, Hattie suggests that:

• Lessons have clear learning intentions.
• Lessons have clear success criteria.
• The success criteria indicate what quality looks like.
• Students know where they stand in relation to the criteria for success (p. 136).

With this in place, teachers only need a pre-assessment and a post-assessment score to be able to calculate effect size.

Below, I demonstrate how to calculate effect size by analyzing students' thinking from an inquiry lesson I recently taught with third graders. The lesson is fully described here: Inquiry into Moon Phases.

What did I want students to learn?

During the lesson, our goal was to answer this essential question:

What would success look like?

A successful response will ...

• Contain academic science vocabulary related to the lesson (1 pt awarded for inclusion of each of the following:
• observe, Earth, Moon, change, orbit, shadow, light, new moon, crescent moon, full moon, phases*, Sun*, reflecting*, darker*, lighter*
• *Not introduced during the lesson, but still important scientific terms that came out during the post-assessment.
• Contain different ideas (1 pt awarded for each complete idea)
• examples of complete ideas are: the moon orbits the earth, the moon reflects the sun's light, the shadow gets bigger as the light gets smaller, the full moon is when the moon is all it up).
• Be accurate
• 4 pts awarded for accurate statements with details/evidence,
• 2 pts awarded for semi-accurate statements with little details/evidence and some misconceptions
• 0 pts awarded for inaccurate statements with no details/evidence and many misconceptions

How do I know they've learned?

Assessment task: The task for the pre-assessment (measure of what students initially understood, knew and could do) was the same as the post-assessment (measure of what they learned). For both, I prompted:

• "Write what you think the answer to our essential question is. Make sure to include scientific vocabulary in your response."

To calculate effect size (p 138)

1. Analyze the pre- and post-assessments.

This step was a snap, thanks to the pre-established success criteria. I recorded these results in a Google Sheet:

	Total pre	Total post
Student A	9	13
Student B	8	12
Student C	5	11
Student D	3	6
Student E	6	14
Student F	2	12
Student G	3	13
Student H	7	13
Student I	7	10
Student J	6	10
Student K	3	16
Student L	3	12
Student M	2	13
Student N	3	13
Student O	2	11
Student P	6	4
Student Q	2	14
Student R	4	11

2. Find the average of the pre- and post-assessments

Using the average formula (=AVERAGE) this step was easy too!

• Average pre: 4.50
• Average post: 11.56

3. Calculate the standard deviation for the pre- and post-assessment and then find the average of the two standard deviations.

This step was super simple too, as the Standard Deviation formula is just (=STDEV).

• Standard Deviation pre: 2.28
• Standard Deviation post: 2.83
• Average Standard Deviation: 2.56

4. Determine effect size

Using Hattie's formula: (Average Post - Average Pre) / Average Standard Deviation

• Effect size: 2.76

This effect size is quite sizable and is most definitely off the scale of the Barometer of Influence that Hattie presents in his work. Some things to consider:

• Whereas this is a large effect size, it is just a number. With this quantitative data, a teacher should also reflect qualitatively:
• In what ways did the students grow the most?
• What about the lesson was successful that should be replicated?
• What wasn't successful that can be eliminated?
• This was just one lesson and the sample size is minute, compared to the studies Hattie typically meta-analyzes. Therefore, little relative importance should be placed on this effect size. After several weeks of working on making scientific observations using scientific language, another assessment could be administered to see if students have continued to show growth with this skill.

5. Determine individual effect sizes

Although the impact of this lesson was quite high on average, that is not necessarily the case for all students. By calculating individual effect sizes using individual assessment scores and the average Standard Deviation, you can see for whom this lesson was successful and for whom it was not.

	Individual Effect Sizes
Student A	1.56
Student B	1.56
Student C	2.35
Student D	1.17
Student E	3.13
Student F	3.91
Student G	3.91
Student H	2.35
Student I	1.17
Student J	1.56
Student K	5.08
Student L	3.52
Student M	4.30
Student N	3.91
Student O	3.52
Student P	-0.78
Student Q	4.69
Student R	2.74

The effect sizes of these individual students is also beyond the scale of Hattie's Barometer of Influence presented in his work. I don't believe that is important though. What is important is to look at the individual effect sizes in relation to one another along with looking at students' thinking and reflect:

• What causes one student (student F, for instance) to make sizable gains, whilst another student (like A or B) just grew marginally?
• What kinds of thinking are these students demonstrating?
• What about the teaching made such an impact on these students that could be replicated in the future?
• What about the teaching caused other students to not gain as much that should be avoided or adapted in the future?
• What do these students need next in their learning?

Below, I've included some samples of students' thinking:

Student F Pre:

Student F Post:

Student N Pre:

Student N Post:

Student O Pre:

Student O Post:

Although most students showed they learned a great deal during this lesson, Student P did not do as well on the post-assessment as he did on the pre. Using this quantitative data, a teacher must look more deeply at the student's response and reflect:

• What kinds of thinking is this student demonstrating?
• What about the teaching had a negative effect on this student's learning that should be avoided in the future?
• What does this student need next in their learning?

Student P Pre:

Student P Post:

Being able to calculate effect size to determine what works best for individual students is powerful and has the potential to transform the ways which we respond to students. How could you use Hattie's math to determine the impact of your instruction on individual students' achievement?

Helping Students be Successful by using the Gradual Release of Responsibility Model

I recently had the opportunity to teach an inquiry lesson to third graders, during which we explored how the moon appears to change during the month. (The lesson is fully described here: Inquiry into Moon Phases). The essential question that we were seeking to answer during the lesson was:

One of the goals of the lesson was that students would be able to accurately describe how the moon looks like it changes during the month and provide evidence/details about what is really happening. In order to get them to meet this goal, I used the Gradual Release of Responsibility model, as described by Doug Fisher in the article Effective Use of the Gradual Release of Responsibility Model.

Focus Lesson/Modeling (I DO IT)
After walking around a model of the moon in the dark (with a flashlight pointed at it) and observing how the moon looks like it changes, we returned to the classroom and began to draw our observations out on a Moon Calendar (we used pictures to help us remember). After week one, we paused so I could model how to make a scientific statement.

On the board, I wrote the sentence stem, "I observe ..." and modeled how I would describe how the moon looked like it changed during week #1.

I said, "I observe that the moon looks like it is changing because the shadow on the moon is getting bigger."

I asked students to notice which scientific words they heard and added those to a bank of science words on the board, by the sentence stem. I added moon, changing and shadow.

Guided Instruction (WE DO IT)

We continued to draw our observations out on the Moon Calendar. After week two, we paused to make a scientific statement together.

I asked for volunteers to describe how the moon looked like it changed during week #2. I guided the volunteers to use the sentence stem, "I observe ..." and as many scientific words as possible. When students said scientific words that weren't yet in our word bank, we added them (eventually that list grew to 9 words).

As I guided different volunteers to describe how the moon looked like it changed during week two, my responsibility as the teacher was to:

• encourage them
• celebrate their effort and successes
• give feedback on how they could improve by
• adding more scientific words
• including additional new ideas
• addressing misconceptions

Collaborative Learning (YOU DO IT TOGETHER)

We continued to draw our observations out on the Moon Calendar. After week three, we paused to so that pairs of students could make a scientific statement together.

I asked the students to think about how the moon looked like it changed during week #3. I reminded them to use the sentence stem, "I observe ..." and as many scientific words as possible. After students had a chance to think, I invited them to pair up and share their scientific sentence with a partner.

After 1-2 minutes, I signaled all the students back together and randomly chose 3 different pairs. As one partner said their scientific sentence, the other partner was in charge of counting how many scientific words they used. 

As the different pairs described how the moon looked like it changed during week three, my responsibility as the teacher was to:

• encourage them
• celebrate their effort and successes
• give feedback on how they could improve by
• adding more scientific words
• including additional new ideas
• addressing misconceptions

Independent Learning (YOU DO IT ALONE)

We continued to draw our observations out on the Moon Calendar. After week four, we paused to so that individual students could make a scientific statement.

I asked the students to think about how the moon looked like it changed during week #4. I reminded them to use the sentence stem, "I observe ..." and as many scientific words as possible. After students had a chance to think, I randomly chose 3 different students to share their scientific sentences with the class. As each student shared, the rest of the class counted how many scientific words they used.

As the different students described how the moon looked like it changed during week four, my responsibility as the teacher was to:

• encourage them
• celebrate their effort and successes
• give feedback on how they could improve by
• adding more scientific words
• including additional new ideas
• addressing misconceptions

Post-Assessment

After gradually releasing responsibility to the students to describe how the moon looked like it changed during the month, I asked them to respond to our essential question (just as I had at the onset of the lesson for a pre-assessment):

The difference between students' response to this question at the beginning of the lesson versus the ending was astonishing. Below is a sample of students' responses. It is clearly evident how much they grew during this lesson in their ability to accurately describe their scientific observations.

Student #1 PRE:

Student #1 POST:

Student #2 PRE:


Student #2 POST:


Student #3 PRE: 


Student #3 POST: 

By slowing and gradually releasing responsibility to students, they were ultimately able to independently describe how the moon changes during the month. How do you successfully use the gradual release of responsibility model in your own classroom?

Word Wallets

At a recent professional development, teachers were given the opportunity to REFLECT on what they had learned, brainstorm a list of actions they could CHOOSE based on those reflections and then pick one and ACT on it.

One kindergarten teacher collaborated with a colleague to use Word Wallets as a way to track student progress and to celebrate the successful learning of sight words. Using these wallets also builds confidence and encourages students to continue to push themselves.

Being able to read sight words is an important reading skill. In the PYP, reading is categorized as a communication skill, one of the five sets of approaches to learning necessary for students to become life-long learners.

The teacher took a folder, cut about two thirds the way down the fold, rounded off the tops of both the sides and folded the tops down. She labeled one side "working on" and the other side "done ☺"

Students are working on learning sight words, five at a time. When they've mastered a word, they celebrate by coloring in the box.

When all five words are colored, they celebrate and move the words over to the "done ☺" side. The students continue to practice the sight words even when they're on the "done ☺" side.

Once students have mastered several strips of words, the teacher plans to make a Sight Word Crown so that students can prominently and proudly display the words they've worked so hard to learn.

When students are reading books appropriate for their developmental and skill level, they'll have an easier time identifying these sight words in context, because they have build up their knowledge of these words and their confidence in their ability to read them.

Word Wallets can, of course, be used to track progress and celebrate successes with other kinds of learning. How might you use these wallets in your classroom? How do you track students' progress and celebrate students' efforts in other ways?

Teaching writing authentically: the WHAT, the WHY, & the HOW

This post was written collaboratively by elementary teachers enrolled in the Alternative Pathway To Teaching program at the University of Minnesota in partnership with Teach For America. The post was edited & stitched together by Ryan Higbea, one of their instructors. As part of CI 5214: Elementary Education Content and Pedagogy IV, teachers are working to understand writing and genre instruction in the elementary school. This post is a synthesis of chapters 1, 2, 4 & 6 of Reading and Writing Genre with Purpose in K-8 Classrooms by Duke, Caughlan, Juzwik, and Martin. Page references throughout the post refer to this text.

Introductory Thoughts

by Gwen

Teaching writing with purpose allows students to see connections between the writing they are learning and their daily lives, while explicitly learning key skills. Genres are not taught in a specific sequence but rather reflect the mixture of text types they’ll see in their daily lives. Writing with purpose allows students to become motivated to write and will engage students to work hard and become more curious about their writing.

The 5 principles (p. 3) that help guide writing instruction are:

• Create an environment that welcomes all communication
• Introduce, through exposure, different types of text
• Explicitly teach genre features.
• Explicitly teach genre-specific or genre-sensitive strategies
• Offer ongoing coaching and feedback

These principles are designed to engage students at different levels to learn these main types of texts: narrative, informative/exclamatory, and persuasive.

What are narrative texts?

by Hannah Bates

Narrative texts share stories about a variety of experiences, for a variety of reasons. Narrative writing is authored by people who have knowledge about a particular experience and can be fiction or nonfiction. It's important to teach narrative writing to empower students to write about their own lived experiences, or historical experiences that have affected them.

When teaching narrative writing, create meaningful assignments that are "larger" than the students themselves. Make them collaborate about something meaningful that will allow them to truly engage authentically. Use mentor texts and modeling to expose students to the components of narrative. Teach genre features, such as setting, plot, details, language, etc. Teach strategies for effectively making sense of narrative texts, like visualizing or rereading for clarity, (p. 22-51).

In the Minnesota Academic Standards for English Language Arts (ELA) K-12, the third benchmark in the writing strand (x.6.3.3) is related to narrative writing.

Narrative – the HOW

by Sarah Ehlen

In my own classroom, after explaining the “what” of narrative, using mentor texts in a variety of narrative types has been crucial in teaching the “HOW.” Young students are natural storytellers, filled with experiences and stories to share. By reading mentor texts and highlighting published authors' “craft moves,” students are able to see different methods of bringing stories to life on the page.

Exemplar mentor texts (like On My Way to Buy Eggs by Chih-Yuan Chen) exposes students to different aspects of narrative writing and gives them an anchor when they want to try a new element of narrative writing themselves. A set of mentor texts is necessary, as well as a variety of paper choices, illustration tools and writing instruments. Students of all stages of writing development should have access to writing materials that allow them to test out craft moves and tell their own stories. Older students, should also use different forms of technology.

Once students begin reading, publishing and sharing their writing, other ELA benchmarks are addressed (x.8.1.1 for example).

The WHAT of Informative/Explanatory

by Callea

“Informational and explanatory” is a broad genre that consists of many sub genres, including but not limited to:

• Textbooks
• Newspaper articles
• Magazines
• Journals
• Websites
• Letters
• Informational pamphlets

Authors of these texts use strategies that we should teach students to use:

• Research by reading texts, interviewing and observing
• Preview texts by skimming and scanning
• Make clear and concise notes
• Summarize
• Capture the gist
• Organize information

The intended audience should be the driving force behind deciding what type of informational text the author will create.

• A textbook for children?
• An informational pamphlet for parents?
• A guidebook for visitors?

Students should get exposure to the text types they are creating and take note of the text features that are predominantly used. These text features can include:

• Information boxes
• Pictures
• Graphics
• Graphs and data
• Paragraphs carefully organized by topic

MN ELA benchmarks related to reading and writing information texts are:

• Informational text benchmarks:
• x.2.1.1
• x.2.1.2
• x.2.5.5
• x.2.6.6
• Writing:
• x.6.2.2
• x.6.4.4
• x.6.8.8

The “How” of Informational Writing

by Will McDuffie

There are five “principles” for teaching informational writing to students (p. 84-108):

• Create a compelling, meaningful environment 
• Introduce interesting topics and graphics
• Design projects around pressing issues
• Provide a real audience
• Follow through (i.e. if you tell students you’re sending their writing to the president, really send it)
• Provide exposure and experience
• Give kids either a teacher model or a mentor text
• Explicitly teach the features of informational texts
• Examples of these features:
• Table of contents, headings/subheadings, index
• Final summary and closing statement
• Frequent repetition of the topic of the text
• Technical vocabulary
• Graphical devices like timelines, diagrams, and flowcharts
• Be influenced by the students’ writing, the needs of their audience, and state benchmarks - not the sequence of mini-lessons from the published curriculum.
• Explicitly teach genre-specific or genre-sensitive strategies
• Strategies for reading and listening
• Strategies for writing and speaking
• Researching
• Planning
• Revising
• Offer Ongoing Coaching and Feedback
• Small group and one-on-one settings

The WHAT of Persuasive

by Freda

The purpose of persuasive text is to influence the reader’s opinion - either subtly or blatantly. Persuasive text is found in many different types in our daily life:

• Editorials
• Blogs
• Magazine articles
• Pamphlets
• Literary essays
• Poetry
• Letters
• Speeches
• Surveys
• Commercials
• Grocery flier
• Campaign flier

Persuasive assignments are often given to students without any real purpose or audience. If students feel their writing has a legitimate purpose and an audience beyond simply the teacher for a grade, they are more likely to be passionate, excited and highly involved in the actual writing because it matters.

Years ago, my son’s teacher challenged her second graders to think of their favorite restaurant and the reasons why it is their favorite (related MN ELA related benchmark: 2.6.1.1). Students wrote a persuasive piece attempting to influence one of their classmates. My son was so excited about this assignment. 

My son picked Smashburger. Students got to share their essays with one another, they were posted in the classroom and copies were mailed to the local restaurants.

Weeks later, my son asked if I would take him to Smashburger for dinner. Upon receipt of my son’s work, the local Smashburger sent coupons to the school as a way of thanking him for his writing. When we redeemed the coupon, I thanked the cashier and she thanked my son for his writing sample!

Now that I am a teacher myself, I think of the extra effort that my son’s teacher went to for this persuasive writing piece to ensure that the assignment had actual purpose to increase student involvement and passion about the writing.

Features that we should teach include:

• Knowing your audience
• Using vocabulary 
• Being creative with arguments
• Beginning with a great launch or “hook” 
• Including rebuttal responses 
• Ending with a strong conclusion

The HOW of Persuasive

by Colin

Several actions MUST be done to teach students HOW to write with persuasion and to support them in their knowledge of a persuasive text. Finding a situation that engages students is essential and once the problem has been identified, find something your students can do to contribute to a solution (p. 142). To identify those problems, think global, act local; in other words, address a real need in their community.

Provide

• Access to resources
• reading texts, watching videos, listening to others’ experiences
• Opportunities to collaborate
• get students talking so they can communicate thoughts and opinions
• Modeling
• model texts from the teacher or the students to show what their work could look like. This will help students produce quality work and meet expected outcomes
• Opportunities to practice
• Give students the chance to communicate effectively, construct arguments with purpose and reasoning, and connect with their audience
• Supports
• graphic organizers (like this one or a bubble map) and support systems can help students structure their arguments

Concluding Thoughts

by Katie

Teachers should keep in mind what being authentic looks like in practice. We may teach the same benchmarks, but we do not use the same goal posts. The needs of students are not necessarily equal. The innately personal aspect of writing necessitates careful contemplation of the needs of our specific students, and just as students must think of their audience when writing, so too should us teachers when planning a project.

The more impassioned by the topic students are, the closer we inch to true engagement. Conversely, without analysis of what an authentic topic means to our specific students, the amount of engagement plummets.

“Children aren’t born knowing”, (p 94), but what they are born with is a need to express, and teaching writing authentically does just that.