How to define improvement
Claims of school improvement are a dime a dozen.
Most claims I see are not grounded in any kind of reasonable evidence. Worse than that, the claims are often made without a logical definition of improvement.
In this article, I’m going to teach you a three-part definition that will help you easily tell when improvement has occurred and when it hasn’t.
But first, let’s start with a challenge
An Improvement Challenge
Get a piece of paper and pen.
Think about the successful improvement efforts that you’ve led throughout your career.
Maybe you have led an effort to increase state test scores, student enrollment, or teacher retention rates. Or, maybe you have led an effort to decrease the number of discipline referrals or perhaps the chronic absenteeism rates in your school.
Create a list of all of those instances. How many examples do you have on your list? I’m guessing many of you could come up with 10 or 15 with no problem.
Now, pause and think about what it means to “improve”.
What is Improvement?
This brings us to an important question, that is: What is improvement?
My hunch is that the vast majority of educational leaders do not have an answer for this seemingly simple question. I know I didn’t for the first part of my career.
For this I turn to a seminal work in the field of improvement science, The Improvement Guide.
The Improvement Guide, 1st & 2nd Editions
Notice what it says up by the arrow on the left side of the figure above. The book on the left is the second edition of The Improvement Guide.
There was a first edition, but soon after they wrote it, the authors received a note from a professor in Brazil. It read, “I know you guys and you're really big on operational definitions, but you've written this book on improvement and nowhere have you defined what you mean by improvement.”
Talk about a swing and a miss!
In retrospect, it’s obvious that a book called The Improvement Guide should have a clear definition for the central concept that is right in the title. It should be just as obvious that leaders need a definition of improvement, and that the definition should precede any improvement claims.
What definition did the authors come up with in their improved 2nd edition?
Defining Improvement
The definition they come up with has three parts. It is the same definition I use in my own improvement work.
Improvement is:
A change that alters how work is done or the makeup of a tool, that produces visible, positive differences relative to historical norms in relevant measures, sustained into the future.
Let’s break down the three parts of the definition:
Part 1: You can point to a change that was made that led to better results.
Part 2: Performance improved after the change compared to past results.
Part 3: The improvement after the change was sustained into the future.
Think back to your list of successful improvement projects. Would you revise your answer after reading this definition of improvement?
It sets a high bar, but it’s the right bar if you are interested in actually improving outcomes.
Putting the Definition to Work
Here’s a simplified version of most improvement claims.
Claim: “Our state test scores improved.”
Translation: This year’s scores are higher than last year’s scores.
The problem is that claims like this fall far short of our definition.
Let’s instead turn to an example that meets the definition. It will help you understand how powerful this can be in practice.
Let’s suppose that we have been working to increase some student outcome measure that we gather on a monthly basis. The chart below has the data plotted over time, and the three parts of the definition of improvement have been labeled.
Definition of Improvement
It is clear from this illustration that a claim of improvement has been substantiated with evidence. First, there’s a clear point-in-time when a change was introduced (Part 1). Second, there’s a clear difference in results after the change was introduced as compared to historical results (Part 2). Third, the improvement is sustained into the future (Part 3).
There’s a profound difference between most improvement claims and one supported by the three-part definition of improvement.
Summary
Most educational leaders do not have a clear definition for the concept of improvement. Three big ideas from this article can help put you on the right track.
Big Idea 1: Most improvement claims lack evidence.
Big Idea 2: A clear definition must precede any improvement claims.
Big Idea 3: The three-part definition makes it easy to tell when things have improved.
After applying these big ideas to your school system, you’ll now be able to answer the question - Have we improved? - with conviction.
***
Whenever you’re ready, there are 3 ways I can help you:
Transformation requires a whole new way of thinking. Understanding variation in your most important data is a good place to start. In this free, 20-minute introductory course I will teach you a method that will allow you to react less and improve more, a win-win for educational leaders with limited time and resources.
Learn how to use improvement science rooted in the Deming philosophy to design simple experiments that lead to solutions that actually work in your schools, all without wasting time and money and burning out frontline educators. For education leaders who’ve heard of improvement science but aren’t sure where to begin, I can support you on this journey.
Win-Win is the improvement science text for education leaders. The aim of the book is to equip you with the knowledge and skills needed to use the System of Profound Knowledge, a powerful management philosophy, to lead and improve school systems.
***
John A. Dues is the Chief Learning Officer for United Schools, a nonprofit charter management organization that supports four public charter school campuses in Columbus, Ohio. He is also the author of the award-winning book Win-Win: W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools. Send feedback to jdues@unitedschools.org.