Numerical Naiveté

Note: This is the first of a five-part series on understanding the concept of variation. Knowledge about variation is one of the components of W. Edwards Deming’s System of Profound Knowledge and is fundamental to correctly interpreting data. 

Dr. Russell Ackoff, the eminent systems thinker said, “Managers do not solve problems, they manage messes.” My take would be more optimistic than Ackoff’s assuming managers knew something about variation. This is in fact why I’ve spent so much time over the last year studying this idea as it fits into W. Edwards Deming’s theoretical framework, the System of Profound Knowledge.

Do Something!

Leaders of most modern organizations-in education, in industry, in healthcare, and in government-encounter an overwhelming amount of information in their day-to-day work, and much of this information comes to us in the form of numbers. Data is literally coming at us in torrents like a rushing river! This implies that one of the challenges for leaders of today’s organizations is the sheer volume of data we have at our fingertips. Technological advancements have allowed us to organize this data information in sophisticated spreadsheets and databases and fancy visualization tools such as data dashboards sit on top of those storage tools. But what do we do with this data as it inevitably moves up and down from week-to-week, month-to-month, and year-to-year? 

It is very likely that most of us are doing what Deming cautioned against when he said:

If you don’t understand variation and how it comes from the system itself, you can only react to every figure. The result is you often overcompensate, when it would have been better to just leave things alone.

Think about what happens as you look at data that comes to you regularly in a school setting. The monthly attendance report for the school in which you are the principal has come across the superintendent’s desk. The attendance rate is down this month! Or, maybe the annual state test scores have just been released. Reading scores decreased in 6th grade from last year to this year! The form of what comes next from the top is something along the lines of “Don’t just stand there, do something!” 

The delivery of that message may vary a bit depending on your district’s culture, but the gist of the message is likely unchanged from this characterization. And so, you move into action. You may shoulder the blame yourself or look for a scapegoat. You may type up a lengthy email with detailed bullet points attempting to explain the results. You may go back to your team with steadfast resolve to get to the bottom of the decline in attendance rate or test scores and to set new goals for the next month or the next year. It is very likely then that this process repeats itself month after month and year after year. This year it was a decline in 6th grade reading scores and perhaps next year those scores rebound but now the 6th grade math scores have declined. However, have you ever stopped to ask yourself: “does the decline in my data represent something meaningful?” And just as important, you’d ask yourself that same question even if the data was moving in a positive direction.

Understanding Variation

For a moment, let’s take it as fact that the attendance rate and test score declines were not meaningful. I’ll show you how you can tell for sure later in this series. For now, taking this point as true imagine the waste in the form of time and effort that is spent worrying about, explaining, and redesigning plans all in the name of something that didn’t represent a statistically significant change in performance. Scurrying around in reaction to the call to “Just do something!” is likely the modus operandi in most schools and other organizations. Why is this the case if we now have these sophisticated data tools at our fingertips? What underlies all of this wasted time and energy? The answer at least in part is “numerical naiveté” a concept Dr. Donald Wheeler captured perfectly in his book Understanding Variation, aptly subtitled The Key to Managing Chaos. Wheeler puts it this way:

Before information can be useful it must be analyzed, interpreted, and assimilated. In short, raw data have to be digested before they can be useful.

This process of digesting data has been widely neglected at all levels of our educational system. Managers and workers, educators and students, accountants and businessmen, financial analysts and bankers, doctors and nurses, and especially lawyers and journalists all have one thing in common. They come out of their educational experience knowing how to add, subtract, multiply, and divide, yet they have no understanding of how to digest numbers to extract the knowledge that may be locked up inside the data. In fact, this shortcoming is also seen, to a lesser extent, among engineers and scientists.

This deficiency has been called “numerical naiveté.” Numerical naiveté is not a failure with arithmetic, but it is instead a failure to know how to use the basic tools of arithmetic to understand data. Numerical naiveté is not addressed by the traditional courses in the primary or secondary schools, nor is it addressed by advanced courses in mathematics. This is why even highly educated individuals can be numerically naive.

The antidote to this terrible affliction is the component of the System of Profound Knowledge that Deming called Knowledge about Variation. The statistical techniques that underlie understanding variation were pioneered by Dr. Walter Shewhart in the 1920s and thoroughly vetted over the next century by Deming and many others across all sectors. Practicing these techniques with real data at United Schools Network has completely shifted how I use data in my work. I’ll attempt to make the case that they are worthy of your study over the course of this series. Until then, you can learn more about variation in an article I posted in December called Variation is the enemy.

***

John A. Dues is the Chief Learning Officer for United Schools Network, a nonprofit charter-management organization that supports four public charter schools in Columbus. Send feedback to jdues@unitedschoolsnetwork.org.