When I was 12, I decided to help out a friend by reattaching the plug that had accidently been severed from her curling iron. I figured that having a father who was an electrician qualified me for the job and it really didn’t seem that complicated. How much harm could I possibly do? I learned two things that day: Knowing what an electrician does is not the same as being an electrician and incorrectly wired home appliances start fires.
I was reminded of this story this week when I read about the Mississauga realtor who has taken it upon herself to compile a weighted ranking of schools in her region and, as a result, has been fighting some fires of her own.
Susi Kostyniuk saw a deficiency in Fraser Institute school rankings; those scores lacked what she calls consistency because they capture only one year of information (despite being available annually from 2008 to 2012). And so in order to make it easier for families to prediction the quality of schools into the future she created a weighted average of scores over the past seven years using her own (unexplained) weighting system.
Unfortunately, she didn’t stop there.
Convinced through “casual conversations” that some elements of the student population might make teaching difficult, she then decided to reweight the scores a second time to correct for socio-economic conditions in the community.
Basically, some schools had their scores weighted up to give them credit for making the most of a bad situation while other schools had their scores weighted down to take into account that some teachers were lucky enough to be teaching children whose parents “produce students that are easier to teach.”
She describes this reweighting process as “leveling the playing field”, but in her individual school reports she posts her “Teacher Difficulty Index” front and center giving anyone visiting the site the impression that the key piece of information is that some schools are full of difficult kids.
But that is not what her measure captures at all.
Based on the methodology described on her website, Ms. Kostyniuk appears to have used data complied from the 2011 Census of Canada that describes not the children in the specific school, but rather the socio-economic characteristics of households in the neighbourhood as a whole.
What Teacher Difficulty Index actually captures is the number of lone-parent households, the median household income, the percentage of the population with a university degree and the number of households headed by immigrants in that census region – not in the student population for a specific school.
Susi Kostyniuk may know what economists do, but she is definitely not an economist. If she was, she would have known that the information she is providing is not a measure of the quality of children in schools, but rather a measure of the nature of households in the neighbourhood.
And she has done it using characteristics that not all Canadian’s would agree make a community undesirable.
Herein lies the problem. When it comes to markets, accurate information improves the efficiency of those markets in that all players are making informed decisions. On the other hand, inaccurate or misinterpreted information has the potential to distort markets in a way that does not promote the public interests.
If buyers and sellers on real estate markets believed Ms. Kostyniuk’s ranking of schools was credible, this would drive up the value of homes in the communities with high rankings and drive down the value of homes in communities with low rankings.
Given that the family home is the most important asset held by most Canadian families, her ranking has the power to redistribute wealth from household s in low income neighbourhoods to households in high income neighbourhoods.
That’s a lot of power for an index that was not generated by a trained statistician and has never been vetted by the peer review process.
In the past, parents choose the best neighbourhoods on the advice of friends and families. That’s no longer good enough; today, we all want to know what the data has to say before making a major decision. Given that trend, I think we can expect to see more amateur statisticians turning to the publicly available data to provide information to their clients. There is nothing inherently wrong with using data for these purposes, but it is important that everyone (including the researcher) appreciates that it’s not always easy to unravel the stories data has to tell.
Marina Adshade is the author of Dollars and Sex: How Economics Influences Sex and Love and teaches at the University of British Columbia’s Vancouver School of Economics.