The Past Speaks

The Globe and Mail and other Canadian media sources recently carried stories about how Canadian universities are falling in the global rankings. (see “Canadian universities slide down world ranking scale” 1 October). The Agenda, a Canadian public affairs TV show, appears to have plans for an episode on the subject. The publication of the Times Higher Education rankings has sparked this debate.

Speaking as a Canadian-educated academic who is familiar with both the British higher education system and the management issue of creating incentives for accuracy in ratings, I find it difficult to believe that people still take these rankings seriously.

In thinking about the rankings of universities, we can divide the ranking agencies into three categories: low credibility, medium credibility, high credibility.

Let’s talk about the totally bogus rankings. There are a number of organizations and individuals who produce world rankings of universities. Some lack any credibility. Consider the Shanghai World Rankings of Universities. These are produced by a team of researchers in a dictatorship that is known for widespread corruption and internet censorship. China’s firewall prevents people in the country from accessing Western media websites, including student newspapers. Given that such websites contain a great deal of information that would be relevant to evaluating universities,  the evaluators in Shanghai are in a poor position to do rankings. Another website with world university rankings is produced by an Italian gentleman who lives in London. The said gentleman offers “consulting” services for universities that wish to move up his rankings!!! Moreover, his rankings  lack credibility since so little reputational capital is at stake.

The Times Higher Education rankings are somewhat more credible, as they are produced by big media conglomerate that has some reputational capital to loose: businesspeople pay a premium to access Times data in the belief it is accurate. We can be pretty certain that outright bribery have never influenced the position of a university in the Times HES rankings. However, even these rankings are problematic, as they appear to be skewed towards UK universities, which is natural given that the rankings are made under the supervision of Phil Baty, a graduate of one of the colleges of the University of London! To be fair to Mr Baty and other patriotic British university graduates, the UK does have a handful of good universities.  I’m proud to work at one of the British universities that provides North-American quality education.  The experience of most British students is very different however.

I suspect that the low rankings of Canadian universities are a function of lack of knowledge of Canada rather than malicious bias on the part of the people at the THES.  Although British people are well informed about the United States, they know little about Canada, in part because the BBC does not employ any full-time journalists in Canada, despite having dozens of people in the United States. Many ordinary British people are surprised when they see pictures of Canada’s urban skylines, since this visual evidence of urbanization conflicts with the prevailing stereotypes of Canada.  Academics in the UK display similar, although less extreme, knowledge gaps. A senior British political scientist once remarked to me that he didn’t know the name of “Canada’s President.” The widespread British ignorance of Canada may well reflect a rational allocation of mental effort. However, it means that Canadians should probably ignore whatever British newspapers say about Canadian universities, at least until they start reporting on, say, Canadian federal elections.

The more fundamental problem with any newspaper and magazine ranking universities is the incentive structure, as the rater faces few if any financial consequences for inaccuracy or bias. After the financial meltdown, investors became sceptical of the rankings issued by bond rating agencies such as S&P. Unlike Warren Buffett who puts his money where his mouth is, the analysts at ratings agencies gave triple-A ratings to securities without investing any of their own money. Any person can predict that the price of wheat will go up next year, but I’m more likely to pay attention to him if he’s bet some money on Chicago wheat futures.  These analogies show what is wrong with the Times Higher, Guardian, and other newspaper-generated ratings.

There are some credible ratings of universities, however. The staff of charities such as the Ford and Carnegie Foundations do categorize and evaluate universities before making decisions about the allocation of billions of dollars of funding. Since these organizations have “skin in the game”, the opinions they express via ratings are credible. The newspaper rankings should be ignored, since individuals such as Phil Baty face no penalties for error.