The Past Speaks

AS: Last week, I blogged about some of the systems used for evaluating the quality of management-school journals. Laurent Ortmans, the FT journalist responsible for the rankings, tried to post a comment in response but for unknown technical reasons neither it nor my (lengthy) response showed up in the Comments section. I’m reposting his comments and an abbreviated version of my reply here.

Hello Andrew,

Your post contains factual errors about the Financial Times.

It is misleading to state that the FT inadvertently revealed my name. It is public information that I am the statistician in charge of the FT business school rankings.

The list of 50 research journals (FT50 list) itself is not a ranking. We do not rank research journals.

We use the articles published in these journals to produce the FT research rank, one of about 20 different criteria that inform our three MBA rankings. See our MBA methodology https://www.ft.com/content/72b3a752-d9be-11e6-944b-e7eb37a6aa8e

The very first list was initially put together by the business schools that took part in our first MBA ranking back in 1999. Since then, the FT has consulted participating schools every time the list was updated, on the basis of one vote per school.

It is correct that some journals contacted me. However, the revised list is based on the schools’ votes only. Incidentally, Liverpool School of Management was consulted and did contribute to our review process.

For your reference, we have published a short methodology https://www.ft.com/content/3405a512-5cbb-11e1-8f1f-00144feabdc0

Hi Laurent, Thanks for getting in touch. That takes guts.

I suppose the decision of the FT to publish your name along with the list wasn’t inadvertent.  It’s true they didn’t really say much about the people making the list.

I’m glad to hear that the lobbying by journals and scholarly organizations didn’t cause you to bend the rules outlined in your methodology.

There was a “short” methodology published, as you note. As I and other said online at the time of its publication, it is far too brief. Here is why I think it is far too brief. There are thousands of biz schools in the world and you only contacted 200. That’s fine, but neither the sampling methodology nor the geographical distribution of the schools is mentioned. What percentage are in each country? How is the weighting done?  Moreover, the way in which the “schools” were contacted wasn’t explained and the text of your communication (probably a mass email) wasn’t published, so we can’t examine its wording, which is crucial in opinion polling. The 67% response rate is mentioned, which is great, but the differences in the response rates between management schools in different countries, etc isn’t specified. To be fully transparent, you should list all of the management schools contacted and all of the one’s that didn’t reply. The date of contact should also probably be mentioned too, since this could affect the response rate. Was the response rate for non-US schools different from US schools?  That’s the type of thing people want to know.

How did each school go about forming its opinion? What were your instructions to the school? How was your contact person in each school? That’s something that should be discussed here as well.

I’m not going to compare this survey of management school opinion to the absurd online polls that tabloid papers run to generate statistics that support their editorial positions.  It reminds me of the Literary Digest polls that were used in the 1920s to predict the results of US presidential elections—these were polls of newspaper editors across the country and each editor was asked “how do people in your town plan to vote?” Gallup polling came along later and got steadily more scientific.

I suggest that next time your preregister your study and its methodology and use Open Data to increase the transparency and legitimacy of your findings. There is a cool organization that promotes the pre-registration of social scientific research.

https://cos.io/prereg/

I will say that the FT polling methodology has less room for bias than the some of the other journal lists that I mocked in my post. And the FT50 is useful for people trying to get promoted- useful intel.