Book Review “The Numbers Game”

I wrote a review on “The Numbers Game” a book on football and statistics for Planet CEU: http://planet.ceu.hu/node/484

Why Large Welfare States Should Use Different University Rankings

Rankings are very fashionable among politicians and policy experts. Universities are no exception to this. Yet they are very controversial, and there are endless fights about a ‘fair’ methodology. Recently, the German political science association even recommended its member institutions not to participate in the most important German ranking any longer

Indeed, loosers of these rankings often complain and maybe sometimes for good reasons. One option is not to produce rankings, but another option is to produce other rankings. I, for once, would love to see an international ranking of the average university in a country. I think this would make a big difference. The reason is that many countries, especially in Europe, but maybe also in Asia have social and political preferences for redistribution and the equalization of standards of living. These countries carefully avoid too much social and regional heterogeneity. Money flows from rich to poor individuals, from strong to weak regions. Under these circumstances, we should not expect universities to reach the top of international rankings. A fairer measure would be how good a university on average would do, compared to an average university in another country.

Currently there is not enough data to do this. But we can do some simple exercises with the available rankings. I choose the Times Higher Education Ranking for 2012-3 with detailed info for the top 200 and less detailed info for the next 200. From this I compute the country average of those universities listed. This gives a different view on who is top and who is not.

We all know that US universities dominate the top. But how do they do on average? If we use the detailed info for the top 200 we see that the US is not top any more but third (See table 1). China and Singapore rank 1 and 2 respectively. This seems exaggerated, and indeed it is. The problem is, we do not get information about the weakest universities, since they do not appear in the ranking. Stats people call this selection bias.

There is no way to avoid the problem, but we can at least extend the list to the top 400 universities (see table 2). If we look at the averages for the top 400 we see that the US is now fifth and Singapore is top (after all there are not that many universities in Singapore). China goes down the table. The Netherlands is now second. These are mock results. I am not saying that these are the real country rankings. But they illustrate the idea.

In more general we see that many European countries improve once we look at the averages of a larger number of universities. Countries that redistribute less go down in the ranking. Japan, Australia are extreme cases. There are important exceptions to this rule, such as Switzerland or Israel, not the most benevolent welfare state, but shooting up in the rankings. And yet, for countries in which the whole polity is built around avoiding excess inequality, it would be wise to focus on averages and not the champions.

 

 

Table 1: Countries Ranked by Average University’s Position in Top200

location	avg. university rank	country rank
China	49	1
Singapore	58	2
United States	86	3
Republic of Korea	90	4
Australia	93	5
Canada	93	6
Sweden	97	7
Japan	99	8
Switzerland	100	9
Netherlands	100	10
Hong Kong	102	11
Finland	109	12
United Kingdom	111	13
South Africa	113	14
France	116	15
Germany	122	16
Belgium	127	17
Denmark	132	18
Taiwan	134	19
Republic of Ireland	149	20
Brazil	158	21
New Zealand	161	22
Austria	162	23
Israel	163	24

 

 

 

 

Table 2 Countries Ranked by Average University’s Position in Top400

location	avg. university rank	country rank
Singapore	58	1
Netherlands	109	2
Switzerland	128	3
Republic of Korea	135	4
United States	155	5
Israel	163	6
Hong Kong	166	7
Sweden	182	8
United Kingdom	183	9
France	188	10
Canada	191	11
Germany	204	12
Belgium	207	13
Brazil	211	14
Denmark	212	15
Russian Federation	226	16
China	228	17
Japan	230	18
Australia	232	19
South Africa	248	20
Turkey	253	21
Norway	260	22
Iceland	263	23
Republic of Irel	265	24
Austria	269	25
Finland	280	26
New Zealand	282	27
Taiwan	291	28
India	292	29
Spain	299	30
Italy	315	31
Czech Republic	326	32
Greece	326	32
Iran	326	32
Saudi Arabia	326	32
Colombia	376	36
Estonia	376	36
Mexico	376	36
Poland	376	36
Portugal	376	36
Thailand	376	36

 

Statistical vs. Substantive Significance

Many pundits, journalists, and interested laymen confuse two important but very different scientific concepts: substantive vs. statistical significance. They are hardly to blame, since scientists often make it very difficult for them to understand what scientists do with statistics. Sometimes scientists do this on purpose because they want to obfuscate, but often it is the result of their own ignorance or sloppiness.

What is at stake? Well, simply said substantive significance is about the size of a relationship/ an effect, whereas statistical significance is about measurement precision (usually based on a sample). These two concepts are answers to two very different questions. Substantive significance asks how much, statistical significance asks how sure we are about it. If you think about it statistical significance is much less interesting for the general public, because it essentially only tells you one thing: the ‚finding‘ a statistician identified is pretty certain. We still don’t know whether the finding is interesting, it may be way too small for you to bother!

The two concepts sometimes are related to each other with substantive significance often implying statistical significance. But they don’t have to. Stephen Ziliak and Deirdre McCloskey have written a fascinating book on the topic showing how the confusion between the two concepts can lead to some fatal consequences. They give examples about medical research in which a statistically significant reduction in depression is substantively too small to justify the purchase of the more expensive drug. In another example, research funded by pharma industry even suppresses fatalities just because there were too few of these to be counted as statistically significant.

But it is not only about health, it happens in all kind of research from natural to social sciences, from engineering to linguistics. In my own field, political science, it seems very frequent, and I don’t deny that I myself get sometimes confused. Media coverage is full of examples. It often starts with ‚new research shows X‘. For instance, new research shows that home teams in football are more likely to win. For a statistician it is important to show that the home advantage is precisely measured and identified. For a club owner this is only of secondary importance. True he also needs to be sure that the statistician did his job well. However, for him it is much more important to know how large the home advantage is, and how safely he or she can rely on that. Do I have a 50% higher chance of winning, or just a 5% chance? In both cases home advantage may be statistically significant, but the substantive difference decides on championship or relegation.

Test yourself. How would you interpret the following sentence full of statistical jargon? ‚In a randomized controlled trial we find that the difference between income levels of those who participated in a microfinance program and those who did not was statistically significant at the 1% level?‘ If you answer here something like microfinance makes a huge difference for people, you were a) wrong because we cannot know on basis of this sentence, b) clearly you are not alone as this happens to many. What this sentence should have read is something like this: ‚We compare income levels between those who participate in a microfinance program and those who don’t. We find that the average difference is large, about 200 Euros, which is equivalent to roughly half the average wage in society X. We also have found that this average difference is not a result due to chance, i.e. it is statistically significant, and not due to other distortions. Hence, we trust our findings.‘

What to do? Well pundits, journalists, policy experts need to sharpen their critical reading of scientific literature. They need to vex scientists to tell them quantities of interest which are easy to interpret. As for us scientists, we need to communicate our findings more clearly.