I wrote a review on “The Numbers Game” a book on football and statistics for Planet CEU: http://planet.ceu.hu/node/484
Tag Archives: statistics
Why Large Welfare States Should Use Different University Rankings
Rankings are very fashionable among politicians and policy experts. Universities are no exception to this. Yet they are very controversial, and there are endless fights about a ‘fair’ methodology. Recently, the German political science association even recommended its member institutions not to participate in the most important German ranking any longer
Indeed, loosers of these rankings often complain and maybe sometimes for good reasons. One option is not to produce rankings, but another option is to produce other rankings. I, for once, would love to see an international ranking of the average university in a country. I think this would make a big difference. The reason is that many countries, especially in Europe, but maybe also in Asia have social and political preferences for redistribution and the equalization of standards of living. These countries carefully avoid too much social and regional heterogeneity. Money flows from rich to poor individuals, from strong to weak regions. Under these circumstances, we should not expect universities to reach the top of international rankings. A fairer measure would be how good a university on average would do, compared to an average university in another country.
Currently there is not enough data to do this. But we can do some simple exercises with the available rankings. I choose the Times Higher Education Ranking for 2012-3 with detailed info for the top 200 and less detailed info for the next 200. From this I compute the country average of those universities listed. This gives a different view on who is top and who is not.
We all know that US universities dominate the top. But how do they do on average? If we use the detailed info for the top 200 we see that the US is not top any more but third (See table 1). China and Singapore rank 1 and 2 respectively. This seems exaggerated, and indeed it is. The problem is, we do not get information about the weakest universities, since they do not appear in the ranking. Stats people call this selection bias.
There is no way to avoid the problem, but we can at least extend the list to the top 400 universities (see table 2). If we look at the averages for the top 400 we see that the US is now fifth and Singapore is top (after all there are not that many universities in Singapore). China goes down the table. The Netherlands is now second. These are mock results. I am not saying that these are the real country rankings. But they illustrate the idea.
In more general we see that many European countries improve once we look at the averages of a larger number of universities. Countries that redistribute less go down in the ranking. Japan, Australia are extreme cases. There are important exceptions to this rule, such as Switzerland or Israel, not the most benevolent welfare state, but shooting up in the rankings. And yet, for countries in which the whole polity is built around avoiding excess inequality, it would be wise to focus on averages and not the champions.
Table 1: Countries Ranked by Average University’s Position in Top200
location | avg. university rank | country rank |
China |
49 |
1 |
Singapore |
58 |
2 |
United States |
86 |
3 |
Republic of Korea |
90 |
4 |
Australia |
93 |
5 |
Canada |
93 |
6 |
Sweden |
97 |
7 |
Japan |
99 |
8 |
Switzerland |
100 |
9 |
Netherlands |
100 |
10 |
Hong Kong |
102 |
11 |
Finland |
109 |
12 |
United Kingdom |
111 |
13 |
South Africa |
113 |
14 |
France |
116 |
15 |
Germany |
122 |
16 |
Belgium |
127 |
17 |
Denmark |
132 |
18 |
Taiwan |
134 |
19 |
Republic of Ireland |
149 |
20 |
Brazil |
158 |
21 |
New Zealand |
161 |
22 |
Austria |
162 |
23 |
Israel |
163 |
24 |
Table 2 Countries Ranked by Average University’s Position in Top400
location | avg. university rank | country rank |
Singapore |
58 |
1 |
Netherlands |
109 |
2 |
Switzerland |
128 |
3 |
Republic of Korea |
135 |
4 |
United States |
155 |
5 |
Israel |
163 |
6 |
Hong Kong |
166 |
7 |
Sweden |
182 |
8 |
United Kingdom |
183 |
9 |
France |
188 |
10 |
Canada |
191 |
11 |
Germany |
204 |
12 |
Belgium |
207 |
13 |
Brazil |
211 |
14 |
Denmark |
212 |
15 |
Russian Federation |
226 |
16 |
China |
228 |
17 |
Japan |
230 |
18 |
Australia |
232 |
19 |
South Africa |
248 |
20 |
Turkey |
253 |
21 |
Norway |
260 |
22 |
Iceland |
263 |
23 |
Republic of Irel |
265 |
24 |
Austria |
269 |
25 |
Finland |
280 |
26 |
New Zealand |
282 |
27 |
Taiwan |
291 |
28 |
India |
292 |
29 |
Spain |
299 |
30 |
Italy |
315 |
31 |
Czech Republic |
326 |
32 |
Greece |
326 |
32 |
Iran |
326 |
32 |
Saudi Arabia |
326 |
32 |
Colombia |
376 |
36 |
Estonia |
376 |
36 |
Mexico |
376 |
36 |
Poland |
376 |
36 |
Portugal |
376 |
36 |
Thailand |
376 |
36 |
Filed under Uncategorized
Statistical vs. Substantive Significance
Many pundits, journalists, and interested laymen confuse two important but very different scientific concepts: substantive vs. statistical significance. They are hardly to blame, since scientists often make it very difficult for them to understand what scientists do with statistics. Sometimes scientists do this on purpose because they want to obfuscate, but often it is the result of their own ignorance or sloppiness.
What is at stake? Well, simply said substantive significance is about the size of a relationship/ an effect, whereas statistical significance is about measurement precision (usually based on a sample). These two concepts are answers to two very different questions. Substantive significance asks how much, statistical significance asks how sure we are about it. If you think about it statistical significance is much less interesting for the general public, because it essentially only tells you one thing: the ‚finding‘ a statistician identified is pretty certain. We still don’t know whether the finding is interesting, it may be way too small for you to bother!
The two concepts sometimes are related to each other with substantive significance often implying statistical significance. But they don’t have to. Stephen Ziliak and Deirdre McCloskey have written a fascinating book on the topic showing how the confusion between the two concepts can lead to some fatal consequences. They give examples about medical research in which a statistically significant reduction in depression is substantively too small to justify the purchase of the more expensive drug. In another example, research funded by pharma industry even suppresses fatalities just because there were too few of these to be counted as statistically significant.
But it is not only about health, it happens in all kind of research from natural to social sciences, from engineering to linguistics. In my own field, political science, it seems very frequent, and I don’t deny that I myself get sometimes confused. Media coverage is full of examples. It often starts with ‚new research shows X‘. For instance, new research shows that home teams in football are more likely to win. For a statistician it is important to show that the home advantage is precisely measured and identified. For a club owner this is only of secondary importance. True he also needs to be sure that the statistician did his job well. However, for him it is much more important to know how large the home advantage is, and how safely he or she can rely on that. Do I have a 50% higher chance of winning, or just a 5% chance? In both cases home advantage may be statistically significant, but the substantive difference decides on championship or relegation.
Test yourself. How would you interpret the following sentence full of statistical jargon? ‚In a randomized controlled trial we find that the difference between income levels of those who participated in a microfinance program and those who did not was statistically significant at the 1% level?‘ If you answer here something like microfinance makes a huge difference for people, you were a) wrong because we cannot know on basis of this sentence, b) clearly you are not alone as this happens to many. What this sentence should have read is something like this: ‚We compare income levels between those who participate in a microfinance program and those who don’t. We find that the average difference is large, about 200 Euros, which is equivalent to roughly half the average wage in society X. We also have found that this average difference is not a result due to chance, i.e. it is statistically significant, and not due to other distortions. Hence, we trust our findings.‘
What to do? Well pundits, journalists, policy experts need to sharpen their critical reading of scientific literature. They need to vex scientists to tell them quantities of interest which are easy to interpret. As for us scientists, we need to communicate our findings more clearly.
Filed under Uncategorized