Tag Archives: statistics

Book Review “The Numbers Game”

I wrote a review on “The Numbers Game” a book on football and statistics for Planet CEU: http://planet.ceu.hu/node/484

1 Comment

Filed under Uncategorized

Why Large Welfare States Should Use Different University Rankings

Rankings are very fashionable among politicians and policy experts. Universities are no exception to this. Yet they are very controversial, and there are endless fights about a ‘fair’ methodology. Recently, the German political science association even recommended its member institutions not to participate in the most important German ranking any longer

Indeed, loosers of these rankings often complain and maybe sometimes for good reasons. One option is not to produce rankings, but another option is to produce other rankings. I, for once, would love to see an international ranking of the average university in a country. I think this would make a big difference. The reason is that many countries, especially in Europe, but maybe also in Asia have social and political preferences for redistribution and the equalization of standards of living. These countries carefully avoid too much social and regional heterogeneity. Money flows from rich to poor individuals, from strong to weak regions. Under these circumstances, we should not expect universities to reach the top of international rankings. A fairer measure would be how good a university on average would do, compared to an average university in another country.

Currently there is not enough data to do this. But we can do some simple exercises with the available rankings. I choose the Times Higher Education Ranking for 2012-3 with detailed info for the top 200 and less detailed info for the next 200. From this I compute the country average of those universities listed. This gives a different view on who is top and who is not.

We all know that US universities dominate the top. But how do they do on average? If we use the detailed info for the top 200 we see that the US is not top any more but third (See table 1). China and Singapore rank 1 and 2 respectively. This seems exaggerated, and indeed it is. The problem is, we do not get information about the weakest universities, since they do not appear in the ranking. Stats people call this selection bias.

There is no way to avoid the problem, but we can at least extend the list to the top 400 universities (see table 2). If we look at the averages for the top 400 we see that the US is now fifth and Singapore is top (after all there are not that many universities in Singapore). China goes down the table. The Netherlands is now second. These are mock results. I am not saying that these are the real country rankings. But they illustrate the idea.

In more general we see that many European countries improve once we look at the averages of a larger number of universities. Countries that redistribute less go down in the ranking. Japan, Australia are extreme cases. There are important exceptions to this rule, such as Switzerland or Israel, not the most benevolent welfare state, but shooting up in the rankings. And yet, for countries in which the whole polity is built around avoiding excess inequality, it would be wise to focus on averages and not the champions.

 

 

Table 1: Countries Ranked by Average University’s Position in Top200

location avg. university rank country rank
China

49

1

Singapore

58

2

United States

86

3

Republic of Korea

90

4

Australia

93

5

Canada

93

6

Sweden

97

7

Japan

99

8

Switzerland

100

9

Netherlands

100

10

Hong Kong

102

11

Finland

109

12

United Kingdom

111

13

South Africa

113

14

France

116

15

Germany

122

16

Belgium

127

17

Denmark

132

18

Taiwan

134

19

Republic of Ireland

149

20

Brazil

158

21

New Zealand

161

22

Austria

162

23

Israel

163

24

 

 

 

 

Table 2 Countries Ranked by Average University’s Position in Top400

location avg. university rank country rank
Singapore

58

1

Netherlands

109

2

Switzerland

128

3

Republic of Korea

135

4

United States

155

5

Israel

163

6

Hong Kong

166

7

Sweden

182

8

United Kingdom

183

9

France

188

10

Canada

191

11

Germany

204

12

Belgium

207

13

Brazil

211

14

Denmark

212

15

Russian Federation

226

16

China

228

17

Japan

230

18

Australia

232

19

South Africa

248

20

Turkey

253

21

Norway

260

22

Iceland

263

23

Republic of Irel

265

24

Austria

269

25

Finland

280

26

New Zealand

282

27

Taiwan

291

28

India

292

29

Spain

299

30

Italy

315

31

Czech Republic

326

32

Greece

326

32

Iran

326

32

Saudi Arabia

326

32

Colombia

376

36

Estonia

376

36

Mexico

376

36

Poland

376

36

Portugal

376

36

Thailand

376

36

 

1 Comment

Filed under Uncategorized

Statistical vs. Substantive Significance

Many pundits, journalists, and interested laymen confuse two important but very different scientific concepts: substantive vs. statistical significance. They are hardly to blame, since scientists often make it very difficult for them to understand what scientists do with statistics. Sometimes scientists do this on purpose because they want to obfuscate, but often it is the result of their own ignorance or sloppiness.

What is at stake? Well, simply said substantive significance is about the size of a relationship/ an effect, whereas statistical significance is about measurement precision (usually based on a sample). These two concepts are answers to two very different questions. Substantive significance asks how much, statistical significance asks how sure we are about it. If you think about it statistical significance is much less interesting for the general public, because it essentially only tells you one thing: the ‚finding‘ a statistician identified is pretty certain. We still don’t know whether the finding is interesting, it may be way too small for you to bother!

The two concepts sometimes are related to each other with substantive significance often implying statistical significance. But they don’t have to. Stephen Ziliak and Deirdre McCloskey have written a fascinating book on the topic showing how the confusion between the two concepts can lead to some fatal consequences. They give examples about medical research in which a statistically significant reduction in depression is substantively too small to justify the purchase of the more expensive drug. In another example, research funded by pharma industry even suppresses fatalities just because there were too few of these to be counted as statistically significant.

But it is not only about health, it happens in all kind of research from natural to social sciences, from engineering to linguistics. In my own field, political science, it seems very frequent, and I don’t deny that I myself get sometimes confused. Media coverage is full of examples. It often starts with ‚new research shows X‘. For instance, new research shows that home teams in football are more likely to win. For a statistician it is important to show that the home advantage is precisely measured and identified. For a club owner this is only of secondary importance. True he also needs to be sure that the statistician did his job well. However, for him it is much more important to know how large the home advantage is, and how safely he or she can rely on that. Do I have a 50% higher chance of winning, or just a 5% chance? In both cases home advantage may be statistically significant, but the substantive difference decides on championship or relegation.

Test yourself. How would you interpret the following sentence full of statistical jargon? ‚In a randomized controlled trial we find that the difference between income levels of those who participated in a microfinance program and those who did not was statistically significant at the 1% level?‘ If you answer here something like microfinance makes a huge difference for people, you were a) wrong because we cannot know on basis of this sentence, b) clearly you are not alone as this happens to many. What this sentence should have read is something like this: ‚We compare income levels between those who participate in a microfinance program and those who don’t. We find that the average difference is large, about 200 Euros, which is equivalent to roughly half the average wage in society X. We also have found that this average difference is not a result due to chance, i.e. it is statistically significant, and not due to other distortions. Hence, we trust our findings.‘

What to do? Well pundits, journalists, policy experts need to sharpen their critical reading of scientific literature. They need to vex scientists to tell them quantities of interest which are easy to interpret. As for us scientists, we need to communicate our findings more clearly.

Leave a comment

Filed under Uncategorized