Tuesday, September 21, 2010

From The Students' Room 2

Click here

Dear Phil Baty,

I had sent you email to inquire about the ranking performance of Taiwan's universities.
I am waiting for your reply.
Please share your opinion.
Thanks!

best regards,

yuching
From the Students' Room

There are some interesting comments about the latest THE rankings at their "The Students' Room"

Here is one from Martin

I appreciate that measuring the research impact of an institution is difficult. Unfortunately, the THE seems to have got it quite badly wrong this time. The best evidence for this is not that Warwick doesn't make it into the top 200 (even though this hurts; yes I am faculty at Warwick), but the fact that the University of Alexandria makes it number 4, based on the output of one single person.


Some suggestions follow:


1. Do not count self-citations. Even better, it is common sense that a citation by someone further from your field should count more than a citation by your former PhD student. Of course, these things are difficult to figure out in an automated way. But one could for example use the collaboration distance (coauthors have distance one, coauthors of coauthors have distance two, etc) to weight citations (with a cap at 4, say).

2. Quality of research is paramount. As already pointed out, it is easy to get many citations for mediocre research if there are sufficiently many other mediocre researchers working in the same area. Again, this is vastly more common than you may think, for the simple reason that it is easier to perform mediocre research than world-class research. Furthermore, you get more recognition as counted by basic citation count, so why not doing it?

One way of taking this into account is to give higher weight to citations coming from articles published in highly respected journals. (Similarly, when measuring "research output", higher weight should be given to articles published in journals with high reputation.)

However, please *DO NOT* use the impact factor as a measure of the quality of a journal, as it can be (and is!) very easily manipulated, as the story of the journal "Chaos, Solitons and Fractals" shows. Instead, the only reliable way of assessing the quality of a journal within a given field is to ask researchers in that field to provide their own rankings. Yes, this seems subjective, but unfortunately that's all you are ever going to get, and I can assure you that you will get a very consistent picture within each area. The fact that the "Annals of Mathematics" is the most respected journal in mathematics simply cannot be measured in terms of impact factor.

3. Count current citations to older work. If someone's article turns out to spawn an entire new field of research five years later, it will not show up at all in the current metric. This makes simply no sense. Of course, this doesn't happen all that often, but the reason why top institutions have a reputation is precisely because of those instances in which it happens. Furthermore, there are areas of research (like mathematics) in which the "lifespan" of a good article is measured in decades, which goes way beyond the two to five years that you use as a rule. Counting current citations to older articles would be one small but absolutely essential step to correct this.

4. Measure the total impact of the institution in a field, and not its "average" impact. The only way I could see that the output of one single person can count so much is that this person somehow has an abnormally high weight, probably due to the fact that there is very little research output from the U. of Alexandria. If this suspicion is indeed correct (I hope that I am wrong on this one), then this would effectively mean that universities are penalised by having large (and influential!) departments and should rather strive to have only very few but quite prolific researchers on their payroll.

There is probably more, but I am getting hungry now ;-) I very much hope that you will take these comments to heart. Best wishes,

Martin
Back to Citations

I was hoping to get away from the citations indicator in the THE rankings for a while but there are several comments here and elsewhere that need to be discussed.

First, I had assumed that the very high citation scores for Alexandria, and to a lesser extent other universities, were the result of a large number of citations that, while perhaps excessive, were from reputable scholars. Many of the citations to Alexandria University papers are to and from papers by Mohamed El Naschie. I assumed that his CV was evidence that he was a distinguished and esteemed scientist. Looking closely at his CV there seem to be a number of points that require clarification. There are references to academic appointments for a single calendar year, not the academic year as one would expect. There is a reference to a Professorship at "DAMTP, Cambridge" but the university, nor a college, is not mentioned. Also , there seems to be a period when El Naschie was a professor simultaneously at the Free University of Brussels, DAMTP Cambridge and the University of Surrey.

I hope that these points can be clarified. The TR citations indicator would still be a problem even if it was being skewed by heavy citation of groundbreaking research but it would be more of a problem if there any doubts, whether or not justified, about the excellence of that research.

Monday, September 20, 2010

Alexandria Gets the News

The news about being among the top 200 universities in the world has finally reached Alexandria. The reaction is very predictable:


"“I believe we are well deserving of being on the list,” professor of economics
Amr Hussein told Bikya Masr. “We have worked hard to improve our system of
education and it is showing that we are succeeding in doing so.”...

Hend Hanafi, President of Alexandria University, told local media that she is proud
of the ranking and hopes the university will continue to make efforts to
consistently improve the quality of education at the university."

Will somebody please send the President an account of what happened to Universiti Malaya and its Vice-Chancellor after THE and QS put them in the top 100 in 2004.
Omissions

Several writers have noticed the absence of any Israeli universities from the top 200 universities. In the QS rankings this year there three. Those who have downloaded the ipad
app will have noticed that there are only two in the top 400. So what happened to Tel Aviv and the Hebrew University of Jerusalem?

There is an article about this in Haaretz.
"Haaretz has learned that most Israeli universities were not on the list because they failed to respond to repeated requests for information, including on faculty and students, which is necessary for the listing.

TAU and the Hebrew University say that they never received such a request from THE. According to THE, only the Technion and Bar-Ilan University responded with information, but they were ranked 221 and 354 respectively.

As for the other universities, the editor of the ranking, Phil Baty, told Haaretz that although it is upsetting for Israel, he hoped that the Israeli universities would recognize the amount of serious work invested in creating the ranking and the degree to which the methodology was transparent, and would participate in the initiative, like other universities have done. He also expressed certainty that next year "they will be included."

Didn't do their homework

THE says that more than 6,000 universities participated in the ranking and most provided the necessary information.

A spokesperson for the Hebrew University responded that contrary to the claim of
the survey's editors, "following an examination we did not find any such request
[for information]. When we asked for the correspondence to the university on the
subject, they could not provide it. The university is saddened by the fact that
the editors of the ranking did not carry out their work responsibly, and thus
harmed the university." "
There are other surprising omissions such as all the Indian Institutes of Technology, the University at Buffalo: SUNY and the Catholic University of Louvain (the French one -- the Dutch one is there at 120)

Sunday, September 19, 2010

Another Post on Citations (And I hope the last for a while)

I think that it is clear now that the strange results for the citations indicator in the THE rankings are not the result of a systematic error or several discrete errors. Rather they result from problems with the inclusion of journals in the ISI indexes, possibly a failure to exclude self citations, problems with the classification of papers by subject and, most seriously, the interaction of a few highly cited papers with a small number of papers overall. Taken together , they undermine the credibility and validity of the indicator and do little to promote confidence in the rankings as a whole.
More on the THE Citations Indicator

See this comment on a previous post:


As you can see from the following paragraph(http://www.timeshighereducation.co.uk/world-university-rankings/2010-2011/analysis-methodology.html) Thomson has normalised citations against each of their 251 subject categories (it's extremely difficult to get this data directly from WOS).. They have great experience in this kind of analysis.. to get an idea, check their in-cites website http://sciencewatch.com/about/met/thresholds/#tab3 where they have citations thresholds for the last 10 years against broad fields.

Paragraph mentioned above:
"Citation impact: it's all relative
Citations are widely recognised as a strong indicator of the significance and relevance — that is, the impact — of a piece of research.
However, citation data must be used with care as citation rates can vary between subjects and time periods.
For example, papers in the life sciences tend to be cited more frequently than those published in the social sciences.
The rankings this year use normalised citation impact, where the citations to each paper are compared with the average number of citations received by all papers published in the same field and year. So a paper with a relative citation impact of 2.0 is cited twice as frequently as the average for similar papers.
The data were extracted from the Thomson Reuters resource known as Web of Science, the largest and most comprehensive database of research citations available.
Its authoritative and multidisciplinary content covers more than 11,600 of the highest-impact journals worldwide. The benchmarking exercise is carried out on an exact level across 251 subject areas for each year in the period 2004 to 2008.
For institutions that produce few papers, the relative citation impact may be significantly influenced by one or two highly cited papers and therefore it does not accurately reflect their typical performance. However, institutions publishing fewer than 50 papers a year have been excluded from the rankings.
There are occasions where a groundbreaking academic paper is so influential as to drive the citation counts to extreme levels — receiving thousands of citations. An institution that contributes to one of these papers will receive a significant and noticeable boost to its citation impact, and this reflects such institutions' contribution to globally significant research projects."


The quotation is from the bottom of the methodology page. It is easy to miss since it is separate from the general discussion of the citations indicator.

I will comment on Simon Pratt's claim that "An institution that contributes to one of these papers will receive a significant and noticeable boost to its citation impact, and this reflects such institutions' contribution to globally significant research projects."

First, were self-citations included in the analysis?

Second, do institutions receive the same credit for contributing to a research project by providing one out of twenty co-authors that they would for contributing all of them.

Third, since citation scores vary from one subject field to another, a paper will get a higher impact score if it is classified as a subject that typically receives few citations than as one in which citations are plentiful.

Fourth, the obvious problem that undermines the entire indicator is that the impact scores are divided by the total number of papers. A groundbreaking paper with thousands of citations would would make little difference to Harvard. Change the affiliation to a small college somewhere and it would stand out (providing the college could reach 50 papers a year)>

This explains something rather odd about the data for Alexandria University. Mohamed El Naschie has published many papers with several different affiliations. Yet the many citations to these papers produced a dramatic effect only for Alexandria. This, it seems, was because his Alexandria papers had a big effect because the total number of papers was so low.
Highlights from the Research Impact (Citations) Indicator


One very good thing to emerge from the current round of rankings is the iphone/ipad apps from THE and QS. The THE app is especially helpful since it contains the scores for the various indicators for each of 400 universities. It is possible then to construct a ranking for research imact as measured by citations, which gets nearly one third of the weighting.

Some highlights

1st Caltech
4th Alexandria University
9th Harvard
10th UC Santa Barbara
13th Hong Kong Baptist University
20th Bilkent
23rd Oxford
27th Royal Holloway
31st Johns Hopkins
41st University of Adelaide
45TH Imperial College
65th Australian National University
84th Kent State
11oth Mcgill
143th Tokyo Metropolitan University
164th Tokyo University
285th Warwick
302nd Delft University of Technology
368th South Australia
Perverse Incentives

Until we get a clear statement from Thomson Reuters we have to assume that the citations indicator in the recent THE rankings was constructed by counting citations to articles published in the period 2004 - 2008, dividing these by the expected number of articles and then dividing again by the total number of articles.

It seems then that universities could improve their score on this indicator by getting cited more often or by reducing the number of papers published in ISI indexed journals. Doing both could bring remarkable results.

This seems to be what has happened in the case of Alexandria University, which according to the new THE ranking, is fourth best in the world for research impact.

The university has accumulated a large number of citations to papers published by Mohamed El Naschie, mainly in two journals, Chaos, Solitons and Fractals, published by Elsevier, and the International Journal of Nonlinear Mathematics and Numerical Simulation, published by the Israeli company Freund. El Naschie was editor of the first until recently and is co-editor of the second. Many of the citations are by himself.

I am unable to judge the merits of El Naschie's work. I assume that since he has been a Professor at Cambridge, Cornell and the University of Surrey and publishes in journals produced by two very reputable companies that his papers are of a very high quality.

It is not enough, however, to simply get lots of citations. The actual citation/expected citation number will -- if this is what happened -- be divided by the total number of papers. And this is where there is a problem. If a university has very few papers in ISI journals in the relevant period they will end by getting a very good score. A lot of papers and your score goes way down. This probably explains why Warwick ranks 285th for research impact and LSE 193rd: there were just too many people writing papers that were above average but not way above average.

An article by David Glenn in the Chronicle of Higher Education talks about the perverse incentives of the new rankings. Here, we have another. If a university simply stopped publishing for a year its score on this indicator would go up since it would still be accumulating citations for articles published in previous years.

Saturday, September 18, 2010

More on the Citations Indicator in the THE Rankings
I am copying the whole of a comment to the previous post since it might be the key to the strange results of the citations indicator.

Perhaps someone from Thomson Reuters can confirm that this is method they were using.
Pablo said...

My bet is that TR uses the Leiden "Crown indicator" since this is what is embodied in their product InCites.

To cut it short, each paper is linked to a subdiscipline, a type of publication (letter, review, ...) and a year of publication. With this data for the whole world, it is easy to calculate the expected number of citations for a paper of a given type, in a given discipline, in a given year.
For a set of papers (e.g. all the papers of Alexandria university), the indicator is calculated as Sum(received citations)/Sum(expected citations).

This number can become very high if you have a small number of paper or if you look only at recent papers (if, on average you expect 0.1 citations for a recent paper in math, a single citation will give you a score of 10 for this paper!)

Note that Leiden as recently decided to change its favorite indicator for a mean(citations received/citations expected) which gives less weight for a few highly cited papers in a set. But it seems that TR has not implemented yet this new indicator.

Note also that, in order to avoid the overweight given to few papers in a small set, Leiden publish its own ranking of universities with thresholds on the total number of papers published.

Friday, September 17, 2010

The Citations Indicator in the THE World University Rankings

I am sure that many people waited for the new Times Higher Education rankings in the hope that they would be a significant improvement over the old THE-QS rankings.

In some ways there have been improvements and if one single indicator had been left out the new rankings could have been considered a qualified success.

However there is a problem and it is a big problem. This is the citations indicator, which consists of the number of citations to articles published between 2004 and 2008 in ISI indexed journals divided by the number of articles. It is therefore a measure of the average quality of articles since we assume that the more citations a paper receives the better it is.

Giving nearly a third of the total weighting to research impact is questionable. Giving nearly a third to just one of several possible indicators of research impact is dangerous. Apart from anything else, it means that any errors or methodological flaws might undermine the entire ranking.

THE have been at pains to suggest that one of the flaws in the old rankings was that the failure to take account of different citation patterns in different disciplines meant that universities with strengths in disciplines such as medicine where citation is frequent do much better than those that are strong in disciplines such as philosophy where citation is less common. We were told that the new data would be normalized by disciplinary group so that a university with a small number of citations in the arts and humanities could still do well if the number of citations was relatively high compared to the number of citations for the highest scorer in that disciplinary cluster.

I think we can assume that this means that in each of the six disciplinary groups, the number of citations per paper was calculated for each university. Then the mean for all universities in the group was calculated. Then the top scoring university was given a score of 100. Then Z scores were calculated, that is the number of standard deviations from the mean. Then the score for the whole indicator was found by calculating the mean score for the six disciplinary groups.

The crucial point here is the rather obvious one that no university can get more that 100 for each disciplinary group. If it were otherwise then Harvard, MIT and Caltech would be getting scores well in excess of 100.

So, let us look at some of the highest scores for citations per paper . First the university of Alesxandria, which is not listed in the ARWU top 500 and is not ranked by QS.and which is ranked 5,882nd in the world by Webometrics.

The new rankings put Alexandria in 4th place in the world for citations per paper. This meant that with the high weighting given to the citations indicator the university achieved a very respectable overall place of 147th.

How did this happen? For a start I would like to compare Alexandria with Cornell, an Ivy League university with a score of 88.1 for citations, well below Alexandria’s

I have used data from the Web of Science to analyse citation patterns according to the disciplinary groups indicated by Thomson Reuters. These scores may not be exactly those calculated by TR since I have made some instantaneous decisions about allocating subjects to different groups and TR may well have done it differently. I doubt though that it would make any real difference if I put biomedical engineering in clinical and health subjects and TR put it in engineering and technology or life sciences. Still I would welcome it if Thomson Reuters could show how their classification of disciplines into various groups produced the score that they have published.

So where did Alexandria’s high score come from. It was not because Alexandria does well in the arts and humanities. Alexandria had an average of 0.44 citations per paper and Cornell 0.85.

I was not because Alexandria is brilliant in the social sciences. It had 4.21 citations per paper and Cornell 7.98.

Was it medicine and allied disciplines? No. Alexandria had 4.97 and Cornell 11.53.

Life sciences? No. Alexandra had 5.30 and Cornell 13.49.

Physical Sciences? No. Alexandria had 6.54 and Cornell 16.31.

Engineering, technology and computer science? No. Alexandria had 6.03 and Cornell 9.59.

In every single disciplinary group Cornell is well ahead of Alexandria. Possibly, TR did something differently. Maybe they counted citations to papers in conference proceedings but that would only affect papers published in 2008 and after. At the moment, I cannot think of anything that would substantially affect the relative scores.

Some further investigation showed that while Alexandria’s citation record is less than stellar in all respects there is precisely one discipline, or subdiscipline or even subsubdiscipline, where it does very well. Looking at the disciplines one by one, I found that there is one where Alexandria does seem to have an advantage, namely mathematical physics. Here it has 11.52 citations per paper well ahead of Cornell with 6.36.

Phil Baty in THE states:

“Alexandria University is Egypt's only representative in the global top 200, in joint 147th place. Its position, rubbing shoulders with the world's elite, is down to an exceptional score of 99.8 in the "research-influence" category, which is virtually on a par with Harvard University.

Alexandria, which counts Ahmed H. Zewail, winner of the 1999 Nobel Prize for
Chemistry, among its alumni, clearly produces some strong research. But it is a
cluster of highly cited papers in theoretical physics and mathematics - and more
controversially, the high output from one scholar in one journal - that gives it
such a high score.

Mr Pratt said: "The citation rates for papers in these fields may not appear exceptional when looking at unmodified citation counts; however, they are as high as 40 times the benchmark for similar papers. "The effect of this is particularly strong given the relatively low number of papers the university publishes overall."

This is not very convincing. Does Alexandria produce strong research? Overall, No. It is ranked 1014 in the world for total papers over a ten year period by SCImago.

Let us assume, however, that Alexandria’s citations per paper were such that it was the top scorer not just in mathematical or interdisciplinary physics, but also in physics in general and in the physical sciences, including maths (which, as we have seen it was not anyway)

Even if the much cited papers in mathematical physics did give a maximum score of 100 for the physical sciences and maths group, how could that compensate for the low scores that the university should be getting for the other five groups? To attain a score of 99.8 Alexandria would have to be near the top for each of the six disciplinary groups. This is clearly not the case. I would therefore like to ask someone from Thomson Reuters to explain how they got from the citation and paper counts in the ISI database to an overall score.

Similarly we find that Bilkent University in Turkey had a score for citations of 91.7, quite a bit ahead of Cornell.

The number of citations per paper in each disciplinary group is as follows:

Arts and Humanities: Bilkent 0.44, Cornell 0.85
Social Sciences: Bilkent 2.92, Cornell 7.98
Medicine etc: Bilkent 9.42 Cornell 11.53
Life Sciences: Bilkent 5.44 Cornell 13.49
Physical Sciences: Bilkent 8.75 Cornell 16.31
Engineering and Computer Science: Bilkent 6.15 Cornell 9.59

Again, it is difficult to see how Bilkent could have surpassed Cornell. I did notice that one single paper in Science had received over 600 citations. Would that be enough to give Bilkent such a high score?

It has occurred to me that since this paper was listed under “multidisciplinary sciences” that maybe its citations have been counted more than once. Again, it would be a good idea for TR to explain step by step exactly what they did.

Now for Hong Kong Baptist University. It is surprising that this university should be in the top 200 since in all other rankings it has lagged well behind other Hong Kong universities. Indeed it lags behind on the other indicators in this ranking.

The number of citations per paper in the various disciplinary groups is as follows:

Arts and Humanities: HKBU 0.34, Cornell 0.85
Social Sciences: HKBU 4.50 Cornell 7.98
Medicine etc: 7.82 Cornell 11.53
Life Sciences: 10.11 Cornell 13.49
Physical Sciences: HKBU 10.78 Cornell 16.31
Engineering and Computer Science: HKBU 8.61 Cornell 9.59

Again, there seems to be a small group of prolific and highly accomplished and reputable researchers especially in chemistry and engineering who have boosted HKBU’s citations. But again, how could this affect the overall score. Isn’t this precisely what normalization by discipline was supposed to prevent?

There are other universities with suspiciously high scores for this indicator. Also one wonders whether among the universities that did not make it into the top 200 there were some unfairly penalized. Now that the iphone app is being downloaded across the world this may soon become apparent.

Once again I would ask TR to go step by step through the process of calculating these scores and to assure us that they are not the result of an error or series of errors. If they can do this I would be very happy to start discussing more fundamental questions about these rankings.
From the Chronicle of Higher Education

An article by Aishah Laby contains this news that I have not heard anywhere else:

Quacquarelli Symonds has continued to produce those rankings, now called the QS World University Rankings, and is partnering with U.S. News and World Report for their publication in the United States.

The relationship between the former collaborators has deteriorated into
barely veiled animosity. QS has accused Times Higher Education of unfairly
disparaging the tables they once published together. This week the company
threatened legal action against the magazine over what Simona Bizzozero, a QS
spokeswoman, described as "factually inaccurate" and misleading statements by
representatives of Times Higher Education. She said THE's role in the
collaboration was limited to publishing the rankings based on a methodology that
QS had developed. "What they're producing now is a brand-new exercise. A totally
brand-new exercise, with absolutely no links whatsoever to what QS produced and
is producing," she said. "So when they refer to their old methodology, that is
not correct."

Phil Baty, editor of the rankings for Times Higher Education, declined to respond to QS's complaints: "We are now looking forward, not looking backward."

I didn't know that the animosty was veiled, even barely.

There are some comments from Ellen Hazelkorn

"Really, nothing has changed," said Ellen Hazelkorn, executive director of the Higher Education Policy Research Unit at the Dublin Institute of Technology, whose book "Rankings and the Battle for Worldclass Excellence: The Reshaping of Higher Education" is due to be published in March.

Despite Times Higher Education's assurances that the new tables represent a much more rigorous and reliable guide than the previous rankings, the indicators on which the new rankings are based are as problematic in their own way, she believes. The heavily weighted measure of teaching, which she described as subjective and based on reputation, introduces a new element of
unreliability.

Gauging research impact through a subjective, reputation-based measure is troublesome enough, and "the reputational aspect is even more problematic once you extend it to teaching," she said.

Ms. Hazelkorn is also troubled by the role Thomson Reuters is playing through its
Global Institutional Profiles Project, to which institutions provide the data
used in the tables. She dislikes the fact that institutions are going to great
effort and expense to compile data that the company could then sell in various
ways.

"This is the monetarization of university data, like Bloomberg
made money out of financial data," she said.


Powered by Thomson Reuters

Thanks to Kris Olds in Global higher Ed for noticing the above in the THE World UniversityRankings banner.

A quotation from his article.

Thomson Reuters is a private global information services
firm, and a highly respected one at that. Apart from ‘deep pockets’, they have
knowledgeable staff, and a not insignificant number of them. For example, on 14
September Phil Baty, of Times Higher Education sent out this fact via their
Twitter feed:

2 days to #THEWUR. Fact: Thomson Reuters involved more
than 100 staff members in its global profiles project, which fuels the rankings

The incorporation of Thomson Reuters into the rankings games by Times
Higher Education was a strategically smart move for this media company for it
arguably (a) enhances their capacity (in principle) to improve ranking
methodology and implementation, and (b) improves the respect the ranking
exercise is likely to get in many quarters. Thomson Reuters is, thus, an
analytical-cum-legitimacy vehicle of sorts.

Thursday, September 16, 2010

Comment on the THE rankings

From The Age (Australia)

Les Field, the deputy vice-chancellor (research) at the University of NSW, said the new Times methodology had produced some curious results, such as Hong Kong Baptist University ranking close behind Harvard on citations.

''There are some anomalies which to my mind don't pass the reasonableness test,'' he said.

And Alexandria University, UC Santa Cruz, UC Santa Barbara, Pohang University of Science and Technology, Bilkent University, William & Mary, Royal Holloway, University of Barcelona, University of Adelaide.
Alexandria University

According to the THE rankings Alexandria University in Egypt (no. 147 overall) is the fourth university in the world for research impact, surpassed only by Caltech, MIT and Princeton.

Alexandria is not ranked by Shanghai Jiao Tong University or HEEACT. It is way down the SCImago rankings. Webometrics puts it in 5,882nd place and 7,253rd for the "Scholar" indicator.

That is not the only strange result for this indicator, which looks as though it will spoil the rankings as a whole.

More on Alexandria and some other universities in a few hours.
The Good News

There are some worthwhile improvements in the new THE World University Rankings.

First the weighting given to the subjective opinion survey has been reduced although probably not by enough. Very sensibly, the survey asked respondents to evaluate teaching as well as research.

The task ahead for THE now is to refine the sample of respondents and the questions they are invited to answer. It would make sense to exclude those with a non-university affiliation from answering questions about teaching. Similarly, there ought to be some way of eliciting the views of university teachers who do not do research, perhaps by some sort of rigorously validated sign up system. Something like this might also be developed to discover the views of students, at least graduate students.

The weighting given to international students has been reduced from five to two per cent.

There is a substantial weighting for a mixed bag of teaching indicators, including the survey. Some of these are questionable though such as the ratio of doctoral to undergraduate students.

For most indicators, the present rankings represent a degree of progress.

The problem with these rankings is the Citations Indicator, which has produced results that, to say the least, are bizarre.
First the Bad News about the THE Rankings

There is something seriously wrong with the citations indicator data. I am doing some checking right now.
Highlights of the THE Rankings

The top ten are:
1. Harvard
2. Caltech
3. MIT
4. Stanford
5. Princeton
6. Cambridge
6. Oxford
8. UC Berkeley
9. Imperial College
10. Yale

The best Asian university is the University of Hong Kong. Sao Paulo is best in South America and Melbourne in Australia. Cape Town is top in Africa followed by the University of Alexandria which is ranked 149th, a rather surprising result.

Wednesday, September 15, 2010

The new THE World University Rankings are out. Discussion follows in a few hours.
HEEACT Rankings Out

The Taiwan Rankings are out. They are based on articles and citations over the last eleven years and the last year, highly cited articles, articles in high impact journals and the H-index. Essentially they measure research productivity, research excellence and research impact.

The top 10 are:

1. Harvard
2. Stanford
3. Johns Hopkins
4. University of Washington -- Seattle
5. UCLA
6. UC Berkeley
7. MIT
8. University of Michigan -- Ann Arbor
9. Tornto
10. Oxford

Tokyo is 14th, Cambridge 16th, University College London 17th, Yale 18th, Imperial 21st, Caltech 31st, Melbourne 43rd, Seoul National University 67th.
Academic Fraud in China

An article by Sam Geall in The New Humanist shows something of the other side of China's rapid scientific development in recent years.

Tuesday, September 14, 2010

Access to Rankings Data

Global Higher Ed has an excellent article by Kris Olds and Susan Robertson about the need for transparency in the collection and distribution of ranking data.
Yet More Reactions to the QS

As the world holds its breath waiting for the THE World University Rankings here are a few more reactions to the QS World University Rankings.

AUSTRALIAN universities have responded with a deafening silence to their contentious downgrading in last week's Quacquarelli Symonds World University Rankings.

The Australian


World's top universities: Four IITs slip in rankings

Sify Finance


Ranking is not everything

The Nation (Thailand)


University climbs fourteen places in world rankings

leedsstudent


A slow but steady climb

Malaysia Star Online






















Saturday, September 11, 2010

Ranking Research Impact

A team at the University of Western Australia has ranked the world' top 500 universities by research impact. The table is based on citations data derived from Scopus and covers the period 2000 to 2009. The ranking seems technically to be very competent.

The top five are:

1. Harvard
2. Stanford
3. MIT
4. UCLA
5. UC Berkeley

Cambridge is 13th and University College London 36th.



.

Thursday, September 09, 2010

More Reactions to the QS Rankings

Australian Higher Education Sector Down in Rankings and Nervous on International Enrolments
AIEC QUEST Australian International Education

4 Chinese universities rank among world's top 50
Peoples Daily Online

Wednesday, September 08, 2010

Some Reactions to the 2010 QS Rankings

Cambridge Knocks Harvard Off Top in University League

Nine Taiwan universities listed among the world's top 500
Radio Taiwan International


Israeli universities drop in international rankings



Trinity and UCD slip down rankings of top universities


Cambridge Beats Harvard -- Sort of

The big news from the QS World University Rankings today is that Cambridge is finally top after trailing Harvard for six years.

This seems a little odd since Cambridge is way behind Harvard, and a few other places, on all the indicators in the Shanghai rankings. So what happened? Looking at the indicator scores we find that on the "Academic Peer Review" -- more accurately called an Academic Reputation Index elsewhere on the site -- Cambridge is first and Harvard second. For the Employer Review Cambridge is third and Harvard first, reversing their places last year. For citations per faculty Harvard was third and Cambridge 36th, behind Tufts, Emory and UC Santa Cruz among others. For student faculty ratio, Cambridge was 18th and Harvard 40th. At the time of writing data was not available for International Faculty and Students.

It seems that the main factor in Cambridge's success was the academic survey. QS indicates the sources of the survey.
  • 1,648 previous respondents who returned. If QS have continued the practice of previous years , they also counted respondents from 2009 and 2008 even if they did not submit a form.
  • 180,00 out of 300,000 persons on the mailing list of World Scientific, a Singapore-based publishing company with links to Imperial College London. World Scientific, by the way, claim to have 400,000 subscribers.
  • 48,125 records from Mardev-DM2
  • 2,000 academics who signed up at the QS site
  • Lists provided by institutions. In 2010 160 universities provided more than 40,000 names.

I will let readers decide how representative or accurate such a survey can be.

Incidentally, QS should be given credit for the detailed description of the methodology of this criterion.

Saturday, September 04, 2010

QS announces Date

Times Higher Education have already announced that their World University Rankings will be published on September 16th.

This morning QS indicated on their topuniversities site that theirs will be out on September 8th.

Friday, September 03, 2010

World Class Universities as a Measure of System Quality

This a list of the percentage of each country's universities that are included in the top 500 of the Academic Ranking of World Universities produced by Shanghai Jiao Tong University. It might be considered a limited indicator of the overall quality of a country's higher education system.

The number of universities in each country included in the ARWU Top 500 is from ARWU . The total number of universities in each country is from Webometrics. A university is simply defined by the possession of a distinct URL.

It is of course easier to start a university in the US than in Israel where the country's first Arabic speaking university has only just been approved. However, this table does put the large number of American universities in global rankings in a different perspective.


1. Israel 21.21
2. Sweden 20.37
3. Australia 19.77
4. UK 16.17
5= Finland 11.76
5= Singapore 11.76
7. South Africa 11.54
8. Canada 11.33
9. New Zealand 11.11
10. Italy 10.89
11. Austria 10.61
12. Germany 9.75
13. Netherlands 8.21
14. Belgium 7.14
15. Switzerland 6.67
16. Ireland 6.00
17. Norway 5.89
18. USA 4.70
19. Spain 4.59
20. Saudi Arabia 4.44
21. Hungary 3.85
22. France 3.77
23. Denmark 3.57
24. Japan 3.50
25= Greece 3.125
25= Slovenia 3.125
27. South Korea 2.55
28. China 2.52
29. Chile 2.47
30. Portugal 1.79
31. Czech Republic 1.75
32. Argentina 0.95
33. Turkey 0.67
34. Poland 0.46
35. Brazil 0.40
36. Russia 0.30
37. Iran 0.19
38. India 0.13
39. Mexico 0.11

Thursday, September 02, 2010

New Rankings on the Way

Times Higher Education have announced that their new rankings will be published on September 16th and have revealed the outline of their methodology.

The rankings will include five groups of indicators as follows:


A new broad category, called "Teaching - the learning environment", will be
given a weighting of 30 per cent.

Using five separate indicators, this category will use data on an institution's income, staff-student ratios and undergraduate-postgraduate mix, as well as the results of the first-ever global academic reputation survey examining the quality of teaching.

A further 30 per cent of the final rankings score will be based on another new indicator, "Research - volume, income and reputation".

This category will use four separate indicators, including data on research income, research output (measured by publications in leading peer-reviewed journals) and the results of the academic reputation survey relating to research.

The highest-weighted category is "Citations - research influence".

This category will examine a university's research influence, measured by the number of times its published work is cited in other academics' papers.

Based on the 12,000 journals indexed by Thomson Reuters' Web of Science, and taken over a five-year period, the citations data will be normalised to take account
of different volumes of citations between disciplines.

Reflecting the high levels of correlation between citations data and research excellence, this category will be given a weighting of 32.5 per cent.

A fourth category, "International mix - staff and students", will use data on the proportion of international staff and students on campus. This indicator will be given a 5 per cent weighting.

Knowledge transfer activities will be reflected in "Industry income - innovation", a new category worth 2.5 per cent of the total rankings score. This will be based on just one measure in 2010 - research income from industry.

There is still a lot apparently left undecided such as the distribution of indicators within the groups and exactly what faculty will count for scaling. In general, though, the broad outlines of the new ranking look promising with the exception of the large weighting -- nearly one third -- assigned to a single indication, citations. Certainly citations are a good measure of research impact and more difficult to manipulate than some others but putting so much emphasis on just one indicator will be a problem for face validity and will also amplify any data entry errors should they occur.

Finally, I wonder if it is a good idea to refer to the "seventh annual survey". Wouldn't it better to start all over again with the First THE Rankings?

Saturday, August 28, 2010

From THE

I am reproducing Phil Baty's column from Times Higher Education in its entirety


One of the things that I have been keen to do as editor of the Times Higher
Education World University Rankings is to engage as much as possible with our
harshest critics.

Our editorial board was trenchant in its criticism of our old rankings. In particular, Ian Diamond, principal of the University of Aberdeen and former chief executive of the Economic and Social Research Council, was scathing about our use of research citations.

The old system failed to normalise data to take account of the dramatically different citation volumes between different disciplines, he said - unfairly hitting strong work in fields with lower average figures. We listened, learned and have corrected this
weakness for the 2010 rankings.

Another strong critic is blogger Richard Holmes, an academic at the Universiti Teknologi MARA in Malaysia. Through his University Ranking Watch blog, he has perhaps done more than anyone to highlight the weaknesses in existing systems: indeed, he highlighted many of the problems that helped convince us to develop a new methodology with a new data provider, Thomson Reuters.

He has given us many helpful suggestions as we develop our improved methodology. For example, he advised that we should reduce the weighting given to the proportion of international students on campus, and we agreed. He added that we should increase the weighting given to our new teaching indicators, and again we concurred.

Of course, there are many elements that he and others will continue to disagree with us on, and we welcome that. We are not seeking anyone's endorsement. We simply ask for open engagement - including criticism - and we expect that process will continue long after the new tables are published.


There are still issues to be resolved but it does appear that the new THE rankings are making progress on several fronts. There is a group of indicators that attempts to measure teaching effectiveness. The weighting given to international students, an indicator that is easily manipulable and that has had very negative backwash effects, has been reduced. The inclusion of funding as a criterion, while obviously favouring wealthy regions, does measure an important input. The weighting assigned to the subjective academic survey has been reduced and it is now drawn from a clearly defined and at least moderately qualified set of respondents.



There are still areas where questions remain. I am not sure that citations per paper is the only way to measure impact. At the very least, the h-index could be added, which would add another ingredient to the mix.



Also, there are details that need to be sorted out. Exactly what sort of faculty will be counted in the various scalings? Is self-citation be counted? I also suspect that not everybody will be enthusiastic about using statistics from UNESCO for weighting the results of the reputational survey. That is not exactly the most efficient organization in the world. There is also a need for a lot more information about the workings of the reputational survey. What was the response rate and exactly how many responses were there from individual countries?

Something that may well cause problems in the future is the proposed indicator of the ratio of doctoral degrees to undergraduate degrees. if this is retained it is easy to predict that universities everywhere will be encouraging or coercing applicants to master's programs to switch to doctoral programs.

Still, it does seem that THE is being more open and honest about the creation of the new rankings than other ranking organizations and that the final result will be a significant improvement.

Monday, August 23, 2010

America' Best Colleges 2011

US News and World Report's Ameica's Best Colleges 2011 is now out.

The top ten National Universities are:

1. Harvard
2. Princeton
3. Yale
4. Columbia
5. Stanford
6. University of Pennsylvania
7 = Caltech
7 = MIT
9 = Dartmouth
9 = Duke
9 = Chicago
Shanghai Rankings: Shifting Research Landscape

My article in University World News on the 2010 Academic Ranking of World Universisities can be viewed here.

Thursday, August 19, 2010

More from THE

Phil Baty in Times Higher Education gives us some clues about what the forthcoming THE World University Rankings will contain.


"While all the self-reported material bears the imprimatur of the supplying
institutions (and our tables include only those that have cooperated with our exercise) and it has been vetted for quality, the consultation had some concerns
about its consistency and robustness - especially in this inaugural year. For example, not all institutions could provide a clear or internationally comparable figure for their research income from industry.

For maximum robustness, we plan to give extra weighting to data that have been sourced independently of the institutions themselves and are globally consistent.

Citations data, for example, which are widely accepted as a strong proxy for research quality, will have a high weighting - perhaps about 30 per cent of the total ranking score.

We also have high confidence in the validity and independence of the results of our reputation survey. Although we may yet adjust its weighting, this subjective measure will not be weighted as highly as it was in our old methodology (2004-09), where reputation was worth 40 per cent."

It looks as though citations per paper, a measure of its influence throughout a research community, will count for a lot in the forthcoming rankings. It is questionable whether such a high weighting for a single component is justified. At the very least it could be combined with other measures of quality such as the h-index which is, in effect, a measure of both productivity and impact.

The reluctance to place too much emphasis on research income and perhaps other types of income, is understandable but perhaps unfortunate. This indicator would give the new rankings a distinctive feature and might also allow us to see whether institutions are giving value for money.

It is inevitable that the reputational survey would never be given the same weight that its predecessors received in the THE-QS rankings. Whether its results are really valid -- we still do not know the response rate -- remains to be seen.

Monday, August 16, 2010

Why international students are not a good indicator of quality

Times Higher Education describes a dispute between Coventry University and a recruiting agent in Chennai. According to the article, Ram Beegala was hired as a recruiting consultant and would only be paid if he succeeded in getting the number of Indian students above 450.

There is a comment by "To John" which might be slightly exaggerated:

"It is no secret that the Indian students who cannot get into any of their universities and colleges are the ones that are willing to come to the UK. Their intention is the 20 hour/week work allowed and assume rightly once they use the university route to get into UK they can stay in the country to work. In my university which recruits these students, the drop out rates for such students is high as they work more than 20 weeks to meet their expenses. Their attendance drops down after a few months. I have yet to come across a single non-EU student who comes with enough funds to complete a 3 UG degree. They are told by agents that they can work in the UK to meet part of their fees and all the living expenses. The students coming in to do MSc are poorly equipped and struggle to pass their modules and write project proposals."

Big Names and Unsung Heroes

In Times Higher Education, Phil Baty hints that the reduction in the weighting for subjective indicators in the forthcoming THE rankings will mean that those dominant in the past will suffer a decline and that there may be some new schools at the top.

"We can expect some big-name institutions to take a hit in the new World
University Rankings.

Why? Because the rankings we will publish this autumn will be based less on subjective opinion and more on objective evidence".

..........................................................................................................

"Under the initial proposals for our methodology, currently being refined in line with responses from the global academy, reputational measures are worth no more than 20 per cent of overall scores.

I have also set a cap to ensure that subjective elements are never again anywhere near the 50 per cent used in our previous methodology. This means that big names with big reputations that lack world-class research output and influence to match will suffer in comparison with previous exercises. Conversely, unsung heroes have a better chance of recognition".

Another Ranking

The ic4u ranking of 200 top universities is based on web popularity.

The top five are:

1. Stanford
2. MIT
3. National Autonomous University of Mexico
4. Berkeley
5. Peking
The Forbes Ranking

The 2010 edition of the Forbes College Rankings is now out. These are basically an evaluation from the students' viewpoint. The criteria are the number of alumni in Who's Who in America, ratings in RatemyProfessor, graduation rates, number of students and faculty winning national awards and accumulated student debt.

There are some surprises. Top place goes to Williams College a private liberal arts college that does not even get into Shanghai's top 500. The service academies do very well. On the other hand, Harvard is 8th, Yale 10th and Chicago 2oth.

Saturday, August 14, 2010

Long Term Trends in the Shanghai Rankings

The Shanghai Rankings are noted for their methodological stability. Whereas frequent changes combined with the insertion and removal of errors produced wild fluctuations in the THE-QS rankings, the ARWU have remained essentially the same since they started. The Shanghai index never aroused as much public interest as the now defunct THE-QS league table but over the long run it is more likely to reveal real and significant trends.

If we compare the 2004 rankings with those just announced there are some noticeable changes over six years. Cambridge and Oxford have each slipped a couple of places while Imperial College and University College London have moved up a bit, although not as high as their implausible position in THE-QS. Tokyo has slipped from 14th to 20th and Kyoto from 21st to 24th. The leading Australian university has also fallen.

Russia has stagnated with only two institutions in the top 500 in 2004 and 2010. India has fallen back with the University of Calcutta dropping out of the rankings. The rising stars for scientific research are Mainland China (8 in 2004 and 22 in 2010), South Korea (7 in 2004 and 10 in 2010), Brazil (4 in 2004 and 6 in 2010) and the Middle East (none in 2004 and 4 from Saudi Arabia, Turkey and Iran in 2010).
The Shanghai Rankings

The 2010 Academic Ranking of World Universities by Shanghai Jiao Tong University is now out. See here.

Friday, August 06, 2010

From QS

A media advisory has been sent by Martin Ince, Chair of the Advisory Board of QS World University Rankings. See here.

The document describes the structure of the current rankings. Something interesting is that apparently the number of responses has increased to over 13,000, although about half of those would be from people who filled out the form in 2009 and 2008 and did not update their forms this year. The number of respondents is now about the same as that reported by Times Higher for their survey, although THE will no doubt point out that they can be fairly confident that their respondents are still alive and working in academia.

The number of respondents is less important than the response rate and so far neither QS or THE have said how many forms were distributed.

Thursday, July 29, 2010

An Epic?

Times Higher Education (THE) has announced the completion of the collection of data for its forthcoming World University Rankings:
An epic effort by our world university rankings data supplier, Thomson Reuters, to collect information from hundreds of universities around the world concluded successfully last week.
I am not sure whether "epic" is the right word. The number of universities in the database does not seem much higher than that for which QS has collected information. The data does apparently include some information that QS has ignored such as institutional income and research income but has not included items counted by QS such as total student numbers or the number of postgraduate students other than doctoral candidates. Meanwhile, the number of respondents to the opinion survey has fallen far short of the original target of 25,000, even with a bit of topping up, like QS, from the Mardev mailing lists.

A proposal to rank universities by disciplines as specific as Agriculture has been dropped. Now, THE will rank universities in six disciplinary clusters, up from five in the THE-QS and QS rankings.

THE also give some idea of errors will be detected. That might be an improvement although I suspect that in many countries third party sources may not be as reliable as THE thinks.

One thing that is not mentioned is whether any universities have refused to participate in the data collection and what THE will do if there are any abstentions.

Monday, July 26, 2010

Discrimination In Top US Colleges

Russell K. Nieli in Minding the Campus discusses a study by Thomas Espenshade and Alexandria Radford that details the extent and depth of the racial and social discrimination practiced by America's top colleges.

"Consistent with other studies, though in much greater detail, Espenshade and Radford show the substantial admissions boost, particularly at the private colleges in their study, which Hispanic students get over whites, and the enormous advantage over whites given to blacks. They also show how Asians must do substantially better than whites in order to reap the same probabilities of acceptance to these same highly competitive private colleges. On an "other things equal basis," where adjustments are made for a variety of background factors, being Hispanic conferred an admissions boost over being white (for those who applied in 1997) equivalent to 130 SAT points (out of 1600), while being black rather than white conferred a 310 SAT point advantage. Asians, however, suffered an admissions penalty compared to whites equivalent to 140 SAT points.

The box students checked off on the racial question on their application was thus shown to have an extraordinary effect on a student's chances of gaining admission to the highly competitive private schools in the NSCE database. To have the same chances of gaining admission as a black student with an SAT score of 1100, an Hispanic student otherwise equally matched in background characteristics would have to have a 1230, a white student a 1410, and an Asian student a 1550. Here the Espenshade/Radford results are consistent with other studies, including those of William Bowen and Derek Bok in their book The Shape of the River, though they go beyond this influential study in showing both the substantial Hispanic admissions advantage and the huge admissions penalty suffered by Asian applicants. Although all highly competitive colleges and universities will deny that they have racial quotas -- either minimum quotas or ceiling quotas -- the huge boosts they give to the lower-achieving black and Hispanic applicants, and the admissions penalties they extract from their higher-achieving Asian applicants, clearly suggest otherwise."

The advantage accorded to Non-Asian minority students, even those whose claim to moral reparation for generations of slavery or dispossession is questionable, is well known. What is surprising about Espenshade and Radford's study is the extent of the discrimination against poor, rural and working class whites.

In part, this is a consequence of the indicators used by American ranking organizations. Selective colleges are apparently reluctant to offer places to students who might not take up an offer for financial reasons since this would push down their acceptance rates and yield scores.

But there is more. Espenshade and Radford found that less affluent whites were dramatically less likely to be offered a place in a competitive private college even when SAT scores, a reasonable proxy for general intelligence, and high school grades were controlled for. In addition, they found evidence of serious discrimination against students who were involved in incorrect activities such as ROTC and Future Farmers of America, especially those holding leadership positions. Apparently "feeding the homeless" will boost one's chances of getting into a top private college if it means doling out soup in between starring in the school play and AP English classes but not if means showing an interest in growing the stuff that the homeless eat.

As cognitive skills become increasingly irrelevant to admission into America's best schools, it seems almost certain that US higher education will be less and less able to compete with those countries that continue to recruit those students most capable of demanding college-level work.





Wednesday, July 21, 2010

New Webometrics Rankings

The new Webometrics rankings are out.
There are few surprises. Here are the top universities in various categories.

World: Harvard
North America: Harvard
Latin America: Universidad Nacional Autonoma de Mexico
Europe: Cambridge
Central and Eastern Europe: Charles University, Prague
Asia: Tokyo
South East Asia: National University of Singapore
South Asia: Indian University of Technology, Bombay
Arab World: King Saud University
Oceania: Australian National University
Africa: Cape Town

One interesting feature of the Arab World rankings is that universites in the Palestinian territories do very well in comparison with many in more affluent countries. Would anyone like to suggest an explanation?




Tuesday, July 20, 2010

Global-Rankings Ping Pong

Ben Wildavsky has an article in the Chronicle of Higher Education about the competition between Times Higher Education and QS over this year's university rankings. It is actually called Global-Rankings Smackdown! but the smackdown bit is rather exaggerated and the exclamation mark is unnecessary. There are some well informed and incisive comments on recent developments in international university ranking, including the divorce between THE and QS.

He concludes:

Will a redemption narrative help Times Higher earn credibility for its new rankings? Perhaps. It should certainly be applauded for its openness to criticism, and for all it is doing to inform the public about its next moves in what its editor characterizes, with appropriate caution, as “a decent first step” at improvement. But ultimately, debating tactics notwithstanding, the global league tables will be judged on their merits. As the wars over league tables continue, the next rankings season should be well worth watching.


I am not entirely sure about how much THE, or more accurately their new partners, Thomson Reuters are doing to inform the public about what they are doing. At the moment there are some things we know about the QS survey that we do not know about Thomson Reuters' -- number of forms sent out, response rate, number of responses from individual countries. Still, all that could change within a few weeks and it did take QS a couple of years before they gave out anything beyond the bare minimum about their survey.
The Avalanche

A short article in the Chronicle of Higher Education ,by Mark Bauerlein, Mohamed Gad-el-Hak, Wayne Grody, Bill McKelvey, and Stanley W. Trimble, 'We Must Stop the Avalanche of Low-Quality Research', calls for a halt to the seemingly inexorable rise in the production of uncited and unread scholarly and scientific papers.

While brilliant and progressive research continues apace here and there, the amount of redundant, inconsequential, and outright poor research has swelled in recent decades, filling countless pages in journals and monographs. Consider this tally from Science two decades ago: Only 45 percent of the articles published in the 4,500 top scientific journals were cited within the first five years after publication. In recent years, the figure seems to have dropped further. In a 2009 article in Online Information Review, Péter Jacsó found that 40.6 percent of the articles published in the top science and social-science journals (the figures do not include the humanities) were cited in the period 2002 to 2006.

As a result, instead of contributing to knowledge in various disciplines, the increasing number of low-cited publications only adds to the bulk of words and numbers to be reviewed. Even if read, many articles that are not cited by anyone would seem to contain little useful information. The avalanche of ignored research has a profoundly damaging effect on the enterprise as a whole. Not only does the uncited work itself require years of field and library or laboratory research. It also requires colleagues to read it and provide feedback, as well as reviewers to evaluate it formally for publication. Then, once it is published, it joins the multitudes of other, related publications that researchers must read and evaluate for relevance to their own work. Reviewer time and energy requirements multiply by the year. The impact strikes at the heart of academe.


Unfortunately, now that authorship of an ISI-indexed article has become the qualification for participation n the reputational survey section of the THE World University Rankings I suspect that universities will go on encouraging their staff to produce more and more articles of questionable quality. Or perhaps we should say more and more email addresses in the ISI database.

Friday, July 16, 2010

An Unwelcome Message

Phil Baty, who is in charge of world university rankings at Times Higher Education, writes about an email that he has received.


I was disturbed by an email that dropped into my in-box late last
month.

No, it was not another offer of cheap Viagra, or an announcement that I
had won an overseas lottery. It was more unsettling than that.

"Dear academic," it began. The greeting alone was a surprise, given
that I am a journalist with little more than a bachelor's degree by way of
academic credentials.

But my unease grew with each line of the message. The email was from a
major education information company inviting me to take part in an online survey
that would be used to create a university ranking.

It said that my role as a leading educationalist combined with my
subject focus made my opinion very important. It even offered to enter me into a
prize draw if I passed on my great wisdom and spent 10 minutes filling in the
form.

It would be amusing if the implications were not so serious. As the
email claimed, the audience for the company's annual exercise is in the
millions, and it is clear that university league tables in various forms have
become a very big business with wide influence.

Any organisation, such as Times Higher Education, that seeks to create
rankings must accept its responsibility to conduct thorough research and to
employ sound data.

There is a responsibility on companies doing such surveys that
academics are selected carefully by discipline, and by country and continent if
appropriate. If compilers want universities and students to see their league
table as robust the onus is on them to take a rigorous approach. When rankings can make or break a university's reputation, or influence multimillion-pound strategic decisions, anything less will simply not do.

I am sure that anyone reading this blog has received the message by now and knows that the mysterious sender is not Voldemort but QS, who are now producing their own university rankings independently of THE.

The sending of the message and form to Phil Baty actually represents an improvement for the QS survey. Even without a doctorate, he is probably better qualified to evaluate universities than most subscribers to the World Scientific mailing list, of whom nearly 200,000 receive the form every year. Subscription requires nothing more than the ability to click a mouse a few times.

I wonder though whether those who completed the THE survey form sent out by Thomson Reuters to authors who have published in ISI indexed journals are significantly better qualified. I have heard that there are many parts of the world where the granting of co-authorship of research papers is simply a perquisite of seniority within a department and nomination as corresponding author, the one who gets to go to conferences and do a bit of shopping, is decided partly or largely by political pressures.

It may be that the time has come for a greater variety of reputational surveys to be conducted. There is certainly room for a QS - style survey, essentially open to anyone who, for whatever reason, is interested. After all, that is a constituency that deserves some consideration . But equally, perhaps more so, we need as survey of research excellence that targets demonstrably competent researchers. The ability to be nominated as corresponding author -- I assume that is the one whose email addresses is entered in the ISI archives -- of a paper once in an academic career mught not be sufficient evidence of competence to evaluate university research and teaching. There is a case for a survey based on a more rigorous working definition of research competence, such as inclusion in the ISI list of highly cited researchers. Another possiblty might be to survey editors of academic journals. Response rates could be boosted by publishing the journals who took part. There is also an obvious niche for a student based survey of teaching.

Anyway, Phil, you might as well do the survey. There are many people less knowledgable than you filling out the form and, for that matter, the one for THE . You might even be the one who wins the BlackBerry.

Thursday, July 08, 2010

Presentation by Phil Baty

A presentation by Phil Baty of Times Higher Education at the ISTIC meeting in Beijing reviewed the background of the now defunct THES-QS World University Rankings and the rationale for the development of a new ranking system.

There are some quotations that highlight familiar complaints about the THE-QS rankings:


“Results have been highly volatile. There have been many sharp rises and falls… Fudan in China has oscillated between 72 and 195…” Simon Marginson, University of Melbourne.


“Most people think that the main problem with the rankings is the opaque way it constructs its sample for its reputational rankings”. Alex Usher, vice president of Educational Policy Institute, US.


“The logic behind the selection of the indicators appears obscure”. Christopher Hood, Oxford University.



Baty also indicates several problems with the "peer review", citations, faculty student ratio and internationalisation indicators.



All of this is very sound. But it is not yet certain how much of an improvement the new THE rankings will be.



THE will now obtain citations and publication data from Thomson Reuters rather than Scopus. The Thomson Reuters data is based on the ISI indexes, which are somewhat more selective than the Scopus database. There is, however, a great deal of overlap and simply using ISI data rather than Scopus will not in itself make very much difference except perhaps that there will be a somewhat greater bias towards English using researchers and the research output that is measured may be of a somewhat higher quality. We should also remember that from 2004 and 2006, the THE-QS citations data were collected by the very same Jonathon Adams who is now overseeing the development of the new THE rankings.



Some of the "confirmed improvements" noted by Baty are certainly that. Normalising citation scores between various disciplinary groups to take account of varying patterns of publication and citation is something overdue. The presentation of information about various types of income will, if the raw data is publicly available, make it possible to evaluate universities in terms of value for money.


In some ways the reputational survey may be better then the QS "peer review" but exactly how much better is not yet clear. Baty says that only published researchers were asked to take part but this apparently could mean no more than being listed as the corresponding author for an article once in a lifetime. No doubt this yields a better qualified group of respondents than that made up those with the energy to sign up with World Scientific but is it really significantly better?


Also, there is much that we have not been told about the reputational survey. We know the total number of respondents, which was much lower than the original target, but not the response rate. Nor has there been indication of the number of responses from individual countries. This is particularly irksome since rumour and subjective impression suggest that many countries have been neglected by the recently closed THE survey.


The methodology still appears in need of refinement. Research income of various kinds appears four times as an indicator or part of an indicator: research income from industry as the sole indicator in the Economic Activity/Innovation category; as part of overall research income and as part of research income from industry and public sources in Research Indicators; and as part of total institutional income in Institutional Indicators. This is a bit messy.

There is still time for THE to produce an improved ranking system. Let's hope they can do it.

Monday, July 05, 2010

Presentation by Jonathon Adams

A summary of progress so far on Thomson Reuters' Global Institutional Profiles Project an be found here.

Saturday, June 19, 2010

A Bit More About the THE Survey

Thomson Reuters have released a bit more information about the reputational survey they recently conducted for the 2010 Times Higher Education World University Rankings.

They managed to get 13,388 responses. This is quite a lot less than the original target of 25,000 although it is higher than the 9,000 plus respondents to the 2009 THE-QS rankings. This means that QS, who are preparing their own rankings, now have an opportunity to boost the numbers of their respondents by using the usual devices -- reminders, extended deadlines, a chance to win an iPad instead of a Blackberry and so on . Thomson Reuters may have made a mistake by closing their survey so early.

Still, numbers are not everything. Thomson Reuters can claim that their survey, which uses the ISI database of authors published in reputable academic journals, targets people who know something about research. The QS survey, on the other hand, consists merely of those who have managed to get on the mailing list of World Scientific.

Thomson Reuters have also provided some information about the regional and disciplinary distributions of their respondents. The largest group is from the Americas. While most disciplinary clusters are well represented, there is a very small number from the arts and humanities. Respondents spend slightly more than half their time doing research and slightly less than a third teaching.

Is this really enough? It would be interesting to know how many forms were sent out and what the response rate was. Also, how far back in time did Thomson Reuters go in collecting respondents? If they went back five or ten years many respondents might have retired or lost interest in research since publishing.

It also would be helpful if more information were given about the geographical distribution of the survey. One notable absurdity of the THE-QS surveys of 2004-2009 was the marked bias in favor of particular countries – more respondents from Indonesia than from Germany, more from the UK plus Australia than from the US, more from Ireland (just the Republic?) than from Russia. Thomson Reuters have probably overcome these biases but have new ones emerged? Has there been an adequate response from Southeast Asia outside Singapore? Have Russia and Central Asia and the Middle East outside Israel been affected by the omission of Russian and Arabic from the list of languages in which the forms can be completed?

It is good that Thomson Reuters have released some information but if they are to fulfill their promise of greater transparency more is needed.