Martin Ince
Published: 06 October 2006
Peer review is once again a key criterion in this years rankings. But research quality is now gauged on five rather than ten years of citations, making it more topical, says Martin Ince
The tables on pages 3-5 are the third edition of The Times Higher/QS World University Rankings. As in 2004 and 2005, they list the worlds top 200 universities according to a range of qualitative and quantitative criteria. Our methodology this year follows that we used in 2005 very closely.
Qualitative and quantitative forms of data each account for half the total score. The qualitative data is based on our belief that the people who know most about university quality are those who work in them or are closely connected to them.
For this reason, 40 per cent of the score allotted to each university is derived from peer review carried out among academics by QS Ltd, partners with The Times Higher in compiling the World University Rankings. This has involved gathering data from 3,703 academics around the world. Each was asked which area of academic life science, medicine, technology, the social sciences or the arts and humanities they are expert in, and then asked to name up to 30 universities they regard as the top institutions in their area. This is a robust and simple test, and is almost immune to fraud. To achieve this large total of participants, we amalgamated data from our surveys in 2004 and 2005 with this years responses. However, only the most recent response was used from any individual. In future years, we shall not use data more than three years old.
This peer review shows that, although there are a few dozen universities that are plainly world leaders, there are also well-regarded universities in a surprisingly large variety of countries, in both the rich and developing worlds. Indeed, Top Universities Guide, the book that accompanies this supplement, shows that the top 500 universities in the world all have their supporters. The top 200 come from 30 countries, while the top 500 come from 51.
This peer review is enhanced by a further 10 per cent of the score based on the opinion of a vital group of outsiders who observe the worlds universities closely. These are graduate recruiters, especially those who work internationally or on a substantial national scale. The sample includes people from companies in manufacturing, services, finance and transport, as well as from the public sector. They were asked which universities they like to recruit from, a question that we hope reveals something about the quality of the students an institution can attract and the teaching they receive there. We sampled 736 recruiters.
Peer review is the standard way in which the quality of individual pieces of academic work is judged. We believe that applying it to institutions in the controlled way we have done provides an up-to-date measure of the dynamism of whole institutions and of wide groups of subjects in them.
The other half of the rankings scores are made up of quantitative measures. As with the whole of this exercise, the problem is to obtain a measure of university quality that can be calculated on a consistent basis in widely differing environments. This means developing questions that can be answered in a valid and informative way in Norway as well as in Brazil.
Teaching and research are the main activities that occur in universities. Measures designed to capture the quality of these activities account for 40 per cent of the total score in our rankings.
We measure teaching by the classic criterion of staff-to-student ratio. This is captured by asking universities how many staff and students they have, and dividing one by the other. In practice, things are not quite so simple. One complication is to decide exactly who is a student.
We ask universities to count people studying towards degrees or other substantial qualifications, not those taking short courses. Staff numbers, too, can be a matter of opinion. We ask universities to submit a figure based on staff with some regular contractual relationship with the institution. A guest lecturer, however distinguished, should not count. This measure is also prone to subject bias. Teaching people to be surgeons or musicians is inherently more person-intensive than transmitting some other forms of knowledge. But because our analysis deals mainly with large general universities, this variation should even itself out.
The measure of staff-to-student ratio is intended to determine how much attention a student can hope to get at a specific institution, by seeing how well stocked it is with academic brainpower relative to the size of its student body. It accounts for 20 per cent of the possible score.
Our next measure, relating to research, is intended to examine how much intellectual power a university has relative to its size. It is based on citations of academic papers, since these are regarded as the most reliable measure of a papers impact. The worlds accepted authority on citations is Thomson Scientific in Philadelphia, formerly the Institute of Scientific Information. We use data from Thomsons Essential Science Indicators database, processed by Evidence Ltd in Leeds.
The ESI concentrates on the worlds most highly cited and influential research. Our analysis uses data covering 2001-06. This is a change from the first two editions of the World University Rankings, which used ten years of data. Using five years increases the dynamism and rate of change of this measure, but still provides a statistically valid amount more than 40,000 papers and more than a million citations each for Texas and Harvard universities, the worlds top two generators of scholarship on this measure.
To compile our analysis, we divide the number of citations by staff numbers to correct for institution size and to give a measure of how densely packed each university is with the most highly cited and impactful researchers.
There are well-known problems with citations as a measure of research. One is the underrepresentation of papers in languages other than English in citations data. Thomson is addressing this issue by sampling more journals in Asian and continental European languages. But it is also becoming less of a factor as English becomes the language of choice for academic publishing across the world.
As our introduction on page 2 makes clear, the increasingly international nature of higher education is a key reason for the existence of the World University Rankings. The final 10 per cent of our score is intended to determine how global universities are:
5 per cent is awarded on the basis of the percentage of overseas staff each university has, and a further 5 per cent for its percentage of overseas students. This measure is intended to help mobile staff and students by giving them an impression of how international a university may be. But because this measure counts for only 10 per cent of the total score, it is not possible for an institution to do well in the overall table on this measure without being excellent in other categories.
There are many measures we do not attempt to capture in these pages. We gather data on universities that teach undergraduates only. This eliminates many high-quality specialist institutions such as Rockefeller University and the University of California, San Francisco, both of which are postgraduate medical institutions.
We have considered a wide range of other criteria, such as graduate employment and entry standards, as possible quality measures. But these have all failed the test of being applicable evenly around the world. For example, a university in a particular country could show poor graduate employment figures because of the state of its national economy, not because it provided a bad education.
Likewise, universities are under pressure to produce spin-off companies and other forms of knowledge transfer. But their success in doing so will depend to a large extent on the economic system in which they are embedded. In the same way, it is impossible to devise a universal measure for entry standards. However, we are always interested in readers suggestions for new measures we could consider applying.
We regret that there are no data on Royal Holloway, University of London. We plan to include the institution in the rankings for 2007.