1 Procedures

The panel held ten meetings and each member spent at least a month in libraries and elsewhere reading publications. Following its published criteria the panel took publications as the main body of evidence. Each department or institution was examined initially by two panel members, who read as many titles as could be located. We saw over three-quarters of the total number of publications submitted and double-read many of them (we are very grateful to institutions for responding so readily to our requests for items). Each publication was graded on a five-point scale from excellent to poor. We attempted to maintain consistency and comparability of grades by pairing each member with a different person in connection with different institutions. There was a reassuring degree of unanimity in the assessments of publications, but in the minority of cases where grades seemed odd to the panel as a whole, they were justified or reassessed. All work in the field of Conservation and in Art and Design was referred to an outside expert or another panel.

Next, we used the grades for publications to arrive at a grade for each researcher, not by averaging or other mechanical means, but by assessing the overall impact of the grades. Books, again according to the criteria, were accorded a greater weight than articles, all other things being equal.

By working in this way we hoped, firstly, to avoid being influenced by pre-conceived ideas as to the worth of institutions, and secondly to have established a large and reliable body of assessments on which to proceed to the next phase. Departments presented sub-areas in a variety of different ways, some treating periods as sub-areas and others not. To ensure comparability we treated each researcher as a sub-area.

The next phase involved the translating of ratings for individual researchers into a grade for each institution. We did this by establishing equivalents for international, national and sub-national and the resulting spreads of scores across gradings, and by examining the contributions of forms RA1 and 3-6. Assessing the contents of these other forms proved more difficult than assessing RA2s. With the publications we had a common understanding of the kind of object, the book or article, for example, we were dealing with, whereas with the other forms there was a much greater variation in the information involved.

RA1
Because category C staff are often less central to a department's academic standing than its category A staff, we gave credit for the good work by C staff but did not penalise work of poorer quality.

RA3
We paid special attention to high absolute numbers of post-graduates as well as to increases in numbers, even where these had not begun to reach completion.

RA4
We gave greatest weight to research income received as a result of peer review, but we were also aware that income acquired from within institutions could indicate commitment and faith in the future.

RA5 & 6
The panel sought three things from these forms: i) explanations for aspects of RA1 and RA2 which appeared to need explaining;
ii) clear indications of the existence of a research culture; and
iii) plans to sustain or create such a culture.

These discussions produced a set of provisional ratings, which we then re-examined, paying special attention to cases on borderlines.

2 Advice to individuals and institutions

Three observations in particular may be of use to institutions and to the discipline as a whole. First, the panel wish to stress again the fact that we gave what we judged to be the quality of published research pride of place at every opportunity, so that publication for publication's sake was of little or no advantage to a researcher or institution.
Secondly, we note that a number of departments graded 1, 2 and 3b have on their staff individuals and groups of scholars of high standing, whose contribution needs to be protected by their institutions.

Thirdly, where institutions have set up in-house publishing houses it is essential that they attract a significant proportion of articles from researchers at other institutions and conversely that they encourage their own staff to publish the bulk of their work elsewhere. The most respectable journal would lay itself open to question if a high proportion of its articles was by members of its editorial board.

3 Results

There were 47 submissions in all, with numbers in each grade as follows, along with figures for the 1992 exercise (45 in all):

The results were therefore similar except for an increase of one 5 and five 3s, and decreases in the two lowest categories. They are not, however, strictly comparable, since, to name only one reason, institutions did not necessarily submit the same proportions of staff in both exercises. There were 9 new institutions in 1996 against 12 in 1992. There were 9 institutions that had lower grades in 1992 than in 1996 and 9 that had higher grades.

Conclusion

The panel considered that the form of assessment represented by this exercise was defensible in that it dealt with outcomes rather than processes. In other words, it was of no interest to the panel how researchers arrived at their results. In this the Research Assessment Exercise differs fundamentally from the teaching Quality Assessment Exercise, which concerns itself primarily with processes.

We have submitted a full report on the exercise to the HEFCE.

Professor Dawn Ades, University of Essex
Professor Stephen Bann, University of Kent
Professor Tim Benton, Open University
Professor Diana Donald, Manchester Metropolitan University
Dr Caroline Elam, Burlington Magazine
Professor Eric Fernie (Chairman), Courtauld Institute
Professor John Onians, University of East Anglia
Professor Adrian Rifkin, University of Leeds