Lessons for University Ranking
Content
Evaluation Practices and Methodologies: Lessons for University Ranking
By Bertrand BELLON
1. Introduction
Hierarchical ranking is the most common and simplest instrument of comparison between discrete indicators. It is the quickest way to obtain a comparison between competing entities, as long as objectives, rules of behavior and the relevant measurement tools are shared within a community (school class, athletic group, business, technology… measured by marks, speed, profit, financial assets, etc.). As an instrument of evaluation, ranking can be applied to almost any criteria. From school marks to the book of records (that hold the attention on the individual who first performed record figures), ranking is used everywhere as a way of measurement and of comparison. How can one improve this unavoidable measure? If one agrees with the choice of data, ranking presents no difficulties excepting the quality of data collection. According to a criterion or a set of criteria, the researcher ranks an entity number one, until another entity surpasses it under the accepted criteria. This process has been accelerated within an environment marked by increasingly open societies and expanding economies and societies, providing ranking with a new era of development. Even so, ranking appears highly problematic when dealing with complex and intangible goods such as knowledge, science, technology and innovation. In the case of the “production” of universities, simple criteria do not apply due to the high complexity level (which is the case in most dimensions of Social Sciences). Ranking may help at first in making crude distinctions, but it immediately becomes a limited instrument, for there is no “unique best way” to apply it in any human activity. Given the fact that there are many possibilities to improve the ranking process within its own rules and limits, this chapter intends to “drain” from the methodology of evaluation several elements with which to improve ranking of “world-class universities”. The author will begin with the
2
Evaluation Practices and Methodologies
extension of needs regarding a better understanding of university structures and strategies in the present times in Section 1. He will then underline the diversity of the objectives of evaluation, comparing them to the simpler, thus easier to understand, objectives of ranking in Section 2. Then, he will recall a selected panel of indicators, which can be managed within evaluation process in Section 3. Section 4 will show lessons drawn from evaluation indicators to improve the ranking activity. In conclusion, the author will revisit a few core questions about the goals of ranking and evaluation.
2. The Increasing Need for a Better Characterization of Universities
In an open economy and society, the characterization of academic activity and of performances is not only a concern for transversal authorities [for years, the OECD has published indicators on education and research and UNESCO has produced an important study on performance indicators and universities (Fielden and Abercromby, 1969)] but also an increasing need for each individual University. Characterization is thereby jointly related to measures of absolute excellence, and toward self-improvement within each specific context. Better understanding of a university’s stand-point, better management of the given assets, better efficacy and output, are thus mandatory, both for the locally embedded as well as for the world-class university. Yet, characterization raises two different questions: ─ how well do universities perform when their goals and means are taken into account?; and, ─ how is a particular university better, equivalent or worse than its competitors? Interestingly, however, these questions are not the same according to everybody, for they inevitably differentiate according to the values of the universities” stakeholders. 2.1. External stakeholders Universities gather a wide variety of stakeholders (internal and external) who are increasingly active and concerned with the way they are managed
Bertrand BELLON
3
and with their results. These partners becoming “drivers”, their requirements differ widely from one to the other. That is to say: Public authorities involved into university support are increasingly concerned with the use of public money. The main share of universities’ budgets still depends mainly on public decisions (in coherence with Adam Smith and Alfred Marshal theory of external effects being attributed to knowledge and education). In a period of relative shortage of public budgets – due to increasing competition between states, to the non interventionism ideology and to cuts into budget deficits – an increasing responsibility is put on public project managers, with more attention being paid to results (and consequently to the universities’ management mode). Taxpayers are increasingly reactive to the way their money is used by public as well as private research institutions. This often justifies political and public financial short-term views, in contrast to the long-term dimension of the research process and the complex articulation between Fundamental and Applied Sciences. Universities have become increasingly decisive tools for economic competitiveness, knowledge and innovation. This has led industries to be directly concerned with university possesses (e.g., hiring skilled students and benefiting as directly as possible from new research). This concerns not only high tech industries, but also includes every “mid-tech” and “low-tech” business that is involved into innovation and to increasing use of general purpose technology (Helpman, 1998). Finally, journalists and other opinion makers are very active in universities’ visibility. They create and convey the images – given to people as proven reality – emphasizing both fictive and real strengths and weaknesses of universities. So, one can well see that universities have become increasingly in debt to, or at least dependent upon an increasing number of external partners, such as taxpayers, government administration and politicians, national and international organizations, business managers, journalists, as well as foundations and NGOs, etc. For various reasons, those external stakeholders focus on the final “outputs” of universities. At best, they require information concerning the relation between material and labor “inputs” (what they have paid for) and “output” or “production”. Thus, external stakeholders are largely unconcerned with the two central processes of university activity, i.e., the production of new knowledge, and the teaching-learning process
4
Evaluation Practices and Methodologies
between professors and students. Indeed, in most cases, the university remains a mysterious “black box” to them, a vision reinforced by the very complexity of these two intangible, ambiguous (and therefore hard-toevaluate) production processes. Hence, these complex problems of education, learning, researching, and governing these institutions are left to specialists. 2.2. Internal interest and need for self-evaluation of universities External interests are not the only university’s partners, however. University managers, students, professors, researchers, administrative staffs, are the other major internal partners that come to the institution with their specific interests and objectives. Students also participate in the openness of economies. This is done by their “shopping” among universities worldwide, according to their own objectives, capabilities and means. Students might choose a university because it is nearest to their home, but, more and more often, they will make their choices according to institutions’ and diplomas’ fame, given that their main concern is to increase their chance of finding rewarding jobs after graduation. They are found to be more discriminating as concerns foreign universities than between their own country’s universities, which increases the artificial differences carried by reputation and image, as compared to real relative capabilities. Researchers and professors tend to field multiple job applications – simultaneously among diverse universities – looking for the “most famous” one or, barring that, the one that provides them the best facilities for research and teaching. “Quality of life” issues, in their various dimensions, from day-to-day particulars to lifelong career prospects, are the main determinants of their final choice. Working amid such “driver behaviors”, university managers carry the responsibility to achieve the optimum of production and productivity out of the two above mentioned groups, by building a working coherence from heterogeneous and highly individualistic behaviors. Each internal partner is thus a stakeholder, carrying forward his or her own objectives, governed, in part, by an interfacing of personal and social opinions and abilities of self and others. This makes the university resemble a network of competing interests. In this regard, the ultimate responsibility
Bertrand BELLON
5
of its management is to provide a minimum degree of satisfaction to each partner. Therefore, there is a great need by universities for finer abilities of inward understanding and evaluation. It is known that academic ranking has different meanings according to who is looking at it. In this context it is not surprinzing that university managers interest in academic ranking has been two-fold, i.e., fostering better management of universities and consolidating a new field of research about the production and diffusion of knowledge. Other objectives can be added to these, each one bringing its own consequences to bear upon the work to be done, but the “ranking user” issue remains the central one. As such, it will have effects on the whole process of academic measurement, including the choice of indicators and methods of data collection. 2.3. The Multi-helix model The new exigency of information and control, initiated from the double “inside-outside” stakeholder demand is strengthened by the increasing interest of researchers in Science and Technology production and in learning activities and processes. The character of this interest is situated at the intersection of four vectors. A growing interest in “macro” studies, that is, making large comparisons of data, and/or providing general historical perspectives of trends (Mowery and Sampat, 2004; Martin and Etzkowitz, 2000). Many trend studies are based on historical cases of specific university (Jacob, Lundqvist and Hellsmark, 2003). The renewed attention to “excellence” among competing universities, can also be associated with scientometric benchmarking and patent analysis (Noyons, Buter, van Raan, Schmoch, Heinze, Hinze, Rangnow, 2003), as well as, in the United States of America, Canada and in Europe (Balconi, Borghini, Moisello, 2002), (Carayol and Matt, 2003-2004), (Azagra Caro et al., 2001). Emerging questions on the strategies of universities (European Commission, 2004) and related issues including the governance of universities (Musselin and Mignot Gerard, 2004; Reale and Potí, 2003) and the organization of research activities: laboratories and research centres versus teaching departments, interdisciplinary versus disciplinary
6
Evaluation Practices and Methodologies
organizations, allocation and rewarding mechanisms, articulation between teaching and research positions, role of research in career stages, etc.). Finally, every public institution involved in R&D and education is increasingly interested in studies on the production process of knowledge. Regional observatories have therefore been created as instruments for orientation of funding decisions (as has been done at the European, national and local levels, even including medium-sized cities). This representation is currently enlarged with the relations developed between the university and the public at large and the multiplicity of organizations that belong neither to government nor to business (e.g., NGOs, international organizations, foundations, multilateral, European and regional entities, cities, etc. Except for the hardware necessary to conduct research, both academic inputs and outputs are intangibles. In consequence, only a small part of such intangibles are identified and thus very limited instruments exist to measure them (Cañibano and Sánchez, 2004). Furthermore, research in such a science requires multidisciplinary work: Sociology, Economics, Science Policy, Management, etc. Yet, one finds recent theoretical developments have brought some interesting benefit to this field of study. Partha Dasgutpa and Paul David (1994) have suggested a framework for a new Economics of Science, Michael Gibbons (2004) has identified the “Mode 2” concept of research, Henry Etzkowitz and Loet Leydesdorff have popularized the “Triple Helix” concept as a way to see government, academia and industry as parts of one structure (1997). In sum, the complex relation system (University-Industry-Government) “increasingly provides the knowledge infrastructure of society” (Etzkowitz and Leydesdorff, 2000; p.1). The model verifies that: 1) the relationships among universities, industry and government are changing; and, 2) there are internal transformations in each of these individual sectors as well. Consequently, universities are not just teaching or research institutions, but combine teaching, research and service toward and for society. In other words, within a knowledge-based economy the noted triple helix model turns into a multi-helix one, with the main function being given over to universities. It is this growing sphere of social function and responsibility that explains growing pressures for an accounting of resources employed and deployed by universities, yet they are given no unique set of criteria given by which to measure their performances.
Commen gath er the auth or mea ns orga niza tion s belo ng neit her to nati onal gov ern men t nor busi ness in this pass age, give n som e of the exa mpl es liste d (i.e. , citie s).
Bertrand BELLON
7
Figure 1. The Multi-helix Model
2. Ranking versus Evaluation Processes
In this context, evaluation processes will take different forms and include different objectives according to different problems and different missions of institutions. These can be distinguished thusly: ─ evaluation (to fix the value, and measure the position regarding objectives or partners); ─ monitoring (to verify the process of the activity; to admonish, alert and remind);
8
Evaluation Practices and Methodologies
─ control (to verify an experiment by duplication of it and comparison); ─ accountability (to be able to provide explanations to stakeholders for actions taken); ─ ranking (to put individual elements in order and relative position; to classify according to certain criteria). Complicating matter, however, is the fact that the “field” of the university is composed of ideas, knowledge, information, communication, etc., which are typically unique non-positional, hence, “non-rival goods”. 2.1. Taking into account the diversity of higher educational missions Because universities deal with the creation and the diffusion of knowledge, the varieties of their missions are endless. These composite varieties are the joint result of multiple knowledge characteristics and of the variety of each university’s stakeholders and their concerns. The first partner will focus on the ability to train wider numbers of students; the second to increase the international research network; the third will consider as a priority to produce Nobel Prize or Fields Medal winners, etc. However, whatever the diversity, every stake holding group will be concerned in gaining better recognition and visibility of its university. In addition, university functions are ever growing in diversity. This is due to the enlargement of the boundaries of scientific thought well into the area beyond the laboratory, thus facing increasing pressures to introduce new applied technologies into day-to-day life (i.e., into the production of goods and services). On a synthetic level, a university can be characterized as having a double mission of training (basic and continuing education) and of researching (production and diffusion of new knowledge). Beyond these, a “third mission” of universities (Spru, 2002) of providing services to society is growing in importance, with broad social-economic impacts, encompassing both profit and non-for-profit output). 2.2. Evaluation as a starting point When using different individual evaluations to compare universities, researchers are faced with a strong difficulty to agree upon indicators and to proceed to efficient benchmarking. At best, universities will be comparable when they share similar goals and they benefit from similar means – which
Bertrand BELLON
9
is very rarely the case. Yet, one can consider the ongoing processes of evaluation as trials to identify the useful indicators for a given set of questions and for a set of universities. The characterization of universities is thus a first step toward the benchmarking of universities worldwide. In the evaluation processes, various types of practices and meanings can be considered: ─ discipline-based versus institution-based evaluation; ─ inputs-based versus outputs-based evaluation; ─ internal versus external evaluation; ─ qualitative versus quantitative evaluation. 2.3. The ranking process The worldwide liberalization of markets and societies has created a new global competition among universities. When considering research and teaching, universities considered as the “best” universities will attract more talented students; attracting the “best” students, they will thus be able to reinforce their capabilities for autonomous selection-processes for their own benefit. Academic ranking intends to provide means of comparison and identification between universities according to their academic or research performance. Yet, the result will be twofold: on the one side, an increasing need for worldwide universities of excellence, on the other side, an increasing need also for local universities that will be specialized at providing college-level (rather than at doctoral-level) training, including a strong commitment to regional issues and development, corresponding to the eventual creation of disciplinary “niches” for research at a level of excellence. There is a gap, and often an unbridgeable one, between evaluation processes and ranking processes. As far as this work is concerned, it will focus on ranking considerations. Given that ranking must be based upon incontestable (or at least objective) indicators, the objective of this author’s contribution is to take into consideration a set of experiences growing from specific evaluation processes toward the methodology of benefit-ranking production.
10
Evaluation Practices and Methodologies
3. Evaluation Indicators
This section will present data and indicators commonly used for evaluation, the objective being to deduct a few especially grounded data that can be adapted and used for general ranking processes. In so doing, however, it is vital to keep in mind that ranking on a worldwide scale induces very strict constraints that limit the number of possible indicators that can be employed. Therefore, indicators will be limited to those that are: already existing, comparable, and, easy to collect. This explains why the selection of indicators will be very limited. This brings new credit to some very restrictive measures (such as the Nobel Prize award) due to their worldwide recognition and already existing status. 3.1. Criteria for data collection Criteria for data collection within evaluation process have a much wider basis. Data are primarily selected at the level of each institution, for its specific purposes. The objective is to evaluate the variety and weight of universities’ inputs and outputs, drawing out the relations between them. The university evaluation compares its “production” with its own goals and means, as converse to its counterparts. An important share of the information process is done at the level of the university itself, allowing limited comparisons with other partners. The main objective is to identify indicators that represent the most complete range of intellectual activity: from production of knowledge to its use. Evaluation indicators must be both feasible and useful. For this, they must answer a set of criteria. Below are mentioned the criteria adopted by a European project, the “Meritum”, that agreed upon a set of characteristics leading to a set of quantitative indicators (MERITUM, 2002).
Figure 2. Characteristics required for evaluation indicators
USEFUL Relevant Comparable Reliable SignificantUnderstandableTimely Objective Truthful Verifiable Feasible
Bertrand BELLON
11
The degree of fulfillment of these eleven characteristics induces the degree of quality of the overall process. Details of each of these characteristics are: ─ useful: allows decision making both by internal and external users to occur. ─ relevant: provides information that can be modified or affirm the expectations of decision-makers. In such cases the information should be: ─ significant: related to issues critical for universities; ─ understandable: presented in a way it can easily be understood by potential users; ─ timely: available when it is required for analysis, comparison or decision-making purposes. Turning to the development of specific indicators, they should be comparable, that is, indicators should follow criteria generally accepted by all implicated organizations in order to allow for comparative analysis and benchmarking; and reliable, that is, users need to be able to trust them. To meet these criteria, indicators are required to be: ─ objective: the value is not affected by any bias arising from the parties involved in the preparation of the information’s interests; ─ Truthful: the information reflects the real situation; ─ Verifiable: it is possible to assess the credibility of the information it provides. Finally, calculation of all indicators should be cost-efficient, or feasible. That is to say, the information required for the proposed indicator and its computation should be easily obtained. The information from the university’s information system, or the cost of modifying those systems in order to obtain the required information should be lower than the benefits (whether private or social) arising from the use of this indicator.
12
Evaluation Practices and Methodologies
3.2. A matrix of indicators The European project on the Observatory of European University (OEU) has developed a framework to characterize universities’ research activities (at the moment, it does not include teaching activities) (see <www.prime.org>). The result is a two dimensional matrix, devoted to: ─ characterizing the status of university research management; ─ identifying the best performing universities; and, ─ comparing the settings within which universities operate.
Figure 3. Observatory of European University evaluation matrix
Tools Objectives Attractiveness Autonomy Strategic capabilities Differentiation profile Territorial embedding Funding Human resources Academic outcomes Third mission Governance
Source: Observatory of European University: PRIME Network <http://www.prime-noe.org>.
The matrix and its elements are as follows: ─ The first dimension of the matrix analyses thematic aspects of university management. The OEU research has considered five themes herein: the first two representing “inputs; the next two representing “outputs” and the fifth one representing the governance of the institution: ─ Funding: includes all budget elements, both revenues and expenses (total budget, budget structure, sources of funding, rules for funding and for management); ─ Human Resources: includes professors, researchers, research engineers and administrative staff, plus PhDs and post-docs (number, distribution, functions between research, teaching and management, staff turnover, and visiting and foreign fellows).
Bertrand BELLON
13
Human resources must be considered both as labor stocks (numbers of people) and as labor flows (human flows, mobility); ─ Academic Outcomes: includes articles and books, other academic publications, citations, and the knowledge embodied in PhDs being trained through research activities; ─ Third Mission: (the university “third mission” is noted as an addition to the two other “traditional” university missions: teaching and research) concerns the service outreach linkages between the university and its non-academic partners, e.g., industry, public authorities, international organizations, NGOs and the public-at-large (covering activities such as employment of alumni, patent and licenses, spin-off and job creation, support of public policy, consultancy and promotion and diffusion of Science and Research activities); ─ Governance: includes the process by which the university converts its inputs (funding and human resources) into research outputs (academic outcomes and third mission). It concerns the management of institutions, from both above the university (as in its manner of relations with government and other finance providers) and within the university. ─ The second dimension of the matrix deals with transversal issues that can be applied to each thematic category, identifying or measuring the capabilities of the university regarding its various stakeholders. The OEU research team has considered five transversal issues: ─ Attractiveness: Each university’s capacity to attract different resources (money, people, equipment, collaboration, etc.) within a context of scarcity. ─ Autonomy: Measures each university’s margin of maneuver, formally defined as the limits, established from external partners (mainly government and other finance providers), to which a university must conform. ─ Strategic Capabilities: Indicates each university’s actual ability to implement its strategic choices.
14
Evaluation Practices and Methodologies
─ Differentiation Profile: The main features of each university that distinguishes it from other strategic actors (competing universities and other research organizations) by its degree of specialization and degree of interdisciplinarity, etc. ─ Territorial Embedding: The geographical distribution of each university’s involvements, contacts, collaborations, and so on within a defined locale, i.e., being a measure of the “territorial utility” of the university activity. In the actual process of using the matrix, however, many adjustments have to be made, mainly to adjust complexity and feasibility. One of the most complex examples is the service or “third mission” dimension, for it requires adjusting business-type dimensions (intellectual property, contracts with industry, spin-offs, etc.) with social and policy dimensions (public understanding of science, involvement into social and cultural life, participation to policy-making, etc.). Thus, the following chart detailing this sphere of activity recalls various dimensions of the previous one, with concise added presentations of relevant data.
Figure 4. The “third mission” dimension, eight items for data collection
1. Human resources – Competencies trained through research transferred to industry (typical case of “embodied knowledge”). The essential indicator is: PhD students who work in industry, built upon numbers and ratios. The combination is important, since having a ratio of 100 percent, i.e., all work with industry with one PhD delivered might be far less relevant for industry than ratio of 25 percent based on twenty PhD students. 2. Ownership – Research leading to publications or patents; with a changing balance between them. The key indicators are: patent inventors (number and ratio) and returns to the university (via licenses form patents, copyrights, etc., calculated as a total amount/ratio to non-public resources). Other complementary indicators reflect the proactive attitude of the university (existence of patent office, numbers of patents taken by university). 3. Spin-offs – Indicators relevant here are composite ones, that is to say they take into consideration three following entries: – the number of incorporated firms; – the number of permanent staff involved;
Bertrand BELLON
15
– more qualitative involvement such as: the existence of support staff funded by university; the presence of business incubators; incentives for creation, funds for seed capital; strategic alliances with venture capital firms, etc. 4. Contracts with industry – The traditional indicators are number of contracts (some prefer number of partners, which is more difficult to assess), and total financial assets generated, the ratio of which be calculated vis-à-vis external resources. 5. Contracts with public bodies – With this axis, the “societal” dimension is entered. The key indicators here are contracts asked for by a public body in order to solve problems (versus academic research funding) – It is important here to differentiate “local” (or nearby environment) from “other” (mostly national in large countries, may be quite international in small countries) contracts. – Elements for analysis are the same as for industrial contracts, i.e., number, volume, ratio 6. Participation into policy-making – Qualitative context: to build a composite index based on involvement in certain activities, with yes/no entries and measures of importance included. – List of activities to consider includes: norms/standards/regulation committees, expertise, formalized public debates. 7. Involvement in social and cultural life – Qualitative context: a composite index concerning specific investments, existence of dedicated research teams, or involvement in specific cultural and social developments. 8. Promoting the public’s understanding of science – Qualitative context: another composite index built on specific events to promote science, to classical involvement of researchers into dissemination and other forms of public understanding of science, including articles, TV appearances; books, films, etc.
Source: Observatory of European University: PRIME Network. <http://www.primenoe.org>.
4. Lessons Drawn from Evaluation Processes
Based on the author’s considerations of the evaluation processes, this fourth section will suggest ideas to improve the academic ranking processes, not with the aim of creating new conclusions, but to provide new elements for consideration in the ranking versus evaluation debate. The multiplication of evaluation processes facilitates new competition between universities, greatly modifying the existing dynamics of science. Researchers are now faced with a multiple model, which challenges “big
16
Evaluation Practices and Methodologies
science” (and the Nobel Prizes it brings) with new forms of “co-operative science” and more “internally driven” research strategies. This new landscape, with a wider variety of dynamic models, must now be taken into account. At this point, strong arguments exist to advocate a radical divergence between evaluation (and its characterization of universities) and ranking. On a strictly critical approach, there exist, on one side, ranking processes, limited to structurally crude bibliometric approaches, based on the smallest most visible parts of “output” of the complex process of knowledge. The risk appears that such a limited focus will lead to a caricatured vision of university missions, providing almost no possibility to draw useful relations between input and output. The existing set of indicators for the Jiao Tong University ranking is:
Criterion Quality of Education Quality of Faculty Research Output Size of Institution
Indicator Alumni of an institution winning Nobel Prizes and Fields Medals Staff of an institution winning Nobel Prizes and Fields Medals Highly cited researchers in 21 broad subject categories Articles published in Nature and Science* Articles in Science Citation Index-expanded and Social Science Citation Index Academic performance with respect to the size of an institution
On the other side, there exist evaluation processes, which are overcomplex, too qualitative and subjective, appearing restricted to internal use by each individually evaluated university. External comparisons are thus limited to the benchmarking of specific functions between selected universities. At first glance, therefore, it seems evaluation processes may not be well adapted to making global comparisons between universities. From this perspective, ranking and evaluation processes stand opposed to one another. Yet, from the perspective of their objectives, they seem very close. That is, both aim to meet the need of better efficiency via a better management of university missions. The “ways and means” for institutional ranking have already progressed, but they can still be greatly improved. In this respect, ranking can greatly benefit from certain indicators being used in the act of evaluation, but only when they can be generalized. The question now remaining is how to best
Bertrand BELLON
17
identify relevant indicators and discover the best way to produce them within the strict governing limit of means. 4.1. Ranking must reflect a minimum of diversity The “Nobel Prize” model’s main limitation is its strong reference to the “one best way” model, which is conceptually inadequate, considering the actual worldwide competition of universities, based on differentiation of competencies and a competition in a limited number of specific fields (e.g., Nano- and Macro-Sciences). At this point, a preliminary debate would be needed, which would clarify the objectives of academic ranking by moving away from a monolithic vision of “world-class universities” toward a set of criteria adequate to measure the diverse strategic objectives of universities with differentiated development trajectories. Moreover, ranking processes must also take into account the variety of meanings given to each indicator. An indicator may be efficient in one case and totally misused in another. Thus it can rightly be argued that: ─ What is useful or relevant for one university, in one scientific field, is not systematically useful or relevant for another university specialized in other domains, with other constraints and objectives; and, ─ What is useful or relevant for public authority or other stakeholders is not systematically useful or relevant for the university itself. The choice of a “universal” set of defining characteristics of “excellence” will nonetheless end with the splitting of universities into different categories; as it is the case for any organized championship match. 4.2. The set of characteristics should fulfill input, output and governance indicators The “relative utility” or relevance of an indicator is its ability to be used as a tool for university management (finance, governance and work). Indicators must provide access to the university’s production spectrum and differentiation profile. The first two improvements concern the discontinuance of some existing indicators and the adoption of more appropriate ones, for example:
18
Evaluation Practices and Methodologies
─ Differentiation by discipline or scientific field (including Social Sciences). ─ Introduction of significant input data and production of some “input/output” ratio. ─ Development of indicators for local “embedded ness” and global reach (i.e., local and global impact of universities). ─ Enlargement towards effective teaching indicators (as compared to research). 4.3. These changes will require specific collections of data: new indicators mean new work Such a renewal project will demand specific computation of existing information as well as the creation of new information. New methodological work has to be done, in addition to the creation of normalized measures necessary for the rebuilding of global indicators. For this to happen, effective connections with the OECD are crucial. 4.4. Enrich the debate in order to enrich the process and the result The debate on academic ranking will grow in importance in the future, and will not be limited to a simple evaluation-ranking dispute. At this point, four questions arise: ─ What does excellence mean, and what is its impact on research and teaching orientations and activities? ─ How important is the degree of diversity within globalization (not being limited to the dualistic global/local debate)? ─ What are the differences and specificities within processes of production, productivity and visibility? ─ Finally (under a transversal approach), a debate appears, questioning the quality of data themselves and their adaptation to the diversity of legal, financial and administrative structure of the bodies that form “universities”.
Bertrand BELLON
19
5. Conclusion: The Missing Link
In looking over the ranking versus evaluation debate, a factor has come to be seen by the author as central: the impact of the “world ranking” process upon the development dynamics of universities. He posits this because the vast majority of universities in the world is not, and has no chance to be, listed within any “world-class” list. They may possess niches of world-class excellence and they may produce excellent output. But, for these universities, the impact of academic ranking is either non-existent or negative (why would a university fight to get “in” if there is no hope to “win” or even to be visible and respected?). At the opposite side, the “elected” universities (those that find themselves within the 1000; 500; 100 “top” universities, in one or many different rankings) will incorporate ranking commitment and criteria, both within their daily management and within their long-term development strategy. As a result, they will “naturally” select the indicators that have been already selected by the ranking producers and will make them mandatory to their component group members (e.g., professors will be pushed to “publish or perish” even if the resulting research is less than useful). In such cases, artificial ranking criteria become “the” new rules that will be adopted and enforced by the universities themselves. The movement’s ideology and methods are thereby self-reinforced (as in the case of the heavy elemental weight given to the Nature and Science reviews in the Shanghai ranking) with possible negative effects on the generation of new hypotheses and academic fields, on the diversity of supported research, and on interdisciplinary co-operation. In some cases, academic ranking may have an unexpected structural side effect. If “university” becomes the unit of evaluation and of action, the current fragmentation of the French higher education system into universities, Grandes écoles and specialized research bodies (they may share academic staff and research activities with the University as well as research bodies such as CNRS, the ENSCM have particular status as independent bodies with their own research laboratories) is made to appear to be completely outdated, whatever its actual rationality. On the other hand, one of the results may be the reinforcement of recent moves to increase the size of universities by merging existing organizations with limited consideration to their real coherence and synergies.
20
Evaluation Practices and Methodologies
The evident impact on university management is of important consequence. University activity is increasingly embedded into a multiactor social space that modifies the governance of research, of innovation and of teaching, taking part within a new dynamic within the public sector. Consequently, institutional ranking processes, along with other tools for unit characterization, may provide original and useful information in the difficult process of university management: to create and consolidate platforms of quantitative data in the act of measuring the multidimensional nature of performance. Regarding external stakeholders, ranking introduces new rationales for public intervention and for the incorporation of new actors. Considering its implications on policy-making both for governments and for the universities themselves, ranking opens a whole new field of research. In short, the debate on university ranking (and on differentiating characterizations in general) is just beginning.
References
Azagra Caro, J. M., Fernández de Lucio, I. and A. Gutiérrez Gracia (2001) “University patent knowledge: the case of the Polytechnic University of Valencia”, 78th International Conference AEA, November 22-23, 2001, Brussel, Belgium. Balconi, M., Borghini, S., and A. Moisello (2003). “Ivory tower vs. spanning university: il caso dell’Università di Pavia”, in, Bonaccorsi, A. (ed.), Il sistema della ricerca pubblica in Italia. Milano: Franco Angeli, pp. 133-175. Carayol, N. and M. Matt (2004) “Does Research Organization Influence Academic Production? Laboratory Level Evidence from a Large European University”, Research Policy 33, p. 1081-1102. Dasgupta, P., and P. A. David (1994) “Towards a New Economics of Science”, Research Policy 23 5, p. 487-521. Etzkowitz, H. and L. Leydesdorff (2000) “The dynamics of innovation: from National Systems and “Mode 2” to a Triple Helix of universityindustry-government relations”, Research Policy 29, p. 109-123.
Bertrand BELLON
21
Etzkowitz, H., and L. Leytesdorff (1997) Universities in the Global Economy: A Triple Helix of academic-industry-government relation. London: Croom Helm. European Commission (2003) “Downsizing And Specializing: The University Model For The 21st Century?”, Snap Shots from the Third European Report on Science and Technology Indicators 2003, p. 1-2. <ftp://ftp.cordis.lu/pub/indicators/docs/3rd_report_snaps10.pdf> Fielden, J. and K. Abercromby (1969) “Accountability and International Co-Operation in the Renewal of Higher Education”, in, UNESCO Higher Education Indicators Study. UNESCO and ACU-CHEMS. <http://unesdoc.unesco.org/images/0012/001206/120630e.pdf>. Gibbons, M. (2004) The New Production of Knowledge. London: Sage. Helpman, E. (1998) General Purpose Technologies and Economic Growth, Cambridge, MA: Massachusetts Institute of Technology Press. Jacob, M., Lundqvist, M. and H. Hellsmark (2003) “Entrepreneurial transformations in the Swedish University system: the case of Chalmers University of Technology”, Research Policy 32, p. 1555–1568 Martin, B. R. and H. Etzkowitz (2000) “The Origin and Evolution of the University Species”, VEST 13 3-4, p. 9-34. MERITUM. Cañibano, L., Sánchez, P., García-Ayuso, M. and C. Chaminade (Eds.) (2002). Guidelines for Managing and Reporting on Intangibles: Intellectual Capital Statements. Madrid: Vodafone Foundation. Mowery, D. C, and B. N. Sampat (2004) “Universities in national innovation systems”, Chapter 8, in, J. Fagerberg, D. C. Mowery and R.R. Nelson (eds.), Oxford Handbook of Innovation, Oxford: Oxford University Press. Musselin, C. and S. Mignot Gerard. (2004) Analyse comparative du gouvernement de quatre universities (Comparative analysis of the government of four universities). <http://www.cpu.fr/Telecharger/ rapport_qualitatif_gvt.pdf>. Noyons, E. C. M., Buter, R. K., van Raan, A. F. J., Schmoch, U., Heinze, T., Hinze, S., and R. Rangnow (2003) “Mapping Excellence in Science
22
Evaluation Practices and Methodologies
and Technology across Europe”. Life Sciences. Report to the European Commission. Leiden University. Observatory of European University (2005). PRIME Network of Excellence. October. <http://www.prime-noe.org>. Reale, E. and B. Potì (2003) “La ricerca universitaria”, in, Scarda A. M. (ed.), Rapporto sul sistema scientifico e tecnologico in Italia, Milano:Angeli, p. 79-99.
By Bertrand BELLON
1. Introduction
Hierarchical ranking is the most common and simplest instrument of comparison between discrete indicators. It is the quickest way to obtain a comparison between competing entities, as long as objectives, rules of behavior and the relevant measurement tools are shared within a community (school class, athletic group, business, technology… measured by marks, speed, profit, financial assets, etc.). As an instrument of evaluation, ranking can be applied to almost any criteria. From school marks to the book of records (that hold the attention on the individual who first performed record figures), ranking is used everywhere as a way of measurement and of comparison. How can one improve this unavoidable measure? If one agrees with the choice of data, ranking presents no difficulties excepting the quality of data collection. According to a criterion or a set of criteria, the researcher ranks an entity number one, until another entity surpasses it under the accepted criteria. This process has been accelerated within an environment marked by increasingly open societies and expanding economies and societies, providing ranking with a new era of development. Even so, ranking appears highly problematic when dealing with complex and intangible goods such as knowledge, science, technology and innovation. In the case of the “production” of universities, simple criteria do not apply due to the high complexity level (which is the case in most dimensions of Social Sciences). Ranking may help at first in making crude distinctions, but it immediately becomes a limited instrument, for there is no “unique best way” to apply it in any human activity. Given the fact that there are many possibilities to improve the ranking process within its own rules and limits, this chapter intends to “drain” from the methodology of evaluation several elements with which to improve ranking of “world-class universities”. The author will begin with the
2
Evaluation Practices and Methodologies
extension of needs regarding a better understanding of university structures and strategies in the present times in Section 1. He will then underline the diversity of the objectives of evaluation, comparing them to the simpler, thus easier to understand, objectives of ranking in Section 2. Then, he will recall a selected panel of indicators, which can be managed within evaluation process in Section 3. Section 4 will show lessons drawn from evaluation indicators to improve the ranking activity. In conclusion, the author will revisit a few core questions about the goals of ranking and evaluation.
2. The Increasing Need for a Better Characterization of Universities
In an open economy and society, the characterization of academic activity and of performances is not only a concern for transversal authorities [for years, the OECD has published indicators on education and research and UNESCO has produced an important study on performance indicators and universities (Fielden and Abercromby, 1969)] but also an increasing need for each individual University. Characterization is thereby jointly related to measures of absolute excellence, and toward self-improvement within each specific context. Better understanding of a university’s stand-point, better management of the given assets, better efficacy and output, are thus mandatory, both for the locally embedded as well as for the world-class university. Yet, characterization raises two different questions: ─ how well do universities perform when their goals and means are taken into account?; and, ─ how is a particular university better, equivalent or worse than its competitors? Interestingly, however, these questions are not the same according to everybody, for they inevitably differentiate according to the values of the universities” stakeholders. 2.1. External stakeholders Universities gather a wide variety of stakeholders (internal and external) who are increasingly active and concerned with the way they are managed
Bertrand BELLON
3
and with their results. These partners becoming “drivers”, their requirements differ widely from one to the other. That is to say: Public authorities involved into university support are increasingly concerned with the use of public money. The main share of universities’ budgets still depends mainly on public decisions (in coherence with Adam Smith and Alfred Marshal theory of external effects being attributed to knowledge and education). In a period of relative shortage of public budgets – due to increasing competition between states, to the non interventionism ideology and to cuts into budget deficits – an increasing responsibility is put on public project managers, with more attention being paid to results (and consequently to the universities’ management mode). Taxpayers are increasingly reactive to the way their money is used by public as well as private research institutions. This often justifies political and public financial short-term views, in contrast to the long-term dimension of the research process and the complex articulation between Fundamental and Applied Sciences. Universities have become increasingly decisive tools for economic competitiveness, knowledge and innovation. This has led industries to be directly concerned with university possesses (e.g., hiring skilled students and benefiting as directly as possible from new research). This concerns not only high tech industries, but also includes every “mid-tech” and “low-tech” business that is involved into innovation and to increasing use of general purpose technology (Helpman, 1998). Finally, journalists and other opinion makers are very active in universities’ visibility. They create and convey the images – given to people as proven reality – emphasizing both fictive and real strengths and weaknesses of universities. So, one can well see that universities have become increasingly in debt to, or at least dependent upon an increasing number of external partners, such as taxpayers, government administration and politicians, national and international organizations, business managers, journalists, as well as foundations and NGOs, etc. For various reasons, those external stakeholders focus on the final “outputs” of universities. At best, they require information concerning the relation between material and labor “inputs” (what they have paid for) and “output” or “production”. Thus, external stakeholders are largely unconcerned with the two central processes of university activity, i.e., the production of new knowledge, and the teaching-learning process
4
Evaluation Practices and Methodologies
between professors and students. Indeed, in most cases, the university remains a mysterious “black box” to them, a vision reinforced by the very complexity of these two intangible, ambiguous (and therefore hard-toevaluate) production processes. Hence, these complex problems of education, learning, researching, and governing these institutions are left to specialists. 2.2. Internal interest and need for self-evaluation of universities External interests are not the only university’s partners, however. University managers, students, professors, researchers, administrative staffs, are the other major internal partners that come to the institution with their specific interests and objectives. Students also participate in the openness of economies. This is done by their “shopping” among universities worldwide, according to their own objectives, capabilities and means. Students might choose a university because it is nearest to their home, but, more and more often, they will make their choices according to institutions’ and diplomas’ fame, given that their main concern is to increase their chance of finding rewarding jobs after graduation. They are found to be more discriminating as concerns foreign universities than between their own country’s universities, which increases the artificial differences carried by reputation and image, as compared to real relative capabilities. Researchers and professors tend to field multiple job applications – simultaneously among diverse universities – looking for the “most famous” one or, barring that, the one that provides them the best facilities for research and teaching. “Quality of life” issues, in their various dimensions, from day-to-day particulars to lifelong career prospects, are the main determinants of their final choice. Working amid such “driver behaviors”, university managers carry the responsibility to achieve the optimum of production and productivity out of the two above mentioned groups, by building a working coherence from heterogeneous and highly individualistic behaviors. Each internal partner is thus a stakeholder, carrying forward his or her own objectives, governed, in part, by an interfacing of personal and social opinions and abilities of self and others. This makes the university resemble a network of competing interests. In this regard, the ultimate responsibility
Bertrand BELLON
5
of its management is to provide a minimum degree of satisfaction to each partner. Therefore, there is a great need by universities for finer abilities of inward understanding and evaluation. It is known that academic ranking has different meanings according to who is looking at it. In this context it is not surprinzing that university managers interest in academic ranking has been two-fold, i.e., fostering better management of universities and consolidating a new field of research about the production and diffusion of knowledge. Other objectives can be added to these, each one bringing its own consequences to bear upon the work to be done, but the “ranking user” issue remains the central one. As such, it will have effects on the whole process of academic measurement, including the choice of indicators and methods of data collection. 2.3. The Multi-helix model The new exigency of information and control, initiated from the double “inside-outside” stakeholder demand is strengthened by the increasing interest of researchers in Science and Technology production and in learning activities and processes. The character of this interest is situated at the intersection of four vectors. A growing interest in “macro” studies, that is, making large comparisons of data, and/or providing general historical perspectives of trends (Mowery and Sampat, 2004; Martin and Etzkowitz, 2000). Many trend studies are based on historical cases of specific university (Jacob, Lundqvist and Hellsmark, 2003). The renewed attention to “excellence” among competing universities, can also be associated with scientometric benchmarking and patent analysis (Noyons, Buter, van Raan, Schmoch, Heinze, Hinze, Rangnow, 2003), as well as, in the United States of America, Canada and in Europe (Balconi, Borghini, Moisello, 2002), (Carayol and Matt, 2003-2004), (Azagra Caro et al., 2001). Emerging questions on the strategies of universities (European Commission, 2004) and related issues including the governance of universities (Musselin and Mignot Gerard, 2004; Reale and Potí, 2003) and the organization of research activities: laboratories and research centres versus teaching departments, interdisciplinary versus disciplinary
6
Evaluation Practices and Methodologies
organizations, allocation and rewarding mechanisms, articulation between teaching and research positions, role of research in career stages, etc.). Finally, every public institution involved in R&D and education is increasingly interested in studies on the production process of knowledge. Regional observatories have therefore been created as instruments for orientation of funding decisions (as has been done at the European, national and local levels, even including medium-sized cities). This representation is currently enlarged with the relations developed between the university and the public at large and the multiplicity of organizations that belong neither to government nor to business (e.g., NGOs, international organizations, foundations, multilateral, European and regional entities, cities, etc. Except for the hardware necessary to conduct research, both academic inputs and outputs are intangibles. In consequence, only a small part of such intangibles are identified and thus very limited instruments exist to measure them (Cañibano and Sánchez, 2004). Furthermore, research in such a science requires multidisciplinary work: Sociology, Economics, Science Policy, Management, etc. Yet, one finds recent theoretical developments have brought some interesting benefit to this field of study. Partha Dasgutpa and Paul David (1994) have suggested a framework for a new Economics of Science, Michael Gibbons (2004) has identified the “Mode 2” concept of research, Henry Etzkowitz and Loet Leydesdorff have popularized the “Triple Helix” concept as a way to see government, academia and industry as parts of one structure (1997). In sum, the complex relation system (University-Industry-Government) “increasingly provides the knowledge infrastructure of society” (Etzkowitz and Leydesdorff, 2000; p.1). The model verifies that: 1) the relationships among universities, industry and government are changing; and, 2) there are internal transformations in each of these individual sectors as well. Consequently, universities are not just teaching or research institutions, but combine teaching, research and service toward and for society. In other words, within a knowledge-based economy the noted triple helix model turns into a multi-helix one, with the main function being given over to universities. It is this growing sphere of social function and responsibility that explains growing pressures for an accounting of resources employed and deployed by universities, yet they are given no unique set of criteria given by which to measure their performances.
Commen gath er the auth or mea ns orga niza tion s belo ng neit her to nati onal gov ern men t nor busi ness in this pass age, give n som e of the exa mpl es liste d (i.e. , citie s).
Bertrand BELLON
7
Figure 1. The Multi-helix Model
2. Ranking versus Evaluation Processes
In this context, evaluation processes will take different forms and include different objectives according to different problems and different missions of institutions. These can be distinguished thusly: ─ evaluation (to fix the value, and measure the position regarding objectives or partners); ─ monitoring (to verify the process of the activity; to admonish, alert and remind);
8
Evaluation Practices and Methodologies
─ control (to verify an experiment by duplication of it and comparison); ─ accountability (to be able to provide explanations to stakeholders for actions taken); ─ ranking (to put individual elements in order and relative position; to classify according to certain criteria). Complicating matter, however, is the fact that the “field” of the university is composed of ideas, knowledge, information, communication, etc., which are typically unique non-positional, hence, “non-rival goods”. 2.1. Taking into account the diversity of higher educational missions Because universities deal with the creation and the diffusion of knowledge, the varieties of their missions are endless. These composite varieties are the joint result of multiple knowledge characteristics and of the variety of each university’s stakeholders and their concerns. The first partner will focus on the ability to train wider numbers of students; the second to increase the international research network; the third will consider as a priority to produce Nobel Prize or Fields Medal winners, etc. However, whatever the diversity, every stake holding group will be concerned in gaining better recognition and visibility of its university. In addition, university functions are ever growing in diversity. This is due to the enlargement of the boundaries of scientific thought well into the area beyond the laboratory, thus facing increasing pressures to introduce new applied technologies into day-to-day life (i.e., into the production of goods and services). On a synthetic level, a university can be characterized as having a double mission of training (basic and continuing education) and of researching (production and diffusion of new knowledge). Beyond these, a “third mission” of universities (Spru, 2002) of providing services to society is growing in importance, with broad social-economic impacts, encompassing both profit and non-for-profit output). 2.2. Evaluation as a starting point When using different individual evaluations to compare universities, researchers are faced with a strong difficulty to agree upon indicators and to proceed to efficient benchmarking. At best, universities will be comparable when they share similar goals and they benefit from similar means – which
Bertrand BELLON
9
is very rarely the case. Yet, one can consider the ongoing processes of evaluation as trials to identify the useful indicators for a given set of questions and for a set of universities. The characterization of universities is thus a first step toward the benchmarking of universities worldwide. In the evaluation processes, various types of practices and meanings can be considered: ─ discipline-based versus institution-based evaluation; ─ inputs-based versus outputs-based evaluation; ─ internal versus external evaluation; ─ qualitative versus quantitative evaluation. 2.3. The ranking process The worldwide liberalization of markets and societies has created a new global competition among universities. When considering research and teaching, universities considered as the “best” universities will attract more talented students; attracting the “best” students, they will thus be able to reinforce their capabilities for autonomous selection-processes for their own benefit. Academic ranking intends to provide means of comparison and identification between universities according to their academic or research performance. Yet, the result will be twofold: on the one side, an increasing need for worldwide universities of excellence, on the other side, an increasing need also for local universities that will be specialized at providing college-level (rather than at doctoral-level) training, including a strong commitment to regional issues and development, corresponding to the eventual creation of disciplinary “niches” for research at a level of excellence. There is a gap, and often an unbridgeable one, between evaluation processes and ranking processes. As far as this work is concerned, it will focus on ranking considerations. Given that ranking must be based upon incontestable (or at least objective) indicators, the objective of this author’s contribution is to take into consideration a set of experiences growing from specific evaluation processes toward the methodology of benefit-ranking production.
10
Evaluation Practices and Methodologies
3. Evaluation Indicators
This section will present data and indicators commonly used for evaluation, the objective being to deduct a few especially grounded data that can be adapted and used for general ranking processes. In so doing, however, it is vital to keep in mind that ranking on a worldwide scale induces very strict constraints that limit the number of possible indicators that can be employed. Therefore, indicators will be limited to those that are: already existing, comparable, and, easy to collect. This explains why the selection of indicators will be very limited. This brings new credit to some very restrictive measures (such as the Nobel Prize award) due to their worldwide recognition and already existing status. 3.1. Criteria for data collection Criteria for data collection within evaluation process have a much wider basis. Data are primarily selected at the level of each institution, for its specific purposes. The objective is to evaluate the variety and weight of universities’ inputs and outputs, drawing out the relations between them. The university evaluation compares its “production” with its own goals and means, as converse to its counterparts. An important share of the information process is done at the level of the university itself, allowing limited comparisons with other partners. The main objective is to identify indicators that represent the most complete range of intellectual activity: from production of knowledge to its use. Evaluation indicators must be both feasible and useful. For this, they must answer a set of criteria. Below are mentioned the criteria adopted by a European project, the “Meritum”, that agreed upon a set of characteristics leading to a set of quantitative indicators (MERITUM, 2002).
Figure 2. Characteristics required for evaluation indicators
USEFUL Relevant Comparable Reliable SignificantUnderstandableTimely Objective Truthful Verifiable Feasible
Bertrand BELLON
11
The degree of fulfillment of these eleven characteristics induces the degree of quality of the overall process. Details of each of these characteristics are: ─ useful: allows decision making both by internal and external users to occur. ─ relevant: provides information that can be modified or affirm the expectations of decision-makers. In such cases the information should be: ─ significant: related to issues critical for universities; ─ understandable: presented in a way it can easily be understood by potential users; ─ timely: available when it is required for analysis, comparison or decision-making purposes. Turning to the development of specific indicators, they should be comparable, that is, indicators should follow criteria generally accepted by all implicated organizations in order to allow for comparative analysis and benchmarking; and reliable, that is, users need to be able to trust them. To meet these criteria, indicators are required to be: ─ objective: the value is not affected by any bias arising from the parties involved in the preparation of the information’s interests; ─ Truthful: the information reflects the real situation; ─ Verifiable: it is possible to assess the credibility of the information it provides. Finally, calculation of all indicators should be cost-efficient, or feasible. That is to say, the information required for the proposed indicator and its computation should be easily obtained. The information from the university’s information system, or the cost of modifying those systems in order to obtain the required information should be lower than the benefits (whether private or social) arising from the use of this indicator.
12
Evaluation Practices and Methodologies
3.2. A matrix of indicators The European project on the Observatory of European University (OEU) has developed a framework to characterize universities’ research activities (at the moment, it does not include teaching activities) (see <www.prime.org>). The result is a two dimensional matrix, devoted to: ─ characterizing the status of university research management; ─ identifying the best performing universities; and, ─ comparing the settings within which universities operate.
Figure 3. Observatory of European University evaluation matrix
Tools Objectives Attractiveness Autonomy Strategic capabilities Differentiation profile Territorial embedding Funding Human resources Academic outcomes Third mission Governance
Source: Observatory of European University: PRIME Network <http://www.prime-noe.org>.
The matrix and its elements are as follows: ─ The first dimension of the matrix analyses thematic aspects of university management. The OEU research has considered five themes herein: the first two representing “inputs; the next two representing “outputs” and the fifth one representing the governance of the institution: ─ Funding: includes all budget elements, both revenues and expenses (total budget, budget structure, sources of funding, rules for funding and for management); ─ Human Resources: includes professors, researchers, research engineers and administrative staff, plus PhDs and post-docs (number, distribution, functions between research, teaching and management, staff turnover, and visiting and foreign fellows).
Bertrand BELLON
13
Human resources must be considered both as labor stocks (numbers of people) and as labor flows (human flows, mobility); ─ Academic Outcomes: includes articles and books, other academic publications, citations, and the knowledge embodied in PhDs being trained through research activities; ─ Third Mission: (the university “third mission” is noted as an addition to the two other “traditional” university missions: teaching and research) concerns the service outreach linkages between the university and its non-academic partners, e.g., industry, public authorities, international organizations, NGOs and the public-at-large (covering activities such as employment of alumni, patent and licenses, spin-off and job creation, support of public policy, consultancy and promotion and diffusion of Science and Research activities); ─ Governance: includes the process by which the university converts its inputs (funding and human resources) into research outputs (academic outcomes and third mission). It concerns the management of institutions, from both above the university (as in its manner of relations with government and other finance providers) and within the university. ─ The second dimension of the matrix deals with transversal issues that can be applied to each thematic category, identifying or measuring the capabilities of the university regarding its various stakeholders. The OEU research team has considered five transversal issues: ─ Attractiveness: Each university’s capacity to attract different resources (money, people, equipment, collaboration, etc.) within a context of scarcity. ─ Autonomy: Measures each university’s margin of maneuver, formally defined as the limits, established from external partners (mainly government and other finance providers), to which a university must conform. ─ Strategic Capabilities: Indicates each university’s actual ability to implement its strategic choices.
14
Evaluation Practices and Methodologies
─ Differentiation Profile: The main features of each university that distinguishes it from other strategic actors (competing universities and other research organizations) by its degree of specialization and degree of interdisciplinarity, etc. ─ Territorial Embedding: The geographical distribution of each university’s involvements, contacts, collaborations, and so on within a defined locale, i.e., being a measure of the “territorial utility” of the university activity. In the actual process of using the matrix, however, many adjustments have to be made, mainly to adjust complexity and feasibility. One of the most complex examples is the service or “third mission” dimension, for it requires adjusting business-type dimensions (intellectual property, contracts with industry, spin-offs, etc.) with social and policy dimensions (public understanding of science, involvement into social and cultural life, participation to policy-making, etc.). Thus, the following chart detailing this sphere of activity recalls various dimensions of the previous one, with concise added presentations of relevant data.
Figure 4. The “third mission” dimension, eight items for data collection
1. Human resources – Competencies trained through research transferred to industry (typical case of “embodied knowledge”). The essential indicator is: PhD students who work in industry, built upon numbers and ratios. The combination is important, since having a ratio of 100 percent, i.e., all work with industry with one PhD delivered might be far less relevant for industry than ratio of 25 percent based on twenty PhD students. 2. Ownership – Research leading to publications or patents; with a changing balance between them. The key indicators are: patent inventors (number and ratio) and returns to the university (via licenses form patents, copyrights, etc., calculated as a total amount/ratio to non-public resources). Other complementary indicators reflect the proactive attitude of the university (existence of patent office, numbers of patents taken by university). 3. Spin-offs – Indicators relevant here are composite ones, that is to say they take into consideration three following entries: – the number of incorporated firms; – the number of permanent staff involved;
Bertrand BELLON
15
– more qualitative involvement such as: the existence of support staff funded by university; the presence of business incubators; incentives for creation, funds for seed capital; strategic alliances with venture capital firms, etc. 4. Contracts with industry – The traditional indicators are number of contracts (some prefer number of partners, which is more difficult to assess), and total financial assets generated, the ratio of which be calculated vis-à-vis external resources. 5. Contracts with public bodies – With this axis, the “societal” dimension is entered. The key indicators here are contracts asked for by a public body in order to solve problems (versus academic research funding) – It is important here to differentiate “local” (or nearby environment) from “other” (mostly national in large countries, may be quite international in small countries) contracts. – Elements for analysis are the same as for industrial contracts, i.e., number, volume, ratio 6. Participation into policy-making – Qualitative context: to build a composite index based on involvement in certain activities, with yes/no entries and measures of importance included. – List of activities to consider includes: norms/standards/regulation committees, expertise, formalized public debates. 7. Involvement in social and cultural life – Qualitative context: a composite index concerning specific investments, existence of dedicated research teams, or involvement in specific cultural and social developments. 8. Promoting the public’s understanding of science – Qualitative context: another composite index built on specific events to promote science, to classical involvement of researchers into dissemination and other forms of public understanding of science, including articles, TV appearances; books, films, etc.
Source: Observatory of European University: PRIME Network. <http://www.primenoe.org>.
4. Lessons Drawn from Evaluation Processes
Based on the author’s considerations of the evaluation processes, this fourth section will suggest ideas to improve the academic ranking processes, not with the aim of creating new conclusions, but to provide new elements for consideration in the ranking versus evaluation debate. The multiplication of evaluation processes facilitates new competition between universities, greatly modifying the existing dynamics of science. Researchers are now faced with a multiple model, which challenges “big
16
Evaluation Practices and Methodologies
science” (and the Nobel Prizes it brings) with new forms of “co-operative science” and more “internally driven” research strategies. This new landscape, with a wider variety of dynamic models, must now be taken into account. At this point, strong arguments exist to advocate a radical divergence between evaluation (and its characterization of universities) and ranking. On a strictly critical approach, there exist, on one side, ranking processes, limited to structurally crude bibliometric approaches, based on the smallest most visible parts of “output” of the complex process of knowledge. The risk appears that such a limited focus will lead to a caricatured vision of university missions, providing almost no possibility to draw useful relations between input and output. The existing set of indicators for the Jiao Tong University ranking is:
Criterion Quality of Education Quality of Faculty Research Output Size of Institution
Indicator Alumni of an institution winning Nobel Prizes and Fields Medals Staff of an institution winning Nobel Prizes and Fields Medals Highly cited researchers in 21 broad subject categories Articles published in Nature and Science* Articles in Science Citation Index-expanded and Social Science Citation Index Academic performance with respect to the size of an institution
On the other side, there exist evaluation processes, which are overcomplex, too qualitative and subjective, appearing restricted to internal use by each individually evaluated university. External comparisons are thus limited to the benchmarking of specific functions between selected universities. At first glance, therefore, it seems evaluation processes may not be well adapted to making global comparisons between universities. From this perspective, ranking and evaluation processes stand opposed to one another. Yet, from the perspective of their objectives, they seem very close. That is, both aim to meet the need of better efficiency via a better management of university missions. The “ways and means” for institutional ranking have already progressed, but they can still be greatly improved. In this respect, ranking can greatly benefit from certain indicators being used in the act of evaluation, but only when they can be generalized. The question now remaining is how to best
Bertrand BELLON
17
identify relevant indicators and discover the best way to produce them within the strict governing limit of means. 4.1. Ranking must reflect a minimum of diversity The “Nobel Prize” model’s main limitation is its strong reference to the “one best way” model, which is conceptually inadequate, considering the actual worldwide competition of universities, based on differentiation of competencies and a competition in a limited number of specific fields (e.g., Nano- and Macro-Sciences). At this point, a preliminary debate would be needed, which would clarify the objectives of academic ranking by moving away from a monolithic vision of “world-class universities” toward a set of criteria adequate to measure the diverse strategic objectives of universities with differentiated development trajectories. Moreover, ranking processes must also take into account the variety of meanings given to each indicator. An indicator may be efficient in one case and totally misused in another. Thus it can rightly be argued that: ─ What is useful or relevant for one university, in one scientific field, is not systematically useful or relevant for another university specialized in other domains, with other constraints and objectives; and, ─ What is useful or relevant for public authority or other stakeholders is not systematically useful or relevant for the university itself. The choice of a “universal” set of defining characteristics of “excellence” will nonetheless end with the splitting of universities into different categories; as it is the case for any organized championship match. 4.2. The set of characteristics should fulfill input, output and governance indicators The “relative utility” or relevance of an indicator is its ability to be used as a tool for university management (finance, governance and work). Indicators must provide access to the university’s production spectrum and differentiation profile. The first two improvements concern the discontinuance of some existing indicators and the adoption of more appropriate ones, for example:
18
Evaluation Practices and Methodologies
─ Differentiation by discipline or scientific field (including Social Sciences). ─ Introduction of significant input data and production of some “input/output” ratio. ─ Development of indicators for local “embedded ness” and global reach (i.e., local and global impact of universities). ─ Enlargement towards effective teaching indicators (as compared to research). 4.3. These changes will require specific collections of data: new indicators mean new work Such a renewal project will demand specific computation of existing information as well as the creation of new information. New methodological work has to be done, in addition to the creation of normalized measures necessary for the rebuilding of global indicators. For this to happen, effective connections with the OECD are crucial. 4.4. Enrich the debate in order to enrich the process and the result The debate on academic ranking will grow in importance in the future, and will not be limited to a simple evaluation-ranking dispute. At this point, four questions arise: ─ What does excellence mean, and what is its impact on research and teaching orientations and activities? ─ How important is the degree of diversity within globalization (not being limited to the dualistic global/local debate)? ─ What are the differences and specificities within processes of production, productivity and visibility? ─ Finally (under a transversal approach), a debate appears, questioning the quality of data themselves and their adaptation to the diversity of legal, financial and administrative structure of the bodies that form “universities”.
Bertrand BELLON
19
5. Conclusion: The Missing Link
In looking over the ranking versus evaluation debate, a factor has come to be seen by the author as central: the impact of the “world ranking” process upon the development dynamics of universities. He posits this because the vast majority of universities in the world is not, and has no chance to be, listed within any “world-class” list. They may possess niches of world-class excellence and they may produce excellent output. But, for these universities, the impact of academic ranking is either non-existent or negative (why would a university fight to get “in” if there is no hope to “win” or even to be visible and respected?). At the opposite side, the “elected” universities (those that find themselves within the 1000; 500; 100 “top” universities, in one or many different rankings) will incorporate ranking commitment and criteria, both within their daily management and within their long-term development strategy. As a result, they will “naturally” select the indicators that have been already selected by the ranking producers and will make them mandatory to their component group members (e.g., professors will be pushed to “publish or perish” even if the resulting research is less than useful). In such cases, artificial ranking criteria become “the” new rules that will be adopted and enforced by the universities themselves. The movement’s ideology and methods are thereby self-reinforced (as in the case of the heavy elemental weight given to the Nature and Science reviews in the Shanghai ranking) with possible negative effects on the generation of new hypotheses and academic fields, on the diversity of supported research, and on interdisciplinary co-operation. In some cases, academic ranking may have an unexpected structural side effect. If “university” becomes the unit of evaluation and of action, the current fragmentation of the French higher education system into universities, Grandes écoles and specialized research bodies (they may share academic staff and research activities with the University as well as research bodies such as CNRS, the ENSCM have particular status as independent bodies with their own research laboratories) is made to appear to be completely outdated, whatever its actual rationality. On the other hand, one of the results may be the reinforcement of recent moves to increase the size of universities by merging existing organizations with limited consideration to their real coherence and synergies.
20
Evaluation Practices and Methodologies
The evident impact on university management is of important consequence. University activity is increasingly embedded into a multiactor social space that modifies the governance of research, of innovation and of teaching, taking part within a new dynamic within the public sector. Consequently, institutional ranking processes, along with other tools for unit characterization, may provide original and useful information in the difficult process of university management: to create and consolidate platforms of quantitative data in the act of measuring the multidimensional nature of performance. Regarding external stakeholders, ranking introduces new rationales for public intervention and for the incorporation of new actors. Considering its implications on policy-making both for governments and for the universities themselves, ranking opens a whole new field of research. In short, the debate on university ranking (and on differentiating characterizations in general) is just beginning.
References
Azagra Caro, J. M., Fernández de Lucio, I. and A. Gutiérrez Gracia (2001) “University patent knowledge: the case of the Polytechnic University of Valencia”, 78th International Conference AEA, November 22-23, 2001, Brussel, Belgium. Balconi, M., Borghini, S., and A. Moisello (2003). “Ivory tower vs. spanning university: il caso dell’Università di Pavia”, in, Bonaccorsi, A. (ed.), Il sistema della ricerca pubblica in Italia. Milano: Franco Angeli, pp. 133-175. Carayol, N. and M. Matt (2004) “Does Research Organization Influence Academic Production? Laboratory Level Evidence from a Large European University”, Research Policy 33, p. 1081-1102. Dasgupta, P., and P. A. David (1994) “Towards a New Economics of Science”, Research Policy 23 5, p. 487-521. Etzkowitz, H. and L. Leydesdorff (2000) “The dynamics of innovation: from National Systems and “Mode 2” to a Triple Helix of universityindustry-government relations”, Research Policy 29, p. 109-123.
Bertrand BELLON
21
Etzkowitz, H., and L. Leytesdorff (1997) Universities in the Global Economy: A Triple Helix of academic-industry-government relation. London: Croom Helm. European Commission (2003) “Downsizing And Specializing: The University Model For The 21st Century?”, Snap Shots from the Third European Report on Science and Technology Indicators 2003, p. 1-2. <ftp://ftp.cordis.lu/pub/indicators/docs/3rd_report_snaps10.pdf> Fielden, J. and K. Abercromby (1969) “Accountability and International Co-Operation in the Renewal of Higher Education”, in, UNESCO Higher Education Indicators Study. UNESCO and ACU-CHEMS. <http://unesdoc.unesco.org/images/0012/001206/120630e.pdf>. Gibbons, M. (2004) The New Production of Knowledge. London: Sage. Helpman, E. (1998) General Purpose Technologies and Economic Growth, Cambridge, MA: Massachusetts Institute of Technology Press. Jacob, M., Lundqvist, M. and H. Hellsmark (2003) “Entrepreneurial transformations in the Swedish University system: the case of Chalmers University of Technology”, Research Policy 32, p. 1555–1568 Martin, B. R. and H. Etzkowitz (2000) “The Origin and Evolution of the University Species”, VEST 13 3-4, p. 9-34. MERITUM. Cañibano, L., Sánchez, P., García-Ayuso, M. and C. Chaminade (Eds.) (2002). Guidelines for Managing and Reporting on Intangibles: Intellectual Capital Statements. Madrid: Vodafone Foundation. Mowery, D. C, and B. N. Sampat (2004) “Universities in national innovation systems”, Chapter 8, in, J. Fagerberg, D. C. Mowery and R.R. Nelson (eds.), Oxford Handbook of Innovation, Oxford: Oxford University Press. Musselin, C. and S. Mignot Gerard. (2004) Analyse comparative du gouvernement de quatre universities (Comparative analysis of the government of four universities). <http://www.cpu.fr/Telecharger/ rapport_qualitatif_gvt.pdf>. Noyons, E. C. M., Buter, R. K., van Raan, A. F. J., Schmoch, U., Heinze, T., Hinze, S., and R. Rangnow (2003) “Mapping Excellence in Science
22
Evaluation Practices and Methodologies
and Technology across Europe”. Life Sciences. Report to the European Commission. Leiden University. Observatory of European University (2005). PRIME Network of Excellence. October. <http://www.prime-noe.org>. Reale, E. and B. Potì (2003) “La ricerca universitaria”, in, Scarda A. M. (ed.), Rapporto sul sistema scientifico e tecnologico in Italia, Milano:Angeli, p. 79-99.
