Abstract

Globalization and market-based orientation of higher education institutions has increased interest of students, parents, employers, universities, funding agencies, governments, and relevant stakeholders in knowing the rank of their concerned universities at national/global level. This has led to the emergence of several global university ranking systems. Aligned with international trends of ranking, Higher Education of Pakistan [HEC] also initiated ranking of universities at the national level in Pakistan. Subsequently, HEC designed comprehensive ranking criteria for ranking of universities and has implemented it since 2010. This study analyzes the nature of HEC ranking criteria and its constituent indicators from the perspective of global university ranking systems. Using content and thematic analysis, this study found that global university ranking systems mainly focus quality of research and teaching, while HEC additionally focuses effective and efficient use of resources, provision of facilities, social integration, and impact on community development.

Key Words

HEC Ranking, Universities, Quality Assurance

Introduction

Market-based orientation of higher education institutions and their international nature around the world has led students, parents, employers, universities, funding agencies, governments, and other stakeholders to take a huge interest in knowing the rank of their concerned university in comparison with national or global universities. This led to the emergence of several ranking systems. Ranking systems can be, therefore, viewed as a standard tool for inter-university comparison around the world (Hazelkorn, 2011). As a result, global university rankings are no longer a matter of concern only for leading universities but for most universities as they either want to seek their names in ranking or to improve their place/position in rankings (Rauhvargers, 2013). In this background, it is helpful to understand the origin of rankings.

While talking about the origin of ranking universities at national or international level, higher education institutions around the globe have not been ranked since long time. During the last couple of decades, however, an increased focus has been observed on ranking of universities. Ranking of universities has also been served as an evaluation tool. In past, staff surveys have been used for evaluating the quality of graduate programs in USA since 1920 (Harvey, 2008). Later in 1983, publication of US best colleges was initiated by World Report and US News (Clarke, 2002; Hou, Morse, & Chiang, 2012). Whereas in UK, ranking formally started in 1990 (Bowden, 2000). Later, ranking of universities got prominent position and within few years it became unavoidable part of academic life. Currently, universities are considering the notion of ranking on serious note.

The literature reveals several purposes of university ranking systems. For example, Vernon, Balas, and Momani (2018) asserted that most university ranking systems are used to identify the top universities for stakeholders, to classify universities on basis of research/teaching, and for comparing educational institutions around the world or within countries. Likewise, Sadlak (2006) argued that ranking is recognized approach, which includes subsequent methodologies to exhibit comparative picture of an organization or some areas of its performance. Ranking systems arrange organizations in each classification group by assessing their performance by indicators such as teaching, research, external judgment, and peer review (Hou et al., 2012).

The ranking of universities has also emerged as a result of several other reasons. There reasons include provision of vigorous competition among universities, development of excellence in universities, contributing supplementary basis for allocation of government funds, and for the provision of information to general public (Sadlak, 2006) from accountability perspectives. Vernon et al. (2018) stated that university ranking systems are usually used by administrators and academic 

institutions for getting reliable evaluation indicators of academic quality and research. Presently, ranking is also recognized as an external tool of quality assurance as it is a source of information for public and it provides opportunities for the assessment of higher education and functioning of its various aspects (Hou et al., 2012).

University rankings have also got prime position among the academia due to increasing access, for making comparisons among the institutions, and for the satisfaction of diverse stakeholders. For example, Hazelkorn (2011) argued that increasing access to higher education and economic and technological expansions have led universities to compare themselves with other national and international universities. This eventually led to the need of university ranking systems. Harvey (2008) stated that the ranking system is a standard way to make comparison between universities. Due to these reasons, ranking systems have consequently attracted universities to engage them in rankings. Ranking of universities has also emerged because of the concern of stakeholders such as students, employers, and parents (Bischoff, Gassmann, & Emrich, 2017).

Globalization has created environment for international rankings (Hazelkorn, 2011; Hou et al., 2012). In age of globalization, accountability also served as an important driver behind ranking (Altbach, 2012). Demands for ranking of universities were from stakeholders such as students, employers, researchers, institutions, funding agencies, and government (Bischoff et al., 2017). These groups are now more aware than ever before in past, as a result of rankings, about making decisions of studying, teaching, research and funding. For making good decision, stakeholders need correct data regarding the quality and nature of programs and institutions (Williams & Van Dyke, 2008).

For the last two decades, the world has witnessed some environmental changes, especially the information explosion to the public has made higher education system marketized (Grewal, Dearden, & Llilien, 2008). Universities are acting like firms in competitive market places and striving for effectiveness. Competition among universities for students’ admission, recruitment of qualified faculty, more fund raising, and improvement of rankings has been rapidly increased in recent years (Bischoff et al., 2017). Rankings are used as a tool by universities for attracting students, bringing alumni donations, hiring administrators/faculty, and for attracting potential donors (Bischoff et al., 2017; Grewal et al., 2008).

The above-mentioned set of drivers led to creation of several systems for ranking universities, developed by various agencies throughout the world. The most well-known rankings include, the ARWU (Shanghai Jiao Tong) ranking, Times Higher Education (THE) ranking system, and the Quacquarelli Symonds (QS) ranking (Shin, Toutkoushian, & Teichler, 2011). Other from these three ranking systems, more than 30 systems exist around the globe (Frenken, Heimeriks, & Hoekman, 2017). These university rankings have attracted stakeholders on basis of several indicators. These indicators of university ranking systems are of diverse nature. For example, Van Raan (2005) stated that bibliometric elements of rankings are publications and citations.

Overall review of literature revealed that university ranking systems generally comprise several indicators. These include teaching and learning environment; staff-student ratios; PhD faculty; reputational surveys; doctorate students; institutional income; international outlook; citations; research productivity and reputation; industry income; etc. (Frenken et al., 2017; Huang, 2011; Sheil, 2010; Shin et al., 2011). Globalization of higher education has also led to development of national ranking systems for making comparisons among universities at national level. In the way as the global rankings enable students to make quicker comparisons about international universities after getting information (Soo, 2013), national university rankings enable students to make quicker comparisons at national level. Moreover, academic debates about international systems of rankings (Taylor & Braddock, 2007) have also led to national ranking systems for making comparisons of universities at national level.

In striving to achieve international academic standards, HEC has taken several initiatives for uplifting higher education institutions (HEIs) of Pakistan at par with international universities by strengthening their quality. One of the measures taken by HEC to strengthen the quality of HEIs in Pakistan was initiation of ranking of universities at national level. Initiation of ranking by HEC is intended to enhance quality of teaching, research and innovation. Despite difficulties in designing robust ranking criteria and controversies associated with it, HEC has designed comprehensive ranking criteria and implemented it for ranking of universities in Pakistan since 2010. Indicators of HEC ranking criteria mainly comprise quality, teaching, and research.

Since the development of criteria by HEC for ranking of universities in Pakistan, these criteria have been subject to amendments over the period in response to feedback of academics and to make these criteria more compatible with global ranking systems. Initial HEC ranking criteria comprised indicators of quality assurance, teaching, and research. These criteria were revised from time to time and latest criteria comprised indicators of quality; teaching; research; finance and facilities; and social integration (community development). Though many researches have been conducted to examine international ranking systems, no comprehensive research has been conducted to examine the nature of HEC ranking criteria and its constituent indicators, from perspective of global ranking systems.

Main purpose of this study was to analyze the nature of HEC ranking criteria and its constituent indicators from perspective of global university ranking systems. Global ranking systems, in context of this study, include three most renowned ranking systems of Times Higher Education, ARWU (Shanghai Jiao Tong), and Quacquarelli Symonds (QS) ranking. This research further examines similarities/differences between indicators of HEC ranking criteria and indicators of global university ranking systems, along with weightage of each indicator. Analysis of ranking criteria would provide in-depth insight for HEC to design a more robust criteria for ranking of universities in Pakistan. Moreover, the study would be significant for academics, policymakers, administrator, HEC, universities, and the government of Pakistan in addition to its utility at international level.

Objectives of Research

Key objective of this study was to analyze the nature of HEC ranking criteria and its constituent indicators from perspective of global university ranking systems. Global ranking systems, in context of this study, include three most renowned ranking systems of Times Higher Education, ARWU (Shanghai Jiao Tong), and Quacquarelli Symonds (QS) ranking. To be more specific, this study aimed at following objectives.

• To analyze the nature, focus, and weightage of the major indicators of global university ranking systems, and their constituent indicators.

• To analyze the nature, focus, and weightage of the major indicators of the HEC ranking criteria for universities of Pakistan, and its constituent indicators.

• To analyze the nature and focus of HEC ranking criteria and its constituent indicators from perspective of global university ranking systems.




Research Design

This study used content analysis research design, a form of mixed methods research. Firstly, a set of policy and practice documents, related to ranking of universities, from three well-known global ranking systems was retrieved from their respective websites from 2010 to 2017. These three ranking systems include Times Higher Education, ARWU (Shanghai Jiao Tong), and the Quacquarelli Symonds (QS) ranking system. Similarly, policy and practice documents related to HEC ranking criteria of universities in Pakistan from 2010 to 2017, were retrieved from the website of HEC. All these documents were available for public access.

Analysis of data was done in three stages. First, nature, focus and weightage of major indicators of global ranking systems, and their constituent indicators, was examined longitudinally from years 2010 to 2017. Second, nature, focus, and weightage of major indicators of HEC ranking criteria, and its constituent indicators, was examined longitudinally from years 2010 to 2017 in the same way as analysis of worldwide university ranking systems. Third, nature and focus of HEC ranking criteria and its constituent indicators from the perspective of global university ranking systems was explored horizontally. Moreover, similarities and differences in nature, focus and weightage of major indicators of HEC ranking criteria (and its constituent indicators) were explored with those of worldwide university ranking systems. For data analysis, thematic and content analysis techniques were used. Results are presented in the following section.

Data Analysis and Findings

This section explains data analysis and findings in three sections. First section examines

 nature, focus, and weightage of the major indicators of global ranking systems, and their constituent indicators longitudinally from years 2010 to 2017. Second section examines nature, focus, and weightage of the major indicators of HEC ranking, and its constituent indicators, longitudinally from years 2010 to 2017. Final section explores the nature and focus of HEC ranking criteria and its indicators from perspective of global university ranking systems, horizontally.




Analysis of the Indicators of Global University Ranking Systems

This section presents data analysis and findings to examine nature, focus, and weightage of the major indicators of global university ranking systems, and their constituent indicators. Analysis was delimited to policy and practice documents, related to ranking of universities, from three ranking systems i.e., Times Higher Education, ARWU and QS ranking system. Analysis was conducted longitudinally, and covered indicators followed in rankings of years from 2010 to 2017. In this way, changes in indicators of worldwide university ranking systems were explored in terms of changes in time, in addition to examining their nature, focus, and weightage.

Content analysis revealed that Times Higher Education Ranking focuses on five key indicators for ranking of universities. These five indicators include teaching and learning environment; research, reputation, and income; citations; international outlook; and industry income, with weightages of 30%; 30%; 30%; 07.50%; and 02.50% respectively. First main indicator of teaching and learning environment (30%) further includes sub indicators of reputation surveys, staff-student ratio, doctorate-bachelors ratio, PhD awarded to faculty and institutional income with weightages of 15%, 4.50%, 2.25%, 6.00% and 02.25% respectively. It shows that teaching and learning indicator of THE mainly focuses on teaching reputational surveys, followed by ratio of doctoral students.

Analysis found that second main indicator of research (30%) further includes sub indicators of reputation surveys, research income and research productivity with weightages of 18%, 6%, and 6% respectively. It shows that the research indicator of Times Higher Education Ranking System mainly focuses on research reputational surveys. Focus of third main indicator is solely on citation with weightage of 30%. Fourth key indicator of international outlook (30%) further includes indicators of international/domestic student ratio, international-domestic staff ratio, and international collaborations with the weightages of 2.50%, 2.50%, and 2.50% respectively. Focus of fifth main indicator is solely on industry income with weightage of 2.50%. It has been further observed from longitudinal analysis that THE Ranking has not made any key changes in main indicators of its ranking systems in years from 2010 to 2017.

Content analysis also showed that ARWU (Shanghai Jiao Tong) Ranking focuses on four main indicators for ranking of universities. These five indicators include quality teaching; quality of research and faculty; research output; and per capita an institution’s academic performance, with weightages of 30%; 40%; 20%; and 10.00% respectively. First main indicator of quality of education (30%) further includes sub indicators of medals/prizes won by alumni and medals/ prizes won by staff, with weightages of 10% and 20% respectively. Analysis further shows that second main indicator of quality faculty/research (40%) includes sub indicators of highly cited researchers in various subjects and papers published in nature/science with weightages of 20% for each indicator. It shows that this indicator of ARWU Ranking mainly focuses on quality of research. Focus of third main indicator is solely on research output with a weightage of 20%, measured through indexing of papers in social (science) citations index. Focus of fourth main indicator is solely on institution's per capita academic performance with weightage of 10.00%. It has been further observed from longitudinal data analysis that ARWU Ranking has not made any key changes in main indicators of its ranking in years 2010-2017.

Analysis shown that QS Ranking emphasizes on six key indicators for ranking of universities. These six indicators include academic repute; employer repute; faculty-student ratio; citations; international faculty; and ratio of international students, with weightages of 40%; 10%; 20%; 20%; 5%; and 5% respectively. Focus of first/second indicator is on academic and employers' repute, which are generally measured through various surveys (collectively 50%). Next focus of indicators of QS Ranking is on staff-student ratios and international to national staff/student ratios. Third focus is on quality research, measured through citations. It has been also observed from longitudinal analysis that QS Ranking has not made any key changes in main indicators of its ranking systems in years 2010-2017. Following section examines key indicators of HEC ranking criteria, and its constituent indicators, longitudinally for years 2010-2017.

Analysis of the Indicators of HEC Ranking Criteria

This section presents data analysis and findings to examine nature, focus, and weightage of key indicators of HEC ranking criteria and its constituent indicators. Longitudinal analysis covered indicators followed in ranking of years 2010-2017. Changes in indicators of HEC ranking were also explored in terms of changes in time. Table 1 presents longitudinal analysis of data and findings.

Table 1. Indicators of HEC Ranking (For years 2010-2017)

Main Indicators	Sub Indicators (HEC Ranking Criteria) *	Weightage
Implementation of QA criteria (15.00%)	·          Faculty appointments ·          M.Phil/MS/PhD admissions ·          Plagiarism policy ·          QECs rating ·          Students’ awards ·          Accreditation ·          International rankings of HEIS		04.00% 02.00% 02.00% 03.00% 01.00% 02.00% 01.00%
Quality teaching (30.00%)	·          Permanent/total faculty ·          PhDs/total faculty ·          Teacher/student ratio ·          Selectivity ·          Faculty trainings ·          Faculty’s terminal degrees ·          National/international awards by faculty ·          PhD faculty		03.00% 07.00% 07.00% 04.00% 02.00% 02.00% 03.00% 02.00%
Quality research (41.00%)	·          National/international patents/commercialization ·          Industry linkages ·          PhD/total students ·          Research/travel grants ·          ISI/Impact factor publications ·          Citations/H Index ·          W/X-category journals ·          Internet/digital library utilization ·          National/international conferences ·          PhD output		03.00% 02.00% 03.00% 06.00% 09.00% 06.00% 03.00% 02.00% 03.00% 04.00%
Finance/Facilities (10.00%)	·          Non-salary expenditures ·          Money generated ·          Research/library budget ·          Computers/books/scholarships		02.00% 02.00% 02.00% 04.00%
Social integration and community development (04.00%)	·          Outreach programs ·          Collaboration/exchange ·          Foreign students ·          Foreign faculty		01.00% 01.00% 01.00% 01.00%
Total Weightage			100.00%

* Longitudinal analysis of indicators of HEC ranking criteria for years 2010-2017.

Table 2 shows that HEC Ranking criteria, from 2015 to 2017, focused on five main indicators. These five indicators include implementation of quality assurance; quality teaching; indicators of research; finances and facilities; and social integration and community development, with weightages of 15%; 30%; 41%; 10%; and 4% respectively. First indicator of implementation of QA criteria (18%) refers to implementation of plagiarism policy; use of standard criteria for faculty appointment and for running M.Phil/PhD programs; ratings of QECs; accreditation of programs; etc. Second indicator of teaching (42%) further includes sub indicators of student-teacher ratio, PhD faculty, PhD output, selectivity, computer labs, books in library, full time PhD faculty, etc.

Third indicator of research (40%) further includes sub indicators of HEC approved supervisors, enrollment of indigenous scholars, research grants, travel grants, impact factor publications, papers published by faculty, citations, H Index, number of HEC recognized journals, internet bandwidth utilization, number of conferences organized by university, etc. Fourth indicator of HEC ranking focuses on effective and efficient use of finances and on provision of facilities. Focus of fifth indicator is on social integration and community development, by sub indicators of community outreach programs, international collaboration and on number of foreign faculty and students. It has been further observed from longitudinal analysis of data that HEC Ranking Criteria has been subject to change in indicators. It was consistent from 2012-2016 and was revised in years from 2015 to 2017.

 

Analysis of Indicators of HEC Ranking Criteria from Perspective of Global University Ranking Systems

This section presents data analysis and findings to examine nature, focus, and weightage of key indicators of HEC ranking criteria, and its constituent indicators from the perspective of global ranking systems. Analysis was delimited to ranking documents of HEC, THE, ARWU, and QS ranking systems. Table 2 presents horizontal comparison of indicators of HEC ranking with those of global university ranking systems.

Table 2. Comparison of Major Indicators of HEC Ranking with Global University Rankings

Ranking system	Major Indicators for Ranking	Weighting
Times Higher Education	·          Teaching/learning environment ·          Research repute ·          Citations ·          International outlook ·          Industry income	30.00% 30.00% 30.00% 07.50% 02.50%
ARWU	·          Quality education ·          Quality faculty ·          Research output ·          Per capita performance	30.00% 40.00% 20.00% 10.00%
QS Ranking	·          Academic repute ·          Employer repute ·          Faculty/student ratio ·          Citations ·          International faculty ·          Faculty/student ratio	40.00% 10.00% 20.00% 20.00% 05.00% 05.00%
HEC Pakistan	·          Implementation of QA criteria ·          Teaching quality ·          Research quality ·          Finances/facilities ·          Social integration/community development	15.00% 30.00% 41.00% 10.00% 04.00%

Table 2 shows that focus of Times Higher Education Ranking System is on quality of teaching and research, usually measured by research citations and teaching and research reputational surveys. ARWU System mainly focuses on quality of research and its output, usually measured through research citations and indexes. Other focus of ARWU System is on quality of faculty and alumni. Focus of QS Ranking is mainly on academic and employers' reputation, measured through various surveys. Second focus of indicators of QS Ranking is on staff-student ratios and international to national staff/student ratios. Third focus is on quality of research, measured through citations.

Table 2 also shows that focus of HEC Ranking Criteria is on quality of teaching and research, followed by sub themes of student-teacher ratio, PhD faculty, and publications and journals. Moreover, focus is on effective and efficient use of finances and on provision of facilities. HEC ranking also focuses on social integration and community development, with sub indicators of community outreach programs, international collaboration and on number of foreign faculty and students. In-depth analysis of Table 2 shows that two indicators of HEC ranking criteria (i.e., quality teaching and quality research) are highly similar with global university ranking systems. Table 2 further shows that two indicators of HEC ranking criteria (i.e., implementation of QA and social integration and community development) have few similarities with all three global university ranking systems.

Figure 1

Comparison of HEC Ranking Criteria with Global University Ranking Systems



Analysis of data further revealed that one indicator of HEC ranking criteria (i.e., finances and facilities) has distinct parameters than those of three global ranking systems. In-depth analysis further revealed that nature of the distinct indicators of HEC ranking criteria is developmental. It can be, therefore, concluded that HEC has included these distinct indicators in its ranking criteria in context of developing nature of universities and country. Figure 1 presents graphical form of comparison between HEC ranking criteria and global 

It is clear from Figure 1 that focus of HEC ranking on quality of teaching, linkage with industry, and on academic reputation is aligned with global university ranking systems. Figure 1 further shows that HEC ranking criteria places more emphasis on implementation of QA procedures; effective/efficient use of finances; and on community development in comparison with global ranking systems. HEC ranking is, however, lacking in indicators of citations and international outlook in comparison with global university ranking systems. Focus of HEC ranking criteria on implementation of QA procedures; effective/efficient use of finances; and on community development, in comparison with global ranking systems, is reasonable as most universities of Pakistan are in developmental phase. This sort of mix approach, comprising accountability and improvement is likely to be useful for universities of Pakistan.

Discussion and Conclusions

This research drawn valuable conclusions. First, it was concluded that focus of Times Higher Education Ranking System is on quality of teaching and research, generally measured through research citations and through teaching and research reputational surveys. Second, it was concluded that focus of ARWU System is mainly on research quality and its output, measured through research citations and indexes. ARWU System also focuses on quality of faculty and alumni. Focus of Quacquarelli Symonds is mainly on academic and employers' reputation, measured through various surveys. Second focus of indicators of QS System is on staff-student ratios and international to national staff/student ratios. Third focus is on quality of research, as measured through the citations. It was also concluded that focus of HEC Ranking is on quality of teaching and research, trailed by focus on student-teacher ratio, PhD faculty, publications and journals. Moreover, focus of HEC ranking is on effective and efficient use of finances and on provision of facilities. HEC further focuses on social integration, community development, outreach programs, international collaboration and foreign faculty and students.

Based on findings of this study, it is concluded that various global university ranking systems focus on different academic areas. Some of these areas are common in every ranking system. As Times Higher Education ranking emphasis on teaching, research and citations while ARWU focuses more on quality of education and faculty. Conversely, QS ranking places emphasis on academic reputation and HEC of Pakistan is prioritizing quality of research and teaching. It is thus concluded that all global ranking systems, including HEC, are highly focusing on quality of teaching and research. These findings are consistent with the findings of many researchers (Frenken et al., 2017; Huang, 2011; Sheil, 2010; Shin et al., 2011). It is, therefore, suggested that such ranking procedures be adopted that are likely to be helpful for not only in teaching and research but also put strong focus on assuring and enhancing quality of education in HEIs.

References

• Altbach, P. G. (2012). The globalization of college and university rankings. Change: The Magazine of Higher Learning, 44(1), 26-31.
• Bischoff, F., Gassmann, F., & Emrich, E. (2017). Demand for and Satisfaction with Places at University--An Empirical Comparative Study. International Journal of Higher Education, 6(2), 59-74.
• Bowden, R. (2000). Fantasy higher education: University and college league tables. Quality in higher education, 6(1), 41-60.
• Clarke, M. (2002). Some guidelines for academic quality rankings. Higher Education in Europe, 27(4), 443-459.
• Frenken, K., Heimeriks, G. J., & Hoekman, J. (2017). What drives university research performance? An analysis using the CWTS Leiden Ranking data. Journal of Informetrics, 11(3), 859-872.
• Grewal, R., Dearden, J. A., & Llilien, G. L. (2008). The university rankings game: Modeling the competition among universities for ranking. The American Statistician, 62(3), 232-237.
• Harvey, L. (2008). Rankings of higher education institutions: A critical review: Routledge.
• Hazelkorn, E. (2011). Globalization and the reputation race in rankings and the reshaping of Higher Education: the battle for world wild excellence: Palgrave: Mac Millan.
• Hou, A. Y. C., Morse, R., & Chiang, C. (2012). An analysis of mobility in global rankings: making institutional strategic plans and positioning for building world-class universities. Higher Education Research & Development, 31(6), 841-857.