Saturday 23 May 2015

A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar

Source: http://link.springer.com/article/10.1007/s11192-015-1608-4

A comparison of 17 author-level
bibliometric indicators for researchers in Astronomy, Environmental
Science, Philosophy and Public Health in Web of Science and Google
Scholar

Abstract

Author-level bibliometric indicators are becoming a standard tool in research assessment. It is important to investigate what these indicators actually measure to assess their appropriateness in scholar ranking and benchmarking average individual levels of performance. 17 author-level indicators were calculated for 512 researchers in Astronomy, Environmental Science, Philosophy and Public Health. Indicator scores and scholar rankings calculated in Web of Science (WoS) and Google Scholar (GS) were analyzed. The indexing policies of WoS and GS were found to have a direct effect on the amount of available bibliometric data, thus indicator scores and rankings in WoS and GS were different, correlations between 0.24 and 0.99. High correlation could be caused by scholars in bottom rank positions with a low number of publications and citations in both databases. The hg indicator produced scholar rankings with the highest level of agreement between WoS and GS and rankings with the least amount of variance. Expected average performance benchmarks were influenced by how the mean indicator value was calculated. Empirical validation of the aggregate mean h-index values compared to previous studies resulted in a very poor fit of predicted average scores. Rankings based on author-level indicators are influenced by (1) the coverage of papers and citations in the database, (2) how the indicators are calculated and, (3) the assessed discipline and seniority. Indicator rankings display the visibility of the scholar in the database not their impact in the academic community compared to their peers. Extreme caution is advised when choosing indicators and benchmarks in scholar rankings.

References:
  1. Alonso, S, Cabreriazo, F, Herrera-Viedma, F (2009) H-index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics 3: pp. 273-289 CrossRef
  2. Alonso, S, Cabreriazo, F, Herrera-Viedma, F (2010) hg-index: A new index to characterize the scientific output of researchers based on the h- and g-indices. Scientometrics 82: pp. 391-400 CrossRef
  3. Altman, D (2005) Standard deviations and standard errors. British Medical Journal 331: pp. 903 CrossRef
  4. Archambault, E., & Gagné, E. V. (2004). The Use of Bibliometrics in Social Sciences and Humanities (Social Sciences and Humanities Research Council of Canada, SSHRCC), Montreal: Science Metrix. http://www.science-metrix.com/pdf/SM_2004_008_SSHRC_Bibliometrics_Social_Science.pdf. Accessed 30 March 2015.
  5. Bar-Ilan, J (2008) Which h-index? A comparison of WoS, Scopus and Google Scholar. Scientometrics 74: pp. 257-271 CrossRef
  6. Berk, RA, Freedman, DA Statistical assumptions as empirical commitments. In: Blomberg, TG, Cohen, S eds. (2003) Punishment and social control. Walter de Gruyter, New York, pp. 235-254
  7. Bornmann, L, Marx, W (2011) The h-index as a research performance indicator. European Science Editing 37: pp. 77-80
  8. Bornmann, L, Marx, W (2013) Evaluating individual researchers’ performance. Euorpean Science Editing 39: pp. 30-40
  9. Cabrerizo, FJ, Alonso, S, Herrera-Viedmac, E, Herrera, F (2012) Q2-index: Quantitative and qualitative evaluation based on the number and impact of papers in the hirsch core. Journal of Informetrics 4: pp. 23-28 CrossRef
  10. Connor, J. (2011). Google scholar citations open to all. Resource document. http://googlescholar.blogspot.dk/2011/11/google-scholar-citations-open-to-all.html. Accessed 30 March 2015.
  11. Costas, R, Bordons, M (2007) The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics 1: pp. 193-203 CrossRef
  12. Courtault, JM, Hayek, N (2008) On the robustness of the h-index: A mathematical approach. Economics Bulletin 3: pp. 1-9
  13. Battisti, F, Salini, S (2012) Robust analysis of bibliometric data. Statistical Methods and Applications 22: pp. 269-283 CrossRef
  14. Egghe, L (2006) Theory and practise of the g-index. Scientometrics 69: pp. 131-152 CrossRef
  15. ESO. (2011). Some statistics on recent job hires in Astronomy: Who is getting the jobs and how do you compare? Science News. Resource document. https://www.eso.org/wiki/pub/Science/ScienceNewsLetter/ScienceNews-May-Jun2011.pdf. Accessed 30 March 2015.
  16. Farhadi, F, Salehi, H, Yunus, MM, Chadegani, AA, Farhabi, M, Fooladi, M (2013) Does it matter which citation tool is used to compare the h-index of a group of highly cited researchers?. Australian Journal of Basic and Applied Sciences 7: pp. 198-202
  17. Fiorenzo, F., Domenico, M., & Luca, M. (2013). The effect of dirty data on h-index calculation. Scientometrics, 95(3), 1179–1188.
  18. Franceschet, M (2010) A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar. Scientometrics 83: pp. 243-258 CrossRef
  19. Franceschini, F, Maisano, D (2011) Criticism on the hg.index. Scientometrics 86: pp. 339-346 CrossRef
  20. Freedman, DA, Pisani, R, Purves, R (2007) Statistics. W.W. Norton & Company, New York, NY
  21. Gorard, S (2003) Quantitative methods in social science: the role of numbers made easy. Continuum, London
  22. Gorard, S (2006) Towards judgement-based statistical analysis. British Journal of Sociology of Education 27: pp. 67-80 CrossRef
  23. Gratton, R. (2014). Evaluation of Italian atronomical production: 2010–2012. Resource document. arXiv: arXiv:1402.4080v1 [astro-ph.IM]. Accessed 30 March 2015.
  24. Hagen, NT (2008) Harmonic allocation of authorship credit: Source-level correction of bibliometric bias assures accurate publication and citation analysis. PLoS ONE.
  25. Harzing, A. (2008). Google Scholar—A new data source for citation analysis. Resource document, http://www.harzing.com/pop_gs.htm. Accessed 30 March 2015.
  26. Harzing, A. (2012). Publish or Perish user’s manual. Resource document, http://www.harzing.com/pop.htm#metrics. Accessed 30 March 2015.
  27. Harzing, A (2014) A longitudinal study of Google Scholar coverage between 2012 and 2013. Scientometrics 98: pp. 565-575 CrossRef
  28. Haustein, S, Larivière, V The use of bibliometrics for assessing research: Possibilities, limitations and adverse effects. In: Welpe, IM, Wollersheim, J, Ringelhan, S, Osterloh, M eds. (2015) Incentives and performance: Governance of research organizations. Springer International Publishing, Switzerland, pp. 121-139 CrossRef
  29. Hirsch, J (2005) An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America (PNAS) 102: pp. 16569-16572 CrossRef
  30. Jacsó, P (2005) As we may search? Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science 89: pp. 1537-1547
  31. Jascó, P (2008) The plausibility of computing the h-index of scholarly productivity and impact using reference-enhanced databases. Online Information Review 32: pp. 266-283 CrossRef
  32. Kamphuis, P., & van der Kruit, P. C. (2010). Citations and impact of Dutch astronomy. Resource document arXiv:1011.5311v1 [astro-ph.IM]. Accessed 30 March 2015.
  33. Kendall, MG (1955) Rank correlation methods. Hafner Publishing Co, New York
  34. Kosmoloulos, C., & Pumain, D. (2014). Citation, citation, citation: Bibliometrics, the web and the Social Sciences and Humanities. Resource document, Cybergeo: European Journal of Geography. http://www.cybergeo.eu/index15463.html. Accessed 30 March 2015.
  35. Kosmulski, M (2006) A new type Hirsch-index saves time and works equally well as the original h-index. ISSI Newsletter 2: pp. 4-6
  36. Lancho-Barrentes, B, Guerrero, VP (2010) What lies behind the averages and significance of citation indicators in different disciplines?. Journal of Information Science 36: pp. 371-382 CrossRef
  37. Li, J, Burnham, JF, Lemley, T, Britton, RM (2010) Citation analysis: Comparison of Web of Science®, Scopus™, SciFinder®, and Google Scholar. Journal of Electronic Resources in Medical Libraries 7: pp. 196-217 CrossRef
  38. Meera, B. M., & Manjunath, M., (2012). h-index of astrophysicists at raman research institute: performance of different calculators, Working Group on Libraries, IAU Division XII Commission 5, In L. Utley (Ed.) Special session 15: Data-intensive astronomy, Beijing, China, 23-24 August 2012. http://iau-commission5.wikispaces.com/LibWG+Beijing+2012. Accessed 30 March 2015.
  39. Meho, LI, Yang, K (2007) Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology 58: pp. 2105-2125 CrossRef
  40. Miller, C. W., (2006). Superiority of the h-index over the impact factor for physics. Resource document, arXiv: physics/0608183 [physics.soc-ph]. Accessed 30 March 2015.
  41. Neuhaus, C, Neuhaus, E, Asher, A, Wrede, C (2006) The depth and breadth of Google Scholar: An empirical study. Portal: Libraries and the Academy 6: pp. 127-141 CrossRef
  42. Panaretos, J, Malesios, C (2009) Assessing scientific research performance and impact with single indicators. Scientometrics.
  43. Patel, VM, Ashrafian, H, Almoudaris, A, Makanjuola, J, Bucciarelli-Ducci, C, Darzi, A (2013) Measuring academic research performance for healthcare researchers with the h index: Which search tool should be used?. Medical Principles and Practice 22: pp. 178-183 CrossRef
  44. Redner, S. (2010). On the meaning of the h-index. Journal of Statistical Mechanics: Theory and Experiment. doi:10.1088/1742-5468/2010/03/L03005.
  45. Roediger, H. L. (2006). The h index in Science: A new measure of scholarly contribution. Resource document, APS Observer: The Academic Observer. http://www.psychologicalscience.org/index.php/publications/observer/2006/april-06/the-h-index-in-science-a-new-measure-of-scholarly-contribution.html. Accessed 30 March 2015.
  46. Schneider, JW (2013) Caveats for using statistical significance tests in research assessments. Journal of Informetrics 7: pp. 50-62 CrossRef
  47. Schneider, JW (2015) Null hypothesis significance tests. A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations. Scientometrics 102: pp. 411-432 CrossRef
  48. Schreiber, M (2013) A case study of the arbitrariness of the h-index and the highly-cited-publications indicator. Journal of Informetrics 7: pp. 379-387 CrossRef
  49. Schreiber, M (2013) How to derive an advantage from the arbitrariness of the g-index. Journal of Informetrics 7: pp. 555-561 CrossRef
  50. Schreiber, M, Malesios, CC, Psarakis, S (2012) Exploratory factor analysis for the hirsch-index, 17 h-type variants, and some traditional bibliometric indicators. Journal of Informetrics 6: pp. 347-358 CrossRef
  51. Sidiropoulos, A, Katsaros, D, Manolopoulos, Y (2007) Generalized hirsch h-index for disclosing latent facts in citation networks. Scientometrics 72: pp. 253-280 CrossRef
  52. Smith, AG (2008) Benchmarking Google Scholar with the New Zealand PBRF research assessment exercise. Scientometrics 74: pp. 309-316 CrossRef
  53. Testa, J. (2012). The Thomson Reuters Journal Selection Process. Resource document, http://wokinfo.com/essays/journal-selection-process/. Accessed 30 March 2015.
  54. Tol, RSJ (2009) Of the H-index and its alternatives: An application to the 100 most prolific economists. Scientometrics 80: pp. 317-324 CrossRef
  55. Leeuwen, T (2008) Testing the validity of the hirsch-index for research assessment purposes. Research Evaluation 17: pp. 157-160 CrossRef
  56. Raan, AFJ (2006) Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics 67: pp. 491-502 CrossRef
  57. Vanclay, JK (2007) On the robustness of the h-index. Journal of the American Society for Information Science and Technology 58: pp. 1547-1550 CrossRef
  58. Waltman, L, Schreiber, M (2013) On the calculation of percentile-based bibliometric indicators. Journal of the American Society for Information Science and Technology 64: pp. 372-379 CrossRef
  59. Waltman, L, Eck, NJ (2012) The inconsistency of the h-index. Journal of the American Society for Information Science and Technology 63: pp. 406-415 CrossRef
  60. Wildgaard, L, Schneider, JW, Larsen, B (2014) A review of the characteristics of 108 author-level bibliometric indicators. Scientometrics 101: pp. 125-158 CrossRef
  61. Wilson, V (2007) A content analysis of Google Scholar: Coverage varies by discipline and by database. Evidence Based Library and Information Practice 2: pp. 134-136
  62. Yang, K., & Meho, L. I. (2006). Citation analysis: A comparison of Google Scholar, Scopus, and Web of Science., 2006. In 69th annual meeting of the American Society for Information Science and Technology (ASIST), Austin (US), 3–8 November 2006. http://eprints.rclis.org/8605/1/Yang_citation.pdf. Accessed 9 April 2015.
  63. Zhang, C (2009) The e-index, complementing the h-index for excess citations. PLoS ONE 4: pp. e5429 CrossRef



    A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar - Online First - Springer
    Posted by at

    No comments:

    Post a Comment

    Subscribe to: Post Comments (Atom)