Towards research assessment reform for the social sciences and humanities:
progress and insights from the GraspOS Project
DOI:
https://doi.org/10.15291/libellarium.4548Keywords:
open science, research culture, research evaluation, responsible research assesment, social sciences and humanitiesAbstract
Purpose. This study focuses on developing research assessment criteria for the Social Sciences and Humanities (SSH) through the GraspOS project. The aim is to reform traditional evaluation methods that rely heavily on bibliometric indicators and to produce recommendations tailored to the SSH community, addressing the limitations of current practices that predominantly emphasise quantitative metrics.
Approach/methodology/design. This study presents outcomes from three workshops with participants from various SSH backgrounds. The workshops facilitated discussions on values-driven assessment, output diversity, and the integration of qualitative indicators. The data collected were analysed using the deductive method, with a priori codes established.
Findings. The analysis revealed consensus on the need for a more inclusive evaluation framework that recognises diverse outputs beyond traditional journal articles. Participants emphasised the importance of qualitative measures, such as peer review, and proposed using narrative CVs to better reflect individual contributions.
Research limitations. The study was conducted within a specific context of the Responsible Research Assessment (RRA) movement and the GraspOS project, which may not fully represent the broader SSH landscape. Generalisation to all SSH disciplines is not possible.
Practical implications. The study provides recommendations for formulating generic research assessment criteria as suggested by SSH scholars.
Originality value. This work provides valuable insights into developing more equitable and effective assessment practices in academia for the SSH community.
Downloads
References
Abramo, Giovanni. 2024. “The forced battle between peer-review and scientometric research assessment: Why the CoARA initiative is unsound.” Research evaluation, May. https://doi.org/10.1093/reseval/rvae021.
Agate, Nicky, Christopher P. Long, Bonnie Russell, Rebecca Kennison, Penelope Weber, Simone Sacchi, Jason Rhody, and Bonnie Thornton Dill. 2022. “Walking the talk: Toward a values- aligned academy.” Knowledge commons. https://hcommons.org/deposits/objects/hc:44632/datastreams/CONTENT/content.
Aksnes, D. W., Langfeldt, L., & Wouters, P. 2019. Citations, citation indicators, and research quality: an overview of basic concepts and theories. Sage Open, 9(1). https://doi.org/10.1177/2158244019829575
Archambault, Éric, and Étienne Vignola Gagné. 2004. “The use of bibliometrics in the social sciences and humanities: final report.” Science-Metrix. https://www.science-metrix.com/pdf/SM_2004_008_SSHRC_Bibliometrics_Social_Science.pdf.
Archambault, Éric, Étienne Vignola-Gagné, Grégoire Côté, Vincent Larivière, and Yves Gingrasb. 2006. “Benchmarking scientific output in the social sciences and humanities: the limits of existing databases.” Scientometrics 68 (3): 329–42. https://doi.org/10.1007/s11192-006-0115-z.
Australian Research Council. 2019. “Non-traditional research outputs (NTROs)”. State of Australian University Research. https://dataportal.arc.gov.au/era/nationalreport/2018/pages/section1/non-traditional-research-outputs-ntros/
Balboa, Luciana, Elizabeth Gadd, Janne Pölönen, Karen Stroobants, and Erzsébet Tóth-Czifra. 2024. “The role of scientometrics in the pursuit of responsible research assessment.” LSE Impact Blog. September 4, 2024. https://blogs.lse.ac.uk/impactofsocialsciences/2024/09/04/the-role-of-scientometrics-in-the-pursuit-of-responsible-research-assessment/.
Bingham, Andrea J. 2023. “From data management to actionable findings: a five-phase process of qualitative data analysis.” International journal of qualitative methods 22 (22).
Checco, Alessandro, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield, and Giuseppe Bianchi. 2021. “AI-assisted peer review.” Humanities and social sciences communications 8 (1): 1–11. https://doi.org/10.1057/s41599-020-00703-8.
CoARA. 2022. “Agreement on reforming research assessment.” CoARA. July 20, 2022. https://coara.eu/app/uploads/2022/09/2022_07_19_rra_agreement_final.pdf.
Curry, Stephen, Sarah de Rijcke, Anna Hatch, Dorsamy (Gansen) Pillay, Inge van der Weijden, and James Wilsdon. 2020. “The changing role of funders in responsible research assessment: progress, obstacles and the way ahead.” Rori.figshare.com. November 18, 2020. https://doi.org/10.6084/m9.figshare.13227914.v2.
DORA. 2012. “San Francisco Declaration on Research Assessment.” https://sfdora.org/read/.
Engels, Tim C. E., Truyken L. B. Ossenblok, and Eric H. J. Spruyt. 2012. “Changing publication patterns in the social sciences and humanities, 2000–2009.” Scientometrics 93 (2): 373–90. https://doi.org/10.1007/s11192-012-0680-2.
European Commission, European Research Executive Agency, A Oancea, and S Wilson. 2024. “Report on research assessment - Publications Office of the EU.” Publications Office of the EU. 2024. https://op.europa.eu/en/publication-detail/-/publication/219aa5ea-fae2-11ee-a251-01aa75ed71a1/language-en.
Galleron, Ioana, Michael Ochsner, Jack Spaapen, and Geoffrey Williams. 2017. “Valorizing SSH research: towards a new approach to evaluate SSH research’ Value for Society.” Fteval Journal for research and technology policy evaluation 44 (September). https://doi.org/10.22163/fteval.2017.274.
Hicks, Diana. 2005. “The four literatures of social sciences.” Journal of management and social sciences 1 (1). https://ibtjbs.ilmauniversity.edu.pk/journal/jbs/1.1/1.%20The%20Four%20Literatures%20of%20Social%20Sciences.pdf.
Himanen, Laura, and Iiris Liinamaa. 2024. “GraspOS Deliverable D5.2: Pilot Findings and Progress Report.” Zenodo, August. https://doi.org/10.5281/zenodo.13629147.
Himanen, Laura, Erica Conte, Marianne Gauffriau, Tanja Strøm, Baron Wolf, and Elizabeth Gadd. 2024. “The SCOPE framework – implementing ideals of responsible research assessment.” F1000Research 12 (May): 1241–41. https://doi.org/10.12688/f1000research.140810.2.
Himanen, Laura. 2023. “GraspOS Deliverable 5.1 ‘Report on pilot setup, current practices and initial requirements.’” Zenodo, August. https://doi.org/10.5281/zenodo.11091718.
Information Commissioners Office. 2021. “Introduction to anonymisation draft anonymisation, pseudonymisation and privacy enhancing technologies guidance.” https://ico.org.uk/media/about-the-ico/consultations/2619862/anonymisation-intro-and-first-chapter.pdf.
Ioannidis, John P. A., and Zacharias Maniadis. 2023. “In defense of quantitative metrics in researcher assessments.” PLOS Biology 21 (12): e3002408–8. https://doi.org/10.1371/journal.pbio.3002408.
Joachim Schöpfel, and Otmane Azeroual. 2024. “Responsible research assessment and research information management systems.” Encyclopedia 4 (2): 915–22. https://doi.org/10.3390/encyclopedia4020059.
Krüegel, Sybil. 2019. The informed consent as legal and ethical basis of research data production. FORS Guide No. 05, Version 1.0. Lausanne: Swiss Centre of Expertise in the Social Sciences FORS. doi:10.24449/FG-2019-00005
Leeuwen, van. 2013. “Bibliometric research evaluations, Web of Science and the social sciences and humanities: a problematic relationship?” Bibliometrie Scientometrics 2 (September): 1–18. https://doi.org/10.5283/bpf.173.
Mystakopoulos, Fotis. 2024. “The role of open science in fostering a more inclusive research assessment process in social sciences and humanities.” Zenodo. https://doi.org/10.5281/zenodo.10562683.
National Research Council. 2012. Improving measures of science, technology, and innovation. National Academies Press EBooks. https://doi.org/10.17226/13358.
Ochsner, Michael, Sven Hug, and Ioana Galleron. 2017. “The future of research assessment in the humanities: bottom-up assessment procedures.” Palgrave Communications 3 (1). https://doi.org/10.1057/palcomms.2017.20.
Pinto da Costa, Mariana. 2021. “Conducting cross-cultural, multi-lingual and multi-country focus groups: guidance for researchers.” International Journal of Qualitative methods 20 (January): 160940692110499. https://doi.org/10.1177/16094069211049929.
Rafols, Ismael. 2024. “The UNESCO Open Science Outlook: OS progresses, but unequally.” Leidenmadtrics.nl. February 2024. https://www.leidenmadtrics.nl/articles/the-unesco-open-science-outlook-there-is-progress-in-os-but-it-is-unequal.
Reale, Emanuela, Dragana Avramov, Kubra Canhial, Claire Donovan, Ramon Flecha, Poul Holm, Charles Larkin, et al. 2017. “A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research.” Research Evaluation 27 (4): 298–308. https://doi.org/10.1093/reseval/rvx025.
Rijcke, Sarah de, Clemencia Cosentino, Robin Crewe , Carlo D’Ippoliti, Shaheen Motala-Timol , Noorsaadah Binti A Rahman , Laura Rovelli , David Vaux, and Yao Yupeng . 2023. “The future of research evaluation: a synthesis of current debates and developments,” May. https://doi.org/10.24948/2023.06.
Rodriguez, Katrina L., Jana L. Schwartz, Maria K. E. Lahman, and Monica R. Geist. 2011. “Culturally responsive focus groups: reframing the research experience to focus on participants.” International journal of qualitative methods 10 (4): 400–417. https://doi.org/10.1177/160940691101000407.
Ross-Hellauer, Tony. 2017. “What is open peer review? A systematic review.” F1000Research 6 (588): 588. https://doi.org/10.12688/f1000research.11369.2.
Schmoch, Ulrich, Torben Schubert, Dorothea Jansen, Richard Heidler, and Regina von Görtz. 2010. “How to use indicators to measure scientific performance: a balanced approach.” Research evaluation 19 (1): 2–18. https://doi.org/10.3152/095820210x492477.
Schönbrodt, Felix D, Anne Gärtner, Maximilian Frank, Mario Gollwitzer, Malika Ihle, Dorothee Mischkowski, Le Vy Phan, et al. 2022. “Responsible research assessment I: implementing DORA for hiring and promotion in psychology.” psyArXiv (OSF Preprints), November. https://doi.org/10.31234/osf.io/rgh5b.
Sharma, Mohit, Anurag Sarin, Priyanka Gupta, Shobhit Sachdeva, and AnkurV Desai. 2014. “Journal impact factor: its use, significance and limitations.” World journal of nuclear medicine 13 (2): 146. https://doi.org/10.4103/1450-1147.139151.
Sivertsen, Gunnar. 2016. “Patterns of internationalization and criteria for research assessment in the social sciences and humanities.” Scientometrics 107 (2): 357–68. https://doi.org/10.1007/s11192-016-1845-1.
Snijder, Ronald, 2019. "The deliverance of open access books: examining usage and dissemination". https://doi.org/10.26530/oapen_1004809
Sorbonne University. 2024. “Sorbonne University unsubscribes from the Web of Science.” Sorbonne Université. 2024. https://www.sorbonne-universite.fr/en/news/sorbonne-university-unsubscribes-web-science.
Stam, Alexandra, and Pablo Diaz. 2023. “Qualitative data anonymisation: theoretical and practical considerations for anonymising interview transcripts.” FORS Guide No. 20 Version 1.0 (March). https://forscenter.ch/wp-content/uploads/2023/03/qualitative-data-anonymisation_final.pdf.
Tatum, Clifford, Zeynep Anli, Ludo Waltman, Anna-Kaisa Hyrkkänen, Janne Pölönen, and Josephine Nordling. 2023. “GraspOS Deliverable 2.2: Open Science Assessment Framework (OSAF).” Zenodo, December. https://doi.org/10.5281/zenodo.11091512.
UNESCO. 2023. “Open Science Outlook 1: status and trends around the world.” UNESDOC Digital Library. https://doi.org/10.54677/giic6829.
Vergoulis, Thanasis, Serafeim Chatzopoulos, Giulia Malaguarnera, Zenia Xenou, Leonidas Pispiringas, Harry Dimitropoulos, Silvio Peroni, et al. 2024. “GraspOS Deliverable D3.2: Tools and Services.” Zenodo, August. https://doi.org/10.5281/zenodo.13619176.
Vergoulis, Thanasis, Serafeim Chatzopoulos, Kleanthis Vichos, Ilias Kanellos, Andrea Mannocci, Natalia Manola, and Paolo Manghi. 2022. “BIP! SCHOLAR: a service to facilitate fair researcher assessment.” In JCDL ’22: Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries. https://doi.org/10.1145/3529372.3533296.
Wang, Ruishu. 2023. “Research on problems of scientific research performance evaluation mechanism of humanities and social sciences.” Frontiers in Educational Research 6 (18). https://doi.org/10.25236/fer.2023.061823.
White, Howard D., Sebastian K. Boell, Hairong Yu, Mari Davis, Concepción S. Wilson, and Fletcher T.H. Cole. 2009. “Libcitations: a measure for comparative assessment of book publications in the humanities and social sciences.” Journal of the American Society for Information Science and Technology 60 (6): 1083–96. https://doi.org/10.1002/asi.21045.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Fotis Mystakopoulos, Carol Delmazo
This work is licensed under a Creative Commons Attribution 4.0 International License.
