What Makes Good Honey?

Download the King’s Digital Lab Checklist for Digital Outputs Assessment in the REF here.

In previous REF assessment exercises, the Faculty of Arts & Humanities at King’s College London has submitted a substantial number of digital outputs which have been well received by the relevant REF panels, scoring highly with respect to quality. It was the Faculty’s request to KDL - as its team of experts in Research Software Engineering (RSE) best practices - to produce the checklist we publish with this blog. It aims to help guide Faculty researchers  conduct peer or self assessment with confidence and in a holistic manner in keeping with both academic and RSE criteria.

For researchers not based in a UK research or higher education institution, REF probably resonates as no more than an overused acronym. Or does it? A glance at the multilingual references in Wiktionary reveals fun ramifications: Does it stands for referee? An abbreviation for reference or refectory? A declination for an Icelandic fox? What about its corresponding Middle English form?

Well, if we stick to a UK research or higher education context, none of the above. REF stands for the well known Reference Excellence Framework (previously called RAE, Research Assessment Exercise) which drives the regular assessment of research performance in universities and other research institutions in regular cycles of 6 years. Research is assessed and measured on the basis of institutional submissions. These include qualitative and quantitative information collecting research outputs, impact case studies, environmental statements and other accompanying material.  Despite due differences, many European countries have implemented some form of performance-based research funding (see Zacharewicz et al. 2019). The process is also similar to those used in countries such as Australia (the ERA) and New Zealand (the PBRF).

Substantial effort is devoted to prepare submissions at the institutional level for the REF exercise. This includes internal assessment of research outputs and products of different kind (the REF is quite permissive with respect to eligible submissions, including printed reference publications as well more ephemeral research products such as exhibitions and design artefacts). Digital outputs of different kinds - whether digital research outputs such as digital publications, digital research products underpinning impact case studies, websites mentioned in environmental statements or in other narrative parts of an institutional submission - are also eligible for submission.

While researchers in all disciplines have developed refined mechanisms to peer or self assess ‘traditional’ research outputs, when it comes to digital outputs, it is often unclear what assessment criteria to apply. The Digital Humanities community have reflected on this issue and published papers and reports even if not directly in the context of the REF. The KDL checklist attempts to systematise, update, adapt and enhance these efforts. It aligns with our ongoing archiving and sustainability efforts, and will be included in a proposed ‘KDL Handbook’, alongside our SDLC templates (via DESIR project) and other resources: a complete guide to our Research Software Engineering (RSE) model. While designed specifically to help colleagues prepare for the UK's REF exercise, then, we hope it could end up being useful to colleagues elsewhere too.