KDL's pragmatic approach to managing 100 Digital Humanities projects, and more
Archiving and sustainability is integral to KDL’s Software Development Life Cycle (SDLC). Discussions about archiving and forward planning inform our pre-project phases and determine release and post-project phases. When and how archiving and sustainability considerations are introduced and discussed with partners depends on the scope, size and type of project, but are typically discussed and evaluated in early conversations, as part of our preliminary feasibility assessment.
Part of those discussions focus on what KDL can offer with respect to data hosting and maintenance, explaining what a Service Level Agreement (SLA) with KDL entails (what it includes and indicative costs). If the partner is a colleague within the same University (King’s College London) and if the research data component of the project is substantial -- as opposed to, for example, an experimental project where tests on the data are deemed ephemeral -- options for data management with the University Library Services King's Open Research Data System (KORDS) are also discussed. Subject repositories or other external research infrastructures connected to relevant data management might also be taken into consideration at this stage. The KDL feasibility document includes a section titled Forward Planning Definition which, amongst other things, describes the hosting and infrastructure SLA along with archiving and sustainability and research data management options.
At its inception in 2015, KDL inherited just under 100 digital research projects and websites. Supporting that number of projects (most of which had no funding attached) was a challenging proposition, but KDL accepts its responsibility to the community to manage them in a fair and transparent manner. This in itself is significant: accepting responsibility entailed shouldering responsibility for issues that were often outside the control of the team, resting with external Principal Investigators or PIs and their institutions, communities of scholars, and funding bodies. The archiving process entailed far more than merely mechanical technical work, in other words. It required sometimes lengthy discussions with research partners (usually PIs) and in some cases stakeholder communities to assess the options available and choose the most appropriate one for their particular project.
Ideally projects were placed on an SLA to ensure ongoing hosting and maintenance, but this was not always either feasible or sensible. In many cases PIs responded promptly, asking us to gracefully archive their project because it had fulfilled its purpose. In other situations it became apparent that migration to a new host was possible, or conversion to a simpler format that substantially reduces the cost of hosting and maintenance. The archiving options offered by the team have evolved over time, to suit the needs of our research community. Currently, we offer the options outlined below.
Archiving and Sustainability Options
SLA with KDL
Maintaining the project under a costed Service Level Agreement with KDL - we offer agreements of variable durations depending on project age, infrastructure requirements, and PI preference but our standard SLA duration is five years.
Migration to ITS
Migration of the project (and associated data) to a King’s College London ITS microsite if that is agreeable to both PIs and ITS.
Migration to external host
Migration of the project (and associated data) to an external host, such as the PI’s new institution or a commercial hosting provider.
Conversion of project website to a static website. This is offered as a free service for suitable projects. It can be a very cost effective solution, and maintains public access to legacy projects and datasets that are no longer updated but continue to have value for the PI and research community.
These options are evolving, in collaboration with the Department of Digital Humanities and other partners: KDL-DDH CKAN data catalogue and repository (https://data.kdl.kcl.ac.uk/); KCL Research Data Management System (https://www.kcl.ac.uk/library/researchsupport/research-data-management/Preserve/Deposit-your-data-with-Kings3.aspx); External institutional or subject specific data repository.
Minimal storage (for two years minimum): project and data are archived on our infrastructure and a placeholder page is shown at a project URL. This placeholder includes a description of the project and other metadata, as well as links to snapshots at the Internet Archive and other web archives (such as webarchive.org.uk) where the project has been archived.
We are also interested in options such as providing projects as virtual machines (VMs) for offline use, or capturing their last good state in recording tools such as webrecorder.io but (interestingly enough) those options have not yet been requested. Where PIs or Co-Is are not able to be reached, options are considered within KDL and with appropriate Department of Digital Humanities staff and a decision made on a per project basis.
Projects under SLA
Institutions that support SLAs for projects hosted and maintained by KDL are included in brackets, below. FAH stands for Faculty of Arts & Humanities at King’s College London.
- AHRC Research Centre for the History and Analysis of Recorded Music (FAH)
- Applying AI to Storytelling (FAH)
- Art of Making in Antiquity (FAH)
- Atlantic Europe in the Metal Ages (University of Wales)
- British Printed Images to 1700 (FAH)
- Cambridge Edition of the Works of Ben Jonson (Cambridge University)
- Catalogue of English Literary Manuscripts (University of Reading)
- Clergy of the Church of England Database (FAH)
- The Community of the Realm in Scotland, 1249-1424 (AHRC)
- CultureCase (Cultural Institute, King’s College London)
- DigiPal (FAH)
- Digital Ghost Hunt (FAH, University of Sussex)
- Digitising the Prosopography of the Roman Republic (FAH)
- Distant Reading Across Languages (FAH)
- Exon Domesday (FAH)
- Henry III Fine Rolls Project (dataset) (FAH)
- Henslowe-Alleyn (University of Reading)
- Heritage Gazetteer of Cyprus (FAH)
- Heritage Gazetteer of Libya (FAH)
- Historical Pageants (FAH/AHRC)
- Hofmeister XIX (Royal Holloway, University of London)
- Inquisitions Post Mortem (dataset) (FAH)
- Inscriptiones antiquae Orae Septentrionalis Ponti Euxini (FAH)
- Inscriptions of Aphrodisias: ALA (2004) + IAph (2007) (FAH)
- Inscriptions of Roman Cyrenaica (FAH)
- Inscriptions of Roman Tripolitania (British Academy)
- Inventory of Byzantine Churches (FAH)
- Jane Austen's Fiction Manuscripts (FAH; Oxford University)
- Language of Langscape (FAH)
- Languages Acts and Worldmaking (AHRC)
- Making of Charlemagne's Europe (FAH)
- Mao Era in Objects (AHRC; FAH)
- Masonic Periodicals Online (The Library and Museum of Freemasonry)
- Medieval Francophone Literary Culture Outside France (FAH)
- Ministry of Information (AHRC)
- Models of Authority (FAH)
- Mozart and Material Culture (FAH)
- Nineteen Century Serials Edition (FAH; Birkbeck)
- Online Chopin (FAH)
- People of Medieval Scotland (dataset) (AHRC)
- Profile of a Doomed Elite + PASE (FAH; University of Oxford)
- Prosopography of the Byzantine World (British Academy)
- Radical Translations: The Transfer of Revolutionary Culture between Britain, France and Italy (1789-1815) (AHRC)
- Renaissance Skin (Wellcome Trust)
- Schenker Documents Online (dataset) (University of Southampton)
- Scrambled Messages (FAH)
- Sensing Place (HEI Fund)
- Sharing Ancient Wisdoms (FAH)
- The Values of French (dataset) (FAH)
If researchers wanted to maintain dynamic functionality for their projects but were unable or uninterested in financially supporting a Service Level Agreement with KDL then we sometimes helped them to migrate their projects to other institutional or commercial infrastructure. King’s College London academics with relatively straightforward and current (in that they are still actively updated), websites were offered the opportunity to migrate to KCL ITS infrastructure where appropriate. We are always sad to see projects leave KDL, but view migration as a positive outcome: the long-term interests of the project should always come first.
- AHRC Crowd Sourcing Study Report
- Corpus of Romanesque Sculpture in Britain and Ireland
- Corpus Vitrearum Medii Aevi of Great Britain
- Cultural Exchange in a Time of Global Conflict: Colonials, Neutrals and Belligerents during the First World War
- Cultural Exchange in a Time of Global Conflict Sourcebook
- Current Epigraphy
- Digital Classicist
- Early Modern London Theatres
- Electronic Sawyer
- Fortune Theatre Records (eREED) - migrated to be archived at the University of Toronto
- German Screen Studies
- Golden Pages for Musicologists
- Gough Map of Britain
- Greek Bible in Byzantine Judaism
- Hieroglyphic Hands
- Humanist listserv
- J.P.E. Harper-Scott (personal website)
- Jonathan Swift Archive
- Mapping Digital Heritage in Jordan data repository
- Modern Languages Open/Online
- Museum of Writing
- Public Monuments and Sculptures Association
- Relics & Selves (scripts and image metadata catalogue only; not full site) migrated to NYU Libraries
- The Sanskrit Knowledge-Systems Project was migrated prior to KDL establishment and is now hosted at Columbia University: http://www.columbia.edu/itc/mealac/pollock/sks/ (KDL also holds basic archival copy)
- Society for the Promotion of Roman Studies was migrated to private hosting prior to KDL establishment: http://www.romansociety.org/ (KDL also holds a basic archival copy)
- Securing a Hybrid Environment for Research Preservation and Access (SHERPA) was migrated prior to KDL establishment and is now hosted at the University of Nottingham: http://www.sherpa.ac.uk/ (KDL also holds basic archival copy)
- THATCamp London 2010
- Victorian Network
- Visigothic PAL
- Warburg Islandora
Migration in progress
- Early English Laws
- Gascon Rolls (dataset)
- Humanist Archive (1987-2021)
- Production and Reading of Music Sources
Projects converted to static sites
Projects converted to static sites remain on KDL infrastructure, but no longer offer search or other dynamic functionality. These are hosted for free by KDL, although we cannot promise to fix issues that appear that are outside our control, such as updates to browsers by external vendors.
- 3D Visualisation in the Arts Network
- Anglo-Saxon Charters
- Anglo-Saxon Cluster
- Body and Mask in Ancient Theatre Space
- Breaking of Britain: Cross Border Society and Scottish Independence 1216-1314
- City Witness
- Computers, Literature and Philology (CLiP) conference 2006
- Cultures of the Lusophone Black Atlantic
- Digital Humanities 2010
- Discover Medieval Chester
- The Durham Liber Vitae Project
- Early Modern Spain
- La Entretenida
- Epidoc Aphrodisias
- Girls and Digital Culture: Transnational Reflections on Girlhood 2012
- Historic Weather
- Investigating the Significant Properties of Electronic Content over Time
- King's Visualisation Lab
- London Charter
- The Making Space Project
- Mapping Digital Heritage in Jordan blogsite
- Mapping Medieval Chester
- Methods Network
- Museum of Writing blogsite
- Out of the Wings (dataset)
- Pompey Project
- Prosopography of the Byzantine Empire
- The Skenographia Project
- Standards for Networking Ancient Prosopographies
- Supporting Productive Queries for Research
- Theatron was developed in HTML and VRML, a now defunct standard for Virtual Reality which no longer has supported plugins for any modern browsers. Content has been converted to static where possible: Theatron 2 and Theatron 3
Minimally archived projects
Minimally archived projects are no longer accessible. We maintain their original URL if it is under our control, and content files and data are archived for a minimum of two years on KDL’s infrastructure (we have had to limit this to two years for practical purposes but aim to keep data longer when feasible). We provide a static placeholder page for historic purposes, with basic information about the project and (where possible) screenshots and links to web archive snapshots. The reasons why a project would be archived in this fashion are diverse:
- The project had reached end-of-life;
- There was no interest in funding or seeking funding to maintain website;
- Project was not appropriate for conversion to a static site;
- Unable to reach Principal Investigator;
- Principal Investigator unknown;
- Project was migrated at some time in the past but KDL retained a copy; or,
- Intention is for project to migrate but there are no current resources to be able to do so at recipient institution.
- Arts and Humanities Data Service: Enabling Digital Resources for the Art and Humanities
- Commodities and Culture in the Colonial World, 1851-1914
- The Commonplacer
- Complete Works of James Shirley
- Digital Cultures of #covid19: Digital Methods for Investigating Online Engagement with Coronavirus
- Digital Exposure of English Place-names
- Digital Humanities: Work in Progress
- Early London Theatre Archive/Online Theatre Histories Archive are two related projects which will be migrated to the University of East London in due course: http://www.elta-project.org/ and http://www.otha.org.uk/
- English Subject Centre - Why Study English?
- État Civil: French Expatriates in Ottoman Egypt
- Forensic Investigation of Digital Objects
- Impact Tracer
- InScribe Module 2
- Interdisciplinary Science Reviews (content also migrated to the current publisher)
- Late-Life Creativity and the New Old Age
- Linking Parliamentary Records through Metadata
- London Association of Classical Teachers Online Records
- Map of Shakespeare's London
- Mapping Global Object Networks and Diasporas
- Mapping Migration in Kastoria, Macedonia
- National Gallery Dealer's Archive
- People of North England
- Poetic Forms Online
- Preservation Exemplar at King’s
- Promoting and Enhancing Reuse of Information through the Content Lifecycle taking account of Evolving Semantics
- SCRIBA, Anar a cal notary a la Catalunya medieval
- Shipping Archives and Integrated Logbooks of Ships: Linking WW1 Naval Records
- Stormont Papers: 50 Years of Northern Ireland Parliamentary Debates Online
- Theoretical Roman Archaeology Conference 2013 was a duplicate of information included at: http://trac.org.uk/
- TEI2008 Conference; a copy can be found at: https://www.tei-c.org/Vault/MembersMeetings/2008/cocoon/tei2008/
- UK Research Information Shared Service Project
- Visual History of War Refugees/Spanish Civil War
Responsibility for KDL's archiving and sustainability strategy rests with its Director, in consultation with the Faculty of Arts & Humanities, King's College London, via the overseeing role of the SLA committee which meets annually. For further information see:
- Ciula, A., & Smithies, J. (2023). Sustainability and Modelling at King's Digital Lab: Between Tradition and Innovation. Nyhan, Rockwell, Sinclair, Ortolja-Baird eds. On Making in Digital Humanities, University College London Press.
- Paul Caton and Brian Maher discuss how KDL approached managing our backlog of now-unfunded legacy software in After the grant, paper presented at the Sixth Annual Conference for Research Software Engineering, Newcastle Upon Tyne, September 2022.
- Smithies, J., Sichani, A.M., & Westling, C. (2017). Preserving 30 years of Digital Humanities Work: The Experience of King’s College London Digital Lab. Presented at the DPASSH: Digital Preservation for Social Sciences and Humanities, University of Sussex.
- Smithies, J., P. Caton, G. Ferraro, L. Figueira, E. Hall, N. Jakeman, P. Mellen, A.M. Sicani, M. Vieira, T. Watts, Carina Westling. (2017). Mechanizing the Humanities? King’s Digital Lab as Critical Experiment. Presented at the DH2017, McGill University, Montreal.
- Smithies, J., Westling, C., Sichani, A.M., Mellen, P., & Ciula, A. Managing 100 Digital Humanities Projects: Digital Scholarship & Archiving in King’sDigital Lab. Digital Humanities Quarterly 13.1 (2019).
Research Software Analyst
I've been responsible for documenting the decommission process for around 70+ projects that have been migrated, converted to static sites or minimally archived. Having worked in the information management sector for several years, the KDL Archiving and Sustainability project is a fantastic opportunity to come to grips with digital archiving of frequently complex digital objects. One lesson that we have taken from this work is to "document as you go"!
Director and Senior Research Software Analyst
As the lists above witness, not only the scale of the archiving process is substantial but as new projects start and end, it lives in the background of KDL daily operations. So it was important to consult with other analysts and feed our perspective to the lab manager, so to align the workflow with KDL Software Development Life Cycle.
Senior Research Software Engineer and Systems Administrator
Through our Archiving & Sustainability efforts, we have been able to greatly improve the security of our systems. I have worked to update or archive as many websites as possible, converting many to static sites, in order to keep important academic research online for as long as possible.
Research Software Lab Manager
I've primarily worked on integrating the Archiving and Sustainability project with the larger KDL Software Development Lifecycle, as well as supporting implementation at various points throughout the project. With a goal of building Archiving and Sustainability into KDL's 'business as usual'; this has involved working with with the team processes and documentation which are robust and standardised. As we reach the stage where SLA maintenance is integrated into the normal work of the Lab, I have implemented processes to ensure that we maintain good oversight of our portfolio of projects under SLA.
Senior Research Software UI/UX Designer
Student Intern - Archiving and Sustainability Assistant
I began the decommissioning documenting process over a two week period in 2018.
Digital Methods Lead (2019-2020)
As part of my role as Digital Methods Lead at KDL in 2019-2020, I assisted the decommissioning analyst in preparing decommissioning documentation and, wherever appropriate, placeholder pages for a number of legacy projects and projects reaching the end of their SLA period. As part of this work, I often had to look into the history of the project and liaise with project partners and technical teams.
Anna Maria Sichani
Marie Curie Postdoctoral Researcher (2016)
I was involved in the KDL Archiving and Sustainability project at its initial stages during my KDL PhD research fellowship. I captured as much information we could for legacy projects so as to sketch an informed sustainability strategy. It was a great school for me in understanding several crucial qualities of a DH project: documentation, documentation, documentation.
Principal Research Software Engineer
My work on Archiving & Sustainability focused on creating simple and easily deployable tools and scripts, that could be incorporated into legacy projects, in order to integrate them smoothly and efficiently into KDL Software Development Lifecycle.
Senior Research Software Systems Manager
As part of our infrastructure renewal, I incorporated some cost effective additional storage into our SAN (disk array) which is just a fancy way of saying: we added some high volume slower disks to complement the super fast SSDs we use for our main service. We designed in the same robust level of fault protection into this archival group (RAID6 plus 2 hotspares - which means any 2 disks can fail simultaneously without loss of data and there are 2 spare disks available to take over from the failed disks). In addition to this, our archival datasets are backed up to the same enterprise grade backup disk systems using the same backup software and monitoring as our main service. At least one of the backup storage systems is located in a remote datacenter to reduce risk. I designed a very simple and effective curation process for archiving whole virtual servers and any associated components such as databases, image sets etc.
Project Manager (2016-17)
I worked with James Smithies and the wider team at KDL to structure and define the problem space of the Archiving and Sustainability project at KDL, and devise a strategy for negotiating onward solutions for ca. 100 legacy projects inherited from the Department of Digital Humanities. Understanding and costing the resources involved in the ongoing maintenance of digital research projects was a big part of this work, and contributed to the development of KDL Software Development Lifecycle processes.