The Electronic Records Problem
What's at Risk?
What's Being Done?
The InterPARES Project is a major international research initiative in which archival scholars, computer engineering scholars, national archival institutions and private industry representatives are collaborating to develop the theoretical and methodological knowledge required for the permanent preservation of authentic records created in electronic systems.
This research is urgently needed as the information technology revolution has dramatically altered the way in which governments, corporations, and individuals communicate and carry out their daily activities. As a result, many records that would have traditionally been created and preserved on paper are now in electronic form. These records may have to be preserved permanently for administrative, legal or cultural reasons.
However, technological obsolescence, storage media fragility, and the manipulability of electronic systems challenge our capacity to guarantee the long-term preservation as well as the authenticity of electronic records.
This is an immense problem for governments, industries, researchers, and the citizenry as it affects accountability, productivity, commerce and the ability to use records as reliable sources of information.
As the traditional custodians of society's records, members of the world's archival community have come together under the InterPARES umbrella to tackle the "electronic records problem" in a comprehensive, collaborative manner.
Major funding contributions have been made by Canada's Social Science and Humanities Research Council, the American National Historical Publications and Records Commission, the National Archives and Records Administration of the United States, and the Italian National Research Council. As well, universities and national archival institutions from around the world have committed financial resources and research talent to the project. Participating countries include Canada, the United States, Italy, The Netherlands, Sweden, Finland, France, Portugal, England, Scotland, Ireland, Australia, China and Hong Kong (see
Participants for the full list of InterPARES researchers).
The InterPARES Project is borne out of previous research carried out at the University of British Columbia's School of Library, Archival and Information Studies. "The Preservation of the Integrity of Electronic Records" (a.k.a. "The UBC Project") defined the requirements for creating, handling and preserving reliable and authentic electronic records in active recordkeeping systems.
The UBC Project researchers, Dr. Luciana Duranti and Professor Terry Eastwood, worked in close collaboration with the U.S. Department of Defense Records Management Task Force to identify requirements for Records Management Applications (RMA). The resulting 5015.2 standard is now being used by the U.S. Defense Information Systems Agency to certify RMA vendors.
The second phase of the UBC Project was intended to address the long-term preservation of inactive electronic records (ie. records which are no longer needed for day-to-day business but which must be preserved for operational, legal, or historical reasons). The immense scope and ubiquity of the issues surrounding the long-term preservation of authentic electronic records made evident the need for an interdisciplinary, international approach.
Between the period of March 1997 and December 1998, Luciana Duranti formally invited scholars, archival institutions and private sector representatives to join a collaborative research project under her direction. The project's researchers met for the first time as a group at a preliminary planning meeting sponsored by the U.S. National Archives and Records Administration in Washington, D.C. (June, 1998).
At the Washington meeting a research plan was drafted and "InterPARES" was chosen as the project name. An acronymn for "International Research on Permanent Authentic Records in Electronic Systems," inter pares is also latin for "amongst peers," an appropriate name given the collaborative nature of the project.
Following another strategic planning meeting in Cagliari, Italy in October 1998, the InterPARES Project was officially launched on January 1, 1999.
InterPARES Researchers at the project's
inaugural meeting. June 1998,
The Electronic Records Problem
The last decade has generated more recorded information than any previous decade of human activity. The fact that the majority of these data is less accessible than ever before is one of the ironies of the modern information age. Idiosyncratic software systems generate, manage, and store digital information using proprietary technologies and media that are subject to the dynamism of the computer industry. This digital information gets lost in a self-perpetuating and expensive cycle of obsolescence and incompatibility. This problem is self-evident to anyone who attempts to recover an old letter from an outdated 5 1/4" floppy disk which was written with an obsolete word processing program on a personal computer whose manufacturer is no longer in business.
Furthermore, organizations and individuals generate records in a variety of media and formats. In most modern offices it is quite common for records related to a single matter to exist partly in traditional paper format, partly in an email box, word processing file, spreadsheet file, multiple database tables, etc.. Moreover, using technologies such as hypertext linking or Object Linking and Embedding (OLE), individual electronic records are increasingly composed of a variety of different formats which may or may not reside on different media. It is difficult enough to establish and maintain the essential links among and within these records while they are being actively used. One of the main challenges facing archivists today is how to preserve such links over the long term so that, one hundred years from now, one will still be able to view the complete file or "dossier" relating to a particular matter, let alone all the essential components of the compound records which might be contained within that dossier.
To deal with the hybrid nature of modern recordkeeping, some organizations have attempted to reduce all records produced by an office to a single medium, for example, by printing out email and inserting it in a paper file, by scanning paper documents into electronic systems, or by converting electronic and paper records to microfilm. These attempts have been unsuccessful for a number of reasons. Firstly, the conversion of records purely for preservation reasons tends to hamper the workflow in an office and, consequently, its implementation is often sporadic and inconsistent. Secondly, many records do not lend themselves to such conversion. For example, hypertext records cannot be printed out to paper and scanned maps or photographs are not always reliable substitutes for the paper originals. Thirdly, recent court decisions have rejected the practice of converting electronic records to other media on the grounds that the converted records lack elements critical to their use as evidence.
What's at Risk?
Virtually every private and public organization which uses information technology to facilitate its recordkeeping functions has experienced the undesirable effects of adopting new technologies without forecasting and planning for the consequences of the proprietary nature of software applications, media and digital obsolescence, and hybrid paper/digital environments. One of the more dramatic examples includes the thousands of databases and digital files of the government of the former East Germany. Archivists at the German Federal Archives headquarters in Koblenz have attempted to save these records which include agricultural files and labour statistics, penal registration lists, and personnel files of Communist party functionaries. However, the documentation of the digital systems on which the records were generated is missing, the software codes are unknown, and the storage media themselves are obsolete and in poor condition. Consequently, the electronic records of East Germany are lost to the new German government that needs the information they contain for administrative purposes, to the citizens whose interests are implicated in those records, and to present and future researchers the world over.
Lack of authenticity presents a problem as serious as lack of accessibility. During the spring of 1996, for example, the inadequacy of procedural mechanisms for ensuring the authenticity of electronic records became a focal point of hearings held by the Canadian Commission of Inquiry into the Deployment of Canadian Forces to Somalia. The commission was investigating the shooting of Somali intruders at a Canadian Forces compound, the beating death of a teenager in the custody of the Canadian Airborne Regiment and an apparent suicide attempt by one of these soldiers. As part of its investigation, the Commission requested access to National Defense Operations Centre (NDOC) logs, which were maintained in an automated database and which contained a record of all message traffic coming into National Defense headquarters from Canadian Forces' theatres of operation. During its review of the logs, the Commission discovered several anomalies, including entries containing no information, missing serial numbers, or entries with duplicate serial numbers. The Commission was concerned that there may have been deliberate tampering with these logs. Although subsequent investigations were unable to show evidence of tampering, they could not exclude the possibility of it because of the absence of standard operating procedures with regard to the log, the complete ineffectiveness of the security system in place, a lack of system audits, and the tendency of officers to bypass the awkward system. Therefore, the Commissioners concluded that NDOC logs were not a reliable record of transactions at the operations centre either for present investigators or future researchers.
Records, therefore, need to be trustworthy and accessible to protect rights and to uphold responsibilities but also to carry out business functions and to make informed decisions. Accessibility to authentic records over the long term is a crucial requirement for public and private organizations around the world. For example, most governments are bound by archival laws which require the permanent preservation of their records, the pharmaceutical and bio-chemical industries are required by law to preserve research notebooks for a minimum of one hundred years; the mining industry is required to maintain documentation of excavations for as long as mining sites exist; agencies concerned with environmental protection and energy are required to permanently preserve records on the location of waste. Moreover, research and cultural institutions as well as their clientele need to be able to trust records that have been reproduced from one medium to another or from one digital technology to another.
What's Being Done?
The archival profession has long been aware of the issues surrounding the long-term preservation of electronic records. Nevertheless, only a small number of the larger archival institutions have had the resources and expertise to deal with them. Although their efforts are at the forefront of current methodological knowledge about electronic records preservation, they are far from comprehensive and often encompass a limited domain of electronic records. Moreover, these programs have been reactive and have tended to deal with specific issues as they present themselves. Given the magnitude and the multi-faceted nature of the issues involved, they have also been slow in developing effective strategies, policies or standards for the long-term preservation of electronic records.
The scholarly research in the field of electronic records has dealt predominantly with the creation and maintenance of electronic records that are in active use (ie. the UBC Project, the Pittsburgh Project). This research has primarily produced theory and methodology for ensuring that electronic records are reliably generated and that their authenticity is preserved by the organization needing them in the usual and ordinary course of business. Given the predominance of research on active records, there have been repeated calls by the archival community for scholarly research on the long-term preservation of inactive electronic records.
The technological aspects of long-term preservation are being explored in a number of studies of migration methods (i.e. ways of moving records from an obsolescent digital technology to a new one). As well, the software industry is increasingly concerned with developing open system architectures and promoting inter-compatibility amongst new technologies. Nevertheless, it is quite clear that technology alone is incapable of solving all the problems presented by the long-term preservation of authentic electronic records.
The theories and methodologies for the long-term preservation of authentic electronic records must be
centered on the nature and meaning of the record itself. Despite the new media and formats of electronic records, the integral components which identify and authenticate a record have not changed from the perspective of archival science.
The goal of the InterPARES Project will be, therefore, to use the tools of archival science and diplomatics to develop the theoretical and methodological knowledge essential to the permanent preservation of electronically generated records. On the basis of this knowledge it will formulate model strategies, policies and standards capable of ensuring their preservation.