Authors: Clare Tagg (Tagg Oram Partnership, www.taggoram.co.uk), Somia Nasim and Peter Goff (The Qualifications & Curriculum Development Agency, www.qcda.org.uk)

Setting up the project - what is REMS?
In 2005 the British Government published proposals for a radical reform of the education system for 14-19 year olds - the 14-19 Reform. The Qualifications & Curriculum Development Agency (QCDA) is the public body tasked with this redevelopment of the curriculum, qualifications and their assessment. The Reforms are being introduced into schools, colleges and training centres between 2008 and 2013. The work to achieve this is being undertaken by strand teams within QCDA and a range of other Government agencies. Each team is undertaking research, evaluation and monitoring activities to ascertain the extent to which their work is meeting the aims.

REMS is a Research Evidence Management System designed to make strategic use of the evidence produced on the 14-19 Reform, to influence and inform policy development and ensure it is evidence-based. REMS is a complete process where evidence data sources are collected, classified, coded and analysed and finally, the trends and key messages generated from the evidence reported to colleagues.

This ambitious and innovative project is only possible by using NVivo to allow data to be stored, classified, coded and analysed in one place. However the size of the project, its high visibility and longitudinal nature bring a series of technical and methodological challenges.

To deliver this remit a project team was set up which included: Somia Nasim - Project Manager, Peter Goff - Senior Researcher and Clare Tagg - NVivo Expert. The team is further supported by the Research and Evaluation team, coding team and a project steering group. [More...]

The data
The objective of the evidence base in REMS is to provide a comprehensive library of current and background evidence on 14-19 education from internal and external sources. Very early on the REMS team decided to only incorporate secondary sources into REMS, due to the difficulty of managing an evidence base with many varied data sources. The NVivo software is able to import both Word and PDF documents retaining most of the original formatting.

A range of data sources are being input into the database, including policy documents, monitoring, research, evaluation reports, and commentary and media articles. This means the users of the system are presented with evidence that provides a complete picture of progress and developments in the 14-19 Reform programme. Having a diverse range of documents made the coding policy and actual coding difficult as the content and style of the documents varied.

The evidence base is intended to be a live system where there is constant addition of new reports and evidence sources ensuring currency of data. A comprehensive data collection and stakeholder management strategy are in place to manage the volume of data. Buy-in from QCDA colleagues and external stakeholders is needed on a continuous basis, to ensure the system has all relevant evidence. REMS is an integrated system that will incorporate evidence until 2013, and between now and then there will be crucial QCDA and Government review points, the system will be utilised at these stages.

This remit proved to be very challenging for the team, particularly the best way to use NVivo. As a result, a pilot was conducted to test every function of the system. [More...]

Working with data
To make the system work each piece of evidence was read, classified and coded by professional researchers following agreed protocols. The NVivo software was used to store, sort, manage, retrieve and search data on the 14-19 Reform. The features used are:

• Using a specific naming convention; here the importance of storing and indexing all the documents in NVivo, by date and 14-19 Reform strand was a key characteristic.
• Classifying each document including: date of document, author, source, type of evidence (including research method, where appropriate). This enables a set of a certain criteria of documents to be selected.
• Developing a comprehensive coding framework, with clear definitions, a through coding process and quality assurance procedures. To train the coders, a coding training manual was devised. Coding enables easy selection of relevant sections of evidence relating to all major themes or issues in 14-19 Reform.
• Merging individual projects to allow several coders to work on the project at the same time.

Analysis processes
The system enables:

• Fast and effective searching of a complex evidence environment
• Identification of emerging themes and issues, including areas where evidence is scarce or missing
• Development of evidence-informed policy during hectic period of early implementation and evaluation
• A single evidence base can be probed by different stakeholders to provide tailored perspectives.

• Quarterly reports - stats, trends, gaps
• Trend reports
• Strand feedback and sign-off reports
• Adhoc thematic reports
• Analysis manual

Reporting the project
A robust reporting strategy was devised and implemented, to ensure all stakeholders were aware of progress as well as the service REMS could offer. In addition a web-page is being developed, to communicate the progress, reports produced, key evidence output and the REMS processes.

The Research and Evaluation team has demonstrated the REMS potential to internal colleagues and external partners, and there has been strong interest in its capability both in terms of providing an evidence service, and in extending the scope of REMS to other project areas of QCDA's work. [More...]

List of References