3 Methodology

The project used a case study design as this allows for both breadth and depth of investigation. This was necessary as the analysis needed to explore the impact of large-scale networked learning on individual institutions as well as on a range of institutions across the sectors. Within this design both qualitative and quantitative data was gathered along with analysis of secondary sources using institutional documentation. This mixed methods approach allowed for triangulation of data within each case study. Replication by multiple case studies allowed some degree of external validity to allow tentative building up an explanatory theory (Yin, 2003). These multiple case studies were selected so that they replicate each other “either predicting similar results (literal replication) or contrasting results for predictable reasons (theoretical replication)” (Yin, 2003).

Thus, whilst it is not possible to generalise from a single case study our design allows us to relate our findings to the literature in order to develop an explanatory analysis and to identify research questions and hypotheses for future research.

3.1 Selection of case study institutions

It was determined that twenty case studies could be accommodated in the study. The primary criteria for our selection of the case study institutions were the possession of a VLE, and large-scale use of the VLE by departments across the institution or large-scale use by the student population.

Information on these criteria was collected from as many institutions as possible by sending out an e-mail request, via NILTA , FERL , SFEU and JISC RSCs with an accompanying short questionnaire to a wide range of HE and FE institutions asking whether the institution used a VLE and if so what percentage of staff and students used it. This resulted in information from 115 institutions, 65 of which were FE and 50 of which were HE. From these, a shortlist of the top 20 FE and HE institutions was created in terms of percentage use of the VLE. Other criteria were then considered to produce an illustrative range of institutions. These criteria, included location, size, research versus teaching balance, pre and post 1992 Universities, single-site versus multi-site, urban versus rural, and FE-HE hybrids. The case study institutions were selected from within this group to provide the widest range of illustrative examples possible.

These institutions were then divided into two groups, preferred choices, and understudies. The preferred institutions, 10 FE and 10 HE with the highest level of VLE usage, were approached in the first instance. Where a preferred institution was unable to take part, an alternative similar institution from the backup list was approached.

Within each institution, we required access to a sample of staff who were involved in particular aspects of the development of networked learning. Staff who had responsibilities or experience in the main areas of investigation (listed below) were selected by the institution.

• Academic staff implementing networked learning,

• Quality manager,

• Registry staff responsible for student records,

• Senior management with responsibility for networked learning and academic quality,

• Support staff providing a range of services for academic staff and students engaged in networked learning,

• Technical staff with responsibility for infrastructure support for networked learning.

These staff were asked to complete a questionnaire and to take part in an interview. Figure 3-1 gives a breakdown of the numbers of staff representing each role from each sector. It should be noted that in a few instances interviewees failed to return their completed questionnaire despite repeated reminders. The exact number of interviewees and questionnaire responses are recorded within the individual case studies.

Figure 3-1: A breakdown of number of participants according to role and institution

Each of these roles was assigned a code number to ensure anonymity of the respondent when quoted in the individual case studies. The numbers do not reflect the order of roles as shown above; however, the numbering allows for tracking of the responses that come from a particular individual within a specific case study, and from similar roles in across all the case studies. In cases where there are two or more individuals interviewed from the same role this is indicated by adding a letter for each individual.

3.2 Data collection

An initial pilot study was conducted, using two institutions to refine data collection plans of both content and procedures, and to help in developing relevant lines of questions and conceptual clarity. The pilot report provided a basis for both research design and fieldwork procedures. The pilot study indicated that the relevance and importance of particular issues is likely to vary between institutions, and that the main study required three different types of evidence:

• documentary evidence including strategy and policy documents,

• semi-structured interviews,

• questionnaires.

The pilot study also indicated that interviews and questionnaires needed to include a range of individuals having different roles and positions within each institution.

Two different interviewers carried out the data collection due to the workload involved in gathering data from such a large number of institutions that were geographically dispersed. One interviewer carried out case studies F1 to F5 and H1 to H5 and the second interviewer carried out case studies F6 to F10.and H6 to H10. The use of two different interviewers inevitably impacts of issues of reliability in data collection and efforts were made to minimise the impact. However, it has led to some differences in the questions used during the semi-structured interviews and this should be noted in relation to the analysis of the data.

Collecting these three different types of data allowed the triangulation of different types of evidence, and the construction of ‘snapshots’ of the current situation within the institutions. These led to an examination of:

• the process(es) the institution went through to arrive at its current state,

• the current state of the institution in relation to successes, barriers and critical developments,

• progression - where the institution plans to go next and why.

It is important to note that the interview data represents the views of particular individuals within each institution at a particular stage of the process of development of networked learning. Technology is changing, as is the context within which educational institutions operate and it is possible that specific situations referred to within these case studies have also changed. The interviews were carried out during the summer and autumn of 2003.

3.3 Data Analysis

Each of the case studies uses data from institutional documents, questionnaires, and semi-structured interviews. A study of pertinent documents was used to identify institutional policies and strategies within each case study organisation; the questionnaires were used to answer some of the factual questions arising from the institutional issues; and the interviews were used to explore the experiences and perceptions of the staff in the institution.

The researchers have taken an interpretive approach to the data analysis. This approach emphasises that there is no objective ‘truth’ in the data; rather data can be interpreted from a range of perspectives (Bryman, 2001). It also emphasises that all knowledge is constructed in and through interactions between human beings and their world, and developed and transmitted within an essentially social context (Crotty, 1998). The interpretation has been based on the need to address the research questions identified and refined during the early stages of the project and the main issues that emerged in the interviews in relation to these issues.

Interview transcripts and the written responses to the questionnaires were used with documentary evidence to provide some degree of triangulation to ensure that significant changes were not missed and were adequately understood (Ackroyd and Hughes, 1992).

The interview data was analysed qualitatively and coded using N6 QSR software, two different coders were involved in the main coding of the data with a third coder sampling the coding of both coders. This approach allowed for comparison between coders and also for an overall coding frame to be developed.

Transparency and reliability were important issues in the analysis of the interview transcripts that proved to be a particular challenge and which resulted in two slightly different datasets. To facilitate these aspects version 6 of NUDIST (Fielding and Lee, 1998) was used in an attempt to provide transparency of analysis and to ensure consistent data exchange between team members. In addition, the following steps were taken to improve reliability between interviewers and coders:

• three meetings to develop and agree an interview schedule and questionnaire,

• two meetings to discuss and agree a coding frame,

• one meeting to check a common understanding and comprehension of the coding frame,

• limited double coding to check consistency of interpretation of coding frame (due to constraints of personnel and time).