Phoebe Phase 2 Evaluation Plan

Document history

• Version 1: 03/10/07
• Minor revisions: 10/01/08, 18/03/08

Research questions for the Phase 2 evaluation

As in Phase 1 we are evaluating Phoebe as a proof-of-concept tool; however, this time we are testing it in more authentic environments, in order to address the following questions (adapted from Phase 1):

I Is Phoebe a tool that practitioners in post-compulsory education find usable, helpful and relevant to their needs, whether they are

a) beginning or experienced teachers looking to use technology in their teaching for the first time, or
b) are familiar with e-learning but are looking for new ideas re technology and pedagogy?

Specifically, does it encourage practitioners to think about their practice in a structured way?

II Can one tool address all sectors of post-16 education, or are separate tools required for, say ACL, WBL, FE versus HE?

III Is Phoebe suitable as a tool for teacher education and/or a tool for supporting everyday practice?

IV Is there a perceived need for Phoebe to be customised to meet the needs of individual organisations (i.e. to function as a community-owned artefact)?

Specific questions to ask (stemming from the research questions)

1. Key aspects of usability

1.1 Ease of learning to use

1.1.1 What "getting started" guidance is needed?

1.2 Ease of use once learned

1.2.1 Does Phoebe work? 1.2.2 What does Phoebe do well? 1.2.3What does Phoebe do badly?

1.3 Productivity/usefulness/effectiveness (task accomplishment) (also covered in practice-related issues below)

1.4 Affective response (enjoy using it/choose to use it/recommend to others)

1.5 Sustainability? (Not just new features, but also projections re effort likely to be involved in maintaining currency of content and examples.)

2. Integration into individual pedagogic practice

2.1 Mapping to their established practice (processes, functionality)

2.2 Usefulness at different levels: course/scheme of work planning vs topic planning vs session planning

2.3 To what extent does Phoebe encourage them to think about their practice in a structured way? If yes, why and how? If no, why not?

2.4 Relevance to their established practice (content) (focus on generic vs examples vs institution-specific examples)

2.5 Extent to which they feel they have learned something new/been inspired to experiment/innovate in their own practice

2.6 Relationship between their use of Phoebe and their continued use of existing tools: Do practitioners combine the use of Phoebe's planning functionality with their familiar tools?

2.7 Would practitioners want to use Phoebe in their normal work?

2.7.1 If yes, what role would Phoebe play? 2.7.2 If no, why not and what, if anything, would induce them to use it again?

2.8 Do they consider Phoebe to be useful to others? At what stage of career/at what level of experience wrt e-learning?

3. Integration into pedagogic practice of the institution (community dimension)

3.1 Staff trainers: does it map to the models of pedagogy which they are trying to communicate to their trainees?

3.1.1 If no, how much effort is required to make it map? Could simple customisation options (e.g. ability to edit "default" guidance, add own case studies and examples, components of plan, terminology?) work, or would they need a totally customised tool?

3.2 Can the tool be used by practitioners working alone or does it work best in an institutional setting, as a community artefact?

3.3 To what extent does Phoebe facilitate or, conversely, impede established practice relating to session planning in the user's institution?

Back Top

Evaluators for Phase 2

1. E-learning experts:
   • Teaching staff who are already familiar with the use of technology in their teaching/learning
   • Staff developers with responsibility for promoting e-learning and tutors in ITT who are actively promoting
   • IT support officers and learning technologists who are responsible for disseminating the use of technology in their institution/department and/or are involved in specific 'projects' (e.g. creating an image bank; redesigning a F2F course to run in a blended or fully online environment)
2. E-learning novices:
   • Teachers undergoing CPD aimed at introducing them to e-learning
   • Teaching staff who are new to technology but not participating in a CPD e-learning programme
   • Students undergoing ITT (NB they may or may not be experienced in using digital technologies)

Possible sources of evaluators:

NB This is just a list of 'possibles': none of these groups has yet been approached.

• Members of JISC Pedagogy Experts group
• Selected participants from the Learning Design Tools project (mailing list is still open)
• PIs and/or their colleagues (esp. where originally approached to take part in 'embedding' workshops)
• TechDis
• Serendipitous contacts, including those who find out about Phoebe through third-party connections or Web searches (see also "Friends of Phoebe" and list of people who contacted us for user accounts in Phase 1)
• Workshop participants in the University of Greenwich staff conference September 2007 who attended a demonstration of Phoebe and expressed interest in another workshop

We envisage recruiting only a small minority of evaluators from the D4L programme itself as we feel that the evaluation will be more fruitful (and objective?) if evaluators are from outside the D4L "community."

Back Top

Evaluation methods

We will use methods that focus on capturing a rich set of qualitative data from a small group of (experienced) people.

The proposed methods are shown in the following table:

Instrument	Subjects	Purpose	Qual/Quant? data?	Notes & issues
Heuristic evaluation	Project team	Optimise usability of interface so as to eradicate unwanted factors in evaluation of pedagogical aspects of tools.	Qual + Quant	Base on user-centred and learner-centred principles; modify, extend and/or replace the Nielsen and Sharples & Beale guidelines. (See other lit on UI design in e-learning, incl. Hedberg et al. paper.)
Observation, optionally with recorded think-alouds or short semi-structured interviews	Evaluators	Affective responses to experience; Usability problems	Qual	Feed into case studies
Semi-structured group discussions	Evaluators	Affective responses to experience (reflective)	Qual	Feed into case studies
Session plans created during workshops?	Evaluators	Capture examples of output - components used in different plans, structures (ordering of components)	Qual	Feed into case studies
Questionnaires	Evaluators	Affective responses to Phoebe; other attitudinal data	Qual + Quant	Draw for inspiration from questionnaires written for LAMS evaluation, LD Tools project; feed into case studies?
Webserver request logs	Evaluators	Investigate users' behaviour re the planning and reference aspects of Phoebe	Quant	From Dave B. 28/08/07: "with the guidance and functionality running as independent websites, correlating users in their logs will be necessary if we wish to examine paterns connecting the two parts of Phoebe."

Back Top

Phase 2 evaluation schedule

The Phase 2 project plan identifies five key 'events' in the evaluation schedule: workshops involving trainee teachers, in-service training (CPD), a workshop for experienced e-learners, and workshops to evaluate the customisation features of Phoebe and its support for collaborative planning. On reflection, some of these can be combined (e.g. experienced e-learners and customisation) or may prove infeasible to run as "live" workshops (e.g. collaborative planning). On consultation with Glenaffric, we now also propose to capture feedback from practitioners who ask us for a Phoebe account (30+ in Phase 1) through an online survey. However, we have yet to ascertain the likely value of such data in relation to the "live" face-to-face events.

Timing	Event	Factor to Evaluate	Questions to Address	Method(s)	Measure of Success
Oct-Nov 2007	Internal usability tests	UI and functionality of prototype tool	Consistency, usability, error-free functioning	Walkthrough by project team members	Number of issues identified and resolved
31 Oct 2008	Trainee teachers' workshop (HE), University of Greenwich	Usability; embedding in ITT; suitability for novice teachers	Usability; value of structured support for the planning process; value of the reference materials in inspiring thinking	Observation, discussion, session plans, Web request logs	Levels of satisfaction and acceptability expressed by participants; extent to which the tool is judged to be of use to trainees
16 Nov 2007; 28-30 Jan 2008	In-service training: staff development workshops at Oxford Brookes University	Embedding in CPD; usability and usefulness of tool	Usability; integration into personal pedagogic practice; integration into institutional practice	Observation, discussion, session plans, questionnaires (? to capture current practice?), Web request logs	Levels of satisfaction and acceptability expressed by participants; extent to which the tool is judged to be of use to individuals (esp. in helping them to engage with e-learning and to plan in a structured way) and the community
14 Jan 2008	Experienced practitioners' workshop held at Oxford University Dept of Continuing Education; attended by practitioners in FE, HE, ACL, WBL	Sustainability; suitability for experienced practitioners; customisation; support for collaborative planning(?)	Integration into personal pedagogic practice; integration into institutional practice	Observation, discussion, session plans, Web request logs	Levels of satisfaction and acceptability expressed by participants; knowledge about the ways in which practitioners might want to customise Phoebe; input on the additional functionality needed to support customisation at an acceptable level; ?input into additional functionality needed to support collaborative planning (incl. enhanced understanding of the ways in which practitioners plan collaboratively)
29 Jan 2008	CPD workshop, University of Brighton (led by Oxford Brookes University)	Sustainability; suitability for experienced practitioners; customisation; support for collaborative planning(?)	Integration into personal pedagogic practice; integration into institutional practice	Observation, discussion, session plans, Web request logs	Levels of satisfaction and acceptability expressed by participants; knowledge about the ways in which practitioners might want to customise Phoebe; input on the additional functionality needed to support customisation at an acceptable level; ?input into additional functionality needed to support collaborative planning (incl. enhanced understanding of the ways in which practitioners plan collaboratively)
6 Feb 2008	Trainee teachers' workshop (FE), Swansea College	Usability; embedding in ITT; suitability for novice teachers	Usability; value of structured support for the planning process; value of the reference materials in inspiring thinking	Observation, discussion, session plans, Web request logs	Levels of satisfaction and acceptability expressed by participants; extent to which the tool is judged to be of use to trainees
4 Mar 2008	Embedding and sustainability	Suitability for embedding in staff development and/or initial teacher-training context; suitability across sectors and domains	Strategic review meeting with representatives from HEA, Becta, ACLearn (joint event with the London Pedagogy Planner project; organised by JISC)	Surveys	Quality of feedback re suitability
Feb-Mar 2008	Online evaluation by interested (mainly experienced?) users	Usability; usefulness	Integration into personal pedagogic practice	Questionnaires, Web request logs(?)	Positive feedback on tool + constructive suggestions for future directions (both functional and in terms of deployment)

Back Top