On evaluation design

Third set of principles

The other thing to remember is that even if you’re leading the evaluation, it’s not your evaluation. One thing you don’t want to create is an “us and them” division within a project, where teachers provide data for the researchers. Education research should be designed with the end user in mind, which is educators, and they know better what they need to know. And everyone in the project is bound to have a good idea about research questions (I refrained from saying “better idea” but that’s probably true too). So the research questions, survey design, sources of data, all need to be collaboratively created, with practitioners and (if they’re interested) students. If there are other practitioners who want to contribute to the evaluation and writing of the report and any papers coming out of the project, they also have a right to do that, and should be included. I know of some projects (none I’ve been involved with thankfully) where academics have just gone to ground with the results and months later have a paper published, without offering anyone else within the project the opportunity to be involved and get a publication out of it. Which isn’t on. The AMORES project brought some of the schoolchildren along to the final conference. This shouldn’t really be exceptional, but it still is. Arguably it’s the learners who are the rationale behind doing all of the research in the first place. (Competing arguments are that it’s our mortgage lenders who are the rationale for doing it, but that’s another post entirely.)

So… #3 evaluation design should be egalitarian, inclusive, participative.

Now would be a good time to mention ethics, probably as it brings together all of the principles we’ve discussed so far.

Obviously everyone who takes part in the project needs to be protected. Everyone has the right to anonymity if they are taking part, so usually I get students to adopt a pseudonym for all interactions. There’s a piece of paper somewhere that matches pseudonym to real name (in case the student forgets and needs to look it up) but that never goes online and never leaves the classroom. Protecting the identities of staff is also important if that’s what they want, but also acknowledging their participation if that’s what they want too. Just remember to ask which it is. But ethics is really the underlying reason why you want the evaluation to be useful (you’re obliged ethically to put something back into the sector if you’re taking time and resources from it) and to be egalitarian (everyone deserves a chance to be published and have a creative input to the process.

So #4 Be ethical.

Tags: / / / /

About the author

Mark Childs

As Senior Lecturer in Technology Enhanced Learning, Mark’s role is to help deliver the Technology Enhanced Learning Framework across Oxford Brookes and to support OCSLD and its staff with their online presence.

Mark’s career in Higher Education has two complementary strands, as a researcher in TEL since 1997 and as an educational developer in TEL since 2003. He has worked at the University of Wolverhampton, the University of Warwick and Coventry University. Between 2011 and arriving at Brookes in 2015 Mark worked as a “freelance academic” providing educational research, consultancy and training for a range of clients including the Open University, Hewlett Packard, The Field Museum of Natural History in Chicago, Ravensbourne College, Worcester University and the Tablet Academy where he is currently its research director.

Mark’s educational development work is informed by, and provides a goal for, his research. The core of these research interests is the use of a wide-range of synchronous communication platforms for education, including social media, videoconferencing, virtual worlds and games-based learning. His most recent work is in the area of online collaboration for design using social media and videoconferencing, where he has evaluated the learner experience of students in distributed teams in projects led by Loughborough University and by CARNet in Zagreb, Croatia. In parallel to this he has a wide-ranging interest in many other fields of research; for example, his most recent publication is a DVD with the OU on Ethiopia’s progress towards reducing child and maternal mortality.

You may also like...