On evaluation design

Some thoughts on my approach to evaluation design.

I’ve just finished another internal evaluation of a project. This time it’s the AMORES project (www.amores-project.eu). Reflecting on the evaluation, and the similarities with the previous evaluation I did, led me to some realisations about the sort of evaluations I conduct, how they are designed, and what their essential elements are. I thought I’d collect these together into a couple of blog posts, mainly so that the next time I design one, I can remember the best of what I did before.

I should specify that I’m discussing particularly internal evaluation. For those not familiar with educational projects, most of them have two evaluation strands. One is the external evaluation; this is conducted by someone outside of the project who examines how well the project functioned, whether it met its goals or not, how well communications worked within it, and so on. It’s part of the Quality Assurance, compliance and accountability process.

The internal evaluation asks questions of the learners, teachers and anyone else involved with the educational aspects to identify good practice, look for tips that can be passed on, and encapsulate the overall experience for the learners and educators. In short, it’s there to answer the research questions addressed by the project.

There’s a good deal of overlap between the two, but they are essentially different things, and should be done by different people. You merge the two at your peril, as part of the external evaluation is to address the success of the internal evaluation. And you do really need both to be done.

I’ve been the internal evaluator on 13 education projects now, but the last two (the other one was the BIM Hub project, bim-hub.lboro.ac.uk) were very similar in evaluation design; I think I’ve cracked the essential elements of what an internal evaluation should look like.

Part of the issue with being an internal evaluator is that, even though you’re part of the project team, you’re not (usually) one of the teachers. And teachers on projects have their own agenda, which is to teach (obviously) and, quite rightly, this takes precedence over all the analysis, research and general nosiness that a researcher wants to conduct.

For this reason, an evaluation design needs to be as unobtrusive as possible. Most education activities generate a lot of data in themselves, artefacts, essays, recordings of teaching sessions, all of these can be used without placing any additional burden on the learners or teachers. Sometimes the evaluation can drive some of the learning activities. So, for example, you need students’ perceptions of their learning; so you set as an assignment a reflective essay. You need something to disseminate, so you set students the task of creating a video about their experiences, which can also be evaluation data. And when we’ve done this, not only does this prove to be a very useful set of data, it also becomes an excellent learning opportunity for the students. Teaching generates a lot of data already, too, such as grades, results of literacy testing, pupil premium figures, tracer studies. As long as the institution releases the data, then this is stuff you can use with no impact on the learners or teachers.

Tags: / / / /

About the author

Mark Childs

As Senior Lecturer in Technology Enhanced Learning, Mark’s role is to help deliver the Technology Enhanced Learning Framework across Oxford Brookes and to support OCSLD and its staff with their online presence.

Mark’s career in Higher Education has two complementary strands, as a researcher in TEL since 1997 and as an educational developer in TEL since 2003. He has worked at the University of Wolverhampton, the University of Warwick and Coventry University. Between 2011 and arriving at Brookes in 2015 Mark worked as a “freelance academic” providing educational research, consultancy and training for a range of clients including the Open University, Hewlett Packard, The Field Museum of Natural History in Chicago, Ravensbourne College, Worcester University and the Tablet Academy where he is currently its research director.

Mark’s educational development work is informed by, and provides a goal for, his research. The core of these research interests is the use of a wide-range of synchronous communication platforms for education, including social media, videoconferencing, virtual worlds and games-based learning. His most recent work is in the area of online collaboration for design using social media and videoconferencing, where he has evaluated the learner experience of students in distributed teams in projects led by Loughborough University and by CARNet in Zagreb, Croatia. In parallel to this he has a wide-ranging interest in many other fields of research; for example, his most recent publication is a DVD with the OU on Ethiopia’s progress towards reducing child and maternal mortality.

You may also like...