On evaluation design

So here’s the first set of criteria. Evaluations must be:

Unobtrusive, opportunistic, aligned with teaching practice

The second set of criteria is related to actually having an evaluation that makes sense. There’s no point gathering a set of data that are more than you can deal with (having said that, every project I’ve done has). Also the data you collect have to be targeted towards finding out something that will be of use to other practitioners once you’ve finished the project (I’ll come to outputs later). The RUFDATA approach is a good one here. There’s also no point trying to gather so many data that no-one will look at the surveys you’re distributing, or complete them if they start them. For length of survey principles that seem to work are:

Quantitative questions
no more than one page (and use a 5-point Likert scale obviously — anything else looks ridiculous — but add “don’t know” and “N/A” as options too.
Free text questions
Well, no-one wants to write an essay, and if it’s on paper you’ll have to transcribe them at some point anyway. As far as numbers go, a good rule of thumb is that if it’s a number you’d see in a movie title it’s OK. So seven, or a dozen, or even 13, is fine. More than that is pushing it (and if you’re going to ask about 451 or 1138 questions then full marks for movie trivia, but minus several million for being a smart person. The point of the movie title thing is that if you see your research questions as characters in the narrative you’re going to weave, then you don’t want to overcrowd your story anyway. Putting too many in then becomes pointless. You want all your questions to be Yul Brynners, rather than Brad Dexters.

So: useful, targeted, light touch, practicable.

A third set of principles is based around whose research is it anyway? Which will be covered when we reconvene on the next page.

Tags: / / / /

About the author

Mark Childs

As Senior Lecturer in Technology Enhanced Learning, Mark’s role is to help deliver the Technology Enhanced Learning Framework across Oxford Brookes and to support OCSLD and its staff with their online presence.

Mark’s career in Higher Education has two complementary strands, as a researcher in TEL since 1997 and as an educational developer in TEL since 2003. He has worked at the University of Wolverhampton, the University of Warwick and Coventry University. Between 2011 and arriving at Brookes in 2015 Mark worked as a “freelance academic” providing educational research, consultancy and training for a range of clients including the Open University, Hewlett Packard, The Field Museum of Natural History in Chicago, Ravensbourne College, Worcester University and the Tablet Academy where he is currently its research director.

Mark’s educational development work is informed by, and provides a goal for, his research. The core of these research interests is the use of a wide-range of synchronous communication platforms for education, including social media, videoconferencing, virtual worlds and games-based learning. His most recent work is in the area of online collaboration for design using social media and videoconferencing, where he has evaluated the learner experience of students in distributed teams in projects led by Loughborough University and by CARNet in Zagreb, Croatia. In parallel to this he has a wide-ranging interest in many other fields of research; for example, his most recent publication is a DVD with the OU on Ethiopia’s progress towards reducing child and maternal mortality.

You may also like...