Excel autograding: alternative e-assessment

This poster on Autograding describes an alternative method of e-assessment which was developed by one of my colleagues.

It is designed for assessing practical write-ups, mostly in the first year of a Life Sciences degree, where all students follow the same experimental protocol, but are required to record and analyse their own results.

The drivers for the introduction of this new assessment method were:

  • to deal with increasing numbers – from 110 to 220 students per class – meaning many more papers to mark;
  • to improve the quality of feedback, making it more extensive, more consistent and more quickly delivered.

Another benefit is that students become familiar with using Excel, which is a useful transferable skill, and they receive detailed and consistent instruction in preparing graphs and other aspects of recording and analysing data.

Students like this method of assessment because they receive lots of feedback with a quick turnaround.

Staff appreciate that the autograding process makes marking quicker, simpler and more objective.

This is a useful alternative method of assessment, but it can’t be the only assessment method used on a course. Writing up a traditional lab report is still a required skill which must be practised and assessed.

 

Declaring UDL – Universal Design for Learning

In spite of a lack of evidence that the use of learning style inventories can improve teaching and learning (see my previous post for more on this), the idea still has supporters. I think that this is because intuitively it seems to make sense. We relate to the idea that different people prefer to learn in different ways – although we may add the qualification that this is not fixed, but varies depending on context and motivation. It is difficult to understand if and how we can use these differences in teaching; maybe it’s just that learners need to be aware of the different approaches and incorporate them into their learning.

While reading about learning styles, I came across another concept, which intuitively makes sense, the idea that:

The references provided in the original website were not informative, but some googling led me to information on UDL – Universal Design for Learning – for example, the CAST organisation, which defines UDL as follows:

Universal Design for Learning is a set of principles for curriculum development that give all individuals equal opportunities to learn.

UDL provides a blueprint for creating instructional goals, methods, materials, and assessments that work for everyone–not a single, one-size-fits-all solution but rather flexible approaches that can be customized and adjusted for individual needs.

This is based on the principles of universal design in architecture and other domains, which suggest that designing for all users, including those with accessibility needs, from the outset will lead to a better result for everyone.

The use of UDL is widespread in the US and Canada, being enshrined in accessibility legislation, and there is much very good material on the topic available online. For example, I spent some time exploring an openly available UDL course for educators in Ontario.

One of the recommendations of this course resonated with me, since I greatly appreciated the multiple ways of viewing learning materials offered on an xMOOC that I undertook.

an accessible course design improves the learning opportunity for all students and significantly reduces the need for accommodation [for students with a disability].

For example, if you are using videos in your course, those videos may require captioning depending on the needs of the students (eg. a student who is hard of hearing or deaf). What research has found is that captioning not only benefits students who have hearing disabilities, but also assists students whose first language is not English, those who may have processing difficulties, or those who may be watching the video in a noisy area such as a bus, cafe or cafeteria where hearing it may be difficult.

For me, the accessible design of the xMOOC allowed great flexibility in when and where I studied the materials, including on my daily commute. And the captioning helped me to understand the American accents!

Overall I think that there is a lot of merit in the UDL approach, although as yet experimental evidence of the effectiveness of the technique in improving learning is somewhat limited. This is acknowledged in this review, which discusses how the technique should develop as it enters its second decade of use, and a body of research is building up, with information being collated on the UDL Center website.

My favourite suggestion from my readings on UDL was the use of a graphic syllabus to bring clarity to course structure.

Maryellen Weimer (at http://www.facultyfocus.com/articles/teaching-professor-blog/a-graphic-syllabus-can-bring-clarity-to-course-structure/) begins by describing a novel use of mind-mapping in class:

Not being a visual learner, I always struggled with ways of graphically representing course content. I was never very successful until I discovered that students could do what I couldn’t. During those summary times at the end of a class session, I often asked them to show graphically their sense of how the ideas related. I was surprised how clearly those visual representations showed whether or not they understood. Even more surprising, they sometimes depicted relationships I hadn’t thought of or positioned ideas so that they highlighted different aspects of a relationship.

The author then introduces Linda Nilson’s related idea of including a mind map in a course syllabus, as described in this presentation:

This includes several examples, and I love how they make the learning aims of the course clearer. If you only have time to look at one example, go to the final slide and see how this brings order to a very complex course. (Actually, I think this may be intended as an example of overcomplexity, but it works for me!)

Learning about learning styles

Explorer Activity 3.4 of the ocTEL course involved looking at learning styles. This was a great opportunity to catch up on the debate, and here are my findings in Storify fashion.

I started by reading the abstract of the paper “Learning Styles: Concepts and Evidence” by Pashler et al.

This introduces the topic as follows:

The term “learning styles” refers to the concept that individuals differ in regard to what mode of instruction or study is most effective for them. Proponents of learning-style assessment contend that optimal instruction requires diagnosing individuals’ learning style and tailoring instruction accordingly.

Pashler et al. looked for experimental evidence to support these claims, specifically for studies which demonstrated that

the instructional method that proves most effective for students with one learning style is not the most effective method for students with a different learning style.

They found

virtually no evidence for the interaction pattern mentioned above, which was judged to be a precondition for validating the educational applications of learning styles.

and therefore concluded that

at present, there is no adequate evidence base to justify incorporating learning-styles assessments into general educational practice. Thus, limited education resources would better be devoted to adopting other educational practices that have a strong evidence base, of which there are an increasing number. 

This is not denying that there are differences in learning style or preference, but that there is no clear benefit to be gained from evaluating this, because we can’t use the information to improve learning.

I next looked at Should we be using learning styles? What research has to say to practice by Frank Coffield, David Moseley, Elaine Hall and Kathryn Ecclestone.

A summary of this work paints a picture of the confusion surrounding the topic. For example, it raises the question, in relation to “a well known learning styles inventory”

does it really measure learning style at all, or is it actually a personality test? And even if the test does do what it says on the bottle, how should teachers respond?

One positive recommendation of the report relates to self-awareness and metacognition:

A reliable and valid instrument which measures learning styles and approaches could be used as a tool to encourage self-development, not only by diagnosing how people learn, but by showing them how to enhance their learning.

But…

One of the main aims of encouraging a metacognitive approach is to enable learners to choose the most appropriate learning strategy from a wide range of options to fit the particular task in hand; but it remains an unanswered question as to how far learning styles need to be incorporated into metacognitive approaches.

And this is not a recommendation for changing practice, because…

In a synthesis of 630 studies, Hattie (1992) [Towards a model of schooling: a synthesis of meta-analyses. Australian Journal of Education, 36, 5–13.] found an average effect size of only 0.14 for individualised teaching in schools.

In contrast, Marzano (1998) [A theory-based meta-analysis of research on Instruction. Aurora, CO: Mid-continent Regional Educational Laboratory] reported…

Interventions targeted at improving metacognition produced an average gain of 26 percentile points [0.26 effect size]

While Black and Wiliam (1998) [Assessment and classroom learning. Assessment in Education, 5(1), 7–73], reported

formative assessment experiments produce typical effect sizes of between 0.4 and 0.7.

As Pashler et al. concluded, effort is better expended on implementing changes in practice for which there is more evidence of learning gain.

Coffield et al. are positive about two instruments, and recommend that

the concepts developed by Entwistle and others, of deep, surface and strategic approaches to learning, and by Vermunt of meaning-directed, application-directed and reproduction-directed learning styles, be adopted for general use in post-16 learning.

The concepts of Entwistle and others and Vermunt relate to context and motivation, which underlines that learners can adopt different strategies or approaches at different times. This contrasts with many of the other instruments which attach labels to learners, suggesting that they are of a particular fixed type.

As “Edutopia staff” say in this article on Learning Styles and Multiple Intelligences: What Does the Research Say? – http://www.edutopia.org/multiple-intelligences-learning-styles-research

Labeling creates limits, and when it comes to learning, boundaries are the last thing we want.

A final positive recommendation from the Coffield report is that

a discussion of learning styles may prove to be the catalyst for individual, organisational or even systemic change

 

A Khan Academy Experience

A review of an online educational resource from the Khan Academy.

I started this task on my smartphone, searching within the Khan Academy Biology videos on YouTube. I chose to look at “Fun with Punnett Squares”, because my background is in Genetics so I was interested in the topic and ready for a fun-filled video. I was hoping it would be engaging, because at 25 minutes it seemed quite long.

The embedded video below provides an experience similar to that on a smartphone.

 

Unfortunately I had a few issues with the video, as follows.

1. General navigation in the Khan Academy
This was the opening screen:

image

and the lecturer began, “In the last video I drew this grid…”

This was quite a confusing image to start out with, and immediately raised the question: should I watch the previous video before starting on this one? With no structured navigation, I didn’t know how to find “the last video”, so I just carried on and hoped that things would become clear.

2. Quality – Factual errors / Lack of clarity
Next, the lecturer started talking about the derivation of “Punnett square” and said that it was named after punnets in a farmers’ market, which is incorrect; it is named after the person who developed the concept, Reginald Punnett.

Overall, I found the production poor. The lecturer did not follow a script and so some of the voiceover was a little confusing. It is difficult to be clear and accurate when discussing genetics, constantly referring to “Big B, little b”, etc, so a script would be especially helpful for this topic. It may make the lecture less engaging and spontaneous, but it would ensure that everything was described clearly and correctly.

3. Internal navigation in the video
I wanted to skip forward in the video to look at dihybrid crosses, but there were no navigation clues, such as headings, so I either had to scroll forward until I found the topic by chance or watch all the way through.

Watching the same video on the YouTube site on a PC provided some better functionality which helped with this navigation issue. (You can experience this by clicking on the YouTube icon at the bottom right of the embedded video above; this will open the video in YouTube in a new tab or window.)

Firstly there is the ability to watch the video at a faster speed, to whizz through the video when searching for a particular part. Secondly, a transcript is available, so you can scan through the text to find the part of interest and click on the text to go straight to that part of the video.

In addition, there are comments on the YouTube site which point out some of the inaccuracies in the video, for example restoring Mr Punnett to his rightful place as the inventor of the square.

Watching the video on the YouTube site on a PC provides an enhanced experience compared with watching in the embedded viewer or on a smartphone, and overcomes some of the drawbacks of the production.

Subsequently I found same video on the Khan Academy website – http://www.khanacademy.org/science/biology/heredity-and-genetics/v/punnett-square-fun

This enhances the experience still further; for example, it sets the video in context with others in the series, so it is possible to watch them in sequence.

As on YouTube, the webpage includes the ability to vary the speed of the video and view the interactive transcript, but it also includes comments, questions and answers, and the ability to report errors. On the Khan website, the video has been annotated to include details of the Punnett error (at around 0:53 in the video).

When directing students to an external resource, is it better to embed the resource within your own content (e.g. in a VLE) or to link out? There may be increased functionality on the original website, but also there may be distractions, especially on YouTube.

And how do we deal with the quality issue? I would have expected this material to be error free. Is it up to lecturers to check all material that they link to and provide corrections where necessary?

 

 

 

Is the traditional lecture outmoded?

Active Lecture Video

This video (length 2:40, no audio) incorporates my reflections on the theories of Eric Mazur, BF Skinner and xMOOC designers and some of the recent commentary in the press suggesting that traditional lectures are outmoded.

I don’t think we should be too quick to change, but rather consider the benefits and drawbacks of each method of teaching/learning and then use the most appropriate method for the context.

The video was made using a three different tools – http://www.storyboardthat.com/, Powerpoint and http://www.screencast-o-matic.com/.

It is also available at http://screencast-o-matic.com/watch/c2hYDNnaW9.

Comments very welcome. Is it too ‘Janet and John’? It looked very different in my head.

Surface and deep learning

In response to the thought-provoking thread at http://octel.alt.ac.uk/2014/forums/topic/deep-learning-is-not-the-ideal/#post-17535, here’s my understanding of surface and deep approaches to learning, using Activity 1.3 from the ocTEL course as an example.

Feedback very welcome on whether you agree with my way of describing this.

A surface learner might say…

This activity involved learning about B.F. Skinner’s Teaching Machine and several key educational thinkers and educational approaches.

I didn’t get the point of looking at the video – it was just weird – or of looking at other people’s posts about what they thought about the different theories. I just read the articles that were linked and made my own notes.

If we were to sit an exam at the end of this course, I would use flashcards to memorise the names, dates and theories of the different people and approaches.

I don’t really see how any of this relates to online learning, and I’m not sure which of the theories described is the correct one.

A deep learner might say…

This activity involved thinking about B.F. Skinner’s Teaching Machine and how this method of teaching relates to different educational approaches and the views of several key educational thinkers.

It was very interesting to explore the different theories and to think about how the advocates of these approaches would view the teaching machines. This gave a context in which to think about their theories.

I liked reading a range of views from other learners who had different experiences and viewpoints; it made me think more widely about the subject.

This also led me to think about how online courses are structured and, for example, the importance of self-pacing, peer interaction, questioning, who is the teacher?, etc.

As Rose mentioned, the ‘assessment’ – i.e. what we were asked to do in this activity – is very important. It was structured to make us look at the question deeply. This is a lot more interesting and meaningful than just reading/learning a set of descriptions of educational theories.

An online environment can help with a task of this nature by making it easy to view and comment on other people’s contributions.

Beyond deep, surface or strategic: cheating in a free online course

Beyond deep, surface or strategic: cheating in a free online course

This screenshot shows an online learner (“Customer”) in a free online course (http://www.edx.org/course/harvardx/harvardx-ph207x-health-numbers-354), purchasing answers to the final exam questions.

This prompted quite a debate in the discussion forums of the course.

What is the point in achieving a course completion certificate by cheating in this way? The certificate does not carry any credit; the only benefit of taking the course is the learning.

Revelation

The following task was set on the ocTEL course (Explorer activity 1.4. What’s the theory? – http://octel.alt.ac.uk/2014/course-materials/week-1-concepts-and-strategies-for-learning-technology/):

Here are five stories about how technology has enhanced learning. Pick two that interest you. Review the evidence we’ve provided and decide which one you think is more powerful and relevant for you.

I decided to focus on Mazur and Mitra – the clickers and ‘the hole in the wall’.

I thought I was quite familiar with the work of Eric Mazur. If called upon, I would have described his approach as:

  • taking the transmission of information out of the lecture theatre, using pre-lecture reading (an early example of flipping the classroom, but using old technology) 
  • using the class time to explore the material, using clickers

When I viewed the video, I was surprised to hear him speak about the key importance of peer instruction in his approach. So I went back and looked at some of his papers, which I’d first read many years ago, and it was all there, but I had completely forgotten the main message, that peer instruction improves learning.

This realisation was very powerful and shocking for me. I have for years devalued and misrepresented the work of Mazur by focusing on the technology rather than the pedagogy.

So, these are not stories about how technology has enhanced learning, but about how great and innovative teachers have enhanced learning, by thinking about the learning process.

And the key message for me comes from Mazur:

I often meet people who tell me they have implemented this “clicker method” in their classes, viewing my approach as simply a technological innovation. However, it is not the technology but the pedagogy that matters.

“Farewell, Lecture?” Eric Mazur, Science, 323, 50-51 (2009).
Available from – http://mazur.harvard.edu/publications.php?function=display&rowid=635

“It is not the technology but the pedagogy that matters”: even though I say this all the time, I still get it wrong.

Communication Breakdown

My personal thoughts on various channels for communication with fellow ocTEL participants – what I plan to use and why.

Twitter
I am already a Twitter user (@msars) and have a good network of TEL contacts there. I like the way that discussions can develop quickly, but the 140 character limit is certainly limiting for in-depth exploration of a topic. It is good for discovery, via the hashtags and recommendations. I will use Twitter for ocTEL to some extent, because I’m comfortable with it and it is already part of my toolset, but I need to do some curation of my followers and lists to keep things manageable.

The ocTEL forums and ocTEL groups
I think that these channels are ideal for course-specific discussions, as well as more general topics. I explored the forums first and saw that they were extremely busy. This may have been first-week enthusiasm, but I have found previously that it can be difficult to ‘get to know people’ in very active forums, and I felt that I wanted to have a more tight-knit group for discussions.

So, I took a look at the list of groups that had already been formed and, finding none that matched my interests, I set one up called ‘Scientists’. I hoped that this would provide a focused discussion area for people teaching in scientific disciplines or with a scientific bent. And things have started off well – thanks, Peter and Paul! More participants in the group are very welcome, so do come and join us, if science is your thing.

I will keep an open mind about the forums and about joining other groups as things settle. I am still trying to find my way around all the functionality in the ocTEL site, and finding things a little hard to manage, especially using multiple devices (I read something on my phone and think I’ll reply later from my desk, but then can’t find it again…), but I am sure that I’ll get to grips with it all and the forums, groups, course reader, notifications, etc will all be very useful as the course progresses.

The ocTEL JISCMail list
I subscribe to a number of mailing lists already and like the fact that the posts arrive in my email in-box, where I spend much of my working life. I therefore thought of using this mailing list as my primary medium for longer, more reflective posts. However, I was a bit put off by the warnings that this may not be a well-used communication channel. Also, I thought that, while there may be replies to a posting on a mailing list, there isn’t really any other indication that people have read or enjoyed it. Hits and click-throughs on a blog post add some welcome feedback and motivation – likewise favourites and retweets in Twitter. I have made one post to the mailing list already (with no response) and will persevere for a bit longer, but will abandon this channel if it doesn’t offer much return.

Blogging
I decided to take the plunge and start a new blog for my ocTEL reflections. I had a Posterous blog previously, but hadn’t made another since the demise of that platform. I like the fact that blogs offer the flexibility of RSS feeds, so people can access them in multiple ways, according to their preferences. I also like the commenting feature, which allows discussions to take place on the post in question, keeping everything in one place. Another benefit to me, is that I haven’t used WordPress before, so this is an opportunity to learn about a new tech tool. Incidently, I am also enjoying the diversity of blogging platforms that are in use by ocTEL participants, giving a chance to experience a wide range of options.

Other tools
I intend to get a good understanding of the communication methods above before I explore any of the many other tools in use on the course. This is partly because of time constraints, but also because, for now, I’m not entirely sure what features and functionality may be useful to me. So, I would like to get more experience of the course before looking at any other technology options. 

Do you have any suggestions for things that I should explore? If so, please let me know in the comments.

Big Q/ little Q

My primary reason for signing up for ocTEL was because I wanted to experience a cMOOC. I have tried to get along with this format before, but have always dropped out quite early, overwhelmed by the quantity of material and the unstructured nature of the ‘course’.

I suspect that this is because my existing PLN is well developed and largely meets my CPD needs. It is well focused for my interests, but I wonder if I am in danger of not hearing alternative voices/ideas.

In contrast with my previous experience of cMOOCs, I’ve really enjoyed a couple of xMOOCs that I’ve taken. On reflection, this may have been because I went into the cMOOCs in an exploratory way – let’s see what this is about – rather than with a learning aim in mind, whereas I joined the xMOOCs because I was interested in the specific content. So this time I will follow the advice “Set your goals, so that they may guide your path” and then try to be more structured in my engagement.

I have decided to take as the focus of my study in ocTEL, my ‘big question’:

In relation to technology enhanced learning in higher education, what is ‘enhanced’ and how do we know?

This question is addressed specifically in a recent paper by by Kirkwood & Price (http://dx.doi.org/doi:10.1080/17439884.2013.770404), but I think it also provides a good framework for general reflection.

And this makes my ‘small question’:

What is a cMOOC and how does it work?