Project Stream

In 2009, because of accidental over-recruitment, we had to put in place a system to allow streaming of material from the main biology lecture theatre to an overflow room. There were two phases to the project: first, developing and setting up the solution and then delivering the solution over the course of the academic year.

The stakeholders were the students who attended the lectures and the staff who delivered the lectures. We had to ensure that the lecture experience was acceptable for all students, both those in the lecture theatre and those in the overflow room. For staff we had to make sure that they were able to focus on delivering their lectures, minimising the disruption caused by the need for streaming.

The technical solution used LiveMeeting (a previous version of Microsoft Lync software) to stream voice and data from the lecture theatre to the overflow room. All video content which appeared on the PC was streamed; this included Powerpoint, web pages, etc. eBeam software was used to display material written on the whiteboard and ELMO software routed the visualiser image via the PC. Dedicated admin sign-ins were needed for the transmitting and receiving PCs. The local ICT support team was very helpful in helping to set up the systems and test the functionality and I delivered training and prepared all the documentation.

The most troublesome part of resourcing the project was the requirement for ongoing support on a daily basis. My time was already fully committed, and I had to make it clear that I was unable to provide support myself. This is hard to do, especially in the ‘matrix management’ situation which often applies for learning technologists – i.e. everyone thinks they are your boss and most important customer! 

In the end we recruited a part-time Streaming Assistant, who set up the equipment before each lecture and monitored the streaming in the overflow room, and an Undergraduate IT Assistant (a volunteer student from the class), who acted as a back-up and provided assistance to the lecturer if required.  I felt it was vital to have this level of support in situ because of the serious consequences if there were problems with the streaming; we would have had to cancel and rearrange the lecture, causing serious disruption to all the students and the lecturer, and a logistical and timetabling headache.

The project plan was clear and achievable, but it was very important to secure adequate resources to deliver a reliable service.

The evaluation of the project was straightforward: was the lecture available and acceptable for the students in the overflow room? It was a reliable solution and the students were happy with the quality of the presentation in the overflow. In fact, some students preferred to watch the lecture there because there was more space (especially if students were using a laptop for notetaking), better sound and video quality than at the back of the main lecture theatre and less chance of being asked a question by the lecturer.  

The results of the project were disseminated at a college Education Day. In my view, the technical details of the project were less significant that the importance of:

  • thinking about the consequences of failure and putting appropriate backup plans in place
  • making sure that staffing resources were adequate

I didn’t use any particular tools for this project, but I can see the value of tools such as a risk register or timesheets. Following a proper project planning process, with appropriate tools and techniques, would make sure that nothing was missed, and would provide documentation of the thinking behind the project plan.

Advertisements

YouTube Automatic Captioning

Did you know about the automatic captioning facility on YouTube?

This is described on the YouTube help pages as follows:

Even if you haven’t added captions to your video, YouTube may use speech recognition technology to automatically make captions available.

Sounds good!

However…

Since these are automatically generated, the quality of the captions may vary from video to video. As the video owner, you can always edit the captions to improve accuracy, or remove them from your video if you do not want them to be available for your viewers.

Hmm…

Here’s an example from the recent ocTEL webinar on assessment and feedback. (Click on the image to see the detail of the captions.)

YouTubeAutoCaptioning

Assessment: Poor.
Feedback: The automatic speech recognition requires more work.

Learning Behaviour

Are you a Viewer, Listener, Optimizer or Completionist?

These terms have been coined by researchers at MIT and Harvard to describe the behaviour of students who participated in the first wave of EDx MOOCs. Links to the research and a de-identified dataset are available in this news report.

The image below shows grades attained and chapters viewed by 53,340 participants on one of these courses, Health and Environmental Change.

Learning Behaviour

From Reich, J., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., Chuang, I., & Ho, A.D. (2014). Health in Numbers and Human Health and Global Environmental Change: 2012-2013 Harvard School of Public Health course reports (HarvardX Working Paper Series No. 2).

The categories of students identified are:

  • Completionists (at the top right of the graph) – students who viewed most of the material and achieved a high grade.
  • Optimizers (top left) – students who did enough to pass the course, but viewed only a limited amount of course material.
  • Viewers (bottom left) – students who viewed a limited amount of material and did not achieve a passing grade. These students may still have achieved all that they wanted from the course.
  • Listeners (bottom right) – students who viewed more than half of the chapters, but did not achieve a passing grade on the course. Some of these students did not engage with the formal assessment at all.

As the report states:

One of the signature features of these plots is that students can be found at nearly every possible location in the possibility space. Some students focused on earning a certificate by targeting assessment questions; some students viewed all parts of the course, eschewing all assessment; some students dabbled in various dimensions; and some students successfully completed all parts of the course.

On future courses, students will be asked about their motivation for taking the course.

Excel autograding: alternative e-assessment

This poster on Autograding describes an alternative method of e-assessment which was developed by one of my colleagues.

It is designed for assessing practical write-ups, mostly in the first year of a Life Sciences degree, where all students follow the same experimental protocol, but are required to record and analyse their own results.

The drivers for the introduction of this new assessment method were:

  • to deal with increasing numbers – from 110 to 220 students per class – meaning many more papers to mark;
  • to improve the quality of feedback, making it more extensive, more consistent and more quickly delivered.

Another benefit is that students become familiar with using Excel, which is a useful transferable skill, and they receive detailed and consistent instruction in preparing graphs and other aspects of recording and analysing data.

Students like this method of assessment because they receive lots of feedback with a quick turnaround.

Staff appreciate that the autograding process makes marking quicker, simpler and more objective.

This is a useful alternative method of assessment, but it can’t be the only assessment method used on a course. Writing up a traditional lab report is still a required skill which must be practised and assessed.

 

Declaring UDL – Universal Design for Learning

In spite of a lack of evidence that the use of learning style inventories can improve teaching and learning (see my previous post for more on this), the idea still has supporters. I think that this is because intuitively it seems to make sense. We relate to the idea that different people prefer to learn in different ways – although we may add the qualification that this is not fixed, but varies depending on context and motivation. It is difficult to understand if and how we can use these differences in teaching; maybe it’s just that learners need to be aware of the different approaches and incorporate them into their learning.

While reading about learning styles, I came across another concept, which intuitively makes sense, the idea that:

The references provided in the original website were not informative, but some googling led me to information on UDL – Universal Design for Learning – for example, the CAST organisation, which defines UDL as follows:

Universal Design for Learning is a set of principles for curriculum development that give all individuals equal opportunities to learn.

UDL provides a blueprint for creating instructional goals, methods, materials, and assessments that work for everyone–not a single, one-size-fits-all solution but rather flexible approaches that can be customized and adjusted for individual needs.

This is based on the principles of universal design in architecture and other domains, which suggest that designing for all users, including those with accessibility needs, from the outset will lead to a better result for everyone.

The use of UDL is widespread in the US and Canada, being enshrined in accessibility legislation, and there is much very good material on the topic available online. For example, I spent some time exploring an openly available UDL course for educators in Ontario.

One of the recommendations of this course resonated with me, since I greatly appreciated the multiple ways of viewing learning materials offered on an xMOOC that I undertook.

an accessible course design improves the learning opportunity for all students and significantly reduces the need for accommodation [for students with a disability].

For example, if you are using videos in your course, those videos may require captioning depending on the needs of the students (eg. a student who is hard of hearing or deaf). What research has found is that captioning not only benefits students who have hearing disabilities, but also assists students whose first language is not English, those who may have processing difficulties, or those who may be watching the video in a noisy area such as a bus, cafe or cafeteria where hearing it may be difficult.

For me, the accessible design of the xMOOC allowed great flexibility in when and where I studied the materials, including on my daily commute. And the captioning helped me to understand the American accents!

Overall I think that there is a lot of merit in the UDL approach, although as yet experimental evidence of the effectiveness of the technique in improving learning is somewhat limited. This is acknowledged in this review, which discusses how the technique should develop as it enters its second decade of use, and a body of research is building up, with information being collated on the UDL Center website.

My favourite suggestion from my readings on UDL was the use of a graphic syllabus to bring clarity to course structure.

Maryellen Weimer (at http://www.facultyfocus.com/articles/teaching-professor-blog/a-graphic-syllabus-can-bring-clarity-to-course-structure/) begins by describing a novel use of mind-mapping in class:

Not being a visual learner, I always struggled with ways of graphically representing course content. I was never very successful until I discovered that students could do what I couldn’t. During those summary times at the end of a class session, I often asked them to show graphically their sense of how the ideas related. I was surprised how clearly those visual representations showed whether or not they understood. Even more surprising, they sometimes depicted relationships I hadn’t thought of or positioned ideas so that they highlighted different aspects of a relationship.

The author then introduces Linda Nilson’s related idea of including a mind map in a course syllabus, as described in this presentation:

This includes several examples, and I love how they make the learning aims of the course clearer. If you only have time to look at one example, go to the final slide and see how this brings order to a very complex course. (Actually, I think this may be intended as an example of overcomplexity, but it works for me!)

Learning about learning styles

Explorer Activity 3.4 of the ocTEL course involved looking at learning styles. This was a great opportunity to catch up on the debate, and here are my findings in Storify fashion.

I started by reading the abstract of the paper “Learning Styles: Concepts and Evidence” by Pashler et al.

This introduces the topic as follows:

The term “learning styles” refers to the concept that individuals differ in regard to what mode of instruction or study is most effective for them. Proponents of learning-style assessment contend that optimal instruction requires diagnosing individuals’ learning style and tailoring instruction accordingly.

Pashler et al. looked for experimental evidence to support these claims, specifically for studies which demonstrated that

the instructional method that proves most effective for students with one learning style is not the most effective method for students with a different learning style.

They found

virtually no evidence for the interaction pattern mentioned above, which was judged to be a precondition for validating the educational applications of learning styles.

and therefore concluded that

at present, there is no adequate evidence base to justify incorporating learning-styles assessments into general educational practice. Thus, limited education resources would better be devoted to adopting other educational practices that have a strong evidence base, of which there are an increasing number. 

This is not denying that there are differences in learning style or preference, but that there is no clear benefit to be gained from evaluating this, because we can’t use the information to improve learning.

I next looked at Should we be using learning styles? What research has to say to practice by Frank Coffield, David Moseley, Elaine Hall and Kathryn Ecclestone.

A summary of this work paints a picture of the confusion surrounding the topic. For example, it raises the question, in relation to “a well known learning styles inventory”

does it really measure learning style at all, or is it actually a personality test? And even if the test does do what it says on the bottle, how should teachers respond?

One positive recommendation of the report relates to self-awareness and metacognition:

A reliable and valid instrument which measures learning styles and approaches could be used as a tool to encourage self-development, not only by diagnosing how people learn, but by showing them how to enhance their learning.

But…

One of the main aims of encouraging a metacognitive approach is to enable learners to choose the most appropriate learning strategy from a wide range of options to fit the particular task in hand; but it remains an unanswered question as to how far learning styles need to be incorporated into metacognitive approaches.

And this is not a recommendation for changing practice, because…

In a synthesis of 630 studies, Hattie (1992) [Towards a model of schooling: a synthesis of meta-analyses. Australian Journal of Education, 36, 5–13.] found an average effect size of only 0.14 for individualised teaching in schools.

In contrast, Marzano (1998) [A theory-based meta-analysis of research on Instruction. Aurora, CO: Mid-continent Regional Educational Laboratory] reported…

Interventions targeted at improving metacognition produced an average gain of 26 percentile points [0.26 effect size]

While Black and Wiliam (1998) [Assessment and classroom learning. Assessment in Education, 5(1), 7–73], reported

formative assessment experiments produce typical effect sizes of between 0.4 and 0.7.

As Pashler et al. concluded, effort is better expended on implementing changes in practice for which there is more evidence of learning gain.

Coffield et al. are positive about two instruments, and recommend that

the concepts developed by Entwistle and others, of deep, surface and strategic approaches to learning, and by Vermunt of meaning-directed, application-directed and reproduction-directed learning styles, be adopted for general use in post-16 learning.

The concepts of Entwistle and others and Vermunt relate to context and motivation, which underlines that learners can adopt different strategies or approaches at different times. This contrasts with many of the other instruments which attach labels to learners, suggesting that they are of a particular fixed type.

As “Edutopia staff” say in this article on Learning Styles and Multiple Intelligences: What Does the Research Say? – http://www.edutopia.org/multiple-intelligences-learning-styles-research

Labeling creates limits, and when it comes to learning, boundaries are the last thing we want.

A final positive recommendation from the Coffield report is that

a discussion of learning styles may prove to be the catalyst for individual, organisational or even systemic change

 

A Khan Academy Experience

A review of an online educational resource from the Khan Academy.

I started this task on my smartphone, searching within the Khan Academy Biology videos on YouTube. I chose to look at “Fun with Punnett Squares”, because my background is in Genetics so I was interested in the topic and ready for a fun-filled video. I was hoping it would be engaging, because at 25 minutes it seemed quite long.

The embedded video below provides an experience similar to that on a smartphone.

 

Unfortunately I had a few issues with the video, as follows.

1. General navigation in the Khan Academy
This was the opening screen:

image

and the lecturer began, “In the last video I drew this grid…”

This was quite a confusing image to start out with, and immediately raised the question: should I watch the previous video before starting on this one? With no structured navigation, I didn’t know how to find “the last video”, so I just carried on and hoped that things would become clear.

2. Quality – Factual errors / Lack of clarity
Next, the lecturer started talking about the derivation of “Punnett square” and said that it was named after punnets in a farmers’ market, which is incorrect; it is named after the person who developed the concept, Reginald Punnett.

Overall, I found the production poor. The lecturer did not follow a script and so some of the voiceover was a little confusing. It is difficult to be clear and accurate when discussing genetics, constantly referring to “Big B, little b”, etc, so a script would be especially helpful for this topic. It may make the lecture less engaging and spontaneous, but it would ensure that everything was described clearly and correctly.

3. Internal navigation in the video
I wanted to skip forward in the video to look at dihybrid crosses, but there were no navigation clues, such as headings, so I either had to scroll forward until I found the topic by chance or watch all the way through.

Watching the same video on the YouTube site on a PC provided some better functionality which helped with this navigation issue. (You can experience this by clicking on the YouTube icon at the bottom right of the embedded video above; this will open the video in YouTube in a new tab or window.)

Firstly there is the ability to watch the video at a faster speed, to whizz through the video when searching for a particular part. Secondly, a transcript is available, so you can scan through the text to find the part of interest and click on the text to go straight to that part of the video.

In addition, there are comments on the YouTube site which point out some of the inaccuracies in the video, for example restoring Mr Punnett to his rightful place as the inventor of the square.

Watching the video on the YouTube site on a PC provides an enhanced experience compared with watching in the embedded viewer or on a smartphone, and overcomes some of the drawbacks of the production.

Subsequently I found same video on the Khan Academy website – http://www.khanacademy.org/science/biology/heredity-and-genetics/v/punnett-square-fun

This enhances the experience still further; for example, it sets the video in context with others in the series, so it is possible to watch them in sequence.

As on YouTube, the webpage includes the ability to vary the speed of the video and view the interactive transcript, but it also includes comments, questions and answers, and the ability to report errors. On the Khan website, the video has been annotated to include details of the Punnett error (at around 0:53 in the video).

When directing students to an external resource, is it better to embed the resource within your own content (e.g. in a VLE) or to link out? There may be increased functionality on the original website, but also there may be distractions, especially on YouTube.

And how do we deal with the quality issue? I would have expected this material to be error free. Is it up to lecturers to check all material that they link to and provide corrections where necessary?

 

 

 

Is the traditional lecture outmoded?

Active Lecture Video

This video (length 2:40, no audio) incorporates my reflections on the theories of Eric Mazur, BF Skinner and xMOOC designers and some of the recent commentary in the press suggesting that traditional lectures are outmoded.

I don’t think we should be too quick to change, but rather consider the benefits and drawbacks of each method of teaching/learning and then use the most appropriate method for the context.

The video was made using a three different tools – http://www.storyboardthat.com/, Powerpoint and http://www.screencast-o-matic.com/.

It is also available at http://screencast-o-matic.com/watch/c2hYDNnaW9.

Comments very welcome. Is it too ‘Janet and John’? It looked very different in my head.

Surface and deep learning

In response to the thought-provoking thread at http://octel.alt.ac.uk/2014/forums/topic/deep-learning-is-not-the-ideal/#post-17535, here’s my understanding of surface and deep approaches to learning, using Activity 1.3 from the ocTEL course as an example.

Feedback very welcome on whether you agree with my way of describing this.

A surface learner might say…

This activity involved learning about B.F. Skinner’s Teaching Machine and several key educational thinkers and educational approaches.

I didn’t get the point of looking at the video – it was just weird – or of looking at other people’s posts about what they thought about the different theories. I just read the articles that were linked and made my own notes.

If we were to sit an exam at the end of this course, I would use flashcards to memorise the names, dates and theories of the different people and approaches.

I don’t really see how any of this relates to online learning, and I’m not sure which of the theories described is the correct one.

A deep learner might say…

This activity involved thinking about B.F. Skinner’s Teaching Machine and how this method of teaching relates to different educational approaches and the views of several key educational thinkers.

It was very interesting to explore the different theories and to think about how the advocates of these approaches would view the teaching machines. This gave a context in which to think about their theories.

I liked reading a range of views from other learners who had different experiences and viewpoints; it made me think more widely about the subject.

This also led me to think about how online courses are structured and, for example, the importance of self-pacing, peer interaction, questioning, who is the teacher?, etc.

As Rose mentioned, the ‘assessment’ – i.e. what we were asked to do in this activity – is very important. It was structured to make us look at the question deeply. This is a lot more interesting and meaningful than just reading/learning a set of descriptions of educational theories.

An online environment can help with a task of this nature by making it easy to view and comment on other people’s contributions.

Beyond deep, surface or strategic: cheating in a free online course

Beyond deep, surface or strategic: cheating in a free online course

This screenshot shows an online learner (“Customer”) in a free online course (http://www.edx.org/course/harvardx/harvardx-ph207x-health-numbers-354), purchasing answers to the final exam questions.

This prompted quite a debate in the discussion forums of the course.

What is the point in achieving a course completion certificate by cheating in this way? The certificate does not carry any credit; the only benefit of taking the course is the learning.