Critics of Massive Open Online Courses (MOOCs) have long raised concerns about low overall completion rates by learners. A new study in the December 8 issue of Educause Review Online (ERO) by Harvard University researcher Justin Reich reveals that these rates, hovering between 2 and 10 percent, often do not account for one crucial factor: student intentions.
It turns out that completion rates among students intending to complete a course are significantly higher than those for students intending to audit or browse a course. The finding suggests that a more nuanced view of the definitions for success in the evolving online learning space is necessary, taking into account how and why learners engage with MOOCs.
“We know from prior studies that when it comes to MOOCs, students engage for a wide variety of reasons,” said Reich, the Menschel Research Fellow at HarvardX, the University-wide strategic initiative focused on open online learning experiences and research. “With that in mind, we set out to understand what specific factors might be influencing completion rates in a more systematic way.”
MOOC completion rates are conventionally calculated as the number of certificate earners (those who successfully met the completion criteria of a course) divided by the total number of people who registered for a course. In the case of HarvardX courses the average completion rate is around 6 percent, on par with the overall “low” MOOC completion number often cited by the media.
Prior data analysis on HarvardX learners showed that many sign up for courses without ever planning to complete them and, about half the time, without even logging in. The research team for that study, including Reich, concluded that completion rates should not necessarily be considered as a metric of success for MOOCs and recommended that more work be done to understand those who did and did not finish courses.
To build on that work and delve into the “why” of learner behavior, Reich’s Educause paper used survey and log data from nine HarvardX open online courses offered on the edX learning platform during 2013-14. The study investigated how completion and attrition rates differed based on learners’ self-reported intentions towards course participation.
Based upon a pre-course survey of HarvardX learners in those courses, 58 percent reported that they intended to earn a certificate, 25 percent that they intended to audit, 14 percent that they were unsure of their intentions, and 3 percent that they intended to browse.
“When intention is put into the mix, the picture of MOOC completion rates starts to become more complex, and more fluid,” Reich said. “Across these courses, on average 22 percent of learners who said they intended to complete a course earned a certificate, and these students are estimated to be 4.5 times more likely to earn a certificate than students who only intended to browse a course.”
Moreover, the median lifetime of intended-completers—the time at which half of students have stopped participating—was 35 percent of a course, compared with a median lifetime of 10 percent of a course for intended-browsers.
During the run of a course a learners’ intentions may also end up changing, as initial browsers may end up becoming so engaged that they end up completing the course. With this in mind, Reich came up with several key takeaways for use by course developers and course providers to gauge success.
A New Benchmark for Completion Rates: Those reporting MOOC completion rates should factor in the percentage of students who intended to complete a course and actually went on to do so. Other MOOC course teams may find the HarvardX completion study a useful benchmark, in the context of all the particular details of every unique learning experience, to help characterize the success of a course.
Learner Intentions Can Change: On average across the HarvardX courses in the study, 6 percent of intended-browsers, 7.5 percent of intended-auditors, and 10 percent of students with unsure commitments to the course earned a certificate. Some of the people who did not initially intend to complete a course were convinced to do so. These “intention flips” may be a greater indicator of course success that the students who inevitably attrite.
Attrition Happens Early: Course Beginnings are Important: Regardless of a student’s stated intentions, attrition rates are highest in the early part of the course. Course developers should recognize that for many students, the first unit of the course is the only part they will see. Course teams should consider allocating resources to making that beginning unit inviting and compelling.
“More broadly, efforts to personalize MOOCs based on self-reported intentions should be conducted with care,” said Reich. “Many students who do not intend to complete a MOOC end up doing so, and the majority of students who intend to complete a MOOC are not successful.”
“While just one study, it does suggest that we cannot judge or determine success with MOOCs by relying upon older classroom models or without really digging into what learners bring to the table.”