Big Data and student performance: Insights from an LMS

In our recent story about Google buying BrightBytes, there was a lot of discussion about what kind of data is useful for understanding learning, and the general limits that educational data has on measuring the things that we really care about.

One area where there might be some promise is with predicting whether a student is likely to fail a course or drop out of college. A recent research paper tested whether “clickstream data” from a Learning Management System (LMS) in community colleges could outperform “administrative data” like HS GPA, previous course grades, and demographic information.

The most striking finding is that previous college course grades outperformed any “clickstream data”. This means that for any students who had previous experience in college courses, those course grades were going to be far more informative and useful than information about what the student was clicking on or where in the LMS they were spending their time.

But for students who were arriving at their first semester of college, adding the LMS “clickstream data” to the administrative data that followed the student increased the accuracy of the prediction.

These sorts of prediction models are already being used at many universities around the US, although most of them only use the “administrative data” to identify people who might be struggling or forecast failure rates and dropout rates for the campus as a whole. Introducing data from an LMS 

One of the hard parts about educational data sometimes is knowing what outcomes to optimize for, but student failure rates and dropout rates can be a useful place to start and the implication behind this data is that high school performance is a far less accurate predictor of success than college performance.

The paper does not address why HS GPAs are less useful than college GPAs. is could be a question of rigor, and in some ways that would validate the concerns around grade inflation. although that doesn’t mean that standardized tests are better at judging first year student performance. It just forces us to re-evaluate some of the ways that we prepare our students for college.

It should also be noted that the study was only done within the Virginia Community College System (VCCS) because that was the system that the researchers had a high degree of access to. So the point about HS GPAs being a poor predictor could be hiding a finding that the GPA earned within one system is a mediocre predictor of the GPA earned in another system. Since transfer rates from post-secondary institutions into community colleges are low, and often little GPA data accompanies a student who takes a community college course while also being enrolled at a 4-year institution, it is unclear whether a student in their first semester within a VCCS school could experience a performance gap irregardless of whether they had previous college experience or just high school experience.