Increasing course completion rates by 41% with learner-aware AI
A B2C professional-skills learning platform (~180k MAU, engagement 2023) where the previous completion-prediction model had decent held-out accuracy but was useless in practice, it flagged at-risk learners on day 28 of a 30-day course, leaving no time to intervene. Our three-component architecture, a learner state model (knowledge graph + item response theory), an early-warning system tuned to leading indicators (session cadence, replay rate, hint usage) rather than lagging outcome data, and an intervention recommendation engine, now flags at-risk learners by day 5. Instructors respond to 82% of recommendations; completion rates +41% on flagged-and-intervened cohorts vs. control.
+41%
Completion rate
3wk
Earlier at-risk detection
82%
Intervention response rate
The Challenge
The platform had invested heavily in new content while completion rates continued declining. A previous "completion prediction model" had decent held-out accuracy but was useless in practice: it flagged learners at day 28 of a 30-day course, leaving no time to intervene.
Our Approach
The AI Readiness Audit identified three core issues: (1) engagement data existed in silos across LMS, email, and app; (2) the previous model used outcome data that arrived too late; (3) content difficulty metadata was missing.
Discovery & Blueprint produced a three-component architecture: a learner state model (knowledge graph + item response theory), an early warning system using leading indicators tuned to flag at-risk learners by day 5, and an intervention recommendation engine for instructors.
Outcome
Completion rates increased 41% on flagged-and-intervened cohorts vs. control. The early warning system surfaces at-risk learners 3 weeks earlier. Instructors respond to 82% of intervention recommendations. The content difficulty calibration layer became a standalone product feature.
What We Learned
01
Predictive accuracy is not the same as actionability.
02
Leading indicators (session patterns, replay rate) outperform lagging outcome data for early warning.
03
Content metadata quality is a prerequisite for any learning AI.
Stages Engaged
AI Readiness Audit
Discovery & Blueprint
Concept Validation
Production Build
Total Duration
6 months total
Artifacts Delivered
PRD
Learner State Model Spec
Early Warning Architecture
WBS
Instructor UX Brief
Start with a Feasibility Call
2 hours. No cost. We'll tell you honestly whether AI makes sense for your case.