Author: Denis Avetisyan
New research demonstrates that intelligently applied AI support can significantly improve outcomes for at-risk undergraduate students.
A doubly robust evaluation, using propensity score matching and framed by activity theory, confirms the positive impact of AI-guided student support on retention and qualification progression.
Despite increasing reliance on predictive modelling in higher education, rigorous causal evidence of intervention effectiveness remains scarce. This study, ‘Stabilising Learner Trajectories: A Doubly Robust Evaluation of AI-Guided Student Support using Activity Theory’, addresses this gap by evaluating an AI-guided support system using a robust quasi-experimental design. Results demonstrate that the intervention effectively stabilised at-risk students’ academic trajectories, significantly reducing failure rates and improving grades. However, translating this stability into accelerated degree completion requires a broader alignment of institutional structures – can AI-enabled support become a catalyst for systemic change in student success?
Beyond Lagging Indicators: Towards Predictive Student Support
Conventional student early warning systems predominantly function by monitoring lagging indicators – metrics like failing grades, increased absences, or disciplinary actions – which essentially signal problems only after they have begun to significantly impact a student’s academic trajectory. This reactive approach inherently limits the potential for proactive intervention, as educators are consistently positioned to address consequences rather than prevent them. While these indicators offer valuable insight into existing difficulties, they provide little information about the underlying causes or emerging risk factors, hindering the implementation of targeted support strategies. Consequently, interventions often lack the necessary nuance to address individual student needs, leaving at-risk students to fall further behind before assistance is effectively deployed, and underscoring the need for systems that prioritize predictive, rather than purely reactive, measures.
Current student risk assessment tools often fall short by treating diverse challenges with generalized interventions. While these systems may flag a student as ‘at risk’ based on attendance or grades, they frequently fail to identify the specific barriers hindering progress – whether it’s a gap in foundational literacy, difficulty with a particular math concept, social-emotional challenges, or external factors like food insecurity. This lack of nuance results in broad-stroke support strategies – such as mandatory tutoring for all flagged students – which can be inefficient, fail to address the root cause, and even inadvertently stigmatize students who require different forms of assistance. Consequently, resources are often spread thinly, diminishing the potential for targeted, impactful interventions that truly address individual student needs and foster meaningful academic improvement.
Determining the root causes of academic difficulty presents a significant hurdle for educators. While identifying students who are falling behind – through metrics like attendance or grades – is a crucial first step, observing a correlation between these lagging indicators and poor performance doesn’t illuminate the underlying reasons for the struggle. A student’s declining grades, for example, could stem from a lack of foundational knowledge, an undiagnosed learning disability, challenging home circumstances, or even temporary factors like illness or anxiety. Without investigating these potential causal factors, interventions risk addressing symptoms rather than the core issues, leading to ineffective or misdirected support. Consequently, a nuanced understanding of why a student is struggling is paramount to designing targeted and impactful strategies that move beyond simply recognizing the presence of a problem.
Modeling Student Success with Artificial Intelligence
AI-based risk models utilize machine learning algorithms to analyze student data and identify those likely to encounter academic difficulties. This predictive capability extends beyond simple identification, offering increased lead time for intervention strategies. By assessing numerous variables – including enrollment data, course performance, and engagement metrics – these models generate risk scores allowing institutions to proactively offer targeted support such as tutoring, counseling, or adjusted learning pathways. The benefit lies in shifting from reactive problem-solving to preventative measures, ultimately improving student outcomes and retention rates through early and focused assistance.
The Probability of Success (PoS) represents a quantified assessment of a student’s anticipated completion of their enrolled qualification. This score is generated by the AI-based risk model, utilizing a range of student data points as inputs. The PoS is not a deterministic prediction, but rather a probabilistic estimate, ranging from 0 to 1, where higher values indicate a greater likelihood of successful qualification completion. The model dynamically adjusts the PoS throughout the student’s learning journey, incorporating new data and refining its assessment of their progress and potential challenges. This continuous evaluation allows for timely identification of students who may benefit from targeted interventions.
The predictive performance of our AI-based risk model was evaluated using an out-of-sample F1 score of 0.871. The F1 score is the harmonic mean of precision and recall, providing a balanced measure of the model’s accuracy in identifying at-risk students. A score of 0.871 indicates a high degree of overlap between predicted and actual at-risk classifications within the held-out dataset, demonstrating the model’s ability to generalize beyond the training data and reliably forecast student outcomes. This metric confirms the model’s strong discriminatory power and its potential for effective student support interventions.
The predictive model demonstrates a balanced performance in identifying at-risk students, as evidenced by its precision and recall scores. Precision of 0.829 indicates that 82.9% of students flagged as at-risk by the model were, in fact, genuinely at risk. Simultaneously, the model achieves a recall of 0.942, meaning it correctly identified 94.2% of all students who were ultimately at risk of not completing their qualification. These scores suggest a low rate of both false positives – incorrectly flagging students as at-risk – and false negatives – failing to identify students who require support, representing a robust and reliable predictive capability.
AI-Guided Support utilizes predictive modeling, specifically the Probability of Success (PoS) score, to proactively identify students requiring assistance. This system moves beyond reactive interventions by delivering personalized support resources – such as targeted tutoring, supplemental materials, or proactive outreach from advisors – based on an individual student’s predicted risk factors. The aim is to address academic or engagement challenges in their early stages, preventing these issues from developing into more substantial obstacles to qualification completion. By focusing on preventative measures, AI-Guided Support seeks to improve student outcomes and reduce attrition rates through timely and individualized assistance.
Establishing Causal Links: Rigorous Evaluation of AI Support
Propensity Score Matching (PSM) was implemented to address selection bias and establish a quasi-experimental framework for evaluating the impact of AI-Guided Support. This statistical technique creates comparable groups of students – those who received AI support and those who did not – by matching individuals based on their propensity score. The propensity score represents the probability of receiving AI support, conditional on observed pre-treatment characteristics. By equating these scores between groups, PSM minimizes confounding and allows for a more accurate estimation of the causal effect of the AI intervention. This approach is particularly valuable in observational studies where random assignment to treatment is not feasible, allowing us to approximate the conditions of a randomized controlled trial.
Immortal Time Bias, a common issue in observational studies evaluating interventions over time, arises when including individuals in the comparison group who have not yet experienced the possibility of receiving the intervention or failing at the outcome of interest. To mitigate this bias, our analysis restricts the comparison of AI-supported students to those who were eligible for support at the same point in time as the treated group. Specifically, we only include students in the control group who had reached the stage in their academic journey where the AI support would have been offered, effectively ensuring both groups are subject to the same time-at-risk for both receiving the intervention and experiencing the measured outcomes. This temporal restriction strengthens the validity of our causal inferences by preventing comparisons between students at different stages of their academic careers and avoids artificially inflating the observed effects of AI support.
Propensity Score Matching (PSM) was implemented with a caliper width of 0.20, a parameter determined by the standard deviation of the logit-transformed propensity scores. This caliper restricts the matching process to observations where the propensity scores differ by no more than 0.20 standard deviations, effectively reducing confounding bias. The selection of this value balances the need for sufficient overlap between treatment groups – ensuring adequate matching is possible – with the desire to minimize the inclusion of dissimilar observations. A narrower caliper improves match quality but may reduce sample size due to insufficient matches, while a wider caliper increases sample size at the cost of potentially introducing bias from poorer matches. The chosen value of 0.20 represents an empirically-justified balance, contributing to the robustness of the causal estimates.
The propensity score matching process prioritized the AI-derived Probability of Success (PoS) score as the primary matching variable. This score, representing the AI’s prediction of student success, was assigned a weight of 0.75 within the composite distance metric used to determine match quality. This weighting scheme reflects the importance of the AI prediction in creating comparable groups; the remaining 0.25 of the metric was distributed among other relevant student characteristics. By prioritizing the PoS score, the matching algorithm focused on minimizing differences in predicted outcomes between students receiving AI support and those who did not, thereby strengthening the validity of the causal inference.
To refine the propensity score matching process, both Logit Transformation and Gower Distance were implemented. Logit Transformation, applying the natural logarithm of the odds, normalizes the propensity scores, addressing potential skewness and improving the performance of distance metrics. Gower Distance, unlike Euclidean distance, accommodates both continuous and categorical covariates without requiring assumptions about data distribution or scaling. This is particularly relevant given the diverse range of student characteristics used in matching. The combination of these techniques enhances the quality of matched pairs, reducing bias and increasing the robustness of the causal effect estimation by providing a more accurate measure of similarity between students receiving and not receiving AI support.
Doubly Robust Estimation (DRE) is employed to estimate the causal effect of AI support due to its statistical properties in the presence of potential model misspecification. DRE achieves this by providing a consistent estimator even if either the propensity score model or the outcome model is incorrectly specified, provided the other model is correct. Specifically, the estimator relies on the inverse probability of treatment weighting (IPTW) and requires accurate estimation of both the propensity score – the probability of receiving AI support given observed covariates – and the conditional mean outcome. The DRE estimator is calculated as $ \sum_{i=1}^{N} \frac{Y_i \cdot I(T_i = 1)}{P(T_i = 1)} – \sum_{i=1}^{N} \frac{Y_i \cdot I(T_i = 0)}{P(T_i = 0)}$, where $Y_i$ represents the outcome for individual $i$, $T_i$ indicates treatment assignment, and $P(T_i = t)$ is the propensity score for receiving treatment $t$. This approach enhances the reliability of causal inference in observational studies by mitigating bias stemming from confounding variables and model uncertainty.
The Institutional Ecosystem: Aligning AI with Systemic Support
The efficacy of AI-guided student support is fundamentally interwoven with the existing Institutional Activity System – a complex network encompassing established roles, governing rules, and utilized tools. AI interventions do not function as isolated solutions; instead, their success hinges on seamless integration within the institution’s established workflows. Predictive algorithms, for example, are most impactful when their outputs directly inform the actions of advisors, tutors, and support staff, leveraging pre-existing communication channels and protocols. A disconnect between AI-driven insights and the capacity of the institution to respond effectively diminishes the potential benefits, highlighting the need to view AI not as a replacement for human interaction, but as a tool to enhance and streamline existing support structures. Consequently, understanding this systemic interplay is crucial for maximizing the impact of AI and fostering sustainable improvements in student outcomes.
Activity Theory offers a valuable lens through which to view the integration of artificial intelligence into established educational frameworks. This framework posits that human activity – in this case, student support – isn’t simply a series of individual actions, but a complex system comprised of interconnected components: the subject (students, staff), the object (student success), the tools (AI prediction models, communication platforms), the rules (institutional policies, data privacy regulations), a community (staff collaboration, student networks), and the division of labor (roles and responsibilities). Successful AI implementation, therefore, necessitates a thorough understanding of how these elements currently interact, identifying potential points of synergy or friction. Rather than treating AI as a standalone solution, Activity Theory emphasizes the importance of adapting existing workflows and support structures to seamlessly incorporate AI’s capabilities, ensuring it functions as an integral part of a holistic system designed to enhance student outcomes.
For artificial intelligence to truly enhance student success, its predictive capabilities must be interwoven with established proactive support systems, rather than functioning as a standalone solution. Effective implementation demands a cohesive framework where AI-driven insights directly inform interventions – such as targeted tutoring, personalized advising, or early alert systems – before students encounter significant academic challenges. This synergistic approach necessitates clear communication between the AI and support staff, ensuring that predictions are translated into actionable strategies. A fragmented system, where AI identifies at-risk students but lacks a corresponding support infrastructure, diminishes its potential impact; conversely, a well-aligned system amplifies the value of both the technology and the human expertise, fostering a more responsive and effective learning environment.
Realizing the full potential of artificial intelligence in student retention demands a thorough comprehension of the institution’s existing framework. Simply deploying predictive algorithms isn’t enough; sustained improvement hinges on how those insights integrate with established roles, resources, and procedures. A nuanced understanding of the institutional activity system – the interconnected web of policies, personnel, and tools – allows for the strategic alignment of AI-driven predictions with proactive support initiatives. This cohesive approach ensures that early alerts translate into meaningful interventions, fostering a responsive environment where students receive the right assistance at the right time. Ultimately, prioritizing this contextual awareness moves beyond short-term gains and cultivates a self-sustaining cycle of improved student outcomes and institutional effectiveness.
The study’s focus on stabilizing learner trajectories through AI-guided support resonates with a systems-level perspective on educational interventions. It acknowledges that a student’s success isn’t isolated but emerges from a complex interplay of activities and contextual factors-a core tenet of activity theory. As Vinton Cerf observed, “The Internet treats everyone the same.” This principle, though applied to network neutrality, elegantly mirrors the study’s approach: AI-driven support isn’t a one-size-fits-all solution, but rather a system designed to adapt to individual student needs within their broader learning ecosystem. A fragile intervention attempts to force a solution; a robust one, like that evaluated here, strengthens the existing system.
Where Do We Go From Here?
The demonstrated association between AI-guided support and improved student outcomes, while encouraging, merely clarifies the surface of a complex system. Propensity score matching provides a valuable corrective, yet it inherently relies on observed variables – a precarious foundation when attempting to model the motivations and circumstances of a learner. Documentation captures structure, but behavior emerges through interaction. The true challenge lies not in perfecting the algorithm, but in constructing a more complete representation of the activity system itself – one that accounts for the often-unarticulated needs and pre-existing support networks that profoundly influence a student’s trajectory.
Future work should resist the temptation to treat ‘at-risk’ as a fixed state. Instead, investigations might fruitfully explore the dynamics of risk – how interventions alter the underlying conditions that initially defined a student as vulnerable. Moreover, a singular focus on qualification progression risks obscuring the more subtle, yet potentially vital, effects of support on student agency and self-efficacy. A system designed to merely ‘retain’ may, ironically, diminish the very qualities it seeks to nurture.
Ultimately, the field must confront the uncomfortable truth that predictive models, however sophisticated, are always approximations. The elegance of a solution is not measured by its ability to predict, but by its capacity to adapt – to learn, not just from data, but from the unexpected consequences of its own interventions. The goal is not to control learner trajectories, but to create a more responsive and equitable learning environment – a space where success is not predetermined, but emerges from the complex interplay of support, agency, and opportunity.
Original article: https://arxiv.org/pdf/2512.11154.pdf
Contact the author: https://www.linkedin.com/in/avetisyan/
See also:
- Super Animal Royale: All Mole Transportation Network Locations Guide
- Shiba Inu’s Rollercoaster: Will It Rise or Waddle to the Bottom?
- The best Five Nights at Freddy’s 2 Easter egg solves a decade old mystery
- Zerowake GATES : BL RPG Tier List (November 2025)
- Avengers: Doomsday Trailer Leak Has Made Its Way Online
- Daisy Ridley to Lead Pierre Morel’s Action-Thriller ‘The Good Samaritan’
- Pokemon Theme Park Has Strict Health Restrictions for Guest Entry
- Wuthering Waves version 3.0 update ‘We Who See the Stars’ launches December 25
- xQc blames “AI controversy” for Arc Raiders snub at The Game Awards
- LINK PREDICTION. LINK cryptocurrency
2025-12-16 02:19