Enhancing Student Performance Using AI: Ethical Issues

In 2013, University ABC, with nearly 3,000 students and 180 classroom instructors, reached a
depressing milestone. That year, the student dropout rate hit its highest point ever at nine percent.
This meant that nearly one out of every ten students who entered the school as a freshman left
without graduating at all. In addition, only 55 percent of the students were able to graduate on
time, compared to the state average of 76.4 percent. The school board responded to these dismal
statistics by demanding that the Provost, Dr. Johnson, address this growing problem, or risk the
loss of funding – even threatening a possible shutdown. Dr. Johnson feared for the University’s
future, but he was at a loss. Students were habitually disengaged, and instructors had been
demotivated by frequently shifting incentives, the imposition of new approaches and increasingly
negative assessments.
Dr. Johnson met with school board members who suggested he put the vast and varied datasets
the school had already collected about its students’ behavior to use. In addition to the typical
performance, disciplinary and attendance records, the school had given students scannable ID
cards that registered when they were used to open the library doors, confirm attendance in class,
purchase snacks or lunches, and so forth. Even the school’s Wi-Fi network tracked students’
internet use and monitored their movements throughout the campus with an impressive degree of
accuracy, using their mobile phones as proxies. These measures produced large quantities of
data. The school board members suggested that developments in data science and machine
learning could be applied to this information in order to shed light on the causes of what
appeared to be an irreversible trend towards high dropout rates. Understanding what causes
students to drop out might suggest appropriate interventions and could inform the creation of
new incentive structures for instructors and students.
Dr. Johnson took these suggestions to heart and contracted a local data science company,
DATAVOLVED, which promised insights into business processes through novel approaches
using artificial intelligence. Together they formulated these specific goals:
1. To identify predictors of student disengagement as an indicator for dropping out, and to
apply machine learning tools to these predictors in order to flag at-risk students;
2. To equip instructors with fine-grained information to allocate resources and assist at-risk
students by suggesting specific interventions, such as talking to the student directly,
adjusting their workload and schedule, or meeting with the student’s counselor; and
3. To be transparent in the way the system is utilized.
Dr. Johnson and the school board agreed to provide DATAVOLVED with their existing
databases, spanning several years, and gave them access to new data as it was collected. Students
were not notified of this agreement, nor were they given the opportunity to opt out. Knowing
from previous experience how difficult it is to get students and administrators to come to a
consensus on any new initiative, and given the urgency of the situation, Dr. Johnson believed this
was for the best. Besides, he argued, this decision was supported by the school board and fell
within his general mandate to promote positive educational outcomes for all.

Adapted from: UCHV & CITP (Princeton) – AI Ethics Cases.

Discussion Question #1:
How should decisions to adopt AI technologies be made? In this case, it came from above –
a suggestion from the school board, implemented by the Provost. Who are the other
relevant stakeholders? Should they have been involved in the decision to utilize
DATAVOLVED? To what extent? How does the decision not to include an opt-out option
affect the legitimacy of the school’s actions?
Upon receipt of the student data, DATAVOLVED began with a broad policy of data analysis
looking at a large number of predictors, ranging from various student demographics (e.g. race,
ethnicity, gender, mobility, address, home life) to academic factors (e.g. grades, GPA, test
results, history of disciplinary action, attendance) to instructor statistics (e.g. certifications,
degrees, percent of students failing per class, years of teaching). DATAVOLVED then harvested
new data for a full academic year, allowing its machines to correlate the data of students that
previously dropped out with information about current students in order to recognize patterns.
Ultimately, DATAVOLVED was able to produce its own synthetic data by generating inferences
that would not have been possible without inputting the University’s original data into its
algorithms.
Discussion Question #2:
Did the school violate the privacy of its students by sharing their data with
DATAVOLVED? If so, is this breach justifiable, and on what grounds? Use
consequentialism and deontology to inform your analysis.
Using all this data, DATAVOLVED was able to identify eight key indicators that, in
combination, predicted whether a student would drop out with 92 percent accuracy. The reasons
ranged from predictable administrative issues (e.g. an overly ambitious or unsuitable
combination of chosen courses, course scheduling) to external factors (e.g. balancing a job with
school, domestic responsibilities), as well as previously unconsidered factors. DATAVOLVED
then supplied instructors with profiles of at-risk students, which in turn helped them better
understand why an individual student might be struggling and suggested targeted approaches for
helping him or her. These treatment protocols included actions, such as tutoring, modifying
assignments, and talking with the students.
Some instructors readily followed the recommendations made by DATAVOLVED, and there
was an immediate boost in student engagement. At the same time, the administration used the
information provided by DATAVOLVED to adjust certain aspects of the campus environment in
order to nudge students towards better behavior. For example, students were encouraged to
change some of their courses to make their schedules more manageable. All instructors were also
instructed to pay closer attention to struggling students, rather than the students who were
already expected to do well.
By the end of the 2016-17 academic year, University ABC appeared to have made an impressive
turnaround. The graduation rate had risen from 55 percent to 85 percent – a praiseworthy
increase, which was distinctly higher than the district’s average. The dropout rate similarly

Adapted from: UCHV & CITP (Princeton) – AI Ethics Cases.

improved, from nine percent to only five percent. Nearly all students now left school with their
diplomas. In an interview with a local news reporter, Dr. Johnson praised the work done by
DATAVOLVED, which he argued was instrumental to this improvement in student outcomes.
Discussion Question #3:
How might we define a successful outcome for University ABC? What was the school
hoping to optimize for by contracting DATAVOLVED? To what extent is this end
legitimate? In order to achieve this end, did DATAVOLVED ask the right questions and
use the right proxies?
Discussion Question #4:
Graduation statistics did, indeed, improve after DATAVOLVED came on the scene;
however, correlation doesn’t necessarily imply causation. What are some alternative
explanations for the improvement in dropout rates?
The seeming success of DATAVOLVED’s approach was overshadowed to some extent by
concerns raised by students when they were finally told about DATAVOLVED’s involvement.
Not only did they learn that DATAVOLVED had been using student data to make its
recommendations, but they were informed that the school’s data collection and use policies were
going to continue into the foreseeable future in order to maintain the system’s accuracy.
Effective as DATAVOLVED had been in improving educational outcomes, this decision,
delivered from above, made many students and their families uncomfortable. Emboldened by
their critiques, some instructors and administrators also began to publicly voice criticisms of and
opposition to the new system.
In addition to these ethical considerations, the University was criticized for over-enthusiasm
about using artificial intelligence as a way to modernize education. Several instructors argued
that the machines were just telling administrators the same things they had been saying for years.
Working on the frontlines, professors often know why individual students are failing and what to
do about it. They just rarely have sufficient institutional support to make the necessary changes.
Thus, these instructors attributed much of the improvement in graduation statistics after the
introduction of DATAVOLVED merely to the “Consultant Effect.”
One of the school’s statistics instructors had a slightly different take, but similarly questioned the
“wisdom” of AI systems such as DATAVOLVED. She wrote an op-ed in the local newspaper,
describing her concern that the new system was simply rehashing statistical methods, rather than
using a new form of intelligence. The instructor explained that all the flaws, limits and biases
that are well-known in statistics were being swept under the rug by rebranding the system as
artificial intelligence, thereby “blackboxing” the processes. She argued that blind faith in such an
unchecked system of statistical governance may lead to the kinds of long-term problems that still
plague the field of statistics.

Adapted from: UCHV & CITP (Princeton) – AI Ethics Cases.

Discussion Question #5:
The rhetorical decision to call a technology “AI” imbues it with a certain mystique. But,
quantitative and statistical methods, such as those used in many AI systems (including
DATAVOLVED) inherently involve generalizations. While there is value in using statistics
to understand social problems and make predictions, the methodologies may not be useful
on an individual basis. What is the danger of calling a system that deals in particulars
“AI”? What are the advantages?
Representatives from the University and DATAVOLVED met with concerned students, families,
and faculty to respond to their worries about the new system. While the school admitted to
handing over vast amounts of data, they considered the privacy concerns unfounded. First, the
databases were all pseudonymized and no record was being kept of the link between data points
and students’ identities in the raw dataset. Second, even if the data had been identifiable, they
argued that they did not need to seek consent from students because the data was collected for
the legitimate purpose of improving educational outcomes.
For its part, DATAVOLVED resisted calls to release its proprietary algorithms. Representatives
from the company argued that the source code would be meaningless to the public. The original
code was developed as a means to enable DATAVOLVED’s algorithms to learn from patterns
and correlations in the existing data, but the code had long since evolved into complex neural
networks with millions or even billions of nodes. It would be impossible for DATAVOLVED (or
any other machine learning experts) to adequately interpret the current functioning of such a
system. However, DATAVOLVED did provide some tools that would allow students to
understand which data points were used to recommend a particular course of action. Given the
state of the art of such technology, this was the best they could feasibly do.
DATAVOLVED’s representatives agreed with the statistics instructor that their system was
largely based on statistical methods. However, they noted that the system makes use of some
recent developments in the field of machine learning, which is both a subfield of artificial
intelligence and predictive statistics. Since the term “artificial intelligence” is more commonly
used, they decided to use that labeling, while making sure to correct their processes and
calculations for the known limitations in the field. They did not intend to mislead.

Adapted from: UCHV & CITP (Princeton) – AI Ethics Cases.