Abstract

In the new era of big educational data, learning analytics (LA) offer the
possibility of implementing real–time assessment and feedback systems
and processes at scale that are focused on improvement of learning,
development of self–regulated learning skills, and student success. How-
ever, to realize this promise, the necessary shifts in the culture, techno-
logical infrastructure, and teaching practices of higher education, from
assessment–for–accountability to assessment–for–learning, cannot be
achieved through piecemeal implementation of new tools. We propose
here that the challenge of successful institutional change for learning
analytics implementation is a wicked problem that calls for new adap-
tive forms of leadership, collaboration, policy development and strategic
planning. Higher education institutions are best viewed as complex sys-
tems underpinned by policy, and we introduce two policy and planning
AUTHORS
frameworks developed for complex systems that may offer institutional
Leah P. Macfadyen
teams practical guidance in their project of optimizing their educational
The University of
systems with learning analytics.
British Columbia

Shane Dawson
University of South Australia

Abelardo Pardo
The University of Sydney

Dragan Gašević
Athabasca University

Embracing Big Data in Complex

Educational Systems: The Learning Analytics
Imperative and the Policy Challenge

In education, we are awash in data about our learners and educators, our
technologies and activities, achievements and performance. To date these data have
rarely been mined intelligently with the goal of improving learning and informing teaching
practice, although evidence from other sectors such as marketing, sports, retail, health
and technology suggests that the effective use of big data can offer the education sector
the potential to enhance its systems and outcomes (Manyika et al., 2011). Norris and Baer
(2013) have noted that, “Data expands the capacity and ability of organizations to make
sense of complex environments” (p. 13). In this context, learning analytics (LA) offers
the capacity to investigate the rising tide of learner data with the goal of understanding
the activities and behaviors associated with effective learning, and to leverage this
knowledge in optimizing our educational systems (Bienkowski, Feng, & Means, 2012;
Campbell, DeBlois, & Oblinger, 2007). Indeed, in a world of larger and larger data sets,
increasing populations of increasingly diverse learners, constrained education budgets
and greater focus on quality and accountability (Macfadyen & Dawson, 2012), some argue
CORRESPONDENCE that using analytics to optimize learning environments is no longer an option but an
imperative. The value of such analytics is highlighted by the authors of the McKinsey
Email Global Institute (Manyika et al., 2011) noting that, “In a big data world, a competitor that
[email protected] fails to sufficiently develop its capabilities will be left behind…Early movers that secure
access to the data necessary to create value are likely to reap the most benefit” (p. 6).
Education can no longer afford not to use learning analytics. As Slade and Prinsloo (2013)
maintain, “Ignoring information that might actively help to pursue an institution’s goals
seems shortsighted to the extreme” (p. 1521).

Volume Nine | Winter 2014 17

 In this article we consider ways in which learning analytics can support and contribute
to the development of new approaches to the assessment of learning, and the degree to
which new adaptive policy and planning approaches will be needed to bring about the kind
of institutional change such proposals demand. We emphasize that successful institutional
adoption demands comprehensive development and implementation of policies to address
LA challenges of learning design, leadership, institutional culture, data access and security,
data privacy and ethical dilemmas, technology infrastructure, and a demonstrable gap in
institutional LA skills and capacity (Siemens, Dawson, & Lynch, 2013). Moreover, we take the
position that educational institutions are complex adaptive systems (Gupta & Anish, 2009;
MacLennan, 2007; Mitleton–Kelly, 2003), and therefore that simplistic approaches to policy
development are doomed to fail. Instead, we will introduce strategy and policy frameworks and
approaches developed for complex systems, including frameworks that offer the potential to
identify points of intervention (Corvalán, Kjellström, & Smith, 1999), with the goal of offering
educational institutions practical guidance.

Assessment Practices: A Wicked Problem in a Complex System

Indeed, in a world There is no better exemplar in higher education than assessment to demonstrate
of larger and larger how institutional policy can impact practice both positively and negatively. The practice
data sets, increasing of assessment has for some time been mired in debate over its role as either a measure of
populations of increas- accountability or a process for learning improvement. While the majority of education
ingly diverse learners, practitioners lean towards assessment as a process for improving student learning, assessment
constrained education nonetheless remains firmly positioned as an important tool for determining accountability
budgets and greater and demonstrating quality. As McDonnell (1994) previously argued, assessment policies
focus on quality and function as a mechanism to provide government with a high level of influence over classroom
accountability, some practice. In essence, assessment acts as a powerful tool to manage aspects of learning and
argue that using analyt- teaching. It is not surprising, then, that assessment policy has numerous invested stakeholders
ics to optimize learn- – learners, educators, administrators and government – all vying for a larger stake in the game.
ing environments is no The diversity of stakeholders, priorities, outcomes and needs make any substantial change to
longer an option but an assessment policy and practice a considerable challenge to say the least.
imperative.
Assessment practice will continue to be intricately intertwined both with learning
and with program accreditation and accountability measures. Such interconnectedness in
educational systems means that narrow efforts to implement changes in policy and practice
in one area (for example, by introducing new approaches to tracking and measuring learning)
may have unanticipated consequences elsewhere in the system. For example, the US
education policy No Child Left Behind drastically reshaped not only the testing processes
employed to identify poor literacy and numeracy standards, but also affected what was taught
and how it was taught. Jacob (2005) documented the unintentional outcomes of this new
accountability policy classroom practice, noting, for example that such high–stakes testing
encouraged teachers to steer low–performing students away from subjects that were included
in the accountability program. While the ethos of the policy had some merit in attempting
to address declining numeracy and literacy skills in the US, the associated incentives and
measures resulted in crossed performance indicators. Dworkin (2005) also expands on this
point, noting that teacher promotion standards were linked to class performance in the high
stakes tests. This practice essentially encouraged teachers to narrow the curriculum and teach
to the test, beautifully illustrating Goodhart’s Law, which states that when a measure becomes
a target it ceases to be a useful measure (Elton, 2004).
In the complex systems of higher education, current performance assessment and
accountability policies may be the forces driving (Corvalán et al., 1999) the continued focus
on high–stakes snapshot testing as a means of producing comparative institutional data, in
spite of the well–articulated weakness of such an approach for understanding student learning.
The continuing primary use of grades in determining entry to university, the Australian
Government’s National Assessment Plan for Literacy and Numeracy (NAPLAN)1 measures,
the OECD’s Programme for International Student Assessment (PISA)2 and similar programs,
demonstrate that there is much invested in the retention of these measures for benchmarking
individuals, schools, districts, states and countries. Wall, Hursh and Rodgers (2014) have
1
http://education.qld.gov.au/naplan/
2
http:/www.oecd.org/pisa/
18 Volume Nine | Winter 2014
argued, on the other hand, that the perception that students, parents and educational leaders
can only obtain useful comparative information about learning from systematized assessment
is a false one. Instead, alternate complementary assessment practices – practices that make
better use of the rich array of educational data now available – may well offer more effective
approaches to improving learning, especially processes that reveal development of student
understanding over time (Wiliam, 2010).
In his criticism of assessment practices, Angelo (1999) suggested that as educators
we must emphasize assessment as a means for improving student learning rather than a
mechanistic, technical process used to monitor performance. He argued that assessing for
learning necessitates a focus on developing practices that help the educator and learner gather
evidence of learning progress, rather than on identifying the students that produce the “right”
or “wrong” answers. The importance of developing better formative or embedded assessment
models has also been reiterated by the OECD Innovative learning environments project
(Dumont, Istance, & Benavides, 2010) and educational researchers have similarly illuminated
that regular feedback at the process level is more effective for enhancing deeper learning (for
review, see Hattie & Timperley, 2007).
Despite the widespread recognition of the need for a more effective assessment
paradigm, implementation is a challenge, and calls for development of new policies and
implementation strategies directed at improving accountability for learning though
practices driven by learning. Differentiating assessment–for–learning from assessment–for–
accountability within the educational system forms part of the wicked problem of institutional
change in higher education that we seek to explore here. As with all complex systems, even
a subtle change may be perceived as difficult, and be resisted (Head & Alford, 2013). For
example, under normal classroom circumstances the use of assessment at the process level
for improving learning requires substantial and sustained engagement between the educator
and students and can be an extremely time intensive process. Implementing such time
intensive assessment models for large (and growing) university classes is not feasible, and
typically scalable models of assessment such as multiple choice exams are implemented
instead. It is unrealistic to consider that educators will adopt time–consuming longitudinal
and personalized assessment models given the massive increase in resources and workload
that would be required.

Learning Analytics and Assessment–for–Learning

A wide range of authors in this special issue illustrate ways in which learning There is no better
analytics – which comes with its own set of implementation challenges and hurdles – exemplar in higher
has the potential to provide learners with sustained, substantial and timely feedback to education than
aid understanding and improve student learning skills, while circumventing the challenge of assessment to
educator workload. We also offer a discussion of how learning analytics may support development demonstrate how
of self–regulated learning in Box 1, inset. Analytics can add distinct value to teaching and institutional policy
learning practice by providing greater insight into the student learning process to identify the can impact practice
impact of curriculum and learning strategies, while at the same time facilitating individual both positively and
learner progress. Nor does the adoption of learning analytics preclude traditional or alternate negatively. The practice
assessment practices that may be required by accreditation and accountability policies. While of assessment has for
current assessment policy may be driven by conflicting intentions – accountability and quality some time been mired
assurance requirements versus promotion of student learning – learning analytics can meet in debate over its role
both. More simply put, LA addresses the need for quality assurance and learning improvement. as either a measure
of accountability or a
Technological Components of the Educational System process for learning
and Support of Learning Analytics improvement.
The LA–supported approaches to assessment of learning envisioned in this
article – indeed, in this entire edition – assumes a technological layer that is capable of
capturing, storing, managing, visualizing and processing big educational data – the millions
of events occurring in diverse learning scenarios and platforms. Transformation of assessment
practices to embrace and integrate learning analytics tools and strategies in support of
teaching and learning therefore demands effective institutional technology infrastructures.
The production of data in every technology–mediated interaction occurring in a learning
environment, the need for more effective provision of feedback, and the need for more
Volume Nine | Winter 2014 19
 Box 1
Learning Analytics for Assessing Student Learning

Differentiating assess-
ment–for–learning from Provision (to learners and educators) of automated analytics that provide feedback
assessment–for–account- on learner study behaviors, progress and outcomes will not only enhance academic
ability within the educa- performance but also develop student self–regulated learning (SRL) skills, and SRL
tional system forms part proficiency has been demonstrated to be a significant predictor of academic success
of the wicked problem (e.g., Butler & Winne, 1995; Pintrich, 1999; Zimmerman, 2002). Student motivation
of institutional change in
and capacity to undertake accurate self–monitoring had a direct impact on the level
higher education that we
and quality of their study and therefore, their overall learning progression and academic
seek to explore here.
achievement (Dunlosky & Thiede, 1998). Conversely, poor performers are poor at
evaluating their own ability or judging their own learning skills (Kruger & Dunning,
1999). For these reasons, it is argued that a core goal of any effective pedagogical
strategy must include the development of student meta–cognitive skills or judgment of
(own) learning (JOL). Feedback on assessment is one key approach that is often adopted
to assist students in developing meta–cognitive skills, but because provision of relevant
feedback can be labor–intensive, it is often delayed and provided at a time when it is no
longer useful to the student to aid their learning.
Recent research posits that SRL is a process of temporal events that evolve during
learning (Azevedo & Aleven, 2013). This research, alongside recent developments in
learning analytics, data mining and machine learning is providing new methods for
developing novel insights into student learning processes. Historically, assessment
and development of student SRL has made use of tasks associated with JOL which
generally involve asking a student to assess how effectively they have understood a
particular concept (Dunlosky & Lipko, 2007). This self–reported rating is then correlated
against their overall test performance to gain insight into the student’s meta–cognitive
proficiency. While JOL has commonly relied on self–report methodologies such as think
aloud protocols and surveys, these have inherent limitations such as poor recall, and
biased responses (Richardson, 2004).
New options for assessing student learning behaviors are emerging as a result of advances
in learning analytics and natural language processing (NLP), and alternate models have
sought to capture actual learner behavior (in lieu of self–reports) from interactions with
technology–based learning activities. For example, oft–cited SRL researcher Phil Winne
has previously reported that student online interaction data can provide significant
indicators of SRL proficiency (e.g., Winne, 2010; Zhou & Winne, 2012). Winne has
developed the software application nStudy as a web tool that can collect very fine
grained, time stamped data about individual learner interactions with online study
materials. The trace data is then used to provide insight and feedback into the learner’s
cognitive choices as they interact with the online information. Essentially, the tool
makes data for reflection available to both the individual learner and the educator.

comprehensive formative and summative assessment translates into a rich set of requirements
of the current technological infrastructures. Although learning management systems (LMSs)
still host a large percentage of technology–mediated educational activities, educational
institutions are recognizing the need to re–assess the concept of teaching and learning space
to encompass both physical and virtual locations, and adapt learning experiences to this new
context (Thomas, 2010). Thus, together with the need for cultural change and a focus on
pedagogical relevance, an additional sociotechnical factor critical to the adoption of learning
analytics is technology itself (Box 2).
The evolution of technology in recent years offers an unprecedented capacity to store
large data sets, and applications using big data are well established in contexts such as business
intelligence, marketing and scientific research (Dillon, Wu, & Chang, 2010). Education faces a
particular challenge that derives from the rich variety of technological affordances emerging in
20 Volume Nine | Winter 2014
learning environments. From an LMS–centric approach consolidated in the early 2000s, we are
now entering an era in which learning may occur anywhere, at any time, with multiple devices,
over a highly heterogeneous collection of resources, and through multiple types of interactions. In
this new scenario, learning analytics tools need to comply with requirements in the following areas:
1. Diverse and flexible data collection schemes: Tools need to adapt to increasing
data sources, distributed in location, different in scope, and hosted in any platform.
2. Simple connection with institutional objectives at different levels:
information needs to be understood by stakeholders with no extra effort. Upper
management needs insight connected with different organizational aspects
than an educator. User–guided design is of the utmost importance in this area.
3. Simple deployment of effective interventions, and an integrated and
sustained overall refinement procedure allowing reflection.
From the technological point of view, learning analytics is an emerging discipline
(Siemens, 2013) and its connection with assessment remains largely unexplored (Ellis, 2013).
This situation is even more extreme when considering the assessment of competences and
learning dispositions (Buckingham Shum, 2012). Educational institutions need technological

Box 2
Sociotechnical Infrastructure Needs for Effective Learning Analytics

It is unrealistic to
Several initiatives are already tackling the problem of flexible data collection schemes. consider that educators
For example the ADL Experience API3 released in 2013 has been proposed as a solution will adopt time–
that can promote interoperability between data collected in different environments consuming longitudinal
and platforms. The interface offers the possibility of capturing a wide variety of and personalized
assessment models given
events in experiences with heterogeneous scenarios (Glahn, 2013). Similarly, the
the massive increase in
IMS Global Consortium has proposed that the Learning Measurement Framework
resources and workload
IMS Caliper4 – containing descriptions to represent metrics, sensor API and learning
that would be required.
events – will facilitate the representation and processing of big data in the learning
field. In parallel, the concept of a Learning Record Store (LRS) has been proposed as
a framework for storing and manipulating data from distributed events in a learning
environment, encoding not only interaction among stakeholders, but among resources.
This information is then made available through a service–based interface to other
systems within an institution (or across multiple institutions) for further analysis and
processing.
Numerous attempts have been made to meet diverse stakeholder reporting and data
access needs by production of so–called dashboards that show a canvas of multiple
visualizations. Common limitation of these graphical representations, however, are their
actual utility and usability (Verbert, Duval, Klerkx, Govaerts, & Santos, 2013). Adapting
presentation of information to user context, needs and interests is another important
factor that must be taken into account if we wish to facilitate the uptake of learning
analytics solutions.
The third requirement for technology supporting learning analytics is that it can
facilitate the deployment of so–called interventions, where intervention may mean any
change or personalization introduced in the environment to support student success,
and its relevance with respect to the context. This context may range from generic
institutional policies, to pedagogical strategy in a course. Interventions at the level of
institution have been already studied and deployed to address retention, attrition or
graduation rate problems (Ferguson, 2012; Fritz, 2011; Tanes, Arnold, King, & Remnet,
2011). More comprehensive frameworks that widen the scope of interventions and
adopt a more formal approach have been recently proposed, but much research is still
needed in this area (Wise, 2014).

3
http://www.adlnet.gov/tla
4
http:/www.imsglobal.org/IMSLearningAnalyticsWP.pdf Volume Nine | Winter 2014 21
 solutions that are deployed in a context of continuous change, with an increasing variety of data
sources, that convey the advantages in a simple way to stakeholders, and allow a connection
with the underpinning pedagogical strategies.
In turn, these technological requirements point to a number of other critical contextual
factors that must form part of any meaningful policy and planning framework for employing
learning analytics in service of improved assessment. Foremost among these is the question
of access to data, which needs must be widespread and open. Careful policy development is
also necessary to ensure that assessment and analytics plans reflect the institution’s vision
for teaching and strategic needs (and are not simply being embraced in a panic to be seen to
be doing something with data), and that LA tools and approaches are embraced as a means of
engaging stakeholders in discussion and facilitating change rather than as tools for measuring
performance or the status quo.

The Challenge: Bringing about Institutional Change in Complex Systems

While the vision of improving student learning and assessment through implementation
of effective learning analytics tools and approaches holds promise, the real challenges of
implementation are significant. In this article we have identified only two of the several critical
and interconnected socio–technical domains that need to be addressed by comprehensive
institutional policy and strategic planning to introduce such attractive new systems: the
challenge of influencing stakeholder understanding of assessment in education, and the
challenge of developing the necessary institutional technological infrastructure to support
the undertaking. Meanwhile, of course, any such changes must coexist with the institution’s
business as usual obligations (Head & Alford, 2013).
It may not be surprising, then, that globally, education lags behind all other sectors in
harnessing the power of analytics. A preliminary analysis indicates that educational institutions
simply lack the practical, technical and financial capacity to effectively gather, manage and
Transformation of mine big data (Manyika et al., 2011). As Bichsel (2012) notes, much concern revolves around
assessment practices to “the perceived need for expensive tools or data collection methods” (p. 3). Certainly, evidence
embrace and integrate suggests that data access and management are proving to be significant hurdles for many
learning analytics tools institutions. The first survey of analytics implementation in US higher education in 2005 found
and strategies in support that of 380 institutions, 70% were at Stage 1 of a five–stage implementation process: “Extraction
of teaching and learning and reporting of transaction–level data” (Goldstein & Katz, 2005). Four years later, a study
therefore demands effec- of 305 US institutions found that 58% continued to wrangle data in Stage 1, while only 20%
tive institutional technol- reported progress to Stage 2: “Analysis and monitoring of operational performance” (Yanosky,
ogy infrastructures 2009). More recently, investigators have reported that while some 70% of surveyed institutions
agreed that analytics is a major priority for their school, the majority of respondents suggested
that data issues (quality, ownership, access, and standardization) were considerable barriers to
analytics implementation, and as such most were yet to make progress beyond basic reporting
(Bichsel, 2012; Norris & Baer, 2013).
To further unpack the complexities of analytics adoption a growing number of
commentators are exploring the more nuanced sociotechnical factors that are the most likely
barriers to institutional LA implementation. For instance, elements of institutional “culture,
capacity and behavior” (Norris, Baer, Leonard, Pugliese, & Lefrere, 2008). There is recognition
that even where technological competence and data exist, simple presentation of the facts (the
potential power of analytics), no matter how accurate and authoritative, may not be enough to
overcome institutional resistance (Macfadyen & Dawson, 2012; Young & Mendizabal, 2009).

Why Policy Matters for Learning Analytics

Higher education institutions are a superb example of complex adaptive systems
(CASs) (Cilliers, 1998; Gupta & Anish, 2009; MacLennan, 2007; Mitleton–Kelly, 2003) and
exist in a state that some have characterized as organized anarchy (Cohen & Marsh, 1986).
Together with institutional history and differences in stakeholder perspectives (Kingdon, 1995;
Sabatier, 2007), policies are the critical driving forces that underpin complex and systemic
institutional problems (Corvalán et al., 1999) and that shape perceptions of the nature of
the problem(s) and acceptable solutions. Below, we argue that it is therefore only through
implementation of planning processes driven by new policies that institutional change can
come about.
22 Volume Nine | Winter 2014
 The challenge of bringing about institution–wide change in such complex and
anarchic adaptive systems may rightly be characterized as a “wicked problem”– a problem
that is “complex, unpredictable, open ended, or intractable” (Churchman, 1967; Head &
Alford, 2013; Rittel & Webber, 1973). Like all complex systems, educational systems are
very stable, and resistant to change. They are resilient in the face of perturbation, and exist …we are now entering
far from equilibrium, requiring a constant input of energy to maintain system organization an era in which learning
(see Capra, 1996). As a result, and in spite of being organizations whose business is research may occur anywhere, at
and education, simple provision of new information to leaders and stakeholders is typically any time, with multiple
insufficient to bring about systemic institutional change. One factor hindering institutional devices, over a highly
change for better use of analytics by educational institutions appears to be their “lack of heterogeneous collection
data–driven mind–set and available data” (Manyika et al., 2011, p. 9). Interestingly, this of resources, and
observation is not new, and was reported with dismay in 1979 by McIntosh, in her discussion through multiple types of
of the failure of institutional research to inform institutional change. Ferguson et al. (in press) interactions.
reprise McIntosh’s arguments in relation to learning analytics, suggesting that additional
barriers to adoption include academics’ unwillingness to act on findings from other disciplines;
disagreement over the relative merits of qualitative vs. quantitative approaches to educational
research; a tendency to base decisions on anecdote; the reality that researchers and decision
makers speak different languages; lack of familiarity with statistical methods; a failure to
effectively present and explain data to decision makers; and the reality that researchers tend to
hedge and qualify conclusions. Norris and Baer (2013) meanwhile note that the analytics IQ of
institutional leaders is typically not high, precluding effective planning. In other words, a range
of political, social, cultural and technical norms shape educational systems and contribute to
their stability and resistance to change.
Elsewhere, we reported on a case study failure of learning analytics to inform
institutional planning (Macfadyen & Dawson, 2012), and noted that the culture of educational
institutions has historically valorized educator/faculty autonomy and resisted any administrative
efforts perceived to interfere with teaching and learning practice. We proposed that in order
to overcome institutional resistance to innovation and change driven by learning analytics,
educational institutions urgently need to implement planning processes that create conditions
that allow stakeholders across the institution to both think and feel positively about change –
conditions that appeal to both the heart and the head.
Social marketing theorists (Kotler & Zaltman, 1971) and change management experts
(Kavanagh & Ashkanasy, 2006; Kotter, 1996) similarly argue that social and cultural change
(that is, change in habits, practices and behaviors) is not brought about by simply giving
people large volumes of logical data (Kotter & Cohen, 2002). Social theorists have argued
that since value perspectives ground the major social issues of modern life, scientific analyses
and technical rationality are insufficient mechanisms for understanding and solving complex
problems (Head & Alford, 2013; Rein, 1976; Schon & Rein, 1994). Instead, what is needed are
comprehensive policy and planning frameworks to address not simply the perceived shortfalls
in technological tools and data management, but the cultural and capacity gaps that are the
true strategic issues (Norris & Baer, 2013).

Policy and Planning Approaches for Wicked Problems in Complex Systems

Policies are, simply, principles developed to guide subjective and/or objective decision
making, with the goal of achieving rational and desirable outcomes. They are statements of
intent that capture organizational goals, and are typically implemented via planned procedures
or protocols. A large and established literature on policy development already exists in fields
such as political science and business, from which have emerged a range of classical policy
cycle tools and heuristics that have been highly influential (Nakamura, 1987). Contemporary
critics from the planning and design fields argue, however, that these classic, top–down,
expert–driven (and mostly corporate) policy and planning models are based on a poor and
homogenous representation of social systems mismatched with our contemporary pluralistic
societies, and that implementation of such simplistic policy and planning models undermines
chances of success (for review, see Head & Alford, 2013). Importantly, they also insist that
modern policy problems are not technical puzzles that can be solved through the application
of scientific knowledge, but instead exist in continuous states of flux within dynamic systems
and have communicative, political and institutional elements. Solutions to such ill–defined
and multi–factorial challenges, they argue, will always be provisional, and must be negotiated
Volume Nine | Winter 2014 23
 between multiple stakeholders in situations of ambiguity, uncertainty and values disagreement
(Rittel & Webber, 1973). A number of theorists have also emphasized that solutions to wicked
problems – actually complex systems of inter–related problems – “can seldom be obtained
by independently solving each of the problems of which it is composed . . . Efforts to deal
separately with such aspects of urban life as transportation, health, crime, and education seem
to aggravate the total situation” (Ackoff, 1974, p. 21).
Systems theory offers two key areas of insight that are significant for policy development
for complex educational systems. First, systems theorists recognized that while systems –
from a single atom to a universe – may appear to be wildly dissimilar, they are all governed by
common patterns, behaviors and properties: their component parts are multiply interconnected
From the technological by information flows, with identifiable and predictable feedbacks, inputs, outputs, controls and
point of view, learning transformation processes; they are dynamic, differentiated and bounded; they are hierarchically
analytics is an emerg- organized and differentiated; and new properties can arise within them as a result of interactions
ing discipline and its between elements. Second, systems theory observes that systems tend to be stable, and that their
connection with assess- interconnectedness facilitates resilience (for a review of systems theory, see Capra, 1996).
ment remains largely
These observations not only illuminate why piecemeal attempts to effect change in
unexplored.
educational systems are typically ineffective, but also explains why no one–size–fits–all prescriptive
approach to policy and strategy development for educational change is available or even possible.
Usable policy frameworks will not be those which offer a to do list of, for example, steps in learning
analytics implementation. Instead, successful frameworks will be those which guide leaders and
participants in exploring and understanding the structures and many interrelationships within
their own complex system, and identifying points where intervention in their own system will be
necessary in order to bring about change.
Drawing on systems and complexity theory, a new generation of authors have begun
to develop accounts of so–called adaptive approaches to policy and planning for complex
systems which can allow institutions to respond flexibly to ever–changing social and
institutional contexts and challenges (Berkhout, Leach, & Scoones, 2003; Haynes, 2003;
Milliron, Malcolm, & Kil, 2014; Tiesman, van Buuren, & Gerrits, 2009; Young & Mendizabal,
2009). A full review of adaptive management strategies is beyond the scope of this paper, and
has been comprehensively undertaken by Head and Alford (2013), who highlight the critical
roles of cross–institutional collaboration, new forms of leadership (moving beyond the
orthodox model of transformational leadership) and the development of enabling structures
and processes (for example, budgeting and finance systems, organizational structure,
human resources management, and approaches to performance measurement and program
evaluation). We offer here two sample policy and planning models that may offer valuable
practical guidance for collaborative teams and leaders in higher education seeking to bring
about systemic institutional change to support learning analytics.

Figure 1. The RAPID Outcome Mapping Approach (ROMA)

24 Volume Nine | Winter 2014

 First, and as we have proposed elsewhere (Ferguson et al., in press) we offer a modification
of Young and Mendizabal’s (2009) Rapid Outcome Mapping Approach (ROMA) model (Figure 1)
as a policy and planning heuristic for learning analytics implementation. Originally developed
to support policy and strategy processes in the complex field of international development, the
seven–step ROMA model is focused on evidence–based policy change. It is designed to be used
iteratively, and to allow refinement and adaptation of policy goals and the resulting strategic plans
over time and as contexts change, emphasizing the provisional nature of any solutions arrived at.
Importantly, the ROMA process begins with a systematic effort at mapping institutional context
(for which these authors offer a range of tools and frameworks) – the people, political structures,
policies, institutions and processes that may help or hinder change. This critical activity allows
institutions to identify the key factors specific to their own context that may influence (positively
or negatively) the implementation process, and therefore also has the potential to illuminate points
of intervention and shape strategic planning.

Figure 2. Cause–effect (DPSEEA) framework for institutional assessment and technology policies
(modified from Corvalan et al., 1999).

Second, Corvalán et al.’s (1999) “cause–effect framework” (or DPSEEA framework) usefully
assists in identifying the multiple linkages that may exist between the driving forces underpinning It may not be surprising,
complex systems, illuminating the multiple points in a complex system of relationships where then, that globally,
action may be needed to effect change. Such a framework can, they suggest, “be used to weigh education lags behind
alternatives and to design step–by–step programs for dealing with a particular…problem” (p. 659). all other sectors in
Figure 2 offers a preliminary modification of this framework to represent institutional effects of, harnessing the power of
for example, technology and assessment policies, and may be a useful context mapping tool in the analytics. A preliminary
ROMA process. analysis indicates that
educational institutions
Use of these models for institutional LA policy development is only in the very early
simply lack the practical,
stages, although we have explored elsewhere (Ferguson et al., in press) the ways in which a small
technical and financial
number of apparently successful institutional LA policy and planning processes have pursued
capacity to effectively
change management approaches that map well to such frameworks. In future work, we hope to
gather, manage and mine
be able to present more robust and critical review of real–time application of these frameworks in
big data.
institutional planning, and their possible effectiveness or limitations.
In the meantime, readers may review both frameworks and immediately dispute the
stages, levels, linkages, effects or impacts in relation to their own institutional context. But this is, of
course, the very point of such adaptive models, which can and should be disputed, negotiated and
modified as needed for local institutional contexts, to guide relevant local action. To paraphrase
Head and Alford (2013), when it comes to wicked problems in complex systems, there is no one–
size–fits–all policy solution, and there is no plan that is not provisional.
Rather, the more important role of such frameworks is to continuously remind us of the
need for a holistic understanding of institutional context if the goal is institutional change, including
external and internal influences, political and cultural context, the evidence itself, and the links:
Volume Nine | Winter 2014 25
 “All of the other actors and mechanisms that affect how the evidence gets into the policy process” (Young
& Mendizabal, 2009). They can assist in identifying points of intervention (Corvalán et al., 1999) in the
complex adaptive system that is education, to offer leaders and practitioners additional insight and tools in
their project of optimizing the system with learning analytics.

References
Ackoff, R. L. (1974). Redesigning the future. New York, NY: Wiley.
Angelo, T. A. (1999). Doing assessment as if learning matters most. AAHE Bulletin, 51(9), 3–6.
Azevedo, R., & Aleven, V. (Eds.). (2013). International handbook of metacognition and learning technologies.
Amsterdam: Springer.
Berkhout, F., Leach, M., & Scoones, I. (Eds.). (2003). Negotiating environmental change: New perspectives from social
science. Cheltenham, UK: Edward Elgar.
Bichsel, J. (2012). Analytics in Higher Education: Benefits, Barriers, Progress, and Recommendations (Research
Report). Louisville, CO: EDUCAUSE Center for Applied Research. http://net.educause.edu/ir/library/pdf/
ERS1207/ers1207.pdf
Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining
and learning analytics: An issue brief. Washington, DC: U.S. Department of Education Office of Educational
Technology. http://www.ed.gov/edblogs/technology/files/2012/03/edm–la–brief.pdf
Buckingham Shum, S. (2012). Learning dispositions and transferable competencies: Pedagogy, modelling and learning
analytics. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), International conference on learning
analytics and knowledge (pp. 92–101). New York, NY: ACM Press.
Butler, D. L., & Winne, P. H. (1995). Feedback and self–regulated learning: A theoretical synthesis. Review of
Educational Research, 65(3), 245–281.
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE
Review, 42(4), 42–57.
Capra, F. (1996). The web of life. New York, NY: Doubleday.
Churchman, C. W. (1967). Free for all. Management Science, 14, B141–B142.
Cilliers, P. (1998). Complexity and postmodernism: Understanding complex systems. London, UK: Routledge.
Cohen, M. D., & Marsh, J. G. (1986). Leadership and ambiguity: The American college president. New York, NY:
McGraw Hill.
Corvalán, C. F., Kjellström, T., & Smith, K. R. (1999). Health, environment and sustainable development: Identifying
links and indicators to promote action. Epidemiology, 10(5), 656–660.
Dillon, T., Wu, C., & Chang, E. (2010). Cloud computing: Issues and challenges. In IEEE International Conference on
Advanced Information Networking and Applications (pp. 27–33). New York, NY: IEEE Press.
Dumont, H., Istance, D., & Benavides, F. (Eds.). (2010). The nature of learning: Using research to inspire
practice. Educational Research and Innovation series, OECD Publishing. http://www.oecd.org/edu/ceri/
thenatureoflearningusingresearchtoinspirepractice.htm
Dunlosky, J., & Lipko, A. R. (2007). Metacomprehension: A brief history and how to improve its accuracy. Current
Directions in Psychological Science, 16(4), 228–232.
Dunlosky, J., & Thiede, K. W. (1998). What makes people study more? An evaluation of factors that affect self–paced
study. Acta Psychologica, 98(1), 37–56.
Dworkin, G. (2005). The No Child Left Behind act: Accountability, high–stakes testing and roles for sociologists.
Sociology of Education, 78(2), 170–174.

26 Volume Nine | Winter 2014

Ellis, C. (2013). Broadening the scope and increasing the usefulness of learning analytics: The case for assessment
analytics. British Journal of Educational Technology, 44(4), 662–664.
Elton, L. (2004). Goodhart's Law and performance indicators in higher education. Evaluation & Research in Education,
18(1–2), 120–128.
Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology
Enhanced Learning, 4(5/6), 304–317.
Ferguson, R., Macfadyen, L. P., Clow, D., Tynan, B., Alexander, S. & Dawson, S. (in press). Setting learning analytics in
context: Overcoming the barriers to large–scale adoption. Invited paper for the Journal of Learning Analytics
based on a paper presented at the 2014 Learning Analytics and Knowledge, Indianapolis, IN.
Fritz, J. (2011). Classroom walls that talk: Using online course activity data of successful students to raise self–awareness
of underperforming peers. The Internet and Higher Education, 14(2), 89–97.
Glahn, C. (2013). Using the ADL experience API for mobile learning, sensing, informing, encouraging, orchestrating. In
International conference on next generation mobile apps, services and technologies (pp. 268–273). New
York, NY: IEEE Press.
Goldstein, P. J., & Katz, R. N. (2005). Academic analytics: The uses of management information and technology in
higher education (Vol. 8). Louisville, CO: EDUCAUSE Center for Applied Research.
Gupta, A., & Anish, S. (2009). Insights from complexity theory: Understanding organisations better. IIMB Management
Review. http://tejas.iimb.ac.in/articles/12.php
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
Haynes, P. (2003). Managing complexity in the public services. Maidenhead, UK: Open University Press.
Head, B. W., & Alford, J. (2013). Wicked problems: Implications for public policy and management. Administration &
Society. doi:10.1177/0095399713481601
Jacob, B. (2005). Accountability, incentives and behavior: Evidence from school reform in Chicago. Journal of Public
Economics, 89(5–6), 761–796.
Kavanagh, M. H., & Ashkanasy, N. M. (2006). The impact of leadership and change management strategy on
organizationsal culture and individual acceptance of change during a merger. British Journal of Management,
17, S81–S103.
Kingdon, J. (1995). Agendas, alternatives and public policies (2nd ed.). Boston, MA: Longman.
Kotler, P., & Zaltman, G. (1971). Social marketing: An approach to planned social change. Journal of Marketing, 35, 3–12.
Kotter, J. P. (1996). Leading change. Boston, MA: Harvard Business School Press.
Kotter, J. P., & Cohen, D. S. (2002). The heart of change. Boston, MA: Harvard Business School Press.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How differences in recognizing one's own incompetence
lead to inflated self–assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134.
Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough: Why e–learning analytics failed to inform an
institutional strategic plan. Educational Technology & Society, 15(3), 149–163.
MacLennan, B. (2007). Evolutionary psychology, complex systems, and social theory. Soundings: An Interdisciplinary
Journal, 90(3/4), 169–189.
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Byers, A. H. (2011). Big data: The next frontier
for innovation, competition and productivity. McKinsey Global Institute. http://www.mckinsey.com/insights/
business_technology/big_data_the_next_frontier_for_innovation
McDonnell, L. M. (1994). Assessment policy as persuasion and regulation. American Journal of Education, 102(4), 394–320.
McIntosh, N. E. (1979). Barriers to implementing research in higher education. Studies in Higher Education, 4(1), 77–86.
Milliron, M. D., Malcolm, L., & Kil, D. (2014). Insight and action analytics: Three case studies to consider. Research &
Practice in Assessment, 9(2), 70-89.
Mitleton–Kelly, E. (2003). Ten principles of complexity & enabling infrastructures. In E. Mitleton–Kelly (Ed.), Complex
systems & evolutionary perspectives of organisations: The application of complexity theory to organisations
(pp. 23–50). Philadelphia, PA: Elsevier.

Volume Nine | Winter 2014 27

 Nakamura, R. T. (1987). The textbook policy process and implementation research. Review of Policy Research, 7(1), 142–154.
Norris, D., & Baer, L. L. (2013). Building organizational capacity for analytics. Louisville, CO: EDUCAUSE. https://net.
educause.edu/ir/library/pdf/PUB9012.pdf
Norris, D., Baer, L., Leonard, J., Pugliese, L., & Lefrere, P. (2008). Action analytics: Measuring and improving
performance that matters in higher education. EDUCAUSE Review, 43(1), 42–67.
Pintrich, P. R. (1999). The role of motivation in promoting and sustaining self–regulated learning. International Journal
of Educational Research, 31(6), 459–470.
Rein, M. (1976). Social science and public policy. Harmondsworth, UK: Penguin.
Richardson, J. (2004). Methodological issues in questionnaire–based research on student learning in higher education.
Educational Psychology Review, 16(4), 347–358.
Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155–169.
Sabatier, P. A. (Ed.). (2007). Theories of the policy process (2nd ed.). Boulder, CO: Westview Press.
Schon, D. A., & Rein, M. (1994). Frame reflection: Toward the resolution of intractable policy controversies. New York,
NY: Basic Books.
Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–
1400. doi:10.1177/0002764213498851
Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the productivity of the higher education sector: Policy and
strategy for systems–level deployment of learning analytics. Sydney, Australia: Society for Learning
Analytics Research for the Australian Government Office for Learning and Teaching. http://solaresearch.
org/Policy_Strategy_Analytics.pdf
Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10),
1510–1529.
Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using signals for appropriate feedback: Perceptions and
practices. Computers & Education, 57(4), 2414–2422. http://www.itap.purdue.edu/learning/docs/research/1–
s2.0–S0360131511001229–main.pdf
Thomas, H. (2010). Learning spaces, learning environments and the dis'placement' of learning. British Journal of
Educational Technology, 41(3), 502–511.
Tiesman, G. R., van Buuren, A., & Gerrits, L. (Eds.). (2009). Managing complex governance systems. London, UK: Routledge.
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications.
American Behavioral Scientist. doi: 10.1177/0002764213479363
Wall, A. F., Hursh, D., & Rodgers, J. W. III. (2014). Assessment for whom: Repositioning higher education assessment as
an ethical and value–focused social practice. Research and Practice in Assessment, 9(1), 5–17.
Wiliam, D. (2010). The role of formative assessment in effective learning environments. In H. Dumont, D. Istance,
& F. Benavides (Eds.), The nature of learning: Using research to inspire practice (pp. 135–155): Paris, France:
OECD Publishing.
Winne, P. H. (2010). Improving measurements of self–regulated learning. Educational Psychologist, 45(4), 267–276.
Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In A. Pardo & S. D.
Teasley (Eds.), Proceedings of the International Conference on Learning Analytics and Knowledge. New
York, NY: ACM Press.
Yanosky, R. (2009). Institutional data management in higher education. Boulder, CO: EDUCAUSE Center for Applied Research.
Young, J., & Mendizabal, E. (2009). Helping researchers become policy entrepreneurs: How to develop engagement
strategies for evidence–based policy–making. (Briefing Paper) ODI briefing Papers. London, UK.
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self–reported versus traced goal orientation.
Learning and Instruction, 22(6), 413–419.
Zimmerman, B. J. (2002). Becoming a self–regulated learner: An overview. Theory Into Practice, 41(2), 64–70. doi:
10.1207/s15430421tip4102_2

28 Volume Nine | Winter 2014