Measuring Impact: The Importance of Evaluation For Documentary Film Campaigns
Measuring Impact: The Importance of Evaluation For Documentary Film Campaigns
net/publication/256545144
CITATIONS READS
19 9,807
2 authors:
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Beth Karlin on 01 June 2014.
1
Karlin
&
Johnson.
(2011).
M/C
Journal
14(6)
sentiment, arguing that, "despite widespread Documentary film has undergone many
speculation about the impact of documentaries, significant changes since its inception, from the
the topic has received relatively little systematic heavily staged romanticism movement of the
attention" ("Evolving"). And earlier this year, the 1920s to the propagandist tradition of
introduction to a special issue of Mass governments using film to persuade individuals to
Communication and Society on documentary film support national agendas to the introduction of
stated, "documentary film, despite its growing cinéma vérité in the 1960s and historical
influence and many impacts, has mostly been documentary in the 1980s (cf. Barnouw).
overlooked by social scientists studying the However, the recent upsurge in popularity of
media and communication" (Nisbet and documentary media, combined with technological
Aufderheide 451). advances of internet and computers have opened
Film has been studied extensively as up a whole new set of opportunities for film to
entertainment, as narrative, and as cultural event, serve as both art and agent for social change.
but the study of film as an agent of social change One such opportunity is in the creation of film-
is still in its infancy. This paper introduces a based social action campaigns. Over the past
systematic approach to measuring the social decade, filmmakers have taken a more active role
impact of documentary film aiming to: (1) discuss in promoting social change by coordinating film
the context of documentary film and its potential releases with action campaigns. Companies such
impact; and (2) argue for a social science as Participant Media (An Inconvenient Truth,
approach, discussing key issues about conducting Food Inc., etc.) now create "specific social action
such research. campaigns for each film and documentary
CHANGES IN DOCUMENTARY PRACTICE designed to give a voice to issues that resonate in
Documentary film has been used as a tool for the films" (Participant Media). In addition, a new
promoting social change throughout its history. sector of "social media" consultants are now
John Grierson, who coined the term offering services, including "consultation,
"documentary" in 1926, believed it could be used strategic planning for alternative distribution,
to influence the ideas and actions of people in website and social media development, and
ways once reserved for church and school. He complete campaign management services to
presented his thoughts on this emerging genre in filmmakers to ensure the content of nonfiction
his 1932 essay, First Principles of Documentary, media truly meets the intention for change"
saying, "We believe that the cinema's capacity for (Working Films).
getting around, for observing and selecting from The emergence of new forms of media and
life itself, can be exploited in a new and vital art technology are changing our conceptions of both
form" (97). Richard Barsam further specified the documentary film and social action. Technologies
definition of documentary, distinguishing it from such as podcasts, video blogs, internet radio,
non-fiction film, such that all documentaries are social media and network applications, and
non-fiction films but not all non-fiction films are collaborative web editing "both unsettle and
documentaries. He distinguishes documentary extend concepts and assumptions at the heart of
from other forms of non-fiction film (i.e. travel 'documentary' as a practice and as an idea"
films, educational films, newsreels) by its (Ellsworth). In the past decade, we have seen new
purpose; it is a film with an opinion and a specific forms of documentary creation, distribution,
message that aims to persuade or influence the marketing, and engagement. Likewise, film
audience. And Bill Nichols writes that the campaigns are utilizing a broad array of strategies
definition of documentary may even expand to engage audience members, including "action
beyond the film itself, defining it as a kits, screening programs, educational curriculums
"filmmaking practice, a cinematic tradition, and and classes, house parties, seminars, panels" that
mode of audience reception" (12). often turn into "ongoing 'legacy' programs that
2
Karlin
&
Johnson.
(2011).
M/C
Journal
14(6)
are updated and revised to continue beyond the strengthen a program, and summative evaluation
film's domestic and international theatrical, DVD examines program outcomes. In terms of
and television windows" (Participant Media). documentary film, these two goals can be
This move towards multi-media documentary described as follows:
film is becoming not only commonplace, but Formative Evaluation: Informing the Process
expected as a part of filmmaking. NYU film As programs (broadly defined as an intentional
professor and documentary film pioneer George set of activities with the aim of having some
Stoney recently noted, "50 percent of the specific impact), the people who interact with
documentary filmmaker's job is making the them, and the cultures they are situated in are
movie, and 50 percent is figuring out what its constantly changing, program development and
impact can be and how it can move audiences to evaluation is an ongoing learning cycle. Film
action" (qtd. in Nisbet, "Gasland"). In his book campaigns, which are an intentional set of
Convergence Culture, Henry Jenkins, coined the activities with the aim of impacting individual
term "transmedia storytelling", which he later viewers and broader cultures, fit squarely within
defined as "a process where integral elements of a this purview. Without formulating hypotheses
fiction get dispersed systematically across about the relationships between program
multiple delivery channels for the purpose of activities and goals and then collecting and
creating a unified and coordinated entertainment analyzing data during implementation to test
experience" ("Transmedia"). When applied to them, it is difficult to learn ways to improve
documentary film, it is the elements of the "issue" programs (or continue doing what works best in
raised by the film that get dispersed across these the most efficient manner). Attention to this
channels, coordinating, not just an entertainment process enables those involved to learn more
experience, but a social action campaign. about, not only what works, but how and why it
works and even gain insights about how program
DIMENSIONS OF EVALUATION
outcomes may be affected by changes to resource
It is not unreasonable to assume that such film
availability, potential audiences, or infrastructure.
campaigns, just like any policy or program, have
the possibility to influence viewers' knowledge, Filmmakers are constantly learning and honing
attitudes, and behavior. Measuring this impact their craft and realizing the impact of their
has become increasingly important, as funders of practice can help the artistic process. Often faced
documentary and issue-based films want look to with tight budgets and timelines, they are forced
understand the "return on investment" of films in to confront tradeoffs all the time, in the writing,
terms of social impact so that they can compare production and post-production process.
them with other projects, including non-media, Understanding where they are having impact can
direct service projects. Although we "feel" like improve their decision-making, which can help
films make a difference to the individuals who both the individual project and the overall field.
also see them in the broader cultures in which Summative Evaluation: Quantifying Impacts
they are embedded, measurement and empirical Evaluation is used in many different fields to
analysis of this impact are vitally important for determine whether programs are achieving their
both providing feedback to filmmakers and intended goals and objectives. It became popular
funders as well as informing future efforts in the 1960s as a way of understanding the impact
attempting to leverage film for social change. of the Great Society programs and has continued
This type of systematic assessment, or program to grow since that time (Madaus and
evaluation, is often discussed in terms of two Stufflebeam). A recent White House memo stated
primary goals—formative (or process) and that "rigorous, independent program evaluations
summative (or impact) evaluation (cf. Muraskin; can be a key resource in determining whether
Trochim and Donnelly). Formative evaluation government programs are achieving their
studies program materials and activities to intended outcomes as well as possible and at the
3
Karlin
&
Johnson.
(2011).
M/C
Journal
14(6)
lowest possible cost" and the United States Office DEFINING THE PROJECT
of Management and Budget (OMB) launched an Though this may seem like an obvious step, it is
initiative to increase the practice of "impact essential to determine the nature of the project so
evaluations, or evaluations aimed at determining one can create research questions and hypotheses
the causal effects of programs" (Orszag 1). based on a complete understanding of the
"treatment". One organization that provides a
Documentary films, like government programs,
great example of the integration of documentary
generally target a national audience, aim to serve
film imbedded into a larger campaign or
a social purpose, and often do not provide a
movement is Invisible Children. Founded in
return on their investment. Participant Media, the
2005, Invisible Children is both a media-based
most visible and arguably most successful
organization as well as an economic development
documentary production company in the film
NGO with the goal of raising awareness and
industry, made recent headlines for its difficulty
meeting the needs of child soldiers and other
in making a profit during its seven-year history
youth suffering as a result of the ongoing war in
(Cieply). Owner and founder Jeff Skoll reported
northern Uganda. Although Invisible Children
investing hundreds of millions of dollars into the
began as a documentary film, it has grown into a
company and CEO James Berk added that the
large non-profit organization with an operating
company sometimes measures success, not by
budget of over $8 million and a staff of over a
profit, but by "whether Mr. Skoll could have
hundred employees and interns throughout the
exerted more impact simply by spending his
year as well as volunteers in all 50 states and
money philanthropically" (Cieply).
several countries. Invisible Children
Because of this, documentary projects often rely programming includes films, events, fundraising
on grant funding, and are starting to approach campaigns, contests, social media platforms,
funders beyond traditional arts and media blogs, videos, two national "tours" per year,
sources. "Filmmakers are finding new fiscal and merchandise, and even a 650-person three-day
non-fiscal partners, in constituencies that would youth summit in August 2011 called The Fourth
not traditionally be considered—or consider Estate.
themselves—media funders or partners"
Individually, each of these components might
(BRITDOC 6). And funders increasingly expect
lead to specific outcomes; collectively, they
tangible data about their return on investment.
might lead to others. In order to properly assess
Says Luis Ubiñas, president of Ford Foundation,
impacts of the film "project", it is important to
which recently launched the Just Films Initiative:
take all of these components into consideration
In these times of global economic uncertainty, and think about who they may impact and how.
with increasing demand for limited philanthropic This informs the research questions, hypotheses,
dollars, assessing our effectiveness is more and methods used in evaluation.
important than ever. Today, staying on the
Film campaigns may even include partnerships
frontlines of social change means gauging, with
with existing social movements and non-profit
thoughtfulness and rigor, the immediate and
organizations targeting social change. The
distant outcomes of our funding.
American University Center for Social Media
Es Establishing the need for evaluation is not concluded in a case study of three issue-based
enough—attention to methodology is also critical. documentary film campaigns:
Valid research methodology is a critical
Digital technologies do not replace, but are
component of understanding around the role
closely entwined with, longstanding on-the-
entertainment can play in impacting social and
ground activities of stakeholders and citizens
environmental issues. The following issues are
working for social change. Projects like these
vital to measuring impact.
forge new tools, pipelines, and circuits of
circulation in a multiplatform media
4
Karlin
&
Johnson.
(2011).
M/C
Journal
14(6)
5
Karlin
&
Johnson.
(2011).
M/C
Journal
14(6)
that for some projects and issues there are key directly at web-traffic changes "enables the
indicators of success" (Barrett and Leddy, 7). creation of an information-seeking curve that can
This can take the form of policy or legislative define the parameters of a teachable moment"
change, passed through film-based lobbying (Hart and Leiserowitz 360). Audience reception
efforts, or shifts in public dialogue and behavior. can be measured, not only via interviews and
Legislative change typically takes place beyond focus groups, but also through content and
the social movement stage, when there is enough sentiment analysis of web content and online
support to pressure legislators to change or create analytics. "Sophisticated analytics can
policy. Film-inspired activism has been seen in substantially improve decision making, minimize
issues ranging from environmental causes such as risks, and unearth valuable insights that would
agriculture (Food Inc.) and toxic products (Blue otherwise remain hidden" (Manyika et al. 5).
Vinyl) to social causes such as foreign conflict These new tools are significantly changing
(Invisible Children) and education (Waiting for evaluation, expanding what we can learn about
Superman). Documentary films can also have a the social impacts of film through triangulation of
strong influence as media agenda-setters, as films self-report data with measurement of actual
provide dramatic "news pegs" for journalists behavior in virtual environments.
seeking to either sustain or generation new CONCLUSION
coverage of an issue (Nisbet "Introduction" 5), The changing media landscape both allows and
such as the media coverage of climate change in impels evaluation of film impacts on individual
conjunction with An Inconvenient Truth. viewers and the broader culture in which they are
imbedded. Although such analysis may have
Barrett and Leddy, however, note that not all
previously been limited to box office numbers,
films target all five impacts and that different
critics' reviews, and theater exit surveys, the rise
films may lead to different impacts. "In some
of new media provides both the ability to connect
cases we could look to key legislative or policy
filmmakers, activists, and viewers in new ways
changes that were driven by, or at least supported
and the data in which to study the process. This
by the project... In other cases, we can point to
capability, combined with significant growth in
shifts in public dialogue and how issues are
the documentary landscape, suggests a great
framed and discussed" (7). It is possible that
potential for documentary film to contribute to
specific film and/or campaign characteristics may
some of our most pressing social and
lead to different impacts; this is a nascent area for
environmental needs. A social scientific
research and one with great promise for both
approach, that combines empirical analysis with
practical and theoretical utility.
theory applied from basic science, ensures that
INNOVATIONS IN TOOLS AND METHODS impact can be measured and leveraged in a way
Finally, the selection of tools is a vital component that is useful for both filmmakers as well as
for assessing impact and the new media landscape funders. In the end, this attention to impact
is enabling innovations in the methods and ensures a continued thriving marketplace for
strategies for program evaluation. Whereas the issue-based documentary films in our social
traditional domain of film impact measurement landscape.
included box office statistics, focus groups, and
exit surveys, innovations in data collection and
analysis have expanded the reach of what
questions we can ask and how we are able to
answer them.
For example, press coverage can assist in
understanding and measuring the increase in
awareness about an issue post-release. Looking
6
Karlin
&
Johnson.
(2011).
M/C
Journal
14(6)
7
Karlin
&
Johnson.
(2011).
M/C
Journal
14(6)
Nisbet, Matthew. "Gasland and Dirty Business: Vladica, Florin and Charles Davis. "Business
Documentary Films Shape Debate on Innovation and New Media Practices in
Energy Policy." Big Think. 9 May 2011. Documentary Film Production and
Web. 1 Oct. 2011. Distribution: Conceptual Framework and
<http://bigthink.com/ideas/38345> Review of Evidence." The Media as a
--- "Introduction: Understanding the Social Driver of the Information Society. Ed. Ed
Impact of a Documentary Film." Albarran, Paulo Faustino and R. Santos.
Documentaries on a Mission: How Lisbon, Portugal: Media XXI / Formal
Nonprofits Are Making Movies for Public press. 2009. 299-319.
Engagement. Hirsch, Karen. Center for Whiteman, David. "Out of the Theaters and Into
Social Media. Mar. 2007. Web. 10 Sept. the Streets: A Coalition Model of the
2011. Political Impact of Documentary Film and
<http://aladinrc.wrlc.org/bitstream/1961/4 Video." Political Communication 21.1
634/1/docs_on_a_mission.pdf> (2004): 51-69.
Nisbet, Matthew and Patricia Aufderheide. --- "The Evolving Impact of Documentary Film:
"Documentary Film: Towards a Research Sacrifice and the Rise of Issue-Centered
Agenda on Forms, Functions, and Outreach." Post Script. 22 Jun. 2007.
Impacts." Mass Communication and Web. 10 Sept. 2011.
Society 12.4 (2011): 450-56. Winston, Brian. Claiming the Real: The
Orszag, Peter. Increased Emphasis on Program Documentary Film Revisited. London:
Evaluation. Washington: Office of British Film Institute, 1995.
Management and Budget. 7 Oct. 2009. "Working Films." Foundation Source Access. 31
Web. 10 Oct. 2011. May 2011. Web. 5 Oct. 2011.
<http://www.whitehouse.gov/sites/default/ <http://access.foundationsource.com/nonp
files/omb/assets/memoranda_2010/m10- rofit/working-films/>
01.pdf>
"Our Mission". Participant Media. n.d. Web. 2
Apr. 2011.
<http://www.participantmedia.com/compa
ny/about_us.php.>
Plantinga, Carl. Rhetoric and Representation in
Nonfiction Film. Cambridge: Cambridge
UP, 1997.
Trochim, William and James Donnelly. Research
Methods Knowledge Base. 3rd Edition.
Mason: Atomic Dogs. 2007.
Ubiñas, Luis. "President's Message." 2009
Annual Report. Ford Foundation. Sept.
2010. Web. 10 Oct. 2011.
<http://www.fordfoundation.org/about-
us/2009-annual-report/presidents-
message>