Keegan 2017
Keegan 2017
Keegan 2017
Access to this document was granted through an Emerald subscription provided by emerald-
srm:512739 []
For Authors
If you would like to write for this, or any other Emerald publication, then please use our Emerald
for Authors service information about how to choose which publication to write for and submission
guidelines are available for all. Please visit www.emeraldinsight.com/authors for more information.
About Emerald www.emeraldinsight.com
Emerald is a global publisher linking research and practice to the benefit of society. The company
manages a portfolio of more than 290 journals and over 2,350 books and book series volumes, as
well as providing an extensive range of online products and additional customer resources and
services.
Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the
Committee on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for
digital archive preservation.
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
Evaluation and
Evaluation and decision making decision
in social media marketing making in
SMM
Brendan James Keegan
Marketing, Operations and Digital Business,
Manchester Metropolitan University Business School, Manchester, UK, and
15
Jennifer Rowley Received 7 October 2015
Department of Information and Communications, Revised 13 April 2016
20 August 2016
Manchester Metropolitan University, Manchester, UK Accepted 25 October 2016
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
Abstract
Purpose – As organisations are increasing their investment in social media marketing (SMM), evaluation of
such techniques is becoming increasingly important. The purpose of this paper is to contribute to knowledge
regarding SMM strategy by developing a stage model of SMM evaluation and uncovering the challenges
in this process.
Design/methodology/approach – Interviews were conducted with 18 key informants working for
specialist SMM agencies. Such informants are a particularly rich source, since they manage social media
campaigns for a wide range of clients. An exploratory research was conducted and thematic analysis surfaced
the key components of the SMM evaluation process and associated challenges.
Findings – The SMM evaluation framework is developed. This framework has the following six stages:
setting evaluation objectives, identifying key performance indicators (KPIs), identifying metrics, data
collection and analysis, report generation and management decision making. Challenges associated with each
stage of the framework are identified, and discussed with a view to better understanding decision making
associated with social media strategies. Two key challenges are the agency-client relationship and the
available social analytics tools.
Originality/value – Despite an increasing body of research on social media objectives, KPIs and metrics, no
previous study has explored how these components are embedded in a marketing campaign planning
process. The paper also offers insights in the factors that make SMM evaluation complex and challenging.
Recommendations for further research and practice are offered.
Keywords Social media marketing, Digital marketing, Social media analytics, Agency-client relationship,
Social media marketing evaluation
Paper type Research paper
Introduction
Due to its dynamic and emergent nature, the effectiveness of social media as a marketing
communications channel has presented many challenges for marketers. It is considered to
be different to traditional marketing channels, and even other digital marketing channels,
centring around a two-way conversation or exchange (Bacile et al., 2014; Shih, 2009). Many
organisations are investing in their social media presence because they appreciate the need
to engage in existing social media conversations in order to protect their corporate or brand
reputation (Lee and Youn, 2009), increase customer engagement (Gummerus et al., 2012) or
increase online sales (Tsimonis and Dimitriadis, 2014). As organisations increasingly
develop their social media presence, it is vital to be able to evaluate the impact of this
investment, including its contribution to achieving marketing objectives, as well as more
generally understanding any return-on-investment (ROI) (Pang and Lee, 2008; Fisher, 2009;
Kumar and Mirchandani, 2012; McCann and Barlow, 2015).
Research into social media strategy is limited. There is some research in this area on
some specific aspects of strategy, such as reputation management (Rokka et al., 2014), the Management Decision
drivers, activities and benefits associated with social media (Tsimonis and Dimitriadis, Vol. 55 No. 1, 2017
pp. 15-31
2014), practitioner case studies using one organisation such as Finnair ( Jarvenpaa and © Emerald Publishing Limited
0025-1747
Tuunainen, 2013), B2B companies adoption of social media (Michaelidou et al., 2011) and the DOI 10.1108/MD-10-2015-0450
MD integration of social media into strategic marketing (Choi and Thoeni, 2016). Choi and
55,1 Theoni (2016) in particular identify a number of challenges in the area of social media
marketing (SMM) and suggest that further research is necessary.
There is a growing interest in the evaluation of the impact of SMM including research
driven by the need to demonstrate the ROI from SMM (Fisher, 2009; Hoffman and Fodor, 2010;
Kumar and Mirchandani, 2012; McCann and Barlow, 2015). There is also interest in the
16 potential of SMM to enhance firm and brand equity (Luo et al., 2013; Tirunillai and Tellis, 2012).
However, there is a considerable journey to travel before the impact of SMM can be intelligently
assessed. Some offer insights into the wider aspects of the processes associated with evaluation
of SMM in specific contexts (Kim and Ko, 2012; Michaelidou et al., 2011; Murdough, 2010;
Töllinen et al., 2010), in addition, there has been significant activity on measurement
frameworks and dashboards (Cvijikj et al., 2012; Marklein and Paine, 2013; Peters et al., 2013),
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
and some discussion of the need to establish clearly defined goals, objectives and metrics
related to the use of social media (e.g. Hoffman and Fodor, 2010; Murdough, 2010).
Only Jeffrey (2013) and McCann and Barlow (2015) have proposed frameworks that link
measurement with SMM decision making and campaign planning. Thus far, neither
framework has been empirically tested, and hence is prescriptive in nature rather than
practice-based. Yet, as O’Sullivan et al. (2009) demonstrate, marketing performance
measurement ability or frequency is linked to firm performance. Accordingly, SMM
evaluation offers a pivotal context in which to consider the challenges associated with
SMM decision making and management.
The aim of this research is contribute to knowledge and theory regarding social media
strategy through an exploratory study of the evaluation of SMM, with a view to proposing a
process framework. In addition, this paper presents a distillation of the challenges
associated with the evaluation process. Hence, the objectives of this research are to:
• identify and define the stages of SMM evaluation, as operationalized by practitioners,
and to propose a conceptual framework; and
• identify and summarise the challenges associated with SMM evaluation.
Next, previous research on the importance and potential of SMM and its evaluation is
summarised. Then, the interview-based research methodology is outlined. This is followed
by a report on SMM evaluation processes and a discussion of the associated challenges.
Finally, the conclusion summarises the research and suggests recommendations for
research and practice.
Literature review
SMM
Aral et al. (2013) argue that social media is “fundamentally changing the way we
communicate, collaborate, consume, and create” (p. 3). Defined as “a group of internet-based
applications that build on the ideological and technological foundations of Web 2.0 and that
allow the creation and exchange of user-generated content” (Kaplan and Haenlein, 2010,
p. 61), social media are impacting on a wide range of business processes, from marketing
and operations to finance (Luo et al., 2013) and human resource management (Bolton, 2013).
In the marketing context, social media is seen as essentially different to other forms of
digital media (Hoffman and Novak, 2012; Hennig-Thurau et al., 2010) and as potentially
heralding a paradigm shift in marketing (Hanna et al., 2011).
Indeed, studies demonstrate that participation in a firm’s social media activities positively
affects profitability. For example, Goh et al. (2013) studied the relative impact of social media
on firm profits and established that user-generated content had a greater impact on
profits than firm-created content. Tirunillai and Tellis (2012) demonstrate that online
reviews and “chatter” are indicators of stock market performance, whilst Luo et al. (2013)
show that social media-based metrics are leading indicators of firm equity value. Given Evaluation and
the significance of social media as an essential part of everyday business activities, decision
it is important to consider the attributes of these strategic marketing activities in the modern making in
digital economy.
Honing SMM requires evaluation, but it is evident that development of effective SMM
approaches to evaluation is not straightforward. Online conversations produce large
volumes of semantic data that present considerable challenges to any analysis of social 17
media activity (Larson and Watson, 2011). As such, an on-going debate exists surrounding
the extent to which social media metrics can be aligned with established digital and general
marketing metrics (Hennig-Thurau et al., 2010; Mangold and Faulds, 2009; Weber, 2009).
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
Methodology
Interview process
Since SMM, and more specifically its evaluation, are at a relatively early stage of
development with limited prior research, an exploratory study that adopted an inductive
MD approach was chosen for this research. This approach provided the opportunity to develop a
55,1 framework and gather deep insights into the actions and challenges embedded in the
evaluation of SMM. It also provided structure and flexibility to ensure the coverage of key
themes whilst accommodating unanticipated insights (Bryman and Bell, 2010; Saunders
et al., 2009; Kvale and Brinkmann, 2009).
The study used semi-structured interviews with key informants. The interview protocol
18 was informed by the relevant literature and was further refined through pilot interviews
with four practitioners to test rigour, validity and appropriateness (Bryman and Bell, 2010).
All questions were open-ended, thus not limiting the interviewee’s choice of answers
(Gubrium and Holstein, 2002) and were supplemented by prompts to ensure coverage of key
themes associated with each stage (Creswell, 2013). In-depth interviews were conducted
face-to-face in the informants’ offices, a setting where interviewees could elaborate and show
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
Sample
In identifying informants for this study, a purposive sampling approach was used to seek
out information-rich cases (Patton, 1990) with key informants who were able to comment
on current practice and experience in the evaluation of SMM. Such professionals have
considerable experience of SMM across a wide range of clients. In total, 18 specialist
marketers were interviewed (Table I), all of whom either had responsibility for SMM, or
more generally digital marketing within their agency. This number of informants is
consistent with other qualitative studies in this field (e.g. Veloutsou and Taylor, 2012;
Wallace and Chernatony, 2007).
Using agency practitioners as key informants provides broader insights into SMM
evaluation than would have been possible through direct conversations with brand owners.
The specialist agencies included ranged from multi-national marketing agencies servicing
global client brands, through to small and micro agencies with a UK client base, embracing
UK national, regional and sector-specific brands.
During the interview process, all informants referred to more than one client brand,
such that, in total, perspectives gathered during the interviews encompassed 78 brands, in
Informant Informant role title Size of agency Clients Typical client size
Clarke (2006): familiarisation with data, generating initial codes, searching for themes
among codes, reviewing themes, defining and naming themes and producing the final
account of the findings.
The data were initially analysed interview transcript by transcript, before checking for
verification across transcripts (Miles and Huberman, 1994). The analysis was completed
manually supporting a closeness to the data which allowed distinctive themes to emerge
and encouraged detailed knowledge of each theme (Eisenhardt, 1991). This analysis led to
the identification and emergence of the six stages of the framework, challenges and
responses to challenges associated with each stage (as shown in Figure 1 and Table II).
Findings
Figure 1 shows the stages of SMM evaluation that emerged from the interviews. It is
presented at this point to assist in structuring the details of this section, and was not
pre-determined before the interviews were conducted. In the remainder of this section,
insights offered on the challenges associated with each of these stages are presented.
Setting
Evaluation Identifying KPIs Identifying Metrics
Objectives
Figure 1.
Social media
Management Data Collection marketing evaluation
Report Generation
Decision Making and Analysis conceptual framework
MD Framework
55,1 stages Stage definition Challenges Responses to challenges
Setting Identification of specific and Lack of client understanding Workshop events to further
evaluation clear evaluation objectives, of social media as a the understanding of clients
objectives which support wider marketing channel knowledge of SMM in
marketing, and overall Ad or post hoc evaluation, practice
20 business goals without reference to specific
objectives
Identifying Identification of the most Vagueness regarding Examining previous
KPIs appropriate performance specific or appropriate KPIs campaign reports to clarify
indicators which support the Iterative target setting for performance indicators,
objectives and the campaign KPIs linked with successful
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
Objective setting starts with consideration of the wider business and marketing objectives
and seeks to identify and align appropriate SMM objectives:
The strategy would link the business objectives through their communication and marketing
objectives, to create social [media marketing] objectives (P3).
However, this process is far from straightforward. Several informants suggested that clients Evaluation and
exhibited difficulty in articulating their SMM objectives, due to their lack of understanding decision
of social media as a marketing channel: making in
At the moment, the client is not that digitally savvy and they are trying to rethink their own SMM
marketing plan and how they go about it (P4).
On occasions, this situation is resolved through meetings and negotiation:
21
What they would like is for us to sit down with them and have a strategy meeting where we talk
about what are the most important goals and talk about how we might theoretically go about
achieving them (P12).
It is evident that in this stage and other stages in the evaluation cycle, approaches vary
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
considerably between brands. Some clients and their agencies engaged in a thorough
pre-campaign process that included consideration of objectives:
We would have a workshop so that they can understand what they want and identify their aims
and objectives for their business (P3).
In other instances, planning of evaluation is more ad hoc or post hoc, with the agency taking
the initiative, often without discussion regarding the clients’ marketing objectives:
I will prepare a little mini report and say Facebook fans went from this to this (P4).
Clients expect a fairly basic report, measurement isn’t something they often want (P15).
Developing KPIs
Following on from setting evaluation objectives, identification of KPIs are imperative for
effective evaluation of a campaign. Informants were very aware that their agency was being
judged on their performance as measured by KPIs and hence very conscious of KPIs. As
such, discussion of KPIs constitutes a crucial stage in the establishment of the measurement
framework. Informants recognised the importance of KPIs; many informants referred to
setting KPIs, often linking them to the assessment of performance, and viewing the
achievement of KPIs as an indication of the agency’s value to the client:
[…] we pride ourselves on really knowing our clients; we know that their key KPIs are going to be
x, y and z (P5).
However, few informants could be prompted to further elaborate on specific KPIs. An exception
was P11, who mentioned specific KPIs, including engagement, reach and conversions:
If we have advertising running, how much does that increase our reach? I like to see what the
organic growth rate is like, what the engagement was like, how we sort of build it within the first
month, then, I can forecast how it is going to go for the rest of the campaign. We would set for
conversions, say month 1 in October, they got 50 social conversions.
For each KPI, a target is set. Given the relative limited previous experiences on use of SMM,
target setting can be difficult. For instance, this quote from P12 suggests that target setting
is not necessarily revised as campaigns evolve:
At the end of the year a lot of the targets were over-reached, as we started doing advertising
and competitions and more engaging stuff, so it became apparent that the targets didn’t actually
mean anything.
ROI, rather than KPIs was referred to by some informants:
{The Client} wanted to see the ROI on the (SMM) campaign and to a certain degree, we can say we
expect this to draw this many website visits (P4).
MD They {The Client} love ROI! People go into their website, making a booking which goes to sales
55,1 team. The average booking will normally generate £30, so that is the figure what I have been told to
work to (P16).
One campaign that was specifically designed to support calculation of ROI was mentioned:
We ran a Facebook offer, which we measured separately. That’s in-store redemption only, run at
one store in {Client Store}. We know how much we spent and how much was redeemed so it was
22 easy to run stats. There was a 14% redemption rate and it cost £2.50 per person that bought
something (P10).
Identifying metrics
Informants identified metrics such as the number of mentions, likes, and followers, which
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
Report generation
Once data collection is complete, reports are generated for clients. All agencies engaged with
this process as part of their contractual responsibilities. Reports are compiled of the various
metrics that it has been agreed with the client will be measured:
On a weekly basis for all clients, we will create a weekly set of metrics which includes follower,
social growth, web site traffic referred to from our social media activity, last click revenue (P3).
Informants viewed the reporting process as an important component of ensuring a
productive agency-client relationship. The nature and frequency of the process is shaped by
a combination of what the client requires, and the metrics available. For example:
With {Client}, everything has to go on an A3 piece of paper. They specify that with all your slides.
You have to say what you have to say within twenty seconds (P1).
MD {Client} are quite demanding and their requirements are very specific so that’s very helpful
55,1 because it does structure how we have to approach their reporting (P12).
There was some disagreement as to the optimum frequency of reporting with daily, weekly
and monthly reports being provided for clients. Frequency of reporting differed with the
type and size of the client. Interestingly, informants did not rely to any great extent on the
reporting functions of the metrics and analytics tools (that supported data collection), but
24 rather preferred to structure and format the report for the client. In this manner, it was easy
for them to identify key trends and outcomes, and in some cases to make the link to the
clients’ KPIs:
We don’t use anything that that just pulls the data for us because we have got an amazing
tech guy who built a report that pulls metrics plus KPIs so that everything updates
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
automatically (P11).
In most cases, SMM practitioners use statistical software, often Microsoft Excel, to combine
and distil the key information from the various data sets. Although report generation tools
are available, these were regarded as too expensive:
That technology is still to catch up and unless you can afford to buy one of the big tools that will do
the whole report for you and you can just print it at the end of the month (P11).
Some agencies created real-time dashboards for campaign performance figures for their
large clients such that the clients are able to interrogate the data sets themselves:
They do that by using the dashboards to pull the figures themselves. Every month there is a report
which they go in and type some information into and then the report generates itself ready on A3
format for them to save off as a (Adobe) .pdf and then they email that to the client to talk about at
their monthly meetings (P12).
In terms of the bigger clients we manage, we built digital dashboards which pull in through the
Twitter and the Facebook KPI. It has been running for about a year and a half using those metrics
that we agreed (P8).
Because reporting processes were part of a contractual arrangement with individual
clients, there was considerable variation in reporting practices and report formats, even
within one agency. Clients reporting requirements were seen to be heavily dependent on
their budget:
[…] it really depends on the client and how much they will pay for the evaluation (P8).
However, because reporting was viewed as a pivotal aspect of the contractual relationship
there was sometimes tension between the agency and their clients:
[Client] wanted everything quantified but I think they want us to give them a list of how much
traffic it will produce so that when it doesn’t they can beat us with it and not pay us (P4).
[The Client] wants monthly content plans with every piece of content. It is just absolutely crazy the
level of stuff they need (P9).
Some of them won’t even look at it […] they will circulate it in their office and no one will read
it (P13).
[…] we just send it {SMM evaluation Report} over and we get nothing back (P11).
The smaller SME’s that we work with, they aren’t really bothered, just say thanks and keep on
tweeting! (P13).
Summary
Table II summarises the findings, offering, based on the data from the interviews, a
definition of each stage of the framework, the challenges discussed and approaches adopted
for addressing those challenges.
challenges associated with each stage in the SMM evaluation cycle. Whilst some of the
challenges relate to practical considerations such as the availability of effective analytics
tools, the biggest challenges lie in the evolution of the relationship between the agency and
their clients, in a realm in which marketing and its evaluation continues to require learning
and adaptation on the part of both agencies and clients.
associated challenges are likely to remain relevant emerge in future SMM practice, there is a
general need for continuing research into social media strategies and their impact.
References
Aral, S., Dellarocas, C. and Godes, D. (2013), “Introduction to the special issue – social media and
business transformation: a framework for research”, Information Systems Research, Vol. 24
No. 1, pp. 3-13.
Bacile, T.J., Ye, C. and Swilley, E. (2014), “From firm-controlled to consumer-contributed: consumer
co-production of personal media marketing communication”, Journal of Interactive Marketing,
Vol. 28 No. 2, pp. 117-133.
Bolton, R. (2013), “Understanding Generation Y and their use of social media: a review and research
agenda”, Journal of Service Management, Vol. 24 No. 3, pp. 245-267.
Brady, H.E. and Collier, D. (Eds) (2010), Rethinking Social Inquiry: Diverse Tools, Shared Standards,
Rowman and Littlefield, Plymouth.
Braun, V. and Clarke, V. (2006), “Using thematic analysis in psychology”, Qualitative Research in
Psychology, Vol. 3 No. 2, pp. 77-101.
Bryman, A. and Bell, E. (2010), Business Research Methods, 4th ed., Oxford University Press,
New York, NY.
Charmaz, K. and Belgrave, L. (2002), Qualitative Interviewing and Grounded Theory Analysis,
Sage Publications, London.
Choi, Y. and Thoeni, A. (2016), “Social media: is this the new organizational stepchild?”, European
Business Review, Vol. 28 No. 1, pp. 21-38.
Creswell, J.W. (2013), Research Design: Qualitative, Quantitative, and Mixed Methods Approaches,
2nd ed., Sage Publications, London.
Cvijikj, I., Spiegler, E. and Michahelles, F. (2012), “Evaluation framework for social media brand
presence”, Social Network Analysis and Mining, Vol. 3 No. 4, pp. 1325-1349.
Davies, M. and Prince, M. (2010), “Advertising agency compensation, client evaluation and switching
costs: an extension of agency theory”, Journal of Current Issues & Research in Advertising,
Vol. 32, January, pp. 13-31.
Eisenhardt, K. (1991), “Better stories and better constructs: the case for rigor and comparative logic”,
Academy of Management Review, Vol. 16 No. 3, pp. 620-627.
Fan, W. and Gordon, M. (2014), “The power of social media analytics”, Communications of the ACM,
Vol. 57 No. 6, pp. 74-81.
Fisher, T. (2009), “ROI in social media: a look at the arguments”, Journal of Database Marketing and
Customer Strategy Management, Vol. 16 No. 3, pp. 189-195.
Goh, K.Y., Heng, C.S. and Lin, Z. (2013), “Social media brand community and consumer behavior:
quantifying the relative impact of user- and marketer-generated content”, Information Systems
Research, Vol. 24 No. 1, pp. 88-107.
Grant, I., McLeod, C. and Shaw, E. (2012), “Conflict and advertising planning: consequences of Evaluation and
networking for advertising planning”, European Journal of Marketing, Vol. 46 Nos 1/2, pp. 73-91. decision
Gubrium, J.F. and Holstein, J.A. (2002), Handbook of Interview Research: Context and Method, Sage, making in
London.
SMM
Gummerus, J., Liljander, V., Weman, E. and Pihlström, M. (2012), “Customer engagement in a Facebook
brand community”, Management Research Review, Vol. 35 No. 9, pp. 857-877.
Hanna, R., Rohm, A. and Crittenden, V.L. (2011), “We’re all connected: the power of the social media 29
ecosystem”, Business Horizons, Vol. 54 No. 3, pp. 265-273.
Haytko, D. (2004), “Firm-to-firm and interpersonal relationships: perspectives from advertising agency
account managers”, Journal of the Academy of Marketing Science, Vol. 32 No. 3, pp. 312-328.
Heijnen, J., de Reuver, M., Bouwman, H., Warnier, M. and Horlings, H. (2013), “Social media data
Downloaded by University of British Columbia Library At 05:10 03 January 2019 (PT)
For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: [email protected]
This article has been cited by: