Employing a social media

monitoring tool as an OSINT

platform for Intelligence,
Defence & Security
BY PAUL BRELSFORD

www.talkwalker.com
 TABLE OF CONTENT

1. Foreword
2. Introduction
3. The Intelligence Cycle:
Modern Intelligence Tradecraft
4. The Intelligence Cycle:
4.1. Direction – Stage 1
WHO IS PAUL BRELSFORD? 4.2. Collection – Stage 2
Paul served for 23 years in 4.3. Processing – Stage 3
the UK’s Military Intelligence.
He is counter-terrorism and 4.4. Dissemination – Stage 4
OSINT professional with global
experience. Paul is an expert in
Intelligence Management and
5. Looking Forward
analytical processes and was
awarded the Intelligence Corps
distinguished service award
6. Summary
for his work in establishing the
UK LAND Forces Intelligence 7. Glossary
Information Management
section within the UK LAND
Forces Intelligence Fusion
Centre. Paul now advises
on counterterrorism and
intelligence management
issues.

UK.LINKEDIN.COM/IN/DECISIONSUPPORTSPECIALIST/
 FOREWORD
This whitepaper discusses how social media monitoring tools
can be applied as powerful and cost effective Open Source
Intelligence (OSINT) platforms; and how they can support
collection and analysis of relevant and targeted information
relating to counter-terrorism, criminal and political open
sources. The use of such tools has wide application and
benefits for a number of industries, including; Government and
Defence Intelligence agencies, Law Enforcement, Commercial
Risk Management companies, Private Security Companies
(PMCs) and Non-Government Organisations (NGOs).

Over the last 6 months of 2013 a proof of concept exercise

was undertaken. This exercise was conducted by a
counterterrorism and OSINT expert to test the application of
Talkwalker as an OSINT platform in the role of supporting the
intelligence cycle and to provide a monitoring function as an
Early Warning System (EWS). The capabilities and findings
are discussed within this paper.

EMPLOYING SOCIAL MEDIA MONITORING TOOLS AS AN OSINT PLATFORM FOR INTELLIGENCE, DEFENCE & SECURITY 3
2. INTRODUCTION

Conducting OSINT for wider intelligence, Many readers will be familiar with the analogy
counter-terrorism and risk management that getting information from the internet is
work has become a complex and increasingly like taking a drink from a fire hydrant. This
resource intensive task for both Government feature of information discovery, management
and Defence Intelligence agencies and the and curating presents difficult challenges in
commercial risk management sector alike. terms of the allocation of resources, personnel
This paper covers some of the challenges and time constraints all combined with the
involved in the collection and processing of challenge of being able to filter out the noise
OSINT and how a social media monitoring tool and irrelevant information from the river of
can exponentially enhance those processes information so that the relevant information
when implemented as an OSINT platform. can surface to be discovered, processed and
Moreover, not only do such tools enhance delivered in a timely manner for action by
OSINT collection and processing but it also decision makers.
provides a vastly increased cost/benefit
versus allocated resources ratio through a Within Government and Defence intelligence
tool’s capability to crawl, filter and deliver agencies OSINT work has gained momentum
increasingly relevant results from huge to become recognised as a legitimate area
quantities of data on daily basis, the results of intelligence operation alongside the more
of which are delivered via a consumable and traditional intelligence domains; such as
easy to manage interfaces. These capabilities HUMINT (agent handling) and SIGINT (signals
allow for the delivery of focussed results intelligence) disciplines.
from a larger pool of raw information in a
shorter timeframe. Concurrent to the delivery This is particularly true in the domain of
of results the overall processes requires counter-terrorism. Nearly all Government and
less resources and personnel to manage and Defence intelligence agencies have resources
exploit the information in intuitive interfaces dedicated to the production of OSINT within
that have a virtually non-existent training the framework of the intelligence cycle in
burden to analysts. order to meet their intelligence requirements
and to produce actionable outputs.

4
3. THE INTELLIGENCE CYCLE -
Modern Intelligence Tradecraft

Supporting the intelligence cycle in the

direction, collection, processing and
dissemination of actionable intelligence is
key in the discovery and validation of critical
and relevant information from OSINT sources.
This process requires much more detailed
focus than blindly surfing the internet for
clues or merely typing relevant or target
oriented search queries into a search engine.

Moreover, even conducting manual cursory

Continuous review
trawls for adversarial social media outputs
or other online channels is likely to provide
little value without the technology to cast and
manage a wide net across the online space.

The modern intelligence tradecraft of

exploiting OSINT requires the assistance of
technology that will enhance the precision,
relevance and speed in the collection and
validation of meaningful insight from online
sources.

EMPLOYING SOCIAL MEDIA MONITORING TOOLS AS AN OSINT PLATFORM FOR INTELLIGENCE, DEFENCE & SECURITY 5
4. THE INTELLIGENCE CYCLE

THE 4 STAGES OF THE INTELLIGENCE CYCLE

The intelligence cycle is comprised of 4 key

stages (5 in the USA) with continuous review
of each stage conducted throughout the
process. For the purposes of this paper the
UK model has been adopted to explain the
framework, which consists of the components
below:
• Direction
• Collection
• Processing
• Dissemination

The intelligence cycle is the classic

intelligence management methodology
and doctrinal tradecraft utilised for the
production of intelligence to support decision
WHAT IS THE INTELLIGENCE CYCLE? makers. The chapters below discusses how
METHODOLOGY & APPLICATION a social media monitoring tool can support
each phase of the intelligence cycle as a
powerful and significant OSINT platform with
The intelligence cycle is the framework and application beyond its conceptual design of
methodology used by modern intelligence supporting brand protection and delivering
agencies and professional intelligencers for competitive intelligence insights.
the production of intelligence outputs and
reporting to support decision making. It is a
proven concept that in more recent years has
also been adopted across many commercial
industries that have a requirement for
information management and discovery to
support business processes and decision
making. The basic tenet of the process is
that information requirements are prioritised
and focused against the decision maker’s
objectives. This allows resources to be
matched and allocated accordingly for the
collection and production of intelligence.

6
4.1 DIRECTION
Stage 1 of the Intelligence Cycle
Direction is the first stage of the intelligence A series of terrorism watchpoints was
cycle. This is where the decision makers designed to provide support in answering
define and prioritise their (business or specific information requirements and
mission) objectives. These requirements to provide early warning of developing
are then captured and form the basis of the threat streams from horizon scanning.
Intelligence Collection Plan (ICP), allowing the Watchpoints were based on a target series
analysts to design a series of highly focussed of known terrorist groups and threat actors
information requirements to answer specific to meet the intelligence requirements of the
questions or to maintain a horizon scanning decision makers. This was achieved through
(early warning system) for information that is setting defined search query parameters,
critical in supporting decision making within established to collect against known terrorist
the prioritised business or mission objectives. organisations and associated persons of
The following describes how a social media interest; and to monitor terrorism related
monitoring platform was employed in a event indicator query terms. The watchpoint
counter-terrorism role. query terms were generated in a spreadsheet
and the specific information requirements
to be answered were itemised. This formed
the basis of the collection plan, the ICP which
was then imported and implemented on
Talkwalker.
TERROR GROUP WATCHPOINTS:

EMPLOYING SOCIAL MEDIA MONITORING TOOLS AS AN OSINT PLATFORM FOR INTELLIGENCE, DEFENCE & SECURITY 7
4.2 COLLECTION
Stage 2 of the Intelligence Cycle
Once a collection plan was implemented on The key benefits of this initial multi-profile
the social media monitoring tool. A series of design was to provide a wide ‘capture all’
profiles were setup to meet the intelligence feature that could act as a total information
collection plan requirements. The first profile repository to be subsequently exploited or
was structured as a generic intelligence analysed at any point in order to provide deep
collection watchpoint of about 20 terrorist current understanding of events or to look
organisations. This defined the list of terror back on historical events.
groups to be collected against (as illustrated
above). The ICP was implemented directly from the
spreadsheet import function, a time saving
A secondary profile was then implemented and easy process in itself. Once implemented
with a list of known terror suspects the social media monitoring tool then began
(associated Persons of Interest (POI)) to to produce results from crawling and indexing
be collected against. The POI were known hundreds of millions of online sources per
terrorist suspects with links to terror day.
organisations listed in the ICP. This second
series of watchpoints allowed for the In summary, the following Watchpoints were
building of a deeper layer of linked, relevant established for collection, each Watchpoint
information to be collected. This profile was capable of independent research and
allowed for collection of information linked analysis but had greater value through the
with the terror group watchpoints. The ability to define patterns and conduct cross-
scale of collection from Talkwalker profiles referencing and deep link analysis between
exponentially increased in volume daily. all three Watchpoints:

A third profile was developed as a set of • Terrorist Groups & organisations.

potential event indicators based on very • Known Persons of Interest (POI)
specific positive search query criteria and associated to terrorist groups.
negative search query criteria. This watchpoint • Terrorist incident event indicators.
was designed to add a third layer of relevant
information, which could be linked and cross
referenced with the previous watchpoints.
This profile also monitored pattern changes
in these key terms, which could potentially
surface signals that would highlight new
or building threat streams. In essence this
combined and layered watchpoint approach
enabled the development of an intelligence
picture and the provision of live situational
awareness.

8
SURFACING RELEVANT INFORMATION RHYTHMS AND INTELLIGENCE

SOURCE VALIDATION AND INTEGRITY

Another key component to the collection stage The source management functionality should
of the intelligence cycle is ensuring source not be underestimated and was a ‘force
validation and integrity. This was achieved multiplier’ in providing and refining results
in combination by the selection and refining and sources with a high level of confidence.
of negative search query terms to filter out Over time this builds up into a hugely powerful
the irrelevant results containing these terms. capability in its own domain. For example,
The ability to block persistent sources which by developing graded sources within each
delivered irrelevant information; and through profile or Watchpoint the analyst is essentially
fine tuning and grading positive search developing a high quality reporting and
queries which returned the collected results. monitoring function based on graded and
judged sources with a higher level of integrity.
The simple source management dropdown This high grade source list can be segmented
menu is also a highly efficient capability separately to run against search queries. In
within social media monitoring tool used. essence, this method removes a high volume
This enabled users to grade or block sources. of noise to leave highly focussed and relevant
In addition, sources can be emailed, custom results for further analysis.
tagged or priority marked. They can then be
filtered for subsequent analysis based on In addition to the above functionality other
their grading. This functionality and process sources and files can be added to each
adds a high level of confidence to the source project. External files and documents may
integrity as key sources are subjected to first also be uploaded instantly for indexing with
line interrogation and grading by an analyst. the developing database.

EMPLOYING SOCIAL MEDIA MONITORING TOOLS AS AN OSINT PLATFORM FOR INTELLIGENCE, DEFENCE & SECURITY 9
 THE BENEFITS FOR ANALYSTS IN THE COLLECTION STAGE

With the assistance of a social media monitoring tool, analysts

are no longer physically limited to the number of online sources
and searches that could be conducted on a daily basis through
manual processes. Not only does it discover the content but
it allows the vast array of content to be instantly searched,
segmented and analysed. A further benefit of such tool are that
they can automatically weed out much of the duplication in the
results served and provide a plethora of new and previously
undiscovered relevant sources and material, which would not
surface from manual or semi-automated search efforts.

What does this mean for the collection process within the
intelligence cycle? Using social media monitoring tools
provides fast, relevant and intuitive platforms to load and align
highly defined collection efforts based on the objectives and
requirements of decision makers.

10
4.3 PROCESSING
Stage 3 of the Intelligence Cycle
With 4 million records (and growing) the This type of powerful functionality allows
possibilities of analysis, external to the social analysts to process and easily manage large
media monitoring tool you use are vast. volumes of data in a very short timeframe
Building in depth Social Network Analysis with ability to separate out highly relevant
(SNA) charts, link analysis between POI and results from a substantially bigger repository
terrorist organisations, and geographic and of information than could be achieved from
threat financial linkages has also become an manual or semi-automated processes.
achievable concept.
From a counter-terrorism viewpoint this
However, utilising social media monitoring enabled the rapid segmentation of POI and
tools as OSINT platforms continues beyond groupings as required for subsequent analysis.
the collection stage. With more than a million At the time of writing a significant number of
records to analyse users may require an networks and links have been developed by
external database due to the limitations of building Social Network Analysis diagrams
volume that spreadsheets such as Excel can based on collected data within the results
manage. that have been returned in the Watchpoints.

Ideally, the tool must have the ability to In addition, patterns and trends have also
interrogate the results and drill-down deep emerged in the both the cycle and sources
into the information in an extremely easy of release of information into the OSINT
and intuitive way from the interface as was environment. This pattern analysis allows
the case with Talkwalker. Data should be able analysts to further refine and target collection
to be segmented by a number of different that would not be possible without the use
attributes; such as source type and grade, of technology such as that provided by the
language, region, country, date. The data can Talkwalker interface. Snapshots of the ease
be further segmented by selecting individual with which results can be processed and
Watchpoints, such as POI or by source analysed within the Talkwalker interface to
(collection) channel (Facebook, Twitter, Blog, provide an intelligence picture are highlighted
YouTube etc.). The data can also be segmented on the next page.
by Boolean search, keywords and date range.

EMPLOYING SOCIAL MEDIA MONITORING TOOLS AS AN OSINT PLATFORM FOR INTELLIGENCE, DEFENCE & SECURITY 11
 MONITORING PERSONS OF INTEREST
BUILDING THE INTELLIGENCE PICTURE FROM The simplicity in which events can be
OSINT
visualised and interrogated is one of
The intelligence picture above shows the Talkwalker’s key strengths, allowing
tempo and rhythm of terrorist related rapid analysis and understanding of
events. The snapshot above is based on events that take place. Moreover, by
over 2.7million results and shows the setting Watchpoints to monitor persons
tempo, rhythm and effect of terrorist of interest (POI) any increase in activity
events between August & December related to those being monitored may
2013. Of note this strategic picture can be instantly viewed in a separate graph
be exploited in the future to understand as an overlay.
and anticipate terrorism related events
and activity to mitigate their impact. The POI overlay below was instantly
generated by selecting POI Watchpoint
from the search query column in the
intelligence picture interface below.

12
THE WESTGATE MALL ATTACK IN KENYA

Further illustration of the image below depicts Analysis of this segment of data also leads
a clear and expected spike in OSINT by the to further discovery and identification of
predicted surge in media and online user source types that may be of further interest.
generated content following a major terrorist An individual, even if using false credentials
or crisis event. In this case, the Westgate Mall online will often have published other
attack in Kenya by Al Shabaab in September material which divulges essential elements of
2013. information or links to other online sources or
individuals of interest that feed into building
By analysing the results surrounding this an increasingly accurate intelligence picture
surge in activity new leads and sources could and horizon scanning function.
be identified from within the content. Persons
who generate content which indicates inside
knowledge of the event or being close to a
POI, or those who have connections to terror
groups can be isolated through analysis of
this data for subsequent addition into social
network analysis (SNA) and link analysis.

EMPLOYING SOCIAL MEDIA MONITORING TOOLS AS AN OSINT PLATFORM FOR INTELLIGENCE, DEFENCE & SECURITY 13
EARLY WARNING SYSTEMS

The tool can also collect against identifying

threat streams at the tactical level, allowing
for quick analysis and early warning.

The example below was identified against

query terms for Al Shabaab, Al Mujahideen.
On its own such open messaging may not raise
significant interest. However, when combined
with other specific OSINT signals following the
Al Shabaab attack on the Westgate shopping
Mall and the pressure faced by Al Shabaab
from Kenyan forces in Somalia, it raises the
probability that this is a developing threat
stream surfacing the intent of Al Shabaab to
cultivate an insider threat against the Kenyan
government.

14
4.4 DISSEMINATION
Stage 4 of the Intelligence Cycle
Social media monitoring offers a number of In sum, using a social media monitoring
different methods to distribute results and solution can significantly speed up the process
analysis. Alerts can be setup to immediately of dissemination by packaging and aligning
disseminate results from the Watchpoints the collection results for export to database,
or search query sets of interest that may email, PDF, Word or to other analytical toolsets
be considered as triggers for information and reporting formats.
relevant requirements.

Export functions have the capability to segment

results according to the requirements of
the analyst and to generate the report into
a number of exportable and consumable
formats. Results may then be exported as CSV
datasets, which can be imported into external
databases or external analytical toolsets for
subsequent exploitation. Reports may also be
exported in Word and PDF formats for use as
standalone reports or for inclusion in other
reporting formats.

EMPLOYING SOCIAL MEDIA MONITORING TOOLS AS AN OSINT PLATFORM FOR INTELLIGENCE, DEFENCE & SECURITY 15
 LOOKING FORWARD
Prediction and the art of the future. At the time
of writing testing continues to extrapolate
emerging patterns based on designated query
term sets and phrases, which are common
signals before terrorist events occur. This
may highlight signals of changes in the online
environment, such as a growing threat stream,
or changes in tactical or strategic objectives
of terror groups, or even the ability to predict
an event in advance.

Watch this space!

16
6. SUMMARY

This white paper illustrated the relevance The functionality of social media monitoring
and efficiency of employing a social media seamlessly integrates to support the
monitoring tool application as an OSINT requirements of each stage of the intelligence
platform in support of the Intelligence Cycle. cycle. The implementation of collection plans
Whilst the proof of concept was focussed and Watchpoints and the delivery of relevant,
within the counter-terrorism domain it has graded results with a highly intuitive source
equal application to other sectors; such as, management capability.
law enforcement, NGOs, risk management
and private security companies. Results can be layered and linked as overlays
and datasets may be rapidly segmented or
The implementation of such tools dramatically analysed using powerful drill-down options
enhances the capability to discover, curate in the interface. The distribution of the
and analyse vast volumes of information results and analysis is easily disseminated
to provide monitoring and early warning of in multiple formats and alerts settings allow
developing threat streams. In addition, it also dissemination of reports either across an
delivers deep datasets with rich, relevant organisation or direct to the key decision
content and sources for further processing makers in a timely and efficient manner.
to develop significant social network analysis
on POI and link analysis charts of threat From an organisation viewpoint, a social media
organisations. monitoring tool is a resource multiplier. It is
cost effective, requiring less time to process
more actionable insights which are derived
from a much wider set of results than can be
processed manually or by semi-automated
approaches.

EMPLOYING SOCIAL MEDIA MONITORING TOOLS AS AN OSINT PLATFORM FOR INTELLIGENCE, DEFENCE & SECURITY 17
14, rue Aldringen
L-1118 Luxembourg
Luxembourg

+352 20 33 35 32
[email protected]
www.talkwalker.com