csenotescorner.blogspot.
com UNIT III - CS6010 SNA
UNIT III EXTRACTION AND MINING COMMUNITIES IN WEB SOCIAL
NETWORKS
Extracting evolution of Web Community from a Series of Web Archive - Detecting
communities in social networks - Definition of community - Evaluating communities - Methods
for community detection and mining - Applications of community mining algorithms - Tools for
detecting communities social network infrastructures and communities - Decentralized online
social networks - Multi-Relational characterization of dynamic social network communities.
3.1 Extracting evolution of Web Community from a Series of Web Archive
The extraction of Web community utilizes Web community chart A graph of communities, in
which related communities are connected by weighted edges. The main advantage of the Web
community chart is existence of relevance between communities.
3.1.1 Notations Used
t1, t2, ..., tn: Time when each archive crawled. Currently, a month is used as the unit
time.
W(tk): The Web archive at time tk.
C(tk): The Web community chart at time tk.
c(tk), d(tk), e(tk), ...: Communities in C(tk).
Types of Changes
Emerge
◦ A community c(tk) emerges in C(tk), when c(tk) shares no URLs with any
community in C(tk−1).
Dissolve
◦ A community c(tk−1) in C(tk) has dissolved, when c(tk−1) shares no URLs with
any community in C(tk)
Growth and Shrink
◦ The community grows when new URLs are appeared in c(tk), and shrinks when
URLs disappeared from c(tk−1).
Split
◦ c(tk−1) shares URLs with multiple communities in C(tk)
1 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
Merge
◦ When multiple communities (c(tk−1)), d(tk−1), ...) share URLs with a single
community e(tk), these communities are merged into e(tk)
Evolution Metrics
Evolution metrics measure how a particular community c(tk) has evolved. The metrics are
defined by differences between c(tk) and its corresponding community c(tk−1).
Growth Rate
The growth rate, Rgrow(c(tk−1), c(tk)), represents the increase of URLs per unit time. It allows
us to find most growing or shrinking communities.
Stability
Represents the amount of disappeared, appeared, merged and split URLs per unit time. A stable
community on a topic is the best starting point for finding interesting changes around the topic.
Disappearance rate
The number of disappeared URLs from c(tk−1) per unit time. Higher disappear rate means that
the community has lost URLs mainly by disappearance.
Merge rate
2 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
The number of absorbed URLs from other communities by merging per unit time. Higher
merge rate means that the community has obtained URLs mainly by merging.
Split Rate
The split rate, Rsplit (c(tk−1), c(tk)), is the number of split URLs from c(tk−1) per unit time.
When the split rate is low, c(tk) is larger than other split communities. Otherwise, c(tk) is smaller
than other split communities.
Other Metrics
The novelty metrics of a main line (c(ti), c(ti+1), ..., c(t j)) is calculated as follows.
Web Archives and Graphs
Web archiving is the process of collecting portions of the Web to ensure the information
is preserved in an archive
Web crawlers are used for automated capture due to the massive size and amount of
information on the Web.
From each archive, a Web graph is built with URLs and links by extracting anchors from
all pages in the archive.
The graph included not only URLs inside the archive, but also URLs outside pointed to
by inside URLs.
By comparing these graphs, the Web was extremely dynamic
3 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
The size distribution of communities also follows the power law and its exponent did not change
so much over time. Although the size distribution of communities is stable, the structure of
communities changes dynamically. The structure of the chart changes mainly by split and merge,
in which more than half of communities are involved.
Split and Merged Communities
Both distributions roughly follow the power law, and show that split or merge rate is
small in most cases.
Their shapes and scales are also similar.
This symmetry is part of the reason why the size distribution of communities does not
change so much.
Emerged and Dissolved Communities
The size distributions of emerged and dissolved communities also follow the power law
Contribute to preserve the size distribution of communities.
Small communities are easy to emerge and dissolve
Growth Rate
The growth rate is small for most of communities, and the graph has clear y-axis
symmetry.
size distribution of communities is preserved over time.
Combining evolution metrics and relevance, evolution around a particular community can be
located. The size distribution of communities followed the power-law, and its exponent did not
change so much over time.
3.2 Detecting communities in social networks
Detecting communities from given social networks are practically important for the following
reasons:
1. Communities can be used for information recommendation because members of the
communities often have similar tastes and preferences. Membership of detected communities
will be the basis of collaborative filtering.
4 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
2. Communities will help us understand the structures of given social networks. Communities are
regarded as components of given social networks, and they will clarify the functions and
properties of the networks.
3. Communities will play important roles when we visualize large-scale social networks.
Relations of the communities clarify the processes of information sharing and information
diffusions, and they may give us some insights for the growth the networks in the future.
3.3 Definition of community
The word “community” intuitively means a subnetwork whose edges connecting inside of it
(intracommunity edges) are denser than the edges connecting outside of it (intercommunity
edges). Definitions of community can be classified into the following three categories.
Local definitions
Global definitions
Definitions based on vertex similarity.
Local definitions
The attention is focused on the vertices of the subnetwork under investigation and on its
immediate neighborhood. Local definitions of community can be further divided into
self-referring ones and comparative ones.
The former considers the subnetwork alone, and the latter compares mutual connections
of the vertices of the subnetwork with the connections with external neighbors.
The examples of self referring definitions are clique (a maximal subnetworks where each
vertex is adjacent to all the others), n-clique (a maximal subnetwork such that the
distance of each pair of vertices is not larger than n), and k-plex (a maximal subnetwork
such that each vertex is adjacent to all the others except at most k of them).
The examples of comparative definitions are LS set (a subnetwork where each vertex
hasmore neighbors inside than outside of the subnetwork), and weak community (the
total degrees of the vertices inside the community exceeds the the number of edges lying
between the community and the rest of the network)
Global definitions
Global definitions of community characterize a subnetwork with respect to the network
as a whole. These definitions usually starts from a null model, in another words, a
5 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
network which matches the original network in some of its topological features, but
which does not display community structure.
The simplest way to design a null model is to introduce randomness in the distribution of
edges among vertices.
The most popular null model consists of a randomized version of the original network,
where edges are rewired at random under the constraint that each vertex keeps its degree.
This null model is the basic concept behind the definition of modularity.
Definitions Based on Vertex Similarity
Definitions of the last category is based on an assumption that communities are groups of
vertices which are similar to each other. Some quantitative criterion is employed to evaluate the
similarity between each pair of vertices. Similarity measures are at the basis of themethod of
hierarchical clustering. Hierarchical clustering is a way to find several layers of communities that
are composed of vertices similar to each other.
Repetitive merges of similar vertices based on some quantitative similarity measures will
generate a structure shown in Fig. 3.a. This structure is called dendrogram, and highly similar
vertices are connected in the lower part of the dendrogram. Subtrees obtained by cutting the
dendrogram with horizontal line correspond to communities. Communities of different
granurality will be obtained by changing the position of the horizontal line
Figure 3.a Dendrogram
3.4 Evaluating Communities
It is necessary to establish which partition exihibit a real community structure. Therefore, a
quality function for evaluating how good a partition is needed. The most popular quality function
is the modularity of Newman and Girivan:
6 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
where the sum runs over all pairs of vertices, A is the adjacency matrix, k i is the degree of vertex
i and m is the total number of edges of the network.
Modularity can be rewritten as follows:
where nm is the number of communities, ls is the total number of edges joining vertices of
community s, and ds is the sum of the degrees of the vertices of s.
The first term of each summand is the fraction of edges of the network inside the community,
whereas the second term represents the expected fraction of edges that would be there if the
network were a random network with the same degree for each vertex.
Figure 3.b illustrates the meaning of modularity.
Figure 3.b Modularity
The latter formula implicitly shows the definition of a community: a subnetwork is a community
if the number of edges inside it is larger than the expected number in modularity’s null model.
The modularity of the whole network, taken as a single community, is zero. Modularity is always
smaller than one, and it can be negative as well.
3.5 Methods for community detection and mining
There are naive methods for dividing given networks into subnetworks, such as graph
partitioning, hierarchical clustering, and k-means clustering. The methods for detecting
communities are roughly classified into the following categories:
7 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
(1) divisive algorithms
(2) modularity optimization
(3) spectral algorithms and
(4) other algorithms.
Divisive Algorithms
A simple way to identify communities in a network is to detect the edges that connect vertices of
different communities and remove them, so that the communities get disconnected from each
other.
The steps of the algorithm are as follows:
(1) Computation of the centrality of all edges,
(2) Removal of edge with largest centrality,
(3) Recalculation of centralities on the running network, and
(4) Iteration of the cycle from step (2).
Edge betweenness is the number of shortest paths between all vertex pairs that run along the
edge.
Modularity Optimization
Modularity is a quality function for evaluating partitions. Therefore, the partition corresponding
to its maximum value on a given network should be the best one. This is the main idea for
modularity optimization. It has been proved that modularity optimization is an NPhard problem.
However, there are currently several algorithms that are able to find fairly good approximations
of the modularity maximum in a reasonable time. One of the famous algorithms for modularity
optimization is CNM algorithm. Another examples of the algorithms are greedy algorithms and
simulated annealing.
Spectral Algorithms
Spectral algorithms are to cut given network into pieces so that the number of edges to be cut
will be minimized. One of the basic algorithm is spectral graph bipartitioning. The Laplacian
matrix L of a network is an n * n symmetric matrix, with one row
8 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
and column for each vertex. Laplacian matrix is defined as L =D - A , where A is the adjacency
matrix and D is the diagonal degree matrix with
All eigenvalues of L are real and non-negative, and L has a full set of n real and orthogonal
eigenvectors. In order to minimize the above cut, vertices are partitioned based on the signs of
the eigenvector that corresponds to the second smallest eigenvalue of L. In general, community
detection based on repetative bipartitioning is relatively fast.
Other Algorithms
There are many other algorithms for detecting communities, such as the methods focusing on
random walk, and the ones searching for overlapping cliques.
3.6 Applications of community mining algorithms
Some applications of community mining, with respect to various tasks in social network analysis
are listed below:
Network Reduction
Network reduction is an important step in analyzing social networks. The example discussed
here is taken from the work in which the network was constructed from the bibliography of the
book entitled “graph products: structure and recognition”. The bibliography contains 360 papers
written by 314 authors. Its corresponding network is a bipartite graph, in which each node
denotes either one author or one paper, and link (i, j) represents author i publishing a paper j.
Community structure is detected using a community mining algorithm called ICS. Each
community contains some papers and their corresponding coauthors. Most of the detected
communities are self-connected components.
Moreover, the clustered coauthor network can be reduced into a much smaller one by
condensing each community as one node. Finally, the top-level condensed network
corresponding to a 3-community structure is constructed by using ICS from the condensed
network. From this a dendrogram corresponding to the original coauthor network can be built.
Discovering Scientific Collaboration Groups from Social Networks
9 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
This section show how community mining techniques can be applied to the analysis of scientific
collaborations among researchers. Flink is a social network that describes the scientific
collaborations among 681 semantic Web researchers (http://flink.semanticweb.org/).
The network was constructed based on semantic Web technologies and all related semantic
information was automatically extracted from “Web-accessible information sources”, such as
“Web pages, FOAF profiles, email lists, and publication archives”. The weights on the links
measure the degrees of collaboration.
Mining Communities from Distributed and Dynamic Networks
Many applications involve distributed and dynamically-evolving networks, in which resources
and controls are not only decentralized but also updated frequently. One promising solution is
based on an Autonomy-Oriented Computing (AOC) approach, in which a group of self-
organizing agents are utilized. The agents will rely only on their locally acquired information
about networks.
Intelligent Portable Digital Assistants (or iPDAs for short) that people carry around can form a
distributed network, in which their users communicate with each other through calls or
messages. One useful function of iPDAs would be to find and recommend new friends with
common interests, or potential partners in research or business, to the users.
The way to implement it will be through the following steps:
(1) based on an iPDA user’s communication traces, selecting individuals who have frequently
contacted or been contacted with the user during a certain period of time;
(2) taking the selected individuals as the input to an AOC-based algorithm.
(3) ranking and recommending new persons who might not be included the current acquaintance
book, the user.
In such a way, people can periodically receive recommendations about friends or partners from
their iPDAs.
10 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
3.7 Tools for detecting communities social network infrastructures and
communities
Several tools have been developed for detecting communities. These are roughly classified into
the following two categories: detecting communities from large-scale networks, and interactively
analyzing communities from small networks.
Graph modelling language (GML) is one of the formats for representing networks. The following
shows the processes of community detection of R+igraph. Explanation of each command are
shown after #.
Initial positions of vertices are assigned randomly, and users can select major algorithms for
visualization. Positions of vertices can be adjusted manually.
The command fastgreedy.community is for maximizing modularity greedily, and its results are
stored in variable gr. The variable gr is composed of gr$modularity (a list of modularity values in
the process of maximization) and gr$merge (vertex IDs that are merged at each step).
The command communty.to.membership generates m$membership (membership of each vertex)
and m$csize (size of each community). This example shows that the first eight vertices of lower
numbers belong to community 0 and the sizes of the two communities are both 17. After storing
the membershop m$membership to V(g)$color, the second tkplot command specifies a
visualization algorithm and colors of vertices in order to perform visualization.
The command edge.betweenness.community detect communities by repetitively removing edges
of high edge betweenness. The contents of the obtained variable ed is different from previous
fastgreedy.community: ed$removed.edges (list of removed edges), ed$edge.betw eenness (list of
edge betweenness), ed$merges (list of vertices that aremerged in a dendrogram), and ed$bridges
(list of bridging edges). Edge betweenness divides given network in an top-down manner, while
ed$merges shows list of vertices in a reversed order.
11 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
3.8 Decentralized online social networks
Online social network (OSN) is an online platform that
(1) provides services for a user to build a public profile and to explicitly declare the
connection between his or her profile with those of the other users;
(2) enables a user to share information and content with the chosen users or public; and
(3) supports the development and usage of social applications with which the user can
interact and collaborate with both friends and strangers.
Current online social networks are extended in two main directions towards the capabilities of
the provided services and the decentralization of the supporting infrastructures, as depicted in
Fig. 3.c
Fig 3.c Classification and development trend of online social network services
A decentralized online social network is an online social network implemented on a distributed
information management platform, such as a network of trusted servers or a peer-to-peer
systems. In contrast to centralized OSNs where the vendor bears all the cost in providing the
services, a distributed or peer-to-peer OSN offers a cost-effective alternative.
12 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
3.8.1 Challenges for DOSN
The challenges for decentralized social networks are listed below:
Storage – The requirement for redundancy to provide availability of data depends to a large
extent on the duration and distribution of time peers are online. These activity patterns are also
influenced by the geographic distribution of the peers and shifted by time zones. The distribution
of interested and authorized peers and the desired probability of availability are to be traded off
with storage requirements, especially if the system should allow for storing of media files and
not only links to websites where such media files can be found
Updates – In peer collaboration systems, updates, e.g., of a workplace, are sent to a small group
of peers via a decentralized synchronization mechanism. In P2P social networks, with distributed
storage and replication – and a potential need for scalability, the requirements change. P2P
publish/subscribe mechanisms are a possibility, but their security in terms of access control will
have to be developed further.
Topology – In pure file-sharing networks, the topology does not depend on whether the peers
know each other and nodes exchange content with any other nodes in the network. At the other
end of the spectrum, existing examples of decentralized social networks (in the widest sense)
aremostly platforms for collaboration or media sharing and they tend to consist of collaborative
groups that are relatively closed circles, e.g., using a “ring of trust” or darknets. In contrast,
online social networking services have overlapping circles.
Search, Addressing – Over multiple sessions, peers may change their physical address. In a
typical file sharing network, this is not an issue. One just needs to find some peer with the
content it is looking for. Traditionally, peer identity is tied with an IP address which clearly is
not sufficient.
In social networks, tagging or folksonomies is the basic mechanism to annotate content.
Recently, there have also been advances made in enabling decentralized tagging, which paves
another step towards realizing social networks on top of a P2P infrastructure.
13 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
Openness to New Applications – One of the most alluring features of current online social
networks is that they are open to third-party applications, which enables a constant change of
what a social networking service provides to the users.
In addition, third-party applications provide more and unpredictable ways of contacting users,
finding out about other users’ interests, forming groups and group identities, etc. This openness
to extensions potentially provides great benefits for the users. The price for these benefits is the
risk that comes with opening the service to untrusted third parties, extending the privacy
problem.
In a decentralized environment, if some users choose to enable a third-party application, their
choice should not affect other users or even users connected directly to them.
Security – For distributed storage with other peers that the user not necessarily wants to access
data, the content has to be encrypted, as done for example for file backup or anonymous peer-to-
peer file-sharing. To manage access to encrypted data, key distribution and maintenance have to
be handled such that the social network group can access data but be flexible enough to handle
churn in terms of going offline and coming back, additions and removal to the user’s social
network.
For peer identities, one can take advantage of opportunistic networks and peer authentication by
in-person contact, when friends meet in real life and exchange keys over their phones. For
bootstrapping authentication, a central authority (trusted third party) seems hard to avoid.
Robustness – In a centralized system, one can turn to the provider in case of user misbehavior,
there is usually a process defined for dealing with such complaints. In a decentralized system,
there is no authority that can ban users for misbehavior or remove content. Robustness against
free-riding.
Once access to content is granted, it is difficult to revoke that right. When a user allows a friend
to see a message, the friend can store the message and keep access to it even after a change of
key. Trust has to be at least equal to assigned access rights, due to this difficulty.
14 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
Limited Peers – To take advantage of the decentralized nature of social networks, a mapping of
physical social network to virtual and vice versa enables extensions to offering access via web
browsers by phone applications and direct exchange of data in physical proximity.
Another immediate benefit however of allowing such two-tier system is that users can then
participate in the social network with resource constrained (e.g., mobile) devices, which they
may use as an auxiliary, even when they contribute resources to the core of the system with their
primary device.
Locality – In addition to such opportunistic networks between users, a distributed architecture
also enables us to take advantage of geographic proximity and its correlation with local interests.
For example, most access routers for home Internet access now come with USB slots where
storage can be added or they already have unused storage on the device itself. These routers are
typically always on and thus would provide some stability for availability of data of local
interest. This local interest can arise from the locality of events but also from the locality of
typical real-life social networks of friends and neighbors.
3.8.2 General Purpose DOSNs
A reference architecture of a general-purpose DOSN platform is given in Fig. 3.d. The reference
architecture consists of six layers and provides an architectural abstraction of variety of current
related approaches to decentralized social networking in the research literature.
The lower layer of this architecture is the physical communication network, which can be the
Internet or a (mobile) ad hoc network (in case we consider a mobile online social network). The
distributed or P2P overlay management provides core functionalities to manage resources in
the supporting infrastructure of the system, which can be a distributed network of trusted servers
or a P2P overlay. Specifically, this layer provides higher layers the capabilities of looking up
resources, routing messages, and retrieving information reliably and effectively among nodes in
the overlay.
On top of this overlay is the decentralized data management layer, which implements
functionalities of a distributed or peer-to-peer information system to query, insert, and update
various persistent objects to the systems.
15 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
The social networking layer implements all basic functionalities and features that are provided
by contemporary centralized social networking services.
Among these functionalities the most important ones are namely the capability to search the
system (Distributed search) for relevant information, the management of users and shared space
(User account and share space management), the management of security and access control
issues (Trust management, Access control and security), the coordination and management of
social applications developed by third parties (Application management).
It is expected that the social networking layer exposes and implements an application
programming interface (API) to support the development of new applications as well as to
enable the customization of the social network service to suit various preferences of the user.
The top layer of the architecture includes the user interface to the system and various
applications built on top of the development platform provided by the DOSN.
The DOSN user is expected to provide the user the necessary transparency to use the DOSN as
any other centralized OSN. Applications can be either implemented by the DOSN provider or
developed by third-parties, and can be installed or removed from the system according to user’s
preferences.
Fig. 3.d The general architecture of a distributed online social network
16 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
3.9 Multi-Relational characterization of dynamic social network communities
The characterization of communities in online social media is presented using computational
approaches grounded on the observations from social science.
Motivation: human community as meaning-making eco-system – The semantics is an
emergent artifact of human activity that evolves over time. Human activity is mostly social, and
the social networks of human are conceivable loci for the construction of meaning. Hence, it is
crucial to identify real human networks as communities of people interacting with each other
through meaningful social activities, and producing stable associations between concepts and
artifacts in a coherent manner.
Motivating applications –As new concepts emerge and evolve around real human networks,
community discovery can result in new knowledge and provoke advancements in information
search and decision-making. Example applications include:
Context-sensitive information search and recommendation
Content organization, tracking and monitoring
Behavioral prediction
Data characteristics and challenges – Large volumes of social media data are being generated
from various social media platforms including blogs, FaceBook, Twitter, Digg, Flickr. The key
characteristics of online social media data include:
Voluminous
Dynamic
Context-rich
Approaches to three problems:
1. Mutual awareness: It is a bi-directional relationship indicating how well a pair of bloggers is
aware of each other, as fundamental property of a community. The amount of mutual awareness
is captured expanding on the entire network using a random walk based distance measure,
commute time, which estimates the probability that two bloggers are aware of each other on the
network (Fig. 3.e). The experimental results for community extraction in terms of standard
evaluation metrics are promising.
2. FacetNet: The community structure at a given timestep is determined both by the observed
networked data and by the prior distribution given by historic community structures. The
17 B. Shanmuga Sundari, AP/ IT
csenotescorner.blogspot.com UNIT III - CS6010 SNA
experimental results suggest that this technique is scalable and is able to extract meaningful
communities based on social media context.
3. MetaFac: MetaFac is the first graph-based tensor factorization framework for analyzing the
dynamics of heterogeneous social networks. In this framework, metagraph, is a novel relational
hypergraph representation for modeling multi-relational and multi-dimensional social data.
Extensive experiments on large-scale real-world social media data and from the enterprise data
suggest that this technique is able to extract meaningful communities that are adaptive to social
media context.
Fig 3.e Multiple aspects on community analysis
From the figure :
(a) Mutual awareness – a bi-directional relationship indicating how well a pair of bloggers is
aware of each other, as fundamental property of a community.
(b) Mutual awareness expansion – a random walk based distance measure which estimates the
probability that two bloggers are aware of each other on the network.
(c) FacetNet – for analyzing communities and their evolutions in a unified process.
(d) MetaFac – the first graph-based multi-tensor factorization framework for analyzing the
dynamics of heterogeneous social networks
18 B. Shanmuga Sundari, AP/ IT
