Understanding the Effect of Deplatforming on Social Networks

Shiza Ali Mohammad Hammas Saeed Esraa Aldreabi

Boston University, USA Boston University, USA Binghamton University, USA
[email protected] [email protected] [email protected]

Jeremy Blackburn Emiliano De Cristofaro Savvas Zannettou

Binghamton University, USA University College London, UK Max Planck Institute for Informatics,
[email protected] [email protected] Germany
[email protected]

Gianluca Stringhini
Boston University, USA
[email protected]

ABSTRACT ACM Reference Format:

Aiming to enhance the safety of their users, social media platforms Shiza Ali, Mohammad Hammas Saeed, Esraa Aldreabi, Jeremy Blackburn,
Emiliano De Cristofaro, Savvas Zannettou, and Gianluca Stringhini. 2021.
enforce terms of service by performing active moderation, includ-
Understanding the Effect of Deplatforming on Social Networks. In 13th
ing removing content or suspending users. Nevertheless, we do ACM Web Science Conference 2021 (WebSci ’21), June 21–25, 2021, Virtual
not have a clear understanding of how effective it is, ultimately, to Event, United Kingdom. ACM, New York, NY, USA, 9 pages. https://doi.org/
suspend users who engage in toxic behavior, as that might actu- 10.1145/3447535.3462637
ally draw users to alternative platforms where moderation is laxer.
Moreover, this deplatforming efforts might end up nudging abusive 1 INTRODUCTION
users towards more extreme ideologies and potential radicalization
Over the past years, toxic activity on social media like hate speech,
risks. In this paper, we set to understand what happens when users
cyberbullying, and harassment, has become an increasingly impor-
get suspended on a social platform and move to an alternative one.
tant problem [1, 36]. To curb abuse, social media platforms have
We focus on accounts active on Gab that were suspended from Twit-
adopted different mitigation strategies, including providing users
ter and Reddit. We develop a method to identify accounts belonging
with tools to flag abusive behavior [29]. They also suspend or ban
to the same person on these platforms, and observe whether there
accounts which are deemed unfit for the community. Suspended
was a measurable difference in the activity and toxicity of these
accounts are usually guilty of violating the platform’s terms of
accounts after suspension. We find that users who get banned on
service, which forbid illegal behavior (e.g., sending spam) as well
Twitter/Reddit exhibit an increased level of activity and toxicity on
as partaking in antisocial behavior. Depending on the policy of
Gab, although the audience they potentially reach decreases. Over-
each social network, offending accounts are blocked for some time
all, we argue that moderation efforts should go beyond ensuring
or banned permanently. Suspension actions targeted at individu-
the safety of users on a single platform, taking into account the
als who took part in hateful and harassing conduct are commonly
potential adverse effects of banning users on major platforms.
referred to as deplatforming [49].
Most research in this space has looked at deplatforming in a
CCS CONCEPTS
siloed fashion, evaluating the effect that these actions have on the
• Security and privacy → Social aspects of security and pri- platforms the accounts where banned from [11, 23, 57]. However,
vacy; • Human-centered computing → Empirical studies in users are obviously not bound to a single platform, but can migrate
collaborative and social computing. to other online services where moderation is possibly more lax. In
fact, anecdotal evidence shows that once hateful users get banned
KEYWORDS from Twitter, they often move to Gab, an alternative social network
Online social networks, deplatforming, moderation with an open lack of moderation marketed as protection of “free
speech” [2, 43, 50]. The effect of these migrations is not well un-
Permission to make digital or hard copies of all or part of this work for derstood. In particular, it is not clear if users tend to become more
personal or classroom use is granted without fee provided that copies are not toxic after they move to a more extreme community. What is clear
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components is that these communities have been used as outlets for violent
of this work owned by others than ACM must be honored. Abstracting with actors, e.g., Robert Bowers’ anti-Semitic posting on Gab just prior
credit is permitted. To copy otherwise, or republish, to post on servers or to to murdering eleven and wounding six people at the Tree of Life
redistribute to lists, requires prior specific permission and/or a fee. Request synagogue in Pittsburgh [46].
permissions from [email protected].
In this paper, we aim to measure the effect that deplatforming
WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom
© 2021 Association for Computing Machinery. events have on users, looking at how their activity changes when
ACM ISBN 978-1-4503-8330-1/21/06. . . $15.00 they get suspended on a social network and move to an alternative
https://doi.org/10.1145/3447535.3462637 one. To do so, we focus on users who were suspended on Twitter and
WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom Shiza Ali, et al.

Reddit and moved to Gab, a social network with laxer moderation. (i.e., users perpetrating real-world violence due to online radicaliza-
Overall, we aim to answer the following research questions: tion [24, 28]). These broad implications should be taken into careful
consideration by policymakers, social network operators, and by
• RQ1 – Account Creation: Do suspended users create an the research community studying these problems.
account on an alternative platform after being suspended on
a mainstream one, or did they already have one? 2 RELATED WORK
• RQ2 – Toxicity: Do suspended users become more toxic In this section, we review relevant related work.
after migrating? Platforms might be banning users to enforce
Hate Speech on Social Media. Silva et al. [52] study the main
codes of conduct, perhaps in the hopes that users will reform
targets of hate speech in online social media by analyzing content
(especially with respect to temporary suspensions). However,
shared on Twitter and Whisper, while Mondal et al. [36] analyze
it is possible that users will actually become more toxic
common hate expressions, the effect of anonymity on hate speech,
when migrating to other platforms, especially when the new
and the most hated groups across regions. [33] study the spread
platforms have an emphasis on more lax moderation.
of hate speech in Gab, showing that hateful content has greater
• RQ3 – Activity: Do suspended users become more active
outreach and spreads faster. As hate speech on social media become
after they migrate to other platforms? We want to understand
increasingly popular, previous work has also worked on automated
whether moving to a less moderated platform can contribute
detection using machine learning [6, 9, 14, 18, 48, 61].
to a user’s activity, resulting in them posting more content.
• RQ4 – Audience: Do suspended users get bigger or smaller Malicious Accounts. A wealth of research has studied malicious
audiences after their migration to other platforms? We want accounts on social networks, from those involved in sending spam [22,
to understand whether after moving to an alternative social 54, 57] to performing fraud [15, 26, 63], to taking part in online
network users are able to retain their following. If a user’s harassment [12, 19] Alorainy et al. [5] analyze suspended accounts
audience is reduced, they may reach fewer people and the and argue that suspended accounts are a reliable source for hate
effects of their toxic activity will be contained. Intuitively, speech prediction. They analyze three sources of data sets: sus-
we expect that the audience reachable by a user will shrink pended, active, and neutral ones. Their emotional analysis indicate
after they are banned from a major social network and move that tweets from suspended accounts show more disgust, negative,
to a smaller one. fear, and sadness emotions than the ones from active accounts. In
another study, Volkova et al. [59] predict suspicious, i.e., deleted or
To answer these questions, we first need to identify users on suspended accounts in social media. They analyze multiple datasets
Gab who were suspended from Twitter and Reddit. To do so, we of thousands of active, deleted, and suspended Twitter accounts and
start from a large corpus of 29M posts gathered from Gab, and produce predictive behaviors that lead to the removal or shutdown
cross-reference profile names with those that used to be active on of an account. They observe that the presence of certain terms in
Twitter and Reddit but were suspended. Overall, there are several tweets increases the likelihood for that account to be deleted or
reasons why a user would reuse the same profile name on a dif- suspended.
ferent platform [32], e.g., for continuity and recognizability with Moderation on Social Media. Jhaver et al. [27] analyze users’
their followers [30]. However, it is also possible that, especially for reactions to Reddit’s moderation process. They find that 18% of
“common” profile names, there are accounts on multiple social net- the participants believed that their posts were removed for the
works that are not controlled by the same person. To identify these right reasons, 37% did not know why their post was removed, and
accounts, we manually labeled a sample of the data and developed 29% expressed resentment towards the removal of their posts. Fur-
a classifier which achieves an accuracy of 94.5%. We consider 3,074 thermore, Habib et al. [23] undertake a comprehensive study on
out of 4,790 (64%) suspended Twitter accounts with a corresponding whether it is feasible to proactively moderate Reddit communities.
profile name on Gab and 5,216 out of 6,308 (82.7%) on Reddit as They build a machine learning model to study the characteristics of
controlled by the same person. subreddits and predict the future behavior of that subreddit. They
We find that 73.68% of the accounts on Gab were created after be- also analyze the impact of different events on user behavior and
ing suspended from Twitter or Reddit, highlighting a non-negligible find that banning and quarantining subreddits does not have an im-
user migration as a result of deplatforming. We also find that users pact on the overall civility of the users involved in that community.
tend to become more active and their posts more toxic after moving Unlike our work, their focus is on banning entire subcommunities
to Gab. However, since they lose followers, their audience tends to (i.e., subreddits) and not single users. They suggest that there is
decrease. a need for more active and nuanced intervention strategies to ef-
Overall, our study paints a comprehensive picture of the efficacy fectively moderate malicious accounts. However, [11] study the
of deplatforming on the Web community that suspended the users, 2015 ban of two hate communities on Reddit, /r/fatpeoplehate, and
the one users migrated to, as well as the effects on user behavior. /r/CoonTown; they analyze the effects of the ban on users and the
We argue that deplatforming seems to assist in safeguarding users communities and conclude that quarantining these communities
from the platform that took the moderation action, while substan- was successful for Reddit since users left the site or reduced their
tially reducing the audience of problematic users. Nevertheless, hate speech.
suspended users are becoming more active and more toxic, which Newell et al. [38] explore how a period of community unrest on
is possibly an indication of online radicalization that might have Reddit affected users migration off platform. Similarly to our study,
an overall negative impact both on the online and offline world they use an algorithm (with a lower bound of 0.6 precision) which
Understanding the Effect of Deplatforming on Social Networks WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom

makes use of user names to match users across a variety of other Reddit. To check whether accounts with a certain profile name
platforms (e.g., HackerNews and Voat). While their findings are used to exist on Reddit, we look up reddit.com/u/<profilename>.json.
relevant to the current work, as they expose some of the motiva- If the object that is returned contains the message Forbidden, with
tions behind users migrating off of Reddit, there is a fundamental HTTP error code 403, we consider that user to be suspended. We
difference: our study focuses on users that were forcibly removed find 145,835 users on Reddit with the same profile name on Gab
from Twitter or Reddit. and out of these 6,308 were suspended.
Ribeiro et al. [47] analyze data from two communities r/The_Donald
and r/Incels that were banned from Reddit and migrated to their 3.3 Collecting Data of Suspended Users on
own websites. Compared to this research, our focus is on the mi-
Twitter and Reddit
gration of single users after suspension and not on the reaction of
users to entire communities being banned. Next, we collect data about suspended Twitter/Reddit users with a
matching Gab profile name.
Remarks. To the best of our knowledge, this paper is the first
to study how the activity of online users changes once they get Twitter. Twitter accounts can be suspended or have their activ-
suspended on one platform and migrate to a different one. ity limited for security purposes, or they may have violated the
Twitter Rules, or have some features limited due to suspicious activ-
3 METHODOLOGY ity [58]. Note that Twitter’s Terms of Service prevent researchers
from keeping records of accounts or tweets once they are deleted.
Our main goal is to identify and study accounts that were sus-
Therefore, we cannot use data collected from, e.g., the 1% Streaming
pended on one or more social platforms – namely, Twitter and
API; rather, we use publicly available archives from the Wayback
Reddit – and later moved to alternative platforms such as Gab. We
Machine [62] and the dataset of verified Twitter accounts from
choose Gab because, according to previous work [65], it has laxer
Pushshift [45]. Out of the 20,967 suspended users, we are able to
moderation policies and it attracts accounts suspended from major
find data for 580 users on Pushshift and 4,210 users on the Wayback
social networks.
Machine, ultimately gathering about 1M tweets.
To identify pairs of accounts that were controlled by the same
person (the suspended account on Twitter or Reddit and the one Reddit. As part of their content policy, Reddit uses content re-
on Gab) we proceed backwards. We first identify Gab users from moval (e.g., if it incites violence), including banning users, as well
a large dataset of publicly available Gab posts. Next, we identify as subreddit quarantines. Since these accounts were suspended, we
accounts on Twitter and Reddit that used the same profile name are not able to collect all their posts by using the regular Reddit
as the ones on Gab, and that have since been suspended. To avoid API. Instead, we retrieve their posts by querying the search tool
false positives, we build a classifier to determine whether or not provided by the Pushshift API. We are able to find data of 5,216
two accounts on different platforms were likely controlled by the out of the 6,308 suspended users using the Pushshift API [44] and
same people. In this section, we describe these steps in detail. extract all the posts made by the user.
Ethics. Our study only uses data that is publicly available and, since
3.1 Gab Data Collection we do not interact with users in any way, it is not considered as
We use the Gab dataset made available by PushShift [7]; this con- human subjects research by the IRB at our institution. Nonetheless,
tains 29 million posts made over a 1.5 year period (between 2016 we acknowledge that linking user accounts across different social
and 2018) by 322,397 unique Gab users. We then extract the profile networks may have some ethical implications. We limit our study
names of these accounts and check whether accounts with the same to accounts that used the same profile name on two platforms;
profile names were also present on Twitter and Reddit but have arguably, this excludes from our analysis users who did not want to
been suspended. be found after being suspended on Twitter or Reddit, for example by
changing their profile name. Moreover, we only report aggregated
3.2 Identifying Pairs of Accounts with the statistical information and do not perform any analysis at a single
Same Profile Name account granularity, except the one needed for determining ground
We start by assuming that users who get suspended on Twitter truth. Finally, we do not use the collected information to further
and Reddit and move to Gab will create an account with the same de-anonymize the users.
profile name. Next, for each account in the Gab dataset, we look
up their profile name on Twitter and Reddit, and check whether 3.4 Classifying Accounts as Belonging to the
an account with that profile name existed and was suspended. We Same Person
began collecting our data for Twitter and Reddit in January 2020. Even though two accounts with the same profile name exist on
Twitter. For Twitter, we use the Twython Twitter API [41] user two different platforms, this does not necessarily mean that the
searching functionality [35] and check whether the user is active, accounts belong to the same person. This is particularly true for
suspended (these get a HTTP 403 error), or not found (HTTP 404 er- common profile names. Therefore, to reliably identify accounts
ror), indicating that the account either never existed or was deleted controlled by the same person, we look at features of the online
by its owner. From the 322K Gab profile names, we find 200,303 accounts on the different platforms and build a machine learning
users on Twitter with the same profile name. Of these, 20,967 Twit- classifier that confirms whether or not the two accounts on the
ter accounts were suspended at the time of collection different platforms belong to the same user.
WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom Shiza Ali, et al.

Annotators Cohen’s k
Annotator 1 and 2 0.96
Annotator 1 and 3 0.95
Annotator 2 and 3 0.96
Table 1: Cohen’s k score between the different annotators

Classifier Precision Recall Accuracy F1-Score

KNN 90.2% 92.0% 93.0% 91.1%
Figure 1: An example of Twitter account metadata we ex- Decision Tree 91.4% 90.9% 90.7% 91.1%
Random Forest 93.2% 92.5% 94.6% 92.8%
tract: (1) Profile name, (2) Display name, (3) Description , and
(4) Location. Table 2: Classification scores for the task of predicting if the
profile name on the two different platforms belonged to the
same person.
Preprocessing. We first process the metadata associated to user
accounts by extracting the display name, description, and location of
the user. An example of the Twitter metadata is showed in Figure 1. a labeled dataset, we train classifiers to automatically determine
We also extract any URLs in the profile description. if two accounts belong to the same users. We experiment with
Features. For Twitter, we use the following features to help us Random Forest [8], KNN [17], and Decision Tree [55] classifiers,
characterize whether two accounts with the same profile name on trained using the 400-account annotated dataset discussed above
two platforms belong to the same person: and stratified 10-fold cross-validation. We also use a hold-out vali-
• Jaro Similarity [60] of the display names of the accounts on dation set of 10 users to fine-tune the parameters of the classifiers.
both platforms. We use this metric to match different display To evaluate performance, we rely on accuracy, along with precision,
names because it assigns a positive weight for close strings recall, and F1-score. Table 2 reports the average results on a 10-fold
and normalizes it according to the length of the strings, hence cross validation obtained using different classifier choices. Our best
outperforming other string compare methods [42]. performing model is Random Forest, achieving an average accuracy
• Jaccard similarity [39] of the profile descriptions. We use Jac- of 94.6%.
card Similarity because it performs well for topic modeling We then use Random Forest to classify the accounts on the rest
and comparing keywords [53, 56] of the dataset. This yields the following results: 3,074 out of 4,790
• Mention of the user’s original Twitter handle in the Gab (64%) suspended accounts on Twitter and 5,216 out of 6,308 (82.7%)
description or viceversa. (We use this feature as manual on Reddit have an active (matching) Gab account. We next analyze
inspection indicates that several users had the same descrip- these matched pairs of accounts, to understand how the activity of
tion mentioned on both platforms, possibly to signal the a user changed after they were suspended on Twitter or Reddit and
followers that they are the same person.) moved to Gab.
• Number of matching hashtags in the profile descriptions.
• Matching location mentioned on Twitter with the one men- 4 ANALYSIS
tioned in the profile description on Gab. In this section, we present the results of our analysis aimed to
For Reddit, since the information provided in user profiles is very address our four research questions, using the dataset presented
limited, we only use Jaro similarity between the account’s display above.
names.
Labeling. Identifying if two accounts are controlled by the same 4.1 RQ1: Are Accounts Created on An
person cannot be easily automated. To establish ground truth, we Alternative Platform After Being
had three authors of this paper manually annotate the same subset Suspended?
of 400 randomly selected accounts (200 Twitter+Gab pairs and First, we investigate whether once an account gets suspended on a
200 Reddit+Gab pairs). Each data point was labeled by the three platform, their owner simply keeps using an alternative account
annotators and the label was chosen by majority vote. We then that they already had on another social network, or rather creates
calculate the Cohen’s k score [31] between the annotators (Table 1), a new one.
finding high agreement scores between all the annotators. Based on Unfortunately, neither Twitter nor Reddit provide any informa-
our labeling, we find that 74.5% of the times if the profile name is tion about when accounts get suspended. However, our Gab dataset
the same on the two different platforms, then the account belongs includes the date when an account was created. Therefore, we can
to the same person. compare this to the last post by the corresponding Twitter/Reddit
account on Twitter/Reddit, and use this as a reasonable estimation
3.5 Classification Performance of whether or not the Gab account was created after suspension.
After having identified suitable features to identify accounts con- Since Gab was launched in 2016, to remove any bias in this exper-
trolled by the same user across platforms and having established iment we only consider users whose Twitter and Reddit accounts
Understanding the Effect of Deplatforming on Social Networks WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom

were suspended after January 1, 2017. We find that 58.74% of Twitter Twitter Gab
users in our dataset (1,152 out of 1,961) created their account on Average Number of Followers 7,764 167
Gab after their last active time on Twitter, presumably after their Average Number of Friends 4,475 201
Twitter account was suspended. For Reddit, we have very similar Average Number of Posts by a User 51,443 485
findings, as 75.88% (3,958 out of 5,216) of the suspended Reddit
Table 3: Average user statistics on Twitter and Gab.
users create their account on Gab after their last post on Reddit.
Overall, these results show that most Gab accounts in our dataset
were created by users after being suspended on the other platform,
allowing us to answer RQ1 in the affirmative. 4.3 RQ3: Do Suspended Users Become More
Active if They Move to Another Platform?
Next, we study how user activity changes once they move to an
4.2 RQ2: Do Suspended Users Become More alternative platform after suspension. We do so by looking at the
Toxic if They Move to Another Platform? daily number of posts made by a user on the various platforms.
Next, we investigate whether suspended users who move to an For Twitter, we compare the activity that a user had on Twitter
alternative platform become more toxic. To do so, we use Google’s before being suspended with the activity they have on Gab after
Perspective API, a free Google service developed by Jigsaw [21] suspension. Figure 3(a) shows the CDF of the daily number of posts
which uses a machine learning model trained on comments (man- before and after suspension, showing that activity overall increases.
ually) labeled as toxic or non-toxic [16]. The API returns several By manually inspecting Gab posts right after suspension, we often
scores, ranging from 0 to 1, including “Toxicity” and “Severe Tox- find that users complain about being unfairly suspended on Twitter,
icity.” We use the latter, as prior work [66] shows it to be a more or including quotes about freedom of speech—i.e., how suspending
robust indicator of online toxicity. accounts violates it. A two-sample KS test confirms statistically
significant differences in the distributions (D = 0.242, p < 0.01).
We also illustrate the pre-suspension and post-suspension activity
Platform Toxicity. We first look at the average Severe Toxicity of of users on Twitter and Gab as a scatter plot in Figure 4(a); this
posts and users on the three platforms. We find that the average shows that a large number of users were not very active on Twitter
Severe Toxicity of posts on, respectively, Twitter, Reddit, and Gab but then became so once moving to Gab.
is 0.387, 0.42, and 0.498. The average Severe Toxicity of users on, We then perform the same analysis for Reddit; see Figure 3(b).
respectively, Twitter, Reddit, and Gab is 0.104, 0.3, and 0.334. Overall, Once again, we find that daily activity increases after users are
this somewhat confirms that there is less moderation on Gab than suspended from Reddit and move to Gab. A scatter plot of the pre-
on Twitter and Reddit. suspension and post-suspension activity of users on Reddit and
Also note that on Gab 79% of users had less than 100 posts. This Gab is shown in Figure 4(b). Again, a two-sample KS test confirms
means that most Gab users do not post regularly, which can be statistically significant differences in the distributions (D = 0.132,
explained by the fact that it has been launched in 2016 and did not p < 0.01).
gain popularity, relatively speaking, until recently [20]. Manual Overall, we show that the activity of suspended users does tend
inspection of toxic posts also show that Gab users do complain to increase after they move to the alternative platform.
about other social platforms suspending their accounts, or for not
supporting freedom of speech. 4.4 RQ4: Do Suspended Users Gain More
Followers on the Other Platform?
Change in Toxicity. We then want to understand whether users While suspending accounts can arguably curb hateful and toxic
become more toxic after they get suspended on a platform and content on the platform, there is a chance that this can actually
move to an alternative one. Figure 2(a) shows the Cumulative Dis- facilitate users moving to an alternative service obtain a broader
tribution Function (CDF) of the average Severe Toxicity of accounts audience, more appreciative of whatever behavior got them banned
on Twitter (before suspension) and Gab (after suspension). As it from their previous platform.
can be seen, about 60% of users are slightly less toxic after moving To understand how the size of a user’s audience changes after
to Gab, but 20% of users become much more toxic after moving suspension, we look at the number of followers of Twitter and Gab
to Gab. We also perform a two-sample Kolmogorov Smirnov (KS) accounts. Table 3 reports the average number of followers, friends,
test on the two curves, which shows that the distributions exhibit and the number of posts for the users in our dataset on Twitter and
statistically significant differences (D = 0.193, p < 0.01). Gab. We notice that the number of followers of the suspended users
We do the same for Reddit, with the CDF shown in Figure 2(b). drastically decreased once they moved to Gab.
The figure shows that the toxicity of users increases after mov- We also look at the distribution of followers of users on Twitter
ing to Gab. A two-sample KS test confirms statistically significant and Gab. Figure 5 shows the CDF of the number of followers of
differences in the distributions (D = 0.416, p < 0.01). matched accounts on Twitter and Gab. A two-sample KS confirms
Overall, our analysis shows that users do tend to become more statistically significant differences in the distributions (D = 0.193,
toxic when they are suspended from a platform and are forced to p < 0.01). This shows that, even though users tend to become more
move to another platform. This is true for users moving from Reddit toxic and more active after they move to the alternative platform,
to Gab, and for 20% of the users moving from Twitter to Gab. their audience decreases.
WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom Shiza Ali, et al.

1.0 1.0
Gab Gab
Twitter Reddit
0.8 0.8

0.6 0.6
Users

Users
0.4 0.4

0.2 0.2

0.0 0.0
0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8
Severe Toxicity Severe Toxicity
(a) (b)
Figure 2: Cumulative Distribution Function (CDF) of Severe Toxicity of a) Twitter users before being suspended and then
moving to Gab b) Reddit users before being suspended and then moving to Gab. 60% of users migrating from Twitter become
less toxic on Gab, while 20% become much more toxic. In general, users migrating from Reddit become more toxic when
moving to Gab.

1.0 1.0

0.8 0.8

0.6 0.6
Users

Users

0.4 0.4

0.2 0.2
Gab Gab
0.0 Twitter 0.0 Twitter
0 10 20 30 40 50 0 10 20 30 40 50
Daily Number of Posts Daily Number of Posts
(a) (b)
Figure 3: Cumulative Distribution Function (CDF) of the daily number of posts of a) Twitter users before being suspended and
then moving to Gab. b) Reddit users before being suspended and then moving to Gab. Users become more active after being
suspended on Twitter and Reddit and moving to Gab.

Alas, we cannot perform the same analysis for Reddit because size. This paper provides a first understanding of user-level mi-
there is no equivalent to the follower concept. On Reddit, users gration after being banned on social network platforms, and our
usually join and comment in communities called subreddits, while results suggest that it is important to understand potentially unin-
on Twitter and Gab people follow other users whose posts they are tended consequences when designing moderation and suspension
interest in. mechanisms.

5 DISCUSSION AND CONCLUSION

In this paper, we presented a large-scale study of social network 5.1 Implications of Our Results
users who get suspended from Twitter and Reddit and move to A common solution to tackle malicious users that send out spam and
Gab. Overall, we found that users tend to become more toxic and malware is to block/suspend them. The same solution is adopted
active once they migrate to Gab, but their audience decreases in by several services to mitigate online toxicity, particularly against
Understanding the Effect of Deplatforming on Social Networks WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom

70 70

60 60

50 50
Gab Activity

Gab Activity
40 40

30 30

20 20

10 10

0 0
0 5 10 15 20 25 30 35 40 0 5 10 15 20 25 30 35 40
Twitter Activity Reddit Activity
(a) (b)
Figure 4: Scatter plot of the number of posts on a) Twitter before being suspended as compared to their activity on Gab after
being suspended. b) Reddit before being suspended as compared to their activity on Gab after being suspended. As it can be
seen a number of users became more active after moving to Gab.

Migration to Alternative Communities. Our study finds that

1.0 accounts that are suspended on Twitter and Reddit often migrate
Gab to Gab. In particular, we can reliably map 2.5% of all Gab accounts
Twitter
to a pre-existing account on Twitter and Reddit that was sus-
0.8 pended. This is a lower bound, since our data collection approach
for suspended accounts has limitations (see below) and our track-
0.6 ing methodology is only able to identify accounts whose owners
maintained the same profile name across platforms. This indicates
Users

that users who are suspended for violating the terms of service of
0.4 a social platform often move to a less moderated community, and
the consequences of this action need to be studied.
0.2 Our work focused on Gab, but there are other alternative com-
munities where users migrate, like Parler, WrongThink, Voat, and
PewTube [4, 40, 64]. Future work should focus on measuring mi-
0.0
100 101 102 103 104 105 106 grations to these communities as well, to paint a better picture of
Followers the displacement effect of suspended users and its potential effect
Figure 5: Cumulative Distribution Function (CDF) of the on the online ecosystem.
number of followers on Twitter and Gab. After moving to
Gab, the number of followers that users reach shrinks. Unexpected Consequences of Suspension. Our results show
that users who move to Gab after a suspension tend to become
more active and more toxic. At the same time, we find that the
audience that these users can reach on the alternative platforms
users who post hateful content. This practice has raised some criti- is much smaller than it used to be on Twitter. This quantitative
cism, with people arguing that users should not be blocked because analysis highlights trends that should be studied further, but also
“selective free speech” is a dangerous precedent [34]. leaves many questions that could be answered through qualitative
More importantly, suspension practices by online services are analysis. For example, a manual analysis of the Gab accounts created
usually conducted in isolation, with the goal of keeping the plat- after a suspension on Twitter showed that these users often wear
form safe, and do not take into account broader effects to the online their suspension as a badge of honor, mentioning it in their profile
ecosystem, such as the unintended consequences that might arise description and often talking about it in their posts on Gab. It is
from suspending users and having them more to alternative com- important to understand how this affects the popularity of these
munities. users, in particular in helping them build an engaged following.
In this paper, we shed some light on these deplatforming dynam- Also, our quantitative analysis does not allow us to understand
ics. Next, we discuss the key insights obtained from our analysis, what type of users are following Gab accounts, together with the
together with some open questions and promising future research nature of the toxic speech used by them. Many offline violence
directions. instances have been linked with alternative social platforms [3, 37,
WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom Shiza Ali, et al.

51]. As part of future work, qualitative analysis could shed light [3] ADL. Gab and 8chan: Home to Terrorist Plots Hiding in Plain
on how the type of discussion changes after users get suspended Sight. https://www.adl.org/resources/reports/gab-and-8chan-home-
and move to alternative platforms, together with understanding to-terrorist-plots-hiding-in-plain-sight, 2020.
[4] M. Aliapoulios, E. Bevensee, J. Blackburn, E. De Cristofaro, G. Stringh-
whether there is a link between online speech and radicalization. ini, and S. Zannettou. An early look at the parler online social network.
arXiv preprint arXiv:2101.03820, 2021.
5.2 Limitations [5] W. Alorainy, P. Burnap, H. Liu, A. Javed, and M. L. Williams. Suspended
accounts: A source of Tweets with disgust and anger emotions for
We now discuss some limitations of our analysis. augmenting hate speech data sample. In ICMLC, 2018.
Data. One important aspect of our work is being able to understand [6] P. Badjatiya, S. Gupta, M. Gupta, and V. Varma. Deep Learning for
Hate Speech Detection in Tweets. ArXiv:1706.00188, 2017.
the role and behavior of users before being suspended on Twitter [7] J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, and J. Blackburn.
or Reddit. Twitter’s Terms of Service require users to promptly The Pushshift Reddit Dataset. In ICWSM, 2020.
delete all tweets generated by Twitter accounts as soon as they are [8] L. Breiman. Random forests. Machine Learning, 45(1), 2001.
suspended. To overcome this limitation, we used publicly available [9] P. Burnap and M. L. Williams. Cyber Hate Speech on Twitter: An
Application of Machine Classification and Statistical Modeling for
datasets from the Wayback Machine Twitter archives and Pushshift. Policy and Decision Making. Policy & Internet, 7(2), 2015.
However, this also means that the data we collect is limited, as [10] S. Cao. Google’s AI Hate Speech Detector Has a âĂŸBlack Twitter’
we could only perform analysis on the tweets and posts that were Problem: Study | Observer, 2019.
available from publicly available snapshots. Also, neither Twitter [11] E. Chandrasekharan, U. Pavalanathan, A. Srinivasan, A. Glynn, J. Eisen-
nor Reddit list the suspension date of accounts, and we had to stein, and E. Gilbert. You Can’t Stay Here: The Efficacy of Reddit’s
2015 Ban Examined Through Hate Speech. Proc. ACM Hum.-Comput.
estimate this from the data. Regardless, our data provides us with Interact., 1(CSCW), 2017.
a lower bound of the number of users that migrated to Gab after [12] D. Chatzakou, N. Kourtellis, J. Blackburn, E. De Cristofaro, G. Stringh-
being suspended on Twitter or Reddit. ini, and A. Vakali. Mean birds: Detecting aggression and bullying on
Furthermore, for our dataset we do not take into consideration twitter. In ACM WebSci, 2017.
[13] D. Coldewey. Racial bias observed in hate speech detection algorithm
banned accounts where the usernames are not identical. This bi-
from Google | TechCrunch, 2019.
ases the dataset to more active users who probably reused their [14] T. Davidson, D. Warmsley, M. Macy, and I. Weber. Automated Hate
username to gain back old followers. Speech Detection and the Problem of Offensive Language. In ICWSM,
2017.
Google’s Perspective API. Our toxicity analysis uses the Perspec- [15] E. De Cristofaro, A. Friedman, G. Jourjon, M. A. Kaafar, and M. Z.
tive API and is thus bounded by its limitations. For instance, [25] Shafiq. Paying for likes? understanding facebook like fraud using
discuss how posts can be tampered with in order to reduce the honeypots. In ACM IMC, 2014.
Perspective score, e.g., editing the sentence “Homer Simpson is a [16] P. Delgado. How El País used AI to make their comments section less
toxic. https://blog.google/outreach-initiatives/google-news-initiative/
moron” to “Homer Simpson is a mor.on” lowers the score from 0.93
how-el-pais-used-ai-make-their-comments-section-less-toxic/, 2019.
to 0.12. Furthermore, the API could have racial biases [10, 13] as, [17] Z. Deng, X. Zhu, D. Cheng, M. Zong, and S. Zhang. Efficient kNN
rather than looking at the context of the abusive comment, it may classification algorithm for big data. Neurocomputing, 195, 2016.
just be looking at the potentially hateful words used. While we [18] N. Djuric, J. Zhou, R. Morris, M. Grbovic, V. Radosavljevic, and
acknowledge that the Perspective API has some limitations and N. Bhamidipati. Hate speech detection with comment embeddings. In
WWW Companion, 2015.
biases as reported, we also find that it is the most reliable tool to [19] M. ElSherief, S. Nilizadeh, D. Nguyen, G. Vigna, and E. Belding. Peer to
assess toxicity that is available to the research community. peer hate: Hate speech instigators and their targets. In ICWSM, 2018.
Reason for Suspension. Due to Twitter’s and Reddit’s policies, [20] D. Gilbert. Here’s How Big Far Right Social Network Gab Has Actually
Gotten. https://www.vice.com/en_uk/article/pa7dwg/heres-how-big-
we do not know why users were suspended. This is important, far-right-social-network-gab-has-actually-become, 2019.
because we expect users getting suspended over hateful conduct [21] Google. Perspective API. https://www.perspectiveapi.com, 2020.
to behave differently than those who posted fraudulent links for [22] C. Grier, K. Thomas, V. Paxson, and M. Zhang. @spam: the under-
example. Anecdotally, we find that many Gab posts are about how ground on 140 characters or less. In ACM CCS, 2010.
[23] H. Habib, M. B. Musa, F. Zaffar, and R. Nithyanand. To Act or React?
users were unfairly suspended. In particular, the most toxic posts Investigating Proactive Strategies For Online Community Moderation.
contain hatred against Twitter and that their free speech and rights arXiv:1906.11932, 2019.
have been denied. [24] G. Hassan, S. Brouillette-Alarie, S. Alava, D. Frau-Meigs, L. Lavoie,
A. Fetiu, W. Varela, E. Borokhovski, V. Venkatesh, C. Rousseau, et al.
Exposure to extremist online content could lead to violent radicaliza-
ACKNOWLEDGMENTS tion: A systematic review of empirical evidence. International journal
We thank the anonymous reviewers for their feedback. This work of developmental science, 12(1-2):71–88, 2018.
was partially supported by the National Science Foundation under [25] H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran. Deceiv-
ing google’s perspective API built for detecting toxic comments.
Grants 1827700, 1942610, 2114411, and 2046590. arXiv:1702.08138, 2017.
[26] J. Huang, G. Stringhini, and P. Yong. Quit playing games with my
REFERENCES heart: Understanding online dating scams. In International Conference
[1] K. S. Adewole, N. B. Anuar, A. Kamsin, K. D. Varathan, and S. A. Razak. on Detection of Intrusions and Malware, and Vulnerability Assessment,
Malicious accounts: Dark of the social networks. Journal of Network 2015.
and Computer Applications, 79, 2017. [27] S. Jhaver, D. S. Appling, E. Gilbert, and A. Bruckman. "Did You Sus-
[2] ADL. When Twitter Bans Extremists, GAB Puts Out the Welcome pect the Post Would Be Removed?": Understanding User Reactions
Mat. https://www.adl.org/blog/when-twitter-bans-extremists-gab- to Content Removals on Reddit. Proc. ACM Hum.-Comput. Interact.,
puts-out-the-welcome-mat, 2019. 3(CSCW), 2019.
Understanding the Effect of Deplatforming on Social Networks WebSci ’21, June 21–25, 2021, Virtual Event, United Kingdom

[28] J. Kadivar. Online radicalization and social media: A case study of [53] D. Spina, J. Gonzalo, and E. Amigó. Learning similarity functions for
daesh. International Journal of Digital Television, 8(3):403–422, 2017. topic detection in online reputation monitoring. In Proceedings of the
[29] I. Kayes, N. Kourtellis, D. Quercia, A. Iamnitchi, and F. Bonchi. The 37th international ACM SIGIR conference on Research & development in
Social World of Content Abusers in Community Question Answering. information retrieval, pages 527–536, 2014.
In WWW, 2015. [54] G. Stringhini, C. Kruegel, and G. Vigna. Detecting Spammers on
[30] M. Korayem and D. J. Crandall. De-anonymizing users across hetero- Social Networks. In Proceedings of the 26th Annual Computer Security
geneous social computing platforms. In ICWSM, 2013. Applications Conference, 2010.
[31] T. O. Kvalseth. Note on Cohen’s Kappa. Psychological Reports, 65(1), [55] P. H. Swain and H. Hauska. The decision tree classifier: Design and
1989. potential. IEEE Transactions on Geoscience Electronics, 15(3), 1977.
[32] E. Mariconti, J. Onaolapo, S. Ahmad, N. Nikiforou, M. Egele, N. Niki- [56] M. Swartz and A. Crooks. Comparison of emoji use in names, profiles,
forakis, and G. Stringhini. What’s in a Name?: Understanding Profile and tweets. In 2020 IEEE 14th International Conference on Semantic
Name Reuse on Twitter. In WWW, 2017. Computing (ICSC), pages 375–380, 2020.
[33] B. Mathew, R. Dutt, P. Goyal, and A. Mukherjee. Spread of Hate Speech [57] K. Thomas, C. Grier, D. Song, and V. Paxson. Suspended accounts in
in Online Social Media. In ACM WebSci, 2019. retrospect: an analysis of twitter spam. In ACM IMC, 2011.
[34] B. Mathew, P. Saha, H. Tharad, S. Rajgaria, P. Singhania, S. K. Maity, [58] Twitter. Help with locked or limited account, 2020.
P. Goyal, and A. Mukherje. Thou shalt not hate: Countering Online [59] S. Volkova and E. Bell. Identifying Effective Signals to Predict Deleted
Hate Speech. In ICWSM, 2019. and Suspended Accounts on Twitter Across Languages. In ICWSM,
[35] R. McGrath. Twython. https://twython.readthedocs.io/en/latest/, 2020. 2017.
[36] M. Mondal, L. Silva, and F. Benevenuto. A Measurement Study of Hate [60] Y. Wang, J. Qin, and W. Wang. Efficient approximate entity match-
Speech in Social Media. In WWW, 2017. ing using jaro-winkler distance. In International Conference on Web
[37] J. Morse. Gab Chat ’likely’ to be used by white extremists, according Information Systems Engineering, 2017.
to police. https://mashable.com/article/law-enforcement-documents- [61] Z. Waseem and D. Hovy. Hateful Symbols or Hateful People? Predictive
violent-white-extremists-encrypted-gab-chat/?europe=true, 2020. Features for Hate Speech Detection on Twitter. In NAACL Student
[38] E. Newell, D. Jurgens, H. M. Saleem, H. Vala, J. Sassine, C. Armstrong, Research Workshop, 2016.
and D. Ruths. User Migration in Online Social Networks: A Case Study [62] Wayback Machine. Wayback Machine APIs – Internet Archive. https:
on Reddit During a Period of Community Unrest. In Proceedigns of the //archive.org/help/wayback_api.php, 2020.
Tenth International AAAI Conference on Web and Social Media, ICWSM, [63] J. Weerasinghe, B. Flanigan, A. Stein, D. McCoy, and R. Greenstadt. The
2016. pod people: Understanding manipulation of social media popularity
[39] S. Niwattanakul, J. Singthongchai, E. Naenudorn, and S. Wanapu. Us- via reciprocity abuse. In The Web Conference, 2020.
ing of Jaccard coefficient for keywords similarity. Proceedings of the [64] Wired. The wheels are falling off the alt-right’s version of the in-
international multiconference of engineers and computer scientists, 1(6), ternet. https://www.wired.co.uk/article/alt-right-internet-is-a-ghost-
2013. town-gab-voat-wrongthink, 2020.
[40] A. Papasavva, J. Blackburn, G. Stringhini, S. Zannettou, and [65] S. Zannettou, B. Bradlyn, E. De Cristofaro, H. Kwak, M. Sirivianos,
E. De Cristofaro. "is it a qoincidence?": A first step towards under- G. Stringini, and J. Blackburn. What is Gab: A Bastion of Free Speech
standing and characterizing the qanon movement on voat. co. arXiv or an Alt-Right Echo Chamber. In WWW Companion, 2018.
preprint arXiv:2009.04885, 2020. [66] S. Zannettou, M. ElSherief, E. Belding, S. Nilizadeh, and G. Stringhini.
[41] R. D. Perera, S. Anand, K. Subbalakshmi, and R. Chandramouli. Twitter Measuring and Characterizing Hate Speech on News Websites. In
analytics: Architecture, tools and analysis. In Milcom, 2010. ACM WebSci, 2020.
[42] D. Perito, C. Castelluccia, M. A. Kaafar, and P. Manils. How unique
and traceable are usernames? In International Symposium on Privacy
Enhancing Technologies Symposium, pages 1–17. Springer, 2011.
[43] H. C. Powell and B. Clifford. De-platforming and the Online Extremist’s
Dilemma. https://www.lawfareblog.com/de-platforming-and-online-
extremists-dilemma, 2019.
[44] Pushshift. Reddit Statistics. pushshift.io, 2020.
[45] Pushshift. Search Twitter Users and Discover Interesting Accounts.
https://pushshift.io/twitter-user-search/, 2020.
[46] A. K. Raymond. What We Know About Robert
Bowers, Alleged Pittsburgh Synagogue Shooter.
https://nymag.com/intelligencer/2018/10/what-we-know-about-
robert-bowers-alleged-synagogue-shooter.html, Oct. 2018.
[47] M. Ribeiro, S. Jhaver, S. Zannettou, J. Blackburn, E. De Cristofaro,
G. Stringhini, and R. West. Does platform migration compromise
content moderation?, 2020.
[48] D. Robinson, Z. Zhang, and J. Tepper. Hate Speech Detection on
Twitter: Feature Engineering v.s. Feature Selection. In ESWC Satellite
Events, 2018.
[49] R. Rogers. Deplatforming: Following extreme Internet celebrities to
Telegram and alternative social media. European Journal of Communi-
cation, 35(3), 2020.
[50] G. Ryan. Weighing the Value and Risks of Deplatform-
ing. https://gnet-research.org/2020/05/11/weighing-the-value-and-
risks-of-deplatforming/, 2020.
[51] E. Schumacher. Far-right social network Gab struggles after Pitts-
burgh attack. https://www.dw.com/en/far-right-social-network-gab-
struggles-after-pittsburgh-attack/a-46065847, 2018.
[52] L. Silva, M. Mondal, D. Correa, F. Benevenuto, and I. Weber. Analyzing
the targets of hate in online social media. arXiv:1603.07709, 2016.