Task 1.

Getting data from JIRA

In this part, you will extract the data you need to apply the data mining algorithms. The target is the Spring XD
project (http://projects.spring.io/spring-xd/), which uses the JIRA project management tool
(https://jira.atlassian.com/) to manage their issues.

This Task consist of two parts. In the first part (Task 1.1), you will create a dataset with the issues from the JIRA
repository. In the second one (Task 1.2), you will pre-process and clean the dataset.

[INFO] Using the JIRA Python API

The JIRA Python API library (https://github.com/pycontribs/jira) eases the use of the JIRA REST API from
Python and it has been used in production for years.

See the documentation for full details. http://jira.readthedocs.io/en/latest/ (http://jira.readthedocs.io/en/latest/)

In [1]:

"""
Example of using JIRA Python API
"""

from jira import JIRA

jira = JIRA('https://jira.atlassian.com')

issue = jira.issue('JRA-9')
print issue.fields.project.key # 'JRA'
print issue.fields.issuetype.name # 'Suggestion'
print issue.fields.reporter.displayName # 'Mike Cannon-Brookes [Atlassian]'

JRASERVER
Suggestion
Mike Cannon-Brookes

In [2]:

# Getting the keys of the last 3 reported issues

jira = JIRA('https://jira.spring.io')
print [issue.key for issue in jira.search_issues('project=XD order by created desc', maxRes

[u'XD-3768', u'XD-3767', u'XD-3766']

Task 1.1 Data extraction from JIRA

Issues
Using the JIRA Python API, connect to the Spring XD JIRA repository (https://jira.spring.io) and create a
dataset with the following info about the issues:

Key
Assignee
 Creator
Reporter
Created
Components
Description
Summary
Fix Versions
Subtask
Issuetype
Priority
Resolution
Resolution date
Status
Status Description
Updated
Versions
Watches
Story Points - Hint: the name of the field in which the story points are stored is not enough descriptive
in this case. Thus, you might have a look at the data on each field and their distribution in order to
understand where the story points are stored. Keep in mind that Story Points usually follow the
fibanacci series (i.e. ½, 1, 2, 3, 5, 8, 13, 20, 40, 100)

Important! Since the JIRA API allow you to connect directly with an online repository, there are some
restrictions to avoid the overload of the servers. For example, JIRA API only allow for extracting bulks of issues
(n=1000). Thus, it is important to take this into account when you are making request to some server.

See more details about this on https://confluence.atlassian.com/jirakb/export-more-than-1000-issues-to-excel-

from-issue-navigator-in-jira-cloud-779160811.html (https://confluence.atlassian.com/jirakb/export-more-than-
1000-issues-to-excel-from-issue-navigator-in-jira-cloud-779160811.html)

In [3]:

# Connect with Spring

from jira import JIRA

jira = JIRA('https://jira.spring.io')

In [4]:

# it is a good idea to save the issues in a dataframe

import pandas as pd

issues = pd.DataFrame()
In [5]:

# first solution
# the easy (and lazy) way

issues_in_proj_1 = jira.search_issues('project=XD', startAt=0, maxResults=1000)

issues_in_proj_2 = jira.search_issues('project=XD', startAt=1000, maxResults=1000)
issues_in_proj_3 = jira.search_issues('project=XD', startAt=2000, maxResults=1000)
issues_in_proj_4 = jira.search_issues('project=XD', startAt=3000, maxResults=1000)

In [6]:

print len(issues_in_proj_1)
print len(issues_in_proj_2)
print len(issues_in_proj_3)
print len(issues_in_proj_4)

1000
1000
1000
706

In [7]:

# Second solution
# Getting all the issues in only one block

from jira import JIRA

jira = JIRA('https://jira.spring.io')

block_size = 1000
block_num = 0
allissues = []
while True:
start_idx = block_num*block_size
issues = jira.search_issues('project=XD', start_idx, block_size)
if len(issues) == 0:
# Retrieve issues until there are no more to come
break
block_num += 1
for issue in issues:
#log.info('%s: %s' % (issue.key, issue.fields.summary))
allissues.append(issue)

In [8]:

print 'Number of issues:', len(allissues)

Number of issues: 3706

In [9]:

## Into pandas

import pandas as pd

issues = pd.DataFrame()

for issue in allissues:

d = {
'key': issue.key,
'assignee': issue.fields.assignee,
'creator' : issue.fields.creator,
'reporter': issue.fields.reporter,
'created' : issue.fields.created,
'components': issue.fields.components,
'description': issue.fields.description,
'summary': issue.fields.summary,
'fixVersions': issue.fields.fixVersions,
'subtask': issue.fields.issuetype.subtask,
'issuetype': issue.fields.issuetype.name,
'priority': issue.fields.priority.name,
'resolution': issue.fields.resolution,
'resolution.date': issue.fields.resolutiondate,
'status.name': issue.fields.status.name,
'status.description': issue.fields.status.description,
'updated': issue.fields.updated,
'versions': issue.fields.versions,
'watches': issue.fields.watches.watchCount,
'storypoints': issue.fields.customfield_10142
}

issues = issues.append(d, ignore_index=True)

print len(issues)
issues.head()

3706
Out[9]:

assignee components created creator description fixV

The jobs that appear

2017-07- abhineet
0 None [] under Executions section []
10T12:41:25.000+0000 kumar
...

Working with Spring-XD

2017-06- Manuel
1 None [] version []
26T15:26:26.000+0000 Jordan
1.3.2.RELEASE\n...
 assignee components created creator description fixV

Keerthi
[Stream 2017-05- My project 7 node cluster
2 None kanth []
Module] 19T21:28:43.000+0000 and in that 2 node a...
Nagaraj

See
Janne 2017-03- Mark
3 [] https://github.com/spring- [1.3
Valkealahti 21T16:54:43.000+0000 Pollack
projects/spring-...

2017-03- Pavan I'm trying to run a Job on

4 None [Batch] []
06T20:01:41.000+0000 Srikar SpringXD and the jo...

Save your results to a file

In [10]:

issues.to_csv("issues-xd.csv", encoding='utf-8', header=True, index=False, line_terminator=

Extracting the changelog

There are two options to get the changelog.

1. Getting blocks of issues with additional information (e.g. the changelog)

2. Retrieve each issue individually with its additional information (e.g. the changelog)

If we opt for the first option (1)

issues = jira.search_issues('project=XD', start_idx, block_size, expand='changelo

g')

If we opt for the second option (2)

issue = jira.issue('FOO-100', expand='changelog')

Here, you have to extract data about the changes in the issues (number of fields to extract=6):

Issue Key. The key of the issue that have been updated.
Author: The key of the author who have made the change.
Date. The timestamp indicating when the change have been done.
Field. The updated field.
From. The field value before the update.
To. The field value after the update.
In [11]:

import pandas as pd

changelog = pd.DataFrame()

for issue in allissues:

issue = jira.issue(issue.key, expand='changelog')

achangelog = issue.changelog

for history in achangelog.histories:

for item in history.items:
d = {
'key': issue.key,
'author': history.author,
'date': history.created,
'field': item.field,
'fieldtype' : item.fieldtype,
'from': getattr(item, 'from'), # because using item.from doesn't wor
'fromString' : item.fromString,
'to': item.to,
'toString': item.toString
}

changelog = changelog.append(d, ignore_index=True)

print "Number of records: ", len(changelog)

Number of records: 43222

In [12]:

changelog.head()

Out[12]:

author date field fieldtype from fromString key to

Mark 2017-03- XD-

0 Fix Version jira None None 15491
Pollack 21T16:54:56.170+0000 3765

Mark 2017-03- XD-

1 status jira 10000 To Do 10001
Pollack 21T16:55:23.057+0000 3765

Mark 2017-03- XD-

2 resolution jira None None 8
Pollack 21T16:55:23.057+0000 3765

Fix stream
Mark 2017-03- XD-
3 summary jira None failover on None
Pollack 22T18:27:00.900+0000 3765
YARN

Krzysztof 2017-01- XD-

4 Attachment jira None None 23691
Noga 23T10:21:47.638+0000 3763

Save the dataset to a file

In [13]:

# backup! save to file

changelog.to_csv("changelog-xd.csv", sep=',', encoding='utf-8', doublequote = True, header=

Task 1.2 Pre-processing and cleaning the data

In this part, you will have to apply the following criteria to clean the dataset. The criteria consist of 10 cleaning
steps.

In [14]:

issues = pd.read_csv('issues-xd.csv')
changelog = pd.read_csv('changelog-xd.csv')

In [15]:

issues.columns

Out[15]:

Index([u'assignee', u'components', u'created', u'creator', u'description',

u'fixVersions', u'issuetype', u'key', u'priority', u'reporter',
u'resolution', u'resolution.date', u'status.description',
u'status.name', u'storypoints', u'subtask', u'summary', u'updated',
u'versions', u'watches'],
dtype='object')

Remove all the issues with null or zero values

In [16]:

to_remove = issues[ (issues['storypoints'] == 0.0) | (pd.isnull(issues['storypoints']))]

df = issues[ ~issues['key'].isin(to_remove['key']) ]

print 'before/after =', len(issues), '/', len(df)

before/after = 3706 / 3541

Remove all the issues whose story points have been updated more than once, since they can
represent misleading information. According to most of the estimation techniques, Story points must
have been assigned once and never updated afterward.
In [17]:

# Filtering all the user stories that have been updated in the story points field
issues = df

to_remove = changelog[((changelog['field'] == 'Story Points') | ( changelog['field'] == 'Ac

| ( changelog['field'] == 'Story Size' ) | ( changelog['field'] == '
| ( changelog['field'] == 'Effort points' ) | ( changelog['field'] =
| ( changelog['field'] == 'Effort' ) | ( changelog['field'] == 'Poin
& (changelog['fromString'].notnull()) ]

df = issues[ ~issues['key'].isin(to_remove['key']) ]

print 'before/after =', len(issues), '/', len(df)

before/after = 3541 / 3064

Remove all the issues that have not been addressed. We consider that an issue is addressed when its
Status is set to Closed (or similar, e.g. Fixed, Completed) and its resolution field is set to Fixed (or
similar, e.g. Done, Completed).

In [18]:

print issues['status.name'].unique()
print issues['resolution'].unique()

['To Do' 'Done' 'In PR' 'In Progress']

[nan 'Complete' 'Invalid' 'Works as Designed' 'Deferred' "Won't Fix"
'Cannot Reproduce' 'Incomplete' 'Duplicate' 'Fixed']

In [19]:

issues = df

df = issues[ ~issues['status.name'].isin(['Done']) | ~issues['resolution'].isin(['Complete'

print 'before/after =', len(issues), '/', len(df)

before/after = 3064 / 1035

Remove all the issues whose Story Points or Description fields have been updated after they were
addressed. Note that fields such as Title and Description may be adjusted or updated at any given
time. However, once the issue is addressed updates rarely happen. Issue report that have been
updated after they were addressed; they are likely to be unstable

In [20]:

changelog.columns

Out[20]:

Index([u'author', u'date', u'field', u'fieldtype', u'from', u'fromString',

u'key', u'to', u'toString'],
dtype='object')
In [21]:

issues = df

to_remove = []
for ix, line in issues.iterrows():
date = pd.to_datetime(line['resolution.date'])
key = line['key']
if ( pd.notnull(date) ):
key_remove = changelog[ (changelog['key'] == key) &
((changelog['field'] == 'description') | (changelog['field']
(pd.to_datetime(changelog['date']) > date) ]['key']
to_remove.append(key_remove)

df = issues[ ~issues['key'].isin(to_remove) ]

print 'before/after =', len(issues), '/', len(df)

C:\Anaconda2\envs\gl-env\lib\site-packages\pandas\core\dtypes\missing.py:28
9: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
if left.shape != right.shape:

before/after = 1035 / 1035

Remove all the issues whose story points are not in the Fibonacci series (i.e. ½, 1, 2, 3, 5, 8, 13, 20,
40, 100)

In [22]:

issues = df

FIBO_CARDS=[0.5, 1, 2, 3, 5, 8, 13, 20, 40, 100]

df = issues[ issues['storypoints'].isin(FIBO_CARDS) ]

print 'before/after =', len(issues), '/', len(df)

before/after = 1035 / 943

Remove all the issues whose informative fields are updated after the story points initialization, since
they are considered as unstable issues. We define informative fields: Issue Type, Description,
Summary, and Component/s.

In [23]:

changelog['field'].unique()

Out[23]:
array(['Fix Version', 'status', 'resolution', 'summary', 'Attachment',
'description', 'Version', 'Component', 'priority', 'issuetype',
'Rank', 'labels', 'Link', 'Pull Request URL', 'assignee', 'Sprint',
'Epic Link', 'Actual Story Points', 'Acceptance Criteria',
'Story Points', 'RemoteIssueLink', 'Parent', 'Comment',
'Epic Child', 'Epic Name', 'Parent Issue', 'Epic Colour',
'reporter', 'Reference URL', 'environment', 'Workflow', 'Key',
'project', 'Out of Scope', 'Epic Status', 'Time Spent', 'WorklogId',
'timespent', 'timeestimate', 'Project'], dtype=object)
In [24]:

issues = df

to_remove = []
for ix, line in issues.iterrows():
date = pd.to_datetime(line['created'])
key = line['key']
key_remove = changelog[ (changelog['key'] == key) &
( (changelog['field'] == 'description') | (changelog['field'] ==
(changelog['field'] == 'summary') | (changelog['field'] == 'Co
(pd.to_datetime(changelog['date']) > date) ]['key']
to_remove.append(key_remove)

df = issues[ ~issues['key'].isin(to_remove) ]

print 'before/after =', len(issues), '/', len(df)

before/after = 943 / 943

Remove outliers. Use Tukey’s fences.

In [25]:

issues = df

import numpy as np

def remove_outliers(df, column, k = 1.5):

first_quartile = np.percentile(df[column], 25)
third_quartile = np.percentile(df[column], 75)
iqr = third_quartile - first_quartile

minval = first_quartile - k*iqr

maxval = third_quartile + k*iqr

return df[(df[column] > minval) | (df[column] < maxval)].copy()

# issues = issues[~pd.isnull(issues['storypoints'])]
df = remove_outliers(df, 'storypoints')

print 'before/after =', len(issues), '/', len(df)

before/after = 943 / 943

Remove the stop words from title and the summary.

In [26]:

issues = df

from nltk.corpus import stopwords

cachedStopWords = stopwords.words("english")

# extra cleaning
df = issues[ pd.notnull(issues['description']) ]

print 'before/after =', len(issues), '/', len(df)

issues = df

issues['description'] = issues['description'].apply( lambda x : ' '.join([word for word in

issues['summary'] = issues['summary'].apply( lambda x : ' '.join([word for word in x.split(

before/after = 943 / 869

C:\Anaconda2\envs\gl-env\lib\site-packages\ipykernel\__main__.py:14: Unicode
Warning: Unicode equal comparison failed to convert both arguments to Unicod
e - interpreting them as being unequal

In [27]:

issues[['description', 'summary']].head(2)

Out[27]:

description summary

0 The jobs appear Executions section Jobs spring... How I make job restartable spring xd

Working Spring-XD version 1.3.2.RELEASE admin config timezone command

1
Starti... work

Remove the code snippets from the title and the summary of the issues. The code snippets can be
easily identify by looking for the tag <code></code>

In [28]:

import re

original = pd.read_csv('issues-xd.csv')

issues = issues[ ~pd.isnull(issues['description']) ]

for ix, line in issues.iterrows():

m = re.search('{code}(.*){code}', line['description'], flags=re.DOTALL)
if m:
issues.loc[ix, 'description'] = line['description'][:m.start(0)] + line['descriptio
# issues.loc[ix, 'code'] = line['description'][m.start(0):m.end(0)]
In [29]:

pd.options.display.max_colwidth = -1

# the issue report XD-3751 has code in its description

print original[ original['key'] == 'XD-3751']['description']

print
#print after
print issues[ issues['key'] == 'XD-3751']['description']

17 In a case where reactor's ringbuffer is full and thus handling backpre

ssure by blocking `onNext`, shutdown phase where `onComplete` is send will c
ause a deadlock.\r\n\r\nThis is shown by a thread dump during a shutdown. Th
is will basically break further deployments for this stream in distributed m
ode while single node will show more errors during undeployment.\r\n\r\n{cod
e}\r\n"pool-7-thread-1" #58 prio=5 os_prio=0 tid=0x979fe800 nid=0x54de runna
ble [0x986ad000]\r\n java.lang.Thread.State: TIMED_WAITING (parking)\r\n\t
at sun.misc.Unsafe.park(Native Method)\r\n\tat java.util.concurrent.locks.Lo
ckSupport.parkNanos(LockSupport.java:338)\r\n\tat reactor.jarjar.com.lmax.di
sruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:122)\r\n\t
at reactor.jarjar.com.lmax.disruptor.SingleProducerSequencer.next(SingleProd
ucerSequencer.java:97)\r\n\tat reactor.jarjar.com.lmax.disruptor.RingBuffer.
next(RingBuffer.java:246)\r\n\tat reactor.core.processor.util.RingBufferSubs
criberUtils.onNext(RingBufferSubscriberUtils.java:30)\r\n\tat reactor.core.p
rocessor.RingBufferProcessor.onNext(RingBufferProcessor.java:575)\r\n{code}
\r\n\r\n{code}\r\n"main-EventThread" #19 daemon prio=5 os_prio=0 tid=0x9b93a
400 nid=0x54b1 runnable [0x9aefe000]\r\n java.lang.Thread.State: TIMED_WAI
TING (parking)\r\n\tat sun.misc.Unsafe.park(Native Method)\r\n\tat java.uti
l.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)\r\n\tat react
or.jarjar.com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequ
encer.java:122)\r\n\tat reactor.jarjar.com.lmax.disruptor.SingleProducerSequ
encer.next(SingleProducerSequencer.java:97)\r\n\tat reactor.jarjar.com.lmax.
disruptor.RingBuffer.next(RingBuffer.java:246)\r\n\tat reactor.core.processo
r.util.RingBufferSubscriberUtils.onComplete(RingBufferSubscriberUtils.java:5
4)\r\n\tat reactor.core.processor.RingBufferProcessor.onComplete(RingBufferP
rocessor.java:585)\r\n\tat org.springframework.xd.greenplum.gpfdist.GPFDistM
essageHandler.doStop(GPFDistMessageHandler.java:170)\r\n{code}\r\n\r\nI've b
een crafting workaround for this by trying to wait reactor stream/buffer to
get drained by gpdb and finally as last resort, forcing processor in reactor
to shutdown.
Name: description, dtype: object

17 In case reactor's ringbuffer full thus handling backpressure blocking

`onNext`, shutdown phase `onComplete` send cause deadlock. This shown thread
dump shutdown. This basically break deployments stream distributed mode sing
le node show errors undeployment. I've crafting workaround trying wait reac
tor stream/buffer get drained gpdb finally last resort, forcing processor re
actor shutdown.
Name: description, dtype: object

Remove all the completed issues whose don't have any asignee.
In [30]:

df = issues[issues['assignee'].notnull()]

print 'before/after =', len(issues), '/', len(df)

before/after = 869 / 222

Finally, save the cleaned dataset to a file.

In [31]:

issues = df

issues.to_csv("issues-xd-cleaned.csv", sep=',', encoding='utf-8', doublequote = True, heade

Task 2 Getting data from Github

In this task, you will extract data from the same project used in Task 1 (the Spring-XD project), but now the
source will be the GitHub repository instead of the JIRA project management tool.

Like in Task 1, this Task consist of two parts. In the first part (Task 2.1), you will create a dataset from the
repository whereas in the second one (Task 2.2) you will pre-process and clean the dataset.

[INFO] Creating a GitHub API Connection

Since GitHub implements OAuth, getting API access involves creating an account followed by one of two
possibilities:

1. creating an application to use as the consumer of the API or

2. creating a Personal Access Token that will be linked directly to your account

You can create your 'Personal API Access Token' from the Settings menu in your GitHub account.
https://github.com/settings/tokens (https://github.com/settings/tokens)

We’ll opt to take advantage of a Python library so that we can avoid some of the tedious details involved in
making requests, parsing responses, and handling pagination. In this particular case, we’ll use PyGithub,
which can be installed with the somewhat predictable pip install PyGithub.

https://github.com/PyGithub/PyGithub (https://github.com/PyGithub/PyGithub)
In [32]:

"""
Using PyGithub to query a particular repository
"""

from github import Github

# XXX: Specify your own access token here

ACCESS_TOKEN = '08e610638148003f5098f191324375149ee14d72'

# Specify a username and repository of interest for that user.

USER = 'spring-projects'
REPO = 'spring-xd'

client = Github(ACCESS_TOKEN, per_page=100)

user = client.get_user(USER)
repo = user.get_repo(REPO)

print "Rate limit remaining", client.rate_limiting

#print "Rate limit reached. Reset time: ", datetime.fromtimestamp(client.rate_limiting_rese

Rate limit remaining (4998, 5000)

Task 2.1 Extracting the data

Using the PyGithub (https://github.com/PyGithub/PyGithub) API, connect to the Spring XD Github repository
and extract the following data:

Contributors: a list of users who contribute to the project

Repositories: a list of repositories in which the contributors are involved
Contributors' stats: Weekly stats from the contributors
Repository
Week
Number of additions
Number of deletions
Number of commits
Contributor
Commits:
Author
Date
Changes
Additions
Deletions
Filename
Comment_count
Message

Contributors
In [33]:

# Get a list of people who have contributed to the repo.

contributors = [ s for s in repo.get_contributors() ]

print "Number of contributors: ", len(contributors)

Number of contributors: 44

Contributors' stats
Stats from contributors
http://pygithub.readthedocs.io/en/latest/github_objects/StatsContributor.html#statscontributor
(http://pygithub.readthedocs.io/en/latest/github_objects/StatsContributor.html#statscontributor)

In [36]:

stats_contributors = repo.get_stats_contributors()

stats_contributors_df = pd.DataFrame()
for sc in stats_contributors:
for w in sc.weeks:
d = {
"week" : w.w,
"additions" : w.a,
"deletions" : w.d,
"commits" : w.c,
"author" : sc.author.login,
"total" : sc.total
}
stats_contributors_df = stats_contributors_df.append(d, ignore_index=True)

In [37]:

print "Size: ", len(stats_contributors_df)

stats_contributors_df.head()

Size: 11616
Out[37]:

additions author commits deletions total week

0 0.0 LinkedList 0.0 0.0 1.0 2013-04-07

1 0.0 LinkedList 0.0 0.0 1.0 2013-04-14

2 0.0 LinkedList 0.0 0.0 1.0 2013-04-21

3 0.0 LinkedList 0.0 0.0 1.0 2013-04-28

4 0.0 LinkedList 0.0 0.0 1.0 2013-05-05

Save to a file
In [38]:

# Saving to a file

stats_contributors_df.to_csv("stats-contributors-xd.csv", index=False, encoding='utf-8')

In [39]:

# Aggregated stats

stats_contributors_df[['additions', 'author', 'commits', 'deletions']].groupby(by='author')

Out[39]:

additions commits deletions

author

BoykoAlex 383.0 2.0 84.0

LinkedList 5.0 1.0 5.0

aclement 13151.0 19.0 2109.0

aeisenberg 49221.0 4.0 98.0

agandhinit 244.0 1.0 69.0

Repositories
In [40]:

repositories = pd.DataFrame()
for i, c in enumerate(contributors):
print "Contributor: ", c.login

repos = c.get_repos()

print repos

for r in repos:
d = {
'contributor' : c.login,
'lang' : r.language,
'owner' : r.owner.login,
'repo' : r.name
}
repositories = repositories.append(d, ignore_index=True)
print "Processed ", i+1, " contributors."
print "Rate limit remaining", client.rate_limiting

print "Repositories: ", len(repositories)

repositories.head()
Rate limit remaining (4957, 5000)
Contributor: artembilan
<github.PaginatedList.PaginatedList instance at 0x0000000046181C88>
Processed 27 contributors.
Rate limit remaining (4955, 5000)
Contributor: BoykoAlex
<github.PaginatedList.PaginatedList instance at 0x00000000478C8A88>
Processed 28 contributors.
Rate limit remaining (4954, 5000)
Contributor: gregturn
<github.PaginatedList.PaginatedList instance at 0x0000000046831048>
Processed 29 contributors.
Rate limit remaining (4952, 5000)
Contributor: kdowbecki
<github.PaginatedList.PaginatedList instance at 0x0000000046E77448>
Processed 30 contributors.
Rate limit remaining (4951, 5000)
Contributor: morfeo8marc
<github.PaginatedList.PaginatedList instance at 0x0000000046267408>
Processed 31 contributors.
( )
Save to a file

In [41]:

# save to file
repositories.to_csv("contributors-xd.csv", index=False, encoding='utf-8')

Commits

In [42]:

commitsrepo = repo.get_commits()
In [43]:

import pandas as pd

commits = pd.DataFrame()

for i, commit in enumerate(commitsrepo):

try:
d = {
'author' : commit.author.login if commit.author.login is not None else commit.a
'changes' : commit.stats.total,
'additions' : commit.stats.additions,
'deletions' : commit.stats.deletions,
'files' : [f.filename for f in commit.files],
'author_date' : commit.commit.author.date,
'commiter_date' : commit.commit.committer.date,
'commiter' : commit.commit.committer.name,
'message' : commit.commit.message,
'comments' : [comment for comment in commit.get_comments()]
}
commits = commits.append(d, ignore_index=True)
except Exception, e: #ssl.SSLError:
print "Encountered an error fetching details of commit #", i, " Skipping."
print e

print "Number of commits: ", len(commits)

commits.head()
Encountered an error fetching details of commit # 1361 Skipping.
'NoneType' object has no attribute 'login'
Encountered an error fetching details of commit # 1362 Skipping.
'NoneType' object has no attribute 'login'
Encountered an error fetching details of commit # 1363 Skipping.
'NoneType' object has no attribute 'login'
Encountered an error fetching details of commit # 1364 Skipping.
'NoneType' object has no attribute 'login'
Encountered an error fetching details of commit # 1365 Skipping.
'NoneType' object has no attribute 'login'
Encountered an error fetching details of commit # 1369 Skipping.
'NoneType' object has no attribute 'login'
Encountered an error fetching details of commit # 1382 Skipping.
'NoneType' object has no attribute 'login'
Encountered an error fetching details of commit # 1385 Skipping.
'NoneType' object has no attribute 'login'
Encountered an error fetching details of commit # 1386 Skipping.
'NoneType' object has no attribute 'login'

Save to a file

In [44]:

# save to file
commits.to_csv("commits-xd.csv", index=False, encoding='utf-8')

Task 2.2 Pre-processing and cleaning the data

Apply the following criteria to clean the dataset:
 Remove commits with null message
Remove all the commits related to artifact release -- hint: the message begins with [artifactory-release]
Remove commits from authors who have been commit only once
Remove all the commits from changes that have been done to auxiliar files (i.e. gradle.properties,
README, docfiles)
Remove all the commits that are related to a given release

In [80]:

commits = pd.read_csv("commits-xd.csv")

In [81]:

commits.head()

Out[81]:

additions author author_date changes comments commiter commiter_date

spring- 2017-03-22 Spring 2017-03-22

0 1.0 2.0 []
buildmaster 19:23:27 Buildmaster 19:23:27

spring- 2017-03-22 Spring 2017-03-22

1 1.0 2.0 []
buildmaster 19:23:23 Buildmaster 19:23:23

2017-02-01 Janne 2017-02-01

2 5.0 jvalkeal 10.0 []
14:21:38 Valkealahti 14:21:38

2016-12-23 Mark 2016-12-23

3 1.0 markpollack 2.0 []
15:06:47 Pollack 15:06:47

2016-12-23 Mark 2016-12-23

4 4.0 markpollack 8.0 []
15:06:15 Pollack 15:06:15

1 - Remove commits with null message

In [82]:

print 'before' , len(commits)

commits = commits[ ~pd.isnull(commits['message']) ]

print 'after' , len(commits)

before 2097
after 2093

2 - Remove all the commits related to artifact release -- hint: the message begins with [artifactory-release]
In [83]:

print 'before' , len(commits)

commits = commits[~commits['message'].str.startswith('[artifactory-release]')]

print 'after' , len(commits)

before 2093
after 2064

3 - Remove commits from authors who have been commit only once

In [84]:

commits['author'].unique()

Out[84]:
array(['jvalkeal', 'markpollack', 'garyrussell', 'cppwfs', 'mbogoevici',
'ilayaperumalg', 'artembilan', 'dturanski', 'sabbyanandan',
'ghillert', 'trisberg', 'mminella', 'pperalta', 'aclement',
'BoykoAlex', 'ericbottard', 'LinkedList', 'sworisbreathing',
'htynkn', 'morfeo8marc', 'philwebb', 'fmarchand', 'smaldini',
'kdowbecki', 'twoseat', 'jbrisbin', 'thomasdarimont', 'agandhinit',
'markfisher', 'nebhale', 'spring-buildmaster', 'liujiong1982',
'sathiyas', 'wilkinsona', 'dsyer', 'fbiville', 'parikhkc',
'chrisjs', 'tekul', 'aeisenberg', 'jencompgeek', 'datianshi',
'kashyap-parikh', 'gregturn'], dtype=object)

In [85]:

# Reading from file

import pandas as pd
stats = pd.read_csv("stats-contributors-xd.csv")

In [86]:

stats.head()

Out[86]:

additions author commits deletions total week

0 0.0 LinkedList 0.0 0.0 1.0 2013-04-07

1 0.0 LinkedList 0.0 0.0 1.0 2013-04-14

2 0.0 LinkedList 0.0 0.0 1.0 2013-04-21

3 0.0 LinkedList 0.0 0.0 1.0 2013-04-28

4 0.0 LinkedList 0.0 0.0 1.0 2013-05-05

In [90]:

stats['author'].unique()

Out[90]:

array(['LinkedList', 'sworisbreathing', 'htynkn', 'philwebb', 'agandhinit',

'sathiyas', 'parikhkc', 'chrisjs', 'kashyap-parikh', 'artembilan',
'sabbyanandan', 'BoykoAlex', 'morfeo8marc', 'kdowbecki',
'thomasdarimont', 'nebhale', 'wilkinsona', 'fbiville', 'datianshi',
'gregturn', 'fmarchand', 'smaldini', 'twoseat', 'aeisenberg',
'liujiong1982', 'jbrisbin', 'pperalta', 'aclement', 'dsyer',
'jvalkeal', 'jencompgeek', 'mminella', 'spring-buildmaster',
'mbogoevici', 'cppwfs', 'dturanski', 'tekul', 'ghillert',
'ericbottard', 'markpollack', 'garyrussell', 'trisberg',
'markfisher', 'ilayaperumalg'], dtype=object)

In [91]:

authors_commits = stats.groupby(by='author')['commits'].sum()

In [92]:

authors_commits = authors_commits.reset_index()

In [93]:

authors_commits.head()

Out[93]:

author commits

0 BoykoAlex 2.0

1 LinkedList 1.0

2 aclement 19.0

3 aeisenberg 4.0

4 agandhinit 1.0

In [94]:

authors_commits[authors_commits['commits'] == 1]['author']

Out[94]:

1 LinkedList
4 agandhinit
6 chrisjs
17 htynkn
22 kashyap-parikh
31 parikhkc
32 philwebb
35 sathiyas
38 sworisbreathing
Name: author, dtype: object
In [95]:

print 'before', len(commits)

cs = authors_commits[authors_commits['commits'] == 1]['author']

commits = commits[ ~commits['author'].isin(cs) ]

print 'after' , len(commits)

before 2064
Number of authors to delete from the stats: 9
after 2055

In [96]:

commits['author'].unique()

Out[96]:
array(['jvalkeal', 'markpollack', 'garyrussell', 'cppwfs', 'mbogoevici',
'ilayaperumalg', 'artembilan', 'dturanski', 'sabbyanandan',
'ghillert', 'trisberg', 'mminella', 'pperalta', 'aclement',
'BoykoAlex', 'ericbottard', 'morfeo8marc', 'fmarchand', 'smaldini',
'kdowbecki', 'twoseat', 'jbrisbin', 'thomasdarimont', 'markfisher',
'nebhale', 'spring-buildmaster', 'liujiong1982', 'wilkinsona',
'dsyer', 'fbiville', 'tekul', 'aeisenberg', 'jencompgeek',
'datianshi', 'gregturn'], dtype=object)

4 - Remove all the commits from changes that have been done to auxiliar files (i.e. gradle.properties, README,
docfiles)

In [97]:

to_remove = []

for ix, line in commits.iterrows():

if( ('gradle.properties' in line['files']) | ('README' in line['files']) ):
commits = commits.drop(index=ix)

print 'after' , len(commits)

after 1967
In [98]:

commits.head()

Out[98]:

additions author author_date changes comments commiter commiter_date

2017-02-01 Janne 2017-02-01

2 5.0 jvalkeal 10.0 []
14:21:38 Valkealahti 14:21:38

2016-12-23 Mark 2016-12-23

4 4.0 markpollack 8.0 []
15:06:15 Pollack 15:06:15

2016-05-03 Artem 2016-11-01

5 32.0 garyrussell 36.0 []
18:13:30 Bilan 16:09:41

2016-02-29 Artem 2016-03-01

6 123.0 garyrussell 134.0 []
20:26:42 Bilan 00:39:36

2016-02-23 Michael 2016-02-23

9 16.0 cppwfs 19.0 []
19:25:44 Minella 21:51:09

5 - Remove all the commits that are related to a given release

In [99]:

commits = commits[ ~commits['message'].str.startswith('[Release') ]

print 'after' , len(commits)

after 1967
Finally, save the cleaned dataset to a file.

In [100]:

# save to file
commits.to_csv("commits-xd-cleaned.csv", index=False, encoding='utf-8')