Data Analysis and Machine

Learning with Kaggle

Copyright ©2021 Packt Publishing

This is an Early Access product. Every effort has been made in the preparation
of this book to ensure the accuracy of the information presented. However, the
content and extracts of this book may evolve as it is being developed to ensure it
is up-to-date.

All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.

The information contained in this book is sold without warranty, either express
or implied. Neither the author nor Packt Publishing or its dealers and distributors
will be held liable for any damages caused or alleged to have been caused
directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of

the companies and products mentioned in this book by the appropriate use of
capitals. However, Packt Publishing cannot guarantee the accuracy of this
information.

Early Access Publication: Data Analysis and Machine Learning with Kaggle

Early Access Production Reference: B17574

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham
B3 2PB, UK

ISBN: 978-1-80181-747-9

www.packt.com
Table of Contents
1. Data Analysis and Machine Learning with Kaggle: How to win
competitions on Kaggle and build a successful career in data science
2. Introducing data science competitions
I. The rise of data science competition platforms
II. Kaggle competition platform
III. Other competition platforms
IV. Stages of a competition
V. Types of competitions and examples
VI. Submission and leaderboard dynamics
VII. Computational resources
VIII. Teaming and networking
IX. Performance tiers and rankings
X. Criticism and opportunities
3. Organizing Data with Datasets
I. Setting up a dataset
II. Gathering the data
III. Using the Kaggle datasets outside of Kaggle
IV. Building around datasets
V. Legal caveats
4. Working and Learning with Kaggle Notebooks
I. Setting up a kernel
II. Upgrade to GCP
III. One step beyond
IV. Kaggle courses
Data Analysis and Machine
Learning with Kaggle: How to win
competitions on Kaggle and build a
successful career in data science
Welcome to Packt Early Access. We’re giving you an exclusive preview of this
book before it goes on sale. It can take many months to write a book, but our
authors have cutting-edge information to share with you today. Early Access
gives you an insight into the latest developments by making chapter drafts
available. The chapters may be a little rough around the edges right now, but our
authors will update them over time. You’ll be notified when a new version is
ready. This title is in development, with more chapters still to be written, which
means you have the opportunity to have your say about the content. We want to
publish books that provide useful information to you and other customers, so
we’ll send questionnaires out to you regularly. All feedback is helpful, so please
be open about your thoughts and opinions. Our editors will work their magic on
the text of the book, so we’d like your input on the technical elements and your
experience as a reader. We’ll also provide frequent updates on how our authors
have changed their chapters based on your feedback. You can dip in and out
of this book or follow along from start to finish; Early Access is designed to be
flexible. We hope you enjoy getting to know more about the process of writing a
Packt book. Join the exploration of new topics by contributing your ideas and
see them come to life in print.

1. Introducing Kaggle and Data Science Competitions

2. Organizing Data with Datasets
3. Working and Learning with Kaggle Notebooks
4. Leveraging Discussion Forums
5. Detailing Competition Tasks and Metrics
6. Designing Good Validation
7. Ensembling and Stacking Solutions
8. Modeling for Tabular Competitions
9. Modeling for Image Classification and Segmentation
10. Modeling for Natural Language Processing
11. Handling Simulation and Optimization Competitions
12. Creating Your Portfolio of Projects and Ideas
13. Finding New Professional Opportunities
Introducing data science
competitions
Competitive programming has a long story, starting in the 1970s with the first
editions of the ICPC, the “International Collegiate Programming Contest”.
In the original ICPC, small teams from universities and companies participated
in a competition that required solving a series of problems using a computer
program (at the beginning participants coded in FORTRAN). In order to achieve
a good final rank, teams had to display good skills in team working, problem
solving and programming.

The experience of participating in the heat of such a competition and the

opportunity to have a spotlight for recruiting companies provided the students
enough motivation and it made the competition popular for many years. Among
ICPC finalists, a few ones have become renowned. Among these, there is Adam
D'Angelo, the former CTO of Facebook and founder of Quora, Nikolai Durov,
the co-founder of Telegram Messenger, and Matei Zaharia, the creator of Apache
Spark. Together with many others professionals, they all share the same
experience: having taken part to an ICPC edition.

After ICPC, programming competitions flourished, especially after 2000, when

remote participation become more feasible, allowing international competitions
more easily and at a lower cost. The format is similar and simply the same for
most of these competitions: there is a series of problems and you have to code a
solution to solve them. The winners can then take a prize, but also make
themselves noticed by recruiting companies or simply become famous and
popular among their peers.

In this chapter, we will explore how competitive programming evolved into data
science competitions, why the Kaggle platform is the most popular site for such
competitions and how it works.

The rise of data science competition platforms

Commonly, problems in competitive programming range from combinatory to
number theory, graph theory, algorithmic game theory, computational geometry,
string analysis, and data structures. Recently also problems relative to artificial
intelligence have successfully emerged, in particular after the launch of the KDD
Cup, a contest in knowledge discovery and data mining, held by the Association
for Computing Machinery's (ACM) Special Interest Group (SIG) on
Knowledge Discovery and Data Mining, during its annual conference.

The first KDD cup, held in 1997, involved a problem on direct marketing for lift
curve optimization and it started a long series of competitions (you can find the
archives containing datasets, instructions, and winners at:
https://www.kdd.org/kdd-cup) that continues up to nowadays (here is the latest
available at the time of writing: https://www.kdd.org/kdd2020/kdd-cup). KDD
cups proved quite effective in establishing best practices with many published
papers describing solutions and techniques and competition dataset sharing that
has been useful for many practitioners for experimentation, education and
benchmarking.

The experience of competitive programming and KDD cups together gave rise to
data science competition platforms, platforms where companies can host data
science challenges that are somehow hard to solve and that could benefit from a
crowdsourcing approach. In fact, given the fact that there is no golden approach
for all the problems in data science, many problems require a time-consuming
approach of the kind of try-all-that you-can-try.

In fact, no algorithm on the long run can beat all the others on all the problems,
but each machine learning algorithm performs if and only if its space of
hypothesis comprises the solution. Yet you cannot know that beforehand, hence
you have to try and test to be assured that you are doing the right thing. You can
consult the no free lunch theorem for a theoretical explanation of this practical
truth, here is a complete article from Analytics India Magazine on the topic:
https://analyticsindiamag.com/what-are-the-no-free-lunch-theorems-in-data-
science/.

Crowdsourcing proves ideal in such conditions when you need to test in an

extensively manner algorithms and data transformations in order to find the best
possible combinations but you lack man and computer power for that. That’s
why for instance, governments and companies resort to competitions in order to
advance in certain fields. On the government side, we can quote DARPA and its
many competitions on self-driving cars, robotic operations, machine translation,
speaker identification, fingerprint recognition, information retrieval, OCR,
automatic target recognition, and many others. On the business side, we can
quote a company such as Netflix, which entrusted a competition in order to
improve its algorithm to predict user movie selection.

The Netflix competition was based on the idea of improving existing

collaborative filtering, whose purpose is simply to predict the potential rating of
a user for film, solely based on the previous ratings that the user gave on other
films without knowing in specific who the user is or what the films are. Since no
user description or movie title or description were available (all replaced by
identity codes), the competition required to develop smart ways to use the
available past ratings. The grand prize of USD $1,000,000 was to be assigned
only if the solution could improve the existing Netflix algorithm, Cinematch,
above a certain threshold. The competition run from 2006 to 2009 and in the end
saw a team made up by the fusion of many previously teams in competitions (a
team from Commendo Research & Consulting GmbH, Andreas Töscher and
Michael Jahrer, quite renown also in Kaggle competitions, two researchers from
AT&T Labs and two others from Yahoo!). In the end, winning the competition
required so much computation power and the ensembling of different solutions,
that teams were forced to merge in order to keep pace. Such situation, reflected
also on the actual usage of the solution by Netflix, which preferred not to
implement it, but to simply take the most interesting insight from it in order to
improve its existing Cinematch algorithm (you can read more about it on this
Wired article: https://www.wired.com/2012/04/netflix-prize-costs/). What
mattered more in the end of the Netflix competition has not been the solution per
se, which has been quickly superseded by the change in business of Netflix from
DVDs to online movies. The real benefit for both participants (who gained a
huge reputation in collaborative filtering) and the company (who could transfer
its improved recommendation knowledge to its new business) were the insights
that has been gained from participating in the competition (and we can advance
that knowledge will be also the leitmotiv of most of this book).

Kaggle competition platform

Other companies than Netflix indeed benefitted from data science competitions.
The list is indeed long but we can quote a few examples where the company
holding the competition reported a clear benefit from it. For instance, we can
quote the insurance company AllState that could improve its actuarial models
built by their experts thanks to a competition involving hundreds of data
scientists (https://www.kaggle.com/c/ClaimPredictionChallenge). As another
well documented example, we can also mention about General Electric that
could improve by 40% the industry standard on predicting arrival times of airline
flights thanks to an analogue competition (https://www.kaggle.com/c/flight).
Both these competitions were held on the Kaggle competition platform.

The Kaggle competition platform has until now held hundreds of competitions
and these two are just a couple of examples of companies that successfully used
its competitions to boost their own models and analytics efforts. Let’s take a step
back from specific competitions for a moment and let’s talk about the Kaggle
company, which is the common thread of all this book.

Kaggle took its first steps in February 2010 thanks to the idea of Anthony
Goldbloom, an Australian trained economist (he has a degree in Economics and
Econometrics from Melbourne University). After working at Australia's
Department of Treasury and in the Research department at the Reserve Bank of
Australia, Goldbloom worked in London as an intern at The Economist, the
international weekly newspaper on current affairs, international business,
politics, and technology. At The Economist he had occasion to write an article
about big data that inspired his idea of building a competition platform that
could crowdsource the best analytical experts in solving interesting machine
learning problems (https://www.smh.com.au/technology/from-bondi-to-the-big-
bucks-the-28yearold-whos-making-data-science-a-sport-20111104-1myq1.html).
Since the crowdsourcing dynamics had a relevant part in the business idea for
this platform, he derived the name Kaggle, which recalls by rhyme the term
“gaggle” i.e. a flock of geese (the goose is also the symbol of the platform).

After moving to the Silicon Valley in the USA, his Kaggle start-up received
$11.25 million in Series A funding from a round led by Khosla Ventures and
Index Ventures, two quite renown venture capital firms. First competitions rolled
out, the community grew and some of the initial competitors came to become
quite prominent, such as Jeremy Howards, the Australian data scientist and
entrepreneur, who, after winning a couple of competitions on Kaggle, become
the President and Chief Scientist of the company. Jeremy Howard left his
position as President in December 2013 and thereafter he started a new start-up,
fast.ai (www.fast.ai), offering machine learning courses and a deep learning
library for coders.
At the times there were other prominent Kagglers (the name to indicate frequent
participants to competitions held by Kaggle) such as Jeremy Achin and Thomas
de Godoy. After reaching the top 20 global rankings on the platform, they
promptly decided to retire and to found their own company, DataRobot. They
soon after started hiring their best employers among the participants in the
Kaggle competitions in order to instill the best machine learning knowledge and
practice into the software they were developing. Today DataRobot is an
undoubted leader in autoML (automatic machine learning).

The Kaggle competitions claimed more and more attention from a larger
audience and even Geoffrey Hinton, the Godfather of deep learning participated
(and won) in a Kaggle competition hosted by Merck in 2012
(https://www.kaggle.com/c/MerckActivity/overview/winners). Kaggle has also
been the platform where Francois Chollet launched his deep learning package
Keras during the Otto Group Product Classification Challenge
(https://www.kaggle.com/c/otto-group-product-classification-
challenge/discussion/13632) and Tianqi Chen launched XGBoost, a speedier and
more accurate version of the gradient boosting machines, in the Higgs Boson
Machine Learning Challenge (https://www.kaggle.com/c/higgs-
boson/discussion/10335).

Competition after competition the community revolving around Kaggle grew to

touch one million in 2017, the same year as, during her keynote at Google Next,
Fei-Fei Li, Chief Scientist at Google, announced that Google Alphabet was
going to acquire Kaggle. Since then Kaggle has become part of Google.

Today the Kaggle community is still active and growing. It has offered to many
of his participants opportunities to create their own company, to launch machine
learning software and packages, to get interviews on magazines
(https://www.wired.com/story/solve-these-tough-data-problems-and-watch-job-
offers-roll-in/), to arrange a course on Coursera
(https://www.coursera.org/learn/competitive-data-science), to write machine
learning books (https://twitter.com/antgoldbloom/status/745662719588589568),
to find their dream job and, most important, or just to learn more about skills and
technicalities about data science.

Other competition platforms

Though this book focused on competitions on Kaggle, we cannot forget that
many data competitions are held on private platforms or on other competitions
platforms. In truth, most of the information you will find in this book will hold
also for all the other competitions, since they all basically operate under similar
principals and the benefits for the participants are more or less the same as
Kaggle’s.

Since many other competition platforms are localized in specific countries or are
specialized in certain kinds of competitions, for completeness we will briefly
introduce some of them, at least those we have some experience and knowledge
of.

DrivenData (https://www.drivendata.org/competitions/) is crowdsourcing

competition platform devoted to social challenges (see
https://www.drivendata.co/blog/intro-to-machine-learning-social-impact/). The
company itself is a social enterprise whose aim is to bring data science solutions,
thanks to data scientists building algorithms for social good, to organizations
tackling the world’s biggest challenges. For instance you an read in this article,
https://www.engadget.com/facebook-ai-hate-speech-covid-19-160037191.html,
how Facebook has choosen DrivenData for its competition on building models
against hate speech and misinformation.

Numerai (https://numer.ai/) is an AI-powered, crowd-sourced hedge fund based

in San Francisco which hosts a weekly tournament in which you can submit your
predictions on hedge fund obfuscated data and earn your prizes in the company’s
crypto currency, Numeraire.

CrowdAnalytix (https://www.crowdanalytix.com/community) a bit less active

now, this platform used to host quite a few challenging competitions a short ago,
as you can read from this blog post: https://towardsdatascience.com/how-i-won-
top-five-in-a-deep-learning-competition-753c788cade1. Also the community
blog is quite interesting for having an idea of what challenges you can find on
this platform: https://www.crowdanalytix.com/jq/communityBlog/listBlog.html.

Signate (https://signate.jp/competitions) is a Japanese data science competition

platform. It is quite rich in contests and it offers a ranking system similar to
Kaggle’s one (https://signate.jp/users/rankings).

Zindi (https://zindi.africa/competitions) is a data science competition platform

from Africa. It hosts competitions focused on solving Africa’s most pressing
social, economic and environmental problems.

Alibaba Cloud (https://www.alibabacloud.com/campaign/tianchi-competitions)

is a Chinese cloud computer and AI provider who has launched the Tianchi
Academic competitions, partnering with academic conferences such as
SIGKDD, IJCAI-PRICAI and CVPR and featuring challenges such as image-
based 3D shape retrieval, 3D object reconstruction, or instance segmentation.

Analytics Vidhya (https://datahack.analyticsvidhya.com/) the largest Indian

community for data science, offers a platform for data science hackatons.

CodaLab (https://codalab.lri.fr/) is instead a French-based data science

competition platform, created as a joint venture between Microsoft and Stanford
University in 2013. They feature a similar Kernel feature (here called
Worksheets: https://worksheets.codalab.org/) for knowledge sharing and
reproducible modeling as Kaggle.

Other minor platforms are CrowdAI (https://www.crowdai.org/) from École

Polytechnique Fédérale de Lausanne in Switzerland, InnoCentive
(https://www.innocentive.com/), Grand-Challenge (https://grand-
challenge.org/) for biomedical imaging, DataFountain
(https://www.datafountain.cn/business?lang=en-US), OpenML
(https://www.openml.org/) and the list could go on. You can always find a list of
many on-running major competitions on the Russian community Open Data
Science (https://ods.ai/competitions) and thus discover even new competition
platforms from time to time.

The alternatives and opportunities besides Kaggle are quite a lot. The interesting
aspect of such an abundance of opportunities is that you can find more easily a
competition that could interests you more because of its specialization and data.
Also, expect less competitive pressure on these challenges since they are less
known and advertized. Also, expect less sharing among participants since no
other competition platform up to now has reached the same richness of sharing
and networking tools as Kaggle has.

Stages of a competition
A competition on Kaggle is arranged through different steps. By having a glance
at each of them, you can get a better understanding at how a data science
competition works and what to expect from it.

When a competition is launched, there are usually some posts on social media
(for instance on Kaggle Twitter profile: https://twitter.com/kaggle) that announce
it and a new tab will appear between active competitions at the page
https://www.kaggle.com/competitions. If you click on the competition’s tab,
you’ll be taken to the competition page. Immediately, at a glance you can at least
check if the competition will have prizes (and if it awards points and medals, a
secondary consequence of participating in a competition), how many teams are
at the moment involved and how much is still left for you to work on a solution.

There you can explore the overview menu first which will provide information
to you about the topic of the competition, its evaluation metric (your models will
be evaluated against that metric), the time line of the competition, the prizes and
the legal or competition requirements. Usually the time line is a bit overlook, but
it should be one of the first things you have to check, in fact it doesn’t tell you
simply when the competition starts and ends, but it will provide you with the
rules acceptance deadline, which is usually from seven day to two weeks before
the competitions closes. The rule acceptance deadline marks the time limit you
can join the competition, by accepting its rules, and the team merger deadline:
you can arrange to combine your team with other competitors’ one just before
that deadline, after that it won’t be possible. In addition, the rules menu is quite
often overlooked (with people just jumping to data), but it is important to check
on them because they can tell you about the requirements of the competition.
Among the key information you can get from the rules there is the eligibility for
a prize and a few other important details such as if you can use external data to
improve your score, how many submissions (tests of your solution) a day you
get, how many final solutions you can choose, and so on. Finally, you can have a
look at the data, though you can download it only after accepting the rules of the
competition.

Once you have accepted the rules, you can download any data or directly start
working on Kaggle Kernel, an only notebook, re-using code that others have
made available to other participants or creating your own code from scratch. If
you decide to download the data, also consider that you have a Kaggle API that
can help you to run downloads and submission in an almost automated way. You
can find more details about the API at https://www.kaggle.com/docs/api and you
can get the code from Github from https://github.com/Kaggle/kaggle-api. By the
way, if you closely check on Kaggle Github repo, you can also find all the
docker images they use for their online notebooks the Kaggle Kernels. At this
point, as you develop your solution, it is not a bad idea to contact other
competitors through the discussion forum: there you can ask and answer
questions. Often you will also find useful hints at specific problems with the data
or even useful ideas there to improve your solution.

Once your solution is ready, you can submit it to the Kaggle evaluation engine,
accordingly to the specification of the competition (some competitions will
accept a csv file as a solution, other will require you to code and produce results
in a Kaggle Kernel). During all the competition, you can keep submitting
solutions.

Every time you submit a solution, the leaderboard will provide you soon after,
depending on the computations necessary for the evaluation, with a score and a
position among the competitors. That position is only indicative anyway,
because it reflects the performance of your model on a part of the test set, called
as the public one since the performance on it is made public during the
competition for everyone to know. Only when the competition closes and the
contestants have decided on what among their model have to be scored, it is
reveled their score on another part of the test set, called the private one. This
new leaderboard, the private leaderboard, constitutes the final, effective rankings
of the competition.

When a competition is closed, the Kaggle team will take a certain time to check
that everything is correct and that all contestants have respected the rules of the
competition. After a while (and sometimes after some changes) the private
leaderboard will become definitive, the winners will be declared, and many
among participants, at their own will, will unveil their strategies, their solutions
and their code to others on the competition discussion forum.

Types of competitions and examples

Kaggle competitions are categorized based on “competitions categories” and
each category has different implications in terms of how to compete and what to
expect from each of them. Type of data, difficulty of the problem, awarded
prizes and competition dynamics are quite diverse inside the categories,
therefore it is important to understand beforehand what each implies.

Here are the official categories that you can use to filter out the different
competitions:

Featured
Masters
Annuals
Research
Recruitment
Getting started
Playground
Analytics
InClass

“Featured” are the most common type of competitions, featuring a business

related problem from a sponsor company and a prize for the top performers in
the competition. The winners will grant a non-exclusive license of their work to
the sponsor company and they will have to prepared a detailed report of their
solution and sometimes even participate in meetings with the sponsor company.

There are examples of Featured competitions every time you access to Kaggle.
At the moment, many of them are problems related to the application of deep
learning methods to unstructured data like text, images, videos or sound but in
the past tabular data competitions, that is competitions based on problems related
to structured data that can be found in a database. Now such competitions are
really less required because a crowdsourced solution would not often be such an
advance in respect of what a good team of data scientists or even an autoML
software could reach. First using random forests, then using gradient boosting
methods with clever feature engineering, in the past tabular data solution derived
from Kaggle could really largely improve a solution. Nowadays, given the
spread of better software and good practices, the obtainable increased results
from competitions could be indeed marginal. In the unstructured data world,
instead, a good deep learning solution could still do the difference since, for
instance, pretrained networks such as BERT brought about double-digit
increases in previous standards of many well known with text.

“Masters” are less usual now, but they are private, invite-only competitions. The
purpose was to create competitions only among experts (generally competitors
ranked as Masters or Grandmasters, based on Kaggle medal rankings), based on
their rankings on Kaggle.
“Annuals” are competitions that always appear on a certain period of the year.
Among the Annuals we have the Santa Claus competitions (usually based on an
algorithmic optimization problem) and the March Machine Learning
Competition, run every year since 2014 during the US College Basketball
Tournaments.

“Research” competitions imply a research or science purpose instead of a

business one, sometimes for serving the public good. That’s why these
competitions do not always offer prizes. In addition, these competitions
sometimes require the winning participants to release their solution as open-
source.

Google has released a few Research competitions in the past such as the Google
Landmark Recognition 2020 (https://www.kaggle.com/c/landmark-recognition-
2020) - Label famous (and not-so-famous) landmarks in images

Sponsors that want to test potential job candidates for their ability hold
“Recruitment” competitions. These competitions are limited to teams of one
and offer to best placed competitors an interview with the sponsor as a prize. The
competitors have to upload their curriculum vitae at the end of the competition if
they want to be considered for being contacted.

Examples of recruiting competitions have been:

The Facebook Recruiting Competition

(https://www.kaggle.com/c/FacebookRecruiting but actually Facebook held
a few ones of these kind)
The Yelp Recruiting Competition (https://www.kaggle.com/c/yelp-
recruiting)

“Getting started” competitions do not offer any prize but a friendly and easy
problem for beginners to get accustomed by Kaggle principles and dynamics.
Usually they are semi-permanent competitions whose leaderboard is refreshed
from time to time. If you are looking for a tutorial in machine learning, these
competitions are the right place where to start.

Famous on-running Getting Started competitions are:

Digit Recognizer (https://www.kaggle.com/c/digit-recognizer)

Titanic: Machine Learning from Disaster
 (https://www.kaggle.com/c/titanic) - Predict survival on the Titanic
Housing Prices: Advanced Regression Techniques
(https://www.kaggle.com/c/house-prices-advanced-regression-techniques)

“Playground” competitions are a little bit more difficult than the Getting Started
ones, but they are also meant for having competitors learn and test their abilities
without the pressure of a fully fledged Featured competition (though sometimes
also in Playground competitions the heat of the competition may turn quite
high). The usual prizes of such competitions are just swag or little money.

One famous Playground competition has been the original “Dogs vs Cats”
competition (https://www.kaggle.com/c/dogs-vs-cats) - Create an algorithm to
distinguish dogs from cats

A mention should be spent for Analytics competitions, where the evaluation is

qualitative and participants are required to provide ideas, drafts of solutions,
PowerPoint slides, charts and so on and the InClass which are competitions held
by academic institutions.

Cross-sectional to this taxonomy of Kaggle competitions, you also have to

consider that competitions may have different formats. The usual format is the
so-called “Simple format” where you provide a solution and it is evaluated as we
previously described. More sophisticated, the two-stage competition splits the
contested into two parts and the final dataset is released only after the first part
has completed and only to the participants of the first phase. The two-stage
competition format has emerged in order to limit the chance that some
competitor may cheat and infringe the rules since the evaluation is done on a
completely untried test set that it is available for a short time. For the same
reason, also the Code competitions have recently appeared, where all
submissions are made from a Kaggle Notebook, and any direct upload of
submissions is disabled.

Submission and leaderboard dynamics

Apparently, the way Kaggle works seems simple: the test set is hidden to
participants; you fit your model, if your model is the best in predicting the test
set, then you score high and you possibly win. Unfortunately, such description
renders the inner workings of Kaggle competitions in a too simplistic way and it
doesn’t take into account that there are dynamics regarding the direct and
indirect interactions of competitors between themselves and the nuances of the
problem you are facing and of its training and test set.

A more comprehensive description of how Kaggle works is actually given by

Professor David Donoho, professor of statistics at Stanford University
(https://web.stanford.edu/dept/statistics/cgi-bin/donoho/), in his writing “50
Years of Data Science”. The paper has first appeared in the Journal of
Computational and Graphical Statistics and then subsequently posted on the MIT
Computer Science and Artificial Intelligence Laboratory (see
http://courses.csail.mit.edu/18.337/2015/docs/50YearsDataScience.pdf).
Professor Donoho does not refer to Kaggle specifically, but to all data science
competition platforms generically. Quoting computational linguist Mark
Liberman, he refers to data science competitions and platforms as part of a
Common Task Framework (CTF) paradigm that has been silently and steadily
progressing data science in many fields during the last decades. He states that a
CTF can work incredibly well at improving the solution of a problem in data
science from an empirical point of view. He quotes as successful examples the
NetFlix competition and many DARPA competitions that have reshaped the best
in class solutions for problems in many fields.

A CTF is composed of “ingredients” and a “secret sauce” (see paragraph 6 of

Donoho, David. "50 years of data science." Journal of Computational and
Graphical Statistics 26.4 (2017): 745-766.). The ingredients are simply:

1. A public available dataset and a related prediction task

2. A set of competitors who share the common task of producing the best
prediction for the task
3. A system for scoring the predictions by the participants in a fair and
objective way, without providing too specific hints (or limiting them at
most) about the solution

The system works the best if the task is well defined and the data is of good
quality. On the long run, the performance of solutions improves by small gains
until they reach an asymptote. The process can be speeded up by allowing a
certain amount of sharing among participants (as it happens on Kaggle by means
of discussions and sharing Kernel notebooks and extra data by Datasets). It can
be noticed that the only competitive pressure, in spite of any degree of sharing
among participants doesn’t stop the improvement in the solution, it just makes it
slower.
This is because the secret sauce in the CTF paradigm is the competition itself,
that, in the framework of a practical problem whose empirical performance has
to be improved, always leads to the emergence of new benchmarks, new data
and modeling solutions, and in general to an improved application of machine
learning to the problem objected of the competition. A competition can therefore
provide a new way to solve a prediction problem, new ways of feature
engineering, new algorithmic or modeling solutions. For instance, deep learning
has not simply emerged from academic research, but it has first gained a great
boost because of successful competitions that declared its efficacy (we have
already mentioned for instance the Merck competition won by Geoffrey Hinton’s
team https://www.kaggle.com/c/MerckActivity/overview/winners).

Coupled with the open software movement, which allows everyone to access to
powerful analytical tools (such as Scikit-learn or TensorFlow or PyTorch), the
CTF paradigm brings even better results because all competitors are on the same
line at the start. On the other end, the reliance of a solution to a competition on
specialized or improved hardware can limit the achievable results because it can
prevent competitors without access to such resources to properly participate and
contribute directly to the solution or indirectly by exercising competitive
pressure on the other participants. Understandably, that the reason why Kaggle
started also offering cloud services for free to participants to its competitions
(the Kernels): it can flatten some differences in hardware intense competitions
(like most deep learning ones are) and increase the overall competitive pressure.

There are, anyway, occurrences that can go wrong and instead led to a
suboptimal result in a competition:

1. Leakage from the data

2. Probing from the Leaderboard (the scoring system)
3. Overfitting and consequent shake-up
4. Private Sharing

You have leakage from data when part of the solution is to be retraced in the data
itself. For instance, certain variables are posterior to the target variable (so they
reveal something of it) or the ordering of the training and test examples or some
identifier is evocative of the solution. Such solution leakage, sometimes named
by the competitors as “golden features” (because getting a hint of such nuances
in the data problem could turn into gold prizes for the participants) invariably
leads to a not reusable solution. It also implies a suboptimal result for the
sponsor (who at least should have learned something about leaking features that
should affect a solution to his/her problem).

Another problem is the possibility to probe a solution from the leaderboard. In

such situation, you can snoop the solution by repeated submission trials on the
leaderboard. Again, in this case the solution is completely unusable in different
circumstances. A clear example of such happened in the competition “Don’t
Overfit II” (https://www.kaggle.com/c/dont-overfit-ii). In this competition, the
winning participant, Zachary Mayers, submitted every individual variable as a
single submission, gaining information on the possible weight of each variable
that allowed him to estimate the correct coefficients for his model (you can read
Zach’s detailed solution here: https://www.kaggle.com/c/dont-overfit-
ii/discussion/91766). Generally, time series problems (or other problems where
there are systematic shifts in the test data) may be seriously affected by probing,
since they can help competitors to successfully define some kind of ‘post-
processing’ (like multiplying their predictions by a constant) that is most suitable
for scoring high on the specific test set.

Another form of leaderboard snooping happens when participants tend to rely

more on the feedback from the public leaderboard than their own tests.
Sometimes such turns into a complete failure of the competition, with a wild
shake-up, that is a complete and unpredictable reshuffling of the competitors’
positions on the final leaderboard. The winning solutions, in such a case, may
(but not always) turn out to be not so optimal for the problem or even sometimes
just dictated by chance. This has led to precise analysis by competitors if the
training set is much different from the test set they have to guess. Such analysis,
called “Adversarial testing” can provide insight if to rely on the leaderboard and
if there are features that are too different from the training and test set to be
better avoided completely (For an example, you can have a look at this Kernel
by Bojan Tunguz: https://www.kaggle.com/tunguz/adversarial-ieee). Another
form of defense against leaderboard overfitting is choosing safe strategies to
avoid submitting solutions too based on the leaderboard results. For instance,
since two solutions are allowed to be chosen by each participant for the
evaluation at the end of the competition, a good strategy is to submit the best one
performing on the leaderboard and the best performing one based on one’s own
cross-validation tests.

In order to avoid problems with leaderboard probing and overfitting, Kaggle has
recently introduced different innovations based on Code Competitions, where
the evaluation is split into two distinct stages, that is you have a test set for the
public leaderboard (the leaderboard you follow during the competition) and a
completely hold out test set for the final private leaderboard. In this way,
participants are actually blind to the actual data their solutions will be evaluated
against and they should be forced to consider more their own tests and a more
general solution against a solution specific to the test set.

Finally, another possible distortion of a competition is due to private sharing

(sharing ideas and solutions in a closed circle of participants) and other illicit
moves such as playing by multiple accounts or playing in multiple teams and
stealing ideas from each in favor of another team. All such actions create an
asymmetry in information between participants that can be favorable to a few
and detrimental to the most. Again, the resulting solution may be affected
because sharing has been imperfect during the competition and fewer teams have
been able to exercise full competitive pressure. Moreover, if such situations
appear evident to participants (for instance see https://www.kaggle.com/c/ashrae-
energy-prediction/discussion/122503), it can lead to distrust and less
involvement in the competition or in following competitions.

Computational resources
Some competitions do pose limitations in order to render available to production
feasible solutions, for instance the Bosh Production Line Performance
competition - https://www.kaggle.com/c/bosch-production-line-performance -
had strict limits on execution time, model file output and memory limit for your
solution. Also Kernel based competitions, when requiring both training and
inference to be executed on Kernels, do not pose a problem for the resources you
have to use because Kaggle will provide with all the resources you need (and
this is also intended as a way to put all participants on the same line for a better
competition result).

Problems for you arise when you have kernel competitions just limited to
inference time and therefore you can train your models on your own machine
and the only limit is then based at test time on the number and complexity of
models you produce. Since at moment most competitions require deep learning
solutions, you have to consider that you surely need specialized hardware such
as GPUs in order to achieve some interesting result in a competition. Anyway,
also if you participate in some of the now rare tabular competitions, you’ll soon
realize that you need a strong machine with quite a number of processors and
memory in order to easily apply feature engineering to data, run experiments and
build models quickly.

Standards do change rapidly, therefore it is difficult to mention a standard

hardware that you should have in order to compete at least on the same league
with others. We can anyway take a hint at such standard by looking at what other
competitors are using, as their own machine or as a machine on the cloud.

For instance, recently HP has launched a program where it awarded a HP Z4 or

Z8 to a few selected Kaggle participants in exchange with visibility for its brand.
For instance, a Z8 machine has 56 cores, 3TB of memory, 48TB of storage (a
good share by solid storage hard drives) and a NVIDIA RTX as GPU. We
understand that such could be a bit out of reach for many as well as even also
renting a similar machine for a short time on a cloud instance such as Google’s
GCP or Amazon’s AWS is out of discussion for the consequent expenses for
even a moderate usage.

Our suggestion, unless your ambition is to climb to the top rankings of Kaggle
participants is therefore to go with the machines provided free by Kaggle, the
Kaggle Notebooks (also previously known as the Kaggle Kernels).

Kaggle Notebooks are a versioned computational environment, based on Docker

containers running in cloud machines, which allow you to write and execute
both scripts and notebooks in R and Python languages. The Kaggle Notebooks
are integrated into the Kaggle environment (you can make submissions from
them and keep track what submission refers to what Notebook), they come with
most data science packages pre-installed, and they allow some customization
(you can download files and install further packages). The basic Kaggle
Notebook is just CPU based, but you can have versions boosted by a NVIDIA
Tesla P100 or a TPU v3-8 (TPUs are hardware accelerators specialized in deep
learning tasks). Though bounded by a usage number and time quota limit,
Kaggle Notebooks provide the computational workhorse to build your baseline
solutions on Kaggle competitions:

A CPU Notebook owns 4 CPU cores and 16 GB of memory, you can run 10
Notebooks of this kind at a time but you don’t have any time quote for them
A GPU features 2 CPU cores and 13 GB of memory, you can run 2
Notebooks of this kind at a time and you have a 30 hours weekly quota for
 such kind of Notebook
A TPU features 4 CPU cores and 16 GB of memory, you can run 2
Notebooks of this kind at a time you have a 30 hours weekly quota for such
kind of Notebook

All Notebooks can run for 9 hours maximum, and have 20 GB disk saving
allowance to store your models and results plus an additional scratchpad disk
that can exceed 20 GBs for temporary usage during script running.

In certain cases, the GPU enhanced machine provided by Kaggle kernels may
not be enough. For instance, the recent Deepfake Detection Challenge
(https://www.kaggle.com/c/deepfake-detection-challenge) required to process
data consisting of about around 500 GB of videos. That is especially because of
the time limit of weekly usage, that at the time of this writing is about 30 hours a
week and because of the fact that you cannot have more than two machines with
GPU running at the same time (10 machines at a time is the limit for the CPU
only instances). Even if you can double your machine time by changing your
code to leverage the usage of TPUs instead of GPUs (and you can find some
guidance for achieving that easily here: https://www.kaggle.com/docs/tpu), that
may still prove not enough for fast experimentation on a data heavy competitions
such as the Deepfake Detection Challenge. That’s the reason in the chapter
devoted to Kaggle Kernel we are going to provide you with many tips and tricks
for successful coping with such limitations with decent results without having to
buy a heavy performing machine. We are also going to show you how to
integrate Kaggle Kernels with Google Cloud Services (GCP) or simply how to
move away all your work on another cloud based solution.

Teaming and networking

However, the computational power plays its part, the human expertise and ability
only can make the real difference in a Kaggle competition. Sometimes a
competition to be handled successfully requires the collaborative efforts of a
team of contestants. Apart from recruitment competitions, where the sponsor
may require individual participants for a better evaluation of their abilities,
usually there is no limit to forming teams during competitions. Usually the teams
can be at maximum made of five contestants. Teaming has its own advantages
because it can multiply the efforts on finding a better solution, since a team can
spend more time all together on the problem and different skills can be of great
help: not all data scientists have the same skills or the same level of skill in the
different models and data manipulations.

Anyway, teaming is not all positive. Coordinating different individuals and

different efforts toward a common goal may prove not so easy and some
suboptimal situations may arise. The usual problem with teams is when part of
the participants is not involved or simply idle, but the worst is surely when, a
more rare occurrence, someone infringes the rules of the competition (to the
detrimental of everyone since the all team could be disqualified) or even spies on
the team in order to advantage other ones.

In spite of any negative side, teaming in Kaggle competition is a great

opportunity to know better other data scientists, to collaborate for a purpose and
to achieve more, since Kaggle rules do reward teams in respect of lonely
competitors. Teaming together is not the only possibility of networking in
Kaggle, though it is certainly the more profitable and interesting for the
participants. You can actually network with others by discussions on the forums,
by sharing datasets and notebooks during competitions. All these opportunities
on the platform can help you to know other data scientist and to be recognized
by them.

There are also quite many occasions to network with other Kagglers outside of
the Kaggle platform itself. First of all, there are a few Slack channels that can be
helpful. For instance, KaggleNoobs (see: https://www.kaggle.com/getting-
started/20577) is a channel, opened up 5 years ago, that feature many discussions
about Kaggle competitions and they have a supportive community that can help
you if you have some specific problem with code and models. There are quite a
few other channels around devoted to exchanging opinions about Kaggle
competitions and data science related topics. Some channels are organized on a
regional or national basis. For instance, the Japanese channel Kaggler-ja
(http://kaggler-ja-wiki.herokuapp.com/) or the Russian community, created six
years ago, Open Data Science Network (https://ods.ai/) which later opened also
to non-Russian speaking participants. The Open Data Science Network (mostly
simply known as ODS) doesn’t simply offer a Slack channel but also courses on
how to win competitions, events, and reporting on active competitions around on
all known data science platforms (see https://ods.ai/competitions).

Apart from Slack channels, also quite a lot of local meetups themed about
Kaggle in general or about specific competitions have sprout out, some for short
time, others for longer. A meetup on Kaggle competition, usually built around a
presentation from a competitor who wants to share her or his experience and
suggestions, is the best situation to meet other Kagglers in person, to exchange
opinions and to build alliances for participating together in data science contests.
In this league, a mention apart is for Kaggle Days (https://kaggledays.com/),
built by Maria Parysz and Paweł Jankiewicz, a renowned Kaggle competitor.
The Kaggle Days organization arranged a few events in major locations around
the World (https://kaggledays.com/about-us/) with the aim of bringing together a
conference of Kaggle experts (which had to come to an abrupt stop due to the
COVID-19 pandemic) and it created a network of local meetups in different
countries which are still quite active (https://kaggledays.com/meetups/).

Performance tiers and rankings

Apart from monetary prizes, Kaggle offers many other more immaterial awards
(apart from some material ones such as cups, t-shorts, hoodies and stickers). The
point is that Kagglers, the participants in Kaggle competitions, do spend really a
lot of time and efforts when in competition (not to count in the specialty skills
they put on that in truth are quite rare in the general population). The monetary
prizes usually cover the efforts of the few top ones, if not of the only top one,
leaving the rest with an astonishing amount of hours just voluntary spent for no
return. On the long, being in competition with no tangible result may lead to
disaffection and disinterest, thus lowering the competitive intensity. Hence,
Kaggle has found at least a way to reward competitors with an honor system
based on medals and points. The idea is that the more medals and the more
points, the more relevant are ones skills, opening for opportunities in job search
or any other relevant activity based on reputation.

First, there is a general leaderboard, that combines all the leaderboards of the
single competitions. In this general leaderboard
(https://www.kaggle.com/rankings), one is ranked based on the position on each
single competition she or he took, which awards some points that all summed
together provide the ranking in the general leaderboard. At first glance, the
formula for the scoring of the points in a competition may look a bit complex:
Nevertheless, in reality it is simply based on a few ingredients: the rank in a
competition, your team size, the popularity of the competition and how much the
competition is old.

Intuitively, ranking high in popular competitions brings many points. Less

intuitively, the size of your team matters in a non-linear way. That’s due to the
inverse square root part of the formula since the part of points you have to give
up grows with the number of people involved but it is still quite favorable if your
team is relatively small (2, max 3 people) due the advantage in wits and
computational power brought about by larger collaborative teams.

Another point to keep in mind is that point decay with time. The decaying is not
linear, but as a rule of thumb just think that after a year very little is left of the
points you gained. Therefore, glory on the general leaderboard of Kaggle cannot
last long and it is ephemeral unless you keep on participating on competitions
coming up with similar results as before. As a consolation, on your profile you’ll
always keep the highest rank you ever reach, as a memento of your great
combined results at a certain time.

More last longing is the medal system that covers all the four aspects of
competing in Kaggle. You will be prized with medals in competitions,
notebooks, discussion and datasets based on your results. In competitions,
medals are awarded based on your position on the leaderboard. In other areas
such as discussion, notebook and datasets medals are awarded based on the
upvotes of other competitors (which actually led sometimes to some suboptimal
situation since upvotes are a less objective metric and also depends on
popularity). The more medals you get, the higher ranks of Kaggle mastery you
can enter. The ranks are classified in Novice, Contributor, Expert, Master, and
Grandmaster. The page https://www.kaggle.com/ progression explains
everything about how to get medals and how many and of what kind are needed
to access the different ranks.

Please keep in mind that such ranks and honors are always relative and that they
do change in time. A few years ago, in fact the scoring system and the ranks
were quite different. Most probably in the future, the ranks will change again in
order to keep the higher ones rarer and thus more valuable.

Criticism and opportunities

Kaggle has drawn quite a few criticisms since its start and if participating to data
science competitions is still quite debated today by many claiming different
negative or positive opinions.

On the side of negative criticism:

Kaggle provides a false perception of what machine learning really is since

it is just focused on leaderboard dynamics
Kaggle is just a game of hyper-parameter optimization and ensembling
many models just for scraping a little more accuracy (while in reality
overfitting the test set)
Kaggle is filled with in-experienced enthusiasts who are ready to try
anything under the sky in order to get a score and a spotlight in the hope to
be spotted by recruiters
As a further consequence, competition solutions are too complicated and
often too specific of a test set to be implemented

Many perceive Kaggle, as many other data science competition platform, far
from what data science is in reality. The point they raise are that business
problems are not given from nowhere and you seldom already have a well-
prepared dataset to start with since you usually built it along the way based on
refining the business specifications and understanding of the problem at hand.
Moreover, they emphasize that production is neither considered, since a winning
solution cannot be constrained by resource limits or considerations about
technical debt (though this is not always true for all competitions).

We cannot but not notice how all such criticism is related in the end about both
the fact that Kaggle is a crowdsourcing experience with a purpose (the CTF
paradigm) and how Kaggle ranking standings do relate in the data science world
in comparison with data science education and work experience. One persistent
myth that ailments criticism is in fact that Kaggle competitions may help getting
you a job or a better job in data science or that performing in Kaggle
competitions may put you on another plane in respect of data scientists that do
not participate at all.
Our stance on such a myth is that it is misleading belief that Kaggle rankings do
have an automatic value beyond the Kaggle community. For instance, in a job
search, Kaggle can provide you with some very useful competencies on
modeling data and problems and effective model testing. It can also expose you
to many techniques and different data/business problems (even beyond your
actual experience and comfort zone), but it cannot supplement you with
everything you need to successfully place yourself as a data scientist in a
company.

You can use Kaggle for learning and for differentiating yourself from other
candidates in a job search; however, how this will be considered will
considerably vary from company to company. Anyway, what will learn on
Kaggle will invariably prove useful throughout all of your career and will
provide you a hedge when you’ll have to solve complex and unusual problems
with data modeling because by participating in Kaggle competitions you build
up strong competencies in modeling and validating. You also network with other
data scientists and that can get you a reference for a job more easily and provide
you with another way to handle difficult problems beyond your skills because
you will have access to others’ competencies and opinions.

Hence, our opinion is that Kaggle can more indirectly help you in your career as
a data scientist and that it can do that in different ways. Of course, sometimes
Kaggle will help you directly being contacted as a job candidate based on your
competitions’ successes, but more often Kaggle will be helpful by providing you
with the intellectual and experience skills you need to succeed first as a
candidate then as a practitioner. In fact, after playing with data and models on
Kaggle for a while, you’ll have had the chance to see enough different datasets
problems and ways to deal with them under the pressure of time, that when faced
with similar problems in real settings you’ll get quite skilled in finding solutions
quickly and effectively.

Actually, this latter opportunity of a skill upgrade is why we got motivated

writing this book in the first place and what this book is actually about. In fact,
you won’t exactly find here a guide just about how to win or score high on
Kaggle competitions (there are also online resources that can enlighten you on
that, actually) but you’ll absolutely will find a guide about how to compete better
on Kaggle and how to get back the maximum from your competition experience.

Use Kaggle and other competition platforms in a smart way. Kaggle is not a
passepartout, being first on a competition won't assure you a highly paid job or
glory beyond the Kaggle community. However, consistently participating in
competitions is instead a card to be played smartly to show interest and passion
in your data science job search and to improve some specific skills that can
differentiate you as a data scientist and not make you obsolete in front of
autoML solutions.

If you are going to follow us along this book, we will show you how.
Organizing Data with Datasets
In his story “The Adventure of the Copper Breeches”, Arthur Conan Doyle has
Sherlock Holmes shout “Data! Data! Data! I cannot make bricks without
clay”—and this mindset, which served the most famous detective in literature so
well, should be adopted by every data scientist. For that reason, we begin the
more technical part of this book with a chapter dedicated to data: specifically, in
the Kaggle context, leveraging the power of Kaggle Datasets functionality for
our purposes.

Setting up a dataset
In principle, any data you can use (subject to limitations—see the legal caveats
section below), you can upload to Kaggle. The specific limits at the time of
writing this book are: 20 gigabytes per dataset and 100 gb total quota. Keep in
mind that the size limit per single dataset is calculated uncompressed—
uploading compressed versions speeds up the transfer but does not help against
the limits. You can check the most recent documentation of the datasets at this
link:

https://www.kaggle.com/docs/datasets

Kaggle promotes itself as a “home of open data science” and the impressive
collection of datasets available from the site certainly lends some credence to
that claim: before uploading the data for your project into a dataset, make sure to
check the existing content—for several popular applications, there is a chance it
has already been stored there:
For the sake of this introduction, let us assume the kind of data you will be using
in your project is not already there—so you need to create a new one. When you
head to the menu with three lines on the left-hand side and click on Data you
will be redirected to the dataset page:

When you click on New Dataset you will be prompted for the basics: uploading
the actual data and giving it a title:
Keep in mind that Kaggle is a popular platform, so numerous people upload their
data there—including private (not publicly visible) ones—so try to think of a
non-generic title.

Voila! Your first dataset is ready. You can then head to the Data tab:

In principle you do not have to fill out all the fields—your newly created dataset
is perfectly usable without them (and if it is a private one, you probably do not
care—after all you know what is in it). However, the community etiquette would
suggest filling the info for the ones you make public: the more you specify, the
more usable the data will be to others (and measured by the usability score,
displayed in the upper right corner).

Gathering the data

Apart from legal aspect (see the last section of this chapter), there is no real limit
on the kind of content you can store in the datasets: tabular data, images, text—if
you fit within the size requirements, you can store it. This includes data
harvested from other sources: tweets by hashtag or topic are among the popular
datasets at the time of writing:

Discussion of the different frameworks for harvesting data from social media
(Twitter, Reddit etc) is outside the scope of this book.

Using the Kaggle datasets outside of Kaggle

Kaggle kernels are free to use, but not without limits (more on that in Chapter 4)
—and the first one you are likely to hit is the time limit of 8 hours. A popular
alternative is to move to Google Colab a free Jupyter notebook environment that
runs entirely in the cloud:
https://colab.research.google.com

But even once we move the computations there, we might still want to have
access to the Kaggle datasets—so importing them into Colab is a rather handy
feature.

The first thing we do—since you are reading this, we assume you already are
registered on Kaggle—is head to the account page to generate the API token:

Go to “your account” and click on Create New API Token

A file named kaggle.json containing your username and token will be
created

Next step is to create a folder named “ Kaggle ” in your drive and upload the
.json there

Once done, you need to create a new Colab notebook and mount your drive:
Get the authorization code from the URL prompt and provide in the empty box,
then execute the following code to prove the path to the .json config:

We can download the dataset now: begin by going to Kaggle and copying the
API command:

Run the code:

The dataset will be downloaded as a .zip archive—unpack it and you are good
to go.

Building around datasets

Once you have created a dataset, you probably want to use in your analysis. You
can start a kernel using your dataset as a primary source: go to the Activity tab
in the upper menu of your dataset page and scroll to this block:

Alternatively, you can start a conversation around the data by clicking on Create
a discussion.

Legal caveats
Just because you can put some data on Kaggle does not necessarily mean that
you should—excellent example would be the “People of Tinder dataset”: in
2017, a developer used the Tinder API to scrape the website for semi-private
profiles and uploaded the data on Kaggle. After the issue became known, Kaggle
ended up taking the dataset down. You can read the full story here:

https://www.forbes.com/sites/janetwburns/2017/05/02/tinder-profiles-have-been-
looted-again-this-time-for-teaching-ai-to-genderize-faces/?sh=1afb86b25454

In general, before you upload anything to Kaggle ask yourself two questions: is
it legal (from a copyright standpoint—always check the licenses) and are there
any risks associated with this dataset (privacy or otherwise).
Working and Learning with Kaggle
Notebooks
Kaggle notebooks—which until recently were called kernels, so please excuse
me if I occasionally use those terms interchangeably—are Jupyter notebooks in
the browser that can run free of charge. This means you can execute your
experiments from any device with an internet connection, although something
bigger than a mobile phone is probably a good idea. The technical specification
of the environment (as of the time of this writing) is given below:

Without further ado, let us jump into it. The first thing we do is figure out how to
set up a notebook.

Setting up a kernel
There are two primary methods of creating a notebook: from the front page or
from a dataset level.
To proceed with the first method, go to the Code section of the menu on the left-
hand side of the landing page at https://www.kaggle.com/ and press the New
Notebook button. This is a preferred method if you are planning an experiment
involving uploading your own dataset:

Alternatively, you can go to the page of the dataset you are interested in and
click the New Notebook button there:

Whichever method you chose, after clicking New Notebook you will be taken to
your notebook page:

By default, a new notebook is initialized language set to Python—if you want to

use R instead, click on the Language dropdown on the right-hand side and your
notebook will switch to R:

An important aspect of using notebooks: you can always take an existing one
(created by somebody) and clone it to modify and adjust to your needs. This can
be achieved by pressing the Copy and Edit button on the kernel page, although
in Kaggle parlance, the process is referred to as forking:
A note on etiquette: if you have participated in a Kaggle competition, you
probably noticed that the leaderboard is flooded by forks of forks of well scoring
notebooks. Nothing wrong with building on somebody else’s work—but if you
do, remember to upvote the original author.

A notebook you create is private (i.e. only visible to you) by default. If you want
to make it available to others, you can select

Upgrade to GCP
Sometimes the resources provided freely by Kaggle are not sufficient for the task
you need, and you need to move to a beefier machine. You can setup the whole
environment yourself—or you can stay within the framework of notebooks but
swap the underlying machine. This is there Google Cloud AI Notebooks come
in.

In order to migrate your notebook to the GCP environment, go to the sideline

menu on the left-hand side and click on Upgrade to Google Cloud AI
Notebooks:
You will be greeted by the prompt:
After that, you will be redirected to the Google Cloud Platform console, where
you need to configure your billing options—unlike Kaggle, GCP is not free. If it
is your first time, you will need to complete a tutorial guiding you through the
necessary steps:

One step beyond

As mentioned earlier in this chapter, Kaggle notebooks are a fantastic tool for
education and participating in competitions—but they also serve another
extremely useful purpose, namely a component of a portfolio you can use to
demonstrate your data science skills.

There are many potential criteria to consider when building your data science
portfolio (branding, audience reach, enabling a pitch to your potential employer
etc.) but none of them matter if nobody can find them. Because Kaggle is part of
Google, the notebooks are indexed by the most popular search engine in the
world—so if someone is looking for a topic related to your code, it will show up
in their search results.

Below I show a “personal” example: a few years ago, I wrote a notebook for a
competition—the problem I wanted to tackle was adversarial validation (for
those unfamiliar with the topic: a fairly easy way to see if your training and test
sets have a similar distribution is to build a binary classifier trained to tell them
apart). When writing this chapter, I tried to give it a try and lo and behold, it
shows up high in the search results (notice the fact that I did not mention Kaggle
or any personal details like name in my query):

Moving on to other benefits of using notebooks to demonstrate your skillset: just

like competitions, datasets and discussions, notebooks can be awarded
votes/medals and thus position you in the progression system and ranking. You
can stay away from the competitions track and become an
expert/master/grandmaster purely by focusing on high quality code the
community appreciates. The most up-to-date version of the progression
requirements can be found at https://www.kaggle.com/progression, below we
give a snapshot relevant to notebooks:

Your Kaggle profile comes with followers/following option and gives you a
possibility to link other professional networks like LinkedIn or GitHub, so you
can leverage the connection gained inside the community:
In this day and age, it is easy to be skeptical about claims of “community
building”—but in the case of Kaggle, it happens to actually be true. Their brand
recognition in the data science universe is second to none, both among
practitioners and among recruiters who actually do their homework. In practice,
this means that a (decent enough) Kaggle profile can get you through the door
already—which, as we all know, is frequently the hardest step.

Kaggle courses
A great many things about Kaggle are about acquiring knowledge be it the things
you learn in a competition, datasets you manage to find in the ever-growing
repository or demonstration of a hitherto unknown model class, there is always
something new to find out. The newest addition to that collection are the courses
gathered under the Kaggle Learn label: https://www.kaggle.com/learn. Those are
micro-courses marketed by Kaggle as “the single fastest way to gain the skills
you’ll need to do independent data science projects”, the core unifying theme
being a crash course introduction across a variety of topics. Each course is
divided into small chapters, followed by coding practice questions.
Below, we provide a brief summary of their content:

Python: https://www.kaggle.com/learn/python You will learn the basics of

functions, Boolean variables, loops, lists and dictionaries.
Intro to ML / Intermediate ML: https://www.kaggle.com/learn/intro-to-
machine-learning Those two courses are best viewed as a two-parter: the
first one introduces different classes of models used in machine learning,
followed by discussion of topics common to different models like
under/overfitting or model validation. The second one goes deeper into
feature engineering, dealing with missing values and handling categorical
variables.
Pandas: https://www.kaggle.com/learn/pandas: this course provides a crash
introduction to one of the most fundamental tools used in modern data
science. You first learn how to create / read / write data, moving on to data
cleaning (indexing, selecting, combining, grouping etc).
Data visualization: https://www.kaggle.com/learn/data-visualization
Everybody knows a picture can be worth a thousand words – if you want to
learn how to create such images summarizing your data science results, this
course is for you. You will walk away know how to handle everything from
line charts to heatmaps and scatterplots.
Feature engineering: https://www.kaggle.com/learn/feature-engineering
This short course demonstrates the basic ideas around encoding categorical
data, general feature generation and selection.
Data cleaning: https://www.kaggle.com/learn/data-cleaning Another short
course, which helps address one of the most glaring omissions in the
academic curriculum: making students realize how messy real-life data is.
Intro to SQL / Advanced SQL https://www.kaggle.com/learn/intro-to-sql
In this tandem of courses, you will learn to extract data using SQL. Starting
with basic SELECT variations, you will go through GROUP BY, HAVING,
all the way to JOINs/UNIONs and explore analytic functions and nested
data.
Geospatial Analysis: https://www.kaggle.com/learn/geospatial-analysis
This course will teach you to create your first map using GeoPandas and
introduce ways to create interactive and choropleth maps. Basics of
proximity analysis are introduced as well.
Intro to Deep Learning https://www.kaggle.com/learn/intro-to-deep-
learning This course offers a crash introduction into what is arguably the
most important methodology in modern deep learning. Using structured
data, you will familiarize yourself with fundamental concepts like gradient
 descent, batch normalization and apply this knowledge to vintage problem
of binary classification.
Computer Vision https://www.kaggle.com/learn/computer-vision /NLP
https://www.kaggle.com/learn/natural-language-processing are two crash
courses introducing the most important domains where deep learning has
been successfully applied, producing impressive state-of-the-art results.
Crucial topics of transfer learning and data augmentation are introduced,
giving you the tools to hit the ground running.
Game AI https://www.kaggle.com/learn/intro-to-game-ai-and-
reinforcement-learning This course is a great wrap-up of the tech-focused
part of the curriculum introduced by Kaggle in the learning modules. You
will write a game-playing agent, tinker with its performance and use the
minimax algorithm.
Machine Learning Explainability https://www.kaggle.com/learn/machine-
learning-explainability Building models is fun, but in the real world not
everybody is a data scientist, so you might find yourself in a position when
you need to explain what you have done to others. This is where the mini
course on model explainability comes in: you will learn to assess how
relevant your features are with three different methods: permutation
importance, SHAP and partial dependence plots.
AI Ethics https://www.kaggle.com/learn/intro-to-ai-ethics This last course
is a very interesting addition to the proposition: it discusses the practical
tools to guide the moral design of AI systems. You will learn how to
identify the bias in AI models, examine the concept of AI fairness and find
out how to increase transparency by communicating ML models
information.

Apart from the original content created by Kaggle, there are multiple other
learning opportunities available on the platform using kernels. A prominent
example worth mentioning is the extremely popular fast.ai course:
https://www.kaggle.com/general/63077

In this chapter, we have discussed Kaggle kernels: a multi-purpose, open coding

environment, which can be used for education, experimentation as well
promoting your data science project portfolio.