Building Automated REST APIs with Python

Jeff Knupp
@jeffknupp
jeff@jeffknupp.com
Wharton Web Conference 2014

Author of “Writing Idiomatic Python”
Full-time Python developer @ AppNexus
Blogger at jeﬀknupp.com
Creator of the “sandman” Python library

We're going to use Python to generate a REST
API.
And we're going to do it without writing a single line of code.

We'll go over what a REST API is, how it works, and why it's
useful
We'll review the HTTP protocol and how the web works
We'll see a lot of Python code

Seven letters. Two acronyms. Buy what does it mean?

Programmatic way of interacting with a third-party system.

Way to interact with APIs over HTTP (the communication
protocol the Internet is built on).
"REST" was coined by Roy Fielding in his 2000 doctoral
dissertation. Includes set of design principles and best
practices for designing systems meant to be "RESTful".

In RESTful systems, application state is manipulated by the
client interacting with hyperlinks. A root link (e.g.
) describes what actions can be
taken by listing resources and state as hyperlinks.
http://example.com/api/

HTTP is just a messaging protocol. Happens to be the one
the Internet is based on.
RESTful systems use this protocol to their advantage e.g.
caching resources

To understand how REST APIs work, we have to understand
how the web works.
Everything you see on the web is transferred to your
computer using HTTP.

What happens when we type http://www.jeffknupp.com into
our browser?
Let's trace the lifecycle of a browser's request.

A protocol called the Domain Name Service (DNS) is used to
ﬁnd the "real" (IP) address of jeffknupp.com.

GET
The browser sends a GET request to 192.168.1.1 for the
page at address / (the home or "root" page).

The (a program used to service HTTP requests to
a web site) receives the request, ﬁnds the associated
HTML ﬁle, and sends it as an HTTP Response.

If there are any images, videos, or scripts that the HTML
makes reference to, separate HTTP GET requests are made
for those as well.

Programs, like curl, can also issue HTTP requests

CURL
curl talks to the webserver, using a public API (via HTTP)

A REST API exposes your internal system to the outside
world

It's also a fantastic way to make a system available to other,
internal systems within an organization.

Examples of popular REST APIs:
Twitter
GitHub
Google (for almost all services)

If you're a SaaS provider, you are expected to have a REST
API for people to write programs to interact with your
service.

Four core concepts are fundamental to all REST services
(courtesy Wikipedia)

When using HTTP, this is done using a URI. Importantly, a
resource and are completely orthogonal.
The server doesn't return database results but rather the
JSON or XML or HTML representation of the resource.

When the server transmits the representation of the
resource to the client, it includes enough information for the
client to know how to modify or delete the resource.

Each representation returned by the server includes
information on how to process the message (e.g. using MIME
types

Clients are . They know nothing about how the service
is laid out to begin with. They discover what actions they can
take from the root link. Following a link gives further links,
deﬁning exactly what may be done from that resource.
Clients aren't assumed to know except what
the message contains and what the server already told them.

A REST API allows to send to
manipulate .
...So we need to write a server capable of accepting HTTP
requests, acting on them, and returning HTTP responses.

Yep. A RESTful API Service is just a web application and, as
such, is built using the same set of tools. We'll build ours
using Python, Flask, and SQLAlchemy

Earlier we said a REST API allows clients to manipulate
via HTTP.

Pretty much. If you're system is built using ORM models, your
resources are almost certainly going to be your models.

Web frameworks reduce the boilerplate required to create a
web application by providing:
of HTTP requests to handler functions or classes
Example: /foo => def process_foo()
of HTTP responses to inject dynamic data in
pre-deﬁned structure
Example: <h1>Hello {{ user_name }}</h1>

The more time you spend building REST APIs with web
frameworks, the more you'll notice the subtle (and at times,
glaring) impedance mismatch.
URLs as to processing functions; REST APIs treat
URLs as the address of a resource or collection
HTML templating, while REST APIs rarely. JSON-related
functionality feels bolted-on.

Imagine we're Twitter weeks after launch. Ashton Kutcher
seems to be able to use our service, but what about
?
That's right, we'll need to create an API. Being an internet
company, we'll build a REST API service. For now, we'll focus
on two resources:
user
tweet

All resources must be identiﬁed by a unique address at which
they can be reached, their URI. This requires each resource
contain a unique ID, usually a monotonically increasing
integer or UUID (like a primary key in a database table).
Our pattern for building URLs will be
/resource_name[/resource_id[/resource_attribute]]

Here we deﬁne our resources is a ﬁle called models.py:
class User(db.Model, SerializableModel):
__tablename__ = 'user'
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String)
class Tweet(db.Model, SerializableModel):
__tablename__ = 'tweet'
content = db.Column(db.String)
posted_at = db.Column(db.DateTime)
user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
user = db.relationship(User)

class SerializableModel(object):
"""A SQLAlchemy model mixin class that can serialize itself as JSON."""
def to_dict(self):
"""Return dict representation of class by iterating over database
columns."""
value = {}
for column in self.__table__.columns:
attribute = getattr(self, column.name)
if isinstance(attribute, datetime.datetime):
attribute = str(attribute)
value[column.name] = attribute
return value

Here's the code that handles retrieving a single tweet and
returning it as JSON:
from models import Tweet, User
@app.route('/tweets/<int:tweet_id>', methods=['GET'])
def get_tweet(tweet_id):
tweet = Tweet.query.get(tweet_id)
if tweet is None:
response = jsonify({'result': 'error'})
response.status_code = 404
return response
else:
return jsonify({'tweet': tweet.to_dict()})

Let's curl our new API (preloaded with a single tweet and
user):
$ curl localhost:5000/tweets/1
{
"tweet": {
"content": "This is awesome",
"id": 1,
"posted_at": "2014-07-05 12:00:00",
"user_id": 1
}
}

@app.route('/tweets/', methods=['POST'])
def create_tweet():
"""Create a new tweet object based on the JSON data sent
in the request."""
if not all(('content', 'posted_at', 'user_id' in request.json)):
response = jsonify({'result': 'ERROR'})
response.status_code = 400 # HTTP 400: BAD REQUEST
return response
else:
tweet = Tweet(
content=request.json['content'],
posted_at=datetime.datetime.strptime(
request.json['posted_at'], '%Y-%m-%d %H:%M:%S'),
user_id=request.json['user_id'])
db.session.add(tweet)
db.session.commit()
return jsonify(tweet.to_dict())

In REST APIs, a group of resources is called a . REST
APIs are heavily built on the notion of resources and
collections. In our case, the of tweets is a list of all
tweets in the system.
The tweet collection is accessed by the following URL
(according to our rules, described earlier): /tweets.

@app.route('/tweets', methods=['GET'])
def get_tweet_collection():
"""Return all tweets as JSON."""
all_tweets = []
for tweet in Tweet.query.all():
all_tweets.append({
'content': tweet.content,
'posted_at': tweet.posted_at,
'posted_by': tweet.user.username})

All the code thus far has been pretty much boilerplate. Every
REST API you write in Flask (modulo business logic) will look
identical. How can we use that to our advantage?

We have self-driving cars and delivery drones, why can't we
build REST APIs automatically?

This allows one to work at a higher level of abstraction. Solve
the problem once in a general way and let code generation
solve each individual instance of the problem.
Part of

SANDBOY
Third party Flask extension written by the dashing Jeﬀ
Knupp. Deﬁne your models. Hit a button. BAM! RESTful API
service that .
(The name will make more sense in a few minutes)

Generalizes REST resource handling into notion of a
(e.g. the "Tweet Service" handles all tweet-related actions).
class Service(MethodView):
"""Base class for all resources."""
__model__ = None
__db__ = None
def get(self, resource_id=None):
"""Return response to HTTP GET request."""
if resource_id is None:
return self._all_resources()
else:
resource = self._resource(resource_id)
if not resource:
raise NotFoundException
return jsonify(resource.to_dict())

def _all_resources(self):
"""Return all resources of this type as a JSON list."""
if not 'page' in request.args:
resources = self.__db__.session.query(self.__model__).all()
else:
resources = self.__model__.query.paginate(
int(request.args['page'])).items
return jsonify(
{'resources': [resource.to_dict() for resource in resources]})

Here's how POST works. Notice the verify_fields decorator
and use of **request.json magic...
@verify_fields
def post(self):
"""Return response to HTTP POST request."""
resource = self.__model__.query.filter_by(
**request.json).first()
if resource:
return self._no_content_response()
instance = self.__model__(**request.json)
self.__db__.session.add(instance)
self.__db__.session.commit()
return self._created_response(instance.to_dict())

We have our models deﬁned. How do we take advantage of
the generic Service class and create services from our
models?
def register(self, cls_list):
"""Register a class to be given a REST API."""
for cls in cls_list:
serializable_model = type(cls.__name__ + 'Serializable',
(cls, SerializableModel), {})
new_endpoint = type(cls.__name__ + 'Endpoint',
(Service,), {'__model__': serializable_model, '__db__': self.db})
view_func = new_endpoint.as_view(
new_endpoint.__model__.__tablename__)
self.blueprint.add_url_rule(
'/' + new_endpoint.__model__.__tablename__,
view_func=view_func)
self.blueprint.add_url_rule(
'/{resource}/<resource_id>'.format(
resource=new_endpoint.__model__.__tablename__),
view_func=view_func, methods=[

'GET','PUT','DELETE','PATCH','OPTIONS'])
TYPE
In Python, type with one argument returns a variable's type.
With three arguments,
.

TYPE
serializable_model = type(
cls.__name__ + 'Serializable',
(cls, SerializableModel),
{})
new_endpoint = type(cls.__name__ + 'Endpoint', (Service,),
{'__model__': serializable_model,
'__db__': self.db})

Let's play pretend again. Now we're a IaaS company that lets
users build private clouds. We'll focus on two resources:
cloud and machine

class Cloud(db.Model):
__tablename__ = 'cloud'
name = db.Column(db.String, nullable=False)
description = db.Column(db.String, nullable=False)
class Machine(db.Model):
__tablename__ = 'machine'
hostname = db.Column(db.String)
operating_system = db.Column(db.String)
description = db.Column(db.String)
cloud_id = db.Column(db.Integer, db.ForeignKey('cloud.id'))
cloud = db.relationship('Cloud')
is_running = db.Column(db.Boolean, default=False)

from flask import Flask
from flask.ext.sandboy import Sandboy
from models import Machine, Cloud, db
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///db.sqlite3'
db.init_app(app)
with app.app_context():
db.create_all()
sandboy = Sandboy(app, db, [Machine, Cloud])
app.run(debug=True)

Building Automated REST APIs with Python

In cases where we're building a REST API from scratch, this is
pretty easy. But what if:
We have an existing database
We want to create a RESTful API for it
It has 200 tables

Only downside of Flask-Sandboy is you have to deﬁne your
model classes explicitly. If you have a lot of models, this
would be tedious.
...I don't do tedious

We have private companies building rocket ships and electric
cars. Why can't we have a tool that you point at an existing
database and hit a button, then, BLAM! RESTful API service.

SANDMAN
, a library by teen heartthrob Jeﬀ Knupp, creates a
RESTful API service for with
.

Here's how you run sandman against a mysql database:
$ sandmanctl mysql+mysqlconnector://localhost/Chinook
* Running on http://0.0.0.0:8080/
* Restarting with reloader

$ curl -v localhost:8080/artists?Name=AC/DC
HTTP/1.0 200 OK
Content-Type: application/json
Date: Sun, 06 Jul 2014 15:55:21 GMT
ETag: "cea5dfbb05362bd56c14d0701cedb5a7"
Link: </artists/1>; rel="self"
{
"ArtistId": 1,
"Name": "AC/DC",
"links": [
{
"rel": "self",
"uri": "/artists/1"
}
],
"self": "/artists/1"
}

ETag set correctly, allowing for caching responses
Link Header set to let clients discover links to other
resources
Search enabled by sending in an attribute name and value
Wildcard searching supported

We can curl / and get a list of all available services and their
URLs. We can hit /<resource>/meta to get meta-info about the
service.
Example (the "artist" service):
$ curl -v localhost:8080/artists/meta
HTTP/1.0 200 OK
Content-Length: 80
Content-Type: application/json
Date: Sun, 06 Jul 2014 16:04:25 GMT
ETag: "872ea9f2c6635aa3775dc45aa6bc4975"
Server: Werkzeug/0.9.6 Python/2.7.6
{
"Artist": {
"ArtistId": "integer(11)",
"Name": "varchar(120)"
}
}

And now for a (probably broken) live-demo!

"Real" REST APIs enable clients to use the API using only the
information returned from HTTP requests. sandman tries to
be as "RESTful" as possible without requiring any code from
the user.

Would be nice to be able to visualize your data in addition to
interacting with it via REST API.

1. Code generation
2. Database introspection
3. Lots of magic

sandman came ﬁrst. Has been number one Python project
on GitHub multiple times and is downloaded 25,000 times a
month. Flask-Sandboy is sandman's little brother...

The fact that the end result is a REST API is not especially
interesting
More important are the concepts underpinning sandman
and Flask-Sandboy

Work at higher level of abstraction
Solve a problem once in a generic manner
Reduces errors, improves performance
In general:
Speaking of automation, here's how my book is "built"...

sandman = Flask + SQLAlchemy + Lots of Glue
Requires you know the capabilities of your tools
Part of the UNIX Philosophy

The best programming advice I ever got was to "be lazy"
Sandman exists because I was too lazy to write boilerplate
ORM code for an existing database
Flask-Sandboy exists because I was too lazy to write the
same API services over and over
Being lazy forces you to learn your tools and make heavy
use of them

Contact me at:
jeff@jeffknupp.com
@jeffknupp on Twitter
on the tubeshttp://www.jeﬀknupp.com

Building Automated REST APIs with Python

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to Building Automated REST APIs with Python (20)

Recently uploaded (20)

Building Automated REST APIs with Python