A Qualitative Study of REST API Design and

Specification Practices
Michael Coblenz∗ , Wentao Guo† , Kamatchi Voozhian‡ , Jeffrey S. Foster§
∗ Computer Science & Engineering Department, University of California, San Diego, La Jolla, California, USA
† Department of Computer Science, University of Maryland, College Park, Maryland, USA
‡ Nokia, Inc., Sunnyvale, California, USA (work conducted at the University of Maryland)
§ Department of Computer Science, Tufts University, Medford, MA USA

[email protected], [email protected], [email protected], [email protected]

Abstract—REST APIs expose web services to clients. Although HTTP standard [18]. Moreover, REST APIs are language-
experts have recommended guidelines for REST API design, independent, with both clients and servers written in multiple
there is little empirical evidence regarding the relationship languages such as JavaScript, TypeScript, Python, Ruby, and
between adherence to guidelines and benefits to API consumers.
We interviewed ten REST API designers to understand what Go. Finally, the client/server distinction has significant impli-
practices REST API designers follow, what opinions they have of cations for modifiability, evolvability, and performance, since
existing guidelines, and what challenges they face when designing the organization running the server must pay for hosting costs.
and using APIs. Some guidelines were widely followed, and As a result, it is difficult to predict what kinds of tools would
some were rejected as being inconsistent with good practice. be most useful for REST API developers, and it is difficult for
The participants reported additional challenges: authentication
and authorization are too hard to implement; specifications and designers of REST APIs to choose practices that result in the
documentation are frequently missing, vague, or outdated; and most effective APIs to enable programmers to build reliable
standard error reporting methods are insufficient for end users client applications.
and for debugging purposes. Overall, our observations present In this paper, we address this gap by conducting an interview
opportunities for tool developers to significantly improve the situ-
ation by assessing and assuring conformance with guidelines; by
study that answers two research questions:
aligning documentation with specifications and implementations; RQ1: What opinions and practices do REST API designers
and by standardizing guidelines for API design. and implementors have regarding existing REST API
Index Terms—REST APIs, Web APIs, API design, empirical
studies of programmers
design guidelines?
RQ2: What challenges do REST API designers face when
I. I NTRODUCTION trying to create reliable REST APIs?
Representational State Transfer (REST) is a widely used Answering these two questions enables us to identify oppor-
architectural style for web APIs. First proposed by Fielding tunities to build tools that help designers create better APIs.
more than twenty years ago [1], REST has grown and evolved It is also progress toward an empirical basis for REST API
significantly, and many authors have provided recommenda- design decisions. In the future, developers may be able to
tions on designing and building usable REST APIs [2], [3], [4], choose practices that have been found empirically to be helpful
[5], [6]. The design of REST aims in part to enable separate rather than using opinion-based advice.
evolution and maintenance of client and server components, We interviewed ten experienced REST API designers, im-
improving reliability of services relative to approaches that plementors, and users. We asked about their experience and
might require co-evolution of the two sides of the interface. challenges with following REST API design recommenda-
Unfortunately, poor API design can lead to security prob- tions, implementing and using secure APIs, testing and de-
lems [7] and can cause delays and result in poor user satis- bugging APIs, specifying APIs, and using tools for imple-
faction [8]. As shown by the proliferation of services to help menting APIs. We used open coding to categorize participants’
monitor API errors [9], [10], real-world API failures are a statements, and then we formed a set of themes by grouping
significant concern in industry. However, despite a plethora related codes. Our goal was not to recommend participants’
of design recommendations, there is relatively little empirical opinions for use but rather to leverage their opinions to identify
evidence regarding which recommendations benefit REST API opportunities for empirical evaluation and tool development.
users, and few tools to help ensure developers follow those (Section II describes our methodology.)
recommendations. We believe a key reason is that, while there Some guidelines, such as making APIs stateless and using
is broad knowledge of good API design in general [11], [12], JSON to return responses, were widely endorsed. Others, such
[13], [14], [15], there are few studies of good REST API as using hypertext as the engine of application state (HA-
design [16], [17]. Results from general API design studies TEOAS [1]), were rejected as not being worthwhile or being
may not provide sufficient guidance for REST API design, inconsistent with good practice. Still others, like following
since REST APIs inherit structural requirements from the HTTP verb conventions and making interfaces uniform, were
recognized as useful ideas that were difficult to follow in ticularly challenging or benefit from additional tool support
practice. (Section III-B discusses RQ1.) (results in Section III):
Additionally, participants identified a number of other chal- a) Security: We asked what security problems partici-
lenges in designing and using REST APIs: Even though a pants had seen in their and others’ APIs as well as what
variety of libraries and external services provide these features, techniques participants use to ensure their designs are secure.
authentication and authorization are still difficult to implement. We also asked about the security benefits of frameworks and
Missing, vague, and outdated API documentation is a critical languages, such as Python.
challenge in using REST APIs. HTTP error codes are insuf- b) Bugs: We asked if there were design choices that were
ficient for conveying information about errors to clients and more likely to lead to bugs or vulnerabilities in client code.
end users. And forward and backward compatibility of REST c) Testing and Debugging: We asked about techniques
APIs are essential for API evolution but are difficult to get to make APIs more debuggable and about techniques used for
right. (Sections III-C through III-H discuss RQ2.) testing. We also discussed the requirements that participants
Our findings suggest an opportunity to develop tools to focused on in testing, such as performance and scalability.
make REST APIs easier to develop and use. Participants d) Specifications: We discussed specification approaches
would benefit from tools that check REST API specifications, that participants use, such as OpenAPI; the alignment between
which formally declare properties of API inputs and outputs, specification techniques and participants’ needs; and which
against client and server code. The community could develop aspects participants did not document but wished they had.
a standard approach for relaying more detailed error messages. e) Tools: We asked for opinions about the frameworks,
And, given machine-checkable REST API specifications, tools languages, and other tools participants use when implementing
may be able to identify potentially incompatible changes as APIs and how those tools influenced their designs.
APIs evolve as well as automatically generate examples as f) Errors: We asked how API implementations report
documentation. (Section IV describes proposed tools.) errors and how applications handle errors.
In summary, this paper’s main contributions are g) Data interchange: We asked what data formats par-
1) An analysis of REST API guidelines and practices, as ticipants’ APIs transmit and why.
currently followed by designers and implementors, and
2) Proposals about how languages and tools could make After each interview, the first and third authors discussed
application and REST API development easier and safer. and summarized the findings. We conducted interviews until
Our work is the first of which we are aware to compare and the newest interviews revealed only minimal additional in-
contrast REST API design practice with REST API design formation. The third author, who had extensive experience
guidelines. Although our focus is on REST, REST APIs implementing REST APIs, transcribed the recordings and
share similarities with non-REST APIs, especially those in conducted an initial open coding on the transcripts. Then,
distributed systems, so addressing challenges with REST APIs the authors worked together to review and revise the codes
may also benefit other kinds of APIs. and then identify themes among the codes. Our focus is on
identifying hypotheses, not on making statistical claims, so we
II. M ETHOD did not compute inter-rater reliability [20].
Because of the difficulty of recruiting experts, we recruited a Limitations: Because of the multitude of recommenda-
convenience sample of ten programmers who each had at least tions for REST APIs, we focused on guidelines and challenges
a year of experience developing REST APIs. When recruiting we felt could inform practice and the development of tools.
our participants, we first asked them to complete introductory There is a risk of bias due to participant selection. Al-
surveys. We obtained informed consent and conducted semi- though the practical difficulties of recruiting experienced API
structured interviews via Zoom that lasted up to 90 minutes. designers limited our recruitment to a convenience sample, we
Our study was approved in advance by our IRBs. Our partic- recruited broadly from people in multiple organizations, and
ipants were willing to participate as volunteers. who worked on differently-sized projects, although medium-
We first asked general questions such as “What were the size organizations may be under-represented in our participant
most challenging parts in learning how to develop an API?” pool. There is also a risk of misinterpretation when analyzing
Then, we asked them to focus on a particular REST API they interview data. To mitigate this risk, multiple authors with
had worked on recently. We asked about several REST API different perspectives read and discussed the data, coding
guidelines (RQ1; Section III-B). We used prior work [19], scheme, and resulting themes.
[2], [3], [4] to construct a list of guidelines to ask about. Due to an error, P6’s recording is missing. We instead
We focused on those (Table II) we thought had significant analyzed notes taken during the interview.
implications for API design practice or offered opportunities
for tool development. In the interview, we summarized each III. R ESULTS
guideline and asked for opinions. We also asked participants After describing the participants (Section III-A), we present
how they would improve the guidelines. the results of our analyses of the interview data. Section III-B
Afterward, we asked about various aspects of API design compares and contrasts existing REST API design guidelines
that we hypothesized, based on the literature, might be par- with the participants’ practices.
 TABLE I
D EMOGRAPHICS OF INTERVIEW PARTICIPANTS . ”L ARGE ” MEANS AT LEAST 1000 EMPLOYEES .

ID Education Org type Roles Design exp. Spec. tools Impl. APIs in Invoked APIs in
P1 BS (unspecified Large Software eng. ≥ 3 years OpenAPI, Java Java
field) consultancy RAML
P2 MS communi- Large IT solution analyst ≥ 3 years OpenAPI, Java Java
cation eng. consultancy RAML
P3 MS (not CS) Govt. consulting Personal projects, Software eng. ≥ 3 years None PHP, Python, Go PHP, Python, Go
P4 MS CS Govt. consulting Design, impl., docs. integration ≥ 3 years OpenAPI, Ruby, Python, Ruby, Python,
Stoplight.io Go, JS Go
P5 CS PhD student University Researcher ≥ 3 years OpenAPI JS JS
P6 BS CS Large company Software eng. ≥ 3 years None Java, JS Java, JS
P7 Sc.B. CS University Organization service projects ≥ 3 years None Python, JS Python, Go
P8 BS CS Large company Developer, project manager 1–3 years OpenAPI Python Python, Java, C#
P9 CS PhD Startup Software eng. 1–3 years OpenAPI Go Go, Python, Bash
P10 BS Info & De- Large company Senior software eng. 1–3 years OpenAPI Python Python, C++,
cision Sci. Objective-C

Sections III-C through III-H discuss the challenges partic- We also asked participants about their experience using
ipants identified and participants’ recommendations for ad- REST APIs. Six reported three of more years of experience,
dressing those challenges. Some of those challenges lead three reported one to three years, and one did not respond.
directly to bugs, and others divert developer attention and add They reported experience using REST APIs in Python (6),
unnecessary complexity. Participants described the difficulty of Go (4), Java (3), Bash (1), C# (1), C++ (1), JavaScript (1),
using third-party authentication and authorization frameworks Objective-C (1), and PHP (1).
and security testing. They also cited insufficient documentation
B. API Guidelines
of input requirements as a key source of bugs. In spite of
research proposing automated testing methods, participants RQ1 asks: what opinions and practices do REST API
reported using manual testing methods—finding automated designers and implementors have regarding existing REST API
approaches, such as property-based testing—too complex. design guidelines? We addressed this question by aggregat-
Missing, vague, or outdated documentation and specifications ing guidelines from several sources [1], [5], [6], prioritizing
are key challenges. Participants also told us that generating guidelines that seemed to represent opportunities for tool
and reporting errors is difficult, inconsistent, and bug-prone, development or which we believed had significant impact on
and that forward and backward compatibility remains a key API quality.
challenge despite using versioning techniques. Table II lists guidelines we discussed and summarizes our
The paper supplement includes a list of all the codes we findings about each one. Key findings include the inability of
used and the number of occurrences of each. developers to assess uniformity; the universality of stateless
design; and the ambiguity of HTTP verb semantics. Next, we
A. Participants describe our findings regarding the guidelines.

We recruited ten participants (N = 10). Table I shows key 1) Use a Uniform Interface: Uniformity, i.e., consistency,
characteristics of the participants. Four identified as women is commonly recommended for REST API design, and can
and six as men. Participants were located primarily in the US, enable users to predict how an unfamiliar part of an inter-
though one was in the UK and one was in India. All partici- face will behave based on experience with other parts [21].
pants had bachelor’s degrees, eight of which were in computer However, the open-ended, vague nature of the guideline leaves
science or related disciplines. Three participants had master’s our participants confused about what they could do to obtain
degrees, and one had a Ph.D. Two were Ph.D. students with or check uniformity. When asked how to ensure uniformity,
significant experience with REST API development, and eight P3 said there was a “part of me that is never satisfied with
were employed in the software industry. Participants worked anything I do” regarding naming and other patterns. P3 later
in contexts ranging from individual projects (P3, P5) to large said the hardest part of designing a REST API is structuring
corporations with many stakeholders (P1, P2). and naming the interface. P7 wanted a tool to test and
All participants had some API design or implementation quantify uniformity. P8 mentioned that organizing resources
experience. Six reported three or more years of experience, to be future-proof is challenging: “I feel like you always
three reported between one and three years, and one did not end up eventually wanting to. . . merge something or change
respond on the survey but confirmed experience verbally. The something.”
participants reported experience implementing REST APIs in Uniformity pertains within an API, but uniformity can
Python (5), Go (3), JavaScript (3), Java (2), and PHP (1). apply across APIs as well. However, this uniformity can
They had specified their APIs with tools including OpenAPI trade off with other design decisions. P5 and P3 favored
(7), RAML (2), and Stoplight.io (1). uniformity across APIs over simplicity. “Dailymotion and
 TABLE II request multiple times. P8 said statelessness was required,
S UMMARY OF REST API GUIDELINES AND CORRESPONDING FINDINGS as their hosting method involved Dockerizing and deploying
multiple instances of an API.
Guideline Source Summary of Findings However, P4 said while they “generally do want things to
(a) (b) (c) be stateless where possible,” there are cases where state is
useful: If there is “an accumulation of a complex state” as
Use a uniform in- • • Designers cannot assess unifor-
terface mity or choose the most appro-
the user is “interactively working through something, building
priate uniform approach something”—e.g., a shopping cart for a web store—then some
Use • • Well-defined formats (e.g. state might be stored on both the client and the server. Another
HATEOAS [19] JSON) must be extended with exception is when the server must be authoritative, e.g., a
application-specific semantics; server should not simply trust a client’s authentication cookie.
standard formats, such as HTML,
do not suffice.
Statelessness • • • Good for scalability; can be 4) APIs Can Provide Code on Demand to Clients: Field-
tricky in complex applications ing [1] proposed that APIs can send code to clients, but this
Provide code on • Avoid due to security risks practice was derided by P3, P7, and P10. As P10 put it, “That’s
demand to clients just a huge security hole waiting to happen.”
Use HTTP meth- • • Verb conventions are followed
ods and follow inconsistently; need community
verb conventions standards and checking tools 5) Use HTTP Methods and Follow Verb Conventions:
Filter, sort, and • • A standard is needed for pagi-
HTTP requests are tagged with a verb, such as PUT or POST,
paginate to limit nation parameter naming and se- which indicates some high-level semantics of the request.
response size mantics. However, participants reported that verb conventions are only
Use JSON to • JSON is conveniently human- followed inconsistently, despite their potential benefits.
transfer data readable, but code receiving P9 expressed support for verb conventions because they
JSON inputs must be flexible
reduce developer confusion. P1 said that not following verb
(a) Fielding [1] (b) Microsoft [5] (c) Stack Overflow [6] conventions is bad design and is hard to fix later because it
requires rewriting the API. As an example, P1 described an
API for fetching data that used a PUT method with a user ID
YouTube. . . have pretty much the same data, but they’re com- in the request body. This was less convenient than using GET
pletely different. I’d have to learn them from scratch—they use because the request body had to be handled with every call.
different type of authentication, different endpoints. . . but if In contrast, P2, P7, and P8 gave reasons for breaking
[there’s a choice between] the simple and having standardized verb conventions. P8 reported rarely using PUT and PATCH
API, I would go with the standardized one” (P5). P3 phrased because “myself and everyone I work with are just so used to
the issue in terms of opinions about design: “I think if you using GET and POST that. . . that’s on your fingertips.” P2 uses
provide appropriate tooling that enforces those opinions— POST to fetch data if the number of required query parameters
whatever you decide they are—it is possible to make dramatic exceeds the limit for GET. P7 uses POST for all actions
improvements. . . because it’s ultimately different assumptions that push things, including deletions, because of a particular
that lead to a lot of problems.” implementation choice they made. However, P7 also said that
if they had to redesign their API, they would probably use
2) Use Hypertext as the Engine of Application State (HA- conventional HTTP verbs.
TEOAS): According to Fielding, “A REST API should be P3 observed widespread deviance from HTTP verb conven-
entered with no prior knowledge [of the data format and tions, which they attributed to a lack of enforcement. P3 said,
API semantics] beyond the initial URI (bookmark) and set of as there are no consequences for breaking conventions, “it just
standardized media types. . . ” [19]. In contrast, our participants feels too loose to be a standard, because it’ll just accept any
universally described REST APIs as having documentation kind of behavior that you program it to accept.” P3 blamed
and specifications that describe data formats and API seman- this on the disorganization of the REST community, saying,
tics. Perhaps the large range of possible data types and seman- “it’s kind of ownerless.”
tics have precluded standardization, though JSON is widely P5 argued that REST APIs would be easier to use if they
adopted. This represents a compromise, since standardized were more object-oriented: a client could complete some
tools can parse JSON, even if the semantics are API-specific. action with one call instead of stringing together multiple
HTTP requests. P5 acknowledged that this might not be as
3) APIs Should Be Stateless: P3, P4, P7, and P8 try to efficient and proposed an intermediate API to address the
maximize statelessness. P3 said, “State in general is a pain performance concern.
in the [rear].” P7 described learning from Uber [22], which
transitioned a monolithic codebase to microservices, which 6) Filter, Sort, and Paginate Data to Limit Response Size:
often offer (stateless) REST APIs. P4 said that stateless APIs REST APIs must limit response sizes to maintain good perfor-
were less likely to break if a client accidentally made the same mance. Participants reported that pagination, in which multiple
API calls are used to return different pages of a lengthy forgot to document it or it wasn’t [an] expected constraint, [I]
result, is one area that might benefit from standardization. P5 sort of figured [it] out after the fact.”
said that, frustratingly, different APIs deal with pagination in Six comments extolled the benefits of examples. Four
different ways. P8 indicated that the names used in pagination emphasized the benefits of executable examples in particular.
often differ across APIs: “Some APIs use skip and take “Facebook [has] an example ready to go, and you just type
parameters, specifying where in the results to start and how in your API key. . . and you can just run that against your
many to give.. . . I have seen some applications that just use a own account. . . so you don’t have to try to look up examples
token, like a next page token or last page token.” on Stack Overflow. . . ” (P7). Examples and learning-by-doing
have also been found to be effective in other domains, such
7) Use JSON to Transfer Data: As discussed earlier, JSON as mathematics [24], and there is a substantial body of work
is a widely recommended data transfer language, and our showing that examples can lead to superior skill acquisition
participants indicated it is widely adopted. Several participants compared to studying only abstract knowledge [25].
had strong feelings about JSON compared to other alternatives. P2 explained the challenge of documenting fields appropri-
P3 said they “hate” being required to use XML, and P8 said ately for multiple client perspectives. “I created the API, which
XML is “a pain to work with.” P4 prefers JSON to XML was more looking into one particular line of business. . . but
because it is more compatible with their code. P3 also prefers then when I moved into the corporate world, their description
JSON to CSV, which they find harder to lint and more likely was completely different.”
to yield results that cannot be reliably parsed or interpreted. Fourteen comments reported that documentation is difficult
The human-readable nature of JSON makes it easier to debug. to keep up to date. A common theme was that the only way
Participants also reported problems with JSON. First, client to ensure documentation is up to date is to automatically
applications typically expect a consistent set of fields to be generate it from the source code. “There are people who are
populated, which leads to bugs if the server omits fields (P5). like ‘code should be simple and self documenting,’ and I’m
Code that accepts JSON must be written in a flexible way, like, ‘no, it should generate documentation’” (P10). P8 and
lest it break when the server sends an unexpected response. P10 use tools that extract comments from source code to
P10 said, “you kind of have to be willing to accept a missing generate documentation, but comments can be outdated. P10’s
value or something.” Second, this approach requires carefully group uses a tool to generate reminders every six months for
encoding data, typically as text. This can be particularly engineers to check documentation. P10 also remarked: “I will
inconvenient when both ends of the API expect Java objects certainly start from the documentation. But the minute my
(P2). Third, checking conformance to the specification requires observed behavior deviates, I’m calling the [author].”
additional tools, such as Swagger. Since incorrect or outdated documentation is a key chal-
Some participants use a specification approach such as lenge, tools that automatically generate documentation from
Protocol Buffers [23]. Like Swagger tools, this ensures the data implementations can be valuable. As P2 remarked, “the Swag-
conforms to a specification, but Protocol Buffers represents the ger documentation is the truth.” Seven participants reported
data more compactly (in binary rather than JSON). However, using specification tools for REST APIs: OpenAPI, Swagger,
protobuf data is inconvenient to debug, since it is not human- or RAML. Five had experience with Protocol Buffers [23],
readable. This approach is also inconsistent with Fielding’s which are used to describe binary interfaces in general. These
architectural goal of decoupling clients and servers. In some specification tools can have both significant costs and benefits.
cases, a protobuf layer was wrapped in a JSON layer to provide As P10 explained: “We said, ‘Hey, we’re going to use Swagger
convenient interfaces for both internal and external clients. to make this API easier to consume.’ And then my employees
came back to me and said, ‘Man, Swagger is a pain in the
C. Documentation and specification [rear].’ And I said, ‘is [it worth the hassle]? If we don’t use
Five participants volunteered on the survey that missing, Swagger, and I send every person that has a question about
vague, or outdated documentation is a key challenge when us- this API to your desk, how are you going to feel?’ And they
ing REST APIs, and we received 14 comments about outdated said, ‘well, okay, I’m gonna get Swagger working.’”
documentation in the interviews. P4 and P10 said that keeping Design processes varied. P7, P8, and P10 added endpoints
documentation up to date is a key challenge when maintaining one at a time as needed. P1 favored writing a specification
REST APIs. P3, P5 P7, and P9 recommended including ahead of time: “We should always go to our specification first,
examples as a particularly effective form of documentation. because that’s the entry point of our API development.”
We found support for documenting all aspects of an API
design: the request parameters, the response body, possible D. Errors
error messages, and the meaning of each HTTP verb for each Participants indicated that it can be unclear which error
endpoint. Participants also suggested documenting which APIs codes to use; that writing and reporting human-readable error
can return large volumes of data. P8 mentioned that docu- messages is too hard; and that it can be difficult to write correct
mentation often neglects some important constraints, such as error-handling code in API implementations.
disallowed parameter values or invocation ordering constraints. The HTTP standard [18] specifies a collection of status
P8 explained how these omissions arise: “either because I just codes, which the server sends in responses. For example, code
500 indicates internal server error. Unfortunately, status codes Themes included the challenges of handling authentication and
do not provide enough information for the client. As a result, authorization and unclear best practices for testing.
APIs typically send a natural-language error messages in an Although authors have discussed the challenges of handling
ad hoc format. Several participants (P3, P8, P9) proposed a authentication and authorization since at least 2016 [26],
standard JSON field for a human-readable error message. P2, the situation has not been resolved. Authentication typically
P4, and P9 use standard structures for all error messages. P9, requires using complex external services, such as OAuth [27].
for example, returns a JSON object with three fields: the HTTP According to P7, one tricky aspect is managing state and data
error code, an API-specific error code (whose meaning is across the different services that implement authentication.
explained in API documentation), and a textual error message. To avoid having to store password information, P8 uses
It is not obvious what information to include in error Amazon Cognito [28]. P10 also tried to use Cognito, but
messages. To aid developers of client applications, messages failed: “As somebody developing services. . . I never want to
should be precise and contain detailed information in terms store PII on my system. I want somebody else to store it and
of API abstractions. However, if the client application simply give me some surrogate ID that I can use to track the user
displays such messages to the end user, then a detailed error within my application.” But: “After a week I just gave up.
message may be confusing rather than helpful. Since error . . . There’s all these keys and certificates. . . ” P10 proposed that
messages are not typically localized, they may not even be someone should create a solution analogous to Let’s Encrypt:
readable by the end user. P3 observed that too-detailed error “Somebody needs to do that for app-level authentication and
messages can leak sensitive information about a server. authorization, because it’s never not been a nightmare.”
Another challenge with textual error messages occurs in For authorization, P9 uses Open Policy Agent [29] for
APIs that are implemented in terms of other APIs. If API A defining access control policies. OPA provides a language,
receives an error message from API B, it is not clear how to REGO, to express access control policies. P9 tries to align
use any textual component of the error message to generate the resource paths with the authorization requirements so the
an appropriate error message for A’s clients. system can easily reject inappropriate requests. That is, the
When generating HTTP status codes, it can be unclear software engineering considerations for implementing autho-
whether to return 200 (“OK”) or 500 (“internal server error”). rization impact the API design itself.
P6 said that if 200 is returned when even a minor error For testing security, participants used fuzzing (P3, P6);
happens, the user can get the wrong impression as to the testing by security experts (P3); review by a security team
outcome. In addition, returning 200 for minor errors can cause (P2); and public bug bounties (P3).
tools that track API success rates to fail to raise necessary
alarms. On the other hand, for debugging purposes P3 found it G. Causes of bugs
more convenient to send 200 with an error message in cases of
invalid input. We did not find other participants who seemed We asked participants whether there are particular API de-
to agree, although P7 writes client code that interprets both sign or implementation choices that lead to bugs. Participants
200 with an error message and 500 as errors. reported that insufficient documentation of input requirements
Implementing error handling can be challenging: when leads to bugs. “[The] author thought their API was well-
failures occur, the implementation may need to revert state defined, but you try to use it, and . . . this is an ID. What is it an
back to before the failure. P7 recommended: “Only ever ID of?” (P6). A particularly insidious cause of bugs is when
commit state at one point in your program, or do it in a way misformatted input is not detected, e.g., when an incorrectly
that rollback is cheap.” formatted date is sent (P6). Likewise, API implementations
frequently fail to validate their input sufficiently, causing
E. Forward and backward compatibility unsafe or incorrect behavior on user inputs (P1, P6).
In the pre-interview survey, four participants reported back- Implementation languages and frameworks also introduce
ward compatibility as a primary concern. Typically, main- their own bug patterns. Dynamically typed languages, such
tainers either do not control all API clients or, even if they as Python, can result in incorrect behavior. For example, if a
do, cannot kill and restart all clients at once. If the client user passes a string instead of a collection, iteration is over
is a web app, users may use an old version of a web app the characters in the string rather than the single string in
for a long time if they do not refresh their browser window the input (P3). As another example, in Go, if an invocation’s
(P6). Thus, backward-incompatible changes, such as renam- request context is omitted in a database call, that call cannot
ing fields, are problematic. Introducing incompatible changes be cancelled, which can lead to leaked threads (P9).
requires recognizing the incompatibility and introducing a
versioning scheme, but this recognition can be error-prone. H. Testing Methodology
P6 recommended anticipating future needs and building in
Despite existing research on automated test generation,
opportunities for extensions (forward compatibility).
participants (P1, P3, P4, P8) reported primarily using manually
F. Security written tests. Although some said testing performance is
We asked participants what security problems they had important, none of the participants reported writing dedicated
observed and what they do to make sure their APIs are secure. performance tests; instead, they use tools or re-purpose unit
tests for this purpose. P8 monitors response times once fea- parameters and of API results on server-side code. Such an
tures are deployed and redesigns if they are slow enough to analysis will need to be path-sensitive [38] to separate the
impact user experience. different request–response combinations.
P9 uses PyTest, while P3 writes Bash scripts that use tools Third, given how helpful our participants found examples,
such as curl. Tests both check correctness and help establish we propose tools to automatically generate examples from
rate limits. Participants also use unit testing and fuzzing to specifications. Such a tool would likely need to be dynamic
identify edge cases that cause bugs. P3 observed that testing since the examples would be concrete, and static analysis is
is inevitable because it is challenging to consider all possible less good at tracking concrete values. Interactive authoring via
edge cases when writing a specification. P9 indicated that tools such as CodeScoop [39] might also be possible.
integration tests are critical to check the whole workflow. P8 Automated testing tools could leverage stronger specifica-
uses low-cost tests in continuous integration but uses Docker tions. Schemathesis [32] enables property-based testing for
for more thorough integration tests before deployment. APIs that are specified with OpenAPI, but OpenAPI specifi-
Participants felt that the benefit of property-based test- cations do specify relationships between inputs and outputs.
ing [30] was not worth the cost. P3 said, “I feel like software RESTler [33] is a stateful REST API fuzzer, but it cannot
engineers tend to be curious and attracted to property testing, distinguish between correct and incorrect responses that do
and then they kind of look at it and go, I could spend X not return error codes. We found a need for an approach
amount of time trying to figure out how to do this, or I could that is simpler for users than property-based testing and more
just point a fuzzer at this, and I would get good enough of a thorough than RESTler.
result with less time. . . .” B. Error Handling
IV. D ISCUSSION The loose coupling between REST API servers and clients
Based on the challenges reported by participants, we be- makes propagating errors difficult, as described in Sec-
lieve there are several opportunities to improve REST API tion III-D. HTTP includes standard error codes, but as there
development and use. New language-based tools may be able are relatively few codes, they are generally not very specific.
to infer specifications from implementations, keep documen- This can make it difficult for a client or an end user to handle
tation consistent with implementations, and aid in automatic the error. We believe that a new approach is needed.
generation of test cases. Authentication and authorization We propose that the community develop a standard ap-
frameworks are still too hard to use and represent a research or proach for relaying detailed error information. This could take
design opportunity. Tools could leverage specifications to help the form of a specific JSON response field with an error code
developers identify some cases in which version numbers need whose meaning is documented in the API specification. Since
to change. Finally, although enforcing guidelines currently the documentation is separate, it could be localized to allow
relies on human analysis, new tools may be able to analyze speakers of a variety of languages to understand the meanings
adherence to guidelines automatically. of errors. The field could also include an optional structured
component to carry error-related values, as is common in many
A. API Documentation and Specification languages with exceptions. For example, if a request includes
We found that documentation is a key challenge with several fields of data, one of which is invalid, the error could
using REST APIs (Section III-C), including ensuring that describe which field is invalid. Tools for checking and inferring
documentation stays up to date. There are existing approaches documentation (Section IV-A) could be extended to document
to specifying APIs [31] and performing some limited checking under what conditions errors are emitted.
of APIs using specifications [32], [33], [34], but we believe C. Security
there are several unmet needs. Participants indicated that handling authentication and
First, new tools that use type checking could ensure clients authorization in REST APIs is an important challenge
and servers conform to API specifications. As with compiler (Section III-F). P10’s proposal to make an authentica-
designs, rather than building one tool per combination of client tion/authorization framework akin to Let’s Encrypt, which
and server languages, we propose building one tool for each would make appropriate recommendations for each developer,
client language and one for each server language. Although is a compelling opportunity. Let’s Authenticate [40] may meet
many clients are written in dynamic languages, recent years some of the needs but we are not aware of a usability
have seen widespread introduction of static typing capabilities evaluation of that system. Information flow-based approaches
for such languages, including TypeScript for JavaScript [35], could help developers identify when sensitive information
PyType [36], and Sorbet for Ruby [37]. Thus, we believe the could leak from an API [41]. Best practices for security testing
time is right for building static analysis tools for enforcing should be better defined, particularly for developers who lack
API specifications, leveraging these type systems. access to security experts.
Second, to help make documentation easier to create, new
tools should automatically generate specifications from imple- D. Versioning and Forward and Backward Compatibility
mentations. We believe this can be done using type inference Participants reported that versioning is important for API
with an expressive type system to discover the types of input evolution, as is ensuring forward and backward compatibility
(Section III-E). However, participants did not report using improved APIs and the efficacy of API designers. Our work
any tools (such as exist for Protocol Buffers) to help. We focuses on design decisions, not on processes.
envision that if an API had a machine-checkable specification Liu et al. [11] studied six months of high-severity bugs in
(Section IV-A), a tool could identify potentially incompatible Microsoft’s Azure cloud computing environment. The most
changes, such as changing API parameters, return values, common causes of bugs were incorrect or missing fault de-
or method names. It could then prompt for version number tection and handling; data races; and inconsistent assumptions
and documentation changes. While this approach would not about data formats. 60% of the data-format incidents were due
identify all incompatibilities, it could help API designers to interface problems, typically because of software updates
consider the implications of their changes on existing clients. that changed interfaces without considering existing clients.
OpenAPI [31] is the most prevalent approach for specify-
E. Checking Other Guidelines
ing REST API interfaces. Swagger [45] provides tools for
Machine learning and natural language processing tech- creating, updating, and sharing OpenAPI definitions. JSON
niques may be able to evaluate interface uniformity (Sec- Schema [46] allows annotating and validating JSON doc-
tion III-B1). Program analysis techniques may be able to warn uments against a schema. Sohan et al. [47] found in an
developers about API methods that are not stateless (Sec- experiment that usage examples improve users’ success rates
tion III-B3) and ensure that stateful endpoints are associated in using APIs relative to only providing specifications.
with reasonable HTTP verbs. McLellan et al. [12] showed how API usability studies
V. R ELATED W ORK revealed shortcomings in one API’s design and documentation
and demonstrated that some API users find examples partic-
In 2000, Fielding proposed Representational State Transfer ularly helpful. Robillard [13] found learnability benefits of
(REST) [1] as a set of architectural constraints for web-based examples. Myers and Stylos [14] observed that usability prob-
APIs. These included guidelines such as statelessness (state lems have resulted in bugs and security problems, proposing
relevant to a request should be sent with the request, not user-centered design and evaluation methods to help designers
stored on a server) and uniformity. Since 2000, developers make APIs more usable. Oliveira et al. [15] showed how latent
have adapted Fielding’s original REST guidelines for their security implications of API designs correlate negatively with
own use. The result is a loose set of guidelines and practices developers’ ability to identify security concerns in code.
that differ significantly from Fielding’s vision. Our focus is on Koci et al. [48] used log information to infer usability
identifying current practice and the challenges therein. challenges in APIs; we elicited advice from experts directly.
Murphy et al. [42] studied public REST API design
guidelines, constructing a list of topics authors can consider VI. C ONCLUSION AND F UTURE W ORK
but finding inconsistent specific guidance. Later, Murphy et We interviewed ten experienced REST API designers and
al. [17] interviewed 28 professionals regarding their training implementors to understand opinions and practices about
and design processes for APIs. While the interviews were not REST API design guidelines (RQ1) and to identify challenges
focused on REST API design, sixteen interviewees worked in developing high-quality REST APIs (RQ2). We identified
on REST APIs, and hence the study did include some results opportunities to improve the way REST APIs are designed and
related to our paper. They also found that it can be unclear implemented. New tools could provide client developers with
which fields to return for a given request, since one API may reliable documentation by making it easy for API designers
serve many different clients, and that excess data increases to create specifications for their APIs that are consistent with
costs. Designers found it particularly challenging to discern the API implementations. The tools may be able to automat-
which use cases were most important. Consistent with our ically generate appropriate examples and test cases from the
findings, Murphy et al. found that many organizations develop specifications, reducing the documentation and testing burden
their own API guidelines, but that, because these guidelines are on developers. Clients could rely on the accuracy of the
not well enforced, it is challenging to apply them consistently. specifications and examples, making development easier.
Kotstein and Bogner [43] asked experts to rate the impor- Existing REST API design guidelines are helpful but do
tance of 82 REST API design rules, finding that the experts not provide enough guidance regarding how to make APIs
gave high importance to 28 of the rules. We focus on guide- as usable as possible. Specific opportunities for community
lines that offered opportunities for tool development, since consensus include error handling, reporting, and propagation;
tools could make creating usable APIs easier. Our approach pagination and identification of APIs that can return very large
was also broader, focusing not only on given guidelines but results; and clarity regarding HTTP verb usage.
also on practices that the experts used. These findings represent an opportunity for future research.
Wang et al. [44] studied relationships between REST API Following guidelines has a cost, both in developer time and in
changes across versions and frequency of related questions on effects on the API. We hope that future research will evaluate
StackOverflow, finding that adding methods was associated guidelines in a quantitative way to assess both cost and ben-
with more questions than other kinds of changes. Our study efit. Establishing clearer API design guidelines may improve
focuses on guidelines and practices, not on evolution. Macvean usability, allowing developers of client applications to learn
et al. [16] studied how an API review process at Google new APIs more effectively and use them more successfully.
 R EFERENCES [23] Google. (2022) Protocol buffers. [Online]. Available: https://developers.
google.com/protocol-buffers
[1] R. T. Fielding, “Architectural Styles and the Design of Network-
[24] X. Zhu and H. A. Simon, “Learning mathematics from examples and
based Software Architectures,” PhD, University of California,
by doing,” Cognition and instruction, vol. 4, no. 3, pp. 137–166, 1987.
Irvine, 2000. [Online]. Available: https://www.proquest.com/docview/ [25] A. Renkl, “Instruction based on examples,” in Handbook of research on
304591392/abstract/CF90BE4588104415PQ/1 learning and instruction. Routledge, 2011, pp. 286–309.
[2] M. Massé, REST API Design Rulebook. O’Reilly, 2011. [26] M. Green and M. Smith, “Developers are not the enemy!: The need
[3] H. Subramanian and P. Raj, Hands-On RESTful API Design Patterns and for usable security apis,” IEEE Security & Privacy, vol. 14, no. 5, pp.
Best Practices: Design, develop, and deploy highly adaptable, scalable, 40–46, 2016.
and secure RESTful web APIs. Packt Publishing Ltd, 2019. [27] A. Parecki. (2022) Oauth 2.0. [Online]. Available: https://oauth.net
[4] M. Biehl, “RESTful API design: Best practices in api design with rest,” [28] Amazon. (2022) Amazon cognito. [Online]. Available: https://aws.
2016. amazon.com/cognito/
[5] Microsoft. (2022) RESTful web API design. [Online]. Available: https: [29] O. P. A. contributors. (2022) Open policy agent. [Online]. Available:
//docs.microsoft.com/en-us/azure/architecture/best-practices/api-design https://www.openpolicyagent.org
[6] J. Au-Yeung and R. Donovan. (2020) Best practices for rest [30] G. Fink and M. Bishop, “Property-based testing: a new approach to
api design. [Online]. Available: https://stackoverflow.blog/2020/03/02/ testing for assurance,” ACM SIGSOFT Software Engineering Notes,
best-practices-for-rest-api-design/ vol. 22, no. 4, pp. 74–80, 1997.
[7] I. OWASP Foundation. (2023) Owasp api security project. [Online]. [31] T. O. Initiative. (2022) The openapi initiative. [Online]. Available:
Available: https://owasp.org/www-project-api-security/ https://www.openapis.org
[8] Miley. (2020) 6 api design flaws you should abso- [32] D. Dygalo. (2021) Schemathesis: property-based testing
lutely avoid. [Online]. Available: https://www.dailycupoftech.com/ for API schemas. [Online]. Available: https://dygalo.dev/blog/
6-api-design-flaws-you-should-absolutely-avoid/ schemathesis-property-based-testing-for-api-schemas/
[9] AkitaSoftware.com. (2023) Akita software. [Online]. Available: https: [33] V. Atlidakis, P. Godefroid, and M. Polishchuk, “Checking security prop-
//www.akitasoftware.com erties of cloud service REST APIs,” in 2020 IEEE 13th International
[10] Google. (2023) Build RESTful APIs. [Online]. Available: https: Conference on Software Testing, Validation and Verification (ICST),
//docs.apigee.com 2020, pp. 387–397.
[11] H. Liu, S. Lu, M. Musuvathi, and S. Nath, “What bugs cause [34] R. Waller. (2022) Openapi validator. [Online]. Available: https:
production cloud incidents?” in Proceedings of the Workshop on Hot //github.com/openapi-library/OpenAPIValidators/
Topics in Operating Systems, ser. HotOS ’19. New York, NY, USA: [35] M. Corporation. (2022) Typescript. [Online]. Available: https://www.
Association for Computing Machinery, 2019, p. 155–162. [Online]. typescriptlang.org
Available: https://doi.org/10.1145/3317550.3321438 [36] G. Corporation. (2022) PyType. [Online]. Available: https://github.com/
[12] S. McLellan, A. Roesler, J. Tempest, and C. Spinuzzi, “Building more google/pytype
usable APIs,” IEEE Software, vol. 15, no. 3, pp. 78–86, 1998. [37] S. Corporation. (2022) Sorbet. [Online]. Available: https://sorbet.org
[13] M. P. Robillard, “What makes APIs hard to learn? answers from [38] M. Das, S. Lerner, and M. Seigle, “Esp: Path-sensitive program
developers,” IEEE Software, vol. 26, no. 6, pp. 27–34, 2009. verification in polynomial time,” SIGPLAN Not., vol. 37, no. 5, p. 57–68,
[14] B. A. Myers and J. Stylos, “Improving API usability,” Commun. may 2002. [Online]. Available: https://doi.org/10.1145/543552.512538
ACM, vol. 59, no. 6, p. 62–69, may 2016. [Online]. Available: [39] A. Head, E. L. Glassman, B. Hartmann, and M. A. Hearst, Interactive
https://doi.org/10.1145/2896587 Extraction of Examples from Existing Code. New York, NY, USA:
[15] D. S. Oliveira, T. Lin, M. S. Rahman, R. Akefirad, D. Ellis, E. Perez, Association for Computing Machinery, 2018, p. 1–12. [Online].
R. Bobhate, L. A. DeLong, J. Cappos, Y. Brun, and N. C. Ebner, Available: https://doi.org/10.1145/3173574.3173659
“API blindspots: Why experienced developers write vulnerable code,” in [40] J. Conners, S. Derbidge, C. Devenport, N. Farnsworth, K. Gates,
Proceedings of the Fourteenth USENIX Conference on Usable Privacy S. Lambert, C. McClain, P. Nichols, and D. Zappala, “Let’s authen-
and Security, ser. SOUPS ’18. USA: USENIX Association, 2018, p. ticate: Automated certificates for user authentication,” in Network and
315–328. Distributed Systems Security (NDSS) Symposium, 2022.
[16] A. Macvean, M. Maly, and J. Daughtry, “API design reviews at scale,” [41] J. Parker, N. Vazou, and M. Hicks, “Lweb: Information flow security
in Proceedings of the 2016 CHI Conference Extended Abstracts on for multi-tier web applications,” Proc. ACM Program. Lang., vol. 3, no.
Human Factors in Computing Systems, ser. CHI EA ’16. New York, POPL, jan 2019. [Online]. Available: https://doi.org/10.1145/3290388
NY, USA: Association for Computing Machinery, 2016, p. 849–858. [42] L. Murphy, T. Alliyu, M. B. Kery, A. Macvean, and B. A. Myers,
[Online]. Available: https://doi.org/10.1145/2851581.2851602 “Preliminary analysis of REST API style guidelines,” in PLATEAU 2017,
[17] L. Murphy, M. B. Kery, O. Alliyu, A. Macvean, and B. A. Myers, “API 2017.
designers in the field: Design practices and challenges for creating usable [43] S. Kotstein and J. Bogner, “Which restful api design rules are important
APIs,” in 2018 IEEE Symposium on Visual Languages and Human- and how do they improve software quality? a delphi study with industry
Centric Computing (VL/HCC), 2018, pp. 249–258. experts,” in Service-Oriented Computing, J. Barzen, Ed. Cham: Springer
[18] R. Fielding, M. Nottingham, and J. Reschke. (2022) Rfc 9110, http International Publishing, 2021, pp. 154–173.
semantics. [Online]. Available: https://www.rfc-editor.org/rfc/rfc9110. [44] S. Wang, I. Keivanloo, and Y. Zou, “How do developers react to RESTful
html api evolution?” in International Conference on Service-Oriented Com-
[19] R. Fielding. (2008) REST APIs must be hypertext- puting. Springer, 2014, pp. 245–259.
driven. [Online]. Available: https://roy.gbiv.com/untangled/2008/ [45] S. Software. (2022) Swagger. [Online]. Available: https://swagger.io
rest-apis-must-be-hypertext-driven [46] J. Schema. (2022) Json schema. [Online]. Available: https://json-schema.
[20] N. McDonald, S. Schoenebeck, and A. Forte, “Reliability and inter-rater org
reliability in qualitative research: Norms and guidelines for cscw and [47] S. M. Sohan, F. Maurer, C. Anslow, and M. P. Robillard, “A study of
hci practice,” Proc. ACM Hum.-Comput. Interact., vol. 3, no. CSCW, the effectiveness of usage examples in rest api documentation,” in 2017
nov 2019. [Online]. Available: https://doi.org/10.1145/3359174 IEEE Symposium on Visual Languages and Human-Centric Computing
[21] J. Nielsen and R. Molich, “Heuristic evaluation of user interfaces,” in (VL/HCC), 2017, pp. 53–61.
SIGCHI conference on Human Factors in Computing Systems, ser. CHI [48] R. Koçi, X. Franch, P. Jovanovic, and A. Abelló, “A data-driven approach
1990, 1990. to measure the usability of web apis,” in 2020 46th Euromicro Confer-
[22] Uber. (2020) Introducing domain-oriented microservice ence on Software Engineering and Advanced Applications (SEAA), Aug.
architecture. [Online]. Available: https://www.uber.com/blog/ 2020, pp. 64–71.
microservice-architecture/