Property-Based Testing of Web APIs

Abstract

Fuzzing - whether generating or mutating inputs - has found many bugs and security vulnerabilities in a wide range of domains. Stateful and highly structured web APIs present significant challenges to traditional fuzzing techniques, as execution feedback is usually limited to a response code instead of code coverage and vulnerabilities of interest include silent information-disclosure in addition to explicit errors.

We present schemathesis, a tool for structured fuzzing of web APIs. Schemathesis exploits API schema descriptions in formats like OpenAPI 7 or GraphQL, as well as the rich featureset of the Hypothesis property-based testing library, and is easy to use as either a standalone tool or integrated into a unit-test suite. To generalise our findings we also present a survey of testable generic and context-specific properties of web APIs, a taxonomy of the subtasks of web API fuzzing, and a summary of typical defect types and frequencies for three web API implementation strategies.

Our evaluation shows that Schemathesis finds NNN times more bugs than previous web-API fuzzing tools, including NNN previously unknown issues and NNN CVEs. We demonstrate that much of this improved performance is attributable to features and heuristics in the underlying property-based testing library. We also evaluate the contribution of undirected vs. feedback-directed generation, and find that using API responses finds NNN more bugs/coverage, using branch coverage of the server finds XXX, and using both finds finds XXX.

Introduction

Much modern software communicates over the internet, and each service therefore defines some kind of web API - often via a REST [Fie00] or more recently GraphQL [Fou20] architecture. Many services also provide machine-readable schemas or specifications which describe their input and output contracts and their semantics. Schema-based tools can generate documentation or client code, support interoperability, and provide a very high level of test automation.

OpenAPI [Ini20] (and GraphQL) schemas are often unsound, either because it is uneconomical to work with schemas which express every relevant constraint, or because there are application-level constraints 8 which cannot be expressed in the schema format. Schemas therefore provide a useful basis for testing, but leave plenty of room for human judgement in analysing failures, detecting over-restrictive schemas, and hand-coding additional or more precise tests.

These challenges apply regardless of whether the schema is derived from the application, vice-versa, or neither: code-first, schema-first, and manual approaches have varying characteristic defects, but none are sufficient to maintain a complete and accurate schema as applications evolve. Extensive and highly automated testing is not a silver bullet, but we believe that making defect detection substantially cheaper and easier can help.

To summarise the contributions of this paper:

  • We define the subtasks of web API fuzzing, including a list of testable properties (and corresponding defect types) applicable to HTTP, OpenAPI, and other web APIs.

  • We describe the implementation of Schemathesis as property-based testing, and demonstrate effectiveness as well as ease of development.

  • We provide a large and reusable suite of realistic applications and web API fuzzers, suitable for future research, and report indicative defect rates for various implementation strategies and frameworks.

Defining the task of web API fuzzing

This section is rough notes only so far.

  • Exploiting schemas to generate inputs allows fuzzers to access service logic - traditional fuzzing would be unlikely to make it past protocol logic or input validation.

  • Most useful when langauge-independent, which implies sparse feedback. HTTP responses, not code coverage, though the latter may be exploited where available.

  • Defining subtasks of web API fuzzing is one of our survey/concept contributions. We also generalise from previous work on REST+Swagger APIs to include GraphQL. Zac TODO: I could swear I’ve seen some stuff about other web schema types…

Exploiting API schemas makes the process much more efficient - we can spend our time doing things that should work or almost work, instead of wasting most of it on trivial rejections due to invalid input or failed preconditions.

I think there’s a contribution in defining the components of this testing task. For example: parse the schema, generate matching inputs, make requests, make sequences of requests, detect errors, detect silent security issues. No coincidence that ICST saw so many Swagger-testing papers this year; and IMO presenting a structure of the large problem and describing what each paper was working on would be a solid contribution. We will probably want to flag this in the abstract.

Previous work has fuzzed Swagger (REST) APIs - we categorise the subtasks etc. Generalising this approach to also target GraphQL APIs means … Dmitri: we might see different endpoints, but different logical operations might might be more interesting through batching operations rather than serial. Deep nesting might also make things more interesting.

Property-based testing

Property-based testing (PBT) originated with the Haskell library QuickCheck [CH00], which emphasised testing algebraic properties of functions by generating many random inputs to a test function. PBT generally differs from fuzzing more in workflow and the affordances of tooling 9 than fundamental concept.

Hypothesis [MHDC19] is the definitive PBT library for Python - used by thousands of open source projects and companies all over the world, and with dozens of third-party extensions. Hypothesis is explicitly designed as a library of tools which make it easy to construct fuzzable tests - which can detect errors that direct fuzzing would not - and a non-user-visible fuzzing backend for them 10.

This abstraction has been very successful. Features including integrated shrinking and error-deduplication [MD], targeted property-based testing [LoscherS17] [LS18] [ASAH20], or coverage-guided fuzzing (nb. HypoFuzz paper in progress) can be implemented at the fuzzing level and support any Hypothesis test. Implementing ‘strategies’ 11 for input generation is therefore simplified, leading to a wide range of schema-to-strategy translators for annotations, grammars, and many other kinds of schemas like hypothesis-jsonschema or hypothesis-graphql.

Prior Art: fuzzing REST APIs

While the REST architecture was described in 2000 [Fie00], highly automated web API fuzzers have largely followed the popularity of OpenAPI (formerly Swagger) schemas. The first credible projects date mostly to 2016-17, with capabilities improving rapidly from 2018-2020 and no obvious end in sight. We are unaware of similar tools targeting GraphQL schemas, perhaps because the GraphQL spec is younger, more complex, and less widely used than OpenAPI.

Schemathesis was directly inspired by the earlier Hypothesis-based swagger-conformance project [Pra17], which last updated in 2018. Yelp’s fuzz-lightyear tool [LLZ20] is similarly based on Hypothesis and OpenAPI schemas, and attempts to discover Insecure Direct Object Reference (IDOR) vulnerabilities through stateful fuzzing - a fairly direct implementation of [AGP20] from before RESTler was made open-source.

Microsoft’s RESTler [AGP19] infers dependencies among request types, generates sequences of requests satisfying those dependencies, and learns to predict sequence validity based on the service reponses to these test sequences. They find that these features are required for effective testing of web APIs.

QuickREST [KCS19] takes a broader approach than RESTler, drawing on property-based testing to check for a range of invariants beyond security issues.

Note: QuickREST has an excellent related work section (VI) - Zac to follow up and expand this, especially related to web APIs and generating data from schemas.

[VDC20] proposes some heuristics to recover information about dependencies between operations when the schema does not provide links. We might want to implement these in Schemathesis but I wouldn’t bother citing it otherwise. It also comments on the importance of negative testing, and proposes a structured mutation approach. Dmitri has a prototype of generating-from-mutated schema, which simulates schema errors.

[MPGB20] uses a combination of web-crawling to find URLs which do not require authentication, and a catalog of twenty-two security properties defined in their custom DSL for metamorphic relations. Again, only really of interest if we teach Schemathesis to check these properties; but in that case rather nice since they had a manual step we can probably automate via a ``fuzz-lightyear``-like approach.

Testable properties of web APIs

A substantial contribution of this paper is to summarise the literature on testable properties of web APIs, including proposing some of our own 1. Below, we provide empirical results of the frequency and severity of detected defects across a range of APIs categorised by (TODO) implementation style and domain.

This should probably be a “Defect type” column of the table above, plus an explanatory paragraph or two. Note that we use the summarised version when reporting evaluation results below.

Defects can generally be grouped into one of four categories. In general order of increasing severity and difficulty of detection:

  1. Non-conforming responses,

  2. Malformed responses,

  3. Logically incorrect responses, and

  4. Security vulnerabilities

Note that the severity and categorisation of some specific defect is application-specific and often disputed. For example a HTTP 500 response is by definition an error; but may be a trivial bug or a denial-of-service vulnerability depending on the service. For non-conforming responses, either the code or the response schema may be defective.

I’ve left this as a separate sub-section for now, because I think the specific categorisation we use should be reflected in Schemathesis’ reporting of the errors and/or config options to disable those which may not be of interest to specific users. Something to discuss, but ultimately I’m waiting on some evaluation results.

Table 1: Testable properties of web APIs

Property

Applies to

Checked by

Comments

Never HTTP 500

any HTTP API

Schemathesis, …

“No internal server errors” is table stakes.

HTTP 200 has body

any HTTP API

per RFC 7231; if no payload use 204/205

Empty body on HTTP 204,205

any HTTP API

Dredd, …

per RFC 7231

HTTP 302 POST-to-GET

any HTTP API

per RFC 7231, client may change POST to GET for subsequent request

HTTP 405 list allowed

any HTTP API

per RFC 7231, server must list methods which are allowed

Non-conforming response

OpenAPI, GraphQL

Schemathesis, …

Undeclared status code

OpenAPI, ??

Schemathesis, …

Wrong or missing headers

OpenAPI

Schemathesis, …

Wrong content-type

OpenAPI, ??

Schemathesis, …

Rejects non-conforming input

OpenAPI, GraphQL

[VDC20], …

The basic “negative test”

Very slow responses

Any HTTP API

Schemathesis, …

Generally an optional check. Can detect DOS attacks.

Access after delete

OpenAPI (stateful)

[AGP20], …

“Use-after-free rule”

Access after creation failed

OpenAPI (stateful)

[AGP20], …

“Resource-leak rule”

“Resource-hierarchy rule”

OpenAPI (stateful?)

[AGP20], …

Specific to the structure of their cloud service API

“User-namespace rule”

OpenAPI (stateful?)

[AGP20], …

Specific to the structure of their cloud service API

Insecure direct reference

OpenAPI, GraphQL?

fuzz-lightyear, …

The motivating case for [LLZ20] at Yelp

22 from MR catalog

[MPGB20], …

Zac to read and fill paper, fill table

TODO: Zac to continue expanding this table. https://github.com/SNTSVV/SMRL_Library/tree/master/src-mrs-icst2020/smrl/mr/owasp We may also want to add a column categorising the bug type; and consider columns vs cell content to distinguish stateless/stateful properties, and universal/contextual properties.

1

assuming nobody else has checked for undeclared status code, headers, or content-type

Schemathesis

This section describes the implementation. Dmitri to write most of it; copying freely from existing documentation if that helps.

Includes both how it works, but also what it does and why people use it (ie how it enhances their workflow)

Zac would love to have some discussion of how users can customise the behaviour, too.

Can use targeted PBT to search for denial-of-service attacks; e.g. timeouts or find amplification attacks by targeting response size.

IBM has a schemathesis extension [Sch20] which also checks conformance to the IBM API Handbook.

Hypothesis-JSONschema

hypothesis-jsonschema 2 provides the from_schema() function, which takes an arbitrary JSON Schema 3 and returns a Hypothesis strategy to generate valid instances. JSONschema is something of a lingua fraca for web related schemas; Swagger and OpenAPI use it directly, while others are easy to convert into JSONschemas. Simple schemas admit simple translations:

{"type": "integer", "minimum": 0, "maximum": 10}

while others, especially if they involve oneOf or allOf 4 combinations, defy easy translation. For example, the following schema is exactly equivalent:

{"allOf": [
    {"type": "integer", "minimum": 0},
    {"maximum": 10}
]}

but the naive approach of generating examples from each and filtering by validity under the others would be unacceptably slow 5. Keeping rejection sampling to a minimum therefore makes the fuzzer considerably faster than naive translators such as [BEFHMarino14].

Such nonlocal constraints are common in real-world schemas 6, even before accounting for widespread use of the $ref keyword. We therefore inline non-recursive references, merge overlapping subschemas, and canonicalise the results before converting the schema to a Hypothesis strategy.

As well as generating valid inputs with no relation to ordinary production traffic, this logic gives us an elegant way to synthesise subtly invalid examples: create a set of variant schemas, and then use each of the variants to generate input. For example, we might wish to generate an invalid input of the correct type:

schema = ...
gen_invalid = from_schema({"type": schema["type"], "not": schema})

and our canonicalisation logic will make this simple approach as efficient as anything else we could write by hand.

2

https://pypi.org/project/hypothesis-jsonschema/

3

https://json-schema.org/

4

the anyOf keyword can trivially be satisfied by taking the union of the generators for sub-schemas.

5

a negligible fraction of all integers >= 0 are also <= 10, and vice-versa, while the second sub-schema allows any type JSON object except integers greater than ten.

6

hypothesis-jsonschema is test against every schema in the upstream compatibility test suite, as well as hundreds of larger schemas from https://www.schemastore.org

Evaluation

We evaluate Schemathesis with the goal of answering four questions:

  1. What real-world bugs does Schemathesis detect?

  2. How does Schemathesis compare to previous web API fuzzers?

  3. What contribution do property-based testing tools make?

  4. How much does coverage help in web API fuzzing?

Our evaluation corpus consists of NNN existing web API servers, listed in Appendix One: full list of tested services. These services were chosen to represent a variety of implementation strategies (schema-first, code-first, and manually synced), specification formats (OpenAPI and GraphQL), and programming langauges; and demonstrate a realistic range of sizes, structures, and API complexity.

The full set of containers to reproduce our work, or use it in evaluating future fuzzers, is available from TODO INSERT LINK HERE.

Real-world bug detection

We run Schemathesis on NNN web APIs, and detect a total of NNN defects including NN non-conforming responses, NN malformed responses, NN logically incorrect responses, NN internal errors, and NN potential or actual security vulnerabilities (of which N have been confirmed upstream). We report disaggregated results, which are too long for this paper, in Appendix Two: disaggregated results.

Table 2: Defect counts by implementation strategy

Implementation

Non-conforming

Malformed

Incorrect

Internal error

Security

Schema-first

? / ?

? / ?

? / ?

? / ?

? / ?

Code-first

? / ?

? / ?

? / ?

? / ?

? / ?

Manual sync

? / ?

? / ?

? / ?

? / ?

? / ?

These results suggest that…

Table 3: Defect counts by programming language and framework

Implementation

Non-conforming

Malformed

Incorrect

Internal error

Security

Python (Django)

? / ?

? / ?

? / ?

? / ?

? / ?

Python (Flask)

? / ?

? / ?

? / ?

? / ?

? / ?

Python (other)

? / ?

? / ?

? / ?

? / ?

? / ?

Ruby (Rails)

? / ?

? / ?

? / ?

? / ?

? / ?

Ruby (other)

? / ?

? / ?

? / ?

? / ?

? / ?

Javascript

? / ?

? / ?

? / ?

? / ?

? / ?

Go

? / ?

? / ?

? / ?

? / ?

? / ?

Other

? / ?

? / ?

? / ?

? / ?

? / ?

This suggests that…

Comparison to previous web API fuzzers

We compare the following API fuzzers

Table 4: Testing tools description

Name

Implementation

Spec support

Schemathesis

Python

Open API 2 / 3, GraphQL

Cats

Java

Open API 2 / 3

TnT-Fuzzer

Python

Open API 2

Got-Swag

JavaScript

Open API 2

APIFuzzer

Python

Open API 2

Fuzz-lightyear

Python

Open API 2

Swagger-conformance

Python

Open API 2

OpenAPI3-fuzzer

Python

Open API 3

Effectiveness

i.e. running Schemathesis, and each of the similar tools we can find, on some set of APIs we can find and reporting the runtime, bug-finding ability, and where possible code coverage we get from each.

The contribution of property-based testing

Zac is interested in exploring how insights from PBT help make fuzzing tools more effective. The recent change of implementation for stateful testing in schemathesis from hand-rolled to using Hypothesis’ support for rule-based stateful testing means we can do a realistic experiment! (testing the effect of heuristics and tricks like swarm testing).

Basically: compare new-schemathesis and old-schemathesis as for “Effectiveness”, above.

How much does coverage help?

“Everyone knows” that coverage information is very valuable for a fuzzer. But how useful is it for fuzzing web APIs, and what kind of coverage feedback do you need? We compare ‘blind’ fuzzing to the use of language-independent ‘event coverage’ of API responses (e.g. HTTP status codes, headers, etc.), use of branch coverage for API servers written in Python, and both these forms of feedback.

Same deal: set up our four fuzzer variants; and measure bugs+coverage+speed for some set of APIs we can run against.

EvoMaster (TODO Zac to cite papers) outputs source code for individual test cases, based on the Swagger schema and coverage feedback from a Java API server. They do note that blackbox works for non-Java systems but with considerably worse performance. This is clearly related to e.g. Randoop and EvoSuite unit-test generators (somewhat like the Hypothesis Ghostwriter). We may want to follow up on this as a conceptual link, though I’m inclined to think it would leave the paper unfocussed.

Conclusion

  • Use schemathesis, it’s better than the rest

  • Future research should be based on property-based testing

7

renamed from https://swagger.io/ to https://www.openapis.org/

8

such as database constraints, relations between endpoints, etc.

9

typically focussing on highly-structured and always-valid data, along with integrated shrinking

10

https://hypothesis.works/articles/what-is-property-based-testing/

11

Hypothesis “strategies”, objects which describe the set of values which might be generated, are called “generators” in most PBT frameworks. Unfortunately a “generator” is already a builtin type in Python.

References

ASAH20

C. Aschermann, S. Schumilo, A. Abbasi, and T. Holz. IJON: exploring deep state spaces via fuzzing. In 2020 IEEE Symposium on Security and Privacy (SP), volume, 1597–1612. 2020. URL: https://www.syssec.ruhr-uni-bochum.de/media/emma/veroeffentlichungen/2020/02/27/IJON-Oakland20.pdf.

AGP19

Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. Rest-ler: automatic intelligent rest api fuzzing. In Proceedings of the 41st International Conference on Software Engineering. IEEE Press, 2019. URL: https://www.microsoft.com/en-us/research/uploads/prod/2018/04/restler.pdf.

AGP20(1,2,3,4,5)

Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. Checking security properties of cloud service REST APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE, October 2020. URL: https://www.microsoft.com/en-us/research/uploads/prod/2019/02/paper2.pdf, doi:10.1109/icst46399.2020.00046.

BEFHMarino14

Clara Benac Earle, Lars-Åke Fredlund, Ángel Herranz, and Julio Mariño. Jsongen: a quickcheck based library for testing json web services. In Proceedings of the Thirteenth ACM SIGPLAN Workshop on Erlang, Erlang ‘14, 33–41. New York, NY, USA, 2014. Association for Computing Machinery. URL: https://doi.org/10.1145/2633448.2633454, doi:10.1145/2633448.2633454.

CH00

Koen Claessen and John Hughes. Quickcheck: a lightweight tool for random testing of haskell programs. Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming, 2000. URL: https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf, doi:10.1145/357766.351266.

Fie00(1,2)

Roy Thomas Fielding. REST: Architectural Styles and the Design of Network-based Software Architectures. Doctoral dissertation, University of California, Irvine, 2000. URL: http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm.

Fou20

The GraphQL Foundation. Graphql. 2020. URL: https://graphql.org/.

Ini20

The OpenAPI Initiative. Openapi. 2020. URL: https://www.openapis.org/.

KCS19

Stefan Karlsson, Adnan Causevic, and Daniel Sundmark. Quickrest: property-based test generation of openapi-described restful apis. 2019. arXiv:1912.09686.

LoscherS17

Andreas Löscher and Konstantinos Sagonas. Targeted property-based testing. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2017, 46–56. New York, NY, USA, 2017. Association for Computing Machinery. URL: https://proper-testing.github.io/papers/issta2017.pdf, doi:10.1145/3092703.3092711.

LLZ20(1,2)

Aaron Loo, Joey Lee, and Victor Zhou. Automated idor discovery through stateful swagger fuzzing. 2020. URL: https://engineeringblog.yelp.com/2020/01/automated-idor-discovery-through-stateful-swagger-fuzzing.html.

LS18

A. Löscher and K. Sagonas. Automating targeted property-based testing. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), volume, 70–80. 2018. URL: https://proper-testing.github.io/papers/icst2018.pdf, doi:10.1109/ICST.2018.00017.

MD

David MacIver and Alastair Donaldson. Test-case reduction via test-case generation: insights from the hypothesis reducer. to be published in https://2020.ecoop.org/details/ecoop-2020-papers/13/Test-Case-Reduction-via-Test-Case-Generation-Insights-From-the-Hypothesis-Reducer. URL: https://drmaciver.github.io/papers/reduction-via-generation-preview.pdf.

MHDC19

David MacIver, Zac Hatfield-Dodds, and Many Contributors. Hypothesis: a new approach to property-based testing. Journal of Open Source Software, 4(43):1891, 2019. URL: https://doi.org/10.21105/joss.01891, doi:10.21105/joss.01891.

MPGB20(1,2)

P. X. Mai, F. Pastore, A. Goknil, and L. Briand. Metamorphic security testing for web systems. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), volume, 186–197. 2020. URL: https://arxiv.org/pdf/1912.05278.pdf, doi:10.1109/ICST46399.2020.00028.

Pra17

Oli Pratt. Swagger-conformance. 2017. URL: https://pypi.org/project/swagger-conformance/.

Sch20

Barrett Schonefeld. Ibm-service-validator. 2020. URL: https://pypi.org/project/ibm-service-validator/.

VDC20(1,2)

E. Viglianisi, M. Dallago, and M. Ceccato. Resttestgen: automated black-box testing of restful apis. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), volume, 142–152. 2020. URL: https://sntsvv.github.io/SMRL/, doi:10.1109/ICST46399.2020.00024.

Appendices

Appendix One: full list of tested services

TODO: Dmitry. Order these web services by relevance:
  • docker + OAS schema

  • OAS schema

  • docker

Table 3: Tested web services

Repository URL

Language

Framework

Specification

Specification type

API description location

Has Docker?

Has Docker-compose?

https://github.com/15Dkatz/official_joke_api

JavaScript

Express 4.16.4

No

No

No

No

https://github.com/24pullrequests/24pullrequests

Ruby

Rails 5.2.4.2

No

No

Yes

Yes

https://github.com/aalises/age-of-empires-II-api

Python

Flask

Open API 3.0.0

Static

https://github.com/aalises/age-of-empires-II-api/blob/master/docs/apispecs.yaml

Yes

No

https://github.com/aaronbassett/FARM

Python

FastAPI

Open API 3

Generated

No

No

https://github.com/aarreedd/uuidtools.com

PHP

Laravel 6.2

No

No

No

No

https://github.com/adafruit/adafruit-io-node#server

JavaScript

Express 4.13.3

Swagger 2.0

Static

https://github.com/adafruit/adafruit-io-node/blob/master/server/swagger.json

No

No

https://github.com/AdenFlorian/random.dog

JavaScript

Express 4.15.4

No

No

No

No

https://github.com/afuh/rick-and-morty-api

JavaScript

Express 4.17.1

GraphQL

Generated

Incomplete

Incomplete

https://github.com/ajnisbet/opentopodata/

Python

Flask 1.1.2

No

No

https://github.com/ajnisbet/opentopodata/blob/master/docs/api.md

Yes

No

https://github.com/ajzbc/kanye.rest

JavaScript

No

No

No

No

No

https://github.com/AlbertSuarez/object-cut

Python

Flask

Open API 3.0.1

Static

https://github.com/AlbertSuarez/object-cut/blob/master/multiplexer/src/openapi/openapi.yaml

Yes

Yes

https://github.com/AlbertSuarez/searchly

Python

Flask

Open API 3.0.1

Static

https://github.com/AlbertSuarez/searchly/blob/master/src/searchly/static/openapi/openapi_v1.yaml

Yes

Yes

https://github.com/alexwohlbruck/cat-facts

JavaScript

Express 4.17.1

No

No

No

No

https://github.com/amundsen-io/amundsenmetadatalibrary

Python

Flask

Open API 3.0.2

Generated

flasgger

Yes

No

https://github.com/andyklimczak/TheReportOfTheWeek-API

JavaScript

Express 4.13.4

No

No

No

No

https://github.com/apache/couchdb

Erlang

No

No

No

https://docs.couchdb.org/en/stable/api/index.html

Yes

No

https://github.com/apilayer/geolocationapi

Python

Starlette

No

No

No

No

https://github.com/apilayer/ratesapi

JavaScript

No

No

No

No

No

https://github.com/apilayer/restcountries

Java

Jboss resteasy

Swagger 2.0

Static

https://github.com/IgorJeri/restcountries.eu-OPEN-API-v2

No

No

https://github.com/apis-is/apis

JavaScript

Express 4.16.3

GraphQL

Generated

No

No

https://github.com/argovaerts/hookless.co

PHP

No

No

No

No

No

https://github.com/aunyks/newton-api

JavaScript

Express 4.14.1

No

No

No

No

https://github.com/azrogers/ShibeOnline

Rust

Actix-web 0.7

No

No

No

No

https://github.com/bagelbits/5e-srd-api

JavaScript

Express 4.17.1

REST + GraphQL

Static + Generated

https://www.dnd5eapi.co/docs/

Yes

Yes

https://github.com/bejaneps/agreement-api

Go

Net/HTTP

No

No

No

No

https://github.com/bikeindex/bikewise

Ruby

Rails 4.1

Swagger 1.2

Generated

No

No

https://github.com/CenterForOpenScience/SHARE

Python

Django 1.11.16

JSON API

Generated

https://share.osf.io/api/v2/

Yes

Yes

https://github.com/cezarykluczynski/stapi

Groovy

Spring Boot

Swagger 2.0

Static

https://github.com/cezarykluczynski/stapi/tree/master/contract/src/main/resources/v1/swagger

No

No

https://github.com/Chocobozzz/PeerTube

TypeScript

Express 4.12.4

Open API 3.0.0

Static

https://github.com/Chocobozzz/PeerTube/blob/develop/support/doc/api/openapi.yaml

Yes

Yes

https://github.com/chrisjm/openbrewerydb-rails-api

Ruby

Rails 5.2.4.4

No

No

https://www.openbrewerydb.org/documentation

No

No

https://github.com/chucknorris-io/chuck-api

Java

Spring Boot

Swagger 2.0

Generated

Yes

Yes

https://github.com/ckan/ckan

Python

Flask 1.1.1

No

No

Yes

Yes

https://github.com/Countly/countly-server

JavaScript

Express 4.17.1

No

No

https://api.count.ly/reference

Yes

Yes

https://github.com/creativecommons/cccatalog-api

Python

Django 2.2.13

Swagger 2.0

Generated

DRF-yasg

Yes

Yes

https://github.com/crobertsbmw/deckofcards

Python

Django 2.2.10

No

No

http://deckofcardsapi.com/

Yes

No

https://github.com/darklynx/request-baskets

Go

Net/HTTP

Swagger 2.0

Static

https://github.com/darklynx/request-baskets/blob/master/doc/api-swagger.yaml

Yes

No

https://github.com/davemachado/public-api

Go

Net/HTTP

No

No

Yes

No

https://github.com/derhuerst/vbb-rest

JavaScript

Express 4.16.2

No

No

Yes

No

https://github.com/DiljotSG/MCU-Countdown

Python

Flask 1.1.2

No

No

https://github.com/DiljotSG/MCU-Countdown/blob/develop/docs/API.md

No

No

https://github.com/disease-sh/api

JavaScript

Express 4.16.3

Swagger 2.0

Static

https://github.com/disease-sh/API/tree/master/public/assets/apidocs

Yes

Yes

https://github.com/DMarby/picsum-photos

Go

Net/HTTP

No

No

Yes

No

https://github.com/drewthoennes/Bored-API

JavaScript

Express 4.17.1

No

No

No

No

https://github.com/DukeNgn/Dog-facts-API

Python

Flask 1.1.2

No

No

No

No

https://github.com/dunglas/mercure/

Go

Net/HTTP

Open API 3.0.2

Static

https://github.com/dunglas/mercure/blob/main/spec/openapi.yaml

Yes

No

https://github.com/e1ven/Robohash

Python

Tornado

No

No

No

No

https://github.com/egno/work-calendar

Python

Flask

No

No

Yes

Yes

https://github.com/ElliottLandsborough/dog-ceo-api

PHP

Symfony

No

No

https://github.com/ElliottLandsborough/dog-ceo-api#endpoints

No

No

https://github.com/epam/Indigo

Python

Flask

Swagger 2.0

Generated

flasgger

Yes

Yes

https://github.com/eskerda/citybikes-api/

Python

Flask

No

No

Yes

Yes

https://github.com/etalab/adresse.data.gouv.fr

JavaScript

Express 4.17.1

No

No

No

No

https://github.com/etalab/geo.api.gouv.fr

JavaScript

Express 4.17.1

Swagger 2.0

Static

https://geo.api.gouv.fr/definition.yml

No

No

https://github.com/europeana/api2

Java

Spring

Swagger 2.0

Generated

Yes

Yes

https://github.com/EvilInsultGenerator/website

PHP

No

No

No

No

No

https://github.com/evz/tacofancy-api

Python

Flask 0.10.1

No

No

https://github.com/evz/tacofancy-api#use-this-data

No

No

https://github.com/exchangeratesapi/exchangeratesapi

Python

Sanic

No

No

https://github.com/exchangeratesapi/exchangeratesapi#usage

No

No

https://github.com/exini/slicebox

Scala

Akka-HTTP 10.1.7

Swagger 2.0

Static

https://exini.github.io/slicebox-api/

No

No

https://github.com/fecgov/openFEC

Python

Flask 1.1.1

Swagger 2.0

Generated

https://github.com/jmcarp/flask-apispec

No

No

https://github.com/fedspendingtransparency/usaspending-api

Python

Django 2.2

No

No

Yes

Yes

https://github.com/feedbin/feedbin

Ruby

Rails 6.0.3.4

No

No

Incomplete

Incomplete

https://github.com/ferreirix/restratp

C#

ASP.NET 2.0.0

Swagger 2.0

Generated

https://github.com/domaindrivendev/Swashbuckle.AspNetCore/tree/v1.0.0-rc3

No

No

https://github.com/fiorix/freegeoip

Go

Net/HTTP

No

No

https://github.com/fiorix/freegeoip#api

Yes

No

https://github.com/fisenkodv/dictum

C#

ASP.NET

No

No

https://github.com/fisenkodv/dictum#api-methods

Yes

No

https://github.com/fixerAPI/fixer

Ruby

Sinatra

No

No

https://github.com/fixerAPI/fixer#usage

Yes

Yes

https://github.com/fossasia/pslab-webapp

Python

Flask >= 0.10.1

Swagger 2.0

Generated

flasgger

No

No

https://github.com/GillesWHC/taylor.rest

TypeScript

Oak

No

No

Yes

No

https://github.com/GIScience/openpoiservice

Python

Flask 0.12.2

Swagger 2.0

Generated

Flasgger 0.8.1

Yes

Yes

https://github.com/gita/bhagavad-gita-backend

Python

Flask 1.1.2

No

No

No

No

https://github.com/gita/BhagavadGita

Python

Flask 1.0.2

Open API 3.0.2

Generated

flasgger

No

No

https://github.com/gitfrosh/lotr-api

JavaScript

Express 4.17.1

No

No

https://the-one-api.dev/documentation

No

Incomplete

https://github.com/gogs/gogs

Go

Net/HTTP

No

No

https://github.com/gogs/docs-api

Yes

No

https://github.com/hakanensari/country

JavaScript

Express 4.17.1

No

No

https://github.com/hakanensari/country#usage

Yes

No

https://github.com/hakanensari/frankfurter

Ruby

Roda

No

No

https://github.com/hakanensari/frankfurter#getting-started

Yes

Yes

https://github.com/hamdyaea/SolarSystemPercentage

Python

Flask

No

No

No

No

https://github.com/heycarsten/lcbo-api

Ruby

Rails 5.2.2

No

No

Yes

Yes

https://github.com/Hipo/university-domains-list-api

Python

Flask

No

No

No

No

https://github.com/HSLdevcom/hsl-map-server

JavaScript

Express 4.16.4

No

No

Yes

No

https://github.com/IBM/worklog

Python

Flask 1.0.2

Swagger 2.0

Static

Yes

Yes

https://github.com/IcaliaLabs/sepomex

Ruby

Rails 6.0.3.2

No

No

Yes

Yes

https://github.com/ideal-postcodes/postcodes.io/

JavaScript

Express 4.17.1

No

No

Yes

Yes

https://github.com/igneus/church-calendar-api

Ruby

Grape

Swagger 2.0

Rendered

https://github.com/igneus/church-calendar-api/blob/master/apps/web/views/swagger.yml.erb

Yes

Yes

https://github.com/internetarchive/openlibrary

Python

Web.py 0.62

No

No

https://openlibrary.org/developers/api

Yes

Yes

https://github.com/iRail/hyperRail

PHP

Laravel 5.1

No

No

No

No

https://github.com/jamesRUS52/worktime-api

PHP

Phpfrm 1.5

Open API 3.0.0

Static

https://app.swaggerhub.com/apis-docs/jamesRUS52/worktime/1.0.0#/

No

No

https://github.com/jamesseanwright/ron-swanson-quotes

TypeScript

Express 4.11.0

Open API 3.0.0

Static

https://github.com/jamesseanwright/ron-swanson-quotes/blob/master/src/schema.json

No

No

https://github.com/janaipakos/ghibliapi

JavaScript

JSON-Server 0.8.4

Swagger 2.0

Static

https://github.com/janaipakos/ghibliapi/blob/master/public/swagger.yaml

No

No

https://github.com/jenkinsci/jenkins

Java

?

No

No

https://ci.jenkins.io/api/

Yes

Yes

https://github.com/jikan-me/jikan-rest

PHP

Laravel/Lumen 5.8

API Blueprint

Static

https://github.com/jikan-me/jikan-rest/blob/master/apiary.apib

Yes

Yes

https://github.com/joakimskoog/AnApiOfIceAndFire

C#

ASP.NET Core 2.1.6

No

No

No

No

https://github.com/jsdelivr/data.jsdelivr.com

JavaScript

Koa 2.13.0

No

No

https://github.com/jsdelivr/data.jsdelivr.com#endpoints

Incomplete

No

https://github.com/judge0/judge0

Ruby

Rails 5.0

No

No

Yes

Yes

https://github.com/jupyterhub/jupyterhub

Python

Tornado 5.1

Swagger 2.0

Static

https://github.com/jupyterhub/jupyterhub/blob/master/docs/rest-api.yml

Yes

No

https://github.com/kiwitcms/Kiwi

Python

Django 3.1.3

No

No

Yes

Yes

https://github.com/lelouchB/final-space-api

JavaScript

Express 4.17.1

No

No

No

No

https://github.com/libertysoft3/saidit

Python

Pylons

No

No

No

No

https://github.com/lukePeavey/quotable

JavaScript

Express 4.17.1

No

No

No

No

https://github.com/M-Media-Group/Cartes.io

PHP

Laravel 5.8

No

No

No

No

https://github.com/marciovsena/abibliadigital

JavaScript

Express 4.17.1

No

No

Yes

Yes

https://github.com/Meeshkan/micro-jaymock

TypeScript

Micro 9.3.4

No

No

No

No

https://github.com/meetDeveloper/googleDictionaryAPI

JavaScript

Express 4.16.2

No

No

No

No

https://github.com/microsoft/restler-fuzzer

Python

Flask

Swagger 2.0

Static

https://github.com/microsoft/restler-fuzzer/blob/main/demo_server/swagger.json

No

No

https://github.com/mmazzarolo/numvalidate

JavaScript

Koa 2.3.0

No

No

Yes

Yes

https://github.com/mozilla/http-observatory

Python

Flask 1.0.2

No

No

Yes

Yes

https://github.com/mozilla/tls-observatory#api-endpoints

Go

Net/HTTP

No

No

https://github.com/mozilla/tls-observatory#api-endpoints

Yes

Incomplete

https://github.com/nager/Nager.Date

C#

ASP.NET Core 3.0

OpenAPI 3.0.0

Generated

https://github.com/domaindrivendev/Swashbuckle.AspNetCore/releases/tag/v5.6.3

Yes

No

https://github.com/netbox-community/netbox

Python

Django 3.1

?

Generated

Drf-yasg 1.17.1

Yes

Yes

https://github.com/NTag/lyrics.ovh

JavaScript

Express 4.14.0

API Blueprint

Static

https://github.com/NTag/lyrics.ovh/blob/master/apiary.apib

Yes

Yes

https://github.com/open-notify/Open-Notify-API

Python

Flask 0.11.1

No

No

Incomplete

Incomplete

https://github.com/openaq/openaq-api

JavaScript

HAPI 8.8.0

No

No

Yes

Yes

https://github.com/openchargemap/ocm-system

C#

ASP.NET Core 3.1

No

No

Yes

No

https://github.com/OpendataCH/Transport

PHP

Silex

Swagger 2.0

Generated

Yes

Yes

https://github.com/OpenMined/opus

Python

Flask

Swagger 2.0

Generated

flasgger

Yes

Yes

https://github.com/opennorth/represent-canada

Python

Django 1.11.28

No

No

No

No

https://github.com/osm-search/Nominatim

PHP

No

No

No

https://github.com/osm-search/Nominatim/tree/master/docs/api

Yes

No

https://github.com/PawelPleskaczynski/apod_api

JavaScript

Express 4.16.4

No

No

No

No

https://github.com/pcraig3/hols

JavaScript

Express 4.17.1

Open API 3.0.0

Static

https://github.com/pcraig3/hols/blob/main/reference/Canada-Holidays-API.v1.yaml

Yes

No

https://github.com/phalt/swapi

Python

Django 1.7.4

JSON API

Static

https://github.com/phalt/swapi/tree/master/resources/schemas

No

Incomplete

https://github.com/PokeAPI/pokeapi/

Python

Django 2.1.11

No

No

Yes

Yes

https://github.com/postmanlabs/httpbin

Python

Flask

Swagger 2.0

Generated

flasgger

Yes

Yes

https://github.com/PostmonAPI/postmon

Python

Bottle

Open API 3.0.2

Static

https://github.com/PostmonAPI/postmon/blob/master/docs/openapi.json

Yes

No

https://github.com/pprathameshmore/QuoteGarden

JavaScript

Express 4.17.1

No

No

No

No

https://github.com/r-spacex/SpaceX-API

JavaScript

Koa

No

No

https://github.com/amorriscode/SpaceX-API/tree/master/docs

Yes

No

https://github.com/RandomAPI/Randomuser.me-Node

JavaScript

Express 4.17.1

No

No

Incomplete

Incomplete

https://github.com/rdegges/ipify-api

Go

Net/HTTP

No

No

No

No

https://github.com/redraw/satellite-passes-api

Python

Flask 1.1.2

No

No

https://github.com/redraw/satellite-passes-api#api

Yes

No

https://github.com/reel2bits/reel2bits

Python

Flask 1.1.1

Swagger 2.0

Generated

Flasgger 0.9.3

Yes

Yes

https://github.com/RefugeRestrooms/refugerestrooms

Ruby

Rails 5.2.4.4

Swagger 2.0

Generated

Yes

Yes

https://github.com/robertoduessmann/weather-api

Go

Net/HTTP

No

No

No

No

https://github.com/RocktimSaikia/anime-chan

JavaScript

Express 4.17.1

No

No

https://github.com/RocktimSaikia/anime-chan#routes

No

No

https://github.com/rosshettel/placegoat

JavaScript

Express 4.13.3

No

No

https://placegoat.com/

No

No

https://github.com/RTradeLtd/Temporal

Go

Net/HTTP

No

No

Yes

Yes

https://github.com/ryo-ma/covid19-japan-web-api

Python

Flask

Swagger 2.0

Generated

flasgger

No

No

https://github.com/shevabam/breaking-bad-quotes

JavaScript

Express 4.14.1

No

No

https://breakingbadquotes.xyz/

Yes

No

https://github.com/shevabam/game-of-thrones-quotes-api

JavaScript

Express 4.17.1

No

No

https://github.com/shevabam/game-of-thrones-quotes-api#api

No

No

https://github.com/shuup/shuup

Python

Django 2.2

No

No

Yes

Yes

https://github.com/skolakoda/programming-quotes-api

JavaScript

Express 4.16.2

No

No

No

No

https://github.com/skywave/KV78Turbo-OVAPI

Python

Raw UWSGI

No

No

No

No

https://github.com/slotix/COVID-19

Go

Net/HTTP

No

No

https://github.com/slotix/COVID-19#endpoints

No

No

https://github.com/slotix/dataflowkit

Go

Net/HTTP

No

No

Yes

Yes

https://github.com/soruly/sola

JavaScript

No

No

No

Yes

Yes

https://github.com/sottenad/jService

Ruby

Rails 4.0.2

No

No

No

No

https://github.com/spaceflightnewsapi/spaceflightnewsapi

JavaScript

StrAPI 3.3.3

Open API 3.0.1

Rendered

https://github.com/spaceflightnewsapi/spaceflightnewsapi/blob/main/extensions/documentation/config/settings.json

Yes

No

https://github.com/surhud004/Foodish

JavaScript

Express 4.17.1

No

No

https://github.com/surhud004/Foodish#api-documentation

No

No

https://github.com/Sv443/JokeAPI

JavaScript

No

No

No

https://sv443.net/jokeapi/v2/

No

No

https://github.com/swagger-api/swagger-codegen

Java

?

Swagger 2.0

?

Yes

No

https://github.com/tasdikrahman/plino

Python

Flask 0.12.1

No

No

No

No

https://github.com/technoweenie/guillotine

Ruby

Sinatra 1.4.0

No

No

No

No

https://github.com/theIYD/NaMoMemes

JavaScript

Express 4.17.1

No

No

No

No

https://github.com/thm/uinames

PHP

No

No

No

No

No

https://github.com/timbiles/Breaking-Bad–API

JavaScript

Express 4.16.3

No

No

No

No

https://github.com/tomdionysus/foaas

JavaScript

Express 4.17.1

No

No

Yes

No

https://github.com/transitland/transitland-datastore

Ruby

Rails 4.2.11.1

No

No

Yes

Yes

https://github.com/treflehq/trefle-api

Ruby

Rails 6.0.3.2

Open API 3.0.3

Generated

Yes

No

https://github.com/tronalddump-io/tronald-app

Kotlin

Spring Boot

Open API 3.0.1

Generated

No

Incomplete

https://github.com/tutyamxx/Romanian-Jokes-API

JavaScript

Express 4.17.1

No

No

No

No

https://github.com/typicode/jsonplaceholder

JavaScript

JSON-Server

No

No

Incomplete

No

https://github.com/typpo/quickchart

JavaScript

Express 4.16.4

No

No

Yes

Yes

https://github.com/uhh-lt/targer

Python

Flask 1.1.1

Swagger 2.0

Generated

Flasgger 0.9.3

Yes

Yes

https://github.com/usnationalarchives/federalregister-api-core

Ruby

Rails 6.0.3.2

Open API 3.0.0

Rendered

https://github.com/usnationalarchives/federalregister-api-core/blob/main/data/open_api_v3.yml

Yes

Yes

https://github.com/uttesh/exude-api

Java

Spring Boot

Open API 3.0.0

Static

https://github.com/uttesh/exude-api/blob/master/exude-service-open-api.json

Yes

No

https://github.com/versioneye/versioneye-api

Ruby

Rails 5.1.4

Swagger 1.2

Generated

GrapeSwagger 0.6.0

Yes

Yes

https://github.com/wger-project/wger

Python

Django 3.1

No

No

Yes

Yes

https://github.com/wikimedia/mediawiki

PHP

No

No

No

https://www.mediawiki.org/wiki/API:Main_page

Yes

Yes

https://github.com/workforce-data-initiative/skills-api

Python

Flask 0.11.1

Swagger 2.0

Static

https://github.com/workforce-data-initiative/skills-api/blob/master/skills-api.json

No

No

https://github.com/xinitrc-dev/randomfox.ca

PHP

No

No

No

https://github.com/xinitrc-dev/randomfox.ca/blob/master/api/v1/getfoxes/index.php

No

No

https://github.com/ynnadkrap/balldontlie

Ruby

Rails 5.1.4

No

No

Yes

No

https://github.com/zippopotamus/zippopotamus

Python

Bottle

No

No

No

No

https://gitlab.com/gitlab-org/gitlab

Ruby

Rails 6.0.3.1

Open API 3.0.3 + GraphQL

Static

https://docs.gitlab.com/ee/api/api_resources.html

Yes

Yes

https://opendev.org/openstack/openstack

Many

Many

?

?

https://docs.openstack.org/api-quick-start/

?

?

https://videobin.org/code

Python

Django 1.3

No

No

https://videobin.org/api

No

No

Appendix Three: more bugs

This section will be substantially re-designed and re-written; at present it’s more of a dumping ground so that we don’t lose track of anything.

Schemathesis found two bugs in github.com/redraw/satellite-passes-api, which have since been fixed. However this project is not part of our evaluation suite, because it makes network calls to an external service.