Page MenuHomeSoftware Heritage

[POC] OpenAPI and Django REST Framework to specify / implement API v2
Needs ReviewPublic

Authored by anlambert on Nov 27 2020, 7:22 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

That diff showcases the result of my experiments using OpenAPI and Django REST Framework for the implementation of Software Heritage Web API v2.
I mostly focused on how to plug an already defined OpenAPI specification using Django.
Once all that plumbing implemented, the idea is to drive API v2 development by its specification.

It is NOT intended to be landed. Its purpose is to discuss about the implementation approach.

Introduction to OpenAPI

The OpenAPI Specification is a specification for machine-readable interface files for describing, producing, consuming, and visualizing RESTful web services.

The current version is 3.0.3 and it enables a fine grained specification for a REST API.
It uses an extended subset of JSON Schema Specification Wright Draft 00 (aka Draft 5) to describe the data formats.
It also enables to split API specification in multiple files based on the
JSON reference specification.

It is language-agnostic. With OpenAPI's declarative resource specification, clients can understand and consume services without knowledge of server implementation.

It is a broadly adopted industry standard for describing modern APIs (see GithHub API description for instance).

It exists a lot of tools that works with OpenAPI, notably to:

  • parse and validate specification
  • validate HTTP requests and responses according to an endpoint specification (parameters, headers, body, ...)
  • generate API client libraries or server stubs (OpenAPI Generator for instance) from a specification
  • generate interactive HTML documentation for a specified REST API (the most famous one is swagger-ui)

OpenAPI and Django REST Framework

I did some reviews of open source projects using OpenAPI with DRF and found the following:

However, all those projects follow the mantra: *implement first then generate OpenAPI specification*.

From my point of view, this is not the right one to follow and it should rather be: *specify with OpenAPI first then connect to implementation*.
It is notably used by:

I did not found any OpenAPI based project for Django REST Framework using that latter mantra so I tried to implement it in that POC.

Software Heritage API v2 implementation proposal

As explained above, the idea is to be able to specify first then connect to endpoints implementation.

Proceeding like this has several advantages:

  • specification can be validated for correctness using dedicated tools
  • specification can be organized in multiple logical files in order to share and reuse components (schema, parameters, responses, ...)
  • input HTTP requests can be validated according to endpoint specification (parameters format, expected headers, ...)

In order to ease those tasks, I used two interesting Python modules that I found:

  • prance: OpenAPI specification parser, resolver and validator
  • openapi-core: OpenAPI request and response validator compatible with Django

In that diff, the idea was to specify a couple of endpoints related to Software Heritage content objects and connect these specified endpoints to their implementation.
I inspired from the preliminary work of @douardda on the subject to write the specification but I splitted the specification
into multiple files.

Once the specification written, swh-web will use it to register the API endpoints by following that process at startup:

  1. The specification is parsed and validated using prance
  2. For each specified API path, create a DRF API view wrapping a function implementing

the endpoint logic. The name of that function can be found in the path specification under the operationId property.

That diff also adds the following:

  • HTTP requests are validated using openapi-core prior executing endpoint implementation logic
  • HTTP responses are validated in debug mode
  • Full dereferenced OpenAPI specification is made available under the /api/2/schema path
  • Web API v2 interactive HTML documentation (based on swagger-ui) is made available under the /api/2/doc path (see screenshot below)

To summarize, OpenAPI is quite pleasant to use and the variety of tools working with it will help us to get a properly
specified and documented Web API v2 while being able to change the implementation backend.

Diff Detail

Repository
rDWAPPS Web applications
Branch
openapi-drf-poc
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 20784
Build 32251: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 32250: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4629 (id=16423)

Rebasing onto 8492a4c688...

Current branch diff-target is up to date.
Changes applied before test
commit 2888e11918eb6db6c45deda0c4a2e3b98a523c5c
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Mon Nov 9 16:33:08 2020 +0100

    [POC] OpenAPI and Django REST Framework to specify / implement API v2

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/502/ for more details.

Build is green

Patch application report for D4629 (id=19769)

Rebasing onto b095564e54...

Current branch diff-target is up to date.
Changes applied before test
commit 050f7fcf37fde84edd90bbe25e06fee3cdd9fe51
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Mon Nov 9 16:33:08 2020 +0100

    [POC] OpenAPI and Django REST Framework to specify / implement API v2

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/689/ for more details.