Blog entries

  • Introducing cubicweb-jsonschema

    2017/03/23 by Denis Laxalde

    This is the first post of a series introducing the cubicweb-jsonschema project that is currently under development at Logilab. In this post, I'll first introduce the general goals of the project and then present in more details two aspects about data models (the connection between Yams and JSON schema in particular) and the basic features of the API. This post does not always present how things work in the current implementation but rather how they should.

    Goals of cubicweb-jsonschema

    From a high level point of view, cubicweb-jsonschema addresses mainly two interconnected aspects. One related to modelling for client-side development of user interfaces to CubicWeb applications while the other one concerns the HTTP API.

    As far as modelling is concerned, cubicweb-jsonschema essentially aims at providing a transformation mechanism between a Yams schema and JSON Schema that is both automatic and extensible. This means that we can ultimately expect that Yams definitions alone would sufficient to have generated JSON schema definitions that would consistent enough to build an UI, pretty much as it is currently with the automatic web UI in CubicWeb. A corollary of this goal is that we want JSON schema definitions to match their context of usage, meaning that a JSON schema definition would not be the same in the context of viewing, editing or relationships manipulations.

    In terms of API, cubicweb-jsonschema essentially aims at providing an HTTP API to manipulate entities based on their JSON Schema definitions.

    Finally, the ultimate goal is to expose an hypermedia API for a CubicWeb application in order to be able to ultimately build an intelligent client. For this we'll build upon the JSON Hyper-Schema specification. This aspect will be discussed in a later post.

    Basic usage as an HTTP API library

    Consider a simple case where one wants to manipulate entities of type Author described by the following Yams schema definition:

    class Author(EntityType):
        name = String(required=True)
    

    With cubicweb-jsonschema one can get JSON Schema for this entity type in at different contexts such: view, creation or edition. For instance:

    • in a view context, the JSON Schema will be:

      {
          "$ref": "#/definitions/Author",
          "definitions": {
              "Author": {
                  "additionalProperties": false,
                  "properties": {
                      "name": {
                          "title": "name",
                          "type": "string"
                      }
                  },
                  "title": "Author",
                  "type": "object"
              }
          }
      }
      
    • whereas in creation context, it'll be:

      {
          "$ref": "#/definitions/Author",
          "definitions": {
              "Author": {
                  "additionalProperties": false,
                  "properties": {
                      "name": {
                          "title": "name",
                          "type": "string"
                      }
                  },
                  "required": [
                      "name"
                  ],
                  "title": "Author",
                  "type": "object"
              }
          }
      }
      

      (notice, the required keyword listing name property).

    Such JSON Schema definitions are automatically generated from Yams definitions. In addition, cubicweb-jsonschema exposes some endpoints for basic CRUD operations on resources through an HTTP (JSON) API. From the client point of view, requests on these endpoints are of course expected to match JSON Schema definitions. Some examples:

    Get an author resource:

    GET /author/855
    Accept:application/json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    {"name": "Ernest Hemingway"}
    

    Update an author:

    PATCH /author/855
    Accept:application/json
    Content-Type: application/json
    {"name": "Ernest Miller Hemingway"}
    
    HTTP/1.1 200 OK
    Location: /author/855/
    Content-Type: application/json
    {"name": "Ernest Miller Hemingway"}
    

    Create an author:

    POST /author
    Accept:application/json
    Content-Type: application/json
    {"name": "Victor Hugo"}
    
    HTTP/1.1 201 Created
    Content-Type: application/json
    Location: /Author/858
    {"name": "Victor Hugo"}
    

    Delete an author:

    DELETE /author/858
    
    HTTP/1.1 204 No Content
    

    Now if the client sends invalid input with respect to the schema, they'll get an error:

    (We provide a wrong born property in request body.)

    PATCH /author/855
    Accept:application/json
    Content-Type: application/json
    {"born": "1899-07-21"}
    
    HTTP/1.1 400 Bad Request
    Content-Type: application/json
    
    {
        "errors": [
            {
                "details": "Additional properties are not allowed ('born' was unexpected)",
                "status": 422
            }
        ]
    }
    

    From Yams model to JSON Schema definitions

    The example above illustrates automatic generation of JSON Schema documents based on Yams schema definitions. These documents are expected to help developping views and forms for a web client. Clearly, we expect that cubicweb-jsonschema serves JSON Schema documents for viewing and editing entities as cubicweb.web serves HTML documents for the same purposes. The underlying logic for JSON Schema generation is currently heavily inspired by the logic of primary view and automatic entity form as they exists in cubicweb.web.views. That is: the Yams schema is introspected to determine how properties should be generated and any additionnal control over this can be performed through uicfg declarations [1].

    To illustrate let's consider the following schema definitions which:

    class Book(EntityType):
        title = String(required=True)
        publication_date = Datetime(required=True)
    
    class Illustration(EntityType):
        data = Bytes(required=True)
    
    class illustrates(RelationDefinition):
        subject = 'Illustration'
        object = 'Book'
        cardinality = '1*'
        composite = 'object'
        inlined = True
    
    class Author(EntityType):
        name = String(required=True)
    
    class author(RelationDefinition):
        subject = 'Book'
        object = 'Author'
        cardinality = '1*'
    
    class Topic(EntityType):
        name = String(required=True)
    
    class topics(RelationDefinition):
        subject = 'Book'
        object = 'Topic'
        cardinality = '**'
    

    and consider, as before, JSON Schema documents in different contexts for the the Book entity type:

    • in view context:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "type": "string"
                          },
                          "title": "author",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      },
                      "topics": {
                          "items": {
                              "type": "string"
                          },
                          "title": "topics",
                          "type": "array"
                      }
                  },
                  "title": "Book",
                  "type": "object"
              }
          }
      }
      

      We have a single Book definition in this document, in which we find attributes defined in the Yams schema (title and publication_date). We also find the two relations where Book is involved: topics and author, both appearing as a single array of "string" items. The author relationship appears like that because it is mandatory but not composite. On the other hand, the topics relationship has the following uicfg rule:

      uicfg.primaryview_section.tag_subject_of(('Book', 'topics', '*'), 'attributes')
      

      so that it's definition appears embedded in the document of Book definition.

      A typical JSON representation of a Book entity would be:

      {
          "author": [
              "Ernest Miller Hemingway"
          ],
          "title": "The Old Man and the Sea",
          "topics": [
              "sword fish",
              "cuba"
          ]
      }
      
    • in creation context:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "oneOf": [
                                  {
                                      "enum": [
                                          "855"
                                      ],
                                      "title": "Ernest Miller Hemingway"
                                  },
                                  {
                                      "enum": [
                                          "857"
                                      ],
                                      "title": "Victor Hugo"
                                  }
                              ],
                              "type": "string"
                          },
                          "maxItems": 1,
                          "minItems": 1,
                          "title": "author",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      }
                  },
                  "required": [
                      "title",
                      "publication_date"
                  ],
                  "title": "Book",
                  "type": "object"
              }
          }
      }
      

      notice the differences, we now only have attributes and required relationships (author) in this schema and we have the required listing mandatory attributes; the author property is represented as an array which items consist of pre-existing objects of the author relationship (namely Author entities).

      Now assume we add the following uicfg declaration:

      uicfg.autoform_section.tag_object_of(('*', 'illustrates', 'Book'), 'main', 'inlined')
      

      the JSON Schema for creation context will be:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "oneOf": [
                                  {
                                      "enum": [
                                          "855"
                                      ],
                                      "title": "Ernest Miller Hemingway"
                                  },
                                  {
                                      "enum": [
                                          "857"
                                      ],
                                      "title": "Victor Hugo"
                                  }
                              ],
                              "type": "string"
                          },
                          "maxItems": 1,
                          "minItems": 1,
                          "title": "author",
                          "type": "array"
                      },
                      "illustrates": {
                          "items": {
                              "$ref": "#/definitions/Illustration"
                          },
                          "title": "illustrates_object",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      }
                  },
                  "required": [
                      "title",
                      "publication_date"
                  ],
                  "title": "Book",
                  "type": "object"
              },
              "Illustration": {
                  "additionalProperties": false,
                  "properties": {
                      "data": {
                          "format": "data-url",
                          "title": "data",
                          "type": "string"
                      }
                  },
                  "required": [
                      "data"
                  ],
                  "title": "Illustration",
                  "type": "object"
              }
          }
      }
      

      We now have an additional illustrates property modelled as an array of #/definitions/Illustration, the later also added the the document as an additional definition entry.

    Conclusion

    This post illustrated how a basic (CRUD) HTTP API based on JSON Schema could be build for a CubicWeb application using cubicweb-jsonschema. We have seen a couple of details on JSON Schema generation and how it can be controlled. Feel free to comment and provide feedback on this feature set as well as open the discussion with more use cases.

    Next time, we'll discuss how hypermedia controls can be added the HTTP API that cubicweb-jsonschema provides.

    [1]this choice is essentially driven by simplicity and conformance when the existing behavior to help migration of existing applications.

  • Hypermedia API with cubicweb-jsonschema

    2017/04/04 by Denis Laxalde

    This is the second post of a series about cubicweb-jsonschema. The first post mainly dealt with JSON Schema representations of CubicWeb entities along with a brief description of the JSON API. In this second post, I'll describe another aspect of the project that aims at building an hypermedia API by leveraging the JSON Hyper Schema specification.

    Hypermedia APIs and JSON Hyper Schema

    Hypermedia API is somehow a synonymous of RESTful API but it makes it clearer that the API serves hypermedia responses, i.e. content that helps discoverability of other resources.

    At the heart of an hypermedia API is the concept of link relation which both aims at describing relationships between resources as well as provinding ways to manipulate them.

    In JSON Hyper Schema terminology, link relations take the form of a collection of Link Description Objects gathered into a links property of a JSON Schema document. These Link Description Objects thus describes relationships between the instance described by the JSON Schema document at stake and other resources; they hold a number of properties that makes relationships manipulation possible:

    • rel is the name of the relation, it is usually one of relation names registered at IANA;
    • href indicates the URI of the target of the relation, it may be templated by a JSON Schema;
    • targetSchema is a JSON Schema document (or reference) describing the target of the link relation;
    • schema (recently renamed as submissionSchema) is a JSON Schema document (or reference) describing what the target of the link expects when submitting data.

    Hypermedia walkthrough

    In the remaining of the article, I'll walk through a navigation path that is made possible by hypermedia controls provided by cubicweb-jsonschema. I'll continue on the example application described in the first post of the series which schema consists of Book, Author and Topic entity types. In essence, this walkthrough is typical of what an intelligent client could do when exposed to the API, i.e. from any resource, discover other resources and navigate or manipulate them.

    This walkthrough assumes that, given any resource (i.e. something that has a URL like /book/1), the server would expose data at the main URL when the client asks for JSON through the Accept header and it would expose the JSON Schema of the resource at a schema view of the same URL (i.e. /book/1/schema). This assumption can be regarded as a kind of client/server coupling, which might go away in later implementation.

    Site root

    While client navigation could start from any resource, we start from the root resource and retrieve its schema:

    GET /schema
    Accept: application/schema+json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "links": [
            {
                "href": "/author/",
                "rel": "collection",
                "schema": {
                    "$ref": "/author/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/author/schema"
                },
                "title": "Authors"
            },
            {
                "href": "/book/",
                "rel": "collection",
                "schema": {
                    "$ref": "/book/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Books"
            },
            {
                "href": "/topic/",
                "rel": "collection",
                "schema": {
                    "$ref": "/topic/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/topic/schema"
                },
                "title": "Topics"
            }
        ]
    }
    

    So at root URL, our application serves a JSON Hyper Schema that only consists of links. It has no JSON Schema document, which is natural since there's usually no data bound to the root resource (think of it as empty rset in CubicWeb terminology).

    These links correspond to top-level entity types, i.e. those that would appear in the default startup page of a CubicWeb application. They all have "rel": "collection" relation name (this comes from RFC6573) as their target is a collection of entities. We also have schema and targetSchema properties.

    From collection to items

    Now that we have added a new book, let's step back and use our books link to retrieve data (verb GET):

    GET /book/
    Accept: application/json
    
    HTTP/1.1 200 OK
    Allow: GET, POST
    Content-Type: application/json
    
    [
        {
            "id": "859",
            "title": "L'homme qui rit"
        },
        {
            "id": "858",
            "title": "The Old Man and the Sea"
        },
    ]
    

    which, as always, needs to be completed by a JSON Schema:

    GET /book/schema
    Accept: application/schema+json
    
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "$ref": "#/definitions/Book_plural",
        "definitions": {
            "Book_plural": {
                "items": {
                    "properties": {
                        "id": {
                            "type": "string"
                        },
                        "title": {
                            "type": "string"
                        }
                    },
                    "type": "object"
                },
                "title": "Books",
                "type": "array"
            }
        },
        "links": [
            {
                "href": "/book/",
                "rel": "collection",
                "schema": {
                    "$ref": "/book/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Books"
            },
            {
                "href": "/book/{id}",
                "rel": "item",
                "targetSchema": {
                    "$ref": "/book/schema?role=view"
                },
                "title": "Book"
            }
        ]
    }
    

    Consider the last item of links in the above schema. It has a "rel": "item" property which indicates how to access items of the collection; its href property is a templated URI which can be expanded using instance data and schema (here we only have a single id template variable).

    So our client may navigate to the first item of the collection (id="859") at /book/859 URI, and retrieve resource data:

    GET /book/859
    Accept: application/json
    
    HTTP/1.1 200 OK
    Allow: GET, PUT, DELETE
    Content-Type: application/json
    
    {
        "author": [
            "Victor Hugo"
        ],
        "publication_date": "1869-04-01T00:00:00",
        "title": "L'homme qui rit"
    }
    

    and schema:

    GET /book/859/schema
    Accept: application/schema+json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "$ref": "#/definitions/Book",
        "definitions": {
            "Book": {
                "additionalProperties": false,
                "properties": {
                    "author": {
                        "items": {
                            "type": "string"
                        },
                        "title": "author",
                        "type": "array"
                    },
                    "publication_date": {
                        "format": "date-time",
                        "title": "publication date",
                        "type": "string"
                    },
                    "title": {
                        "title": "title",
                        "type": "string"
                    },
                    "topics": {
                        "items": {
                            "type": "string"
                        },
                        "title": "topics",
                        "type": "array"
                    }
                },
                "title": "Book",
                "type": "object"
            }
        },
        "links": [
            {
                "href": "/book/",
                "rel": "up",
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Book_plural"
            },
            {
                "href": "/book/859/",
                "rel": "self",
                "schema": {
                    "$ref": "/book/859/schema?role=edition"
                },
                "targetSchema": {
                    "$ref": "/book/859/schema?role=view"
                },
                "title": "Book #859"
            }
        ]
    }
    

    Entity resource

    The resource obtained above as an item of a collection is actually an entity. Notice the rel="self" link. It indicates how to manipulate the current resource (i.e. at which URI, using a given schema depending on what actions we want to perform). Still this link does not indicate what actions may be performed. This indication is found in the Allow header of the data response above:

    Allow: GET, PUT, DELETE
    

    With these information bits, our intelligent client is able to, for instance, form a request to delete the resource. On the other hand, the action to update the resource (which is allowed because of the presence of PUT in Allow header, per HTTP semantics) would take the form of a request which body conforms to the JSON Schema pointed at by the schema property of the link.

    Also note the rel="up" link which makes it possible to navigate to the collection of books.

    Conclusions

    This post introduced the main hypermedia capabilities of cubicweb-jsonschema, built on top of the JSON Hyper Schema specification. The resulting Hypermedia API makes it possible for an intelligent client to navigate through hypermedia resources and manipulate them by using both link relation semantics and HTTP verbs.

    In the next post, I'll deal with relationships description and manipulation both in terms of API (endpoints) and hypermedia representation.