Blog entries

  • Towards building a JavaScript user interface to CubicWeb

    2016/01/08 by Denis Laxalde

    This post is an introduction of a series of articles dealing with an on-going experiment on building a JavaScript user interface to CubicWeb, to ultimately replace the web component of the framework. The idea of this series is to present the main topics of the experiment, with open questions in order to eventually engage the community as much as possible. The other side of this is to experiment a blog driven development process, so getting feedback is the very point of it!

    As of today, three main topics have been identified:

    • the Web API to let the client and server communicate,
    • the issue of representing the application schema client-side, and,
    • the construction of components of the web interface (client-side).

    As part of the first topic, we'll probably rely on another experimental work about REST-fulness undertaken recently in pyramid-cubicweb (see this head for source code). Then, it appears quite clearly that we'll need sooner or later a representation of data on the client-side and that, quite obviously, the underlying format would be JSON. Apart from exchanging of entities (database) information, we already anticipate on the need for the HATEOAS part of REST. We already took some time to look at the existing possibilities. At a first glance, it seems that hydra is the most promising in term of capabilities. It's also built using semantic web technologies which definitely grants bonus point for CubicWeb. On the other hand, it seems a bit isolated and very experimental, while JSON API follows a more pragmatic approach (describe itself as an anti-bikeshedding tool) and appears to have more traction from various people. For this reason, we choose it for our first draft, but this topic seems so central in a new UI, and hard to hide as an implementation detail; that it definitely deserves more discussion. Other candidates could be Siren, HAL or Uber.

    Concerning the schema, it seems that there is consensus around JSON-Schema so we'll certainly give it a try.

    Finally, while there is nothing certain as of today we'll probably start on building components of the web interface using React, which is also getting quite popular these days. Beyond that choice, the first practical task in this topic will concern the primary view system. This task being neither too simple nor too complicated will hopefully result in a clearer overview of what the project will imply. Then, the question of edition will come up at some point. In this respect, perhaps it'll be a good time to put the UX question at a central place, in order to avoid design issues that we had in the past.

    Feedback welcome!


  • Using JSONAPI as a Web API format for CubicWeb

    2016/01/26 by Denis Laxalde

    Following the introduction post about rethinking the web user interface of CubicWeb, this article will address the topic of the Web API to exchange data between the client and the server. As mentioned earlier, this question is somehow central and deserves particular interest, and better early than late. Of the two candidate representations previously identified Hydra and JSON API, this article will focus on the later. Hopefully, this will give a better insight of the capabilities and limits of this specification and would help take a decision, though a similar experiment with another candidate would be good to have. Still in the process of blog driven development, this post has several open questions from which a discussion would hopefully emerge...

    A glance at JSON API

    JSON API is a specification for building APIs that use JSON as a data exchange format between clients and a server. The media type is application/vnd.api+json. It has a 1.0 version available from mid-2015. The format has interesting features such as the ability to build compound documents (i.e. response made of several, usually related, resources) or to specify filtering, sorting and pagination.

    A document following the JSON API format basically represents resource objects, their attributes and relationships as well as some links also related to the data of primary concern.

    Taking the example of a Ticket resource modeled after the tracker cube, we could have a JSON API document formatted as:

    GET /ticket/987654
    Accept: application/vnd.api+json
    
    {
      "links": {
        "self": "https://www.cubicweb.org/ticket/987654"
      },
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
          "description": "Well, let's try, at least...",
        },
        "relationships": {
          "concerns": {
            "links": {
              "self": "https://www.cubicweb.org/ticket/987654/relationships/concerns",
              "related": "https://www.cubicweb.org/ticket/987654/concerns"
            },
            "data": {"type": "project", "id": "1095"}
          },
          "done_in": {
            "links": {
              "self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
              "related": "https://www.cubicweb.org/ticket/987654/done_in"
            },
            "data": {"type": "version", "id": "998877"}
          }
        }
      },
      "included": [{
        "type": "project",
        "id": "1095",
        "attributes": {
            "name": "CubicWeb"
        },
        "links": {
          "self": "https://www.cubicweb.org/project/cubicweb"
        }
      }]
    }
    

    In this JSON API document, top-level members are links, data and included. The later is here used to ship some resources (here a "project") related to the "primary data" (a "ticket") through the "concerns" relationship as denoted in the relationships object (more on this later).

    While the decision of including or not these related resources along with the primary data is left to the API designer, JSON API also offers a specification to build queries for inclusion of related resources. For example:

    GET /ticket/987654?include=done_in
    Accept: application/vnd.api+json
    

    would lead to a response including the full version resource along with the above content.

    Enough for the JSON API overview. Next I'll present how various aspects of data fetching and modification can be achieved through the use of JSON API in the context of a CubicWeb application.

    CRUD

    CRUD of resources is handled in a fairly standard way in JSON API, relying of HTTP protocol semantics.

    For instance, creating a ticket could be done as:

    POST /ticket
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "ticket",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
          "description": "Well, let's try, at least...",
        },
        "relationships": {
          "concerns": {
            "data": { "type": "project", "id": "1095" }
          }
        }
      }
    }
    

    Then updating it (assuming we got its id from a response to the above request):

    PATCH /ticket/987654
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "description": "We'll succeed, for sure!",
        },
      }
    }
    

    Relationships

    In JSON API, a relationship is in fact a first class resource as it is defined by a noun and an URI through a link object. In this respect, the client just receives a couple of links and can eventually operate on them using the proper HTTP verb. Fetching or updating relationships is done using the special <resource url>/relationships/<relation type> endpoint (self member of relationships items in the first example). Quite naturally, the specification relies on GET verb for fetching targets, PATCH for (re)setting a relation (i.e. replacing its targets), POST for adding targets and DELETE to drop them.

    GET /ticket/987654/relationships/concerns
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "project",
        "id": "1095"
      }
    }
    
    PATCH /ticket/987654/relationships/done_in
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "version",
        "id": "998877"
      }
    }
    

    The body of request and response of this <resource url>/relationships/<relation type> endpoint consists of so-called resource identifier objects which are lightweight representation of resources usually only containing information about their "type" and "id" (enough to uniquely identify them).

    Related resources

    Remember the related member appearing in relationships links in the first example?

      [ ... ]
      "done_in": {
        "links": {
          "self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
          "related": "https://www.cubicweb.org/ticket/987654/done_in"
        },
        "data": {"type": "version", "id": "998877"}
      }
      [ ... ]
    

    While this is not a mandatory part of the specification, it has an interesting usage for fetching relationship targets. In contrast with the .../relationships/... endpoint, this one is expected to return plain resource objects (which attributes and relationships information in particular).

    GET /ticket/987654/done_in
    Accept: application/vnd.api+json
    
    {
      "links": {
        "self": "https://www.cubicweb.org/998877"
      },
      "data": {
        "type": "version",
        "id": "998877",
        "attributes": {
            "number": 4.2
        },
        "relationships": {
          "version_of": {
            "self": "https://www.cubicweb.org/998877/relationships/version_of",
            "data": { "type": "project", "id": "1095" }
          }
        }
      },
      "included": [{
        "type": "project",
        "id": "1095",
        "attributes": {
            "name": "CubicWeb"
        },
        "links": {
          "self": "https://www.cubicweb.org/project/cubicweb"
        }
      }]
    }
    

    Meta information

    The JSON API specification allows to include non-standard information using a so-called meta object. This can be found in various place of the document (top-level, resource objects or relationships object). Usages of this field is completely free (and optional). For instance, we could use this field to store the workflow state of a ticket:

    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
        },
        "meta": { "state": "open" }
    }
    

    Permissions

    Permissions are part of metadata to be exchanged during request/response cycles. As such, the best place to convey this information is probably within the headers. According to JSON API's FAQ, this is also the recommended way for a resource to advertise on supported actions.

    So for instance, response to a GET request could include Allow headers, indicating which request methods are allowed on the primary resource requested:

    GET /ticket/987654
    Allow: GET, PATCH, DELETE
    

    An HEAD request could also be used for querying allowed actions on links (such as relationships):

    HEAD /ticket/987654/relationships/comments
    Allow: POST
    

    This approach has the advantage of being standard HTTP, no particular knowledge of the permissions model is required and the response body is not cluttered with these metadata.

    Another possibility would be to rely use the meta member of JSON API data.

    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
        },
        "meta": {
          "permissions": ["read", "update"]
        }
      }
    }
    

    Clearly, this would minimize the amount client/server requests.

    More Hypermedia controls

    With the example implementation described above, it appears already possible to manipulate several aspects of the entity-relationship database following a CubicWeb schema: resources fetching, CRUD operations on entities, set/delete operations on relationships. All these "standard" operations are discoverable by the client simply because they are baked into the JSON API format: for instance, adding a target to some relationship is possible by POSTing to the corresponding relationship resource something that conforms to the schema.

    So, implicitly, this already gives us a fairly good level of Hypermedia control so that we're not so far from having a mature REST architecture according to the Richardson Maturity Model. But beyond these "standard" discoverable actions, the JSON API specification does not address yet Hypermedia controls in a generic manner (see this interesting discussion about extending the specification for this purpose).

    So the question is: would we want more? Or, in other words, do we need to define "actions" which would not map directly to a concept in the application model?

    In the case of a CubicWeb application, the most obvious example (that I could think of) of where such an "action" would be needed is workflow state handling. Roughly, workflows in CubicWeb are modeled through two entity types State and TrInfo (for "transition information"), the former being handled through the latter, and a relationship in_state between the workflowable entity type at stake and its current State. It does not appear so clearly how would one model this in terms of HTTP resource. (Arguably we wouldn't want to expose the complexity of Workflow/TrInfo/State data model to the client, nor can we simply expose this in_state relationship, as a client would not be able to simply change the state of a entity by updating the relation). So what would be a custom "action" to handle the state of a workflowable resource? Back in our tracker example, how would we advertise to the client the possibility to perform "open"/"close"/"reject" actions on a ticket resource? Open question...

    Request for comments

    In this post, I tried to give an overview of a possible usage of JSON API to build a Web API for CubicWeb. Several aspects were discussed from simple CRUD operations, to relationships handling or non-standard actions. In many cases, there are open questions for which I'd love to receive feedback from the community. Recalling that this topic is a central part of the experiment towards building a client-side user interface to CubicWeb, the more discussion it gets, the better!

    For those wanting to try and play themselves with the experiments, have a look at the code. This is a work-in-progress/experimental implementation, relying on Pyramid for content negotiation and route traversals.

    What's next? Maybe an alternative experiment relying on Hydra? Or an orthogonal one playing with the schema client-side?


  • Hypermedia API with cubicweb-jsonschema

    2017/04/04 by Denis Laxalde

    This is the second post of a series about cubicweb-jsonschema. The first post mainly dealt with JSON Schema representations of CubicWeb entities along with a brief description of the JSON API. In this second post, I'll describe another aspect of the project that aims at building an hypermedia API by leveraging the JSON Hyper Schema specification.

    Hypermedia APIs and JSON Hyper Schema

    Hypermedia API is somehow a synonymous of RESTful API but it makes it clearer that the API serves hypermedia responses, i.e. content that helps discoverability of other resources.

    At the heart of an hypermedia API is the concept of link relation which both aims at describing relationships between resources as well as provinding ways to manipulate them.

    In JSON Hyper Schema terminology, link relations take the form of a collection of Link Description Objects gathered into a links property of a JSON Schema document. These Link Description Objects thus describes relationships between the instance described by the JSON Schema document at stake and other resources; they hold a number of properties that makes relationships manipulation possible:

    • rel is the name of the relation, it is usually one of relation names registered at IANA;
    • href indicates the URI of the target of the relation, it may be templated by a JSON Schema;
    • targetSchema is a JSON Schema document (or reference) describing the target of the link relation;
    • schema (recently renamed as submissionSchema) is a JSON Schema document (or reference) describing what the target of the link expects when submitting data.

    Hypermedia walkthrough

    In the remaining of the article, I'll walk through a navigation path that is made possible by hypermedia controls provided by cubicweb-jsonschema. I'll continue on the example application described in the first post of the series which schema consists of Book, Author and Topic entity types. In essence, this walkthrough is typical of what an intelligent client could do when exposed to the API, i.e. from any resource, discover other resources and navigate or manipulate them.

    This walkthrough assumes that, given any resource (i.e. something that has a URL like /book/1), the server would expose data at the main URL when the client asks for JSON through the Accept header and it would expose the JSON Schema of the resource at a schema view of the same URL (i.e. /book/1/schema). This assumption can be regarded as a kind of client/server coupling, which might go away in later implementation.

    Site root

    While client navigation could start from any resource, we start from the root resource and retrieve its schema:

    GET /schema
    Accept: application/schema+json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "links": [
            {
                "href": "/author/",
                "rel": "collection",
                "schema": {
                    "$ref": "/author/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/author/schema"
                },
                "title": "Authors"
            },
            {
                "href": "/book/",
                "rel": "collection",
                "schema": {
                    "$ref": "/book/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Books"
            },
            {
                "href": "/topic/",
                "rel": "collection",
                "schema": {
                    "$ref": "/topic/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/topic/schema"
                },
                "title": "Topics"
            }
        ]
    }
    

    So at root URL, our application serves a JSON Hyper Schema that only consists of links. It has no JSON Schema document, which is natural since there's usually no data bound to the root resource (think of it as empty rset in CubicWeb terminology).

    These links correspond to top-level entity types, i.e. those that would appear in the default startup page of a CubicWeb application. They all have "rel": "collection" relation name (this comes from RFC6573) as their target is a collection of entities. We also have schema and targetSchema properties.

    From collection to items

    Now that we have added a new book, let's step back and use our books link to retrieve data (verb GET):

    GET /book/
    Accept: application/json
    
    HTTP/1.1 200 OK
    Allow: GET, POST
    Content-Type: application/json
    
    [
        {
            "id": "859",
            "title": "L'homme qui rit"
        },
        {
            "id": "858",
            "title": "The Old Man and the Sea"
        },
    ]
    

    which, as always, needs to be completed by a JSON Schema:

    GET /book/schema
    Accept: application/schema+json
    
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "$ref": "#/definitions/Book_plural",
        "definitions": {
            "Book_plural": {
                "items": {
                    "properties": {
                        "id": {
                            "type": "string"
                        },
                        "title": {
                            "type": "string"
                        }
                    },
                    "type": "object"
                },
                "title": "Books",
                "type": "array"
            }
        },
        "links": [
            {
                "href": "/book/",
                "rel": "collection",
                "schema": {
                    "$ref": "/book/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Books"
            },
            {
                "href": "/book/{id}",
                "rel": "item",
                "targetSchema": {
                    "$ref": "/book/schema?role=view"
                },
                "title": "Book"
            }
        ]
    }
    

    Consider the last item of links in the above schema. It has a "rel": "item" property which indicates how to access items of the collection; its href property is a templated URI which can be expanded using instance data and schema (here we only have a single id template variable).

    So our client may navigate to the first item of the collection (id="859") at /book/859 URI, and retrieve resource data:

    GET /book/859
    Accept: application/json
    
    HTTP/1.1 200 OK
    Allow: GET, PUT, DELETE
    Content-Type: application/json
    
    {
        "author": [
            "Victor Hugo"
        ],
        "publication_date": "1869-04-01T00:00:00",
        "title": "L'homme qui rit"
    }
    

    and schema:

    GET /book/859/schema
    Accept: application/schema+json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "$ref": "#/definitions/Book",
        "definitions": {
            "Book": {
                "additionalProperties": false,
                "properties": {
                    "author": {
                        "items": {
                            "type": "string"
                        },
                        "title": "author",
                        "type": "array"
                    },
                    "publication_date": {
                        "format": "date-time",
                        "title": "publication date",
                        "type": "string"
                    },
                    "title": {
                        "title": "title",
                        "type": "string"
                    },
                    "topics": {
                        "items": {
                            "type": "string"
                        },
                        "title": "topics",
                        "type": "array"
                    }
                },
                "title": "Book",
                "type": "object"
            }
        },
        "links": [
            {
                "href": "/book/",
                "rel": "up",
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Book_plural"
            },
            {
                "href": "/book/859/",
                "rel": "self",
                "schema": {
                    "$ref": "/book/859/schema?role=edition"
                },
                "targetSchema": {
                    "$ref": "/book/859/schema?role=view"
                },
                "title": "Book #859"
            }
        ]
    }
    

    Entity resource

    The resource obtained above as an item of a collection is actually an entity. Notice the rel="self" link. It indicates how to manipulate the current resource (i.e. at which URI, using a given schema depending on what actions we want to perform). Still this link does not indicate what actions may be performed. This indication is found in the Allow header of the data response above:

    Allow: GET, PUT, DELETE
    

    With these information bits, our intelligent client is able to, for instance, form a request to delete the resource. On the other hand, the action to update the resource (which is allowed because of the presence of PUT in Allow header, per HTTP semantics) would take the form of a request which body conforms to the JSON Schema pointed at by the schema property of the link.

    Also note the rel="up" link which makes it possible to navigate to the collection of books.

    Conclusions

    This post introduced the main hypermedia capabilities of cubicweb-jsonschema, built on top of the JSON Hyper Schema specification. The resulting Hypermedia API makes it possible for an intelligent client to navigate through hypermedia resources and manipulate them by using both link relation semantics and HTTP verbs.

    In the next post, I'll deal with relationships description and manipulation both in terms of API (endpoints) and hypermedia representation.