Blog entries january 2016 [3]
  • Using JSONAPI as a Web API format for CubicWeb

    2016/01/26 by Denis Laxalde

    Following the introduction post about rethinking the web user interface of CubicWeb, this article will address the topic of the Web API to exchange data between the client and the server. As mentioned earlier, this question is somehow central and deserves particular interest, and better early than late. Of the two candidate representations previously identified Hydra and JSON API, this article will focus on the later. Hopefully, this will give a better insight of the capabilities and limits of this specification and would help take a decision, though a similar experiment with another candidate would be good to have. Still in the process of blog driven development, this post has several open questions from which a discussion would hopefully emerge...

    A glance at JSON API

    JSON API is a specification for building APIs that use JSON as a data exchange format between clients and a server. The media type is application/vnd.api+json. It has a 1.0 version available from mid-2015. The format has interesting features such as the ability to build compound documents (i.e. response made of several, usually related, resources) or to specify filtering, sorting and pagination.

    A document following the JSON API format basically represents resource objects, their attributes and relationships as well as some links also related to the data of primary concern.

    Taking the example of a Ticket resource modeled after the tracker cube, we could have a JSON API document formatted as:

    GET /ticket/987654
    Accept: application/vnd.api+json
    
    {
      "links": {
        "self": "https://www.cubicweb.org/ticket/987654"
      },
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
          "description": "Well, let's try, at least...",
        },
        "relationships": {
          "concerns": {
            "links": {
              "self": "https://www.cubicweb.org/ticket/987654/relationships/concerns",
              "related": "https://www.cubicweb.org/ticket/987654/concerns"
            },
            "data": {"type": "project", "id": "1095"}
          },
          "done_in": {
            "links": {
              "self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
              "related": "https://www.cubicweb.org/ticket/987654/done_in"
            },
            "data": {"type": "version", "id": "998877"}
          }
        }
      },
      "included": [{
        "type": "project",
        "id": "1095",
        "attributes": {
            "name": "CubicWeb"
        },
        "links": {
          "self": "https://www.cubicweb.org/project/cubicweb"
        }
      }]
    }
    

    In this JSON API document, top-level members are links, data and included. The later is here used to ship some resources (here a "project") related to the "primary data" (a "ticket") through the "concerns" relationship as denoted in the relationships object (more on this later).

    While the decision of including or not these related resources along with the primary data is left to the API designer, JSON API also offers a specification to build queries for inclusion of related resources. For example:

    GET /ticket/987654?include=done_in
    Accept: application/vnd.api+json
    

    would lead to a response including the full version resource along with the above content.

    Enough for the JSON API overview. Next I'll present how various aspects of data fetching and modification can be achieved through the use of JSON API in the context of a CubicWeb application.

    CRUD

    CRUD of resources is handled in a fairly standard way in JSON API, relying of HTTP protocol semantics.

    For instance, creating a ticket could be done as:

    POST /ticket
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "ticket",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
          "description": "Well, let's try, at least...",
        },
        "relationships": {
          "concerns": {
            "data": { "type": "project", "id": "1095" }
          }
        }
      }
    }
    

    Then updating it (assuming we got its id from a response to the above request):

    PATCH /ticket/987654
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "description": "We'll succeed, for sure!",
        },
      }
    }
    

    Relationships

    In JSON API, a relationship is in fact a first class resource as it is defined by a noun and an URI through a link object. In this respect, the client just receives a couple of links and can eventually operate on them using the proper HTTP verb. Fetching or updating relationships is done using the special <resource url>/relationships/<relation type> endpoint (self member of relationships items in the first example). Quite naturally, the specification relies on GET verb for fetching targets, PATCH for (re)setting a relation (i.e. replacing its targets), POST for adding targets and DELETE to drop them.

    GET /ticket/987654/relationships/concerns
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "project",
        "id": "1095"
      }
    }
    
    PATCH /ticket/987654/relationships/done_in
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "version",
        "id": "998877"
      }
    }
    

    The body of request and response of this <resource url>/relationships/<relation type> endpoint consists of so-called resource identifier objects which are lightweight representation of resources usually only containing information about their "type" and "id" (enough to uniquely identify them).

    Related resources

    Remember the related member appearing in relationships links in the first example?

      [ ... ]
      "done_in": {
        "links": {
          "self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
          "related": "https://www.cubicweb.org/ticket/987654/done_in"
        },
        "data": {"type": "version", "id": "998877"}
      }
      [ ... ]
    

    While this is not a mandatory part of the specification, it has an interesting usage for fetching relationship targets. In contrast with the .../relationships/... endpoint, this one is expected to return plain resource objects (which attributes and relationships information in particular).

    GET /ticket/987654/done_in
    Accept: application/vnd.api+json
    
    {
      "links": {
        "self": "https://www.cubicweb.org/998877"
      },
      "data": {
        "type": "version",
        "id": "998877",
        "attributes": {
            "number": 4.2
        },
        "relationships": {
          "version_of": {
            "self": "https://www.cubicweb.org/998877/relationships/version_of",
            "data": { "type": "project", "id": "1095" }
          }
        }
      },
      "included": [{
        "type": "project",
        "id": "1095",
        "attributes": {
            "name": "CubicWeb"
        },
        "links": {
          "self": "https://www.cubicweb.org/project/cubicweb"
        }
      }]
    }
    

    Meta information

    The JSON API specification allows to include non-standard information using a so-called meta object. This can be found in various place of the document (top-level, resource objects or relationships object). Usages of this field is completely free (and optional). For instance, we could use this field to store the workflow state of a ticket:

    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
        },
        "meta": { "state": "open" }
    }
    

    Permissions

    Permissions are part of metadata to be exchanged during request/response cycles. As such, the best place to convey this information is probably within the headers. According to JSON API's FAQ, this is also the recommended way for a resource to advertise on supported actions.

    So for instance, response to a GET request could include Allow headers, indicating which request methods are allowed on the primary resource requested:

    GET /ticket/987654
    Allow: GET, PATCH, DELETE
    

    An HEAD request could also be used for querying allowed actions on links (such as relationships):

    HEAD /ticket/987654/relationships/comments
    Allow: POST
    

    This approach has the advantage of being standard HTTP, no particular knowledge of the permissions model is required and the response body is not cluttered with these metadata.

    Another possibility would be to rely use the meta member of JSON API data.

    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
        },
        "meta": {
          "permissions": ["read", "update"]
        }
      }
    }
    

    Clearly, this would minimize the amount client/server requests.

    More Hypermedia controls

    With the example implementation described above, it appears already possible to manipulate several aspects of the entity-relationship database following a CubicWeb schema: resources fetching, CRUD operations on entities, set/delete operations on relationships. All these "standard" operations are discoverable by the client simply because they are baked into the JSON API format: for instance, adding a target to some relationship is possible by POSTing to the corresponding relationship resource something that conforms to the schema.

    So, implicitly, this already gives us a fairly good level of Hypermedia control so that we're not so far from having a mature REST architecture according to the Richardson Maturity Model. But beyond these "standard" discoverable actions, the JSON API specification does not address yet Hypermedia controls in a generic manner (see this interesting discussion about extending the specification for this purpose).

    So the question is: would we want more? Or, in other words, do we need to define "actions" which would not map directly to a concept in the application model?

    In the case of a CubicWeb application, the most obvious example (that I could think of) of where such an "action" would be needed is workflow state handling. Roughly, workflows in CubicWeb are modeled through two entity types State and TrInfo (for "transition information"), the former being handled through the latter, and a relationship in_state between the workflowable entity type at stake and its current State. It does not appear so clearly how would one model this in terms of HTTP resource. (Arguably we wouldn't want to expose the complexity of Workflow/TrInfo/State data model to the client, nor can we simply expose this in_state relationship, as a client would not be able to simply change the state of a entity by updating the relation). So what would be a custom "action" to handle the state of a workflowable resource? Back in our tracker example, how would we advertise to the client the possibility to perform "open"/"close"/"reject" actions on a ticket resource? Open question...

    Request for comments

    In this post, I tried to give an overview of a possible usage of JSON API to build a Web API for CubicWeb. Several aspects were discussed from simple CRUD operations, to relationships handling or non-standard actions. In many cases, there are open questions for which I'd love to receive feedback from the community. Recalling that this topic is a central part of the experiment towards building a client-side user interface to CubicWeb, the more discussion it gets, the better!

    For those wanting to try and play themselves with the experiments, have a look at the code. This is a work-in-progress/experimental implementation, relying on Pyramid for content negotiation and route traversals.

    What's next? Maybe an alternative experiment relying on Hydra? Or an orthogonal one playing with the schema client-side?


  • Happy New Year CubicWeb !

    2016/01/25 by Nicolas Chauvat

    This CubicWeb blog that has been asleep for some months, whereas the development was active. Let me try to summarize the recent progress.

    https://upload.wikimedia.org/wikipedia/commons/thumb/f/f1/New_Year_Ornaments_%282%29.JPG/320px-New_Year_Ornaments_%282%29.JPG

    CubicWeb 3.21

    CubicWeb 3.21 was published in July 2015. The announce was sent to the mailing list and changes were listed in the documentation.

    The main goal of this release was to reduce the technical debt. The code was improved, but the changes were not directly visible to users.

    CubicWeb 3.22

    CubicWeb 3.22 was published in January 2016. A mail was sent to the mailing list and the documentation was updated with the list of changes.

    The main achievements of this release were the inclusion of a new procedure to massively import data when using a Postgresql backend, improvements of migrations and customization of generic JSON exports.

    Roadmap and bi-monthly meetings

    After the last-minute cancellation of the may 2015 roadmap meeting, we failed to reschedule in june, the summer arrived, then the busy-busy end of the year... and voilà, we are in 2016.

    During that time, Logilab has been working on massive data import, full-js user interfaces exchanging JSON with the CubicWeb back-end, 3D in the browser, switching CubicWeb to Python3, moving its own apps to Bootstrap, using CubicWeb-Pyramid in production and improving management/supervision, etc. We will be more than happy to discuss this with the rest of the (small but strong) CubicWeb community.

    So let's wish a happy new year to everyone and meet again in March for a new roadmap session !


  • Towards building a JavaScript user interface to CubicWeb

    2016/01/08 by Denis Laxalde

    This post is an introduction of a series of articles dealing with an on-going experiment on building a JavaScript user interface to CubicWeb, to ultimately replace the web component of the framework. The idea of this series is to present the main topics of the experiment, with open questions in order to eventually engage the community as much as possible. The other side of this is to experiment a blog driven development process, so getting feedback is the very point of it!

    As of today, three main topics have been identified:

    • the Web API to let the client and server communicate,
    • the issue of representing the application schema client-side, and,
    • the construction of components of the web interface (client-side).

    As part of the first topic, we'll probably rely on another experimental work about REST-fulness undertaken recently in pyramid-cubicweb (see this head for source code). Then, it appears quite clearly that we'll need sooner or later a representation of data on the client-side and that, quite obviously, the underlying format would be JSON. Apart from exchanging of entities (database) information, we already anticipate on the need for the HATEOAS part of REST. We already took some time to look at the existing possibilities. At a first glance, it seems that hydra is the most promising in term of capabilities. It's also built using semantic web technologies which definitely grants bonus point for CubicWeb. On the other hand, it seems a bit isolated and very experimental, while JSON API follows a more pragmatic approach (describe itself as an anti-bikeshedding tool) and appears to have more traction from various people. For this reason, we choose it for our first draft, but this topic seems so central in a new UI, and hard to hide as an implementation detail; that it definitely deserves more discussion. Other candidates could be Siren, HAL or Uber.

    Concerning the schema, it seems that there is consensus around JSON-Schema so we'll certainly give it a try.

    Finally, while there is nothing certain as of today we'll probably start on building components of the web interface using React, which is also getting quite popular these days. Beyond that choice, the first practical task in this topic will concern the primary view system. This task being neither too simple nor too complicated will hopefully result in a clearer overview of what the project will imply. Then, the question of edition will come up at some point. In this respect, perhaps it'll be a good time to put the UX question at a central place, in order to avoid design issues that we had in the past.

    Feedback welcome!