subscribe to this blog

CubicWeb Blog

News about the framework and its uses.

show 133 results
  • Hypermedia API with cubicweb-jsonschema

    2017/04/04 by Denis Laxalde

    This is the second post of a series about cubicweb-jsonschema. The first post mainly dealt with JSON Schema representations of CubicWeb entities along with a brief description of the JSON API. In this second post, I'll describe another aspect of the project that aims at building an hypermedia API by leveraging the JSON Hyper Schema specification.

    Hypermedia APIs and JSON Hyper Schema

    Hypermedia API is somehow a synonymous of RESTful API but it makes it clearer that the API serves hypermedia responses, i.e. content that helps discoverability of other resources.

    At the heart of an hypermedia API is the concept of link relation which both aims at describing relationships between resources as well as provinding ways to manipulate them.

    In JSON Hyper Schema terminology, link relations take the form of a collection of Link Description Objects gathered into a links property of a JSON Schema document. These Link Description Objects thus describes relationships between the instance described by the JSON Schema document at stake and other resources; they hold a number of properties that makes relationships manipulation possible:

    • rel is the name of the relation, it is usually one of relation names registered at IANA;
    • href indicates the URI of the target of the relation, it may be templated by a JSON Schema;
    • targetSchema is a JSON Schema document (or reference) describing the target of the link relation;
    • schema (recently renamed as submissionSchema) is a JSON Schema document (or reference) describing what the target of the link expects when submitting data.

    Hypermedia walkthrough

    In the remaining of the article, I'll walk through a navigation path that is made possible by hypermedia controls provided by cubicweb-jsonschema. I'll continue on the example application described in the first post of the series which schema consists of Book, Author and Topic entity types. In essence, this walkthrough is typical of what an intelligent client could do when exposed to the API, i.e. from any resource, discover other resources and navigate or manipulate them.

    This walkthrough assumes that, given any resource (i.e. something that has a URL like /book/1), the server would expose data at the main URL when the client asks for JSON through the Accept header and it would expose the JSON Schema of the resource at a schema view of the same URL (i.e. /book/1/schema). This assumption can be regarded as a kind of client/server coupling, which might go away in later implementation.

    Site root

    While client navigation could start from any resource, we start from the root resource and retrieve its schema:

    GET /schema
    Accept: application/schema+json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "links": [
            {
                "href": "/author/",
                "rel": "collection",
                "schema": {
                    "$ref": "/author/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/author/schema"
                },
                "title": "Authors"
            },
            {
                "href": "/book/",
                "rel": "collection",
                "schema": {
                    "$ref": "/book/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Books"
            },
            {
                "href": "/topic/",
                "rel": "collection",
                "schema": {
                    "$ref": "/topic/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/topic/schema"
                },
                "title": "Topics"
            }
        ]
    }
    

    So at root URL, our application serves a JSON Hyper Schema that only consists of links. It has no JSON Schema document, which is natural since there's usually no data bound to the root resource (think of it as empty rset in CubicWeb terminology).

    These links correspond to top-level entity types, i.e. those that would appear in the default startup page of a CubicWeb application. They all have "rel": "collection" relation name (this comes from RFC6573) as their target is a collection of entities. We also have schema and targetSchema properties.

    From collection to items

    Now that we have added a new book, let's step back and use our books link to retrieve data (verb GET):

    GET /book/
    Accept: application/json
    
    HTTP/1.1 200 OK
    Allow: GET, POST
    Content-Type: application/json
    
    [
        {
            "id": "859",
            "title": "L'homme qui rit"
        },
        {
            "id": "858",
            "title": "The Old Man and the Sea"
        },
    ]
    

    which, as always, needs to be completed by a JSON Schema:

    GET /book/schema
    Accept: application/schema+json
    
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "$ref": "#/definitions/Book_plural",
        "definitions": {
            "Book_plural": {
                "items": {
                    "properties": {
                        "id": {
                            "type": "string"
                        },
                        "title": {
                            "type": "string"
                        }
                    },
                    "type": "object"
                },
                "title": "Books",
                "type": "array"
            }
        },
        "links": [
            {
                "href": "/book/",
                "rel": "collection",
                "schema": {
                    "$ref": "/book/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Books"
            },
            {
                "href": "/book/{id}",
                "rel": "item",
                "targetSchema": {
                    "$ref": "/book/schema?role=view"
                },
                "title": "Book"
            }
        ]
    }
    

    Consider the last item of links in the above schema. It has a "rel": "item" property which indicates how to access items of the collection; its href property is a templated URI which can be expanded using instance data and schema (here we only have a single id template variable).

    So our client may navigate to the first item of the collection (id="859") at /book/859 URI, and retrieve resource data:

    GET /book/859
    Accept: application/json
    
    HTTP/1.1 200 OK
    Allow: GET, PUT, DELETE
    Content-Type: application/json
    
    {
        "author": [
            "Victor Hugo"
        ],
        "publication_date": "1869-04-01T00:00:00",
        "title": "L'homme qui rit"
    }
    

    and schema:

    GET /book/859/schema
    Accept: application/schema+json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "$ref": "#/definitions/Book",
        "definitions": {
            "Book": {
                "additionalProperties": false,
                "properties": {
                    "author": {
                        "items": {
                            "type": "string"
                        },
                        "title": "author",
                        "type": "array"
                    },
                    "publication_date": {
                        "format": "date-time",
                        "title": "publication date",
                        "type": "string"
                    },
                    "title": {
                        "title": "title",
                        "type": "string"
                    },
                    "topics": {
                        "items": {
                            "type": "string"
                        },
                        "title": "topics",
                        "type": "array"
                    }
                },
                "title": "Book",
                "type": "object"
            }
        },
        "links": [
            {
                "href": "/book/",
                "rel": "up",
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Book_plural"
            },
            {
                "href": "/book/859/",
                "rel": "self",
                "schema": {
                    "$ref": "/book/859/schema?role=edition"
                },
                "targetSchema": {
                    "$ref": "/book/859/schema?role=view"
                },
                "title": "Book #859"
            }
        ]
    }
    

    Entity resource

    The resource obtained above as an item of a collection is actually an entity. Notice the rel="self" link. It indicates how to manipulate the current resource (i.e. at which URI, using a given schema depending on what actions we want to perform). Still this link does not indicate what actions may be performed. This indication is found in the Allow header of the data response above:

    Allow: GET, PUT, DELETE
    

    With these information bits, our intelligent client is able to, for instance, form a request to delete the resource. On the other hand, the action to update the resource (which is allowed because of the presence of PUT in Allow header, per HTTP semantics) would take the form of a request which body conforms to the JSON Schema pointed at by the schema property of the link.

    Also note the rel="up" link which makes it possible to navigate to the collection of books.

    Conclusions

    This post introduced the main hypermedia capabilities of cubicweb-jsonschema, built on top of the JSON Hyper Schema specification. The resulting Hypermedia API makes it possible for an intelligent client to navigate through hypermedia resources and manipulate them by using both link relation semantics and HTTP verbs.

    In the next post, I'll deal with relationships description and manipulation both in terms of API (endpoints) and hypermedia representation.


  • Introducing cubicweb-jsonschema

    2017/03/23 by Denis Laxalde

    This is the first post of a series introducing the cubicweb-jsonschema project that is currently under development at Logilab. In this post, I'll first introduce the general goals of the project and then present in more details two aspects about data models (the connection between Yams and JSON schema in particular) and the basic features of the API. This post does not always present how things work in the current implementation but rather how they should.

    Goals of cubicweb-jsonschema

    From a high level point of view, cubicweb-jsonschema addresses mainly two interconnected aspects. One related to modelling for client-side development of user interfaces to CubicWeb applications while the other one concerns the HTTP API.

    As far as modelling is concerned, cubicweb-jsonschema essentially aims at providing a transformation mechanism between a Yams schema and JSON Schema that is both automatic and extensible. This means that we can ultimately expect that Yams definitions alone would sufficient to have generated JSON schema definitions that would consistent enough to build an UI, pretty much as it is currently with the automatic web UI in CubicWeb. A corollary of this goal is that we want JSON schema definitions to match their context of usage, meaning that a JSON schema definition would not be the same in the context of viewing, editing or relationships manipulations.

    In terms of API, cubicweb-jsonschema essentially aims at providing an HTTP API to manipulate entities based on their JSON Schema definitions.

    Finally, the ultimate goal is to expose an hypermedia API for a CubicWeb application in order to be able to ultimately build an intelligent client. For this we'll build upon the JSON Hyper-Schema specification. This aspect will be discussed in a later post.

    Basic usage as an HTTP API library

    Consider a simple case where one wants to manipulate entities of type Author described by the following Yams schema definition:

    class Author(EntityType):
        name = String(required=True)
    

    With cubicweb-jsonschema one can get JSON Schema for this entity type in at different contexts such: view, creation or edition. For instance:

    • in a view context, the JSON Schema will be:

      {
          "$ref": "#/definitions/Author",
          "definitions": {
              "Author": {
                  "additionalProperties": false,
                  "properties": {
                      "name": {
                          "title": "name",
                          "type": "string"
                      }
                  },
                  "title": "Author",
                  "type": "object"
              }
          }
      }
      
    • whereas in creation context, it'll be:

      {
          "$ref": "#/definitions/Author",
          "definitions": {
              "Author": {
                  "additionalProperties": false,
                  "properties": {
                      "name": {
                          "title": "name",
                          "type": "string"
                      }
                  },
                  "required": [
                      "name"
                  ],
                  "title": "Author",
                  "type": "object"
              }
          }
      }
      

      (notice, the required keyword listing name property).

    Such JSON Schema definitions are automatically generated from Yams definitions. In addition, cubicweb-jsonschema exposes some endpoints for basic CRUD operations on resources through an HTTP (JSON) API. From the client point of view, requests on these endpoints are of course expected to match JSON Schema definitions. Some examples:

    Get an author resource:

    GET /author/855
    Accept:application/json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    {"name": "Ernest Hemingway"}
    

    Update an author:

    PATCH /author/855
    Accept:application/json
    Content-Type: application/json
    {"name": "Ernest Miller Hemingway"}
    
    HTTP/1.1 200 OK
    Location: /author/855/
    Content-Type: application/json
    {"name": "Ernest Miller Hemingway"}
    

    Create an author:

    POST /author
    Accept:application/json
    Content-Type: application/json
    {"name": "Victor Hugo"}
    
    HTTP/1.1 201 Created
    Content-Type: application/json
    Location: /Author/858
    {"name": "Victor Hugo"}
    

    Delete an author:

    DELETE /author/858
    
    HTTP/1.1 204 No Content
    

    Now if the client sends invalid input with respect to the schema, they'll get an error:

    (We provide a wrong born property in request body.)

    PATCH /author/855
    Accept:application/json
    Content-Type: application/json
    {"born": "1899-07-21"}
    
    HTTP/1.1 400 Bad Request
    Content-Type: application/json
    
    {
        "errors": [
            {
                "details": "Additional properties are not allowed ('born' was unexpected)",
                "status": 422
            }
        ]
    }
    

    From Yams model to JSON Schema definitions

    The example above illustrates automatic generation of JSON Schema documents based on Yams schema definitions. These documents are expected to help developping views and forms for a web client. Clearly, we expect that cubicweb-jsonschema serves JSON Schema documents for viewing and editing entities as cubicweb.web serves HTML documents for the same purposes. The underlying logic for JSON Schema generation is currently heavily inspired by the logic of primary view and automatic entity form as they exists in cubicweb.web.views. That is: the Yams schema is introspected to determine how properties should be generated and any additionnal control over this can be performed through uicfg declarations [1].

    To illustrate let's consider the following schema definitions which:

    class Book(EntityType):
        title = String(required=True)
        publication_date = Datetime(required=True)
    
    class Illustration(EntityType):
        data = Bytes(required=True)
    
    class illustrates(RelationDefinition):
        subject = 'Illustration'
        object = 'Book'
        cardinality = '1*'
        composite = 'object'
        inlined = True
    
    class Author(EntityType):
        name = String(required=True)
    
    class author(RelationDefinition):
        subject = 'Book'
        object = 'Author'
        cardinality = '1*'
    
    class Topic(EntityType):
        name = String(required=True)
    
    class topics(RelationDefinition):
        subject = 'Book'
        object = 'Topic'
        cardinality = '**'
    

    and consider, as before, JSON Schema documents in different contexts for the the Book entity type:

    • in view context:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "type": "string"
                          },
                          "title": "author",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      },
                      "topics": {
                          "items": {
                              "type": "string"
                          },
                          "title": "topics",
                          "type": "array"
                      }
                  },
                  "title": "Book",
                  "type": "object"
              }
          }
      }
      

      We have a single Book definition in this document, in which we find attributes defined in the Yams schema (title and publication_date). We also find the two relations where Book is involved: topics and author, both appearing as a single array of "string" items. The author relationship appears like that because it is mandatory but not composite. On the other hand, the topics relationship has the following uicfg rule:

      uicfg.primaryview_section.tag_subject_of(('Book', 'topics', '*'), 'attributes')
      

      so that it's definition appears embedded in the document of Book definition.

      A typical JSON representation of a Book entity would be:

      {
          "author": [
              "Ernest Miller Hemingway"
          ],
          "title": "The Old Man and the Sea",
          "topics": [
              "sword fish",
              "cuba"
          ]
      }
      
    • in creation context:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "oneOf": [
                                  {
                                      "enum": [
                                          "855"
                                      ],
                                      "title": "Ernest Miller Hemingway"
                                  },
                                  {
                                      "enum": [
                                          "857"
                                      ],
                                      "title": "Victor Hugo"
                                  }
                              ],
                              "type": "string"
                          },
                          "maxItems": 1,
                          "minItems": 1,
                          "title": "author",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      }
                  },
                  "required": [
                      "title",
                      "publication_date"
                  ],
                  "title": "Book",
                  "type": "object"
              }
          }
      }
      

      notice the differences, we now only have attributes and required relationships (author) in this schema and we have the required listing mandatory attributes; the author property is represented as an array which items consist of pre-existing objects of the author relationship (namely Author entities).

      Now assume we add the following uicfg declaration:

      uicfg.autoform_section.tag_object_of(('*', 'illustrates', 'Book'), 'main', 'inlined')
      

      the JSON Schema for creation context will be:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "oneOf": [
                                  {
                                      "enum": [
                                          "855"
                                      ],
                                      "title": "Ernest Miller Hemingway"
                                  },
                                  {
                                      "enum": [
                                          "857"
                                      ],
                                      "title": "Victor Hugo"
                                  }
                              ],
                              "type": "string"
                          },
                          "maxItems": 1,
                          "minItems": 1,
                          "title": "author",
                          "type": "array"
                      },
                      "illustrates": {
                          "items": {
                              "$ref": "#/definitions/Illustration"
                          },
                          "title": "illustrates_object",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      }
                  },
                  "required": [
                      "title",
                      "publication_date"
                  ],
                  "title": "Book",
                  "type": "object"
              },
              "Illustration": {
                  "additionalProperties": false,
                  "properties": {
                      "data": {
                          "format": "data-url",
                          "title": "data",
                          "type": "string"
                      }
                  },
                  "required": [
                      "data"
                  ],
                  "title": "Illustration",
                  "type": "object"
              }
          }
      }
      

      We now have an additional illustrates property modelled as an array of #/definitions/Illustration, the later also added the the document as an additional definition entry.

    Conclusion

    This post illustrated how a basic (CRUD) HTTP API based on JSON Schema could be build for a CubicWeb application using cubicweb-jsonschema. We have seen a couple of details on JSON Schema generation and how it can be controlled. Feel free to comment and provide feedback on this feature set as well as open the discussion with more use cases.

    Next time, we'll discuss how hypermedia controls can be added the HTTP API that cubicweb-jsonschema provides.

    [1]this choice is essentially driven by simplicity and conformance when the existing behavior to help migration of existing applications.

  • Monitor all the things! ... and early too!

    2016/09/16 by Arthur Lutz

    Following the "release often, release early" mantra, I thought it might be a good idea to apply it to monitoring on one of our client projects. So right from the demo stage where we deliver a new version every few weeks (and sometimes every few days), we setup some monitoring.

    https://www.cubicweb.org/file/15338085/raw/66511658.jpg

    Monitoring performance

    The project is an application built with the CubicWeb platform, with some ElasticSearch for indexing and searching. As with any complex stack, there are a great number of places where one could monitor performance metrics.

    https://www.cubicweb.org/file/15338628/raw/Screenshot_2016-09-16_12-19-21.png

    Here are a few things we have decided to monitor, and with what tools.

    Monitoring CubicWeb

    To monitor our running Python code, we have decided to use statsd, since it is already built into CubicWeb's core. Out of the box, you can configure a statsd server address in your all-in-one.conf configuration. That will send out some timing statistics about some core functions.

    The statsd server (there a numerous implementations, we use a simple one : python-pystatsd) gets the raw metrics and outputs them to carbon which stores the time series data in whisper files (which can be swapped out for a different technology if need be).

    https://www.cubicweb.org/file/15338392/raw/Screenshot_2016-09-16_11-56-44.png

    If we are curious about a particular function or view that might be taking too long to generate or slow down the user experience, we can just add the @statsd_timeit decorator there. Done. It's monitored.

    statsd monitoring is a fire-and-forget UDP type of monitoring, it should not have any impact on the performance of what you are monitoring.

    Monitoring Apache

    Simply enough we re-use the statsd approach by plugging in an apache module to time the HTTP responses sent back by apache. With nginx and varnish, this is also really easy.

    https://www.cubicweb.org/file/15338407/raw/Screenshot_2016-09-16_11-56-54.png

    One of the nice things about this part is that we can then get graphs of errors since we will differentiate OK 200 type codes from 500 type codes (HTTP codes).

    Monitoring ElasticSearch

    ElasticSearch comes with some metrics in GET /_stats endpoint, the same goes for individual nodes, individual indices and even at cluster level. Some popular tools can be installed through the ElasticSearch plugin system or with Kibana (plugin system there too).

    We decided on a different approach that fitted well with our other tools (and demonstrates their flexibility!) : pull stats out of ElasticSearch with SaltStack, push them to Carbon, pull them out with Graphite and display them in Grafana (next to our other metrics).

    https://www.cubicweb.org/file/15338399/raw/Screenshot_2016-09-16_11-56-34.png

    On the SaltStack side, we wrote a two line execution module (elasticsearch.py)

    import requests
    def stats():
        return request.get('http://localhost:9200/_stats').json()
    

    This gets shipped using the custom execution modules mechanism (_modules and saltutils.sync_modules), and is executed every minute (or less) in the salt scheduler. The resulting dictionary is fed to the carbon returner that is configured to talk to a carbon server somewhere nearby.

    # salt demohost elasticsearch.stats
    [snip]
      { "indextime_inmillis" : 30,
    [snip]
    

    Monitoring web metrics

    To evaluate parts of the performance of a web page we can look at some metrics such as the number of assets the browser will need to download, the size of the assets (js, css, images, etc) and even things such as the number of subdomains used to deliver assets. You can take a look at such metrics in most developer tools available in the browser, but we want to graph this over time. A nice tool for this is sitespeed.io (written in javascript with phantomjs). Out of the box, it has a graphite outputter so we just have to add --graphiteHost FQDN. sitespeed.io even recommends using grafana to visualize the results and publishes some example dashboards that can be adapted to your needs.

    https://www.cubicweb.org/file/15338109/raw/sitespeed-logo-2c.png

    The sitespeed.io command is configured and run by salt using pillars and its scheduler.

    We will have to take a look at using their jenkins plugin with our jenkins continuous integration instance.

    Monitoring crashes / errors / bugs

    Applications will have bugs (in particular when released often to get a client to validate some design choices early). Level 0 is having your client calling you up saying the application has crashed. The next level is watching some log somewhere to see those errors pop up. The next level is centralised logs on which you can monitor the numerous pieces of your application (rsyslog over UDP helps here, graylog might be a good solution for visualisation).

    https://www.cubicweb.org/file/15338139/raw/Screenshot_2016-09-16_11-30-53.png

    When it starts getting useful and usable is when your bugs get reported with some rich context. That's when using sentry gets in. It's free software developed on github (although the website does not really show that) and it is written in python, so it was a good match for our culture. And it is pretty awesome too.

    We plug sentry into our WSGI pipeline (thanks to cubicweb-pyramid) by installing and configuring the sentry cube : cubicweb-sentry. This will catch rich context bugs and provide us with vital information about what the user was doing when the crash occured.

    This also helps sharing bug information within a team.

    The sentry cube reports on errors being raised when using the web application, but can also catch some errors when running some maintenance or import commands (ccplugins in CubicWeb). In this particular case, a lot of importing is being done and Sentry can detect and help us triage the import errors with context on which files are failing.

    Monitoring usage / client side

    This part is a bit neglected for the moment. Client side we can use Javascript to monitor usage. Some basic metrics can come from piwik which is usually used for audience statistics. To get more precise statistics we've been told Boomerang has an interesting approach, enabling a closer look at how fast a page was displayed client side, how much time was spend on DNS, etc.

    On the client side, we're also looking at two features of the Sentry project : the raven-js client which reports Javascript errors directly from the browser to the Sentry server, and the user feedback form which captures some context when something goes wrong or a user/client wants to report that something should be changed on a given page.

    Load testing - coverage

    To wrap up, we also often generate traffic to catch some bugs and performance metrics automatically :

    • wget --mirror $URL
    • linkchecker $URL
    • for $search_term in cat corpus; do wget URL/$search_term ; done
    • wapiti $URL --scope page
    • nikto $URL

    Then watch the graphs and the errors in Sentry... Fix them. Restart.

    Graphing it in Grafana

    We've spend little time on the dashboard yet since we're concentrating on collecting the metrics for now. But here is a glimpse of the "work in progress" dashboard which combines various data sources and various metrics on the same screen and the same time scale.

    https://www.cubicweb.org/file/15338648/raw/Screenshot_2016-09-13_09-41-45.png

    Further plans

    • internal health checks, we're taking a look at python-hospital and healthz: Stop reverse engineering applications and start monitoring from the inside (Monitorama) (the idea is to distinguish between the app is running and the app is serving it's purpose), and pyramid_health
    • graph the number of Sentry errors and the number of types of errors: the sentry API should be able to give us this information. Feed it to Salt and Carbon.
    • setup some alerting : next versions of Grafana will be doing that, or with elastalert
    • setup "release version X" events in Graphite that are displayed in Grafana, maybe with some manual command or a postcreate command when using docker-compose up ?
    • make it easier for devs to have this kind of setup. Using this suite of tools in developement might sometimes be overkill, but can be useful.

  • Status of the CubicWeb python3 porting effort, February 2016

    2016/02/05 by Julien Cristau

    An effort to port CubicWeb to a dual python 2.6/2.7 and 3.3+ code base was started by Rémi Cardona in summer of 2014. The first task was to port all of CubicWeb's dependencies:

    • logilab-common 0.63
    • logilab-database 1.14
    • logilab-mtconverter 0.9
    • logilab-constraint 0.6
    • yams 0.40
    • rql 0.34

    Once that was out of the way, we could start looking at CubicWeb itself. We first set out to make sure we used python3-compatible syntax in all source files, then started to go and make as much of the test suite as possible pass under both python2.7 and python3.4. As of the 3.22 release, we are almost there. The remaining pain points are:

    • cubicweb's setup.py hadn't been converted. This is fixed in the 3.23 branch as of https://hg.logilab.org/master/cubicweb/rev/0b59724cb3f2 (don't follow that link, the commit is huge)
    • the CubicWebServerTC test class uses twisted to start an http server thread, and twisted itself is not available for python3
    • the current method to serialize schema constraints into CWConstraint objects gives different results on python2 and python3, so it needs to be fixed (https://www.logilab.org/ticket/296748)
    • various questions around packaging and deployment: what happens to e.g. the cubicweb-common package installing into python2's site-packages directory? What does the ${prefix}/share/cubicweb directory become? How do cubes express their dependencies? Do we need a flag day? What does that mean for applications?

  • Using JSONAPI as a Web API format for CubicWeb

    2016/01/26 by Denis Laxalde

    Following the introduction post about rethinking the web user interface of CubicWeb, this article will address the topic of the Web API to exchange data between the client and the server. As mentioned earlier, this question is somehow central and deserves particular interest, and better early than late. Of the two candidate representations previously identified Hydra and JSON API, this article will focus on the later. Hopefully, this will give a better insight of the capabilities and limits of this specification and would help take a decision, though a similar experiment with another candidate would be good to have. Still in the process of blog driven development, this post has several open questions from which a discussion would hopefully emerge...

    A glance at JSON API

    JSON API is a specification for building APIs that use JSON as a data exchange format between clients and a server. The media type is application/vnd.api+json. It has a 1.0 version available from mid-2015. The format has interesting features such as the ability to build compound documents (i.e. response made of several, usually related, resources) or to specify filtering, sorting and pagination.

    A document following the JSON API format basically represents resource objects, their attributes and relationships as well as some links also related to the data of primary concern.

    Taking the example of a Ticket resource modeled after the tracker cube, we could have a JSON API document formatted as:

    GET /ticket/987654
    Accept: application/vnd.api+json
    
    {
      "links": {
        "self": "https://www.cubicweb.org/ticket/987654"
      },
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
          "description": "Well, let's try, at least...",
        },
        "relationships": {
          "concerns": {
            "links": {
              "self": "https://www.cubicweb.org/ticket/987654/relationships/concerns",
              "related": "https://www.cubicweb.org/ticket/987654/concerns"
            },
            "data": {"type": "project", "id": "1095"}
          },
          "done_in": {
            "links": {
              "self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
              "related": "https://www.cubicweb.org/ticket/987654/done_in"
            },
            "data": {"type": "version", "id": "998877"}
          }
        }
      },
      "included": [{
        "type": "project",
        "id": "1095",
        "attributes": {
            "name": "CubicWeb"
        },
        "links": {
          "self": "https://www.cubicweb.org/project/cubicweb"
        }
      }]
    }
    

    In this JSON API document, top-level members are links, data and included. The later is here used to ship some resources (here a "project") related to the "primary data" (a "ticket") through the "concerns" relationship as denoted in the relationships object (more on this later).

    While the decision of including or not these related resources along with the primary data is left to the API designer, JSON API also offers a specification to build queries for inclusion of related resources. For example:

    GET /ticket/987654?include=done_in
    Accept: application/vnd.api+json
    

    would lead to a response including the full version resource along with the above content.

    Enough for the JSON API overview. Next I'll present how various aspects of data fetching and modification can be achieved through the use of JSON API in the context of a CubicWeb application.

    CRUD

    CRUD of resources is handled in a fairly standard way in JSON API, relying of HTTP protocol semantics.

    For instance, creating a ticket could be done as:

    POST /ticket
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "ticket",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
          "description": "Well, let's try, at least...",
        },
        "relationships": {
          "concerns": {
            "data": { "type": "project", "id": "1095" }
          }
        }
      }
    }
    

    Then updating it (assuming we got its id from a response to the above request):

    PATCH /ticket/987654
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "description": "We'll succeed, for sure!",
        },
      }
    }
    

    Relationships

    In JSON API, a relationship is in fact a first class resource as it is defined by a noun and an URI through a link object. In this respect, the client just receives a couple of links and can eventually operate on them using the proper HTTP verb. Fetching or updating relationships is done using the special <resource url>/relationships/<relation type> endpoint (self member of relationships items in the first example). Quite naturally, the specification relies on GET verb for fetching targets, PATCH for (re)setting a relation (i.e. replacing its targets), POST for adding targets and DELETE to drop them.

    GET /ticket/987654/relationships/concerns
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "project",
        "id": "1095"
      }
    }
    
    PATCH /ticket/987654/relationships/done_in
    Content-Type: application/vnd.api+json
    Accept: application/vnd.api+json
    
    {
      "data": {
        "type": "version",
        "id": "998877"
      }
    }
    

    The body of request and response of this <resource url>/relationships/<relation type> endpoint consists of so-called resource identifier objects which are lightweight representation of resources usually only containing information about their "type" and "id" (enough to uniquely identify them).

    Related resources

    Remember the related member appearing in relationships links in the first example?

      [ ... ]
      "done_in": {
        "links": {
          "self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
          "related": "https://www.cubicweb.org/ticket/987654/done_in"
        },
        "data": {"type": "version", "id": "998877"}
      }
      [ ... ]
    

    While this is not a mandatory part of the specification, it has an interesting usage for fetching relationship targets. In contrast with the .../relationships/... endpoint, this one is expected to return plain resource objects (which attributes and relationships information in particular).

    GET /ticket/987654/done_in
    Accept: application/vnd.api+json
    
    {
      "links": {
        "self": "https://www.cubicweb.org/998877"
      },
      "data": {
        "type": "version",
        "id": "998877",
        "attributes": {
            "number": 4.2
        },
        "relationships": {
          "version_of": {
            "self": "https://www.cubicweb.org/998877/relationships/version_of",
            "data": { "type": "project", "id": "1095" }
          }
        }
      },
      "included": [{
        "type": "project",
        "id": "1095",
        "attributes": {
            "name": "CubicWeb"
        },
        "links": {
          "self": "https://www.cubicweb.org/project/cubicweb"
        }
      }]
    }
    

    Meta information

    The JSON API specification allows to include non-standard information using a so-called meta object. This can be found in various place of the document (top-level, resource objects or relationships object). Usages of this field is completely free (and optional). For instance, we could use this field to store the workflow state of a ticket:

    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
        },
        "meta": { "state": "open" }
    }
    

    Permissions

    Permissions are part of metadata to be exchanged during request/response cycles. As such, the best place to convey this information is probably within the headers. According to JSON API's FAQ, this is also the recommended way for a resource to advertise on supported actions.

    So for instance, response to a GET request could include Allow headers, indicating which request methods are allowed on the primary resource requested:

    GET /ticket/987654
    Allow: GET, PATCH, DELETE
    

    An HEAD request could also be used for querying allowed actions on links (such as relationships):

    HEAD /ticket/987654/relationships/comments
    Allow: POST
    

    This approach has the advantage of being standard HTTP, no particular knowledge of the permissions model is required and the response body is not cluttered with these metadata.

    Another possibility would be to rely use the meta member of JSON API data.

    {
      "data": {
        "type": "ticket",
        "id": "987654",
        "attributes": {
          "title": "Let's use JSON API in CubicWeb"
        },
        "meta": {
          "permissions": ["read", "update"]
        }
      }
    }
    

    Clearly, this would minimize the amount client/server requests.

    More Hypermedia controls

    With the example implementation described above, it appears already possible to manipulate several aspects of the entity-relationship database following a CubicWeb schema: resources fetching, CRUD operations on entities, set/delete operations on relationships. All these "standard" operations are discoverable by the client simply because they are baked into the JSON API format: for instance, adding a target to some relationship is possible by POSTing to the corresponding relationship resource something that conforms to the schema.

    So, implicitly, this already gives us a fairly good level of Hypermedia control so that we're not so far from having a mature REST architecture according to the Richardson Maturity Model. But beyond these "standard" discoverable actions, the JSON API specification does not address yet Hypermedia controls in a generic manner (see this interesting discussion about extending the specification for this purpose).

    So the question is: would we want more? Or, in other words, do we need to define "actions" which would not map directly to a concept in the application model?

    In the case of a CubicWeb application, the most obvious example (that I could think of) of where such an "action" would be needed is workflow state handling. Roughly, workflows in CubicWeb are modeled through two entity types State and TrInfo (for "transition information"), the former being handled through the latter, and a relationship in_state between the workflowable entity type at stake and its current State. It does not appear so clearly how would one model this in terms of HTTP resource. (Arguably we wouldn't want to expose the complexity of Workflow/TrInfo/State data model to the client, nor can we simply expose this in_state relationship, as a client would not be able to simply change the state of a entity by updating the relation). So what would be a custom "action" to handle the state of a workflowable resource? Back in our tracker example, how would we advertise to the client the possibility to perform "open"/"close"/"reject" actions on a ticket resource? Open question...

    Request for comments

    In this post, I tried to give an overview of a possible usage of JSON API to build a Web API for CubicWeb. Several aspects were discussed from simple CRUD operations, to relationships handling or non-standard actions. In many cases, there are open questions for which I'd love to receive feedback from the community. Recalling that this topic is a central part of the experiment towards building a client-side user interface to CubicWeb, the more discussion it gets, the better!

    For those wanting to try and play themselves with the experiments, have a look at the code. This is a work-in-progress/experimental implementation, relying on Pyramid for content negotiation and route traversals.

    What's next? Maybe an alternative experiment relying on Hydra? Or an orthogonal one playing with the schema client-side?


  • Happy New Year CubicWeb !

    2016/01/25 by Nicolas Chauvat

    This CubicWeb blog that has been asleep for some months, whereas the development was active. Let me try to summarize the recent progress.

    https://upload.wikimedia.org/wikipedia/commons/thumb/f/f1/New_Year_Ornaments_%282%29.JPG/320px-New_Year_Ornaments_%282%29.JPG

    CubicWeb 3.21

    CubicWeb 3.21 was published in July 2015. The announce was sent to the mailing list and changes were listed in the documentation.

    The main goal of this release was to reduce the technical debt. The code was improved, but the changes were not directly visible to users.

    CubicWeb 3.22

    CubicWeb 3.22 was published in January 2016. A mail was sent to the mailing list and the documentation was updated with the list of changes.

    The main achievements of this release were the inclusion of a new procedure to massively import data when using a Postgresql backend, improvements of migrations and customization of generic JSON exports.

    Roadmap and bi-monthly meetings

    After the last-minute cancellation of the may 2015 roadmap meeting, we failed to reschedule in june, the summer arrived, then the busy-busy end of the year... and voilà, we are in 2016.

    During that time, Logilab has been working on massive data import, full-js user interfaces exchanging JSON with the CubicWeb back-end, 3D in the browser, switching CubicWeb to Python3, moving its own apps to Bootstrap, using CubicWeb-Pyramid in production and improving management/supervision, etc. We will be more than happy to discuss this with the rest of the (small but strong) CubicWeb community.

    So let's wish a happy new year to everyone and meet again in March for a new roadmap session !


  • Towards building a JavaScript user interface to CubicWeb

    2016/01/08 by Denis Laxalde

    This post is an introduction of a series of articles dealing with an on-going experiment on building a JavaScript user interface to CubicWeb, to ultimately replace the web component of the framework. The idea of this series is to present the main topics of the experiment, with open questions in order to eventually engage the community as much as possible. The other side of this is to experiment a blog driven development process, so getting feedback is the very point of it!

    As of today, three main topics have been identified:

    • the Web API to let the client and server communicate,
    • the issue of representing the application schema client-side, and,
    • the construction of components of the web interface (client-side).

    As part of the first topic, we'll probably rely on another experimental work about REST-fulness undertaken recently in pyramid-cubicweb (see this head for source code). Then, it appears quite clearly that we'll need sooner or later a representation of data on the client-side and that, quite obviously, the underlying format would be JSON. Apart from exchanging of entities (database) information, we already anticipate on the need for the HATEOAS part of REST. We already took some time to look at the existing possibilities. At a first glance, it seems that hydra is the most promising in term of capabilities. It's also built using semantic web technologies which definitely grants bonus point for CubicWeb. On the other hand, it seems a bit isolated and very experimental, while JSON API follows a more pragmatic approach (describe itself as an anti-bikeshedding tool) and appears to have more traction from various people. For this reason, we choose it for our first draft, but this topic seems so central in a new UI, and hard to hide as an implementation detail; that it definitely deserves more discussion. Other candidates could be Siren, HAL or Uber.

    Concerning the schema, it seems that there is consensus around JSON-Schema so we'll certainly give it a try.

    Finally, while there is nothing certain as of today we'll probably start on building components of the web interface using React, which is also getting quite popular these days. Beyond that choice, the first practical task in this topic will concern the primary view system. This task being neither too simple nor too complicated will hopefully result in a clearer overview of what the project will imply. Then, the question of edition will come up at some point. In this respect, perhaps it'll be a good time to put the UX question at a central place, in order to avoid design issues that we had in the past.

    Feedback welcome!


  • Serving Cubicweb via WSGI with Pyramid: comparing the options

    2015/04/21 by David Douard

    CubicWeb can now be powered by Pyramid (thank you so much Christophe) instead of Twisted.

    I aim at moving all our applications to CubicWeb/Pyramid, so I wonder what will be the best way to deliver them. For now, we have a setup made of Apache + Varnish + Cubicweb/Twisted. In some applications we have two CubicWeb instances with a naive load balacing managed by Varnish.

    When moving to cubicweb-pyramid, there are several options. By default, a cubicweb-pyramid instance started via the cubicweb-ctl pyramid command, is running a waitress wsgi http server. I read it is common to deliver wsgi applications with nginx + uwsgi, but I wanted to play with mongrel2 (that I already tested with Cubicweb a while ago), and give a try to the circus + chaussette stack.

    I ran my tests :

    • using ab the simple Apache benchmark tool (aka ApacheBench) ;
    • on a clone of our logilab.org forge ;
    • on my laptop (Intel Core i7, 2.67GHz, quad core, 8Go),
    • using a postgresql 9.1 database server.

    Setup

    In order to be able to start the application as a wsgi app, a small python script is required. I extracted a small part of the cubicweb-pyramid ccplugin.py file into a elo.py file for this:

    appid = 'elo2'
    
    cwconfig = cwcfg.config_for(appid)
    application = wsgi_application_from_cwconfig(cwconfig)
    repo = cwconfig.repository()
    repo.start_looping_tasks()
    

    I tested 5 configurations: twisted, pyramid, mongrel2+wsgid, uwsgi and circus+chaussette. When possible, they were tested with 1 worker and 4 workers.

    Legacy Twisted mode

    Using good old legacy twisted setup:

    cubicwebctl start -D -l info elo
    

    The config setting that worth noting are:

    webserver-threadpool-size=6
    connections-pool-size=6
    

    Basic Pyramid mode

    Using the pyramid command that uses waitress:

    cubicwebctl pyramid --no-daemon -l info elo
    

    Mongrel2 + wsgid

    I have not been able to use uwsgi-mongrel2 as wsgi backend for mongrel2, since this uwsgi plugin is not provided by the uwsgi debian packages. I've used wsgid instead (sadly, the project appears to be dead).

    The mongrel config is:

    main = Server(
       uuid="f400bf85-4538-4f7a-8908-67e313d515c2",
       access_log="/logs/access.log",
       error_log="/logs/error.log",
       chroot="./",
       default_host="localhost",
       name="test",
       pid_file="/pid/mongrel2.pid",
       bind_addr="0.0.0.0",
       port=8083,
       hosts = [
           Host(name="localhost",
                routes={'/': Handler(send_spec='tcp://127.0.0.1:5000',
                                     send_ident='2113523d-f5ff-4571-b8da-8bddd3587475',
                                     recv_spec='tcp://127.0.0.1:5001',
                                     recv_ident='')
                       })
               ]
       )
    
    servers = [main]
    

    and the wsgid server is started with:

    wsgid --recv tcp://127.0.0.1:5000 --send tcp://127.0.0.1:5001 --keep-alive \
    --workers <N> --wsgi-app elo.application --app-path .
    

    uwsgi

    The config file used to start uwsgi is:

    [uwsgi]
    stats = 127.0.0.1:9191
    processes = <N>
    wsgi-file = elo.py
    http = :8085
    plugin = http,python
    virtualenv = /home/david/hg/grshells/venv/jpl
    enable-threads = true
    lazy-apps = true
    

    The tricky config option there is lazy-apps which must be set, otherwise the worker processes are forked after loading the cubicweb application, which this later does not support. If you omit this, only one worker will get the requests.

    circus + chaussette

    For the circus setup, I have used this configuration file:

    [circus]
    check_delay = 5
    endpoint = tcp://127.0.0.1:5555
    pubsub_endpoint = tcp://127.0.0.1:5556
    stats_endpoint = tcp://127.0.0.1:5557
    statsd = True
    httpd = True
    httpd_host = localhost
    httpd_port = 8086
    
    [watcher:webworker]
    cmd = /home/david/hg/grshells/venv/jpl/bin/chaussette --fd $(circus.sockets.webapp) elo2.app
    use_sockets = True
    numprocesses = 4
    
    [env:webworker]
    PATH=/home/david/hg/grshells/venv/jpl/bin:/usr/local/bin:/usr/bin:/bin
    CW_INSTANCES_DIR=/home/david/hg/grshells/grshell-jpl/etc
    PYTHONPATH=/home/david/hg/grshells//grshell-jpl
    
    [socket:webapp]
    host = 127.0.0.1
    port = 8085
    

    Results

    The bench are very simple; 100 requests from 1 worker or 500 requests from 5 concurrent workers, getting the main index page for the application:

    One ab worker

    ab -n 100 -c 1 http://127.0.0.1:8085/
    

    We get:

    Synthesis (1 client)

    Response times are:

    Response time (1 client)

    Five ab workers

    ab -n 500 -c 5 http://127.0.0.1:8085/
    

    We get:

    Synthesis (5 clients)

    Response times are:

    Response time (5 clients)

    Conclusion

    As expected, the legacy (and still default) twisted-based server is the least efficient method to serve a cubicweb application.

    When comparing results with only one CubicWeb worker, the pyramid+waitress solution that comes with cubicweb-pyramid is the most efficient, but mongrel2 + wsgid and circus + chaussette solutions mostly have similar performances when only one worker is activated. Surprisingly, the uwsgi solution is significantly less efficient, and especially have some requests that take significantly longer than other solutions (even the legacy twisted-based server).

    The price for activating several workers is small (around 3%) but significant when only one client is requesting the application. It is still unclear why.

    When there are severel workers requesting the application, it's not a surpsise that solutions with 4 workers behave significanly better (we are still far from a linear response however, roughly a 2x better for 4x the horsepower; maybe the hardware is the main reason for this unexpected non-linear response).

    I am quite surprised that uwsgi behaved significantly worse than the 2 other scalable solutions.

    Mongrel2 is still very efficient, but sadly the wsgid server I've used for these tests has not been developed for 2 years, and the uwsgi plugin for mongrel2 is not yet available on Debian.

    On the other side, I am very pleasantly surprised by circus + chaussette. Circus also comes with some nice features like a nice web dashboard which allows to add or remove workers dynamically:

    //www.cubicweb.org/file/5272071/raw //www.cubicweb.org/file/5272077/raw

  • CubicWeb Roadmap meeting on March 5th 2015

    2015/03/11 by David Douard

    The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. The previous roadmap meeting was in January 2015.

    Christophe de Vienne (Unlish) and Aurélien Campéas (self-employed) joined us.

    Christophe de Vienne asked for discussions on:

    • Security Context: settle on an approach, and make it happen.
    • Pyramid Cubicweb adoption: where are we? what authentication stack do we want by default?
    • Package layout (aka "develop mode" friendliness): let's get real
    • Documentation: is the restructuration attempt (https://www.cubicweb.org/ticket/4832808) a credible path for the documentation?

    Aurélien Campéas asked for discussions on:

    • status of integration in the 3.21 branch
    • a new API for cubicweb stores

    Sylvain Thénault asked for discussions on:

    • a new API for dataimport (including cubicweb stores, but not only),
    • new integrators on CW

    Versions

    Cubicweb

    Version 3.18

    This version is stable but old and maintained (current is 3.18.8).

    Version 3.19

    This version is stable and maintained (current is 3.19.9).

    Version 3.20

    This version is now stable and maintained (current is 3.20.4).

    Version 3.21

    See below

    Agenda

    Next roadmap meeting will be held at the beginning of may 2015 at Logilab. Interested parties are invited to get in touch.

    Open Discussions

    New integrators

    Rémi Cardona (rcardona) and Denis Laxaldle (dlaxalde) have now the publish access level on Cubicweb repositories.

    Security context

    Christophe exposed his proposal for a "security context" in Cubicweb, as exposed in https://lists.cubicweb.org/pipermail/cubicweb/2015-February/002278.html and https://lists.cubicweb.org/pipermail/cubicweb/2015-February/002297.html with a proposition of implementation (see https://www.cubicweb.org/ticket/4919855 )

    The idea has been validated based on a substitution variables, which names will start with "ctx:" (the RQL grammar will have to be modified to accept a ":")

    This will then allow to write RQL queries like (API still to be tuned):

    X owned_by U, U eid %(ctx:cwuser_eid)s
    

    Pyramid

    The pyramid-based web server proposed by Christophe and used for its unlish website is still under test and evaluation at Logilab. There are missing features (implemented in cubes) required to be able to deploy pyramid-cubicweb for most of the applications used at Logilab, especially cubicweb-signedrequest

    In order to make it possible to implement authentication cubes like cubicweb-signedrequest, the pyramid-cubicweb requires some modifications. These has been developped and are about to be published, along with a new version of signedrequest that provide pyramid compatibility.

    There are still some dependencies that lack a proper Debian package, but that should be done in the next few weeks.

    In order to properly identify pyramid-related code in a cube, it has been proposed that these code should go in modules in the cube named pviews and pconfig (note that most cube won't require any pyramid specific code). The includeme function should however be in the cube's main packgage (in the __init__.py file)

    There have been some discussions about the fact that, for now, a pyramid-cubicweb instance requires an anonymous user/access, which can also be a problem for some application.

    Layout

    Christophe pointed the fact that the directory/files layout of cubicweb and cubes do not follow current Python's de facto standards, which makes cubicweb hard to use in a context of virtualenv/pip based installation. There is the CWEP004 discussing some aspects of this problem.

    The decision has been taken to move toward a Cubicweb ecosystem that is more pip-friendly. This will be done step by step, starting with the dependencies (packages currently living in the logilab "namespace").

    Then we will investigate the feasibility of migrating the layout of Cubicweb itself.

    Documentation

    The new documentation structure has been approved.

    It has been proposed (and more or less accepted) to extract the documentation in a dedicated project. This is not a priority, however.

    Roadmap for 3.21

    No change since last meeting:

    • the complete removal of the dbapi, the merging of Connection and ClientConnection. remains
    • Integrate the pyramid cube to provide the pyramid command if the pyramid framework can be imported: removed (too soon, pyramid-cubicweb's APIs are not stable enough)
    • Integration of CWEP-003 (FROM clause for RQL): removed (will probably never be included unless someone needs it)
    • CWEP-004 (cubes as standard python packages) is being discussed: removed (not for 3.21, see above)

    dataimports et stores

    A heavy refactoring is under way that concerns data import in CubicWeb. The main goal is to design a single API to be used by the various cubes that accelerate the insertion of data (dataio, massiveimport, fastimport, etc) as well as the internal CWSource and its data feeds.

    For details, see the thread on the mailing-list and the patches arriving in the review pipeline.


  • CubicWeb roadmap meeting on January 8th, 2015

    2015/01/05 by Nicolas Chauvat

    The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. The previous roadmap meeting was in November 2014.

    Here is the report about the January 8th, 2015 meeting.

    Christophe de Vienne (Unlish) and Aurélien Campéas (self-employed) joined us to express their concerns and discuss the future of CubicWeb.

    Versions

    Version 3.18

    This version is stable but old and maintained (current is 3.18.7).

    Version 3.19

    This version is stable and maintained (current is 3.19.8).

    Version 3.20

    This version has been released a few days ago. It has not been deployed on production systems yet.

    Its main features are:

    • virtual relations: a new ComputedRelation class can be used in schema.py; its rule attribute is an RQL snippet that defines the new relation.

    • computed attributes: an attribute can now be defined with a formula argument (also an RQL snippet); it will be read-only, and updated automatically.

      Both of these features are described in CWEP-002, and the updated "Data model" chapter of the CubicWeb book.

    • cubicweb-ctl plugins can use the cubicweb.utils.admincnx function to get a Connection object from an instance name.

    • new 'tornado' wsgi backend

    • session cookies have the HttpOnly flag, so they're no longer exposed to javascript

    • rich text fields can be formatted as markdown

    • the edit controller detects concurrent editions, and raises a ValidationError if an entity was modified between form generation and submission

    • cubicweb can use a postgresql "schema" (namespace) for its tables

    • cubicweb-ctl configure can be used to set values of the admin user credentials in the sources configuration file

    For details read list of tickets for CubicWeb 3.20.0.

    We would have loved to integrate the pyramid cube in this release, but the debian packaging effort needed by the pyramid stack is quite big and is acceptable if we target jessie only (at decent price).

    Version 3.21

    For now, the roadmap for 3.21 is still the complete removal of the dbapi, the merging of Connection and ClientConnection.

    Integrate the pyramid cube to provide the pyramid command if the pyramid framework can be imported.

    Integration of CWEP-003 (FROM clause for RQL) and CWEP-004 (cubes as standard python packages) is being discussed.

    Version 4.0

    We expect to accelerate development of CubicWeb 4, which exact roadmap is still to be discussed, but we may already want:

    • be pyramid-based (remove twisted, auth management, etc.),
    • do not have anything left of old dbapi and ClientConnection,
    • integrate squareui as main (and only) web-ui "template" or remove web generation (almost) completely from cubicweb-core and provide it only through the cube system.

    Agenda

    Next roadmap meeting will be held at the beginning of march 2015 at Logilab. Interested parties are invited to get in touch.

    Open Discussions

    Refactoring the documentation

    Christophe de Vienne suggested to completely revamp the documentation and intends to lead this effort.

    Training material

    Aurélien Campéas asks if Logilab would be willing to share its training material under a free license to help interested parties organize and sell trainings.

    Towards making squareui the default rendering engine for cubicweb

    We are expecting to be able to use squareui/bootstrap as "rendering engine" for our forge applications (like http://www.cubicweb.org and http://www.logilab.org) as soon as possible. However to achieve to goal, there are still too many "visual bugs", some of which may require a discussion.

    Among others:

    • put the ctxtoolbar component in the <nav> div
    • each box component should have an icon (what API for this?)
    • we cannot easily make the left column of the main template responsive-aware (requires to change the html flow), so it's probably best to take inspiration from things like http://wrapbootstrap.com/preview/WB0N89JMK
    • facet boxes are a mess, there is no simple solution to have a "smart layout"

    Migration

    • AppObjects should not be loaded by default
    • Have a look at Alembic the migration tool for SQLAlchemy and take inspiration from there.

show 133 results