subscribe to this blog

CubicWeb Blog

News about the framework and its uses.

show 141 results
  • Report of June 16th Cubicweb Meeting

    2020/06/24 by Henri Cazottes

    Hi everyone,

    Here is the weekly report of last week meeting with some delay...

    Kanban status

    You can check the milestone here

    • Build the new cubicweb image for all the intranet apps on the public head #9
    • Add py{27,3}-from-forge to clients project to ensure we don't break everything when releasing #37
      • MR waiting on francearchive, Laurent is going to talk to Katia about it. Also Simon says that everything is fine with 3.28rc1 on data.bnf
      • all internal apps of logilab runs with 3.28rc1 and there is not bugs to signal

    Todo

    • Setup a demo with a SPARQL API
      • this work has been started few years ago but today clients are interested in this feature
      • option to create an RQL-SPARQL bridge would be too long
      • work in progress to compare RQL and SPARQL expressiveness, not done yet
      • current lead: using rdflib-sqlalchemy by adding tables next to existing Cubicweb tables and creating a prototype of an OWL to YAMS converter
    • Rollback class_deprecated modifications on logilab-common
    • Continue typing other libs such as:
      • cubicweb (complex)
      • rql
      • logilab-mtconverter
        • Patrick will probably start with this before continuing on RQL
      • logilab-constrain
      • logilab-database
    • Identify "good first issue" to ease contributing
    • Think about the documentation structure and what we want to write (for the next release)
      • reduce technical debt
      • spread documentation improvements among several sprints

    Current work

    • working on fixes for YAMS tip for CW
    • reducing the load on the CI by removing some useless tests when triggered from other CI
    • adding needed commits on https://github.com/logilab/yapps to update it for python3 compatibility

    See you very soon for the next report !


  • Report of June 10th Cubicweb Meeting

    2020/06/09 by Laurent Peuch

    Hi everyone,

    We've just published the RC1 for CubicWeb https://pypi.org/project/cubicweb/3.28.0rc1/ and a new version 1.7.0 for logilab-common https://pypi.org/project/logilab-common/1.7.0/

    Our current focus is finishing the last details for the release.

    Milestone update

    • the changelog is now part of the documentation of logilab-common to make sure it is visible
    • test our clients project against our latest version on our repository to ensure we don't break everything when making a release
    • allow 2 randomly breaking tests to fail (those aren't part of the code we are currently working on)

    Current roadmap

    Semver

    One of our focus right now is to make stable releases of our core projects that won't break all the things ™ and we've made a lot of improvement in our testing suit to ensure that we test everything against our latest modifications before a release is made. Another problem we have right now is that CW only depends on a minimum version number for its dependencies, this mean that if we want to make a new release for one of the dependencies that will have some breaking code this introduce the risk of breaking all new CW installations.

    To solve this situation we have decided to implement semantic versioning and only introduction breaking changes in major releases and in addition to only depends on one specific major release at the time in CW dependencies. This way, when we need to make a new release with breaking changes, this will be a major release and we won't break all new CW installations.

    We have planned to start implementing this strategy starting CW version 4.0

    Various updates

    • a lot of fixes have been pushed on YAMS and CubicWeb to make CubicWeb compatible with the latest modifications in YAMS

    See you next week!


  • Report of June 3rd Cubicweb Meeting

    2020/06/03 by Henri Cazottes

    Hi everyrone,

    Version 3.28-rc1 is on its way! First, let's have a look to the issue board state.

    Milestone update

    • Introduced types #10
      • logilab.common.deprecation has been typed (see hackathon report below): done
    • Add tests for the content negociation !20: MR about to be accepted
    • Update logilab-common changelogs #43 : done
    • Add automatic doc re-build to the CubicWeb CI #8 : done

    Todo

    • Review and accept MR !20
    • Release logilab-common and cubicweb 3.28-rc1

    Semver discussions

    Right now, dependencies are only specifying a minimal version. So if we introduce a breaking change in a new version, apps might break too. We plan to follow semver convention to prevent this from happening.

    We also discussed the idea of aligning version between compatible tools, so every major version would work with the same major version of other tools/dependencies.

    This idea will be introduced in 3.29 documentation, but will probably start with the release of Cubicweb version 4.

    Hackathon

    Last Friday we did an internal hackathon at Logilab and Laurent, Noé and I spent time working on Cubicweb. We mainly:

    • wrote changelogs for:
      • logilab-common
      • cubicweb
    • tried to add a Merge Request template on Cubicweb
      • doesn't work on Heptapod actually, we will ask Octobus to have a look (see #46)
    • added annotation types on logilab.common.deprecated
    • improved tox.ini and added a gitlag-ci.yaml file in cube skeleton

    That's all! You should receive and email soon about the rc1 release.

    Thanks for reading,

    Henri


  • Report of May 26th Cubicweb Meeting

    2020/05/26 by Henri Cazottes

    Hi,

    Welcome back to another weekly report! Today, the following topics have been discussed.

    Broken tests situation, follow up

    Migrating CubicWeb to Heptapod and modifications in dependencies resulted in broken tests as it was presented last week. Work has been done on Friday afternoon thanks to Simon and Laurent, but it's not fixed yet. Tox is now happy but we still have a bug on a test that succeeds locally but not when run by the CI job. We do have a lead which may concern firefox usage in headless mode. Jobs logs are available here.

    Milestone update

    • Introduced types
      • Types have been added in Yams, merge request about to be reviewed
      • We choose to dissociate this issue from the actual release as it's more related to Yams and not mandatory to release a 3.28 version.
      • Re-build ReadTheDoc for every release
      • Done for most of the dependencies but not CubicWeb yet
      • Two dependencies, mtconvert and constraint don't have doc nor tests, we think about removing them instead of creating and maintaining this code which is pretty old
    • Move to semantic versionning
      • we talked about improving dependencies requirements to ease the release process (not giving only one version)
      • we think we should stick to semver, but we need to discuss it more (should we bump all major version to the same number for interoperable dependencies, etc...)
      • As those questions need to be discussed, we choose to move this issue to the 3.29 milestone
    • Add tests to content negociation
      • Need to be done
    • Check if ?vid=rdf is still working
      • Done
      • We did spot that requesting vid=rdf or using content negotiation would not return the same RDF. We should fix this to have a more consistent behavior. Will be added for the next release.

    Todo before releasing version 3.28

    To sum up, before releasing the next CubicWeb version, we need to:

    • Fix CI tests
    • Add automatic doc re-build to the CubicWeb CI

    This should be done and released within the next two weeks.

    Side notes

    • For the next release we should align CubicWeb with changes made in Yams
    • Release early, release often

    See you next week!


  • Report of May 19th Cubicweb Meeting

    2020/05/20 by Simon Chabot

    Hi everyone,

    Yesterday we held a new CubicWeb meeting, and we would like to share with you what was said during that meeting :

    • some issues have been migrated from cubicweb.org forge to the heptapod forge. You can find them here. Only the issues which were in the cubicweb-dev-board have been migrated, the others have been considered too old to make their migration worthwhile ;

    • new milestones have been created, and some issues have been affected to them ;

    • if you want to help out with Cubicweb's development, you can have a look to the Issues Board, and pick one task in the “To Do” column (this task should be related to a milestone) ;

    • CW tests are still failing on the forge, and it's also related to other packages that have been released. Fixing those tests is quite urgent now, therefore we suggest to fix them in a sprint this Friday afternoon. Feel
      free to join !

    • On main Cubicweb's dependencies (RQL, YAMS, logilab-common, etc), a heptapod job has been added to trigger CW's tests with the last version of the dependency in the forge and not only the last released version on pypi. This should help to release new versions of CW's dependencies with
      more confidence. (for now, it only triggers the job, a future version of heptapod should provide a better integration of multi-project pipelines) ;

    • The documentations of CubicWeb's dependencies are now automatically published on readthedocs. The work is in progress for CubicWeb itself ;

    Next week, if the tests are successful, we will talk about a release candidate for 3.28.

    Stay tuned :)


  • A roadmap to Cubicweb 3.28 (and beyond)

    2020/05/13 by Simon Chabot

    Yesterday at Logilab we had a small meeting to discuss about a roadmap to Cubicweb 3.28 (and beyond), and we would like to report back to you from this meeting.

    Cubicweb 3.28 will mainly bring the implementation of content negotiation. It means that Cubicweb will handle content negotiation and will be able to return RDF using Cubicweb's ontology when requested by a client.

    The 3.28 will have other features as well (like a new variables attributes to ResultSet that contains the name of the projected variables, etc). Those features will be detailed the release changelog.

    Before releasing this version, we would like to finish the migration to heptapod, to make sure that everything is ok. The remaining tasks are:

    • fixing the CI (there is still some random failings, that need further investigation)
    • migrate the jenkins job that pushes images to hub.docker.com on heptapod, to make everthing available from the forge. It will be explicit for everyone when a job is done, and what is its status.

    Beside of releasing Cubicweb 3.28, its ecosystem will also be updated:

    • logilab-common, a new version will be released very soon, which brings a refactoring of the deprecation system, and annotations (coming from pyannotate)
    • yams, a new version is coming. This version:
    • brings type annotation (manually done, a carefully checked);
    • removes a lot of abbreviation to make the code clearer;
    • removes some magic related to a object which used to behave like a string;

    The goal of these two releases, is to have type annotations in the core libraries used by CubicWeb, and then to be able to bring type annotation into CubicWeb itself, in a future version.

    On those projects, some “modernisation” has been started too ; (fixing flake8 when needed, repaint the code black). This “modernisation” step is still on going on the different projects related to CubicWeb (and achieved for yams, and logilab-common).

    In the medium term, we would like to focus on the documentation of CubicWeb and its ecosystem. We do know that it's really hard for newcomers (and even ourself sometime) to understand how to start, what each module is doing etc. An automatic documentation has been released for some modules (see 1, 2 or 3 for instance). It would be nice to automatize the update of the documentation on readthedocs, update the old examples, and add new ones about the new feature we are adding (like content negotiation, pyramid predicates, etc). This could be done in team Friday's sprint or hackathon for instance. CubicWeb would also need some modernisation (running black ? and above all, make all files flake8 compilant…).

    Regarding CubicWeb development, all (or, at least a lot of) cubes and Cubicweb related projects moved from cubicweb.org's forge to our instance of heptapod (4 and 5). Some issues have been imported from cubicweb.org to heptapod. New issues should be opened on heptapod, and the review should also be done there. We hope that will ease the reappropriation of the code basis and stimulates new merge-requests :)

    To end this report, we like to emphasis that we will try to make a « remote Cubicweb meeting » each Tuesday at 2 pm. If you would like to participate to this meeting, it's with great pleasure (if you need the webconference URL, contact one of us, we will provide it to you). We also created a #Cubicweb channel on matrix.logilab.org ; feel free to ask for an invitation if you'd like to discuss Cubicweb related things with us.

    All the best, and… see you next Tuesday :)


  • What is new in CubicWeb 3.27 ?

    2020/02/03 by Nicolas Chauvat

    Hello CubicWeb community,

    We are pleased to announce the release of CubicWeb 3.27. Many thanks to all the contributors of this release!

    Main changes in this release are listed below. Please note this release drops python2 support.

    Enjoy this new version!

    New features

    • Tests can now be run concurrently across multiple processes. You can use pytest-xdist for that. For tests using PostgresApptestConfiguration you should be aware that startpgcluster() can't run concurrently. Workaround is to call pytest with --dist=loadfile to use a single test process per test module or use an existing database cluster and set db-host and db-port of devtools.DEFAULT_PSQL_SOURCES['system'] accordingly.
    • on cubicweb-ctl create and cubicweb-ctl pyramid, if it doesn't already exist in the instance directory, the pyramid.ini file will be generated with the needed secrets.
    • add a --pdb flag to all cubicweb-ctl command to launch (i)pdb if an exception occurs during a command execution.
    • the --loglevel and --dbglevel flags are available for all cubicweb-ctl instance commands (and not only the pyramid one)
    • following "only in foreground" behavior all commands logs to stdout by default from now on. To still log to a file pass log_to_file=True to CubicWebConfiguration.config_for
    • add a new migration function update_bfss_path(old_path, new_path) to update the path in Bytes File-System Storage (bfss).
    • on every request display request path and selected controller in CLI
    • migration interactive mode improvements:
      • when an exception occurs, display the full traceback instead of only the exception
      • on migration p(db) choice, launch ipdb if it's installed
      • on migration p(db) choice, give the traceback to pdb if it's available, this mean that the (i)pdb interactive session will be on the stack of the exception instead of being on the stack where pdb is launched which will allow the user to access all the relevant context of the exception which otherwise is lost
    • on DBG_SQL and/or DBG_RQL, if pygments is installed, syntax highlight sql/rql debug output
    • allow to specify the instance id for any instance command using the CW_INSTANCE global variable instead of or giving it as a cli argument
    • when debugmode is activated ('-D/--debug' on the pyramid command for example), the HTML generated by CW will contains new tags that will indicate by which object in the code it has been generated and in which line of which source code. For example:
    <div
      cubicweb-generated-by="cubicweb.web.views.basetemplates.TheMainTemplate"
      cubicweb-from-source="/home/user/code/logilab/cubicweb/cubicweb/web/views/basetemplates.py:161"
      id="contentmain">
        <h1
          cubicweb-generated-by="cubicweb.web.views.basetemplates.TheMainTemplate"
          cubicweb-from-source="/home/user/code/logilab/cubicweb/cubicweb/view.py:136">
            unset title
        </h1>
        [...]
    </div>
    

    While this hasn't been done yet, this feature is an open path for building dynamic tools that can help inspect the page.

    • a new debug channels mechanism has been added, you can subscribe to one of those channels in your python code to build debug tools for example (the pyramid custom panels are built using that) and you will receive a datastructure (a dict) containing related information. The available channels are: controller, rql, sql, vreg, registry_decisions
    • add a new '-t/--toolbar' option the pyramid command to activate the pyramid debugtoolbar
    • a series of pyramid debugtoolbar panels specifically made for CW, see bellow

    Pyramid debugtoolbar and custom panel

    The pyramid debugtoolbar is now integrated into CubicWeb during the development phase when you use the 'pyramid' command. To activate it you need to pass the '-t/--toolbar' argument to the 'pyramid' command.

    In addition, a series of custom panels specifically done for CW are now available, they display useful information for the development and the debugging of each page. The available panels are:

    • a general panel which contains the selected controller, the current settings and useful links screenshot1
    • a panel listing all decisions taken in registry for building this page screenshot2
    • a panel listing the content of the vreg registries screenshot3
    • a panel listing all the RQL queries made during a request screenshot4
    • a panel listing all the SQL queries made during a request screenshot5

    Furthermore, in all those panels, next to each object/class/function/method a link to display its source code is available (shown as '[source]' screenshot6) and also every file path shown is a traceback is also a link to display the corresponding file (screenshot7). For example: screenshot8.

    Backwards incompatible changes

    • Standardization on the way to launch a cubicweb instance, from now on the only way to do that will be the used the pyramid command. Therefore:

      • cubicweb-ctl commands "start", "stop", "restart", "reload" and "status" have been removed because they relied on the Twisted web server backend that is no longer maintained nor working with Python 3.
      • Twisted web server support has been removed.
      • cubicweb-ctl wsgi has also been removed.
    • Support for legacy cubes (in the 'cubes' python namespace) has been dropped. Use of environment variables CW_CUBES_PATH and CUBES_DIR is removed.

    • Python 2 support has been dropped.

    • Exceptions in notification hooks aren't catched-all anymore during tests so one can expect tests that seem to pass (but were actually silently failing) to fail now.

    • All "cubicweb-ctl" command only accept one instance argument from now one (instead of 0 to n)

    • 'pyramid' command will always run in the foreground now, by consequence the option --no-daemon has been removed.

    • DBG_MS flag has been removed since it is not used anymore

    • transactions db logs where displayed using the logging (debug/info/warning...) mechanism, now it is only displayed if the corresponding DBG_OPS flag is used

    Deprecated code drops

    Most code deprecated by version 3.25 or older versions has been dropped.


  • Implementing the langserver protocol for RQL

    2019/10/31 by Laurent Peuch

    One of our next project for cubicweb and its ecosystem is to implement the langserver protocol for the RQL language that we are using to query the data stored in CubicWeb. The langserver protocol is an idea to solve one problem: to integrate operation for various languages, most IDE/tools needs to reimplement the wheel all the time, doing custom plugin etc... To solve this issue, this protocol has been invented with one idea: make one server for a language, then all IDE/tools that talks this protocol will be able to integrate it easily.

    language server protocol matrice illustration

    So the idea is simple: let's build our own server for RQL so we'll be able to integrate it everywhere and build tools for it.

    Since RQL has similarities with GraphQL, one of the goals is to have something similar to Graphiql which is for example used by GitHub to expose their API at https://developer.github.com/v4/explorer/

    github graphql explorer

    So this post has several objectives:

    • gather people that would be motivate to work on that subject, for now there is Laurent Wouters and me :)
    • explain to you in more details (not all) how the language server protocol works
    • show what is already existing for both langserver in python and rql
    • show the first roadmap we've discussed with Laurent Wouters on how we think we can do that :)
    • be a place to discuss this project, things aren't fixed yet :)

    So, what is the language server protocol (LSP)?

    It's a JSON-RPC based protocol where the IDE/tool talks to the server. JSON-RPC, said simply, is a bi-directional protocol in json.

    In this procotol you have 2 kind of exchanges:

    • requests: where the client (or server) ask the server (or the server ask the client) something and a reply is expected. For example: where is the definition of this function?
    • notifications: the same but without an expected reply. For example: linting information or error detection

    language server protocol example schema

    The LSP specifications has 3 bigs categories:

    • everything about initialization/shutdown the server etc...
    • everything regarding text and workspace synchronization between the server and the client
    • the actual things that interest us: a list of languages features that the server supports (you aren't in the obligation to implement everything)

    Here is the simplified list of possible languages features that the website present:

    • Code completion
    • Hover
    • Jump to def
    • Workspace symbols
    • Find references
    • Diagnostics

    The specification is much more detailed but way less comprehensive (look at the "language features" on the right menu for more details):

    • completion/completion resolve
    • hover (when you put your cursor on something)
    • signatureHelp
    • declaration (go to...)
    • definition (go to...)
    • typeDefinition (go to...)
    • implementation (go to...)
    • references
    • documentHighlight (highlight all references to a symbol)
    • documentSymbol ("symbol" is a generic term for variable, definitions etc...)
    • codeAction (this one is interesting)
    • codeLens/codeLens resolve
    • documentLink/documentLink resolve
    • documentColor/colorPresentation (stuff about picking colors)
    • formatting/rangeFormatting/onTypeFormatting (set tab vs space)
    • rename/prepareRename
    • foldingRange

    (Comments are from my current understanding of the spec, it might not be perfect)

    The one that is really interesting here (but not our priority right now) is "codeAction", it's basically a generic entry point for every refactoring kind of operations as some examples from the spec shows:

    Example extract actions:

    • Extract method
    • Extract function
    • Extract variable
    • Extract interface from class

    Example inline actions:

    • Inline function
    • Inline variable
    • Inline constant

    Example rewrite actions:

    • Convert JavaScript function to class
    • Add or remove parameter
    • Encapsulate field
    • Make method static
    • Move method to base class

    But I'm not expecting us to have direct need for it but that really seems one to keep in mind.

    One question that I frequently got was: is syntax highlight included in the langserver protocol? Having double checked with Laurent Wouters, it's actually not the case (I thought documentSymbol could be used for that but actually no).

    But we already have an implementation for that in pygments: https://hg.logilab.org/master/rql/file/d30c34a04ebf/rql/pygments_ext.py

    rql pygments syntax highlight

    What is currently existing for LSP in python and rql

    The state is not great in the python ecosystem but not a disaster. Right now I haven't been able to find any generic python implementation of LSP that we could really reuse and integrate.

    There is, right now and to my knowledge, only 2 maintained implementation of LSP in python. One for python and one for ... Fortran x)

    Palantir's one makes extensive use of advanced magic code doesn't seems really necessary but it is probably of higher quality code since the Fortran one doesn't seems very idiomatic but looks much simpler.

    So we'll ever need to extract the needed code from one of those of implement our own, not so great.

    On the RQL side, everything that seems to be useful for our current situation is located in the RQL package that we maintain: https://hg.logilab.org/master/rql

    Roadmap

    After a discussion with Laurent Wouters, a first roadmap looks like this:

    • extract the code from either palantir or fortran LSP implementation and come with a generic implementation (I'm probably going to do it but Laurent told me he his going to take a look too) When I'm talking about a generic implementation I'm talking about everything listed in the big category of the protocol that isn't related to language features which we don't really want to rewrite again.

    Once that's done, start implementing the language features for RQL:

    • the easiest is the syntax errors detection code, we just need to launch to parser on the code and handle the potential errors
    • do that with pretty specific red underline
    • play with RQL AST to extract the symbols and start doing things like codeLens and hover
    • much more complex (and for later): autocompletion (we'll either need a demi compiler or to modify the current one for that)

    Side note

    To better understand the motivation behind this move, it is part of the more global move of drop the "Web" from CubicWeb and replace all the front end current implementation by reactjs+typescript views. In this context CubicWeb (or Cubic?) will only serves as a backend provide with which we will talk in... RQL! Therefor writing and using RQL will be much more important than right now.


  • Hypermedia API with cubicweb-jsonschema

    2017/04/04 by Denis Laxalde

    This is the second post of a series about cubicweb-jsonschema. The first post mainly dealt with JSON Schema representations of CubicWeb entities along with a brief description of the JSON API. In this second post, I'll describe another aspect of the project that aims at building an hypermedia API by leveraging the JSON Hyper Schema specification.

    Hypermedia APIs and JSON Hyper Schema

    Hypermedia API is somehow a synonymous of RESTful API but it makes it clearer that the API serves hypermedia responses, i.e. content that helps discoverability of other resources.

    At the heart of an hypermedia API is the concept of link relation which both aims at describing relationships between resources as well as provinding ways to manipulate them.

    In JSON Hyper Schema terminology, link relations take the form of a collection of Link Description Objects gathered into a links property of a JSON Schema document. These Link Description Objects thus describes relationships between the instance described by the JSON Schema document at stake and other resources; they hold a number of properties that makes relationships manipulation possible:

    • rel is the name of the relation, it is usually one of relation names registered at IANA;
    • href indicates the URI of the target of the relation, it may be templated by a JSON Schema;
    • targetSchema is a JSON Schema document (or reference) describing the target of the link relation;
    • schema (recently renamed as submissionSchema) is a JSON Schema document (or reference) describing what the target of the link expects when submitting data.

    Hypermedia walkthrough

    In the remaining of the article, I'll walk through a navigation path that is made possible by hypermedia controls provided by cubicweb-jsonschema. I'll continue on the example application described in the first post of the series which schema consists of Book, Author and Topic entity types. In essence, this walkthrough is typical of what an intelligent client could do when exposed to the API, i.e. from any resource, discover other resources and navigate or manipulate them.

    This walkthrough assumes that, given any resource (i.e. something that has a URL like /book/1), the server would expose data at the main URL when the client asks for JSON through the Accept header and it would expose the JSON Schema of the resource at a schema view of the same URL (i.e. /book/1/schema). This assumption can be regarded as a kind of client/server coupling, which might go away in later implementation.

    Site root

    While client navigation could start from any resource, we start from the root resource and retrieve its schema:

    GET /schema
    Accept: application/schema+json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "links": [
            {
                "href": "/author/",
                "rel": "collection",
                "schema": {
                    "$ref": "/author/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/author/schema"
                },
                "title": "Authors"
            },
            {
                "href": "/book/",
                "rel": "collection",
                "schema": {
                    "$ref": "/book/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Books"
            },
            {
                "href": "/topic/",
                "rel": "collection",
                "schema": {
                    "$ref": "/topic/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/topic/schema"
                },
                "title": "Topics"
            }
        ]
    }
    

    So at root URL, our application serves a JSON Hyper Schema that only consists of links. It has no JSON Schema document, which is natural since there's usually no data bound to the root resource (think of it as empty rset in CubicWeb terminology).

    These links correspond to top-level entity types, i.e. those that would appear in the default startup page of a CubicWeb application. They all have "rel": "collection" relation name (this comes from RFC6573) as their target is a collection of entities. We also have schema and targetSchema properties.

    From collection to items

    Now that we have added a new book, let's step back and use our books link to retrieve data (verb GET):

    GET /book/
    Accept: application/json
    
    HTTP/1.1 200 OK
    Allow: GET, POST
    Content-Type: application/json
    
    [
        {
            "id": "859",
            "title": "L'homme qui rit"
        },
        {
            "id": "858",
            "title": "The Old Man and the Sea"
        },
    ]
    

    which, as always, needs to be completed by a JSON Schema:

    GET /book/schema
    Accept: application/schema+json
    
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "$ref": "#/definitions/Book_plural",
        "definitions": {
            "Book_plural": {
                "items": {
                    "properties": {
                        "id": {
                            "type": "string"
                        },
                        "title": {
                            "type": "string"
                        }
                    },
                    "type": "object"
                },
                "title": "Books",
                "type": "array"
            }
        },
        "links": [
            {
                "href": "/book/",
                "rel": "collection",
                "schema": {
                    "$ref": "/book/schema?role=creation"
                },
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Books"
            },
            {
                "href": "/book/{id}",
                "rel": "item",
                "targetSchema": {
                    "$ref": "/book/schema?role=view"
                },
                "title": "Book"
            }
        ]
    }
    

    Consider the last item of links in the above schema. It has a "rel": "item" property which indicates how to access items of the collection; its href property is a templated URI which can be expanded using instance data and schema (here we only have a single id template variable).

    So our client may navigate to the first item of the collection (id="859") at /book/859 URI, and retrieve resource data:

    GET /book/859
    Accept: application/json
    
    HTTP/1.1 200 OK
    Allow: GET, PUT, DELETE
    Content-Type: application/json
    
    {
        "author": [
            "Victor Hugo"
        ],
        "publication_date": "1869-04-01T00:00:00",
        "title": "L'homme qui rit"
    }
    

    and schema:

    GET /book/859/schema
    Accept: application/schema+json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    
    {
        "$ref": "#/definitions/Book",
        "definitions": {
            "Book": {
                "additionalProperties": false,
                "properties": {
                    "author": {
                        "items": {
                            "type": "string"
                        },
                        "title": "author",
                        "type": "array"
                    },
                    "publication_date": {
                        "format": "date-time",
                        "title": "publication date",
                        "type": "string"
                    },
                    "title": {
                        "title": "title",
                        "type": "string"
                    },
                    "topics": {
                        "items": {
                            "type": "string"
                        },
                        "title": "topics",
                        "type": "array"
                    }
                },
                "title": "Book",
                "type": "object"
            }
        },
        "links": [
            {
                "href": "/book/",
                "rel": "up",
                "targetSchema": {
                    "$ref": "/book/schema"
                },
                "title": "Book_plural"
            },
            {
                "href": "/book/859/",
                "rel": "self",
                "schema": {
                    "$ref": "/book/859/schema?role=edition"
                },
                "targetSchema": {
                    "$ref": "/book/859/schema?role=view"
                },
                "title": "Book #859"
            }
        ]
    }
    

    Entity resource

    The resource obtained above as an item of a collection is actually an entity. Notice the rel="self" link. It indicates how to manipulate the current resource (i.e. at which URI, using a given schema depending on what actions we want to perform). Still this link does not indicate what actions may be performed. This indication is found in the Allow header of the data response above:

    Allow: GET, PUT, DELETE
    

    With these information bits, our intelligent client is able to, for instance, form a request to delete the resource. On the other hand, the action to update the resource (which is allowed because of the presence of PUT in Allow header, per HTTP semantics) would take the form of a request which body conforms to the JSON Schema pointed at by the schema property of the link.

    Also note the rel="up" link which makes it possible to navigate to the collection of books.

    Conclusions

    This post introduced the main hypermedia capabilities of cubicweb-jsonschema, built on top of the JSON Hyper Schema specification. The resulting Hypermedia API makes it possible for an intelligent client to navigate through hypermedia resources and manipulate them by using both link relation semantics and HTTP verbs.

    In the next post, I'll deal with relationships description and manipulation both in terms of API (endpoints) and hypermedia representation.


  • Introducing cubicweb-jsonschema

    2017/03/23 by Denis Laxalde

    This is the first post of a series introducing the cubicweb-jsonschema project that is currently under development at Logilab. In this post, I'll first introduce the general goals of the project and then present in more details two aspects about data models (the connection between Yams and JSON schema in particular) and the basic features of the API. This post does not always present how things work in the current implementation but rather how they should.

    Goals of cubicweb-jsonschema

    From a high level point of view, cubicweb-jsonschema addresses mainly two interconnected aspects. One related to modelling for client-side development of user interfaces to CubicWeb applications while the other one concerns the HTTP API.

    As far as modelling is concerned, cubicweb-jsonschema essentially aims at providing a transformation mechanism between a Yams schema and JSON Schema that is both automatic and extensible. This means that we can ultimately expect that Yams definitions alone would sufficient to have generated JSON schema definitions that would consistent enough to build an UI, pretty much as it is currently with the automatic web UI in CubicWeb. A corollary of this goal is that we want JSON schema definitions to match their context of usage, meaning that a JSON schema definition would not be the same in the context of viewing, editing or relationships manipulations.

    In terms of API, cubicweb-jsonschema essentially aims at providing an HTTP API to manipulate entities based on their JSON Schema definitions.

    Finally, the ultimate goal is to expose an hypermedia API for a CubicWeb application in order to be able to ultimately build an intelligent client. For this we'll build upon the JSON Hyper-Schema specification. This aspect will be discussed in a later post.

    Basic usage as an HTTP API library

    Consider a simple case where one wants to manipulate entities of type Author described by the following Yams schema definition:

    class Author(EntityType):
        name = String(required=True)
    

    With cubicweb-jsonschema one can get JSON Schema for this entity type in at different contexts such: view, creation or edition. For instance:

    • in a view context, the JSON Schema will be:

      {
          "$ref": "#/definitions/Author",
          "definitions": {
              "Author": {
                  "additionalProperties": false,
                  "properties": {
                      "name": {
                          "title": "name",
                          "type": "string"
                      }
                  },
                  "title": "Author",
                  "type": "object"
              }
          }
      }
      
    • whereas in creation context, it'll be:

      {
          "$ref": "#/definitions/Author",
          "definitions": {
              "Author": {
                  "additionalProperties": false,
                  "properties": {
                      "name": {
                          "title": "name",
                          "type": "string"
                      }
                  },
                  "required": [
                      "name"
                  ],
                  "title": "Author",
                  "type": "object"
              }
          }
      }
      

      (notice, the required keyword listing name property).

    Such JSON Schema definitions are automatically generated from Yams definitions. In addition, cubicweb-jsonschema exposes some endpoints for basic CRUD operations on resources through an HTTP (JSON) API. From the client point of view, requests on these endpoints are of course expected to match JSON Schema definitions. Some examples:

    Get an author resource:

    GET /author/855
    Accept:application/json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    {"name": "Ernest Hemingway"}
    

    Update an author:

    PATCH /author/855
    Accept:application/json
    Content-Type: application/json
    {"name": "Ernest Miller Hemingway"}
    
    HTTP/1.1 200 OK
    Location: /author/855/
    Content-Type: application/json
    {"name": "Ernest Miller Hemingway"}
    

    Create an author:

    POST /author
    Accept:application/json
    Content-Type: application/json
    {"name": "Victor Hugo"}
    
    HTTP/1.1 201 Created
    Content-Type: application/json
    Location: /Author/858
    {"name": "Victor Hugo"}
    

    Delete an author:

    DELETE /author/858
    
    HTTP/1.1 204 No Content
    

    Now if the client sends invalid input with respect to the schema, they'll get an error:

    (We provide a wrong born property in request body.)

    PATCH /author/855
    Accept:application/json
    Content-Type: application/json
    {"born": "1899-07-21"}
    
    HTTP/1.1 400 Bad Request
    Content-Type: application/json
    
    {
        "errors": [
            {
                "details": "Additional properties are not allowed ('born' was unexpected)",
                "status": 422
            }
        ]
    }
    

    From Yams model to JSON Schema definitions

    The example above illustrates automatic generation of JSON Schema documents based on Yams schema definitions. These documents are expected to help developping views and forms for a web client. Clearly, we expect that cubicweb-jsonschema serves JSON Schema documents for viewing and editing entities as cubicweb.web serves HTML documents for the same purposes. The underlying logic for JSON Schema generation is currently heavily inspired by the logic of primary view and automatic entity form as they exists in cubicweb.web.views. That is: the Yams schema is introspected to determine how properties should be generated and any additionnal control over this can be performed through uicfg declarations [1].

    To illustrate let's consider the following schema definitions which:

    class Book(EntityType):
        title = String(required=True)
        publication_date = Datetime(required=True)
    
    class Illustration(EntityType):
        data = Bytes(required=True)
    
    class illustrates(RelationDefinition):
        subject = 'Illustration'
        object = 'Book'
        cardinality = '1*'
        composite = 'object'
        inlined = True
    
    class Author(EntityType):
        name = String(required=True)
    
    class author(RelationDefinition):
        subject = 'Book'
        object = 'Author'
        cardinality = '1*'
    
    class Topic(EntityType):
        name = String(required=True)
    
    class topics(RelationDefinition):
        subject = 'Book'
        object = 'Topic'
        cardinality = '**'
    

    and consider, as before, JSON Schema documents in different contexts for the the Book entity type:

    • in view context:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "type": "string"
                          },
                          "title": "author",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      },
                      "topics": {
                          "items": {
                              "type": "string"
                          },
                          "title": "topics",
                          "type": "array"
                      }
                  },
                  "title": "Book",
                  "type": "object"
              }
          }
      }
      

      We have a single Book definition in this document, in which we find attributes defined in the Yams schema (title and publication_date). We also find the two relations where Book is involved: topics and author, both appearing as a single array of "string" items. The author relationship appears like that because it is mandatory but not composite. On the other hand, the topics relationship has the following uicfg rule:

      uicfg.primaryview_section.tag_subject_of(('Book', 'topics', '*'), 'attributes')
      

      so that it's definition appears embedded in the document of Book definition.

      A typical JSON representation of a Book entity would be:

      {
          "author": [
              "Ernest Miller Hemingway"
          ],
          "title": "The Old Man and the Sea",
          "topics": [
              "sword fish",
              "cuba"
          ]
      }
      
    • in creation context:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "oneOf": [
                                  {
                                      "enum": [
                                          "855"
                                      ],
                                      "title": "Ernest Miller Hemingway"
                                  },
                                  {
                                      "enum": [
                                          "857"
                                      ],
                                      "title": "Victor Hugo"
                                  }
                              ],
                              "type": "string"
                          },
                          "maxItems": 1,
                          "minItems": 1,
                          "title": "author",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      }
                  },
                  "required": [
                      "title",
                      "publication_date"
                  ],
                  "title": "Book",
                  "type": "object"
              }
          }
      }
      

      notice the differences, we now only have attributes and required relationships (author) in this schema and we have the required listing mandatory attributes; the author property is represented as an array which items consist of pre-existing objects of the author relationship (namely Author entities).

      Now assume we add the following uicfg declaration:

      uicfg.autoform_section.tag_object_of(('*', 'illustrates', 'Book'), 'main', 'inlined')
      

      the JSON Schema for creation context will be:

      {
          "$ref": "#/definitions/Book",
          "definitions": {
              "Book": {
                  "additionalProperties": false,
                  "properties": {
                      "author": {
                          "items": {
                              "oneOf": [
                                  {
                                      "enum": [
                                          "855"
                                      ],
                                      "title": "Ernest Miller Hemingway"
                                  },
                                  {
                                      "enum": [
                                          "857"
                                      ],
                                      "title": "Victor Hugo"
                                  }
                              ],
                              "type": "string"
                          },
                          "maxItems": 1,
                          "minItems": 1,
                          "title": "author",
                          "type": "array"
                      },
                      "illustrates": {
                          "items": {
                              "$ref": "#/definitions/Illustration"
                          },
                          "title": "illustrates_object",
                          "type": "array"
                      },
                      "publication_date": {
                          "format": "date-time",
                          "title": "publication_date",
                          "type": "string"
                      },
                      "title": {
                          "title": "title",
                          "type": "string"
                      }
                  },
                  "required": [
                      "title",
                      "publication_date"
                  ],
                  "title": "Book",
                  "type": "object"
              },
              "Illustration": {
                  "additionalProperties": false,
                  "properties": {
                      "data": {
                          "format": "data-url",
                          "title": "data",
                          "type": "string"
                      }
                  },
                  "required": [
                      "data"
                  ],
                  "title": "Illustration",
                  "type": "object"
              }
          }
      }
      

      We now have an additional illustrates property modelled as an array of #/definitions/Illustration, the later also added the the document as an additional definition entry.

    Conclusion

    This post illustrated how a basic (CRUD) HTTP API based on JSON Schema could be build for a CubicWeb application using cubicweb-jsonschema. We have seen a couple of details on JSON Schema generation and how it can be controlled. Feel free to comment and provide feedback on this feature set as well as open the discussion with more use cases.

    Next time, we'll discuss how hypermedia controls can be added the HTTP API that cubicweb-jsonschema provides.

    [1]this choice is essentially driven by simplicity and conformance when the existing behavior to help migration of existing applications.

show 141 results