Blog entries

  • CubicWeb sprint in Paris / Need for Speed

    2011/03/22 by Adrien Di Mascio

    Logilab is hosting a CubicWeb sprint - 3 days in our Paris offices.

    The general focus will be on speed :

    • on cubicweb-server side : improve performance of massive insertions / deletions
    • on cubicweb-client side : cache implementation, HTTP server, massive parallel usage, etc.

    This sprint will take place from in April 2011 from tuesday the 26th to thursday the 28th. You are more than welcome to come along and help out, contribute, but unlike previous sprints, at least basic knowledge of CubicWeb will be required for participants since no introduction is planned.

    Network resources will be available for those bringing laptops.

    Address : 104 Boulevard Auguste-Blanqui, Paris. Ring "Logilab" (googlemap)

    Metro : Glacière

    Contact : http://www.logilab.fr/contact

    Dates : 26/04/2011 to 28/04/2011


  • Summer CubicWeb/Narval Sprint - Final report

    2010/08/18 by Sylvain Thenault

    For that last sprint day, each team made some nice achievements:

    • Steph & Alain worked on the mv/cp actions implementation to makes them working properly and supporting globs. Last but not least, with a full set of tests.
    • Alex & Charles got back what we call apycot 'full' tests, eg running test with coverage enabled, checking that code coverage is greater than a given threshold, but also running pylint and checking that its global evaluation is at least 7 (configurable, of course).
    • Katia & Aurélien provided a sharp implementation of recipe checking, so that we know we don't launch a recipe badly constructed, as well as informing the user nicely from what errors his recipe suffer.
    • Julien managed to set up a recipe managing from Debian package construction to Debian repository publication, going through lintian on the way
    • Pierre-Yves helped other teams to solve the narval related bugs they encountered, and finished by writing a thread-safe implementation of apycot's writer so we can run several checker simultaneously.
    • Celso continued working on a proof of concept blue-theme cube, wondering how to make CubicWeb looks nicer and be easily customisable in future versions.
    • Sylvain helped there and there and integrated patches...

    So we finally didn't get up to the demo. But we now have everything to set it up, so I've a good hope that we will have a beta version of our brand new production chain up and running before the end of August!

    Thanks to everyone for all this good work, and for this time spent all together!


  • What's new in CubicWeb 3.18

    2014/01/10 by Aurelien Campeas

    The migration script does not handle sqlite nor mysql instances.

    New functionalities

    • add a security debugging tool (see #2920304)
    • introduce an add permission on attributes, to be interpreted at entity creation time only and allow the implementation of complex update rules that don't block entity creation (before that the update attribute permission was interpreted at entity creation and update time) (see #2965518)
    • the primary view display controller (uicfg) now has a set_fields_order method similar to the one available for forms
    • new method ResultSet.one(col=0) to retrieve a single entity and enforce the result has only one row (see #3352314)
    • new method RequestSessionBase.find to look for entities (see #3361290)
    • the embedded jQuery copy has been updated to version 1.10.2, and jQuery UI to version 1.10.3.
    • initial support for wsgi for the debug mode, available through the new wsgi cubicweb-ctl command, which can use either python's builtin wsgi server or the werkzeug module if present.
    • a rql-table directive is now available in ReST fields
    • cubicweb-ctl upgrade can now generate the static data resource directory directly, without a manual call to gen-static-datadir.

    API changes

    • not really an API change, but the entity write permission checks are now systematically deferred to an operation, instead of a) trying in a hook and b) if it failed, retrying later in an operation
    • The default value storage for attributes is no longer String, but Bytes. This opens the road to storing arbitrary python objects, e.g. numpy arrays, and fixes a bug where default values whose truth value was False were not properly migrated.
    • symmetric relations are no more handled by an rql rewrite but are now handled with hooks (from the activeintegrity category); this may have some consequences for applications that do low-level database manipulations or at times disable (some) hooks.
    • unique together constraints (multi-columns unicity constraints) get a name attribute that maps the CubicWeb contraint entities to the corresponding backend index.
    • BreadCrumbEntityVComponent's open_breadcrumbs method now includes the first breadcrumbs separator
    • entities can be compared for equality and hashed
    • the on_fire_transition predicate accepts a sequence of possible transition names
    • the GROUP_CONCAT rql aggregate function no longer repeats duplicate values, on the sqlite and postgresql backends

    Deprecation

    • pyrorql sources have been deprecated. Multisource will be fully dropped in the next version. If you are still using pyrorql, switch to datafeed NOW!
    • the old multi-source system
    • find_one_entity and find_entities in favor of find (see #3361290)
    • the TmpFileViewMixin and TmpPngView classes (see #3400448)

    Deprecated Code Drops

    • ldapuser have been dropped; use ldapfeed now (see #2936496)
    • action GotRhythm was removed, make sure you do not import it in your cubes (even to unregister it) (see #3093362)
    • all 3.8 backward compat is gone
    • all 3.9 backward compat (including the javascript side) is gone
    • the twisted (web-only) instance type has been removed

    For a complete list of tickets, read CubicWeb 3.18.0.


  • CubicWeb 3.6 is (almost) out!

    2010/02/10 by Sylvain Thenault

    And that's great news, after several months of development (things started moving in the beginning of august 2009...), it should be available on our Debian repositories and ftp site in the next few hours.

    So, we can say this release contains a (too) large set of improvements and refactorings. I'll talk about the most important ones here.

    Appobject/Entity classes namespace cleanup

    First of all, the namespace cleanup... 3.6 is a step towards cleaning the entity classes (hence more generally appobject), which are used for a lot of things, making it impossible to tell for sure what could be used or not as an attribute or relation name. We decided to declare identifiers starting with \_cw or cw\_ reserved for the core classes. A lot of methods have been deprecated to cleanup the base appobject class namespace. The remaining methods on entity classes will be removed in future version, by the introduction of an ORM for database related methods, and by the (most probable) introduction of ZCA adapters for other aspects. The most notable renaming are:

    • .req -> ._cw
    • .rset -> .cw_rset
    • .row -> .cw_row
    • .col -> .cw_col

    This is probably what you'll see first when upgrading to 3.6: a huge stack of deprecation warnings on your screen :)

    Another step towards a nice and powerful form system

    • cleaner reponsibilities separation between form, field and widget

    • fields and widgets are now responsible for handling POSTed values (the editcontroller was handling this, making things really unflexible). The editcontroller has been rewritten and now properly gets values from fields. Another benefit is that you can now easily have a widget handling multiple inputs (see the new datetime picker for instance, or the custom widget for Bookmark.path)

    • refactored automatic forms:

      • rewrite 'generic relations' as a field
      • inlined forms are now encapsulated into a field

      so you get much more control on these parts of automatic forms by using mechanism provided generally by fields

      • clearer form relations tags: removed autoform_is_inlined, more understandable autoform_field_section

    Hooks refactoring

    Hooks are now regular appobjects, with selectors (don't forget to reuse Hook.__select__, remember that !). They should simply implement __call__ with no argument (well, only self) and will get info previously passed as argument as instance attributes, according to the matching event.

    Test API cleanup

    EnvBasedTC, ControllerTC, WebTest, RepoBasedTC are all gone. Simply use CubicWebTC, with an unified API similar to what you use in cubicweb-ctl shell and in usual development.

    The Bytes File System Storage

    You can now specify a custom storage for attributes of entities stored in the system source. This mechanism is used to provide a way to store Bytes attributes (such as File.data for instance) as files on the file-system instead of BLOBs in the database. You can configure which attributes should use this storage for your instance and then everything is transparent.

    Schema definition changes (yams 0.27)

    In your schema definition file:

    • "symetric" should be correctly spelled "symmetric" :)
    • "permissions" was renamed to "__permissions__"

    Also, permissions for relations are now supported per definition, not per type, at the cost of a visible impact when writing/reading the schema.

    Note about backward compatibility

    We worked hard to keep backward compatibility, but you shouldn't upgrade to 3.6 without checking that everything is fine... Check notably:

    • forms, if you're using custom forms by overriding internal methods
    • import for date functions from cubicweb.utils (they moved to logilab.common.date)

    And also

    CubicWeb 3.6 comes with a set of 37 cubes "3.6"-ready to avoid too much warnings!

    Enjoy!


  • What's new in CubicWeb 3.16

    2013/01/23 by Aurelien Campeas

    What's new in CubicWeb 3.16?

    New functionalities

    • Add a new dataimport store (SQLGenObjectStore). This store enables a fast import of data (entity creation, link creation) in CubicWeb, by directly flushing information in SQL. This may only be used with PostgreSQL, as it requires the 'COPY FROM' command.

    API changes

    • Orm: set_attributes and set_relations are unified (and deprecated) in favor of cw_set that works in all cases.

    • db-api/configuration: all the external repository connection information is now in an URL (see #2521848), allowing to drop specific options of pyro nameserver host, group, etc and fix broken ZMQ source. Configuration related changes:

      • Dropped 'pyro-ns-host', 'pyro-instance-id', 'pyro-ns-group' from the client side configuration, in favor of 'repository-uri'. NO MIGRATION IS DONE, supposing there is no web-only configuration in the wild.
      • Stop discovering the connection method through repo_method class attribute of the configuration, varying according to the configuration class. This is a first step on the way to a simpler configuration handling.

      DB-API related changes:

      • Stop indicating the connection method using ConnectionProperties.
      • Drop _cnxtype attribute from Connection and cnxtype from Session. The former is replaced by a is_repo_in_memory property and the later is totaly useless.
      • Turn repo_connect into _repo_connect to mark it as a private function.
      • Deprecate in_memory_cnx which becomes useless, use _repo_connect instead if necessary.
    • the "tcp://" uri scheme used for ZMQ communications (in a way reminiscent of Pyro) is now named "zmqpickle-tcp://", so as to make room for future zmq-based lightweight communications (without python objects pickling).

    • Request.base_url gets a secure=True optional parameter that yields an https url if possible, allowing hook-generated content to send secure urls (e.g. when sending mail notifications)

    • Dataimport ucsvreader gets a new boolean ignore_errors parameter.

    Unintrusive API changes

    • Drop of cubicweb.web.uicfg.AutoformSectionRelationTags.bw_tag_map, deprecated since 3.6.

    User interface changes

    • The RQL search bar has now some auto-completion support. It means relation types or entity types can be suggested while typing. It is an awesome improvement over the current behaviour !
    • The action box associated with table views (from tableview.py) has been transformed into a nice-looking series of small tabs; it means that the possible actions are immediately visible and need not be discovered by clicking on an almost invisible icon on the upper right.
    • The uicfg module has moved to web/views/ and ui configuration objects are now selectable. This will reduce the amount of subclassing and whole methods replacement usually needed to customize the ui behaviour in many cases.
    • Remove changelog view, as neither cubicweb nor known cubes/applications were properly feeding related files.

    Other changes

    • 'pyrorql' sources will be automatically updated to use an URL to locate the source rather than configuration option. 'zmqrql' sources were broken before this change, so no upgrade is needed...
    • Debugging filters for Hooks and Operations have been added.
    • Some cubicweb-ctl commands used to show the output of msgcat and msgfmt; they don't anymore.

  • CubicWeb 3.6 sprint report

    2009/12/14 by Sylvain Thenault

    Last week we held a cubicweb sprint in our new Paris office !

    We were a nice number of people: 7 from the Logilab's crew, including Sandrine, our US representative, Celso and Carlos from Mexico, plus some others guests and colleagues working on (cubicweb based of course) customer projects.

    The objective of the sprint was to kick out the 3.6 version of cubicweb, a big refactoring release started by Adrien and I a few months ago. Unfortunatly we had been preempted by some other projects and the cubicweb development branch was simply painfully following changes done in the stable branch.

    Also, we decided to start using mq as a basis for code review. The sprint was a nice opportunity to test and see if it was actually usable for both developer and code reviewer. But more on this latter :)

    The tasks to achieve to get this release out were:

    1. resurrect the default branch after 3 months of nasty bugs introduced by simply merging from the stable branch without any time to test
    2. update main cubes to the new test / uicfg / hooks / members api
    3. finish the editcontroller (which handle post of most web forms) refactoring
    4. finish the relation permissions change, including migration
    5. update the documentation
    6. test real applications

    Of course this was ambitious :) Among those point 0. and 1. and 3. took us much more time than I expected. The editcontroller work (2.) has not been finished yet, and we didn't find any time for the documentation (4.).

    Besides this, everyone (well, me at least ;) enjoyed its time while working hard all together in our new meeting room! The 3.6 version still needs a little work before being released, but the development branch is definitly back, with a great bunch of cubes ready. Among them : comment, tag, blog, keyword, tracker, forge, card, nosylist, etc...

    So many thanks to everyone, and particularly to our Mexican friends Carlos and Celso... Tequila! ;)

    By the way the good news is that we plan to do more sprints like this now that we've some room for it!


  • CubicWeb 3.7 released

    2010/03/19

    Hi there !

    I'm pleased to announce the 3.7 release of CubicWeb, after a much shorter development cycle than for the 3.6...

    But it still have some interesting changes:

    • NOW DEPENDS ON PYTHON 2.5
    • use the newly created logilab.database package (you'll have to install it as well as upgrade logilab.common and rql)
    • proper behaviour on the repository side of cubiweb:
      • dropped unsafe_execute, execute is now unsafe by default in hooks and operations. You can still explicitly control security using the enabled_secury context manager
      • proper transaction hooks control using the hooks_control context manager
    • started some transaction undo support (only undo of deletion supported right now)
    • various other bug fixes and improvments

    Notice the 3.6 branch will still be maintained for some time.

    Enjoy!


  • Running CubicWeb on Windows

    2009/09/08

    This was not supported (and still isn't officially). But rumors have been circulating about a port of CubicWeb on Windows.

    http://pc-astuces.seebz.net/images/logo-windows-small.jpg

    I can confirm that there is some truth in this. A few changesets have been circulating, to be merged in the official repository any time now, enabling one to run CubicWeb on Windows. Support for running CubicWeb as a Windows service is not available yet, but should become available in the next few weeks.

    Update: check out the source of the 3.5 branch, and you will be able to create and start instances on Windows. Use cubicweb-ctl start -D for non daemon start.


  • Summer CubicWeb/Narval Sprint - Day 4

    2010/08/13 by Pierre-Yves David

    In this fourth day of the our Summer Sprint important progress have been made.

    • Stéphanie and Alain cleaned up the Apycot's bot sources from deprecated code and rewrite part of the test suite to follow the new way to launch apycot. They cleaned up the handling of VCS sources for tested project taking advantages of the new mercurial cache for vcsfile implemented by Katia and Aurélien last Tuesday. This feature keep a local clone of the remote repository and allow much faster checkout during test runs.
    • Julien made significant progress in the writing of the Debian recipe. A recipes can now successfully build Debian packages of a project and validate them with lintian and lgp. He later paired with Pierre-Yves and they improved the annotation of Apycot's Narval variable to enhance Input validation in Apycot's Narval recipes. For example, the action building a Debian package will explicitly refuse to run on a project not yet checked-out.
    • Aurelien first paired with Pierre-Yves to improve some views and the consistency of the database schema, then he worked on a dashboard displaying various indicators useful to the version publishing process.
    • Pierre-Yves spent some time improving the ability of Narval to recover on errors and to display meaningful logs about them.
    • Alexandre and Charles finished the re-implementation of the full python recipe.They used options at the Narval level to run test suite with the coverage enabled and re-enabled the coverage checker to process the result, discovering some problems in Narval's engine on the way...
    • Celso finished Spanish translation of Cubicweb's core and started to work on a new css theme
    • Sylvain helped several groups along the day and reviewed patches from them.

  • Summer CubicWeb/Narval Sprint - Day 2

    2010/08/11 by Katia Saurfelt

    During the second day of our Summer CubicWeb/Narval Sprint, several tasks started on the first day were completed and new tasks started:

    • Charles, Alexandre and Julien finished writing the "copy" and "move" Narval actions, and then started transforming existing apycot checkers into Narval actions.
    • Pierre-Yves managed to improve Narval reports with more explicit and relevant content.
    • Stéphanie and Alain finished the bot status view as well as the recipe graph view.
    • Katia and Aurélien finished writing the new mercurial cache solution for vcsfile and started improving the security of Narval recipes (i.e. who can start which recipe).
    • Celso kept on his life-long work of translating CubicWeb to Spanish.
    • Sylvain wrote some Narval views, improved Narval execution logs handling and kept on reviewing patches and helping various people...

  • CubicWeb sprint in Paris on january 19/20/21 2011

    2010/12/03 by Sylvain Thenault
    http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

    Almost everything is in the title: we'll hold a CubicWeb sprint in our Paris office after the first French Semantic Web conference, so on 19, 20 and 21 of january 2011.

    The main topic will be to enhance newcomers experience in installing and using CubicWeb.

    If you wish to come, you're welcome, that's a great way to meet us, learn the framework and share thoughts about it. Simply contact us so we can check there is still some room available.

    photo by Sebastian Mary under creative commons licence.


  • Summer CubicWeb/Narval Sprint - Day 3

    2010/08/13

    CubicWeb/Narval Sprint is going on !

    The third day of our sprint focused on the following points:

    • Pierre-Yves worked to prevent duplicate test executions (eg running several time the same test with the same version configuration),
    • Celso has terminated the spanish translation of CubicWeb. He's now working on various cubes translation,
    • Stéphanie and Alain spent some time on the narval bot view. They also modified ProjectEnvironement's attributes in order to use similar information available on the vcsfile repository, hence simplifying the configuration (more to do on this!),
    • Julien worked on the debian package recipe,
    • Katia and Aurélien worked on recipe security (using CWPermission),
    • Alexandre and Charles produced a first template of a full test recipe using pyunit and pycoverage,
    • Finally, our captain, Sylvain, is at the helm !

    We'll hopefuly be able to present a functionnal demo at the end of the week.

    Narval/Cubicweb left off !


  • Summer CubicWeb/Narval Sprint - Day 1

    2010/08/10 by Sylvain Thenault

    We started this first day by several presentations by Sylvain about Logilab's current development process workflow, and compared it to what it should be after the sprint. Sylvain also introduced Narval.

    We then set up a dev environment on everyone's computer: a working forge with a local Narval agent that can be used for tests during the week.

    Regarding more concrete tasks:

    • Charles and Alexandre started writing some basic Narval actions such as move, to move a file from a place to another, and had to grasp narval's concepts on the way.
    • Pierre-Yves dug into the code to understand how exceptions are propagated in the Narval engine, his goal was to get better reports.
    • Stéphanie and Alain worked on a nice bot status view.
    • Katia, Aurélien studied the new mercurial cache solution for vcsfile
    • Julien started some piece of documentation.
    • Celso, our Mexican friend, discovered some new features of recent cubicweb releases and setup his environment to later work on Spanish translation, CSS, etc.
    • Sylvain came with a basically working narval implementation on top of cubicweb, and spent the day helping various people...

  • CubicWeb 3.9 released

    2010/07/12 by Sylvain Thenault

    CubicWeb 3.9.0 went out last week. We now have tested it in production and fixed the remaining bugs, which means it is now show time!

    http://www.cubicweb.org/file/1179905?vid=download

    What's new in CubicWeb 3.9?

    The 3.9 release development was started by a one week long sprint at the beginning of May. The two goals were first to make it easier to customize the look and feel of a CubicWeb application, and second to do a big cleanup of the javascript library. This led to the following major changes.

    • We introduced property sheets, which replace former external_resources file, as well as define some constants that will be used to 'compile' cubicweb and cubes' stylesheets.
    • We started a new, clean cubicweb.css stylesheet, that tries to keep up with the rhythm. This is still a work in progress, and by default the old css is still used, unless specified otherwise in the configuration file.
    • We set the bases for web functional testing using windmill. See test cases in cubicweb/web/test/windmill/ and python wrapper in cubicweb/web/test_windmill/ if you want to use this in your own cube.
    • We set the bases for javascript unit-testing using qunit. See test cases in cubicweb/web/test/jstests/ and python wrapper in cubicweb/web/test_jscript/ if you want to use this in your own cube.
    • We cleaned the javascript code: the generic stuff moved into the cw namespace, the ajax api is now much simpler thanks to more generic and powerful functions. As usual backward compatibility was kept, which means that your existing code will still run, but you will see tons of deprecation warnings in the firebug console.
    • We implemented a simple documentation extraction system for javascript. Just put ReST in javascript comments, and get all the power of sphinx for documenting your javascript code.

    But that's not all! There are also two major changes in 3.9.

    http://www.cubicweb.org/file/1179904?vid=download

    Architectural change: adapters

    The first major change is the introduction of adapters, also found in the Zope Component Architecture and documented in the GoF book. This will allow for better application design and easier code reuse. You can see several usage in the framework, for instance the "ITree" adapter in cubicweb.entities.adapters, the "IBreadCrumbs" adapter in cubicweb.web.views.ibreadcrumbs, or still the "ICalendarable" adapter in cubicweb.web.views.calendar.

    Important full search improvement

    The second major change will benefit directly to end users: we worked with our friends from SecondWeb to expose the ranking feature found in postgres full-text search. This clearly improves the user experience when doing full-text searches. Ranking may be finely tuned by setting different weights to entity types, entity types attributes, or even be dynamically computed per entity instance. Of course, all this is done in an adapter, see "IFTIndexableAdapter" in cubicweb/entities/adapters.py.

    Minor changes

    Other minor changes include:

    • support for wildcard text search for application using postgres >= 8.4 as backend. Try searching for 'cub*' on cubicweb.org for instance.
    • inline edition of composite relation
    • nicer, clickable, schema image of the data model
    • enhanced support for the SQLserver database

    Enjoy!


  • CubicWeb 3.8 released

    2010/04/28 by Sylvain Thenault

    CubicWeb 3.8.0 went out last week, but now we have tested it, produced a 3.8.1, it's show time!

    What's new in CubicWeb 3.8?

    One of the most important change is http server update to move from deadend twisted.web2 to twisted.web. With this change comes the possibility to configure the maximum size of POST request in the configuration file (was hard-coded to 100Mo before).

    Other changes include:

    • CubicWeb should now be installable through pip or easy_install. This is still experimental, and we don't use it that much so please, give us some feedback! Some cubes are now also "pipable" (comment, blog...), but more will come with new releases.
    • .execute() function lost its cache key argument. This is great news since it was a pain to explain and most cubicweb users didn't know how to handle it well (and I'm thre greatest beneficer since I won't have to explain over and over again)
    • nicer schema and workflow views
    • refactored web session handling, which should now be cleaner, clearer, hence less buggy...
    • nicer skeleton generation for new cubes, cleaner __pkginfo__ (you don't have to define both __depends__ / __depends_cubes__ or __recommends__ / __recommends_cubes__ in the general case, and other cleanups)

    Enjoy!


  • What's new in CubicWeb 3.13?

    2011/07/21 by Sylvain Thenault

    CubicWeb 3.13 has been developed for a while and includes some cool stuff:

    • generate and handle Apache's modconcat compatible URLs, to minimize the number of HTTP requests necessary to retrieve JS and CSS files, along with a new cubicweb-ctl command to generate a static 'data' directory that can be served by a front-end instead of CubicWeb
    • major facet enhancements:
      • nicer layout and visual feedback when filtering is in-progress
      • new RQLPathFacet to easily express new filters that are more than one hop away from the filtered entities
      • a more flexibile API, usable in cases where it wasn't previously possible
    • some form handling refactorings and cleanups, notably introduction of a new method to process posted content, and updated documentation
    • support for new base types : BigInt, TZDateTime and TZTime (in 3.12 actually for those two)
    • write queries optimization, and several RQL fixes on complex queries (e.g. using HAVING, sub-queries...), as well as new support for CAST() function and REGEXP operator
    • datafeed source and default CubicWeb xml parsers:
      • refactored into smaller and overridable chunks
      • easier to configure
      • make it work

    As usual, the 3.13 also includes a bunch of other minor enhancements, refactorings and bug fixes. Please download and install CubicWeb 3.13 and report any problem on the tracker and/or the mailing-list!

    Enjoy!


  • CubicWeb roadmap meeting on November 6th, 2014

    2014/11/03 by Nicolas Chauvat

    The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. The previous roadmap meeting was in September 2014.

    Here is the report about the November 6th, 2014 meeting. Christophe de Vienne (Unlish) joined us to express their concerns and discuss the future of CubicWeb. Dimitri Papadopoulos (CEA) could not come.

    Versions

    Version 3.17

    This version is stable but old and maintainance will continue only as long as some customers will be willing to pay for it (current is 3.17.17).

    If you're still using 3.17, you should go directly to 3.19.

    Version 3.18

    This version is stable but old and maintained (current is 3.18.6).

    Version 3.19

    This version is stable and maintained (current is 3.19.5).

    Version 3.20

    This version is still under development but should be released very soon now (expected next week). Its main feature being the inclusion of CWEP-002 (computed attributes and relations), along with many small improvement patches.

    For details read list of tickets for CubicWeb 3.20.0.

    We would have loved to integrate the pyramid cube in this release, but the debian packaging effort needed by the pyramid stack is quite big and is acceptable if we target jessie only (at decent price).

    Version 3.21

    For now, the roadmap for 3.21 is still the complete removal of the dbapi, the merging of Connection and ClientConnection, and possibly including CWEP-003 (adding a FROM clause to RQL).

    Integrate the pyramid cube to provide the pyramid command if the pyramid framework can be imported.

    Integration of CWEP-004 is being discussed.

    Version 4.0

    We expect to accelerate development of CubicWeb 4, which exact roadmap is still to be discussed, but we may already want:

    • be pyramid-based (remove twisted, auth management, etc.),
    • do not have anything left of old dbapi and ClientConnection,
    • integrate squareui as main (and only) web-ui "template" or remove web generation (almost) completely from cubicweb-core and provide it only through the cube system.

    CWEPs

    Here is the status of open CubicWeb Evolution Proposals:

    to be written

    Work in progress

    Some work is in progress around CKAN, DCAT and othr Open Data and Semantic Web related technologies.

    Agenda

    Next roadmap meeting will be held at the beginning of january 2015 at Logilab, and Christophe and Dimitri (or Yann) are invited.

    Open Discussions

    Migration:

    • AppObjects should not be loaded by default
    • Have a look at Alembic the migration tool for SQLAlchemy and take inspiration from there

  • "Data Fast-food": quick interactive exploratory processing and visualization of complex datasets with CubicWeb

    2012/01/19 by Vincent Michel

    With the emergence of the semantic web in the past few years, and the increasing number of high quality open data sets (cf the lod diagram), there is a growing interest in frameworks that allow to store/query/process/mine/visualize large data sets.

    We have seen in previous blog posts how CubicWeb may be used as an efficient knowledge management system for various types of data, and how it may be used to perform complex queries. In this post, we will see, using Geonames data, how CubicWeb may perform simple or complex data mining and machine learning procedures on data, using the datamining cube. This cube adds powerful tools to CubicWeb that make it easy to interactively process and visualize datasets.

    At this point, it is not meant to be used on massive datasets, for it is not fully optimized yet. If you try to perform a TF-IDF (term frequency–inverse document frequency) with a hierarchical clustering on the full dbpedia abstracts dataset, be prepared to wait. But it is a promising way to enrich the user experience while playing with different datasets, for quick interactive exploratory datamining processing (what I've called the "Data fast-food"). This cube is based on the scikit-learn toolbox that has recently gained a huge popularity in the machine learning and Python community. The release of this cube drastically increases the interest of CubicWeb for data management.

    The Datamining cube

    For a given query, similarly to SQL, CubicWeb returns a result set. This result set may be presented by a view to display a table, a map, a graph, etc (see documentation and previous blog posts).

    The datamining cube introduces the possibility to process the result set before presenting it, for example to apply machine learning algorithms to cluster the data.

    The datamining cube is based on two concepts:

    • the concept of processor: basically, a processor transforms a result set in a numpy array, given some criteria defining the mathematical processing, and the columns/rows of the result set to be taken into account. The numpy-array is a polyvalent structure that is widely used for numerical computation. This array could thus be efficiently used with any kind of datamining algorithms. Note that, in our context of knowledge management, it is more convenient to return a numpy array with additional meta-information, such as indices or labels, the result being stored in what we call a cw-array. Meta-information may be useful for display, but is not compulsory.
    • the concept of array-view: the "views" are basic components of CubicWeb, distinguish querying and displaying the data is key in this framework. So, on a given result set, many different views can be applied. In the datamining cube, we simply overload the basic view of CubicWeb, so that it works with cw-array instead of result sets. These array-views are associated to some machine learning or datamining processes. For example, one can apply the k-means (clustering process) view on a given cw-array.

    A very important feature is that the processor and the array-view are called directly through the URL using the two related parameters arid (for ARray ID) and vid (for View ID, standard in CubicWeb).

    http://www.cubicweb.org/file/2154793?vid=download

    Processors

    We give some examples of basic processors that may be found in the datamining cube:

    • AttributesAsFloatArrayProcessor (arid='attr-asfloat'): This processor turns all Int, BigInt and Float attributes in the result set to floats, and returns the corresponding array. The number of rows is equal to the number of rows in the result set, and the number of columns is equal to the number of convertible attributes in the result set.
    • EntityAsFloatArrayProcessor (arid='entity-asfloat'): This processor performs similarly to the AttributesAsFloatArrayProcessor, but keeps the reference to the entities used to create the numpy-array. Thus, this information could be used for display (map, label, ...).
    • AttributesAsTokenArrayProcessor (arid='attr-astoken'): This processor turns all String attributes in the result set in a numpy array, based on a Word-n-gram analyze. This may be used to tokenize a set of strings.
    • PivotTableCountArrayProcessor (arid='pivot-table-count'): This processor is used to create a pivot table, with a count function. Other functions, such as sum or product also exist. This may be used to create some spreadsheet-like views.
    • UndirectedRelationArrayProcessor (arid='undirected-rel'): This processor creates a binary numpy array of dimension (nb_entities, nb_entities), that represents the relations (or corelations) between entities. This may be used for graph-based vizualisation.

    We are also planning to extend the concept of processor to sparse matrix (scipy.sparse), in order to deal with very high dimensional data.

    Array Views

    The array views that are found in the datamining cube, are, for most of them, used for simple visualization. We used HTML-based templates and the Protovis Javascript Library.

    We will not detail all the views, but rather show some examples. Read the reference documentation for a complete and detailed description.

    Examples on numerical data

    Histogram

    The request:

    Any LO, LA WHERE X latitude LA, NOT X latitude NULL, X longitude LO,  NOT X longitude NULL,
    X country C, NOT X elevation NULL, C name "France"
    

    that may be translated as:

    All couples (latitude, longitude) of the locations in France, with an elevation not null
    

    and, using vid=protovis-hist and arid=attr-asfloat

    http://www.cubicweb.org/file/2154795?vid=download

    Scatter plot

    Using the notion of view, we can display differently the same result set, for example using a scatter plot (vid=protovis-scatterplot).

    http://www.cubicweb.org/file/2156233?vid=download

    Another example with the request:

    Any P, E WHERE X is Location, X elevation E, X elevation >1, X population P,
    X population >10, X country CO, CO name "France"
    

    that may be translated as:

    All couples (population, elevation) of locations in France,
    with a population higher than 10 (inhabitants),and an elevation higher than 1 (meter)
    

    and, using the same vid (vid=protovis-scatterplot) and the same arid (arid=attr-asfloat)

    http://www.cubicweb.org/file/2154802?vid=download

    If a third column is given in the result set (and thus in the numpy array), it will be encoded in the size/color of each dot of the scatter plot. For example with the request:

    Any LO, LA, E WHERE X latitude LA, NOT X latitude NULL, X longitude LO,  NOT X longitude NULL,
    X country C, NOT X elevation NULL, X elevation E, C name "France"
    

    that may be translated as:

    All tuples (latitude, longitude, elevation) of the locations in France, with an elevation not null
    

    and, using the same vid (vid=protovis-scatterplot) and the same arid (arid=attr-asfloat), we can visualize the elevation on a map, encoded in size/color

    http://www.cubicweb.org/file/2154805?vid=download

    Another example with the request:

    Any LO, LA LIMIT 50000 WHERE X is Location, X population  >1000, X latitude LA, X longitude LO,
    X country CO, CO name "France"
    

    that may be translated as:

    All couples (latitude, longitude) of 50000 locations in France, with a population higher than 100 (inhabitants)
    
    http://www.cubicweb.org/file/2156095?vid=download

    There also exist some AreaChart view, LineArray view, ...

    Examples on relational data

    Relational Matrix (undirected graph)

    The request:

    Any X,Y WHERE X continent CO, CO name "North America", X neighbour_of Y
    

    that may be translated as:

    All neighbour countries in North America
    

    and using the vid='protovis-binarymap' and arid='undirected-rel'

    http://www.cubicweb.org/file/2154796?vid=download

    Relational Matrix (directed graph)

    If we do not want a symmetric matrix, i.e. if we want to keep the direction of a link (X,Y is not the same relation as Y,X), we can use the directed*rel array processor. For example, with the following request:

    Any X,Y LIMIT 20 WHERE X continent Y
    

    that may be translated as:

    20 countries and their continent
    

    and using the vid='protovis-binarymap' and arid='directed-rel'

    http://www.cubicweb.org/file/2154797?vid=download

    Force directed graph

    For a dynamic representation of relations, we can use a force directed graph. The request:

    Any X,Y WHERE X neighbour_of Y
    

    that may be translated as:

    All neighbour countries in the World.
    

    and using the vid='protovis-forcedirected' and arid='undirected-rel', we can see the full graph, with small independent components (e.g. UK and Ireland)

    http://www.cubicweb.org/file/2154800?vid=download

    Again, a third column in the result set could be used to encode some labeling information, for example the continent.

    The request:

    Any X,Y,CO WHERE X neighbour_of Y, X continent CO
    

    that may be translated as:

    All neighbour countries in the World, and their corresponding continent.
    

    and again, using the vid='protovis-forcedirected' and arid='undirected-rel', we can see the full graph with the continents encoded in color (Americas in green, Africa in dark blue, ...)

    http://www.cubicweb.org/file/2154801?vid=download

    Dendrogram

    For hierarchical information, one can use the Dendrogram view. For example, with the request:

    Any X,Y WHERE X continent Y
    

    that may be translated as:

    All couple (country, continent) in the World
    

    and using vid='protovis-dendrogram' and arid='directed-rel', we have the following dendrogram (we only show a part due to lack of space)

    http://www.cubicweb.org/file/2154806?vid=download

    Unsupervised Learning

    We have also developed some machine learning view for unsupervised learning. This is more a proof of concept than a fully optimized development, but we can already do some cool stuff. Each machine learning processing is referenced by a mlid. For example, with the request:

    Any LO, LA WHERE X is Location, X elevation E, X elevation >1, X latitude LA, X longitude LO,
    X country CO, CO name "France"
    

    that may be translated as:

    All couples (latitude, longitude) of the locations in France, with an elevation higher than 1
    

    and using vid='protovis-scatterplot' arid='attr-asfloat' and mlid='kmeans', we can construct a scatter plot of all couples of latitude and longitude in France, and create 10 clusters using the kmeans clustering. The labeling information is thus encoded in color/size:

    http://www.cubicweb.org/file/2154804?vid=download

    Download

    Finally, we have also implement a download view, based on the Pickle of the numpy-array. It is thus possible to access remotely any data within a Python shell, allowing to process them as you want. Changing the request can be done very easily by changing the rql parameter in the URL. For example:

    import pickle, urllib
    data = pickle.loads(urllib.open('http://mydomain?rql=my request&vid=array-numpy&arid=attr-asfloat'))
    

  • CubicWeb sprint in Paris - 2012/02/07-10

    2011/12/21 by Nicolas Chauvat

    Topics

    To be decided. Some possible topics are :

    • optimization (still)
    • porting cubicweb to python3
    • porting cubicweb to pypy
    • persistent sessions
    • finish twisted / wsgi refactoring
    • inter-instance communication bus
    • use subprocesses to handle datafeeds
    • developing more debug-tools (debug console, view profiling, etc.)
    • pluggable / unpluggable external sources (as needed for the cubipedia and semantic family)
    • client-side only applications (javascript + http)
    • mercurial storage backend: see this thread of the mailing list
    • mercurial-server integration: see this email to the mailing list

    other ideas are welcome, please bring them up on cubicweb@lists.cubicweb.org

    Location

    This sprint will take place from in february 2012 from tuesday the 7th to friday the 10th. You are more than welcome to come along, help out and contribute. An introduction is planned for newcomers.

    Network resources will be available for those bringing laptops.

    Address : 104 Boulevard Auguste-Blanqui, Paris. Ring "Logilab" (googlemap)

    Metro : Glacière

    Contact : http://www.logilab.fr/contact

    Dates : 07/02/2012 to 10/02/2012


  • Cubicweb sprints winter/spring 2014

    2014/01/24 by David Douard

    The Logilab team is pleased to announce two Cubicweb sprints to be held in its Paris offices in the upcoming months:

    February 13/14th at Logilab in Paris

    The agenda would be the FROM clause for which a CWEP is still awaited, and the RQL rewriter according to the CWEP02.

    April 28/30th at Logilab in Paris

    Agenda to be defined.

    Join the party

    All users and contributors of CubicWeb are invited to join the party. Just send an email to contact at Logilab.fr if you plan to come.

    http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

  • What's new in CubicWeb 3.14?

    2011/11/10 by Sylvain Thenault

    The development of CubicWeb 3.14 was rather long and included a lot of API changes detailed here. As usual backward compatibility is provided for public APIs.

    Please note this release depends on yams 0.34 (which is incompatible with prior cubicweb releases regarding instance re-creation).

    API changes

    • Entity.fetch_rql the restriction argument has been deprecated and should be replaced with a call to the new Entity.fetch_rqlst method, get the returned value (a rql Select node) and use the RQL syntax tree API to include the above-mentioned restrictions.

      Backward compat is kept with proper warning.

    • Entity.fetch_order and Entity.fetch_unrelated_order class methods have been replaced by Entity.cw_fetch_order and Entity.cw_fetch_unrelated_order with a different prototype:

      • instead of taking (attr, var) as two string argument, they now take (select, attr, var) where select is the rql syntax tree being constructed and var the variable node.
      • instead of returning some string to be inserted in the 'ORDERBY' clause, it has to modify the syntax tree

      Backward compat is kept with proper warning, except if:

      • custom order method returns something else the a variable name with or without the sorting order (e.g. cases where you sort on the value of a registered procedure as it was done in the tracker for instance). In such case, an error is logged telling that this sorting is ignored until API upgrade.
      • client code uses direct access to one of those methods on an entity (no code known to do that).
    • Entity._rest_attr_info class method has been renamed to Entity.cw_rest_attr_info

      No backward compat since this is a protected method an no code is known to use it outside cubicweb itself.

    • AnyEntity.linked_to has been removed as part of a refactoring of this functionality (link a entity to another one at creation step). It was replaced by a EntityFieldsForm.linked_to property.

      In the same refactoring, cubicweb.web.formfield.relvoc_linkedto, cubicweb.web.formfield.relvoc_init and cubicweb.web.formfield.relvoc_unrelated were removed and replaced by RelationField methods with the same names, that take a form as a parameter.

      No backward compatibility yet. It's still time to cry for it. Cubes known to be affected: tracker, vcsfile, vcreview.

    • CWPermission entity type and its associated require_permission relation type (abstract) and require_group relation definitions have been moved to a new localperms cube. Some functions from the cubicweb.schemas package as well as some views where moved too. This makes cubicweb itself smaller while you get all the local permissions stuff into a single and documented place.

      Backward compat is kept for existing instances, though you should have installed the localperms cubes. A proper error should be displayed when trying to migrate to 3.14 an instance the use CWPermission without the new cube installed. For new instances / test, you should add a dependancy on the new cube in cubes using this feature, along with a dependancy on cubicweb >= 3.14.

    • jQuery has been updated to 1.6.4 and jquery-tablesorter to 2.0.5. No backward compat issue known.

    • Table views refactoring : new RsetTableView and EntityTableView, as well as rewritten an enhanced version of PyValTableView on the same bases, with logic moved to some column renderers and a layout. Those should be well documented and deprecates former TableView, EntityAttributesTableView and CellView, which are however kept for backward compat, with some warnings that may not be very clear unfortunatly (you may see your own table view subclass name here, which doesn't make the problem that clear). Notice that _cw.view('table', rset, *kwargs) will be routed to the new RsetTableView or to the old TableView depending on given extra arguments. See #1986413.

    • display_name don't call .lower() anymore. This may leads to changes in your user interface. Different msgid for upper/lower cases version of entity type names, as this is the only proper way to handle this with some languages.

    • IEditControlAdapter has been deprecated in favor of EditController overloading, which was made easier by adding dedicated selectors called match_edited_type and match_form_id.

    • Pre 3.6 API backward compat has been dropped, though data migration compatibility has been kept. You may have to fix errors due to old API usage for your instance before to be able to run migration, but then you should be able to upgrade even a pre 3.6 database.

    • Deprecated cubicweb.web.views.iprogress in favor of new iprogress cube.

    • Deprecated cubicweb.web.views.flot in favor of new jqplot cube.

    Unintrusive API changes

    • Refactored properties forms (eg user preferences and site wide properties) as well as pagination components to ease overridding.

    • New cubicweb.web.uihelper module with high-level helpers for uicfg.

    • New anonymized_request decorator to temporary run stuff as an anonymous user, whatever the currently logged in user.

    • New 'verbatimattr' attribute view.

    • New facet and form widget for Integer used to store binary mask.

    • New js_href function to generated proper javascript href.

    • match_kwargs and match_form_params selectors both accept a new once_is_enough argument.

    • printable_value is now a method of request, and may be given dict of formatters to use.

    • [Rset]TableView allows to set None in 'headers', meaning the label should be fetched from the result set as done by default.

    • Field vocabulary computation on entity creation now takes __linkto information into accounet.

    • Started a cubicweb.pylintext pylint plugin to help pylint analyzing cubes: you should now use

      pylint --load-plugins=cubicweb.pylintext
      

      to analyse your cubicweb code.

    RQL

    User interface changes

    • Datafeed source now present an history of the latest import's log, including global status and debug/info/warning/error messages issued during imports. Import logs older than a configurable amount of time are automatically deleted.
    • Breadcrumbs component is properly kept when creating an entity with '__linkto'.
    • users and groups management now really lead to that (i.e. includes groups management).
    • New 'jsonp' controller with 'jsonexport' and 'ejsonexport' views.

    Configuration

    • Added option 'resources-concat' to make javascript/css files concatenation optional, making JS debugging a lot easier when needed.

    As usual, the 3.14 also includes a bunch of other minor changes, and bug fixes, though this time an effort has been done so that every API changes / new API should be listed here. Please download and install CubicWeb 3.14 and report any problem on the tracker and/or the mailing-list!

    Enjoy!


  • What's new in CubicWeb 3.17

    2013/06/21 by Aurelien Campeas

    What's new in CubicWeb 3.17?

    New functionalities

    • add a command to compare db schema and file system schema (see #464991)
    • Add CubicWebRequestBase.content with the content of the HTTP request (see #2742453)
    • Add directive bookmark to ReST rendering (see #2545595)
    • Allow user defined final type (see #124342)

    API changes

    • drop typed_eid() in favour of int() (see #2742462)
    • The SIOC views and adapters have been removed from CubicWeb and moved to the sioc cube.
    • The web page embedding views and adapters have been removed from CubicWeb and moved to the embed cube.
    • The email sending views and controllers have been removed from CubicWeb and moved to the massmailing cube.
    • RenderAndSendNotificationView is deprecated in favor of ActualNotificationOp the new operation uses the more efficient data idiom.
    • Looping task can now have an interval <= 0. Negative interval disable the looping task entirely.
    • We now serve html instead of xhtml. (see #2065651)

    Deprecation

    • ldapuser has been deprecated. It will be removed in a future version. If you are still using ldapuser switch to ldapfeed NOW!
    • hijack_user has been deprecated. It will be dropped soon.

    Deprecated Code Drops

    • The progress views and adapters have been removed from CubicWeb. These classes were deprecated since 3.14.0. They are still available in the iprogress cube.
    • The part of the API deprecated since 3.7 was dropped.

  • CubicWeb using Postgresql at its best

    2014/02/08 by Nicolas Chauvat

    We had a chat today with a core contributor to Postgresql from whom we may buy consulting services in the future. We discussed how CubicWeb could get the best out of Postgresql:

    • making use of the LISTEN/NOTIFY mechanism built into PG could be useful (to warn the cache about modified items for example) and PgQ is its good friend;
    • views (materialized or not) are another way to implement computed attributes and relations (see CWEP number 002) and it could be that the Entities table is in fact a view of other tables;
    • implementing RQL as an in-database language could open the door to new things (there is PL/pgSQL, PL/Python, what if we had PL/RQL?);
    • Foreign Data Wrappers written with Multicorn would be another way to write data feeds (see LDAP integration for an example);
    • managing dates can be tricky when users reside in different timezones and UTC is important to keep in mind (unicode/str is a good analogy);
    • for transitive closures that are often needed when implementing access control policies with __permissions, Postgresql can go a long way with queries like "WITH ... (SELECT UNION ALL SELECT RETURNING *) UPDATE USING ...";
    • the fastest way to load tabular data that does not need too much pre-processing is to create a temporary table in memory, then COPY-FROM the data into that table, then index it, then write the transform and load step in SQL (maybe with PL/Python);
    • when executing more than 10 updates in a row, it is better to write into a temporary table in memory, then update the actual tables with UPDATE USING (let's check if the psycopg driver does that when executemany is called);
    • reaching 10e8 rows in a table is at the time of this writing the stage when you should start monitoring your db seriously and start considering replication, partition and sharding.
    • full-text search is much better in Postgresql than the general public thinks it is and recent developments made it orders of magnitude faster than tools like Lucene or Solr and ElasticSearch;
    • when dealing with complex queries (searching graphs maybe), an option to consider is to implement a specific data type, use it into a materialized view and use GIN or GIST indexes over it;
    • for large scientific data sets, it could be interesting to link the numpy library into Postgresql and turn numpy arrays into a new data type;
    • Oh, and one last thing: the object-oriented tables of Postgresql are not such a great idea, unless you have a use case that fits them perfectly and does not hit their limitations (CubicWeb's is_instance_of does not seem to be one of these).

    Hopin' I got you thinkin' :)

    http://developer.postgresql.org/~josh/graphics/logos/elephant.png

  • What's new in CubicWeb 3.19

    2014/05/05 by Aurelien Campeas

    New functionalities

    • implement Cross Origin Resource Sharing (CORS) (see #2491768)
    • system_source.create_eid can return a range of IDs, to reduce overhead of batch entity creation

    Behaviour Changes

    • The anonymous property of Session and Connection is now computed from the related user login. If it matches the anonymous-user in the config the connection is anonymous. Beware that the anonymous-user config is web specific. Therefore, no session may be anonymous in a repository only setup.

    New Repository Access API

    Connection replaces Session

    A new explicit Connection object replaces Session as the main repository entry point. A Connection holds all the necessary methods to be used server-side (execute, commit, rollback, call_service, entity_from_eid, etc...). One obtains a new Connection object using session.new_cnx(). Connection objects need to have an explicit begin and end. Use them as a context manager to never miss an end:

    with session.new_cnx() as cnx:
        cnx.execute('INSERT Elephant E, E name "Babar"')
        cnx.commit()
        cnx.execute('INSERT Elephant E, E name "Celeste"')
        cnx.commit()
    # Once you get out of the "with" clause, the connection is closed.
    

    Using the same Connection object in multiple threads will give you access to the same Transaction. However, Connection objects are not thread safe (hence at your own risks).

    repository.internal_session is deprecated in favor of repository.internal_cnx. Note that internal connections are now safe by default, i.e. the integrity hooks are enabled.

    Backward compatibility is preserved on Session.

    dbapi vs repoapi

    A new API has been introduced to replace the dbapi. It is called repoapi.

    There are three relevant functions for now:

    • repoapi.get_repository returns a Repository object either from an URI when used as repoapi.get_repository(uri) or from a config when used as repoapi.get_repository(config=config).
    • repoapi.connect(repo, login, **credentials) returns a ClientConnection associated with the user identified by the credentials. The ClientConnection is associated with its own Session that is closed when the ClientConnection is closed. A ClientConnection is a Connection-like object to be used client side.
    • repoapi.anonymous_cnx(repo) returns a ClientConnection associated with the anonymous user if described in the config.

    repoapi.ClientConnection replaces dbapi.Connection and company

    On the client/web side, the Request is now using a repoapi.ClientConnection instead of a dbapi.Connection. The ClientConnection has multiple backward compatible methods to make it look like a dbapi.Cursor and dbapi.Connection.

    Sessions used on the Web side are now the same as the ones used Server side. Some backward compatibility methods have been installed on the server side Session to ease the transition.

    The authentication stack has been altered to use the repoapi instead of the dbapi. Cubes adding new elements to this stack are likely to break.

    New API in tests

    All current methods and attributes used to access the repo on CubicWebTC are deprecated. You may now use a RepoAccess object. A RepoAccess object is linked to a new Session for a specified user. It is able to create Connection, ClientConnection and web side requests linked to this session:

    access = self.new_access('babar') # create a new RepoAccess for user babar
    with access.repo_cnx() as cnx:
        # some work with server side cnx
        cnx.execute(...)
        cnx.commit()
        cnx.execute(...)
        cnx.commit()
    
    with access.client_cnx() as cnx:
        # some work with client side cnx
        cnx.execute(...)
        cnx.commit()
    
    with access.web_request(elephant='babar') as req:
        # some work with web request
        elephant_name = req.form['elephant']
        req.execute(...)
        req.cnx.commit()
    

    By default testcase.admin_access contains a RepoAccess object for the default admin session.

    API changes

    • RepositorySessionManager.postlogin is now called with two arguments, request and session. And this now happens before the session is linked to the request.
    • SessionManager and AuthenticationManager now take a repo object at initialization time instead of a vreg.
    • The async argument of _cw.call_service has been dropped. All calls are now synchronous. The zmq notification bus looks like a good replacement for most async use cases.
    • repo.stats() is now deprecated. The same information is available through a service (_cw.call_service('repo_stats')).
    • repo.gc_stats() is now deprecated. The same information is available through a service (_cw.call_service('repo_gc_stats')).
    • repo.register_user() is now deprecated. The functionality is now available through a service (_cw.call_service('register_user')).
    • request.set_session no longer takes an optional user argument.
    • CubicwebTC does not have repo and cnx as class attributes anymore. They are standard instance attributes. set_cnx and _init_repo class methods become instance methods.
    • set_cnxset and free_cnxset are deprecated. The database connection acquisition and release cycle is now more transparent.
    • The implementation of cascading deletion when deleting composite entities has changed. There comes a semantic change: merely deleting a composite relation does not entail any more the deletion of the component side of the relation.
    • _cw.user_callback and _cw.user_rql_callback are deprecated. Users are encouraged to write an actual controller (e.g. using ajaxfunc) instead of storing a closure in the session data.
    • A new entity.cw_linkable_rql method provides the rql to fetch all entities that are already or may be related to the current entity using the given relation.

    Deprecated Code Drops

    • The session.hijack_user mechanism has been dropped.
    • EtypeRestrictionComponent has been removed, its functionality has been replaced by facets a while ago.
    • the old multi-source support has been removed. Only copy-based sources remain, such as datafeed or ldapfeed.

  • CubicWeb sprint in Paris - 2012/12/13-14

    2012/11/11 by Nicolas Chauvat

    Topics

    To be decided. Some possible topics are :

    • Work on CubicWeb front end : Anything related to Themaintemplate, primaryview, reledit, tables handling etc.
    • Share the Evolution and more integration of the OrbUI project for CW
    • Things to do for HTML5 and bootstrap integration
    • Work on ideas from Thoughts on CubicWeb 4
    • ...

    other ideas are welcome, please bring them up on cubicweb@lists.cubicweb.org

    Location

    This sprint will take place in decembre 2012 from thursday the 13th to friday the 14th. You are more than welcome to come along, help out and contribute. An introduction is planned for newcomers.

    Network resources will be available for those bringing laptops.

    Address : 104 Boulevard Auguste-Blanqui, Paris. Ring "Logilab" (googlemap)

    Metro : Glacière

    Contact : http://www.logilab.fr/contact

    Dates : 13/12/2012 to 14/12/2012

    Participants

    • Celso Flores (Crealibre - Mexico)
    • Carine Fourrier (Crealibre - Mexico)
    • ...

  • What's new in CubicWeb 3.15

    2012/05/14 by Sylvain Thenault

    CubicWeb 3.15 introduces a bunch of new functionalities. In short (more details below):

    • ability to use ZMQ instead of Pyro to connect to repositories
    • ZMQ inter-instances messages bus
    • new LDAP source using the datafeed approach, much more flexible than the legacy 'ldapuser' source
    • full undo support

    Plus some refactorings regarding Ajax function calls, WSGI, the registry, etc. Read more for the detail.

    New functionalities

    • Add ZMQ server, based on the cutting edge ZMQ socket library. This allows to access distant instances, in a similar way as Pyro.
    • Publish/subscribe mechanism using ZMQ for communication among cubicweb instances. The new zmq-address-sub and zmq-address-pub configuration variables define where this communication occurs. As of this release this mechanism is used for entity cache invalidation.
    • Improved WSGI support. While there are still some caveats, most of the code which was twisted only is now generic and allows related functionalities to work with a WSGI front-end.
    • Full undo/transaction support: undo of modifications has finally been implemented, and the configuration simplified (basically you activate it or not on an instance basis).
    • Controlling HTTP status code returns is now much easier:
      • WebRequest now has a status_out attribute to control the response status ;
      • most web-side exceptions take an optional status argument.

    API changes

    • The base registry implementation has been moved to a new logilab.common.registry module (see #1916014). This includes code from :

      • cubicweb.vreg (everything that was in there)
      • cw.appobject (base selectors and all).

      In the process, some renaming was done:

      • the top level registry is now RegistryStore (was VRegistry), but that should not impact CubicWeb client code;
      • former selectors functions are now known as "predicate", though you still use predicates to build an object'selector;
      • for consistency, the objectify_selector decorator has hence been renamed to objectify_predicate;
      • on the CubicWeb side, the selectors module has been renamed to predicates.

      Debugging refactoring dropped the need for the lltrace decorator. There should be full backward compat with proper deprecation warnings. Notice the yes predicate and objectify_predicate decorator, as well as the traced_selection function should now be imported from the logilab.common.registry module.

    • All login forms are now submitted to <app_root>/login. Redirection to requested page is now handled by the login controller (it was previously handled by the session manager).

    • Publisher.publish has been renamed to Publisher.handle_request. This method now contains a generic version of the logic previously handled by Twisted. Controller.publish is not affected.

    Unintrusive API changes

    • New 'ldapfeed' source type, designed to replace 'ldapuser' source with data-feed (i.e. copy based) source ideas.
    • New 'zmqrql' source type, similar to 'pyrorql' but using ømq instead of Pyro.
    • A new registry called 'services' has appeared, where you can register server-side cubicweb.server.Service child classes. Their call method can be invoked from a web-side AppObject instance using the new self._cw.call_service method or a server-side one using self.session.call_service. This is a new way to call server-side methods, much cleaner than monkey patching the Repository class, which becomes a deprecated way to perform similar tasks.
    • a new ajaxfunction registry now hosts all remote functions (i.e. functions callable through the asyncRemoteExec JS api). A convenience ajaxfunc decorator will let you expose your python functions easily without all the appobject standard boilerplate. Backwards compatibility is preserved.
    • the 'json' controller is now deprecated in favor of the 'ajax' one.
    • WebRequest.build_url can now take a __secure__ argument. When True, cubicweb tries to generate an https url.

    User interface changes

    A new 'undohistory' view exposes the undoable transactions and gives access to undo some of them.


  • Thoughts on CubicWeb 4.0

    2012/05/14 by Sylvain Thenault

    This is a fairly technical post talking about the structural changes I would like to see in CubicWeb's near future. Let's call that CubicWeb 4.0! It also drafts ideas on how to go from here to there. Draft, really. But that will eventually turn into a nice roadmap hopefully.

    The great simplification

    Some parts of cubicweb are sometimes too hairy for different reasons (some good, most bad). This participates in the difficulty to get started quickly. The goal of CubicWeb 4.0 should be to make things simpler :

    • Fix some bad old design.
    • Stop reinventing the wheel and use widely used libraries in the Python Web World. This extends to benefitting from state of the art libraries to build nice and flexible UI such as Bootstrap, on top of the JQuery foundations (which could become as prominent as the Python standard library in CubicWeb, the development team should get ready for it).
    • If there is a best way to do something, just do it and refrain from providing configurability and options.

    On the road to Bootstrap

    First, a few simple things could be done to simplify the UI code:

    • drop xhtml support: always return text/html content type, stop bothering with this stillborn stuff and use html5
    • move away everything that should not be in the framework: calendar?, embedding, igeocodable, isioc, massmailing, owl?, rdf?, timeline, timetable?, treeview?, vcard, wdoc?, xbel, xmlrss?

    Then we should probably move the default UI into some cubes (i.e. the content of cw.web.views and cw.web.data). Besides making the move to Bootstrap easier, this should also have the benefit of making clearer that this is the default way to build an (automatic) UI in CubicWeb, but one may use other, more usual, strategies (such as using a template language).

    At a first glance, we should start with the following core cubes:

    • corelayout, the default interface layout and generic components. Modules to backport there: application (not an appobject yet), basetemplates, error, boxes, basecomponents, facets, ibreadcrumbs, navigation, undohistory.
    • coreviews, the default generic views and forms. Modules to backport there: actions, ajaxedit, baseviews, autoform, dotgraphview, editcontroller, editforms, editviews, forms, formrenderers, primary, json, pyviews, tableview, reledit, tabs.
    • corebackoffice, the concrete views for the default back-office that let you handle users, sources, debugging, etc. through the web. Modules to backport here: cwuser, debug, bookmark, cwproperties, cwsources, emailaddress, management, schema, startup, workflow.
    • coreservices, the various services, not directly related to display of something. Modules to backport here: ajaxcontroller, apacherewrite, authentication, basecontrollers, csvexport, idownloadable, magicsearch, sessions, sparql, sessions, staticcontrollers, urlpublishing, urlrewrite.

    This is a first draft that will need some adjustements. Some of the listed modules should be split (e.g. actions, boxes,) and their content moved to different core cubes. Also some modules in cubicweb.web packages may be moved to the relevant cube.

    Each cube should provide an interface so that one could replace it with another one. For instance, move from the default coreviews and corelayout cube to bootstrap based ones. This should allow a nice migration path from the current UI to a Bootstrap based UI. Bootstrap should probably be introduced bottom-up: start using it for tables, lists, etc. then go up until the layout defined in the main template. The Orbui experience should greatly help us by pointing at hot spots that will have to be tackled, as well as by providing a nice code base from which we should start.

    Regarding current implementation, we should take care that Contextual components are a powerful way to build "pluggable" UI, but we should probably add an intermediate layer that would make more obvious / explicit:

    • what the available components are
    • what the available slots are
    • which component should go in which slot when possible

    Also at some point, we should take care to separate view's logic from HTML generation: our experience with client works shows that a common need is to use the logic but produce a different HTML. Though we should wait for more use of Bootstrap and related HTML simplification to see if the CSS power doesn't somewhat fulfill that need.

    On the road to proper tasks management

    The current looping task / repo thread mecanism is used for various sort of things and has several problems:

    • tasks don't behave similarly in a multi-instances configuration (some should be executed in a single instance, some in a subset); the tasks system has been originally written in a single instance context; as of today this is (sometimes) handled using configuration options (that will have to be properly set in each instance configuration file);
    • tasks is a repository only api but we also need web-side tasks;
    • there is probably some abuse of the system that may lead to unnecessary resources usage.

    Analyzing a sample http://www.logilab.org/ instance, below are the running looping task by categories. Tasks that have to run on each web instance:

    • clean_sessions, automatically closes unused repository sessions. Notice cw.etwist.server also records a twisted task to clean web sessions. Some changes are imminent on this, they will be addressed in the upcoming refactoring session (that will become more and more necessary to move on several points listed here).
    • regular_preview_dir_cleanup (preview cube), cleanup files in the preview filesystem directory. Could be executed by a (some of the) web instance(s) provided that the preview directory is shared.

    Tasks that should run on a single instance:

    • update_feeds, update copy based sources (e.g. datafeed, ldapfeed). Controlled by 'synchronize' source configuration (persistent source attribute that may be overridden by instance using CWSourceHostConfig entities)
    • expire_dataimports, delete CWDataImport entities older than an amount of time specified in the 'logs-lifetime' configuration option. Not controlled yet.
    • cleanup_auth_cookies (rememberme cube), delete CWAuthCookie entities whose life-time is exhausted. Not controlled yet.
    • cleaning_revocation_key (forgotpwd cube), delete Fpasswd entities with past revocation_date. Not controlled yet.
    • cleanup_plans (narval cube), delete Plan entities instance older than an amount of time specified in the configuration. If 'plan-cleanup-delay' is set to an empty value, the task isn't started.
    • refresh_local_repo_caches (vcsfile cube), pull or clone vcs repositories cache if the Repository entity ask to import_revision_content (hence web instance should have up to date cache to display files content) or if 'repository-import' configuration option is set to 'yes'; import vcs repository content as entities if 'repository-import' configuration option and it is coming from the system source.

    Some deeper thinking is needed here so we can improve things. That includes thinking about:

    • the inter-instances messages bus based on zmq and introduced in 3.15,
    • the Celery project (http://celeryproject.org/), an asynchronous task queue, widely used and written in Python,

    Remember the more cw independent the tasks are, the better it is. Though we still want an 'all-integrated' approach, e.g. not relying on external configuration of Unix specific tools such as CRON. Also we should see if a hard-dependency on Celery or a similar tool could be avoided, and if not if it should be considered as a problem (for devops).

    On the road to an easier configuration

    First, we should drop the different behaviour according to presence of a '.hg' in cubicweb's directory. It currently changes the location where cubicweb external resources (js, css, images, gettext catalogs) are searched for. Speaking of implementation:

    • shared_dir returns the cubicweb.web package path instead of the path to the shared cube,
    • i18n_lib_dir returns the cubicweb/i18n directory path instead of the path to the shared/i18n cube,
    • migration_scripts_dir returns the cubicweb/misc/migration directory path instead of share/cubicweb/migration.

    Moving web related objects as proposed in the Bootstrap section would resolve the problem for the content web/data and most of i18n (though some messages will remain and additional efforts will be needed here). By going further this way, we may also clean up some schema code by moving cubicweb/schemas and cubicweb/misc/migration to a cube (though only a small benefit is to be expected here).

    We should also have fewer environment variables... Let's see what we have today:

    • CW_INSTANCES_DIR, where to look for instances configuration
    • CW_INSTANCES_DATA_DIR, where to look for instances persistent data files
    • CW_RUNTIME_DIR, where to look for instances run-time data files
    • CW_MODE, set to 'system' or 'user' will predefine above environment variables differently
    • CW_CUBES_PATH, additional directories where to look for cubes
    • CW_CUBES_DIR, location of the system 'cubes' directory
    • CW_INSTALL_PREFIX, installation prefix, from which we can compute path to 'etc', 'var', 'share', etc.

    I would propose the following changes:

    • CW_INSTANCES_DIR is turned into CW_INSTANCES_PATH, and defaults to ~/etc/cubicweb.d if it exists and /etc/cubicweb.d (on Unix platforms) otherwise;
    • CW_INSTANCES_DATA_DIR and CW_RUNTIME_DIR are replaced by configuration file options, with smart values generated at instance creation time;
    • the above change should make CW_MODE useless;
    • CW_CUBES_DIR is to be dropped, CW_CUBES_PATH should be enough;
    • regarding CW_INSTALL_PREFIX, I'm lacking experience with non-hg-or-debian installations and don't know if this can be avoided or not.

    Last but not least (for the moment), the 'web' / 'repo' / 'all-in-one' configurations, and the fact that the associated configuration file changes stinks. Ideas to stop doing this:

    • one configuration file per instance, with all options provided by installed parts of the framework used by the application.
    • activate 'services' (or not): web server, repository, zmq server, pyro server. Default services to be started are stored in the configuration file.

    There is probably more that can be done here (less configuration options?), but that would already be a great step forward.

    On the road to...

    The following projects should be investigated to see if we could benefit from them:

    Discussion

    Remember the following goals: migration of legacy code should go smoothly. In a perfect world every application should be able to run with CubicWeb 4.0 until the backwards compatibility code is removed (and CubicWeb 4.0 will probably be released as 4.0 at that time).

    Please provide feedbacks:

    • do you think choices proposed above are good/bad choices? Why?
    • do you know some additional libraries that should be investigated?
    • do you have other changes in mind that could/should be done in cw 4.0?

  • CubicWeb sprint / winter 2014

    2014/02/12 by Nicolas Chauvat

    This sprint took place at Logilab's offices in Paris on Feb 13/14. People from CEA, Unlish, Crealibre and Logilab teamed up to push CubicWeb forward.

    We did not forget the priorities from the roadmap:

    • CubicWeb 3.17.13 and 3.18.3 were released, and CubicWeb 3.19 made progress
    • the branch about ComputedAttributes and ComputedRelations (CWEP-002) is ready to be merged,
    • the branch about the FROM clause (CWEP-003) made progress (the CWEP was reviewed and part of the resulting spec was implemented),
    • in order to reduce work in progress, the number of patches in state reviewed or pending-review was brought down to 243 (from 302, that is 60 or 20%, which is not bad).

  • CubicWeb roadmap meeting on September 4th, 2014

    2014/09/01 by Nicolas Chauvat

    The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. The previous roadmap meeting was in July 2014.

    Here is the report about the September 4th, 2014 meeting. Christophe de Vienne (Unlish) and Dimitri Papadopoulos (CEA) joined us to express their concerns and discuss the future of CubicWeb.

    Versions

    Version 3.17

    This version is stable but old and maintainance will continue only as long as some customers will be willing to pay for it (current is 3.17.16 with 3.17.17 in development).

    Version 3.18

    This version is stable and maintained (current is 3.18.5 with 3.18.6 in development).

    Version 3.19

    This version is stable and maintained (current is 3.19.3 with 3.19.4 in development).

    Version 3.20

    This version is under development. It will try to reduce as much as possible the stock of patches in the state "reviewed", "awaiting review" and "in progress". If you have had something in the works that has not been accepted yet, please ready it for 3.20 and get it merged.

    It should still include the work done for CWEP-002 (computed attributes and relations).

    For details read list of tickets for CubicWeb 3.20.0.

    Version 3.21

    Removal of the dbapi, merging of Connection and ClientConnection, CWEP-003 (adding a FROM clause to RQL).

    Version 4.0

    When the work done for Pyramid will have been tested, it will become the default runner and a lot of things will be dropped: twisted, dead code, ui and core code that would be better cast into cubes, etc.

    This version could happen early in 2015.

    Cubes

    New cubes and libraries

    CWEPs

    Here is the status of open CubicWeb Evolution Proposals:

    CWEP-0002 full-featured implementation, to be merged in 3.20

    CWEP-0003 patches sent to the review. . Champion will be adim.

    Work in progress

    PyConFR

    Christophe will try to present at PyConFR the work he did on getting CubicWeb to work with Pyramid.

    Pip-friendly source layout

    Logilab and Christophe will try to make CubicWeb more pip/virtualenv-friendly. This may involve changing the source layout to include a sub-directory, but the impact on existing devs is expected to be too much and could be delayed to CubicWeb 4.0.

    Pyramid

    Christophe has made good progress on getting CubicWeb to work with Pyramid and he intends to put it into production real soon now. There is a Pyramid extension named pyramid_cubicweb and a CubicWeb cube named cubicweb-pyramid. Both work with CubicWeb 3.19. Christophe demonstrated using the debug toolbar, authenticating users with Authomatic and starting multiple workers with uWSGI.

    Early adopters are now invited to jump in and help harden the code!

    Agenda

    Logilab's next roadmap meeting will be held at the beginning of november 2014 and Christophe and Dimitri were invited.


  • CubicWeb roadmap meeting on July 3rd, 2014

    2014/06/26 by Nicolas Chauvat

    The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. The previous roadmap meeting was in May 2014.

    Here is the report about the July 3rd, 2014 meeting. Christophe de Vienne (Unlish) and Dimitri Papadopoulos (CEA) joined us to express their concerns and discuss the future of CubicWeb.

    Versions

    Version 3.17

    This version is stable but old and maintainance will continue only as long as some customers will be willing to pay for it (current is 3.17.15 with 3.17.16 in development).

    Version 3.18

    This version is stable and maintained (current is 3.18.5 with 3.18.6 in development).

    Version 3.19

    This version was published at the end of April and has now been tested on our internal servers. It includes support for Cross Origin Resource Sharing (CORS) and a heavy refactoring that modifies sessions and sources to lay the path for CubicWeb 4.

    For details read the release notes or the list of tickets for CubicWeb 3.19.0. Current is 3.19.2

    Version 3.20

    This version is under development. It will try to reduce as much as possible the stock of patches in the state "reviewed", "awaiting review" and "in progress". If you have had something in the works that has not been accepted yet, please ready it for 3.20 and get it merged.

    It should still include the work done for CWEP-002 (computed attributes and relations.

    For details read list of tickets for CubicWeb 3.20.0.

    Version 3.21 (or maybe 4.0?)

    Removal of the dbapi, merging of Connection and ClientConnection, CWEP-003 (adding a FROM clause to RQL).

    Cubes

    Cubes published over the past two months

    New cubes

    • cubicweb-frbr: Cube providing a schema based on FRBR entities
    • cubicweb-clinipath
    • cubicweb-fastimport

    CWEPs

    Here is the status of open CubicWeb Evolution Proposals:

    CWEP-0002 only missing a bit of migration support, to be finished soon for inclusion in 3.20.

    CWEP-0003 has been reviewed and is waiting for a bit of reshaping that should occurs soon. It's targeted for 3.21.

    New CWEPs are expected to be written for clarifying the API of the _cw object, supporting persistent sessions and improving the performance of massive imports.

    Work in progress

    Design

    The new logo is now published in the 3.19 line. David showed us his experimentation that modernize a forge's ui with a bit of CSS. There is still a bit of pressure on the bootstrap side though, as it still rely on heavy monkey-patching in the cubicweb-bootstrap cube.

    Data import

    Also, Dimitry expressed is concerns with the lack of proper data import API. We should soon have some feedback from Aurelien's cubicweb-fastimport experimentation, which may be an answer to Dimitry's need. In the end, we somewhat agreed that there were different needs (eg massive-no-consistency import vs not-so-big-but-still-safe), that cubicweb.dataimport was an attempt to answer them all and then cubicweb-dataio and cubicweb-fastimport were more specific responses. In the end we may reasonably hope that an API will emerge.

    Removals

    On his way to persistent sessions, Aurélien made a huge progress toward silence of warnings in the 3.19 tests. dbapi has been removed, ClientConnection / Connection merged. We decided to take some time to think about the recurring task management as it is related to other tricky topics (application / instance configuration) and it's not directly related to persistent session.

    Rebasing on Pyramid

    Last but not least, Christophe demonstrated that CubicWeb could basically live with Pyramid. This experimentation will be pursued as it sounds very promising to get the good parts from the two framework.

    Agenda

    Logilab's next roadmap meeting will be held at the beginning of september 2014 and Christophe and Dimitri were invited.


  • CubicWeb roadmap meeting on January 8th, 2015

    2015/01/05 by Nicolas Chauvat

    The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. The previous roadmap meeting was in November 2014.

    Here is the report about the January 8th, 2015 meeting.

    Christophe de Vienne (Unlish) and Aurélien Campéas (self-employed) joined us to express their concerns and discuss the future of CubicWeb.

    Versions

    Version 3.18

    This version is stable but old and maintained (current is 3.18.7).

    Version 3.19

    This version is stable and maintained (current is 3.19.8).

    Version 3.20

    This version has been released a few days ago. It has not been deployed on production systems yet.

    Its main features are:

    • virtual relations: a new ComputedRelation class can be used in schema.py; its rule attribute is an RQL snippet that defines the new relation.

    • computed attributes: an attribute can now be defined with a formula argument (also an RQL snippet); it will be read-only, and updated automatically.

      Both of these features are described in CWEP-002, and the updated "Data model" chapter of the CubicWeb book.

    • cubicweb-ctl plugins can use the cubicweb.utils.admincnx function to get a Connection object from an instance name.

    • new 'tornado' wsgi backend

    • session cookies have the HttpOnly flag, so they're no longer exposed to javascript

    • rich text fields can be formatted as markdown

    • the edit controller detects concurrent editions, and raises a ValidationError if an entity was modified between form generation and submission

    • cubicweb can use a postgresql "schema" (namespace) for its tables

    • cubicweb-ctl configure can be used to set values of the admin user credentials in the sources configuration file

    For details read list of tickets for CubicWeb 3.20.0.

    We would have loved to integrate the pyramid cube in this release, but the debian packaging effort needed by the pyramid stack is quite big and is acceptable if we target jessie only (at decent price).

    Version 3.21

    For now, the roadmap for 3.21 is still the complete removal of the dbapi, the merging of Connection and ClientConnection.

    Integrate the pyramid cube to provide the pyramid command if the pyramid framework can be imported.

    Integration of CWEP-003 (FROM clause for RQL) and CWEP-004 (cubes as standard python packages) is being discussed.

    Version 4.0

    We expect to accelerate development of CubicWeb 4, which exact roadmap is still to be discussed, but we may already want:

    • be pyramid-based (remove twisted, auth management, etc.),
    • do not have anything left of old dbapi and ClientConnection,
    • integrate squareui as main (and only) web-ui "template" or remove web generation (almost) completely from cubicweb-core and provide it only through the cube system.

    Agenda

    Next roadmap meeting will be held at the beginning of march 2015 at Logilab. Interested parties are invited to get in touch.

    Open Discussions

    Refactoring the documentation

    Christophe de Vienne suggested to completely revamp the documentation and intends to lead this effort.

    Training material

    Aurélien Campéas asks if Logilab would be willing to share its training material under a free license to help interested parties organize and sell trainings.

    Towards making squareui the default rendering engine for cubicweb

    We are expecting to be able to use squareui/bootstrap as "rendering engine" for our forge applications (like http://www.cubicweb.org and http://www.logilab.org) as soon as possible. However to achieve to goal, there are still too many "visual bugs", some of which may require a discussion.

    Among others:

    • put the ctxtoolbar component in the <nav> div
    • each box component should have an icon (what API for this?)
    • we cannot easily make the left column of the main template responsive-aware (requires to change the html flow), so it's probably best to take inspiration from things like http://wrapbootstrap.com/preview/WB0N89JMK
    • facet boxes are a mess, there is no simple solution to have a "smart layout"

    Migration

    • AppObjects should not be loaded by default
    • Have a look at Alembic the migration tool for SQLAlchemy and take inspiration from there.

  • Comparing CubicWeb with Drupal plus CCK extension

    2009/10/29 by Nicolas Chauvat
    http://www.cubicweb.org/file/502151?vid=download

    Drupal is a CMS written in PHP that is getting more and more visibility in the Semantic Web crowd. Several researchers from DERI have been using it as a test bed for their research projects and developed extensions to showcase their ideas. It is for example used to build the Semantic Web Dog Food site that archives the semantic web conferences and publishes them as Linked Open Data. The URL for this year's ISWC is http://data.semanticweb.org/conference/iswc/2009

    This led me to read more about Drupal than I had had the incentive before. I have not had time to give it a try, but I skimmed the documentation and will try to compare it with CubicWeb from a software architecture point of view.

    Drupal defines a Node as an information item. The CCK (aka Content Construction Kit) can be used to define new types of Nodes thru a web interface. Nodes and the bits and pieces used to display them as HTML are not packed together in components. The Features extension is planning on getting this bits packaged.

    If you are a Drupal user/developer and think I am not being fair to Drupal, please comment below.

    On the other hand, CubicWeb has implemented very early the concept of reusable component. What is called a Node in Drupal is an Entity in CubicWeb. By design, CubicWeb does not have a web interface to define entities. The data model is part of the code. To efficiently maintain applications in production, changes to the data model must be tracked with changes to the code. Data model changes imply migration procedures. In CubicWeb, all of this is versionned and made part of the components. Where Drupal needs to grow extensions like CCK and Features, CubicWeb has more advanced possibilities by design, for example the ability to develop featurefull applications by assembling components.

    This was a very short comparison. I'm looking forward to getting a chance of discussing it with knowledgeable Drupal hackers.


  • Monitor all the things! ... and early too!

    2016/09/16 by Arthur Lutz

    Following the "release often, release early" mantra, I thought it might be a good idea to apply it to monitoring on one of our client projects. So right from the demo stage where we deliver a new version every few weeks (and sometimes every few days), we setup some monitoring.

    https://www.cubicweb.org/file/15338085/raw/66511658.jpg

    Monitoring performance

    The project is an application built with the CubicWeb platform, with some ElasticSearch for indexing and searching. As with any complex stack, there are a great number of places where one could monitor performance metrics.

    https://www.cubicweb.org/file/15338628/raw/Screenshot_2016-09-16_12-19-21.png

    Here are a few things we have decided to monitor, and with what tools.

    Monitoring CubicWeb

    To monitor our running Python code, we have decided to use statsd, since it is already built into CubicWeb's core. Out of the box, you can configure a statsd server address in your all-in-one.conf configuration. That will send out some timing statistics about some core functions.

    The statsd server (there a numerous implementations, we use a simple one : python-pystatsd) gets the raw metrics and outputs them to carbon which stores the time series data in whisper files (which can be swapped out for a different technology if need be).

    https://www.cubicweb.org/file/15338392/raw/Screenshot_2016-09-16_11-56-44.png

    If we are curious about a particular function or view that might be taking too long to generate or slow down the user experience, we can just add the @statsd_timeit decorator there. Done. It's monitored.

    statsd monitoring is a fire-and-forget UDP type of monitoring, it should not have any impact on the performance of what you are monitoring.

    Monitoring Apache

    Simply enough we re-use the statsd approach by plugging in an apache module to time the HTTP responses sent back by apache. With nginx and varnish, this is also really easy.

    https://www.cubicweb.org/file/15338407/raw/Screenshot_2016-09-16_11-56-54.png

    One of the nice things about this part is that we can then get graphs of errors since we will differentiate OK 200 type codes from 500 type codes (HTTP codes).

    Monitoring ElasticSearch

    ElasticSearch comes with some metrics in GET /_stats endpoint, the same goes for individual nodes, individual indices and even at cluster level. Some popular tools can be installed through the ElasticSearch plugin system or with Kibana (plugin system there too).

    We decided on a different approach that fitted well with our other tools (and demonstrates their flexibility!) : pull stats out of ElasticSearch with SaltStack, push them to Carbon, pull them out with Graphite and display them in Grafana (next to our other metrics).

    https://www.cubicweb.org/file/15338399/raw/Screenshot_2016-09-16_11-56-34.png

    On the SaltStack side, we wrote a two line execution module (elasticsearch.py)

    import requests
    def stats():
        return request.get('http://localhost:9200/_stats').json()
    

    This gets shipped using the custom execution modules mechanism (_modules and saltutils.sync_modules), and is executed every minute (or less) in the salt scheduler. The resulting dictionary is fed to the carbon returner that is configured to talk to a carbon server somewhere nearby.

    # salt demohost elasticsearch.stats
    [snip]
      { "indextime_inmillis" : 30,
    [snip]
    

    Monitoring web metrics

    To evaluate parts of the performance of a web page we can look at some metrics such as the number of assets the browser will need to download, the size of the assets (js, css, images, etc) and even things such as the number of subdomains used to deliver assets. You can take a look at such metrics in most developer tools available in the browser, but we want to graph this over time. A nice tool for this is sitespeed.io (written in javascript with phantomjs). Out of the box, it has a graphite outputter so we just have to add --graphiteHost FQDN. sitespeed.io even recommends using grafana to visualize the results and publishes some example dashboards that can be adapted to your needs.

    https://www.cubicweb.org/file/15338109/raw/sitespeed-logo-2c.png

    The sitespeed.io command is configured and run by salt using pillars and its scheduler.

    We will have to take a look at using their jenkins plugin with our jenkins continuous integration instance.

    Monitoring crashes / errors / bugs

    Applications will have bugs (in particular when released often to get a client to validate some design choices early). Level 0 is having your client calling you up saying the application has crashed. The next level is watching some log somewhere to see those errors pop up. The next level is centralised logs on which you can monitor the numerous pieces of your application (rsyslog over UDP helps here, graylog might be a good solution for visualisation).

    https://www.cubicweb.org/file/15338139/raw/Screenshot_2016-09-16_11-30-53.png

    When it starts getting useful and usable is when your bugs get reported with some rich context. That's when using sentry gets in. It's free software developed on github (although the website does not really show that) and it is written in python, so it was a good match for our culture. And it is pretty awesome too.

    We plug sentry into our WSGI pipeline (thanks to cubicweb-pyramid) by installing and configuring the sentry cube : cubicweb-sentry. This will catch rich context bugs and provide us with vital information about what the user was doing when the crash occured.

    This also helps sharing bug information within a team.

    The sentry cube reports on errors being raised when using the web application, but can also catch some errors when running some maintenance or import commands (ccplugins in CubicWeb). In this particular case, a lot of importing is being done and Sentry can detect and help us triage the import errors with context on which files are failing.

    Monitoring usage / client side

    This part is a bit neglected for the moment. Client side we can use Javascript to monitor usage. Some basic metrics can come from piwik which is usually used for audience statistics. To get more precise statistics we've been told Boomerang has an interesting approach, enabling a closer look at how fast a page was displayed client side, how much time was spend on DNS, etc.

    On the client side, we're also looking at two features of the Sentry project : the raven-js client which reports Javascript errors directly from the browser to the Sentry server, and the user feedback form which captures some context when something goes wrong or a user/client wants to report that something should be changed on a given page.

    Load testing - coverage

    To wrap up, we also often generate traffic to catch some bugs and performance metrics automatically :

    • wget --mirror $URL
    • linkchecker $URL
    • for $search_term in cat corpus; do wget URL/$search_term ; done
    • wapiti $URL --scope page
    • nikto $URL

    Then watch the graphs and the errors in Sentry... Fix them. Restart.

    Graphing it in Grafana

    We've spend little time on the dashboard yet since we're concentrating on collecting the metrics for now. But here is a glimpse of the "work in progress" dashboard which combines various data sources and various metrics on the same screen and the same time scale.

    https://www.cubicweb.org/file/15338648/raw/Screenshot_2016-09-13_09-41-45.png

    Further plans

    • internal health checks, we're taking a look at python-hospital and healthz: Stop reverse engineering applications and start monitoring from the inside (Monitorama) (the idea is to distinguish between the app is running and the app is serving it's purpose), and pyramid_health
    • graph the number of Sentry errors and the number of types of errors: the sentry API should be able to give us this information. Feed it to Salt and Carbon.
    • setup some alerting : next versions of Grafana will be doing that, or with elastalert
    • setup "release version X" events in Graphite that are displayed in Grafana, maybe with some manual command or a postcreate command when using docker-compose up ?
    • make it easier for devs to have this kind of setup. Using this suite of tools in developement might sometimes be overkill, but can be useful.