Blog entries

Logilab's roadmap for CubicWeb on March 7th, 2014

2014/03/10 by Nicolas Chauvat

The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. Here is the report about the Mar 7th, 2014 meeting. The previous report posted to the blog was the january 2014 roadmap.

Version 3.17

This version is stable but old and maintainance will stop in a few weeks (current is 3.17.13 and 3.17.14 is upcoming).

Version 3.18

This version is stable and maintained (current is 3.18.3 and 3.18.4 is upcoming).

Version 3.19

This version is about to be published. It includes a heavy refactoring that modifies sessions and sources to lay the path for CubicWeb 4.

For details read list of tickets for CubicWeb 3.19.0.

Version 3.20

This version will try to reduce as much as possible the stock of patches in the state "reviewed", "awaiting review" and "in progress". If you have had something in the works that has not been accepted yet, please ready it for 3.20 and get it merged.

It should also include the work done for CWEP-002 (computed attributes and relations) and CWEP-003 (adding a FROM clause to RQL).

For details read list of tickets for CubicWeb 3.20.0.

Cubes

Here is a list of cubes that had versions published over the past two months: addressbook, awstats, blog, bootstrap, brainomics, comment, container, dataio, genomics, invoice, mediaplayer, medicalexp, neuroimaginge, person, preview, questionnaire, securityprofile, simplefacet, squareui, tag, tracker, varnish, vcwiki, vtimeline.

Here are a the new cubes we are pleased to announce:

collaboration is a building block that reuses container and helps to define collaborative workflows where entities are cloned, modified and shared.

Our priorities for the next two months are collaboration and container, then narval/apycot, then mercurial-server, then rqlcontroller and signedrequest, then imagesearch.

Mid-term goals

The work done for CWEP-0002 (computed attributes and relations) is expected to land in CubicWeb 3.20.

The work done for CWEP-0003 (explicit data source federation using FROM in RQL) is expected to land in CubicWeb 3.20.

Tools to diagnose performance issues would be very useful. Maybe in 3.21 ?

Caching session data would help and some work was done on this topic during the sprint in february. Maybe in 3.22 ?

WSGI has made progress lately, but still needs work. Maybe in 3.23 ?

RESTfulness is a goal. Maybe in 3.24 ?

Maybe 3.25 will be in fact 4.0 ?

Events

A spring sprint will take place in Logilab's offices in Paris from April 28th to 30th. We invite all the interested parties to join us there!

Last but not least

As already said on the mailing list, other developers and contributors are more than welcome to share their own goals in order to define a roadmap that best fits everyone's needs.

Logilab's next roadmap meeting will be held at the beginning of may 2014.


CubicWeb sprint / winter 2014

2014/02/12 by Nicolas Chauvat

This sprint took place at Logilab's offices in Paris on Feb 13/14. People from CEA, Unlish, Crealibre and Logilab teamed up to push CubicWeb forward.

We did not forget the priorities from the roadmap:

  • CubicWeb 3.17.13 and 3.18.3 were released, and CubicWeb 3.19 made progress
  • the branch about ComputedAttributes and ComputedRelations (CWEP-002) is ready to be merged,
  • the branch about the FROM clause (CWEP-003) made progress (the CWEP was reviewed and part of the resulting spec was implemented),
  • in order to reduce work in progress, the number of patches in state reviewed or pending-review was brought down to 243 (from 302, that is 60 or 20%, which is not bad).

CubicWeb using Postgresql at its best

2014/02/08 by Nicolas Chauvat

We had a chat today with a core contributor to Postgresql from whom we may buy consulting services in the future. We discussed how CubicWeb could get the best out of Postgresql:

  • making use of the LISTEN/NOTIFY mechanism built into PG could be useful (to warn the cache about modified items for example) and PgQ is its good friend;
  • views (materialized or not) are another way to implement computed attributes and relations (see CWEP number 002) and it could be that the Entities table is in fact a view of other tables;
  • implementing RQL as an in-database language could open the door to new things (there is PL/pgSQL, PL/Python, what if we had PL/RQL?);
  • Foreign Data Wrappers written with Multicorn would be another way to write data feeds (see LDAP integration for an example);
  • managing dates can be tricky when users reside in different timezones and UTC is important to keep in mind (unicode/str is a good analogy);
  • for transitive closures that are often needed when implementing access control policies with __permissions, Postgresql can go a long way with queries like "WITH ... (SELECT UNION ALL SELECT RETURNING *) UPDATE USING ...";
  • the fastest way to load tabular data that does not need too much pre-processing is to create a temporary table in memory, then COPY-FROM the data into that table, then index it, then write the transform and load step in SQL (maybe with PL/Python);
  • when executing more than 10 updates in a row, it is better to write into a temporary table in memory, then update the actual tables with UPDATE USING (let's check if the psycopg driver does that when executemany is called);
  • reaching 10e8 rows in a table is at the time of this writing the stage when you should start monitoring your db seriously and start considering replication, partition and sharding.
  • full-text search is much better in Postgresql than the general public thinks it is and recent developments made it orders of magnitude faster than tools like Lucene or Solr and ElasticSearch;
  • when dealing with complex queries (searching graphs maybe), an option to consider is to implement a specific data type, use it into a materialized view and use GIN or GIST indexes over it;
  • for large scientific data sets, it could be interesting to link the numpy library into Postgresql and turn numpy arrays into a new data type;
  • Oh, and one last thing: the object-oriented tables of Postgresql are not such a great idea, unless you have a use case that fits them perfectly and does not hit their limitations (CubicWeb's is_instance_of does not seem to be one of these).

Hopin' I got you thinkin' :)

http://developer.postgresql.org/~josh/graphics/logos/elephant.png

Cubicweb sprints winter/spring 2014

2014/01/24 by David Douard

The Logilab team is pleased to announce two Cubicweb sprints to be held in its Paris offices in the upcoming months:

February 13/14th at Logilab in Paris

The agenda would be the FROM clause for which a CWEP is still awaited, and the RQL rewriter according to the CWEP02.

April 28/30th at Logilab in Paris

Agenda to be defined.

Join the party

All users and contributors of CubicWeb are invited to join the party. Just send an email to contact at Logilab.fr if you plan to come.

http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

Logilab's roadmap for CubicWeb on January 9th, 2014

2014/01/14 by Nicolas Chauvat

The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. Here is the report about the Jan 9th, 2014 meeting. The previous report posted to the blog was the november 2013 roadmap.

Version 3.17

This version is stable and maintained (current is 3.17.11 and 3.17.12 is upcoming).

Version 3.18

This version was released on Jan 10th. Read the release notes or the details of CubicWeb 3.18.0.

Version 3.19

This version includes a heavy refactoring that modifies sessions and sources to lay the path for CubicWeb 4. It is currently the default development head in the repository and is expected to be released before the end of january.

For details read list of tickets for CubicWeb 3.19.0.

Version 3.20

This version will try to reduce as much as possible the stock of patches in the state "reviewed", "awaiting review" and "in progress". If you have had something in the works that has not been accepted yet, please ready it for 3.20 and get it merged.

For details read list of tickets for CubicWeb 3.20.0.

Cubes

The current trend is to develop more and more new features in dedicated cubes than to add more code to the core of CubicWeb. If you thought CubicWeb development was slowing down, you made a mistake, because cubes are ramping up.

Here is a list of versions that were published in the past two months: timesheet, postgis, leaflet, bootstrap, worker, container, embed, geocoding, vcreview, trackervcs, vcsfile, zone, dataio, mercurial-server, queueing, questionnaire, genomics, medicalexp, neuroimaging, brainomics, elections.

Here are a the new cubes we are pleased to announce:

Bootstrap works and we do not create a new application without it.

relationwidget provides a modal window to edit relations in forms (use uicfg to activate it).

resourcepicker provides a modal window to insert links to images and files into structured text.

rqlcontroller allows to use the INSERT, DELETE and SET keywords when sending RQL queries over HTTP. It returns JSON. Get used to it and you may forget about asking for specific web services in your apps, for it is a generic web service.

imagesearch is an image gallery with facets. You may use it as a demo of a visual search tool.

Mid-term goals

A new repository was created to have all the CubicWeb Evolution Proposals in one place.

CWEP-0002 is a work in progress about computed relations and computed attributes, or maybe more. It will be a focus of the next sprint and is targeted at CubicWeb 3.20.

A new CWEP is expected about the adding FROM keyword to RQL to implement explicit data source federation. It will be a focus of the next sprint and is targeted at CubicWeb 3.21.

Tools to diagnose performance issues would be very useful. Maybe in 3.22 ?

Caching session data would help. Maybe in 3.23 ?

WSGI has made progress lately, but still needs work. Maybe in 3.24 ?

RESTfulness is a goal. Maybe in 3.25 ?

Maybe 3.26 will be in fact 4.0 ?

Events

A sprint will take place in Logilab's offices in Paris around mid-february or at the end of april. We invite all the interested parties to join us there!

Last but not least

As already said on the mailing list, other developers and contributors are more than welcome to share their own goals in order to define a roadmap that best fits everyone's needs.

Logilab's next roadmap meeting will be held at the beginning of march 2014.


What's new in CubicWeb 3.18

2014/01/10 by Aurelien Campeas

The migration script does not handle sqlite nor mysql instances.

New functionalities

  • add a security debugging tool (see #2920304)
  • introduce an add permission on attributes, to be interpreted at entity creation time only and allow the implementation of complex update rules that don't block entity creation (before that the update attribute permission was interpreted at entity creation and update time) (see #2965518)
  • the primary view display controller (uicfg) now has a set_fields_order method similar to the one available for forms
  • new method ResultSet.one(col=0) to retrieve a single entity and enforce the result has only one row (see #3352314)
  • new method RequestSessionBase.find to look for entities (see #3361290)
  • the embedded jQuery copy has been updated to version 1.10.2, and jQuery UI to version 1.10.3.
  • initial support for wsgi for the debug mode, available through the new wsgi cubicweb-ctl command, which can use either python's builtin wsgi server or the werkzeug module if present.
  • a rql-table directive is now available in ReST fields
  • cubicweb-ctl upgrade can now generate the static data resource directory directly, without a manual call to gen-static-datadir.

API changes

  • not really an API change, but the entity write permission checks are now systematically deferred to an operation, instead of a) trying in a hook and b) if it failed, retrying later in an operation
  • The default value storage for attributes is no longer String, but Bytes. This opens the road to storing arbitrary python objects, e.g. numpy arrays, and fixes a bug where default values whose truth value was False were not properly migrated.
  • symmetric relations are no more handled by an rql rewrite but are now handled with hooks (from the activeintegrity category); this may have some consequences for applications that do low-level database manipulations or at times disable (some) hooks.
  • unique together constraints (multi-columns unicity constraints) get a name attribute that maps the CubicWeb contraint entities to the corresponding backend index.
  • BreadCrumbEntityVComponent's open_breadcrumbs method now includes the first breadcrumbs separator
  • entities can be compared for equality and hashed
  • the on_fire_transition predicate accepts a sequence of possible transition names
  • the GROUP_CONCAT rql aggregate function no longer repeats duplicate values, on the sqlite and postgresql backends

Deprecation

  • pyrorql sources have been deprecated. Multisource will be fully dropped in the next version. If you are still using pyrorql, switch to datafeed NOW!
  • the old multi-source system
  • find_one_entity and find_entities in favor of find (see #3361290)
  • the TmpFileViewMixin and TmpPngView classes (see #3400448)

Deprecated Code Drops

  • ldapuser have been dropped; use ldapfeed now (see #2936496)
  • action GotRhythm was removed, make sure you do not import it in your cubes (even to unregister it) (see #3093362)
  • all 3.8 backward compat is gone
  • all 3.9 backward compat (including the javascript side) is gone
  • the twisted (web-only) instance type has been removed

For a complete list of tickets, read CubicWeb 3.18.0.


Logilab's roadmap for CubicWeb on November 8th, 2013

2013/11/11 by Nicolas Chauvat

The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. Here is the report about the Nov 8th, 2013 meeting. The previous report posted to the blog was the september 2013 roadmap.

Version 3.17

This version is stable and maintained (cubicweb 3.17.11 is upcoming).

Version 3.18

This version was supposed to be released in september or october, but is stalled at the integration stage. All open tickets were moved to 3.19 and existing patches that are not ready to be merged will be more aggressively delayed to 3.19. The goal is to release 3.18 as soon as possible.

For details read list of tickets for CubicWeb 3.18.0.

Version 3.19

This version will probably be published early next year (read january or february 2014). it is planned to include a heavy refactoring that modifies sessions and sources to lay the path for CubicWeb 4.

For details read list of tickets for CubicWeb 3.19.0.

Squareui

Logilab is now developping all its new projects based on Squareui (and Bootstrap 3.0). Squareui can be considered as a usable beta, but not as feature-complete.

Logilab is looking for a UX designer to work on the general ergonomy of CubicWeb. Read the job offer.

Mid-term goals

The mid-term goals include better REST support (Representational State Transfer), complete WSGI (Python's Web Server Gateway Interface) and the FROM clause for RQL queries (to reinvent db federation outside of the core).

On the front-end side, it would be nice to be able to improve forms, maybe with client-side javascript and better support for a "json on server, js in browser" separation of concerns.

Cubes

A cube oauth was contributed in large part by Unlish, a startup that is using CubicWeb to implement its service.

A cube vcwiki is being developed by Logilab, to manage the content of a wiki with a version control system (built with the cube vcsfile).

Last but not least

As already said on the mailing list, other developers and contributors are more than welcome to share their own goals in order to define a roadmap that best fits everyone's needs.

Logilab's next roadmap meeting will be held at the beginning of january 2014.


Apache authentication

2013/10/10 by Dimitri Papadopoulos

An Apache front end might be useful, as Apache provides standard log files, monitoring or authentication. In our case, we have Apache authenticate users before they are cleared to access our CubicWeb application. Still, we would like user accounts to be managed within a CubicWeb instance, avoiding separate sets of identifiers, one for Apache and the other for CubicWeb.

We have to address two issues:

  • have Apache authenticate users against accounts in the CubicWeb database,
  • have CubicWeb trust Apache authentication.

Apache authentication against CubicWeb accounts

A possible solution would be to access the identifiers associated to a CubicWeb account at the SQL level, directly from the SQL database underneath a CubicWeb instance. The login password can be found in the cw_login and cw_upassword columns of the cw_cwuser table. The benefit is that we can use existing Apache modules for authentication against SQL databases, typically mod_authn_dbd. On the other hand this is highly dependant on the underlying SQL database.

Instead we have chosen an alternate solution, directly accessing the CubicWeb repository. Since we need Python to access the repository, our sysasdmins have deployed mod_python on our Apache server.

We wrote a Python authentication module that accesses the repository using ZMQ. Thus ZMQ needs be enabled. To enable ZMQ uncomment and complete the following line in all-in-one.conf:

zmq-repository-address=zmqpickle-tcp://localhost:8181

The Python authentication module looks like:

from mod_python import apache
from cubicweb import dbapi
from cubicweb import AuthenticationError

def authenhandler(req):
    pw = req.get_basic_auth_pw()
    user = req.user

    database = 'zmqpickle-tcp://localhost:8181'
    try:
        cnx = dbapi.connect(database, login=user, password=pw)
    except AuthenticationError:
        return apache.HTTP_UNAUTHORIZED
    else:
        cnx.close()
        return apache.OK

CubicWeb trusts Apache

Our sysadmins set up Apache to add x-remote-user to the HTTP headers forwarded to CubicWeb - more on the relevant Apache configuration in the next paragraph.

We then add the cubicweb-trustedauth cube to the dependencies of our CubicWeb application. We simply had to add to the __pkginfo__.py file of our CubicWeb application:

__depends__ =  {
    'cubicweb': '>= 3.16.1',
    'cubicweb-trustedauth': None,
}

This cube gets CubicWeb to trust the x-remote-user header sent by the Apache front end. CubicWeb bypasses its own authentication mechanism. Users are directly logged into CubicWeb as the user with a login identical to the Apache login.

Apache configuration and deployment

Our Apache configuration looks like:

<Location /apppath >
  AuthType Basic
  AuthName "Restricted Area"
  AuthBasicAuthoritative Off
  AuthUserFile /dev/null
  require valid-user

  PythonAuthenHandler cubicwebhandler

  RewriteEngine On
  RewriteCond %{REMOTE_USER} (.*)
  RewriteRule . - [E=RU:%1]
<Location /apppath >

RequestHeader set X-REMOTE-USER %{RU}e

ProxyPass          /apppath  http://127.0.0.1:8080
ProxyPassReverse   /apppath  http://127.0.0.1:8080

The CubicWeb application is accessed as http://ourserver/apppath/.

The Python authentication module is deployed as /usr/lib/python2.7/dist-packages/cubicwebhandler/handler.py where cubicwebhandler is the attribute associated to PythonAuthenHandler in the Apache configuration.


Brainomics / CrEDIBLE conference report

2013/10/09 by Vincent Michel

Cubicweb and the Brainomics project were presented last week at the CrEDIBLE workshop (October 2-4, 2013, Sophia-Antipolis) on "Federating distributed and heterogeneous biomedical data and knowledge". We would like to thank the organizers for this nice opportunity to show the features of CubicWeb and Brainomics in the context of biomedical data.

http://credible.i3s.unice.fr/lib/tpl/credible/images/credible.png

Workshop highlights

  • A short presentation of SHI3LD that defines data access based on conditions that are based on ASK request. The other part was a state of the art of Open data license, and the (poor) existence of licenses expressed in RDF. Future work seems to be an interesting combination of both SHI3LD and RDF-based licenses for data access.
  • MIDAS, an open-source software for sharing medical data. This project could be an interesting source of inspiration for the file sharing part of CubicWeb, even if the (really complicated in my opinion) case of large files downloads is not addressed for now.
  • Federated queries based on FedX - the optimization techniques based on source selection & exclusive groups seems a good approach for avoiding large data transfers and finding some (sub-)optimal ways to join the different data sources. This should be taken into account in the future work on the "FROM" clause in CubicWeb.
  • WebPIE/QueryPIE: a map-reduce-based approach for large-scale reasoning.

CubicWeb and Brainomics

The slides of the presentation can be download as a PDF or viewed on slideshare.

Some people seem confused on the RQL to SQL translation. This relies on a simple translation logic that is implemented in the rql2sql file. This is only an implementation trick, not so different from the one used in RDBMS-based triplestores that have to convert SPARQL into SQL.

RQL inference : there is no magic behind the RQL inference process. As opposed to triplestores that store RDF triples that contain their own schema, and thus cannot easily know the full data model in these triples without looking at all the triples, RQL relies on a relational database with an fixed (at a given moment) data model, thus allowing inference and simple checks. In particular, in this example, we want All the Cities of `Île de France` with more than 100 000 inhabitants ?, which is expressed in RQL:

Any X WHERE X region Y, X population > 100000,
            Y uri "http://fr.dbpedia.org/resource/Île-de-France"

and SPARQL:

select ?ville where {
?ville db-owl:region <http://fr.dbpedia.org/resource/Île-de-France> .
?ville db-owl:populationTotal ?population .
FILTER (?population > 100000)
}

Beside the fact that RQL is less verbose that SPARQL (syntax matters), the simplicity of RQL relies on the fact that it can automatically infer (similarly to SPARQL) that if X is related to Y by the region relation and has a population attribute, it should be a city. If city and district both have the region relation and a population attribute, the RQL inference allows to fetch them both transparently, otherwise one can be specific by using the is relation:

Any X WHERE X is City, X region Y, X population > 100000,
            Y uri "http://fr.dbpedia.org/resource/Île-de-France"

RQL also allows subqueries, union, full-text search, stored procedures, ... (see the doc).

These really interesting discussions convinced us that we should write a journal paper for detailing the theoretical and technical concepts behind RQL and the YAMS schema.


Logilab will be in Toulouse métropole Open Data Barcamp tomorrow

2013/10/08 by Sylvain Thenault

Meet us tomorrow at the Toulouse's Cantine where several people from Logilab will be there for the open data barcamp organized by Toulouse Metropole.

More infos on barcamp.org. We'll probably talk abouthow CubicWeb manages to import large amounts of open-data to reuse.


Logilab's roadmap for CubicWeb on September 6th, 2013

2013/09/17 by Nicolas Chauvat

The Logilab team holds a roadmap meeting every two months to plan its CubicWeb development effort. Here is the report about the Sept 6th, 2013 meeting. The previous report posted to the blog was the february 2013 roadmap.

Version 3.17

This version is now stable and maintained (release 3.17.7 is upcoming). It added a couple features and focused on putting CW to the diet by extracting some functionnalities provided by the core into external cubes: sioc, embed, massmailing, geocoding, etc.

For details read what's new in CubicWeb 3.17.

Version 3.18

This version is now freezed and will be published as soon as all the patches are tested and merged. Since we have a lot of work for clients until the end of the year at Logilab, the community should feel free to help (as usual) if it wants this version to be released rather sooner than later.

This version will remove the ldapuser source that is replaced by ldapfeed, implement Cross Origin Resource Sharing, drop some very old compatibility code, deprecate the old version of the multi-source system and provide various other features and bugfixes.

For details read list of tickets for CubicWeb 3.18.0.

Version 3.19

This version will probably be publish early next year (read january or february 2014) unless someone who is not working at Logilab takes responsibility for its release.

It should include the heavy refactoring work done by Pierre-Yves and Sylvain over the past year, that modifies sessions and sources to lay the path for CubicWeb 4.

For details read list of tickets for CubicWeb 3.19.0 or take a look at this head.

Squareui

Since Orbui changes the organization of the default user interface on screen, it was decided to share the low-level bootstrap related views that could be shared and build a SquareUI cube that would conform design choices of the default UI.

Logilab is now developping all its new projects based on Squareui 0.2. Read about it on the mailing list archives.

Mid-term goals

The mid-term goals include better REST support (Representational State Transfer), complete WSGI (Python's Web Server Gateway Interface) and the FROM clause for RQL queries (to reinvent db federation outside of the core).

Cubes

Our current plan is to extract as much as possible to cubes. We started CubicWeb many years ago with the Python motto "batteries included", but have since realized that having too much in the core contributes to making CubicWeb difficult to learn.

Since we would very much like the community to grow, we are now aiming for something more balanced, like Mercurial does. The core is designed such that most features can be developed as an extension. Once they are stable, popular extensions can be moved to the main library that is distributed with the core, and be activated with a switch in the configuration file.

Several cubes are under active development: oauth, signedrequest, dataio, etc.

Last but not least

As already said on the mailing list, other developers and contributors are more than welcome to share their own goals in order to define a roadmap that best fits everyone's needs.

Logilab's next roadmap meeting will be held at the beginning of November 2013.


Brainomics - A management system for exploring and merging heterogeneous brain mapping data

2013/09/12 by Arthur Lutz

At OBHM 2013, the 19th Annual Meeting of the Organization for Human Brain Mapping, Logilab presented a poster which explains the work done using CubicWeb on brain imaging and genetics data in collaboration with INRIA, INSERM and the CEA during the Brainomics project co-financed by Agence nationale de la Rercherche.

http://www.cubicweb.org/file/3123353/raw/Screenshot%20from%202013-09-12%2010%3A27%3A27.png

You can download this poster and try the demo online.


What's new in CubicWeb 3.17

2013/06/21 by Aurelien Campeas

What's new in CubicWeb 3.17?

New functionalities

  • add a command to compare db schema and file system schema (see #464991)
  • Add CubicWebRequestBase.content with the content of the HTTP request (see #2742453)
  • Add directive bookmark to ReST rendering (see #2545595)
  • Allow user defined final type (see #124342)

API changes

  • drop typed_eid() in favour of int() (see #2742462)
  • The SIOC views and adapters have been removed from CubicWeb and moved to the sioc cube.
  • The web page embedding views and adapters have been removed from CubicWeb and moved to the embed cube.
  • The email sending views and controllers have been removed from CubicWeb and moved to the massmailing cube.
  • RenderAndSendNotificationView is deprecated in favor of ActualNotificationOp the new operation uses the more efficient data idiom.
  • Looping task can now have an interval <= 0. Negative interval disable the looping task entirely.
  • We now serve html instead of xhtml. (see #2065651)

Deprecation

  • ldapuser has been deprecated. It will be removed in a future version. If you are still using ldapuser switch to ldapfeed NOW!
  • hijack_user has been deprecated. It will be dropped soon.

Deprecated Code Drops

  • The progress views and adapters have been removed from CubicWeb. These classes were deprecated since 3.14.0. They are still available in the iprogress cube.
  • The part of the API deprecated since 3.7 was dropped.

We're going to PGDay France, the Postgresql Community conference

2013/06/11 by Arthur Lutz

A few people of the CubicWeb team are going to attend the French PostgreSQL community conference in Nantes (France) on the 13th of june.

http://www.cubicweb.org/file/2932005/raw/hdr_left.png

We're excited to learn more about the following topics that are relevant to CubicWeb's development and features :

https://www.pgday.fr/_media/pgfr2.png

Obviously we'll pay attention to all the talks during the day. If you're attending, we hope to see you there.


OpenData meets the Semantic Web at WOD2013

2013/06/10 by Arthur Lutz

With a few people from Logilab we went to the 2nd International Workshop on Open Data (WOD), on the 3rd of june.

Although the main focus was an academic take on OpenData, a lot of talks were related to the Semantic Web technologies and especially LinkedData.

http://www.logilab.org/file/144837/raw/banniere-wod2013.png

The full program (and papers) is on the following website. Here is a quick review of the things we though worth sharing.

  • privacy oriented ontologies : http://l2tap.org/
  • interesting automations done to suggest alignments when initial data is uploaded to an opendata website
  • some opendata platforms have built-in APIs to get files, one example is Socrata : http://dev.socrata.com/
  • some work is being done to scale processing of linked data in the cloud (did you know you could access ready available datasets in the Amazon cloud ? DBPedia for example )
  • the data stored in wikipedia can be a good source of vocabulary on certain machine learning tasks (and in the future, wikidata project)
  • there is an RDF extension to Google Refine (or OpenRefine), but we haven't managed to get it working out of the box,
  • WebSmatch uses morphological operators (erosion / dilation) to identify grids and zones in Excel Spreadsheets and then aligns column data on known reference values (e.g. country lists).

We naturally enjoyed the presentation made by Romain Wenz about http://data.bnf.fr with the unavoidable mention of Victor Hugo (and CubicWeb).

Thanks to the organizers of the conference and to the National French Library for hosting the event.


data.bnf.fr gets the Stanford Prize for Innovation in Research Libraries

2013/03/01 by Nicolas Chauvat

data.bnf.fr and Gallica just got awarded the Stanford Prize for Innovation in Research Libraries 2013. The CubicWeb community is very pleased to see that data.bnf.fr, which is built with CubicWeb, is being recognized at the top international level as leading innovation its domain! Read the comments of the judges for more details.


CubicWeb at Data Tuesday on Feb 26th 2013

2013/02/15 by Nicolas Chauvat

CubicWeb was showcased at Data Tuesday on Feb 26th 2013. The other presentations were interesting, especially shacache.org, the soon-to-be-launched OpenMeteoData and the very useful scikit.learn.


CubicWeb rewarded at Dataconnexion 2013

2013/02/06 by Nicolas Chauvat

CubicWeb got rewarded yesterday at the award ceremony of the Dataconnexions 2013 contest.

http://www.cubicweb.org/2710848?vid=download

Dataconnexions is a contest organized by Etalab, the organization part of the French State that is in charge of data.gouv.fr, that catalogs the open data published by the french administration.

Congratulations to all the developers and users of CubicWeb and welcome to the people who will join the CW community thanks to the media coverage we are now experiencing.

Read the announce to the press and the slides.


Logilab's roadmap for CubicWeb as of February 2013

2013/02/04 by Nicolas Chauvat

The Logilab team now holds a roadmap meeting every two months to plan its CubicWeb development effort. Here are the decisions that were taken on Feb 1st, 2013.

Version 3.17

This version should be published before the end of March and will finish all the things that are work in progress. It will include:

  • the refactoring necessary to introduce persistant sessions,
  • the shrinking of web/views: everything that does not deserve its own cube (like sioc, embed, geocoding, etc) will go into a cube named legacyui (this will open the door to squareui),
  • stop serving pages with "content-type: application/xhtml",
  • handling postgresql schemas (will require a new version of logilab.database),
  • a new logo.

Squareui

Once the cube legacyui extracted (in version 3.17), it will be possible to move forward swiftly with squareui. Due to its other duties, one can not expect the core CW team to develop squareui. People interested will be in charge and ideally the squareui cube could be released when cubicweb 3.17 will be published.

Cleaning up the backlog

The lead CW developers will spend about 20% of their time cleaning up the ticket backlog at the forge (900 open tickets and 50 in progress !)

The first step will be to reduce the number of tickets "in progress", then to organize the open tickets and merge the duplicates.

Version 3.18

This version is due at the end of may 2013. It will include:

  • persisting sessions,
  • WSGI,
  • RESTfulness: support for HTTP verbs PUT / DELETE, enforcement of the semantics of GET / POST (may be difficult to maintain backward-compatibility)

Mid-term goals

The mid-term goals are:

  • possibility to add new base types (Array, HStore, Geometry, TSVector, etc.) that would use extensions from the SQL backend

  • FROM clause in rql queries

  • websockets

  • defining attribute on relations and defining "virtual" relations or rules:

    class Contribution(EntityType):
        author = SubjectRelation('Person', cardinality='1*', inlined=True)
        book = SubjectRelation('Book', cardinality='1*', inlined=True)
        role = SubjectRelation('Role', cardinality='1*', inlined=True)
    
    preface_writer = VirtualRelation('C is Contribution, C author S, C book O, '
                                     'C role R, R name "preface writer"')
    

    And:

    Any P WHERE B is Book, P preface_writer B
    

    Will we need a materialized view in the database, a standard relation maintained by hooks, rewrite the RQL on-the-fly ? Time will tell.

  • cards with logic (mustache js templates for example)

  • coffeescript ? brython ? javascript ? prototype something with CubicDB + WebService that outputs json + user interface in full javascript

  • package separately Cubic(Web)DB et CubicWeb ?

  • think about the overall architecture (using WSGI, persistent sessions, etc.), and find solutions that fit a distributed architecture (look at paste.deploy, circus, etc.)

  • clean up the javascript en web/data/*.js

  • configurable metadata, managing the size of the entities table

  • more SPARQL

  • namespaces for the data models of the cubes

As already said on the mailing list, other developers and contributors are more than welcome to share their own goals in order to define a roadmap that best fits everyone's needs.

Logilab's next roadmap meeting will be held at the beginning of April 2013.


What's new in CubicWeb 3.16

2013/01/23 by Aurelien Campeas

What's new in CubicWeb 3.16?

New functionalities

  • Add a new dataimport store (SQLGenObjectStore). This store enables a fast import of data (entity creation, link creation) in CubicWeb, by directly flushing information in SQL. This may only be used with PostgreSQL, as it requires the 'COPY FROM' command.

API changes

  • Orm: set_attributes and set_relations are unified (and deprecated) in favor of cw_set that works in all cases.

  • db-api/configuration: all the external repository connection information is now in an URL (see #2521848), allowing to drop specific options of pyro nameserver host, group, etc and fix broken ZMQ source. Configuration related changes:

    • Dropped 'pyro-ns-host', 'pyro-instance-id', 'pyro-ns-group' from the client side configuration, in favor of 'repository-uri'. NO MIGRATION IS DONE, supposing there is no web-only configuration in the wild.
    • Stop discovering the connection method through repo_method class attribute of the configuration, varying according to the configuration class. This is a first step on the way to a simpler configuration handling.

    DB-API related changes:

    • Stop indicating the connection method using ConnectionProperties.
    • Drop _cnxtype attribute from Connection and cnxtype from Session. The former is replaced by a is_repo_in_memory property and the later is totaly useless.
    • Turn repo_connect into _repo_connect to mark it as a private function.
    • Deprecate in_memory_cnx which becomes useless, use _repo_connect instead if necessary.
  • the "tcp://" uri scheme used for ZMQ communications (in a way reminiscent of Pyro) is now named "zmqpickle-tcp://", so as to make room for future zmq-based lightweight communications (without python objects pickling).

  • Request.base_url gets a secure=True optional parameter that yields an https url if possible, allowing hook-generated content to send secure urls (e.g. when sending mail notifications)

  • Dataimport ucsvreader gets a new boolean ignore_errors parameter.

Unintrusive API changes

  • Drop of cubicweb.web.uicfg.AutoformSectionRelationTags.bw_tag_map, deprecated since 3.6.

User interface changes

  • The RQL search bar has now some auto-completion support. It means relation types or entity types can be suggested while typing. It is an awesome improvement over the current behaviour !
  • The action box associated with table views (from tableview.py) has been transformed into a nice-looking series of small tabs; it means that the possible actions are immediately visible and need not be discovered by clicking on an almost invisible icon on the upper right.
  • The uicfg module has moved to web/views/ and ui configuration objects are now selectable. This will reduce the amount of subclassing and whole methods replacement usually needed to customize the ui behaviour in many cases.
  • Remove changelog view, as neither cubicweb nor known cubes/applications were properly feeding related files.

Other changes

  • 'pyrorql' sources will be automatically updated to use an URL to locate the source rather than configuration option. 'zmqrql' sources were broken before this change, so no upgrade is needed...
  • Debugging filters for Hooks and Operations have been added.
  • Some cubicweb-ctl commands used to show the output of msgcat and msgfmt; they don't anymore.