There has been a growing interest in ZMQ in the past months, due to its ability to efficiently deal with message passing, while being light and robust.
We have worked on introducing ZMQ in the CubicWeb framework for various uses :
- As a replacement/alternative to the Pyro source, that is used to connect to distant instances. ZMQ may be used as a lighter and more efficient alternative to Pyro. The main idea here is to use the send_pyobj/recv_pyobj API of PyZMQ (python wrapper of ZMQ) to execute methods on the distant Repository in a totally transparent way for CubicWeb.
- As a JSONServer. Indeed, ZMQ could be used to share data between a server and any requests done through ZMQ. The request is just a string of RQL, and the response is the result set formatted in Json.
- As the building block for a simple notification (publish/subscribe) system between CubicWeb instances. A component can register its interest in a particular topic, and receive a callback whenever a corresponding message is received. At this point, this mechanism is used in CubicWeb to notify other instances that they should invalidate their caches when an entity is deleted.
CubicWeb 3.15 introduces a bunch of new functionalities. In short (more details below):
- ability to use ZMQ instead of Pyro to connect to repositories
- ZMQ inter-instances messages bus
- new LDAP source using the datafeed approach, much more flexible than the legacy 'ldapuser' source
- full undo support
Plus some refactorings regarding Ajax function calls, WSGI, the registry, etc. Read more for the detail.
- Add ZMQ server, based on the cutting edge ZMQ socket
library. This allows to access distant instances, in a similar way as Pyro.
- Publish/subscribe mechanism using ZMQ for communication among cubicweb
instances. The new zmq-address-sub and zmq-address-pub configuration variables
define where this communication occurs. As of this release this mechanism is
used for entity cache invalidation.
- Improved WSGI support. While there are still some caveats, most of the code
which was twisted only is now generic and allows related functionalities to work
with a WSGI front-end.
- Full undo/transaction support: undo of modifications has finally been
implemented, and the configuration simplified (basically you activate it or not
on an instance basis).
- Controlling HTTP status code returns is now much easier:
- WebRequest now has a status_out attribute to control the response status ;
- most web-side exceptions take an optional status argument.
The base registry implementation has been moved to a new
logilab.common.registry module (see #1916014). This includes code from :
- cubicweb.vreg (everything that was in there)
- cw.appobject (base selectors and all).
In the process, some renaming was done:
- the top level registry is now RegistryStore (was VRegistry), but that
should not impact CubicWeb client code;
- former selectors functions are now known as "predicate", though you still use
predicates to build an object'selector;
- for consistency, the objectify_selector decorator has hence been renamed to
- on the CubicWeb side, the selectors module has been renamed to
Debugging refactoring dropped the need for the lltrace decorator. There
should be full backward compat with proper deprecation warnings. Notice the
yes predicate and objectify_predicate decorator, as well as the
traced_selection function should now be imported from the
All login forms are now submitted to <app_root>/login. Redirection to requested
page is now handled by the login controller (it was previously handled by the
Publisher.publish has been renamed to Publisher.handle_request. This
method now contains a generic version of the logic previously handled by
Twisted. Controller.publish is not affected.
- New 'ldapfeed' source type, designed to replace 'ldapuser' source with
data-feed (i.e. copy based) source ideas.
- New 'zmqrql' source type, similar to 'pyrorql' but using ømq instead of Pyro.
- A new registry called 'services' has appeared, where you can register
server-side cubicweb.server.Service child classes. Their call method can be
invoked from a web-side AppObject instance using the new self._cw.call_service
method or a server-side one using self.session.call_service. This is a new
way to call server-side methods, much cleaner than monkey patching the
Repository class, which becomes a deprecated way to perform similar tasks.
- a new ajaxfunction registry now hosts all remote functions (i.e. functions
callable through the asyncRemoteExec JS api). A convenience ajaxfunc
decorator will let you expose your python functions easily without all the
appobject standard boilerplate. Backwards compatibility is preserved.
- the 'json' controller is now deprecated in favor of the 'ajax' one.
- WebRequest.build_url can now take a __secure__ argument. When True, cubicweb
tries to generate an https url.
A new 'undohistory' view exposes the undoable transactions and gives access to undo
some of them.
What's new in CubicWeb 3.16?
- Add a new dataimport store (SQLGenObjectStore). This store enables a fast
import of data (entity creation, link creation) in CubicWeb, by directly
flushing information in SQL. This may only be used with PostgreSQL, as it
requires the 'COPY FROM' command.
Orm: set_attributes and set_relations are unified (and
deprecated) in favor of cw_set that works in all cases.
db-api/configuration: all the external repository connection information is
now in an URL (see #2521848),
allowing to drop specific options of pyro nameserver host, group, etc and fix
broken ZMQ source. Configuration related changes:
- Dropped 'pyro-ns-host', 'pyro-instance-id', 'pyro-ns-group' from the client side
configuration, in favor of 'repository-uri'. NO MIGRATION IS DONE,
supposing there is no web-only configuration in the wild.
- Stop discovering the connection method through repo_method class attribute
of the configuration, varying according to the configuration class. This is
a first step on the way to a simpler configuration handling.
DB-API related changes:
- Stop indicating the connection method using ConnectionProperties.
- Drop _cnxtype attribute from Connection and cnxtype from
Session. The former is replaced by a is_repo_in_memory property
and the later is totaly useless.
- Turn repo_connect into _repo_connect to mark it as a private function.
- Deprecate in_memory_cnx which becomes useless, use _repo_connect instead
the "tcp://" uri scheme used for ZMQ
communications (in a way reminiscent of Pyro) is now named
"zmqpickle-tcp://", so as to make room for future zmq-based lightweight
communications (without python objects pickling).
Request.base_url gets a secure=True optional parameter that yields
an https url if possible, allowing hook-generated content to send
secure urls (e.g. when sending mail notifications)
Dataimport ucsvreader gets a new boolean ignore_errors
- Drop of cubicweb.web.uicfg.AutoformSectionRelationTags.bw_tag_map,
deprecated since 3.6.
- The RQL search bar has now some auto-completion support. It means
relation types or entity types can be suggested while typing. It is
an awesome improvement over the current behaviour !
- The action box associated with table views (from tableview.py)
has been transformed into a nice-looking series of small tabs; it
means that the possible actions are immediately visible and need not
be discovered by clicking on an almost invisible icon on the upper
- The uicfg module has moved to web/views/ and ui configuration
objects are now selectable. This will reduce the amount of
subclassing and whole methods replacement usually needed to
customize the ui behaviour in many cases.
- Remove changelog view, as neither cubicweb nor known cubes/applications
were properly feeding related files.
- 'pyrorql' sources will be automatically updated to use an URL to locate the source
rather than configuration option. 'zmqrql' sources were broken before this change,
so no upgrade is needed...
- Debugging filters for Hooks and Operations have been added.
- Some cubicweb-ctl commands used to show the output of msgcat and
msgfmt; they don't anymore.