Blog entries

  • Profiling your CubicWeb instance

    2009/03/27 by Adrien Di Mascio

    If you feel that one of your pages takes more time than it should to be generated, chances are that you're making too many RQL queries. Obviously, there are other reasons but my personal experience tends to show this is first thing to track down. Luckily for us, CubicWeb provides a configuration option to log rql queries. In your all-in-one.conf file, set the query-log-file option:

    # web application query log file
    query-log-file=~/myapp-rql.log
    

    Then restart your application, reload your page and stop your application. The file myapp-rql.log now contains the list of RQL queries that were executed during your test. It's a simple text file containing lines such as:

    Any A WHERE X eid %(x)s, X lastname A {'x': 448} -- (0.002 sec, 0.010 CPU sec)
    Any A WHERE X eid %(x)s, X firstname A {'x': 447} -- (0.002 sec, 0.000 CPU sec)
    

    The structure of each line is:

    <RQL QUERY> <QUERY ARGS IF ANY> -- <TIME SPENT>
    

    Use the cubicweb-ctl exlog command to examine and summarize data found in such a file:

    adim@crater:~$ cubicweb-ctl exlog < ~/myapp-rql.log
    0.07 50 Any A WHERE X eid %(x)s, X firstname A {}
    0.05 50 Any A WHERE X eid %(x)s, X lastname A {}
    0.01 1 Any X,AA ORDERBY AA DESC WHERE E eid %(x)s, E employees X, X modification_date AA {}
    0.01 1 Any X WHERE X eid %(x)s, X owned_by U, U eid %(u)s {, }
    0.01 1 Any B,T,P ORDERBY lower(T) WHERE B is Bookmark,B title T, B path P, B bookmarked_by U, U eid %(x)s {}
    0.01 1 Any A,B,C,D WHERE A eid %(x)s,A name B,A creation_date C,A modification_date D {}
    

    This command sorts and uniquifies queries so that it's easy to see where is the hot spot that needs optimization.

    Having said all this, it would probably be worth talking about the fetch_attrs attribute you can define in your entity classes because it can greatly reduce the number of queries executed but I'll make a specific blog entry for this.

    I should finally mention the existence of the profile option in the all-in-on.conf. If set, this option will make your application run in an hotshot session and store the results in the specified file.


  • Customizing search box with magicsearch

    2009/12/13 by Adrien Di Mascio

    During last cubicweb sprint, I was asked if it was possible to customize the search box CubicWeb comes with. By default, you can use it to either type RQL queries, plain text queries or standard shortcuts such as <EntityType> or <EntityType> <attrname> <value>.

    Ultimately, all queries are translated to rql since it's the only language understood on the server (data) side. To transform the user query into RQL, CubicWeb uses the so-called magicsearch component which in turn delegates to a number of query preprocessor that are responsible of interpreting the user query and generating corresponding RQL.

    The code of the main processor loop is easy to understand:

    for proc in self.processors:
        try:
            return proc.process_query(uquery, req)
        except (RQLSyntaxError, BadRQLQuery):
            pass
    

    The idea is simple: for each query processor, try to translate the query. If it fails, try with the next processor, if it succeeds, we're done and the RQL query will be executed.

    Now that the general mechanism is understood, here's an example of code that could be used in a forge-based cube to add a new search shortcut to find tickets. We'd like to use the project_name:text syntax to search for tickets of project_name containing text (e.g pylint:warning).

    Here's the corresponding preprocessor code:

    from cubicweb.web.views.magicsearch import BaseQueryProcessor
    
    class MyCustomQueryProcessor(BaseQueryProcessor):
        priority = 0 # controls order in which processors are tried
    
        def preprocess_query(self, uquery, req):
            """
            :param uqery: the query as sent by the browser
            :param req: the standard, omnipresent, cubicweb's req object
            """
            try:
                project_name, text = uquery.split(':')
            except ValueError:
                return None # the shortcut doesn't apply
            return (u'Any T WHERE T is Ticket, T concerns P, P name %(p)s, '
                    u'T has_text %(t)s', {'p': project_name, 't': text})
    

    The code is rather self-explanatory, but here's a few additional comments:

    • the class is registered with the standard vregistry mechanism and should be defined along the views
    • the priority attribute is used to sort and define the order in which processors will be tried in the main processor loop
    • the preprocess_query returns None or raise an exception if the query can't be processed

    To summarize, if you want to customize the search box, you have to:

    1. define a new query preprocessor component
    2. define its priority wrt other standard processors
    3. implement the preprocess_query method

    and CubicWeb will do the rest !


  • CubicWeb documentation sprint in feb. 2010

    2010/01/22 by Nicolas Chauvat
    http://farm4.static.flickr.com/3042/2871708248_950831962c_s.jpg

    On February 2nd, 2010 Logilab will host in its head offices a one-day sprint dedicated to the improvement of the CubicWeb documentation.

    Get in touch with Logilab if you want to participate in person or via the net: contact at logilab dot fr.

    Photo by Adam Hyde from the FLOSS blog


  • CubicWeb documentation mini-sprint report

    2010/02/10 by Sylvain Thenault

    We held a one day sprint last week in our Paris office, trying to improve CubicWeb's documentation.

    There is a huge work to do on this, much more than we can do on a one day sprint, even with many people. But you have to begin with something :)

    So, after a quick meeting to define priorities:

    • Stéphanie, Charles and later Sandrine (from her US home-office), began to add some documentation and screenshots to cubes. They started with the following cubes: addressbook, person, basket, tag, folder, forgotpwd, forge, tracker, vcsfile, keyword, blog and comment.
    • Julien explored sphinx abilities to build the index and extract docstrings. He applied this to improve the documentation of selectors.
    • Adrien (ach) and Celso, our friend from Mexico, tackled the task to improve the tutorial from a beginner's point of view.
    • Arthur added some pieces of documentation found in our intranet, mailing-list...
    • Pyves worked on a cubicweb-ctl command to generate schema images (png) for cubes, to include them in the cube's documentation.
    • Adrien (adim) and I helped the various teams.

    Huum, I think I did not forgot anyone...

    If there is still a lot to do (we need more doc sprints, stay tuned), this is really a nice start! This site should soon be updated to include more valuable cubes description and online documentation extracted from the contributed doc.


  • Documentation progress

    2010/04/20 by Aurelien Campeas

    As part of an effort to improve the documentation (see the cw_course version) a lot of chapters have been completed (and filled with real-world examples). Many more were updated and reorganized.

    I won't list everything but here are the most important improvements:

    picture under creative commons

    Picture under Creative Commons, courtesy of digitalnoise.

    • The publishing process
    • Templates & the architecture of views
    • Primary views customizations (including use of the uicfg module)
    • Controllers
    • Hooks & Operations
    • Proper usage of the ORM
    • Unit tests
    • Breadcrumbs
    • URL rewrite
    • Using the CW javascript library

    Last but not least, a whole new tutorial based on Sylvain's great series Building my photos Web site has been included. It covers some advanced topics such as Operations and sophisticated security settings.

    The visual style has been enhanced a bit to have better readability.

    As always, patches are welcome !

    picture under Creative Commons, courtesy of digitalnoise


  • CubicWeb Monthly news february/march 2021

    2021/04/09 by Simon Chabot

    It has been quite a time since the last public news ; we will try in the letter to summarize what we did during february and march 2021. During this period, we tried to fix issues, migrate a lot of our code base to latest version of CubicWeb. We also did some archeology ; we have been looking at all the old heads in CubicWeb's repository, and we have tried to rebase them on the latest public head. A lot of merge requests have been created and are still under review.

    CubicWeb 3.30 is out!

    We released Cubicweb 3.30 on march 16th. There are no “big” changes in this release. A lot of issues have been fixed, the documentation have been improved − a new tuto is coming ! − and a few new features have been implemented.

    Here is an extract of the changelog:

    • it is now possible to use smtp authentification to send an email (#221)
    • required variables can be read from the environment (#85)
    • one can specify scripts attributes when using the add_js function (#210)
    • GROUP_CONCAT was not returning the expected results when NULL values were encountered (#109)
    • it is now possible not to drop table indexes when using the MassiveStore (#219)

    One can find the full changelog on Cubicweb release page here.

    Adding python types to RQL

    In the past months, python types have progressively been included in RQL. This is a step forward to python types in CubicWeb itself.

    A lot of work has been done, and there is still a lot of work to do. Adding types to RQL help us to simplify the refactorings. As RQL's API is not clearly defined, we will run CubicWeb's tests on each merge request of RQL, to make sure nothing breaks.

    Separate Back and Front

    For some time now, the trend has been towards a clear separation between front and back. We are adding more features to use React in the front.

    CwClientLibJS, a library allowing to use rql in the browser. CwClientLibJS is now in version 1.1.0 and is able to generate query on entity state (see !20).

    react-admin-cubicweb, new tools to generate admin pages. react-admin-cubicweb is still at an early stage but can display entity attribute and relation. It also allows edition as well. The version 0.3.0 has been released !

    Release-new

    Release-new is one of our latest utility to release new versions of our cubes. Assuming that you are using conventional commits, you should really like this tool to release new version of your cubes.

    Release-new takes care to:

    • update the version of the cube in __pkginfo__.py
    • update the debian/control if there is one
    • update the changelog
    • create a new commit with theses changes
    • tag the commit

    By default, it will try to guess if it is a major, minor or patch release (you can specify the release type if need be). The changelog will be updated and you will be able to edit it before the commit is done. The update of the changelog will be eased if you use conventionnal commit as your commit history will be dispatched in the appropriate sections. For instance, the last changelog of CubicWeb has been produced by this tool !

    Docker images

    During our last hackathon, a team worked on our CubicWeb Docker images. The code to generate them has been rewritten to be simpler. During this refactoring, we took care to generate docker images for minor versions of CubicWeb as well. Therefore, on https://hub.docker.com/r/logilab/cubicweb/tags you can find docker images for major and minor versions of CubicWeb with different versions of python, to suit your needs the best we can :)

    See you next month!