Blog entries

  • Profiling your CubicWeb instance

    2009/03/27 by Adrien Di Mascio

    If you feel that one of your pages takes more time than it should to be generated, chances are that you're making too many RQL queries. Obviously, there are other reasons but my personal experience tends to show this is first thing to track down. Luckily for us, CubicWeb provides a configuration option to log rql queries. In your all-in-one.conf file, set the query-log-file option:

    # web application query log file

    Then restart your application, reload your page and stop your application. The file myapp-rql.log now contains the list of RQL queries that were executed during your test. It's a simple text file containing lines such as:

    Any A WHERE X eid %(x)s, X lastname A {'x': 448} -- (0.002 sec, 0.010 CPU sec)
    Any A WHERE X eid %(x)s, X firstname A {'x': 447} -- (0.002 sec, 0.000 CPU sec)

    The structure of each line is:


    Use the cubicweb-ctl exlog command to examine and summarize data found in such a file:

    adim@crater:~$ cubicweb-ctl exlog < ~/myapp-rql.log
    0.07 50 Any A WHERE X eid %(x)s, X firstname A {}
    0.05 50 Any A WHERE X eid %(x)s, X lastname A {}
    0.01 1 Any X,AA ORDERBY AA DESC WHERE E eid %(x)s, E employees X, X modification_date AA {}
    0.01 1 Any X WHERE X eid %(x)s, X owned_by U, U eid %(u)s {, }
    0.01 1 Any B,T,P ORDERBY lower(T) WHERE B is Bookmark,B title T, B path P, B bookmarked_by U, U eid %(x)s {}
    0.01 1 Any A,B,C,D WHERE A eid %(x)s,A name B,A creation_date C,A modification_date D {}

    This command sorts and uniquifies queries so that it's easy to see where is the hot spot that needs optimization.

    Having said all this, it would probably be worth talking about the fetch_attrs attribute you can define in your entity classes because it can greatly reduce the number of queries executed but I'll make a specific blog entry for this.

    I should finally mention the existence of the profile option in the all-in-on.conf. If set, this option will make your application run in an hotshot session and store the results in the specified file.

  • Customizing search box with magicsearch

    2009/12/13 by Adrien Di Mascio

    During last cubicweb sprint, I was asked if it was possible to customize the search box CubicWeb comes with. By default, you can use it to either type RQL queries, plain text queries or standard shortcuts such as <EntityType> or <EntityType> <attrname> <value>.

    Ultimately, all queries are translated to rql since it's the only language understood on the server (data) side. To transform the user query into RQL, CubicWeb uses the so-called magicsearch component which in turn delegates to a number of query preprocessor that are responsible of interpreting the user query and generating corresponding RQL.

    The code of the main processor loop is easy to understand:

    for proc in self.processors:
            return proc.process_query(uquery, req)
        except (RQLSyntaxError, BadRQLQuery):

    The idea is simple: for each query processor, try to translate the query. If it fails, try with the next processor, if it succeeds, we're done and the RQL query will be executed.

    Now that the general mechanism is understood, here's an example of code that could be used in a forge-based cube to add a new search shortcut to find tickets. We'd like to use the project_name:text syntax to search for tickets of project_name containing text (e.g pylint:warning).

    Here's the corresponding preprocessor code:

    from cubicweb.web.views.magicsearch import BaseQueryProcessor
    class MyCustomQueryProcessor(BaseQueryProcessor):
        priority = 0 # controls order in which processors are tried
        def preprocess_query(self, uquery, req):
            :param uqery: the query as sent by the browser
            :param req: the standard, omnipresent, cubicweb's req object
                project_name, text = uquery.split(':')
            except ValueError:
                return None # the shortcut doesn't apply
            return (u'Any T WHERE T is Ticket, T concerns P, P name %(p)s, '
                    u'T has_text %(t)s', {'p': project_name, 't': text})

    The code is rather self-explanatory, but here's a few additional comments:

    • the class is registered with the standard vregistry mechanism and should be defined along the views
    • the priority attribute is used to sort and define the order in which processors will be tried in the main processor loop
    • the preprocess_query returns None or raise an exception if the query can't be processed

    To summarize, if you want to customize the search box, you have to:

    1. define a new query preprocessor component
    2. define its priority wrt other standard processors
    3. implement the preprocess_query method

    and CubicWeb will do the rest !

  • CubicWeb documentation sprint in feb. 2010

    2010/01/22 by Nicolas Chauvat

    On February 2nd, 2010 Logilab will host in its head offices a one-day sprint dedicated to the improvement of the CubicWeb documentation.

    Get in touch with Logilab if you want to participate in person or via the net: contact at logilab dot fr.

    Photo by Adam Hyde from the FLOSS blog

  • CubicWeb documentation mini-sprint report

    2010/02/10 by Sylvain Thenault

    We held a one day sprint last week in our Paris office, trying to improve CubicWeb's documentation.

    There is a huge work to do on this, much more than we can do on a one day sprint, even with many people. But you have to begin with something :)

    So, after a quick meeting to define priorities:

    • Stéphanie, Charles and later Sandrine (from her US home-office), began to add some documentation and screenshots to cubes. They started with the following cubes: addressbook, person, basket, tag, folder, forgotpwd, forge, tracker, vcsfile, keyword, blog and comment.
    • Julien explored sphinx abilities to build the index and extract docstrings. He applied this to improve the documentation of selectors.
    • Adrien (ach) and Celso, our friend from Mexico, tackled the task to improve the tutorial from a beginner's point of view.
    • Arthur added some pieces of documentation found in our intranet, mailing-list...
    • Pyves worked on a cubicweb-ctl command to generate schema images (png) for cubes, to include them in the cube's documentation.
    • Adrien (adim) and I helped the various teams.

    Huum, I think I did not forgot anyone...

    If there is still a lot to do (we need more doc sprints, stay tuned), this is really a nice start! This site should soon be updated to include more valuable cubes description and online documentation extracted from the contributed doc.

  • Documentation progress

    2010/04/20 by Aurelien Campeas

    As part of an effort to improve the documentation (see the cw_course version) a lot of chapters have been completed (and filled with real-world examples). Many more were updated and reorganized.

    I won't list everything but here are the most important improvements:

    picture under creative commons

    Picture under Creative Commons, courtesy of digitalnoise.

    • The publishing process
    • Templates & the architecture of views
    • Primary views customizations (including use of the uicfg module)
    • Controllers
    • Hooks & Operations
    • Proper usage of the ORM
    • Unit tests
    • Breadcrumbs
    • URL rewrite
    • Using the CW javascript library

    Last but not least, a whole new tutorial based on Sylvain's great series Building my photos Web site has been included. It covers some advanced topics such as Operations and sophisticated security settings.

    The visual style has been enhanced a bit to have better readability.

    As always, patches are welcome !

    picture under Creative Commons, courtesy of digitalnoise