Topics

To be decided. Some possible topics are :

• optimization (still)
• porting cubicweb to python3
• porting cubicweb to pypy
• persistent sessions
• finish twisted / wsgi refactoring
• inter-instance communication bus
• use subprocesses to handle datafeeds
• developing more debug-tools (debug console, view profiling, etc.)
• pluggable / unpluggable external sources (as needed for the cubipedia and semantic family)
• client-side only applications (javascript + http)
• mercurial storage backend: see this thread of the mailing list
• mercurial-server integration: see this email to the mailing list

other ideas are welcome, please bring them up on cubicweb@lists.cubicweb.org

Location

This sprint will take place from in february 2012 from tuesday the 7th to friday the 10th. You are more than welcome to come along, help out and contribute. An introduction is planned for newcomers.

Network resources will be available for those bringing laptops.

Address : 104 Boulevard Auguste-Blanqui, Paris. Ring "Logilab" (googlemap)

Metro : Glacière

Contact : http://www.logilab.fr/contact

Dates : 07/02/2012 to 10/02/2012

Geonames

Geonames is an open-source compilation of geographical data from various sources:

"...The GeoNames geographical database covers all countries and contains over eight million placenames that are available for download free of charge..." (http://www.geonames.org)

The data is available as a dump containing different CSV files:

• allCountries: main file containing information about 8,000,000 places in the world. We won't detail the various attributes of each location, but we will focus on some important properties, such as population and elevation. Moreover, admin_code_1 and admin_code_2 will be used to link the different locations to the corresponding AdministrativeRegion, and feature_code will be used to link the data to the corresponding type.
• admin1CodesASCII.txt and admin2Codes.txt detail the different administrative regions, that are parts of the world such as region (Ile-de-France), department (Department of Yvelines), US counties...
• featureCodes.txt details the different types of location that may be found in the data, such as forest(s), first-order administrative division, aqueduct, research institute, ...
• timeZones.txt, countryInfo.txt, iso-languagecodes.txt are additional files prodividing information about timezones, countries and languages. They will be included in our CubicWeb database but won't be explained in more details here.

The Geonames website also provides some ways to browse the data: by Countries, by Largest Cities, by Highest mountains, by postal codes, etc. We will see that CubicWeb could be used to automatically create such ways of browsing data while allowing far deeper queries. There are two main challenges when dealing with such data:

• the number of entries: with 8,000,000 placenames, we have to use efficient tools for storing and querying them.
• the structure of the data: the different types of entries are separated in different files, but should be merged for efficient queries (i.e. we have to rebuild the different links between entities, e.g Location to Country or Location to AdministrativeRegion).

Data model

With CubicWeb, the data model of the application is written in Python. It defines different entity classes with their attributes, as well as the relationships between the different entity classes. Here is a sample of the schema.py that we have used for Geonames data:

class Location(EntityType):
    name = String(maxsize=1024, indexed=True)
    uri = String(unique=True, indexed=True)
    geonameid = Int(indexed=True)
    latitude = Float(indexed=True)
    longitude = Float(indexed=True)
    feature_code = SubjectRelation('FeatureCode', cardinality='?*', inlined=True)
    country = SubjectRelation('Country', cardinality='?*', inlined=True)
    main_administrative_region = SubjectRelation('AdministrativeRegion',
                              cardinality='?*', inlined=True)
    timezone = SubjectRelation('TimeZone', cardinality='?*', inlined=True)
    ...

This indicates that the main Location class has a name attribute (string), an uri (string), a geonameid (integer), a latitude and a longitude (both floats), and some relation to other entity classes such as FeatureCode (the relation is named feature_code), Country (the relation is named country), or AdministrativeRegion called main_administrative_region.

The cardinality of each relation is classically defined in a similar way as RDBMS, where * means any number, ? means zero or one and 1 means one and only one.

We give below a visualisation of the schema (obtained using the /schema relative url)

Any X LIMIT 10 WHERE X is Location, X population > 1000000,
    X country C, C name "France"

SELECT _X.cw_eid FROM cw_Country AS _C, cw_Location AS _X
WHERE _X.cw_population>1000000
      AND _X.cw_country=_C.cw_eid AND _C.cw_name="France"
LIMIT 10

class IGeocodableAdapter(EntityAdapter):
      __regid__ = 'IGeocodable'
      __select__ = is_instance('Location')
      @property
      def latitude(self):
          return self.entity.latitude
      @property
      def longitude(self):
          return self.entity.longitude

http://baseurl:port/?rql=The query that I want&vid=Identifier-of-the-view

class LocationPopulationFacet(facet.RangeFacet):
    __regid__ = 'population-facet'
    __select__ = is_instance('Location')
    order = 2
    rtype = 'population'

Geonames in CubicWeb

The main page of the Geoname application is illustrated in the screenshot below. It provides general information on the database, in particular the number of entities in the different classes:

• 7,984,330 locations.
• 59,201 administrative regions (e.g. regions, counties, departments...)
• 7,766 languages.
• 656 features (e.g. types of location).
• 410 time zones.
• 252 countries.
• 7 continents.

Simple query

We will first illustrate the possibilites of CubicWeb with the simple query that we have detailed before (that could be directly pasted in the url...):

Any X LIMIT 10 WHERE X is Location, X population > 1000000,
    X country C, C name "France"

We obtain the following page:

This is the standard view of CubicWeb for displaying results. We can see (right box) that we obtain 10 locations that are indeed located in France, with a population of more than 1,000,000 inhabitants. The left box shows the search panel that could be used to launch queries, and the facet filters that may be used for filtering results, e.g. we may ask to keep only results with a population greater than 4,767,709 inhabitants within the previous results:

and we obtain now only 4 results. We can also notice that the facets are linked: by restricting the result set using the population facet, the other facets also restricted their possibilities.

Simple query (but with more information !)

Let's say that we now want more information about the results that we have obtained previously (for example the exact population, the elevation and the name). This is really simple ! We just have to ask within the RQL query what we want (of course, the names N, P, E of the variables could be almost anything...):

Any N, P, E LIMIT 10 WHERE X is Location,
    X population P, X population > 1000000,
    X elevation E, X name N, X country C, C name "France"

The empty column for the elevation simply means that we don't have any information about elevation.

Anyway, we can see that fetching particular information could not be simpler! Indeed, with more complex queries, we can access countless information from the Geonames database:

Any N,E,LA,LO ORDERBY E DESC LIMIT 10  WHERE X is Location,
      X latitude LA, X longitude LO,
      X elevation E, NOT X elevation NULL, X name N,
      X country C, C name "France"

which means:

Give me the 10 highest locations (the 10 first when sorting by decreasing elevation) with their name, elevation, latitude and longitude that are in a country named "France"

We can now use another view on the same request, e.g. on a map (view mapstraction.map):

Any X ORDERBY E DESC LIMIT 10  WHERE X is Location,
       X latitude LA, X longitude LO, X elevation E,
       NOT X elevation NULL, X country C, C name "France"

And now, we can add the fact that we want more results (20), and that the location should have a non-null population:

Any N, E, P, LA, LO ORDERBY E DESC LIMIT 20  WHERE X is Location,
       X latitude LA, X longitude LO,
       X elevation E, NOT X elevation NULL, X population P,
       X population > 0, X name N, X country C, C name "France"

... and on a map ...

Source Datasets

We used two of the datasets available on data.gouv.fr:

• Train stations : description of the 6442 train stations in France, including their name, type and geographic coordinates. Here is a sample of the file

  441000;St-Germain-sur-Ille;Desserte Voyageur;48,23955;-1,65358
  441000;Montreuil-sur-Ille;Desserte Voyageur-Infrastructure;48,3072;-1,6741
  

• LevelCrossings : description of the 18159 level crossings on french railways, including their type and location. Here is a sample of the file

  558000;PN privé pour voitures avec barrières sans passage piétons accolé;48,05865;1,60697
  395000;PN privé pour voitures avec barrières avec passage piétons accolé public;;48,82544;1,65795
  

Data Model

Given the above datasets, we wrote the following data model to store the data in CubicWeb:

class Location(EntityType):
    name = String(indexed=True)
    latitude = Float(indexed=True)
    longitude = Float(indexed=True)
    feature_type = SubjectRelation('FeatureType', cardinality='?*')
    data_source = SubjectRelation('DataGovSource', cardinality='1*', inlined=True)

class FeatureType(EntityType):
    name = String(indexed=True)

class DataGovSource(EntityType):
    name = String(indexed=True)
    description = String()
    uri = String(indexed=True)
    icon = String()

The Location object is used for both train stations and level crossings. It has a name (text information), a latitude and a longitude (numeric information), it can be linked to multiple FeatureType objects and to a DataGovSource. The FeatureType object is used to store the type of train station or level crossing and is defined by a name (text information). The DataGovSource object is defined by a name, a description and a uri used to link back to the source data on data.gouv.fr.

Schema of the data model

Data Import

We had to write a few lines of code to benefit from the massive data import feature of CubicWeb before we could load the content of the CSV files with a single command:

$ cubicweb-ctl import-datagov-location datagov_geo gare.csv-fr.CSV  --source-type=gare
$ cubicweb-ctl import-datagov-location datagov_geo passage_a_niveau.csv-fr.CSV  --source-type=passage

In less than a minute, the import was completed and we had:

• 2 DataGovSource objects, corresponding to the two data sets,
• 24 FeatureType objects, corresponding to the different types of locations that exist (e.g. Non exploitée, Desserte Voyageur, PN public isolé pour piétons avec portillons or PN public pour voitures avec barrières gardé avec passage piétons accolé manoeuvré à distance),
• 24601 Locations, corresponding to the different train stations and level crossings.

Data visualization

CubicWeb allows to build complex applications by assembling existing components (called cubes). Here we used a cube that wraps the Mapstraction and the OpenLayers libraries to display information on maps using data from OpenStreetMap.

In order for the Location type defined in the data model to be displayable on a map, it is sufficient to write the following adapter:

class IGeocodableAdapter(EntityAdapter):
      __regid__ = 'IGeocodable'
      __select__ = is_instance('Location')
      @property
      def latitude(self):
          return self.entity.latitude
      @property
      def longitude(self):
          return self.entity.longitude

That was it for the development part! The next step was to use the application to browse the structure of the french train network on the map.

Train stations in use:

Train stations not in use:

Zooming on some parts of the map, for example Brittany, we get to see more details and clicking on the train icons gives more information on the corresponding Location.

Train stations in use:

Train stations not in use:

Since CubicWeb separates querying the data and displaying the result of a query, we can switch the view to display the same data in tables or to export it back to a CSV file.

Querying Data

CubicWeb implements a query langage very similar to SPARQL, that makes the data available without the need to learn a specific API.

• Example 1: http:/some.url.demo/?rql=Any X WHERE X is Location, X name LIKE "%miny"

  This request gives all the Location with a name that ends with "miny". It returns only one element, the Firminy train station.

• Example 2: http:/some.url.demo/?rql=Any X WHERE X is Location, X name LIKE "%ny"

  This request gives all the Location with a name that ends with "ny", and return 112 trainstations.

• Example 3: http:/some.url.demo/?rql=Any X WHERE X latitude < 47.8, X latitude>47.6, X longitude >-1.9, X longitude<-1.8

  This request gives all the Location that have a latitude between 47.6 and 47.8, and a longitude between -1.9 and -1.8.

  We obtain 11 Location (9 levelcrossings and 2 trainstations). We can map them using the view mapstraction.map that we describe previously.

  

• Example 4: http:/domainname:8080/?rql=Any X WHERE X latitude < 47.8, X latitude>47.6, X longitude >-1.9, X longitude<-1.8, X feature_type F, F name "Desserte Voyageur"

  Will limit the previous results set to train stations that are used for passenger service:

  

• Example 5: http:/domainname:8080/?rql=Any X WHERE X feature_type F, F name "PN public pour voitures sans barrières sans SAL"&vid=mapstraction.map

  Finally, one can map all the level crossings for vehicules without barriers (there are 3704):

  

As you could see in the last URL, the map view was chosen directly with the parameter vid, meaning that the URL is shareable and can be easily included in a blog with a iframe for example.

Data sharing

The result of a query can also be "displayed" in RDF, thus allowing users to download a semantic version of the information, without having to do the preprocessing themselves:

<rdf:Description rdf:about="cwuri24684b3a955d4bb8830b50b4e7521450">
  <rdf:type rdf:resource="http://ns.cubicweb.org/cubicweb/0.0/Location"/>
  <cw:cw_source rdf:resource="http://some.url.demo/"/>
  <cw:longitude rdf:datatype="http://www.w3.org/2001/XMLSchema#float">-1.89599</cw:longitude>
  <cw:latitude rdf:datatype="http://www.w3.org/2001/XMLSchema#float">47.67778</cw:latitude>
  <cw:feature_type rdf:resource="http://some.url.demo/7222"/>
  <cw:data_source rdf:resource="http://some.url.demo/7206"/>
</rdf:Description>

Conclusion

For someone who knows the CubicWeb framework, a couple hours are enough to create a CubicWeb application that stores, displays, queries and shares data downloaded from http://www.data.gouv.fr/

The full source code for the above will be released before the end of the week.

If you want to see more of CubicWeb in action, browse http://data.bnf.fr or learn how to develop your own application at http://docs.cubicweb.org/

Non technical (albeit important) considerations

Customers that want to run such a medium traffic web site either tell you which hosting company they partner with, or ask you to find one, so you have no other choice to deal with an external hosting structure to manage the servers. I prefer this by the way because:

1. High Availability (HA) hosting really requires skills and hardware that are neither common nor cheap;
2. HA hosting requires 24/7/365 availability that SecondWeb could not (and did not even want to) offer.

It is clearly difficult for all parties (try to put yourself in the shoes of the customer...) to manage a website with 3 partners involved, each with their own goals. From the development leader point of view, you will notice that the technical people of the hosting company continuously change and you keep seeing the same operational errors even if you provide and keep improving high quality documentation. The software upgrade documentation has to be particularly clear as it greatly influences the overall web site availability. You also have to keep an history of the interventions on the servers yourself and maintain an up-to-date copy of the configuration files.

The overall architecture proposed here partly benefits from this experience with managed hosting company, in that we tried to keep it simple.

Which traffic size ? Why not bigger ?

The architecture proposed here has been successfully tested with sites delivering web pages to up to 2 millions unique visitors per month. It should scale further up depending on your site database access needs: if you need very fresh data and have a lot of write operations to the database, you will need to distribute database access amongst several servers, which is beyond the scope of this post.

This is the main limitation of the proposed architecture and the reason why it is not well-suited for a bigger traffic.

Design choices

http_port 81 defaultsite=www.example.org vhost
acl portA myport 81

http_port 82 defaultsite=www.example.org vhost
acl portB myport 82

acl site1 dstdomain www.example.org

cache_peer 127.0.0.1 parent 8081 0 no-query originserver default name=server_1
cache_peer_access server_1 allow portA site1
cache_peer_access server_1 deny all

cache_peer 127.0.0.1 parent 8082 0 no-query originserver default name=server_2
cache_peer_access server_2 allow portB site1
cache_peer_access server_2 deny all

from cubicweb import typed_eid
from cubicweb.selectors import yes
from cubicweb.web.views.editcontroller import EditController


class CustomEditController(EditController):
    __select__ = EditController.__select__ & yes()

    def handle_updated_image(self, old_eid):
        'modify submitted form to change old_eid into a new entity eid in all key/ values'
        old_eid = unicode(old_eid)
        form = self._cw.form
        new_eid = self._cw.varmaker.next()
        # handle image eid
        del form['__type:%s' % old_eid]
        form['__type:%s' % new_eid] = u'Image'
        # handle eid list
        index = form['eid'].index(old_eid)
        form['eid'] = form['eid'][:index] + [new_eid] + form['eid'][index+1:]
        # handle attribute and relations
        for (k, v) in form.iteritems():
            if v == old_eid:
                form[k] = new_eid
            if k.endswith(u':%s' % old_eid):
                form[k[:-len(old_eid)] + new_eid] = v
                del form[k]

    def _default_publish(self):
        # implement image creation when data image was updated, so that we can use
        # a far expiry date cache on download view
        images = []
        for (k, v) in self._cw.form.iteritems():
            if v != 'Image' or not k.startswith('__type') or k == self._cw.form['__maineid']:
                continue
            try:
                eid = typed_eid(k[7:])
            except ValueError:
                continue
            if self._cw.form.get('data-subject:%s' % eid, None):
                self.handle_updated_image(eid)
                images.append(eid)
        super(CustomEditController, self)._default_publish()
        for eid in images:
            self._cw.execute('DELETE Image I WHERE I eid %(eid)s', {'eid': eid})

from cubicweb.selectors import yes
from cubicweb.web import httpcache
from cubicweb.web.views.idownloadable import DownloadView

class CustomDownloadView(DownloadView):
    __select__ = DownloadView.__select__ & yes()
    http_cache_manager = httpcache.MaxAgeHTTPCacheManager
    cache_max_age = 3600

from socket import gethostname

from cubicweb.view import View


class _MonitoringView(View):
    __abstract__ = True
    __select__ = yes()
    content_type = 'text/plain'
    templatable = False


class PoolMonitoringView(_MonitoringView):
    __regid__ = 'monitor_pool'

    def call(self):
        repo = self._cw.cnx._repo
        max_pool = self._cw.vreg.config['connections-pool-size']
        percent = ((max_pool - repo._available_pools.qsize()) * 100.0) / max_pool
        self.w(u'%s%%' % percent)


class DBMonitoringView(_MonitoringView):
    __regid__ = 'monitor_db'

    def call(self):
        try:
            count = self._cw.execute('Any COUNT(X) WHERE X is CWUser')[0][0]
            self.w(u'ServiceOK : %s users in DB' % count)
        except:
            self.w(u'ServiceKO')


class VersionMonitoringView(_MonitoringView):
    __regid__ = 'monitor_version'

    def versions_text(self, versions):
        return u' | '.join(cube + u': ' + u'.'.join(unicode(x) for x in version)
                           for (cube, version) in versions)

    def call(self):
        config = self._cw.vreg.config
        vc_config = config.vc_config()
        db_config = [('cubicweb', vc_config.get('cubicweb', '?'))]
        fs_config = [('cubicweb', config.cubicweb_version())]
        for cube in sorted(config.cubes()):
            db_config.append((cube, vc_config.get(cube, '?')))
            try:
                fs_version = config.cube_version(cube)
            except:
                fs_version = '?'
            fs_config.append((cube, fs_version))
        db_config = self.versions_text(db_config)
        fs_config = self.versions_text(fs_config)
        if db_config == fs_config:
            self.w(u'ServiceOK : FS config %s == DB config %s' % (fs_config, db_config))
        else:
            self.w(u'ServiceKO : FS config %s !$ DB config %s' % (fs_config, db_config))


class HostnameMonitoringView(_MonitoringView):
    __regid__ = 'monitor_hostname'

    def call(self):
        self.w(unicode(gethostname()))

Sketch of the architecture and conclusion

There is a sketch of the proposed architecture. Please comment on it and share your experience on the topic, I would be happy to learn your tips and tricks.

I would conclude with an important remark regarding performance: a good scalable architecture is of great help to run a busy web site smoothly, however the performance boost you get by optimizing your software performance is usually worth it and must be seriously considered before any hardware upgrade, may it seem costly at first glance.

Step 1: tired of the default look?

OK... Now our site has its most desired features. But... I would like to make it look somewhat like my website. It is not www.cubicweb.org after all. Let's tackle this first!

The first thing we can to is to change the logo. There are various way to achieve this. The easiest way is to put a logo.png file into the cube's data directory. As data files are looked at according to cubes order (CubicWeb resources coming last), that file will be selected instead of CubicWeb's one.

Though there are some cases where you don't want to use a logo.png file. For instance if it's a JPEG file. You can still change the logo by defining in the cube's uiprops.py file:

LOGO = data('logo.jpg')

The uiprops machinery has been introduced in CubicWeb 3.9. It is used to define some static file resources, such as the logo, default Javascript / CSS files, as well as CSS properties (we'll see that later).

The nice thing is that in debug mode, change to a uiprops.py file are detected and then automatically reloaded.

Now, as it's a photos web-site, I would like to have a photo of mine as background... After some trials I won't detail here, I've found a working recipe explained here. All I've to do is to override some stuff of the default CubicWeb user interface to apply it as explained.

The first thing to to get the <img/> tag as first element after the <body> tag. If you know a way to avoid this by simply specifying the image in the CSS, tell me! The easiest way to do so is to override the HTMLPageHeader view, since that's the one that is directly called once the <body> has been written. How did I find this? By looking in the cubiweb.web.views.basetemplates module, since I know that global page layouts sits there. I could also have grep the "body" tag in cubicweb.web.views... Finding this was the hardest part. Now all I need is to customize it to write that img tag, as below:

class HTMLPageHeader(basetemplates.HTMLPageHeader):
    # override this since it's the easier way to have our bg image
    # as the first element following <body>
    def call(self, **kwargs):
        self.w(u'<img id="bg-image" src="%sbackground.jpg" alt="background image"/>'
               % self._cw.datadir_url)
        super(HTMLPageHeader, self).call(**kwargs)


def registration_callback(vreg):
    vreg.register_all(globals().values(), __name__, (HTMLPageHeader))
    vreg.register_and_replace(HTMLPageHeader, basetemplates.HTMLPageHeader)

As you may have guessed, my background image is in a background.jpg file in the cube's data directory, but there are still some things to explain to newcomers here:

• The call method is there the main access point of the view. It's called by the view's render method. It is not the only access point for a view, but this will be detailed later.
• Calling self.w writes something to the output stream. Except for binary views (which do not generate text), it must be passed an Unicode string.
• The proper way to get a file in data directory is to use the datadir_url attribute of the incoming request (e.g. self._cw).

I won't explain again the registration_callback stuff, you should understand it now! If not, go back to previous posts in the series :)

Fine. Now all I've to do is to add a bit of CSS to get it to behave nicely (which is not the case at all for now). I'll put all this in a cubes.sytweb.css file, stored as usual in our data directory:

/* fixed full screen background image
 * as explained on http://webdesign.about.com/od/css3/f/blfaqbgsize.htm
 *
 * syt update: set z-index=0 on the img instead of z-index=1 on div#page & co to
 * avoid pb with the user actions menu
 */
img#bg-image {
    position: fixed;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
    z-index: 0;
}

div#page, table#header, div#footer {
    background: transparent;
    position: relative;
}

/* add some space around the logo
 */
img#logo {
    padding: 5px 15px 0px 15px;
}

/* more dark font for metadata to have a chance to see them with the background
 *  image
 */
div.metadata {
    color: black;
}

You can see here stuff explained in the cited page, with only a slight modification explained in the comments, plus some additional rules to make things somewhat cleaner:

• a bit of padding around the logo
• darker metadata which appears by default below the content (the white frame in the page)

To get this CSS file used everywhere in the site, I have to modify the uiprops.py file introduced above:

STYLESHEETS = sheet['STYLESHEETS'] + [data('cubes.sytweb.css')]

Here we simply want our CSS in addition to CubicWeb's base CSS files, so we redefine the STYLESHEETS variable to existing CSS (accessed through the sheet variable) with our one added. I could also have done:

sheet['STYLESHEETS'].append(data('cubes.sytweb.css'))

But this is less interesting since we don't see the overriding mechanism...

At this point, the site should start looking good, the background image being resized to fit the screen.

The final touch: let's customize CubicWeb's CSS to get less orange... By simply adding

contextualBoxTitleBg = incontextBoxTitleBg = '#AAAAAA'

and reloading the page we've just seen, we know have a nice greyed box instead of the orange one:

This is because CubicWeb's CSS include some variables which are expanded by values defined in uiprops file. In our case we controlled the properties of the CSS background property of boxes with CSS class contextualBoxTitleBg and incontextBoxTitleBg.

Step 2: configuring boxes

Boxes present to the user some ways to use the application. Let's first do a few user interface tweaks in our views.py file:

from cubicweb.selectors import none_rset
from cubicweb.web.views import bookmark
from cubes.zone import views as zone
from cubes.tag import views as tag

# change bookmarks box selector so it's only displayed on startup views
bookmark.BookmarksBox.__select__ = bookmark.BookmarksBox.__select__ & none_rset()
# move zone box to the left instead of in the context frame and tweak its order
zone.ZoneBox.context = 'left'
zone.ZoneBox.order = 100
# move tags box to the left instead of in the context frame and tweak its order
tag.TagsBox.context = 'left'
tag.TagsBox.order = 102
# hide similarity box, not interested
tag.SimilarityBox.visible = False

The idea is to move all boxes in the left column, so we get more space for the photos. Now, serious things: I want a box similar to the tags box but to handle the Person displayed_on File relation. We can do this simply by adding a AjaxEditRelationCtxComponent subclass to our views, as below:

from logilab.common.decorators import monkeypatch
from cubicweb import ValidationError
from cubicweb.web import uicfg, component
from cubicweb.web.views import basecontrollers

# hide displayed_on relation using uicfg since it will be displayed by the box below
uicfg.primaryview_section.tag_object_of(('*', 'displayed_on', '*'), 'hidden')

class PersonBox(component.AjaxEditRelationCtxComponent):
    __regid__ = 'sytweb.displayed-on-box'
    # box position
    order = 101
    context = 'left'
    # define relation to be handled
    rtype = 'displayed_on'
    role = 'object'
    target_etype = 'Person'
    # messages
    added_msg = _('person has been added')
    removed_msg = _('person has been removed')
    # bind to js_* methods of the json controller
    fname_vocabulary = 'unrelated_persons'
    fname_validate = 'link_to_person'
    fname_remove = 'unlink_person'


@monkeypatch(basecontrollers.JSonController)
@basecontrollers.jsonize
def js_unrelated_persons(self, eid):
    """return tag unrelated to an entity"""
    rql = "Any F + ' ' + S WHERE P surname S, P firstname F, X eid %(x)s, NOT P displayed_on X"
    return [name for (name,) in self._cw.execute(rql, {'x' : eid})]


@monkeypatch(basecontrollers.JSonController)
def js_link_to_person(self, eid, people):
    req = self._cw
    for name in people:
        name = name.strip().title()
        if not name:
            continue
        try:
            firstname, surname = name.split(None, 1)
        except:
            raise ValidationError(eid, {('displayed_on', 'object'): 'provide <first name> <surname>'})
        rset = req.execute('Person P WHERE '
                           'P firstname %(firstname)s, P surname %(surname)s',
                           locals())
        if rset:
            person = rset.get_entity(0, 0)
        else:
            person = req.create_entity('Person', firstname=firstname,
                                            surname=surname)
        req.execute('SET P displayed_on X WHERE '
                    'P eid %(p)s, X eid %(x)s, NOT P displayed_on X',
                    {'p': person.eid, 'x' : eid})

@monkeypatch(basecontrollers.JSonController)
def js_unlink_person(self, eid, personeid):
    self._cw.execute('DELETE P displayed_on X WHERE P eid %(p)s, X eid %(x)s',
                     {'p': personeid, 'x': eid})

You basically subclass to configure with some class attributes. The fname_* attributes give the name of methods that should be defined on the json control to make the AJAX part of the widget work: one to get the vocabulary, one to add a relation and another to delete a relation. These methods must start by a js_ prefix and are added to the controller using the @monkeypatch decorator. In my case, the most complicated method is the one which adds a relation, since it tries to see if the person already exists, and else automatically create it, assuming the user entered "firstname surname".

Let's see how it looks like on a file primary view:

Great, it's now as easy for me to link my pictures to people than to tag them. Also, visitors get a consistent display of these two pieces of information.

Step 3: configuring facets

The last feature we'll add today is facet configuration. If you access to the '/file' url, you'll see a set of 'facets' appearing in the left column. Facets provide an intuitive way to build a query incrementally, by proposing to the user various way to restrict the result set. For instance CubicWeb proposes a facet to restrict based on who created an entity; the tag cube proposes a facet to restrict based on tags; the zoe cube a facet to restrict based on geographical location, and so on. In that gist, I want to propose a facet to restrict based on the people displayed on the picture. To do so, there are various classes in the cubicweb.web.facet module which simply have to be configured using class attributes as we've done for the box. In our case, we'll define a subclass of RelationFacet.

views/__init__.py   # uicfg configuration, facets
views/layout.py     # header/footer/background stuff
views/components.py # boxes, adapters
views/pages.py      # index view, 404 view

from cubicweb.web import facet

class DisplayedOnFacet(facet.RelationFacet):
    __regid__ = 'displayed_on-facet'
    # relation to be displayed
    rtype = 'displayed_on'
    role = 'object'
    # view to use to display persons
    label_vid = 'combobox'

Let's say we also want to filter according to the visibility attribute. This is even simpler as we just have to derive from the AttributeFacet class:

class VisibilityFacet(facet.AttributeFacet):
    __regid__ = 'visibility-facet'
    rtype = 'visibility'

Now if I search for some pictures on my site, I get the following facets available:

Conclusion

We started to see the power behind the infrastructure provided by the framework, both on the pure ui (CSS, Javascript) side and on the Python side (high level generic classes for components, including boxes and facets). We now have, with a few lines of code, a full-featured web site with a personalized look.

Of course we'll probably want more as time goes, but we can now concentrate on making good pictures, publishing albums and sharing them with friends...