A few people from Logilab attended the dotjs conference in Paris last week. The conference wasn't exactly what we expected, we were hoping for more technical talks. Nevertheless, some of the things we saw were quite interesting. Some of them could be relevant to CubicWeb.

Here is a raw roundup of links collected last friday :

I attented the conference organised by IRI in a series of conferences about "Muséologie, muséographie et nouvelles formes d’adresse au public" (hashtag #museoweb). This particular occurence was about "Le Web devient audiovisuel" (the web is also audio and video content). Here are a few notes and links we gathered. The event was organised by Alexandre Monnin @aamonnz.

IRI presentation

Vincent Puig @vincentpuig and Raphaël Velt @raphv made a presentation of various projects led by IRI :

• Lignes des temps : http://ldt.iri.centrepompidou.fr/ldtplatform/ldt/ (here is an example of a conference about opendata during which I tweeted : RSLN open data )
• PolemicTweet http://polemictweet.com/
• Bubble-T visualization of tweets about the french presidential elections : http://www.iri.centrepompidou.fr/experimentations/bubble-t/
• HDA Labs http://hdalab.iri-research.org/hdalab/

One team during the CubicWeb sprint looked at issues around monitoring benchmark values for CubicWeb development. This is a huge task, so we tried to stay focused on a few aspects:

• production reponse times (using tools such as smokeping and munin)
• response times of test executions in continuous integration tests
• response times of test instances runinng in continuous integration

We looked at using cpu.clock() instead of cpu.time() in the xunit files that report test results so as to be a bit more independent of the load of the machine (but subprocesses won't be counted for).

Graphing test times in hudson/jenkins already exists (/job/PROJECT/BUILDID/testReport/history/?) and can also be graphed by TestClass and by individual test. What is missing so far is a specific dashboard were one could select the significant graphs to look at.

By the end of the first day we had a "lorem ipsum" test instance that is created on the fly on each hudson/jenkins build and a jmeter bench running on it, it's results processed by the performance plugin.

By the end of the second day we had some visualisation of existing data collected by apycot using jqplot javascript visulation (cubicweb-jqplot):

By the end of the sprint, we got patches submitted for the following cubes :

• apycot
• cubicweb-jqplot
• the original jqplot library (update : patch accepted a few days later)

On the last hour of the sprint, since we had a "lorem ipsum" test application running each time the tests went through the continuous integration, we hacked up a proof of concept to get automatic screenshots of this temporary test application. So far, we get screenshots for firefox only, but it opens up possibilities for other browsers. Inspiration could be drawn from https://browsershots.org/

While at Paris Web 2010 we were all impressed by the presentation and demos by Paul Rouget on HTML5 (tech evangelist must be a hard job!). Here is my take and a few URLs on the things that were presented.

• Websockets with persistent connections between the server and the browser. That way you can avoid pulling information every 5 seconds, the server can tell the web page a new info is available. The immediate uses we have for this are :
  • realtime feed display
  • jabber web chat rooms
  • in cubicweb's forge : new comment indication on a ticket
  • in cubicweb in general : notification that the edited element has been openned by another user (instead of a lock mechanism)
  • real time collaborative editing (etherpad style functionality)
• File upload demo : http://demos.hacks.mozilla.org/openweb/uploadingFiles/
• File EXIF extraction, client side resize or geolocalisation http://demos.hacks.mozilla.org/openweb/FileAPI/ . That could be very cool for things such as resizing an image before it is sent to the server (you know, for your mother who doesn't know how to resize that 2 Mbytes photo before sending it to the site). Reference : https://developer.mozilla.org/en/Using_files_from_web_applications
• Using File IO, you can do some heavy Drag'n'drop from your computer to your browser directly in the browser (yes, you can get rid of that nasty java applet). Apparently Google implemented in Chromium a non-standard drag'n'drop the other way around : from the web app to your desktop, which could be cool as well.

• XHR - XMLHttpRequest. Usually this type of requests is not possible cross-domain. Now they will be (with an authorization mechanism). That way, you will be able to post and control websites from the page in your browser.
• Audio Data API : you can now access & modify audio files directly in your browser (before uploading them server side). This makes me think of the first time I realized people where implementing traditionally "heavy" applications (photo editing, music editing, even movie edition) in web applications. I was (and still am) very surprised and skeptic, but this kind of evolution makes me believe that there can be a day when you don't even need to send massive files to the server to edit them.

Admittedly, you probably need to see the thrilling presentation and demos to be tempted to go and dip into these technologies. Reading the documentation will probably not encourage you to go and code some cool new features.

One of the things that the audience commented about at the end of the presentation is that there was still a huge lack of "authoring tools" for HTML5. For some coders that never leave vim or emacs, this is heresy, but we have to admit that the adoption of flash and silverlight (apparently) is very much driven by simple click'n'program tools.

During the presentation, I used a Chrome 6 that I had lying around on my Ubuntu, but by the end of the presentation I had installed Firefox4 using the mozilla PPA

sudo add-apt-repository ppa:ubuntu-mozilla-daily/ppa
sudo apt-get update
sudo apt-get -uVf install firefox-4.0

The PPA version keeps config files separate so you can easily switch between your "standard" Firefox3 profile and the cutting edge Firefox4 (obviously the big downside is not having all your cool extensions).

The only thing missing from the presentation was the code... a request I hope Paul will grant to the community (a bunch of tweets about that followed the presentation).

For the "Journées Du Logiciel Libre (JDLL)" in Lyon which will take place the 14th, 15th et 16th of octobre 2010, we will be presenting the semantic side of CubicWeb on Friday 15th. There will be a talk and a tutorial. Details can be found here and there.

If you're around, come and see us!

Logilab is once again hosting a sprint around CubicWeb - 5 days in our Paris offices.

The general focus will be around javascript & css :

• easily change the style of an application
• handling of bundles merging javascript and css
• have a clean javascript API, documented and tested
• have documentation about the css & javascript parts in the cubicweb book

This sprint is taking place from thursday the 29th of April 2010 to the 5th of may 2010 (weekend is off limits - the offices will be closed). You are more than welcome to come along and help out, contribute, or just pair program with someone. Coming only for a day, or an afternoon is fine too... Network resources will be available for those bringing laptops.

Address : 104 Boulevard Auguste-Blanqui, Paris. Ring "Logilab".

Metro : St Jacques or Corvisart (Glacière is closest, but will be closed from monday onwards)

Contact : http://www.logilab.fr/contact

Dates : 29/04/2010 to 30/04/2010 and 03/05/2010 to 05/05/2010

Aim : do the migration for N cubicweb instances hosted on a server to another with no downtime.

Prerequisites : have an explicit definition of the database host (not default or localhost). In our case, the database is hosted on another host. You are not migrating your pyro server. You are not using multisource (more documentation on that soon).

Steps :

1. on new machine : install your environment (pseudocode)

   apt-get install cubicweb cubicweb-applications apache2
   

2. on old machine : copy your cubicweb and apache configuration to the new machine

   scp /etc/cubicweb.d/ newmachine:/etc/cubicweb.d/
   scp /etc/apache2/sites-available/ newmachine:/etc/apache2/sites-available/
   

3. on new machine : give new ids to pyro registration so the new instances can register

   cd /etc/cubicweb.d/ ; sed -i.bck 's/^pyro-instance-id=.*$/\02/' */all-in-one.conf
   

4. on new machine : start your instances

   cubicweb start
   

5. on new machine : enable sites and modules for apache and start it, test it using by modifying your /etc/host file.

6. change dns entry from your oldmachine to newmachine

7. shutdown your old machine (if it doesn't host other services or your database)

8. That's it.

Possible enhancements : use right from the start a pound server behind your apache, that way you can add backends and smoothily migrate by shuting down backends that pound will take into account.

Yesterday I had a little quick fun with apycot in the train, using the existing plots infrastructure I managed to quickly add a few graphs to the application. I only had an old dump of our apycot for mercurial (http://apycot.hg-scm.org/) so the timespan is not huge, but I like it anyway! Here are some dev screenshots while you wait for this feature your your application... The pylint grades where pretty constant so I'm not including that graph.

Now, I have to make solid code and integrate it properly.

Since the mercurial 1.5 sprint Pierre-Yves has been working on improving Continuous Integration for Mercurial. All developers are encouraged to run the test suites and code quality checkers but it's no always feasible to test every cases, different OS, different python versions, strange test dependencies, slow coverage run, etc. Moreover it's generally useful to keep track of the results of previous tests, especially for benchmarks.

At http://apycot.hg-scm.org/ you will find a production setup that now runs several variants of the tests-suite for all official repo and checks code style and documentation. Notification by email or RSS is available. For more details check out the FAQ.

apycot is open source and uses the cubicweb platform, if you want to set up one for your project, check out the step by step documentation.

Here is a small example of one the things you can do with cubicweb's scalable architecture when serving a large number of users.

Obviously you can easily add machines hosting CubicWeb to the middle bit to scale up. Adding multiple postgres servers is possible but more tricky. In a later blog I will also show a way of split CubicWeb servers onto multiple servers (separate the web engine from the data repository part). Debian is one of the possible host systems, you can use something else, it's just easier with debian...

If you want a more detailed explanation of how we setup such an environment, please comment and we'll try to find the time to document it.

As a systems administrator, I can then enjoy the use of the following tools :

• clusterssh - to access all machines at once and do common task by only typing it once (a must!)
• htop - to monitor resources in a nicer way than the simple top
• iotop - to monitor input/output load
• varnishist - to check varnish is properly caching some content
• apachetop - to watch in real time what is being accessed on the apache server
• jnettop - to watch network flows
• apt-get (on debian) to install all this in a a few simple commands...