Blog entries by Arthur Lutz [24]

Brainomics - A management system for exploring and merging heterogeneous brain mapping data

2013/09/12 by Arthur Lutz

At OBHM 2013, the 19th Annual Meeting of the Organization for Human Brain Mapping, Logilab presented a poster which explains the work done using CubicWeb on brain imaging and genetics data in collaboration with INRIA, INSERM and the CEA during the Brainomics project co-financed by Agence nationale de la Rercherche.

http://www.cubicweb.org/file/3123353/raw/Screenshot%20from%202013-09-12%2010%3A27%3A27.png

You can download this poster and try the demo online.


We're going to PGDay France, the Postgresql Community conference

2013/06/11 by Arthur Lutz

A few people of the CubicWeb team are going to attend the French PostgreSQL community conference in Nantes (France) on the 13th of june.

http://www.cubicweb.org/file/2932005/raw/hdr_left.png

We're excited to learn more about the following topics that are relevant to CubicWeb's development and features :

https://www.pgday.fr/_media/pgfr2.png

Obviously we'll pay attention to all the talks during the day. If you're attending, we hope to see you there.


OpenData meets the Semantic Web at WOD2013

2013/06/10 by Arthur Lutz

With a few people from Logilab we went to the 2nd International Workshop on Open Data (WOD), on the 3rd of june.

Although the main focus was an academic take on OpenData, a lot of talks were related to the Semantic Web technologies and especially LinkedData.

http://www.logilab.org/file/144837/raw/banniere-wod2013.png

The full program (and papers) is on the following website. Here is a quick review of the things we though worth sharing.

  • privacy oriented ontologies : http://l2tap.org/
  • interesting automations done to suggest alignments when initial data is uploaded to an opendata website
  • some opendata platforms have built-in APIs to get files, one example is Socrata : http://dev.socrata.com/
  • some work is being done to scale processing of linked data in the cloud (did you know you could access ready available datasets in the Amazon cloud ? DBPedia for example )
  • the data stored in wikipedia can be a good source of vocabulary on certain machine learning tasks (and in the future, wikidata project)
  • there is an RDF extension to Google Refine (or OpenRefine), but we haven't managed to get it working out of the box,
  • WebSmatch uses morphological operators (erosion / dilation) to identify grids and zones in Excel Spreadsheets and then aligns column data on known reference values (e.g. country lists).

We naturally enjoyed the presentation made by Romain Wenz about http://data.bnf.fr with the unavoidable mention of Victor Hugo (and CubicWeb).

Thanks to the organizers of the conference and to the National French Library for hosting the event.


Links roundup from dotjs.eu

2012/12/05 by Arthur Lutz

A few people from Logilab attended the dotjs conference in Paris last week. The conference wasn't exactly what we expected, we were hoping for more technical talks. Nevertheless, some of the things we saw were quite interesting. Some of them could be relevant to CubicWeb.

http://www.cubicweb.org/file/2532779?vid=download

Here is a raw roundup of links collected last friday :


Follow up of IRI conference about Museums and the Web #museoweb

2012/04/12 by Arthur Lutz

I attented the conference organised by IRI in a series of conferences about "Muséologie, muséographie et nouvelles formes d’adresse au public" (hashtag #museoweb). This particular occurence was about "Le Web devient audiovisuel" (the web is also audio and video content). Here are a few notes and links we gathered. The event was organised by Alexandre Monnin @aamonnz.

http://polemictweet.com/2011-2012-museo-audiovisuel/images/slide4_museo_fr.png

Yves Raimond from the BBC

Yves Raimond @moustaki made a presentation about his work at the BBC around semantic web technologies and speech recognition over large quantities of digitized archives. Parts of the BCC web sites use semantic web data as the database and do mashups with external sources of data (musicbrainz, dbpedia, wikipedia). For example Tom Waits has an html web page : http://www.bbc.co.uk/music/artists/c3aeb863-7b26-4388-94e8-5a240f2be21b add .rdf at the end of the URL http://www.bbc.co.uk/music/artists/c3aeb863-7b26-4388-94e8-5a240f2be21b.rdf

He also made an introduction about the ABC-IP The Automatic Broadcast Content Interlinking Project and the Kiwi-API project that uses CMU Sphinx on Amazon Web Services to process large quantities of archives. A screenshot of Kiwi-API is shown on the BBC R&D blog. The code should be open sourced soon and should appear on the BBC R&D github page.

Following his presentation, the question was asked if using Wikipedia content on an institutional web site would be possible in France, I pointed to the use of Wikipedia on http://data.bnf.fr , for example at the bottom of the Victor Hugo page.

Raphaël Troncy about Media Fragments

Raphaël Troncy @rtroncy made a presentation about "Media Fragments" which will enable sharing parts of a video on the web. Two major features : the sharing of specific extracts and the optimization of bandwith use when streaming the extract (usefull for mobile devices for example). It is a W3C working draft : http://www.w3.org/TR/media-frags-reqs/. Here are a few links of demos and players :

Part of the presentation was about the ACAV project done jointly with Dailymotion : http://www.capdigital.com/projet-acav/

The slides of his presentation are available here : http://www.slideshare.net/troncy/addressing-and-annotating-multimedia-fragments

IRI presentation

Vincent Puig @vincentpuig and Raphaël Velt @raphv made a presentation of various projects led by IRI :

http://www.iri.centrepompidou.fr/wp-content/themes/IRI-Theme/images/logo-iri-petit_fr_fr.png

Final words

The technologies seen during this conference are often related to semantic web technologies or at least web standards. Some of the visualizations are quite impressive and could mean new uses of the Web and an inspiration for CubicWeb projects.

A few of the people present at the conference will be attending or presenting talks at SemWeb.Pro which will take place in Paris on the 2nd and 3rd of may 2012.


CubicWeb Sprint report for the "Benchmarks" team

2012/02/17 by Arthur Lutz

One team during the CubicWeb sprint looked at issues around monitoring benchmark values for CubicWeb development. This is a huge task, so we tried to stay focused on a few aspects:

  • production reponse times (using tools such as smokeping and munin)
  • response times of test executions in continuous integration tests
  • response times of test instances runinng in continuous integration

We looked at using cpu.clock() instead of cpu.time() in the xunit files that report test results so as to be a bit more independent of the load of the machine (but subprocesses won't be counted for).

Graphing test times in hudson/jenkins already exists (/job/PROJECT/BUILDID/testReport/history/?) and can also be graphed by TestClass and by individual test. What is missing so far is a specific dashboard were one could select the significant graphs to look at.

By the end of the first day we had a "lorem ipsum" test instance that is created on the fly on each hudson/jenkins build and a jmeter bench running on it, it's results processed by the performance plugin.

http://www.cubicweb.org/file/2184036?vid=download

By the end of the second day we had some visualisation of existing data collected by apycot using jqplot javascript visulation (cubicweb-jqplot):

http://www.cubicweb.org/file/2184035?vid=download

By the end of the sprint, we got patches submitted for the following cubes :

  • apycot
  • cubicweb-jqplot
  • the original jqplot library (update : patch accepted a few days later)

On the last hour of the sprint, since we had a "lorem ipsum" test application running each time the tests went through the continuous integration, we hacked up a proof of concept to get automatic screenshots of this temporary test application. So far, we get screenshots for firefox only, but it opens up possibilities for other browsers. Inspiration could be drawn from https://browsershots.org/


HTML5 features presented at Paris Web 2010 by Paul Rouget

2010/10/19 by Arthur Lutz

While at Paris Web 2010 we were all impressed by the presentation and demos by Paul Rouget on HTML5 (tech evangelist must be a hard job!). Here is my take and a few URLs on the things that were presented.

http://hacks.mozilla.org/wp-content/themes/Hacks2010/img/mozilla.png
  • Websockets with persistent connections between the server and the browser. That way you can avoid pulling information every 5 seconds, the server can tell the web page a new info is available. The immediate uses we have for this are :
    • realtime feed display
    • jabber web chat rooms
    • in cubicweb's forge : new comment indication on a ticket
    • in cubicweb in general : notification that the edited element has been openned by another user (instead of a lock mechanism)
    • real time collaborative editing (etherpad style functionality)
  • File upload demo : http://demos.hacks.mozilla.org/openweb/uploadingFiles/
  • File EXIF extraction, client side resize or geolocalisation http://demos.hacks.mozilla.org/openweb/FileAPI/ . That could be very cool for things such as resizing an image before it is sent to the server (you know, for your mother who doesn't know how to resize that 2 Mbytes photo before sending it to the site). Reference : https://developer.mozilla.org/en/Using_files_from_web_applications
  • Using File IO, you can do some heavy Drag'n'drop from your computer to your browser directly in the browser (yes, you can get rid of that nasty java applet). Apparently Google implemented in Chromium a non-standard drag'n'drop the other way around : from the web app to your desktop, which could be cool as well.
http://farm5.static.flickr.com/4147/5085028912_173337f0ba.jpg
  • XHR - XMLHttpRequest. Usually this type of requests is not possible cross-domain. Now they will be (with an authorization mechanism). That way, you will be able to post and control websites from the page in your browser.
  • Audio Data API : you can now access & modify audio files directly in your browser (before uploading them server side). This makes me think of the first time I realized people where implementing traditionally "heavy" applications (photo editing, music editing, even movie edition) in web applications. I was (and still am) very surprised and skeptic, but this kind of evolution makes me believe that there can be a day when you don't even need to send massive files to the server to edit them.
http://farm1.static.flickr.com/191/513636061_98d07f7966_t.jpg

Admittedly, you probably need to see the thrilling presentation and demos to be tempted to go and dip into these technologies. Reading the documentation will probably not encourage you to go and code some cool new features.

One of the things that the audience commented about at the end of the presentation is that there was still a huge lack of "authoring tools" for HTML5. For some coders that never leave vim or emacs, this is heresy, but we have to admit that the adoption of flash and silverlight (apparently) is very much driven by simple click'n'program tools.

http://www.mozilla.org/images/minefield_168.png

During the presentation, I used a Chrome 6 that I had lying around on my Ubuntu, but by the end of the presentation I had installed Firefox4 using the mozilla PPA

sudo add-apt-repository ppa:ubuntu-mozilla-daily/ppa
sudo apt-get update
sudo apt-get -uVf install firefox-4.0

The PPA version keeps config files separate so you can easily switch between your "standard" Firefox3 profile and the cutting edge Firefox4 (obviously the big downside is not having all your cool extensions).

The only thing missing from the presentation was the code... a request I hope Paul will grant to the community (a bunch of tweets about that followed the presentation).


CubicWeb presentation at the JDLL (Lyon)

2010/10/07 by Arthur Lutz

For the "Journées Du Logiciel Libre (JDLL)" in Lyon which will take place the 14th, 15th et 16th of octobre 2010, we will be presenting the semantic side of CubicWeb on Friday 15th. There will be a talk and a tutorial. Details can be found here and there.

If you're around, come and see us!

http://www.jdll.org/sites/default/files/banniere.png

CubicWeb sprint in Paris about js and css

2010/04/29 by Arthur Lutz

Logilab is once again hosting a sprint around CubicWeb - 5 days in our Paris offices.

The general focus will be around javascript & css :

http://www.iconarchive.com/icons/enhancedlabs/lha-objects/128/Filetype-CSS-icon.pnghttp://codesnip.net/wp-content/uploads/javascript.png
  • easily change the style of an application
  • handling of bundles merging javascript and css
  • have a clean javascript API, documented and tested
  • have documentation about the css & javascript parts in the cubicweb book

This sprint is taking place from thursday the 29th of April 2010 to the 5th of may 2010 (weekend is off limits - the offices will be closed). You are more than welcome to come along and help out, contribute, or just pair program with someone. Coming only for a day, or an afternoon is fine too... Network resources will be available for those bringing laptops.

Address : 104 Boulevard Auguste-Blanqui, Paris. Ring "Logilab".

Metro : St Jacques or Corvisart (Glacière is closest, but will be closed from monday onwards)

Contact : http://www.logilab.fr/contact

Dates : 29/04/2010 to 30/04/2010 and 03/05/2010 to 05/05/2010


Migrating cubicweb instances - benefits from a distributed architecture

2010/04/22 by Arthur Lutz

Aim : do the migration for N cubicweb instances hosted on a server to another with no downtime.

Prerequisites : have an explicit definition of the database host (not default or localhost). In our case, the database is hosted on another host. You are not migrating your pyro server. You are not using multisource (more documentation on that soon).

Steps :

  1. on new machine : install your environment (pseudocode)

    apt-get install cubicweb cubicweb-applications apache2
    
  2. on old machine : copy your cubicweb and apache configuration to the new machine

    scp /etc/cubicweb.d/ newmachine:/etc/cubicweb.d/
    scp /etc/apache2/sites-available/ newmachine:/etc/apache2/sites-available/
    
  3. on new machine : give new ids to pyro registration so the new instances can register

    cd /etc/cubicweb.d/ ; sed -i.bck 's/^pyro-instance-id=.*$/\02/' */all-in-one.conf
    
  4. on new machine : start your instances

    cubicweb start
    
  5. on new machine : enable sites and modules for apache and start it, test it using by modifying your /etc/host file.

  6. change dns entry from your oldmachine to newmachine

  7. shutdown your old machine (if it doesn't host other services or your database)

  8. That's it.

Possible enhancements : use right from the start a pound server behind your apache, that way you can add backends and smoothily migrate by shuting down backends that pound will take into account.

http://www.cubicweb.org/image/893561?vid=download

Fun with graphs in apycot

2010/03/24 by Arthur Lutz

Yesterday I had a little quick fun with apycot in the train, using the existing plots infrastructure I managed to quickly add a few graphs to the application. I only had an old dump of our apycot for mercurial (http://apycot.hg-scm.org/) so the timespan is not huge, but I like it anyway! Here are some dev screenshots while you wait for this feature your your application... The pylint grades where pretty constant so I'm not including that graph.

http://www.cubicweb.org/image/779761?vid=downloadhttp://www.cubicweb.org/image/779768?vid=download

Now, I have to make solid code and integrate it properly.


Continuous Integration platform for Mercurial with apycot

2010/03/15 by Arthur Lutz

Since the mercurial 1.5 sprint Pierre-Yves has been working on improving Continuous Integration for Mercurial. All developers are encouraged to run the test suites and code quality checkers but it's no always feasible to test every cases, different OS, different python versions, strange test dependencies, slow coverage run, etc. Moreover it's generally useful to keep track of the results of previous tests, especially for benchmarks.

At http://apycot.hg-scm.org/ you will find a production setup that now runs several variants of the tests-suite for all official repo and checks code style and documentation. Notification by email or RSS is available. For more details check out the FAQ.

apycot is open source and uses the cubicweb platform, if you want to set up one for your project, check out the step by step documentation.

http://www.cubicweb.org/image/749160?vid=download

Distributed scalable architecture using CubicWeb

2010/01/14 by Arthur Lutz

Here is a small example of one the things you can do with cubicweb's scalable architecture when serving a large number of users.

http://www.cubicweb.org/image/619085?vid=download

Obviously you can easily add machines hosting CubicWeb to the middle bit to scale up. Adding multiple postgres servers is possible but more tricky. In a later blog I will also show a way of split CubicWeb servers onto multiple servers (separate the web engine from the data repository part). Debian is one of the possible host systems, you can use something else, it's just easier with debian...

If you want a more detailed explanation of how we setup such an environment, please comment and we'll try to find the time to document it.

As a systems administrator, I can then enjoy the use of the following tools :

  • clusterssh - to access all machines at once and do common task by only typing it once (a must!)
  • htop - to monitor resources in a nicer way than the simple top
  • iotop - to monitor input/output load
  • varnishist - to check varnish is properly caching some content
  • apachetop - to watch in real time what is being accessed on the apache server
  • jnettop - to watch network flows
  • apt-get (on debian) to install all this in a a few simple commands...

Relase early, release often

2009/10/05 by Arthur Lutz

Looking at the releases of the CubicWeb projects for the month of September alone, I think we can conclude that we are applying the Agile Software Development principle quite closely.

http://farm4.static.flickr.com/3025/2732378117_cdd948fd1d_m.jpg
  • 11 releases of the cubicweb framework (now in stable and unstable flavors) : 3.5.2, 3.5.1, 3.5.0, 3.4.11, , 3.4.9, 3.4.8, 3.4.7, 3.4.6, 3.4.5, 3.4.4, 3.4.3
  • 3 releases of cubicweb-vcsfile
  • 4 releases of cubicweb-forge
  • 2 releases of cubicweb-drh
  • 2 releases of cubicweb-workorder
  • 1 release of cubicweb-conference, cubicweb-tracker, cubicweb-registration, cubicweb-timesheet, cubicweb-workcase, cubicweb-task, cubicweb-expense, cubicweb-calendar, cubicweb-invoice, cubicweb-nosylist, etc.

Hope you can keep-up or use the stable versions...

photo by kennymatic under creative commons


Graphing version progress

2009/07/06 by Arthur Lutz

As you might have noticed we've upgraded http://www.logilab.org and http://www.cubicweb.org to CubicWeb 3.3 and a bunch of cubes were upgraded too. We can now benefit from a few cool bugfixes and features on those two forges.

One of them I like and wish to mention is the graphing of a project's progress as a Burn Down Chart, you can see an example below. We're using the some jQuery magic here, and so you can roll over the mouse to get more info on the graph... (not on the screenshot below). This type of graph is generated on all the version views... This is particularly useful on some of our extranets to see the progress of a version (and if tickets were added along the way).

http://www.cubicweb.org/image/344424?vid=download

For the coders out there you can check out cubicweb/web/views/plots.py and the example in the forge cube.


What's new in cubicweb 3.3

2009/06/24 by Arthur Lutz

After the CubicWeb 3.2 blackout, the release early, release often mantra strikes back and CubicWeb 3.3 is out ! A few bugs were fixed, mainly migration scripts bug, and some new functionalities were added among which the long awaited standard plotting feature. We've added piechart support (with gchartwrapper) and standard plots with flot.

under creative commons by jared

Features

  • jquery has been updated to the latest 1.3.x version
  • plotting facilities using Flot and Google Chart have been added (replacing sometimes similar facilities using matplotlib)
  • the i18n command names have been changed
  • also a non-negligible amount of internal refactorings occurred, but this should be quite transparent

Bugs fixed

  • problems with migrations using SQL has been fixed
  • bugs with the multi-source planner have been fixed
  • problems with synchronize-schema and not-null constraints

photo licenced under CreativeCommons by jared


Cubicweb News 09.04

2009/04/28 by Arthur Lutz

In april a bunch of bugs have been corrected on the stable branch of cubicweb (3.1 series) and we've been working on the next generation series : 3.2. Here's a quick summary of what's been going on :

  • cubicweb (the framework) was released twice with 3.1.3 and 3.1.4 which fixed a few bugs in the querier and the management screens
  • cubicweb-blog 1.5.0 was released with some improvements to the graphical rendering
  • cubicweb-tag 1.4.5 was released with notable improvements to tag clouds (added colors and better scaling of tags).
  • cubicweb-file got a bugfix in 1.4.4
  • cubicweb-mailinglist got a bugfix 1.3.1.

Next up, we are working on the 3.2.0 version of cubicweb with some particular focus on :

  • form generation
  • more explicit view registration (less magic)
  • simpler workflow definitions
  • js, css and ajax improvements

Do not hesitate to try the development branch (named tls-sprint at the moment) or read the changes at http://www.logilab.org/hg/cubicweb


Cubicweb to be presented at French Linux Conference

2009/03/31 by Arthur Lutz
http://www.solutionslinux.fr/images/index_07.jpg

The CubicWeb plateform will be on display at the French conference about linux "Solution Linux" hosted in Paris in the next 3 days. You can meet us at the System@tic stand or see us talk about it during a talk about Web2 this afternoon.

More info in french on the Logilab.org Blog.


Using email with CubicWeb

2009/03/18 by Arthur Lutz

You might have noticed here and there the mysterious user "mailbot" on cubicweb.org or logilab.org (both running the CubicWeb web app). Who is this user ?

http://farm4.static.flickr.com/3244/2959912279_8446aa1abd_m.jpg

Well, one of the cool features about cubicweb is that you can interact with it simply by using your email. When you are registered on a site, you can subscribe to a software project for example, from then on, you receive notifications of the new tickets and comments on the project. When you receive such a notification you can simply do an email reply to the new ticket or new comment, and cubicweb on the receiving end will import the content of your email to the website. When the content is imported that way, it's the mailbot doing the job.

This is not rocket science, but it sure is useful. Follow the activity of the site by email and interact directly with comments and tickets from your mail client!

image by husin.sani under creative commons


What's new in CubicWeb 3.1.0

2009/03/04 by Arthur Lutz
http://www.cubicweb.org/image/212907?vid=download

Here is a brief summary of what you get for the new CubicWeb 3.1.0 release. You could obviously go though the tickets on the version page, but here is the short version.

What new features ?

  • a few OWL and Linked_Data functionalities
  • navigation is now more complete on search results
  • when installing a new cube that requires anonymous access (public site) the installer enables that access

What bugs are fixed ?

  • a few things didn't work with opera and IE6
  • json controller conflicts solved
  • the newcube command is working again
  • facets don't get in the way of the association process anymore
  • and more...

Hope you enjoy this version... to see what's coming next, you can check out the planned versions of CubicWeb : 3.1.1 and 3.2.0.


Unittesting with CubicWeb

2009/02/17 by Arthur Lutz

In test driven developpement (TDD), you write the test before you write the code. On a web application, number of levels can be tested. Here are a few hints at how we manage some of the testing with CubicWeb.

We use pytest (which is an extension of python's unittest framework available in logilab-common) to execute all tests across the cubes. Even in the core of cubicweb the tests are spread out across the server, web part, repository, common tools... so a simple pytest command crawls though all theses tests and runs them.

http://www.sqlite.org/images/SQLite.gif

The problem : One of the tricky things with testing CubicWeb is that the structure of the data is imported into the database (which enables us to easily modify the schema on running data), and that test data can be long to generate and fake for a web application that is used to talk to a proper database server (postgres). So we though of inserting test data into an sqlite database. After a bit of work on compatibility, it was up an running. But setting up that database was (and still is) quite long, testing was becoming way too long, TDD (with frequent testing) was becoming impossible.

The solution : we ended up storing the sqlite database in a temporary file which is used up if it's not too old, TDD was back in the loop. So if you're developing for CubicWeb don't worry about those test/tmpdb files, on the contrary, that means you're running tests. For writing tests, check out the content about it in the book.


RSS for latest releases

2009/01/28 by Arthur Lutz
http://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Feed-icon.svg/128px-Feed-icon.svg.png

The Cubicweb Framework can give you an RSS feed of any selection that you make. When you master RQL (Relational Query Language - more on that coming soon) you can build yourself some cool RSS feeds to follow the site's activity.

Here is one that we cooked up for you. The latest releases of packages on cubicweb, all the cubes, the framework releases straight to your RSS reader : subscribe here.


More CubicWeb releases last week

2009/01/19 by Arthur Lutz

We're still busy with the CubicWeb 3.0 releases. We did two releases of cubicweb last week : 3.0.2 and 3.0.3.

These were mainly for bugfixes, particularly about how the multisource functionality was working.


CubicWeb 3.0.1 bugfix release

2009/01/14 by Arthur Lutz

Shortly after the release of CubicWeb under the GPL licence, we've release a quickfix version to correct a few bugs :

  • XHTML validity wasn't always there because of a bug in cutting parts of texts
  • cubicweb-ctl had a few things corrected
  • permissions on certain actions we're properly placed
  • a few bugfix in the generation of the configuration

The new version is 3.0.1, you can see the corrected tickets here.