Blog entries

Continuous Integration platform for Mercurial with apycot

2010/03/15 by Arthur Lutz

Since the mercurial 1.5 sprint Pierre-Yves has been working on improving Continuous Integration for Mercurial. All developers are encouraged to run the test suites and code quality checkers but it's no always feasible to test every cases, different OS, different python versions, strange test dependencies, slow coverage run, etc. Moreover it's generally useful to keep track of the results of previous tests, especially for benchmarks.

At http://apycot.hg-scm.org/ you will find a production setup that now runs several variants of the tests-suite for all official repo and checks code style and documentation. Notification by email or RSS is available. For more details check out the FAQ.

apycot is open source and uses the cubicweb platform, if you want to set up one for your project, check out the step by step documentation.

http://www.cubicweb.org/image/749160?vid=download

Debugging a memory leak in a cube

2010/09/24

We recently discovered that the cubicweb.org site (the one you are probably visiting right now) was suffering from a memory leak. The munin graphs showed a memory consumption steadily increasing soon after the instance was started, and this would only stop when all the memory on the host was exhausted. This was clearly caused by a memory leak somewhere, either in CubicWeb itself or in a cube used by the instance.

Munin graphs showing the memory leak in cubicweb.org

Fig. 1: Munin graphs showing the memory leak in cubicweb.org

Notice the associated service downtimes, and the stabilized memory consumption on Sept 23, after the leak was fixed.

Since Python has a garbage collector, either the leak was occuring in a C extension, or it was caused by some objects which were not garbage collectable. A common cause for the latter, as explained in the gc module documentation, are objects with a __del__ method which are part of a cycle.

We used the "gc" view, which is an administrative view in CubicWeb, reachable by appending "?vid=gc" at the end of the url of the root of your instance, if you are a member of the managers group. This view uses the gc module from the python standard library to see which objects are not garbage collected.

This view showed thousands of instances of mercurial.url.httphandler. This class indeed has a __del__ method and instances have a cycle with urllib2.OpenerDirector. Mercurial is used by the vcsfile cube which regularly polls remote repository over HTTP, which causes httphandler to be instantiated (and a reference to be leaked). This problem had gone undetected in mercurial because most of the time, processes using mercurial over http are shortlived and the leaked memory is quickly collected by the operating system. Discussion ensued on the IRC forum #mercurial with the developers and a patch was submitted which fixes the leak. In order to avoid the problem with versions of mercurial up to the current one, a new version of vcsfile including a monkey patch for mercurial was released and deployed on cubicweb.org!


Fun with graphs in apycot

2010/03/24 by Arthur Lutz

Yesterday I had a little quick fun with apycot in the train, using the existing plots infrastructure I managed to quickly add a few graphs to the application. I only had an old dump of our apycot for mercurial (http://apycot.hg-scm.org/) so the timespan is not huge, but I like it anyway! Here are some dev screenshots while you wait for this feature your your application... The pylint grades where pretty constant so I'm not including that graph.

http://www.cubicweb.org/image/779761?vid=downloadhttp://www.cubicweb.org/image/779768?vid=download

Now, I have to make solid code and integrate it properly.