cubicweb #1058265 upload of large files in forms freeze CW [resolved]

For some customer application, we routinely have to handle the upload of large files in forms. Typical size is 50MB.

When such an upload occurs, CW completely hangs for a significant amount of time (~20s on our test machine, about twice that much on the production server which is significantly slower).

I've instrumented the code, and the culprit seems to be cgi.FieldStorage, which is instanciated in the monkey patched requestReceived method (in etwist.server). The instanciation takes ~19s for a POST of a form with a single 50MB file. As this method is executed synchronously, everything is blocked.

Fixing this is not easy as the FieldStorage is a local variable, used to initialize attributes of the Request class before the process method is called.

This is the stack path leading to requestReceived:

File "C:\python26\lib\site-packages\win32\lib\", line 806, in SvcRun
File "c:\Documents and Settings\alf\My Documents\Pylos\cubicweb\etwist\", line 86, in SvcDoRun
File "C:\python26\lib\site-packages\twisted\internet\", line 1166, in run
File "C:\python26\lib\site-packages\twisted\internet\", line 1178, in mainLoop
File "C:\python26\lib\site-packages\twisted\internet\", line 140, in doSelect
  _logrun(selectable, _drdw, selectable, method, dict)
File "C:\python26\lib\site-packages\twisted\python\", line 84, in callWithLogger
  return callWithContext({"system": lp}, func, *args, **kw)
File "C:\python26\lib\site-packages\twisted\python\", line 69, in callWithContext
  return{ILogContext: newCtx}, func, *args, **kw)
File "C:\python26\lib\site-packages\twisted\python\", line 59, in callWithContext
  return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "C:\python26\lib\site-packages\twisted\python\", line 37, in callWithContext
  return func(*args,**kw)
File "C:\python26\lib\site-packages\twisted\internet\", line 146, in _doReadOrWrite
  why = getattr(selectable, method)()
File "C:\python26\lib\site-packages\twisted\internet\", line 460, in doRead
  return self.protocol.dataReceived(data)
File "C:\python26\lib\site-packages\twisted\protocols\", line 246, in dataReceived
  return self.rawDataReceived(data)
File "C:\python26\lib\site-packages\twisted\web\", line 1570, in rawDataReceived
File "C:\python26\lib\site-packages\twisted\web\", line 1296, in dataReceived
File "C:\python26\lib\site-packages\twisted\web\", line 1510, in _finishRequestBody
File "C:\python26\lib\site-packages\twisted\web\", line 1566, in allContentReceived
  req.requestReceived(command, path, version)
File "c:\Documents and Settings\alf\My Documents\Pylos\cubicweb\etwist\", line 368, in requestReceived
done in3.8.5
load left0.000
closed by<not specified>