cubicweb #3478476 Loading slowly a big table using default view [open]

Create a new cube as:

$ cubicweb-ctl newcube test_big_table
Enter a short description for your cube: dddd

Modify test_big_table/schema.py as:

from yams.buildobjs import (EntityType, RelationDefinition,
                            SubjectRelation, String, RichString,
                            BigInt, Int, Float, Boolean)
class DataEntity(EntityType):
    """ Test big table"""
    name = String(maxsize=256)  # HotSpot
    data = Float(required=True)

Create an instance as i_test_big_table:

$ cubicweb-ctl create test_big_table i_test_big_table
$ cubicweb-ctl start i_test_big_table

Create a python script "import_db_big_table.py" as below:

from cubicweb.dataimport import SQLGenObjectStore
if __name__ == "__main__":
    store = SQLGenObjectStore(session)
    for i in xrange(1000000):
        if i % 100000 == 0:
            print i
            store.flush()
        data = {"name": "%d" % i, "data": float(i)}
        store.create_entity('DataEntity', **data)
    store.flush()

Run this script to import all the data:

$ cubicweb-ctl shell i_test_big_table import_db_big_table.py

Go to the web page (http://127.0.0.1:8080) and click " DataEntity_plural" see DataEntity, it will take more than 5 seconds to open pages with only first 40 data items among 1 million data items. If there are 10 million data, it will take more than 1 minute. However, if we execute rql in shell mode and only get first 40 items, it will show it very fast:

$ cubicweb-ctl shell i_test_big_table
>>> res = session.execute("Any E where E is DataEntity")
>>> entities = res.entities()
>>> entities.next()
<Entity DataEntity 656551 ['cwuri', 'data', 'name', 'modification_date', 'creation_date'] at 55907984>

Back to web interface, I was watching the memory usage when I clicked "DataEntity_plural". The consumed memory didn't increase a lot. It means the system didn't load all the data into memory. It could be problem from too many pagination items (hypothesis from Dimitri P. and Sylvain Thénault).

I have added the codes below from Vincent Michel in test_big_table/views.py to remove not useful facets. I re-created instance and imported all the data. When I click "DataEntity_plural" see DataEntity, it is still slow.

def registration_callback(vreg):
    # Remove facets
    from cubicweb.web.views.facets import (CWSourceFacet, CreatedByFacet,
                                          InStateFacet, InGroupFacet,
                                          ETypeFacet, HasTextFacet)
    vreg.unregister(CWSourceFacet)
    vreg.unregister(CreatedByFacet)
    vreg.unregister(InStateFacet)
    vreg.unregister(InGroupFacet)
    vreg.unregister(ETypeFacet)
    vreg.unregister(HasTextFacet)

from cubicweb.entities import AnyEntity
AnyEntity.fetch_attrs = ()

Feel free to answer the question. Thanks in advance.

prioritynormal
typeenhancement
done in<not specified>
closed by<not specified>