loading the admin page/submitting job when many entities are indexed is too slow

[bradfordcondon]

the problem is the summary table.

On hte one hand i think its very useful information. You can see what your previous indexing looks like or check on the status or get a sense if theres a problem (no bundles indexed for a CV).

On the other, its just too slow to do a county query on the index table like this.

Also the labels dont make sense. I think that the "big" table is total number of entities that are in the index, wheras the mini table is the total number of terms for that content type (ie more than aone term per entity).

[almasaeed2010]

Loading the admin page has become even more of a problem now that we have this many organisms. Trying to the live site takes a few minutes before the page responds.

I think a good solution is to separate the insights page from the indexing page.

[almasaeed2010]

Another good solution is to cache the results after every indexing job.

[almasaeed2010]

As for submitting a job taking this long, I'd just make a dispatcher job that has the form data.

EDIT

this issue might resolve itself once we fix the stats table.

[bradfordcondon]

Are the insights helpful as is? probably not.

really it should tell you which organisms you've indexed and which you've havent...

If we can think of a way to make that work, the ncacheing the results sounds good to me.

Otherwise moving elsewhere or removing altogether seems like the right plan.

[almasaeed2010]

Doing it on a per organism is super slow. My attempt at it at least didn't work out very well. See the automated annotation module for the code.