Display detailed ingestion status to see if everything is alright and when the collection is ready

[felixebert]

Use Case: Add a large collection of documents to aleph using aleph crawldir. As this probably takes multiple weeks, I would like:

• as admin of the system regularly check if everything is alright
• as journalist when I approx. can search through the documents

I would like to suggest the following stats during ingestion:

• Documents per seconds at the moment
• Documents per seconds since start of indexing
• Total number of errors occured, maybe additionally divided by stages (OCR, index, entity extraction)
• Approx. time left (probably complicated to get right, should maybe based on documents per sec per file type and file size)
• if possible: Stats per File Type: Error / Success
• if possible: a browsable list of errors to detect patterns

All these stats should be viewable even after the ingestion got finished, enriched with aleph version and time stats

• as an admin: to compare documents per second after an upgrade to the system
• as an admin: to know if you have to / should re-ingest a document after new features / bugfixes were added to aleph
• as a journalist: to see if you have to perform a new search if new documents were added since then

[simonwoerpel]

Additional data display:

• job/stage history for each document
• collection history / ingestion summary

[PlainSite]

This would be very helpful. Right now, the Aleph UI displays a progress bar under "Tasks," with separate columns for "Finished" and "Pending." A typical user would assume Tasks refers to the Tasks column alone, but it refers to all of these numbers—which is confusing, because there are three types of tasks (reflected nowhere in the UI, only in the status JSON requests coming in on loop) and so the number of completed Tasks is maybe 3X the number of processed files. But which files are being analyzed, and which stages are they at? The UI sort of makes this clear in Notifications, but that's also confusing because sometimes that pages says that files have been ingested "tomorrow." Can't really explain that.

The most frustrating part of this UI is that it never gives you an up-front view of how much work there is to do. The number of pending tasks spends a while (days for large jobs) increasing before it starts decreasing. Users almost assuredly want a total number of tasks to do, the number done, and the number pending.

It's possible this has all been fixed, which would be great, but upgrading Aleph using the commands in the documentation seems to update the ElasticSearch schema, but the version number in the UI footer always remains the same. So it's also not clear if upgrade is really upgrading what it's supposed to or not. There's no useful output (or any output) on the command line, either.