Storing the list of artifacts in the database

[mitya57]

Speaking about performance issues, there is one more thing. It's orthogonal to the previous discussion in #194, so I am creating a new issue for it.

In our project we have 22 GB of artifacts and we would like to move them to an external storage after each job run. The first reason to do that is that if the project keeps growing, eventually we will run out of space on the machine that runs laminar. The second reason is that working with a million of small files with standard tools is quite slow, because readdir(3) has a 32 kB buffer.

It would be nice if laminar stored a list of artifacts for each run in the database. Then it would display a list of artifact links as it usually does, and we would handle the rest using LAMINAR_ARCHIVE_URL setting. Even for those who don't use external storage, it will be a performance improvement, as reading from a database is faster than doing readdir.

Basically, what I'm proposing is moving the readdir call from “user opens the run page” stage to “job run finished” and caching the result in the database. If you agree, I can make a pull request for that. I think it needs to be a separate table with (name, number, filename) columns.

[mitya57]

Oops, @KAction was 5 minutes faster than me. Closing as a duplicate of #195.