About log storage

[erdengk]

I think this issue needs a detailed discussion, so create a new discussion here.

Dynamically load log lines from server

As mentioned in the question, in test scenarios or long-running deployments with higher log levels, the logs can get very large.

When the log file becomes very large, it can be very resource-intensive to read the file directly. If it is designed in this way, when we want to query the logs of the specified date, it is a very troublesome thing.

So I think it is a reasonable and convenient practice to store the logs into the database.
The reasons are as follows:

1. When viewing the log, we can view it by time. For example, view the log of a certain day, or view the log of a certain week.
2. log4j provides the JDBCAppender class, which provides the function of asynchronously writing log information to data. We can use this class to write our log information to the database.

But this design will also have some new problems.

1. When the log is generated and queried, a database connection needs to be established, which may increase the burden on the database.
2. When the user views the log, we should provide the date of viewing the log, these need to add a new interface to query the data. Similarly, the UI also needs to add corresponding components, such as the calendar component.

The solution of one problem may often lead to new problems, and we need to balance the new problems with the old ones.

The above analysis is just my personal analysis, I don't know much about the overall architecture of DB. Perhaps reading the log file directly is a better way? Which plan do you think is better?
@vogti

[hennlo]

Hi @erdengk , thanks for your thoughts and analysis.

Generally I like your approach to optimize the accessibility of the log. That is indeed an open issue and an important addition for the overall operability of Polypheny.

However, I have a few remarks on your idea.
All in all you are correct, the logs can get quite large and can therefore get quite resource consuming to parse them by hand.
But honestly that is exactly one of the problems, why I would avoid loading the log into the db.

• For one it originates out of Polypheny-DB so there would be easier ways to keep track of the logs internally without writing them to disk first and reinserting them again.
• Secondly it is then considered to be a database object and will be restored, once the DB is restored.
  But sometimes you do not want really want this.
  -- Furthermore file operations should always be strictly decoupled from the database itself. Hence also backuped up separately.
  Systems
• Additionally Polypheny is currently not build for working with log metrics. There are other systems that do this inherently like the ELK stack.

The question is always why do we even log and when to we want to use it? And in general it will always come down to investigating a faulty behavior or even a system crash.
For both scenarios it's crucial to keep the log accessible by the system or any 3rd party application, if you cannot access your db again.
Otherwise you will have no chance identifying and eliminating the root cause of any problem.

If in the future the log should be accessible using the db. I would suggest this to be a feature of the UI and a dedicated log & trace file viewer. A common practice is to read the log file chunk wise. That being 1000 lines at a time (or any other arbitrary) number.

To sum things up, I think there are always solution like logrotate.d or similar tools to keep the log "parseable" and neat.
I would therefore suggest to look at this from a UI perspective, once Polypheny-DB writes its log directly without using Control.

[erdengk]

Hi, @hennlo, Thanks for your analysis.

I missed the original role of the log in my analysis.

When the database fails to start and needs to view the log to troubleshoot the problem, it is not a reasonable choice to store the log in the database.

for you mentioned

to look at this from a UI perspective, once Polypheny-DB writes its log directly without using Control

(In the text below, DB stands for Polypheny-DB and UI stands for Polypheny-UI)

I think the DB is an instance, and the UI can view these instances, and when the DB fails and cannot be used, the UI can view the log to solve the problem.
This is how I understand the relationship between DB and UI, I don't know if it is correct, please guide me.

If my understanding is ok. Can this problem be solved in the following way?

• The DB logs are archived and stored on a weekly or daily basis, so that each file will not be too large, and the resources required to read the file will not be too much.
• The UI can access the file by date when it needs to be viewed.
• When viewing a file, if the number of lines in the log file is less than 1000 (or other numbers), the entire log file is directly returned. If the log file is large, log blocks can be returned in sequence.
• If there is a need to optimize the log file size, log4j's RollingFileAppender provides a similar function.

What do you think of this solution? Please help me improve my solution.

[hennlo]

Hi @erdengk

So regarding the architecture. The UI and the DB can be installed separately, meaning they can reside on different systems.
However one UI can only connect to one DB. It cannot view several DBs or instances at once.

Therefore, I would really avoid loading the log into the DB.
For the issue and your initial request lets focus solely on Polypheny-Control. Although we'll consider the DB logging at a later point in time.

So for Control to keep things simple and neat, I would propose based on your suggestions to dynamically load the last 1000 lines or so from the one logfile that is generated form the currently running instance.
That being said I would propose to generate a new file per day

If a user then chooses to access by date you can still do this by looking for the required files in the given directory.
Also I think it would be cool to give the user the opportunity to choose later on if he now wants to view lets say 2000 or 4000 lines of log.