Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log overview and background work #176

Closed
wants to merge 634 commits into from
Closed

Conversation

PatrikBuetler
Copy link
Contributor

@PatrikBuetler PatrikBuetler commented Jul 8, 2024

Not intended to be merged. It is in a working, but not very useful state.
Many of the fixed values such as 'thresholdError' will be modifiable for each system in the future.

Added functionality to search a log for keywords, count occurances. These features will be expanded upon

grafik

Feedback is welcome

@PatrikBuetler PatrikBuetler requested review from vogti and removed request for vogti July 8, 2024 19:36
@vogti
Copy link
Member

vogti commented Jul 9, 2024

Thank you for this draft. I reviewed the code and have some suggestions:

  • LogUtils --> lower case, should go to libraries, file name: logalyzer.php, Class Name: Logalyzer_Library
  • Please do not manually include a file include '../core/LogUtils.php'; this is what the autoloader is for (thus the precise naming rules).
  • Please do not forget to add PHPDoc to the classes and functions.
  • With your current approach, you are analyzing the log every time you open the detail or overview page. With large logs, this will kill performance... Since logs are append-only, it would be better to check when the logs are submitted and only recheck if the patterns have changed (e.g, hashing the pattern and storing it together with the status). Make sure to implement a proper logic for handling rechecks while a job is still running and producing log lines.
  • I like the "longer as usual warning", however, it should be based on a percentage in relation to the other jobs of the same evaluation, not a fixed number of lines.
  • I still think it needs both positiv and negativ patterns.
  • Why keyWordDict? Shouldn't it be two lists, one for warning patterns and one for error patterns?
  • I like the design of the warning and error icons.

$this->job = $job;
$this->system = new System($this->job->getSystemId());
$path = UPLOADED_DATA_PATH . '/log/' . $job->getId() . '.log';
$log = Util::readFileContents($path);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks wrong. Do not read the whole file in the constructor, move to readEntireLog.

Comment on lines 162 to 163
$logUtil = new Logalyzer_Library($subJob);
$logUtil->examineLogAndSetAlert();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an artifact, do not analyze all jobs again every time.

// Grab keyword arrays at creation of this object. Changes during a job run make the result outdated, but consistent
$this->warningKeys = json_decode($this->system->getLogalyzerWarningKeywords());
$this->errorKeys = json_decode($this->system->getLogalyzerErrorKeywords());
$this->calculateAndSetHash();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not update the Hash every time you initialize the Logalyzer Lib. Set hash when creating job (or even better when writing the first log line, thus when creating the log file) and after checking the entire file.

}
public function examineEntireLog() {
// Check if there have been changes to the log
if ($this->checkHashDifference()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove

$warningCount = 0;
$errorCount = 0;
foreach ($this->warningKeys as $key) {
if(str_starts_with($key, "/") || str_starts_with($key, "#") || str_starts_with($key, "~")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Store info whether it is a regex in the json.

// Log is reexamined using up-to-date keys, save new hash.
$this->job->setLogalyzerCountWarnings($warningCount);
$this->job->setLogalyzerCountErrors($errorCount);
$this->calculateAndSetHash();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calculate in the beginning.

public function examineLogLine($logLine) {
if($this->job->getLogalyzerCountWarnings <= 10) {
foreach ($this->warningKeys as $key) {
if (str_starts_with($key, "/") || str_starts_with($key, "#") || str_starts_with($key, "~")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above, store in json

}
}
public function examineLogLine($logLine) {
if($this->job->getLogalyzerCountWarnings <= 10) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Magic number, take from settings / config.

* @param string $key name of new keyword
* @return void
*/
public function addKey(string $identifier, string $key) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addPattern

}

/**
* @param string $identifier add $key as warning or error?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logLevel

* @param string $key name of keyword to be deleted
* @return void
*/
public function removeKey(string $identifier, string $key) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removePattern

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants