Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the AutoscaledPool log understandable #705

Open
Pijukatel opened this issue Nov 18, 2024 · 0 comments
Open

Make the AutoscaledPool log understandable #705

Pijukatel opened this issue Nov 18, 2024 · 0 comments
Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@Pijukatel
Copy link
Contributor

Pijukatel commented Nov 18, 2024

AutoscalePool periodically logs system load information in this function:
AutoscaledPool._log_system_status

This looks for example like this:

2024-11-06T15:11:50.471Z [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 1; desired_concurrency = 1; cpu = 0.581; mem = 0.0; event_loop = 0.227; client_info = 0.0

It shows values that are internally used by the desired_concurrency controller, but those value are hard to interpret by humans and thus not very useful to show in log. Make this log understandable.

On the other hand, the logged values should also be connected to values used by mentioned controller. If it gets readable, but detached from controller, then the log is again not very usable. So there is a risk that making this more readable would require changing the controller itself.

See full discussion in: #662

@Pijukatel Pijukatel added enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team. labels Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

No branches or pull requests

1 participant