Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak by design in the console and log #110

Open
RenWal opened this issue Jun 3, 2019 · 4 comments
Open

Memory leak by design in the console and log #110

RenWal opened this issue Jun 3, 2019 · 4 comments
Assignees
Labels
discussion needed Need to wait until discussion is resolved Priority: High Needs to be addressed as soon as possible Status: Available Ready for pickup Type: Bug This is a confirmed or unconfirmed bug

Comments

@RenWal
Copy link
Contributor

RenWal commented Jun 3, 2019

Describe the bug
Both the Python console and the log have infinite scrollback. This causes high memory consumption of a few tens to hundreds additional MB when you leave HAL running for a long time (I like to have the tool in a VM and just hibernate that VM, i.e. HAL is never closed) or you print an insanely large amount of data from your Python script.

To Reproduce

  1. Leave HAL open long enough (and actively work on a project)
  2. or print large amounts of data from Python
  3. or execute a gigantic Python script, which is copied as a whole into the log every time it runs
  4. run top and see the memory usage of the hal process

Expected behavior
HAL should only keep the last few hundred lines of console and log loaded and visible. Older data should be removed from the widgets to save memory.

If you really, really want an infinite scrollback buffer you could put it on a disk and load it on demand / map it into memory.

@swallat
Copy link
Member

swallat commented Jun 11, 2019

The logger is actually restricted to a maximum of 1000 blocks. @RenWal Can you give us such a large script you use?

@swallat
Copy link
Member

swallat commented Jun 11, 2019

I've just added the maximum block count for the python console as well. @RenWal Does it have an impact for you?

@swallat swallat added discussion needed Need to wait until discussion is resolved Priority: High Needs to be addressed as soon as possible Status: Available Ready for pickup Type: Bug This is a confirmed or unconfirmed bug and removed Substatus: Discussing labels Jun 13, 2019
@RenWal
Copy link
Contributor Author

RenWal commented Jun 14, 2019

I can still see some memory consumption but the rate has decreased a lot.

This is my test case: print('12345\nA'*1000)
When loading this into the editor and hitting run 100 times (do not use a loop), I can see an increase of the resident set size by about 68.6MiB in the unpatched and about 2.4MiB in the patched version.

Note that in this case, the console reaches the block limit but the log does not: Line breaks in print statements are stripped from the log, so only 100 lines (1 for each execution) exist. This would explain the 2.4MiB memory consumption.

This is probably also what made me think that both log and console were leaking, as I was printing large amounts of data with line breaks from a single print statement, thus the log's block limit was never hit.

@swallat
Copy link
Member

swallat commented Jun 17, 2019

@Slowpoke42 could you take a look on this?

@RenWal RenWal added the Status: Abandoned No progress for a long time. Is this still an issue? label Dec 14, 2019
@RenWal RenWal assigned joern274 and unassigned swallat and not-a-trojan Aug 26, 2022
@SJulianS SJulianS removed the Status: Abandoned No progress for a long time. Is this still an issue? label Aug 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion needed Need to wait until discussion is resolved Priority: High Needs to be addressed as soon as possible Status: Available Ready for pickup Type: Bug This is a confirmed or unconfirmed bug
Projects
None yet
Development

No branches or pull requests

5 participants