You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm wondering if anyone else has a similar use case: I want to monitor a sub-process call (e.g. with open_process) and log the output with structlog to go into a centralized monitoring system. I'm calling a program that generates a lot of output so I don't want to log each line at a time, but I also can't just dump the whole stdout output in a single log since it would be too big and get truncated. I saw the docs showing async for text in TextReceiveStream(process.stdout) and was testing it out, but I noticed that it will cut in the middle of a line, which can make following that broken-up output more difficult.
I think it would also work if it would emit each line individually and then build up our own buffer in the application code. This could be used to emit logs on a time basis (e.g. every 10 seconds log what was captured so far), number of log lines (e.g. every 1000 lines), or some other determination (e.g. buffer size). I used run_process before, but that would take the whole logs which would need to be broken up, and if it's going to be broken up I might as well try to log periodically.
Is this something that would make sense to have in AnyIO or should we stick to having external composition of these (e.g. mixing with BufferedByteReceiveStream to get up until \n)?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I'm wondering if anyone else has a similar use case: I want to monitor a sub-process call (e.g. with
open_process
) and log the output with structlog to go into a centralized monitoring system. I'm calling a program that generates a lot of output so I don't want to log each line at a time, but I also can't just dump the whole stdout output in a single log since it would be too big and get truncated. I saw the docs showingasync for text in TextReceiveStream(process.stdout)
and was testing it out, but I noticed that it will cut in the middle of a line, which can make following that broken-up output more difficult.I think it would also work if it would emit each line individually and then build up our own buffer in the application code. This could be used to emit logs on a time basis (e.g. every 10 seconds log what was captured so far), number of log lines (e.g. every 1000 lines), or some other determination (e.g. buffer size). I used
run_process
before, but that would take the whole logs which would need to be broken up, and if it's going to be broken up I might as well try to log periodically.Is this something that would make sense to have in AnyIO or should we stick to having external composition of these (e.g. mixing with
BufferedByteReceiveStream
to get up until\n
)?Beta Was this translation helpful? Give feedback.
All reactions