Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to create one line logs for each ID based on the result? #7395

Open
junam2 opened this issue Jul 1, 2024 · 0 comments
Open

How to create one line logs for each ID based on the result? #7395

junam2 opened this issue Jul 1, 2024 · 0 comments

Comments

@junam2
Copy link

junam2 commented Jul 1, 2024

Is your feature request related to a problem? Please describe.
The results of inferences are not recorded in the log at once, so it is difficult to compare or search for the history of inference times for each model. Currently, the trace output varies widely by category (e.g., HTTP_RECV_START, INFER_RESPONSE_COMPLETE, HTTP_SEND_END), and additional work (like json parsing) is required to calculate the total time.

The --log-verbose option generates too many logs for one request.

Describe the solution you'd like
I would like to obtain the results of input token / output token / inference time / output_text for each ID in one line. (Like vllm, tgi)

Additional context

* tgi logg example

INFO generate{parameters=GenerateParameters { best_of: None, temperature: None, repetition_penalty: None, frequency_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: false, max_new_tokens: Some(512), return_full_text: None, stop: ["//", "[Response]"], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None } total_time="357.867572ms" validation_time="1.96844ms" queue_time="49.382µs" inference_time="355.849981ms" time_per_token="14.827082ms" seed="None"}: text_generation_router::server: router/src/server.rs:309: Success
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant