The confusion about the role of metric parameter in fit

[HuangChiEn]

I have confused about the role of the metric parameter in fit.
In FLAML document :

Optimization metric
The optimization metric is specified via metric argument. It can be either a string which refers to a built-in metric, or a user-defined function.

The metric is not the optimized target and it should be used to objectively measure the performance of the model, which has the different meaning between loss.

So, my question is:

1. Is the metric only used to measure the performance (it's not loss or objective function), and the model will not change the inner-parameters according to the metric ?
2. If the reply of 1. is Yes, then why do we have log_"Loss" as built-in "metric"?

[qingyun-wu]

Hi @HuangChiEn, the answer to question 1 is Yes. For question 2, it seems that you believelog_loss should not be used as one possible metric to measure the performance of the model? Since log_loss it is just one of the many ways to measure the model's predictive performance (on the validation dataset), I think it is a legitimate metric in our case.

Is it because you think the metric for measuring the performance should be a metric that needs to be maximized? For the built-in metrics, we automatically detect whether they are metrics that should be maximized or minimized? The documentation about the mode of the metric (maximization or minimization) indeed should be improved.

Let me know if you have other concerns. Thank you!

[HuangChiEn]

Thanks for your detailed reply. It's no problem if the framework will automatically decide to maximize or minimize the built-in metrics.

I also have an related question, how to indicate the maximize/minimize of customized metrics?
In my case, I need to do some modification of f1 score, so I return "1" - mod_f1(...) to let it "maximize" the custom metric. Is there exists another native way?

[qingyun-wu]

@HuangChiEn, Thank you for your question. Currently, customized metrics only support minimization. Your way of modifying the metric you hope to maximize to a one that should be minimized is proper. There is no other native way for maximization. By the way, can you please take one minute to give us feedback via this survey: https://docs.google.com/forms/d/e/1FAIpQLSd0u4BbSzr-m9hgM8rmPlYB_JJoFXeHhuoXI9YFbHDRWrDlOQ/viewform?usp=sf_link . We appreciate your input! Thank you!

[HuangChiEn]

I have done the survey & close this issue ~