Metrics fixes, tests, refactoring, typing improvement #1214

MorrisNein · 2023-11-29T16:45:27Z

Changes:

Fix F1 & precision evaluation
Remove flattening of target & predict values during metrics evaluation
Unwrap the shape of prediction in pipeline.predict() from (1, N) to (N) for forecasting tasks
Add duck typing for MetricCallable
Multiple inheritance fixes for the inheritors of Metric
Add unit tests for all supported metrics, tasks and data types

pep8speaks · 2023-11-29T16:45:36Z

Hello @MorrisNein! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file fedot/api/builder.py:

Line 277:121: E501 line too long (123 > 120 characters)

In the file test/unit/tasks/test_regression.py:

Line 104:52: F811 redefinition of unused 'data_setup' from line 14

Comment last updated at 2023-11-29 18:17:04 UTC

codecov · 2023-11-29T17:48:26Z

Codecov Report

Attention: 30 lines in your changes are missing coverage. Please review.

Comparison is base (9d9469f) 79.27% compared to head (e1d6f84) 79.42%.
Report is 1 commits behind head on master.

Files	Patch %	Lines
fedot/core/operations/atomized_model.py	0.00%	29 Missing ⚠️
fedot/core/composer/metrics.py	98.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1214      +/-   ##
==========================================
+ Coverage   79.27%   79.42%   +0.14%     
==========================================
  Files         145      145              
  Lines       10048    10045       -3     
==========================================
+ Hits         7966     7978      +12     
+ Misses       2082     2067      -15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fedot/core/repository/quality_metrics_repository.py

fedot/api/builder.py

fedot/core/composer/metrics.py

fedot/core/repository/metrics_repository.py

fedot/core/composer/metrics.py

test/unit/composer/test_metrics.py

kasyanovse · 2023-11-30T10:09:46Z

fedot/core/composer/metrics.py

- # Time series forecasting
- is_ts_forecasting = reference_data.task.task_type == TaskTypesEnum.ts_forecasting
- if is_ts_forecasting or is_multi_target_regression:
- results, reference_data = cls.flatten_convert(results, reference_data)


Почему убрана эта строка?

Потому что она преобразует мульти-таргет предсказания из матрицы в колонку. Вроде как для вычисления метрик нет такой необходимости.

Насчёт преобразования временного ряда из нескольких колонок в одну, мне кажется, тоже очень спорная логика. Но могу ошибаться

Потестировал. Все метрики толерантны к форме данных на входе.

remove default value for metric_by_id

MorrisNein · 2023-12-05T16:03:22Z

@kasyanovse, вынес проблему с бинарной классификацией в отдельную issue.

А пока что вынес этот случай в менее строгий тест.

kasyanovse · 2023-12-06T08:18:12Z

test/unit/composer/test_metrics.py

Я добавил сид для теста метрик

Убрал простой тест для бинарной классификации

Убрал автоматическую перезапись результатов расчета метрик

Перенес перезапись результатов расчета метрик после assert

У меня небольшая проблема возникла, что в результате теста переписываются все метрики для временных рядов. Пока посмотрю из-за чего, потом аппрувну, если все будет ок.

MorrisNein force-pushed the fix-metrics branch from fcd46c7 to 726f3ab Compare November 29, 2023 16:47

MorrisNein changed the title ~~Metrics fixes, tests, refactoring~~ Metrics fixes, tests, typing refactoring Nov 29, 2023

MorrisNein changed the title ~~Metrics fixes, tests, typing refactoring~~ Metrics fixes, tests, refactoring, typing improvement Nov 29, 2023

kasyanovse self-requested a review November 29, 2023 18:00

MorrisNein commented Nov 29, 2023

View reviewed changes

fedot/core/repository/quality_metrics_repository.py Outdated Show resolved Hide resolved

MorrisNein commented Nov 29, 2023

View reviewed changes

fedot/api/builder.py Outdated Show resolved Hide resolved

kasyanovse requested changes Nov 30, 2023

View reviewed changes

MorrisNein force-pushed the fix-metrics branch 3 times, most recently from b6c3c5f to beb6bbe Compare December 4, 2023 11:36

MorrisNein added 18 commits December 4, 2023 14:38

fix f1 & precision evaluation

68f06de

rem flattening from QualityMetric._simple_prediction

d598469

fix metrics hierarchy

3266ad0

improve typing of QualityMetricCallable

e3569c8

add tests for all supported types of metrics

f9ac3a2

return **kwargs to complexity metrics

332eedb

use MetricCallable where it is appropriate

e232425

tests: simplify regression pipeline, type hints & additional assertion

a5a9718

remove __self__ from MetricCallable definition;

e5a957e

remove default value for metric_by_id

rename & redo repository methods

4000e65

rename MetricType -> MetricIDType

ff78901

reorder module

e910544

rename metrics repo module

61de759

rename metrics test module

eddd5cc

update test data pipeline pickle

bd0ea7e

correct typos

dafbc1b

adapt documentation

793dc23

fix inheritance

e18f729

MorrisNein added 6 commits December 4, 2023 14:42

add kwargs and metric() to complexity metrics

b618483

better types for metrics_repository.py

c2f8696

combine all metrics to a single test; add default values check

e4aa6d0

unwrap pipeline predictions for time series

3d133dc

fix confusing naming

a329eb7

add fixture for expected values for faster tests

0aa8acf

MorrisNein force-pushed the fix-metrics branch 5 times, most recently from 92d32b7 to 17073c1 Compare December 4, 2023 12:41

MorrisNein added 2 commits December 5, 2023 18:48

remove test for binary classification

47e1955

add a less strict test for binary classification

c34297d

MorrisNein force-pushed the fix-metrics branch from 17073c1 to c34297d Compare December 5, 2023 16:01

MorrisNein requested a review from kasyanovse December 5, 2023 16:02

kasyanovse added 2 commits December 6, 2023 10:46

fix random seed for binary classification

168b71b

delete simple test for binary classification

8338597

kasyanovse reviewed Dec 6, 2023

View reviewed changes

delete binary test from test_metrics

43b590c

kasyanovse approved these changes Dec 6, 2023

View reviewed changes

MorrisNein added 4 commits December 6, 2023 14:35

fix structural_analysis_example.py

f4cb6ed

make test_metrics fall if update_expected_values=True

62fd7d5

fix unused imports

c88a41f

add str to MetricIDType

e1d6f84

MorrisNein merged commit 4ef1c8e into master Dec 6, 2023
5 of 6 checks passed

MorrisNein deleted the fix-metrics branch December 6, 2023 11:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metrics fixes, tests, refactoring, typing improvement #1214

Metrics fixes, tests, refactoring, typing improvement #1214

MorrisNein commented Nov 29, 2023 •

edited

Loading

pep8speaks commented Nov 29, 2023 •

edited

Loading

codecov bot commented Nov 29, 2023 •

edited

Loading

kasyanovse Nov 30, 2023

MorrisNein Nov 30, 2023

kasyanovse Dec 6, 2023

MorrisNein commented Dec 5, 2023

kasyanovse Dec 6, 2023 •

edited

Loading

Metrics fixes, tests, refactoring, typing improvement #1214

Metrics fixes, tests, refactoring, typing improvement #1214

Conversation

MorrisNein commented Nov 29, 2023 • edited Loading

pep8speaks commented Nov 29, 2023 • edited Loading

Comment last updated at 2023-11-29 18:17:04 UTC

codecov bot commented Nov 29, 2023 • edited Loading

Codecov Report

kasyanovse Nov 30, 2023

Choose a reason for hiding this comment

MorrisNein Nov 30, 2023

Choose a reason for hiding this comment

kasyanovse Dec 6, 2023

Choose a reason for hiding this comment

MorrisNein commented Dec 5, 2023

kasyanovse Dec 6, 2023 • edited Loading

Choose a reason for hiding this comment

MorrisNein commented Nov 29, 2023 •

edited

Loading

pep8speaks commented Nov 29, 2023 •

edited

Loading

codecov bot commented Nov 29, 2023 •

edited

Loading

kasyanovse Dec 6, 2023 •

edited

Loading