Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tests for item validation pipeline #414

Merged
merged 2 commits into from
Aug 30, 2023

Conversation

VMRuiz
Copy link
Collaborator

@VMRuiz VMRuiz commented Aug 30, 2023

The current spidermon behavior for adding _validation field to items seems to be a bit controversial: #379

To make easier to track changes, I have implemented some additional tests that covers the current logic.

@codecov
Copy link

codecov bot commented Aug 30, 2023

Codecov Report

Patch coverage has no change and project coverage change: +0.12% 🎉

Comparison is base (a75cf97) 78.52% compared to head (1347209) 78.64%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #414      +/-   ##
==========================================
+ Coverage   78.52%   78.64%   +0.12%     
==========================================
  Files          73       73              
  Lines        3152     3152              
  Branches      528      528              
==========================================
+ Hits         2475     2479       +4     
+ Misses        600      598       -2     
+ Partials       77       75       -2     

see 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@VMRuiz
Copy link
Collaborator Author

VMRuiz commented Aug 30, 2023

The error raised in these lines https://github.com/scrapinghub/spidermon/pull/414/files#diff-877bfbf0203e928b3c50d4dadb782369861cf83240c9d0721ed1f7dd791a6406R140-R145 may be a bug.

Spidermon should allow setting the default value in classes that supports using additional fields like dicts but that doesn't support __setitem__ method. I can fix it by using setattr(item, self.errors_field, defaultdict(list) instead of item[self.errors_field] = defaultdict(list). However I prefer to merge this first and fix it in a new PR.

Also, it should check that data[self.errors_field] is not None to cover the cases where the field was defined but was given a None value as default.

@VMRuiz VMRuiz merged commit 37c8a3a into scrapinghub:master Aug 30, 2023
7 checks passed
VMRuiz added a commit that referenced this pull request Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants