fix sorting of None values in NMPVCD.do_persist(). #1321
Conversation
| PersistenceUtil.store_json(self.persistence_file_name, sorted(list(self.known_values_set))) | ||
| try: | ||
| PersistenceUtil.store_json(self.persistence_file_name, sorted(list(self.known_values_set), | ||
| key=lambda L: tuple(el if el is not None else b'-' for el in L))) |
There was a problem hiding this comment.
I do not understand why the lambda expression is necessary here. When there is None in the list, sorting will fail.
>>> x = [b'a', None, b'1']
>>> sorted(list(x), key=lambda L: tuple(el if el is not None else b'-' for el in L))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
TypeError: 'NoneType' object is not iterable
Also, when replacing None with '-' this may be problematic if the value observed in the log lines can also be '-' (which is not unusual, e.g. in Apache Access logs). It is therefore important that None is persisted, and not some string replacement.
There was a problem hiding this comment.
Thank you for showing the need to comment this code.
The Lambda function is only used to allow sorting of tuples with None values. This would not be possible otherwise.
The data is later on stored normally as json (None will be null).
The data structure of known_values_set looks different. This test case is covered in the NewMatchPathValueComboDetectorTest.py lines 168-180.
Here is a simple test:
Python 3.8.10 (default, Jul 29 2024, 17:02:10)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
x = {(b"ddd ", b"25538"), (b" pid=", b"25537"), (b"ddd ", None)}
sorted(list(x), key=lambda L: tuple(el if el is not None else b'-' for el in L))
[(b' pid=', b'25537'), (b'ddd ', None), (b'ddd ', b'25538')]
I don't think there is any bug in the code.
Make sure these boxes are signed before submitting your Pull Request -- thank you.
Must haves
Fixes #1314
Submission specific
Describe changes: