Improve complex forms performance

[lindsay-stevens]

Relevant to #739 and significantly improves performance and memory usage, but does not fully resolve the #739 in particular the recursion problem with relative references in repeats. However the changeset is already broad in scope and a large performance improvement so I would like to merge before other PRs come in.

Why is this the best possible solution? Were any other approaches considered?

The problem form mentioned in #739 now converts in about 8 seconds (excl. 6s for Validate) rather than about 5 minutes. In terms of memory usage, from a pre-conversion baseline of 35-40MB (resident/RSS), usage peaks at about 110MB rather than about 145MB.

Here is an outline of the pyxform conversion pipeline and where notable changes were made:

1. xls2xform.py: CLI or API request to read from a file or memory buffer
2. xls2json_backends.py: convert to rows of dict from rectangular data source (XLS/X, CSV, MD)
   1. OrderedDict was being used to support <py3.7 but now dict retains order and uses half the memory per object (64B vs 128B). So if the input XLSForm has 10k rows of questions/choices, that's 640KB difference.
3. xls2json.py: convert rows of dict to untyped dict tree structure
   1. Move validate_choices to the validators module so the relevant data is in scope for as short time as possible
   2. Move "or_other" choice / InputQuestion insertion from builder.py, to allow a row-specific error and earlier finalisation of the choices lists
4. builder.py: convert untyped dict tree structure into SurveyElement-based tree structure
   1. Avoid copying the input dict data, by removing modifications (e.g. pop) that would have made copying necessary
   2. For each Question with an itemset, use the Options (choices) instances from Survey instead of making new ones
   3. For SurveyElement, switch from using a dict to a slotted Mapping (dict interface but not a dict) to save memory and make it clearer what attributes are relevant to each class. For example an Option (which represents a choice) would get about 60 attributes (now ~8) - most of which empty strings but many are dicts or lists. Because a lot of code was not tolerant of None as a default value, all the fields were initialised, which wastes time and memory. This refactoring took a long time because many fields required software archaeology to find where is it used/tested/documented, and/or why/when was it added. Some more fields could probably be culled e.g. SMS/compact fields overlap and might be deprecated, etc.
   4. For Questions, merge the input kwargs with the question type dictionary (QTD - a bunch of bind/control/etc defaults for various question types) during __init__. Before, at the SurveyElement level, every attribute access (__getattr__) was being run through a check or merge against the QTD. For example Option()["label"] checked if "label" was in the QTD - lots of those 60 or so attributes are not in the QTD so a lot of wasted time. A worse situation existed for questions, e.g. for Question()["bind"] there would be a check against QTD, and since most question types have a "bind" default in the QTD, a copy would be created of the "bind" from QTD, add merged with the Question["bind"] value - not just once but every time "bind" was accessed, so a lot of wasted time and memory.
   5. On the QTD topic, the Question class still potentially is wasting some time/memory. The tests (mainly internal APIs) require that QTD default values are not output by SurveyElement.to_json_dict. For example if I create a start-geopoint, the "bind" info {"type": "geopoint"} is excluded. I think it's reasonable to include the QTD defaults since it's part of the Question definition, but to avoid additional confusion from more modified tests, for now the original input kwargs are stored separately from the QTD merged data, so that the QTD data can be excluded.
   6. Avoid iterating the SurveyElement tree structure where possible.
      1. Pass the Survey object into methods that need it, instead of iterating upwards from the current object to find it every time. For example Question.xml_instance gets called for every question, and self.get_root would go through 1 to N iterations of element.parent until it found the survey.
      2. Cache each element xpath in _survey_element_xpath after calculating it for the first time - there is no code that involves changing the tree after it is constructed, but just in case the cache is invalided by __setattr__ if "parent" changes.
      3. Add to iter_descendants a condition option to filter the returned elements during iteration. This doesn't reduce the traversal, but on the caller side fewer objects are returned for iteration/processing. For example in Survey.xml_descendent_bindings, Option and Tag don't have bindings so we can skip xml_bindings for them.
   7. For forms with translations, the language codes are validated against IANA subtags. Previous solution loaded all 8138 options (525KB) into a list by default then ran a membership check (O(n)) to search for each language. New solution loads 2-character codes first (190 items / 8KB) as a set (O(1)) then if that fails load and check the full list as a set. Best case: default, invalid, or short language codes get faster lookup with less memory used than before. Worst case: longer language codes get faster lookup with the same memory used as before.
5. utils.py (node): convert SurveyElement-based tree structure into xml.dom.minidom.Element-based tree structure
   1. Check the type of only as many childNodes as needed instead of all of them
   2. Add combine_lists for coalescing/chaining iterables instead of creating new lists or concatenation
6. survey.py: minidom.Element-based tree structure into XML string
7. survey.py: write XML string to a temp file for ODK Validate to execute javarosa read check
8. xls2xform.py: return converted data or write XML to file

General improvements applied in numerous places:

1. Avoid using an empty dict as a default value for get and instead handle the None case.
2. If a reference container (e.g. in constants.py) is used for a membership check (x in y), convert it to a set (O(1)) instead of a list (O(N)). Compared to a list, a set uses more memory (216B vs 56B) but generally these reference objects have one copy only so the speed tradeoff seems worthwhile - particularly SUPPORTED_MEDIA which is in a hot loop (referenced for each media dict key for any Question or Option).
3. Use f-strings to insert variables instead of join or + to do the same thing
4. yield / from for iteration instead of appending to a list and returning the list to do the same
5. Use tuple instead of list where an ordered sequence is needed 1) save 16 bytes per object (40 vs 56), 2) avoid rogue modifications to sequences that probably shouldn't change mid-pipeline.

Further changes not included in this PR that could further help performance:

1. The input data is copied at pipeline steps 2 (read), 3 (to untyped dict), 4 (to SurveyElement), 5 (to minidom Element), 6 (to string). Similar to my experience with the JVM, a Python process will not reliably release memory (even if after forced garbage collection) so to reduce process memory usage requires a focus on reducing peak usage. Some ideas:
   1. Adapt xls2json_backends and workbook_to_json to emit / accept generators instead of processing and passing the full XLSForm data. Potentially each row could be a generator as well.
   2. The data copying in pipeline step 5 (minidom Element) seems to consume the most memory. The SurveyElement could be adapted to output XML (i.e. avoid copying data to Elements at all), or the Element class could be changed to sax/event-driven incremental string output (Elements created/kept only as long as it takes to produce their XML). The XML probably can't be streamed back to the caller though because the entire document is needed by default for the ODK Validate check.
      1. Update ODK Validate to accept XForm input from stdin instead of reading only from files, to avoid the need for a tempfile during processing.
   3. Adapt workbook_to_json to produce the SurveyElement tree directly, instead of passing the data to the builder module. On that note the SurveyElement classes would need to (and probably should anyway) be able to deal with instantiating themselves without the builder module.
2. As mentioned above I haven't really touched on the "relative references in repeats" recursion problem. For the relevant functions I did reduce the lru_cache arg maxsize values since 128 is the default and probably a reasonable point of diminishing return of performance vs. memory usage (seemed to be the case for parse_expression). About a year ago I changed these maxsize args from None (unbounded cache) to 65536 (a guess but at least not unbounded).
3. Review and optimise strings processing/cleaning. It seems like strings are cleaned or examined multiple times during processing e.g. cleaning in xls2json_backends and workbook_to_json; checking for regex/token matches in workbook_to_json and SurveyElement processing. If it's too hard/awkward to consolidate completely then maybe each string could have metadata noted somehow e.g. "this string was cleaned already", and "this string does/not contain references".
4. Some SurveyElement functions relate only to certain subclasses and could be moved to those classes, or a mixin / intermediate subclass. e.g. a fair bit of awkward "hasattr" or "isinstance" remains, not all classes have children or bind etc.

What are the regression risks?

Minimal tests fixes were required for these changes, so any issues should be more for projects using pyxform internal classes or APIs. It also means that if something is broken then we didn't have a test for it and so it should have a test. Potential compatibility issues:

• removed some SurveyElement fields that seem to not be used at all, and SurveyElement classes will not have all 60+ fields anymore so any calling code will need to be tolerant of that by a) checking hasattr or isinstance and b) checking for None.
• any extra data passed in as kwargs that is not a declared attribute is put into a new extra_data dict, and is excluded from to_json_dict output. The extra_data gets used when specifying extra choices columns, to generate choices output for that data.

Does this change require updates to documentation? If so, please file an issue here and include the link below.

No, but as mentioned above it would be good to check what the support situation is for SMS/compact and whether that is still needed, and check whether all the Survey attributes correspond to documented and supported features (either document them if they are supported, or remove them if they aren't).

Before submitting this PR, please make sure you have:

• included test cases for core behavior and edge cases in tests
• run python -m unittest and verified all tests pass
• run ruff format pyxform tests and ruff check pyxform tests to lint code
• verified that any code or assets from external sources are properly credited in comments