Decode hangs when decoding the output of encode if ASN.1 contains a SEQUENCE of CHOICE of SEQUENCE

[k-channon-PA]

To reproduce:

1. compile the ASN.1 schema from the test_hangs method below using ans1tools.compile_string and the DER coded
2. Encode some data using the compiled schema
3. Decode the bytes you just encoded with ans1tools

Expected outcome:

The bytes are decoded and the resulting data is equal to the data you originally encoded

Actual Outcome:

Decode hangs with this call stack:

  File "/home/user/ans1tools_hang_tests.py", line 63, in test_hangs
    hangs_here = compiled_schema.decode("Element1", encoded)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/compiler.py", line 167, in decode
    decoded = type_.decode(data)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 1550, in decode
    return self.decode_with_length(data)[0]
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 1559, in decode_with_length
    decoded, offset = self._type.decode(bytearray(data), 0)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 551, in decode
    return self.decode_content(data, offset, length)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 755, in decode_content
    offset, out_of_data = self.decode_members(self.root_members, data, values, offset, end_offset)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 802, in decode_members
    value, offset = member.decode(data, offset, values=values)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 551, in decode
    return self.decode_content(data, offset, length)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/der.py", line 92, in decode_content
    decoded_element, offset = self.element_type.decode(data, offset)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 1227, in decode
    decoded, offset = member.decode(data, offset)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 551, in decode
    return self.decode_content(data, offset, length)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/der.py", line 92, in decode_content
    decoded_element, offset = self.element_type.decode(data, offset)
  File "/home/user/.local/lib/python3.6/site-packages/asn1tools/codecs/ber.py", line 540, in decode
    if len(tag_data) != self.tag_len:
KeyboardInterrupt

The keyboard interrupt is me killing the process. At the point of the hang, it's actually stuck in an infinite loop and it just pushing the same thing back into a list of things over and over again. I guess it will eventually crash when the process runs out of memory.

The offending bit of code is this, from ber.py:

which returns the offset without incrementing it to this function in der.py, which pushes it back into a list and then goes around again at the returned offset:

For some reason, the value of self.tag in ber.py is 0x06, which is not what it finds on as the tag at the point of failure (which is on the 0x30 of the inner SEQUENCE that is Element6. For reference, the DER that's produced has a structure that looks like this:

0x00000000 00000000  30 (16):
0x00000002 00000002    a0 (14):
0x00000004 00000004      30 (12):
0x00000006 00000006        06 (02) -->    .2.  OID [2a 03]
0x0000000a 00000010        31 (0c):
0x0000000c 00000012          30 (0a):
0x0000000e 00000014            30 (08):   <<<<=== HERE's WHERE WE GET STUCK, with a self.tag of 0x06, but finding this 0x30 tag
0x00000010 00000016              06 (03) -->    .3.  OID [81 25 06]
0x00000015 00000021              02 (01) -->    .1.  INT [0a]

Test code to reproduce

We're seeing this issue in a much larger schema, but I've tried to boil it down to the minimum pieces that I could. The only difference between the hanging and non-hanging case is the presence of Element3 in the failing case, which is a CHOICE type in a SEQUENCE, and the CHOICE type itself contains a SEQUENCE.

import logging
import unittest
import asn1tools


class ASN1ToolsHangTests(unittest.TestCase):
    logging.basicConfig(level=logging.DEBUG)

    def test_hangs(self):
        schema = """TopLevel DEFINITIONS ::=
            BEGIN
                Element1 ::= SEQUENCE {
                    e1values [0] IMPLICIT Element2
                }
                
                Element2 ::= SET SIZE (1..MAX) OF Element3
                
                Element3 ::= CHOICE {
                    e3choice1 Element4,
                    e3choice2 Element7
                }
            
                Element4 ::= SEQUENCE {
                    e4id OBJECT IDENTIFIER,
                    e4values SET OF Element5
                }
                
                Element5 ::= SEQUENCE {
                    e5value Element6
                }
                
                Element6 ::= SEQUENCE {
                    e6id OBJECT IDENTIFIER,
                    e6value INTEGER (0..MAX)
                }
                
                Element7 ::= SEQUENCE SIZE (1..MAX) OF OBJECT IDENTIFIER
                
            END
        """

        data_to_encode = {
            "e1values": [
                (
                    "e3choice1",
                    {
                        "e4id": "1.2.3",
                        "e4values": [
                            {
                                "e5value": {
                                    "e6id": "4.5.6",
                                    "e6value": 10
                                }
                            }
                        ]
                    }
                )
            ]
        }

        compiled_schema = asn1tools.compile_string(schema, codec='der')
        encoded = compiled_schema.encode(name="Element1", data=data_to_encode)
        hangs_here = compiled_schema.decode("Element1", encoded)

        self.assertEqual(data_to_encode, hangs_here)

    def test_does_not_hang(self):
        schema = """TopLevel DEFINITIONS ::=
            BEGIN
                Element1 ::= SEQUENCE {
                    e1values [0] IMPLICIT Element2
                }
                
                Element2 ::= SET SIZE (1..MAX) OF Element4
            
                Element4 ::= SEQUENCE {
                    e4id OBJECT IDENTIFIER,
                    e4values SET OF Element5
                }
                
                Element5 ::= SEQUENCE {
                    e5value Element6
                }
                
                Element6 ::= SEQUENCE {
                    e6id OBJECT IDENTIFIER,
                    e6value INTEGER (0..MAX)
                }
                
            END
        """

        data_to_encode = {
            "e1values": [
                {
                    "e4id": "1.2.3",
                    "e4values": [
                        {
                            "e5value": {
                                "e6id": "4.5.6",
                                "e6value": 10
                            }
                        }
                    ]
                }
            ]
        }

        compiled_schema = asn1tools.compile_string(schema, codec='der')
        encoded = compiled_schema.encode(name="Element1", data=data_to_encode)
        decoded = compiled_schema.decode("Element1", encoded)

        self.assertEqual(data_to_encode, decoded)


if __name__ == '__main__':
    unittest.main()