Error Decoding Indefinite Length ASN.1 Elements Containing Extension Marker

[aziz-marashly]

I have been using this decoder for around three years. I encountered an error when decoding some files and assumed they were corrupted. However, after a long debugging session, I was able to abstract and detect the leading cause of the failure.

Description:
I encountered a decoding error while using an ASN.1 decoder for elements defined with indefinite length that included an extension marker (...). This error appears specifically when decoding elements labeled with application-specific tags.

Steps to Reproduce:

This is an example asn1 definition and test code to reproduce

import base64
import asn1tools

if __name__ == "__main__":

    SPECIFICATION = """
    HelloWorld DEFINITIONS IMPLICIT TAGS ::= BEGIN
        Message ::= [APPLICATION 1]  SEQUENCE
        {
            id          [APPLICATION 2] UTF8String OPTIONAL,
            header      [APPLICATION 3] UTF8String OPTIONAL,
            content     [APPLICATION 4] UTF8String OPTIONAL,
            ...,
            other       [APPLICATION 5] UTF8String OPTIONAL
        }

    END
    """
    compiler = asn1tools.compile_string(SPECIFICATION)

    base64_message_def = "YRVCAmlkQwZoZWFkZXJEB2NvbnRlbnQ="  # defined lenght
    decoded_from_base64 = compiler.decode(
        "Message", base64.b64decode(base64_message_def.encode("ascii"))
    )
    print("defined length", decoded_from_base64)

    base64_message_ind = "YYBCAmlkQwZoZWFkZXJEB2NvbnRlbnQAAA=="  # indefinite  lenght
    decoded_from_base64 = compiler.decode(
        "Message", base64.b64decode(base64_message_ind.encode("ascii"))
    ) # exception will be raised here 
    print("indefinite length", decoded_from_base64)

Output:

Exception has occurred: OutOfByteDataError
Message: Ran out of data when trying to find End of Contents tag for indefinite length field (At offset: 25)
  File "path\to\lib\asn1tools\asn1tools\codecs\ber.py", line 112, in detect_end_of_contents_tag
    raise OutOfByteDataError(
  File "path\to\lib\asn1tools\asn1tools\codecs\ber.py", line 94, in is_end_of_data
    elif detect_end_of_contents_tag(data, offset):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\lib\asn1tools\asn1tools\codecs\ber.py", line 791, in decode_members
    out_of_data, offset = is_end_of_data(data, offset, end_offset)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\lib\asn1tools\asn1tools\codecs\ber.py", line 759, in decode_content
    offset, out_of_data = self.decode_members(flatten(self.additions), data, values, offset, end_offset,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\lib\asn1tools\asn1tools\codecs\ber.py", line 551, in decode
    return self.decode_content(data, offset, length)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\lib\asn1tools\asn1tools\codecs\ber.py", line 1565, in decode_with_length
    raise e
  File "path\to\lib\asn1tools\asn1tools\codecs\ber.py", line 1565, in decode_with_length
    raise e
  File "path\to\lib\asn1tools\asn1tools\codecs\ber.py", line 1550, in decode
    return self.decode_with_length(data)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\lib\asn1tools\asn1tools\compiler.py", line 167, in decode
    decoded = type_.decode(data)
              ^^^^^^^^^^^^^^^^^^
  File "path\to\lib\asn1tools\fast_example.py", line 28, in <module>
    decoded_from_base64 = compiler.decode(
                          ^^^^^^^^^^^^^^^^
asn1tools.codecs.ber.OutOfByteDataError: Message: Ran out of data when trying to find End of Contents tag for indefinite length field (At offset: 25)

If we change the ANS1 specs and move the ... indicator to the end of the element, then everything will be fine.
This example will work without any exceptions

import base64
import asn1tools

if __name__ == "__main__":

    SPECIFICATION = """
    HelloWorld DEFINITIONS IMPLICIT TAGS ::= BEGIN
        Message ::= [APPLICATION 1]  SEQUENCE
        {
            id          [APPLICATION 2] UTF8String OPTIONAL,
            header      [APPLICATION 3] UTF8String OPTIONAL,
            content     [APPLICATION 4] UTF8String OPTIONAL,
            other       [APPLICATION 5] UTF8String OPTIONAL,
            ...
        }

    END
    """
    compiler = asn1tools.compile_string(SPECIFICATION)

    base64_message_def = "YRVCAmlkQwZoZWFkZXJEB2NvbnRlbnQ="  # defined lenght
    decoded_from_base64 = compiler.decode(
        "Message", base64.b64decode(base64_message_def.encode("ascii"))
    )
    print("defined length", decoded_from_base64)

    base64_message_ind = "YYBCAmlkQwZoZWFkZXJEB2NvbnRlbnQAAA=="  # indefinite  lenght
    decoded_from_base64 = compiler.decode(
        "Message", base64.b64decode(base64_message_ind.encode("ascii"))
    )
    print("indefinite length", decoded_from_base64)

Expected output:

defined length {'id': 'id', 'header': 'header', 'content': 'content'}
indefinite length {'id': 'id', 'header': 'header', 'content': 'content'}

Is this a bug in the decoder's handling of indefinite-length elements with extension markers, or is the ASN.1 specification incorrectly interpreted by the decoder, or is the ASN.1 specification incorrect?

Context:

• ans1tools version: 0.166.0
• Windows 11
• Python 3.12.1

Thank you

[aziz-marashly]

I found here that it could be possible as a use-case, as mentioned here in this comment:
https://sourceforge.net/p/asn1c/discussion/357921/thread/31c6b188/

MyTest ::= SEQUENCE {
  a        INTEGER,
  b        INTEGER,
  …,
  c        INTEGER OPTIONAL
  …
}