new 2.4.0 breaks Prefixed(uint16, 'utf8')

[Niluge-KiWi]

Scenario: Field Prefixed(uint16, 'utf8')

• before: caterpillar version from ~2024-03-18: OK
• after: caterpillar version 2.4.0 (from now): caterpillar.exception.StructException: 'str' object has no attribute '__unpack__'

It may come from #17, the changelog doesn't explicitly document this breaking change: Prefixed now supports arbitrary structs

The fix: Prefixed(uint16, encoding='utf8'): indeed 'utf8' was now accidentally used as struct, not as encoding.

Maybe this change in the PR examples/ is broken too : https://github.com/MatrixEditor/caterpillar/pull/17/files#diff-8f2f4f0eab2df6f648de905b4f6e80be48cd664dc2a7e8651a514dabd155d8c4L88
maybe it should be Prefixed(vint, encoding="utf-8")?

For a data point, what I was expecting as a user:

• (first: SemVer versioning, or something in the README or doc telling otherwise; I understood breaking changes could happen in 2.x with the release description: good enough)
• breaking changes as such documented more explicitly.

Anyway, thank you for this great library!

[Niluge-KiWi]

After more analysis, it seems worse: when no struct is specified, the encoding doesn't seem applied: it returns a bytes, not a decoded str, not sure why.
I'm rolling back to 2.2.0 for now.

[MatrixEditor]

Hey @Niluge-KiWi,

thanks for reporting this - I will get back to this issue in a few days.

[MatrixEditor]

Hi again,

here are some points regarding the issue you've found linked to the Prefixed class:

1. In version 2.4.0 the Prefixed class was changed in order to support arbitrary struct while also providing backwards compatibility support using the encoding parameter. Yes, documentation is lacking and that change should have been documented more explicitly.
2. The issue has been identified and will be fixed with the next patch release.
3. Changes in nibarchive.py still work as expected.
4. The Prefixed class targets situations where a custom structure is prefixed by its length. e.g.: Prefixed(uint8, CString[...]). I'm working on introducing PascalString to replace the Prefixed class when strings are required.

I know documentation of this library is lacking content - trying my best to fill all missing spots.

[MatrixEditor]

Tracking documentation related stuff in #20 for now.

[Niluge-KiWi]

Hello @MatrixEditor ,

2. I confirm that v2.4.1/[FIX] Prefixed encoding Parameter #19 fixes my issue Prefixed(uint16, encoding='utf8'), thank you!

Overall, thanks for the work on the project, it helped me a lot even with the current state of the documentation!