Regression pack framework

[sscobici]

Proposal: Regression Pack Framework for Symphonia
Hi, I’ve been thinking about how a media library can ensure that optimizations and refactorings do not introduce regressions. To address this, I propose a regression pack framework that serves as the foundation for such a test suite.

Inspiration
This concept is not new—the FFmpeg project is a well-established leader in this area. It employs the FATE test suite alongside a collection of media files derived from bug reports: FATE media suite. Additionally, FFmpeg uses descriptive metadata files for assertions, such as this example.

Unfortunately, I haven’t found the corresponding media file for this specific meta description in their GitHub repository.

Proposal for Symphonia
For Symphonia, I propose adopting a similar approach:

1. A folder containing media files.
2. A folder containing descriptions of the media files.
3. A symphonia-regression crate, which includes tests that validate processing a media file against its expected description.
   Why Not symphonia-check?
   The existing symphonia-check crate is useful, but it doesn’t address scenarios where reference decoders (such as FFmpeg or MediaInfo) produce inconsistent results. For instance:

During the implementation of PTS in Symphonia, I observed that

• FFmpeg generates video PTS that don’t start at 0 for some MP4 files.
• MediaInfo sometimes reports differing duration calculations.
• Other inconsistencies may exist.

Key Challenges
The main challenge is determining:

• Where to source media files that are small, diverse, and free of copyright issues.
• Where to store the media files. Media description files can remain in the repository, but media files themselves are better stored externally.

Example Media File Description
Here’s an example YAML description for Symphonia, listing non-None values from the FormatReader. In the future, packet lists could be added for better demuxer coverage. This approach would also allow validation of basic behaviors such as seeking forward and backward, streamed parsing, etc.

Format:
    format_info:
        format: "FLAC"
        short_name: "flac"
        long_name: "Free Lossless Audio Codec Native"
    tracks:
      - id: 0
        codec_params:
            codec_type: "Audio"
            codec: "FLAC"
            sample_rate: 48000
            bits_per_sample: 16
            channels: "[FL,FR]"
            extra_data:
                data_len: 34
        time_base: "1/48000"
        num_frames: 68546
        start_ts: 0

Draft Implementation
A draft implementation is available here, but you will not be able to run it due to the absence of media files.

Test run example:

> cargo test --package symphonia-regression

running 11 tests
test tests::a_2_wav ... ok
test tests::a_cameraclick_caf ... ok
test tests::a_flac_with_id3v2_flac ... ok
test tests::a_untagged_aac ... ok
test tests::v_bunny_mp4 ... ok
test tests::a_nelly_ciobanu_hora_din_moldova_m4a ... ok
test tests::a_07_commence_mp3 ... ok
test tests::a_2_oga ... ok
test tests::a_full_test_aiff ... ok
test tests::a_defeat_ogg ... ok
test tests::v_avatar_mp4 ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.95s

Failed test run example:

> cargo test --package symphonia-regression
running 11 tests
test tests::a_2_wav ... ok
test tests::a_defeat_ogg ... ok
test tests::a_full_test_aiff ... ok
test tests::a_flac_with_id3v2_flac ... ok
test tests::a_untagged_aac ... ok
test tests::a_07_commence_mp3 ... FAILED
test tests::v_bunny_mp4 ... FAILED
test tests::a_nelly_ciobanu_hora_din_moldova_m4a ... ok
test tests::a_cameraclick_caf ... ok
test tests::a_2_oga ... ok
test tests::v_avatar_mp4 ... ok

failures:

---- tests::a_07_commence_mp3 stdout ----
thread 'tests::a_07_commence_mp3' panicked at symphonia-regression\src\lib.rs:169:9:
assertion `left == right` failed: track: 0, format.track.delay
  left: Some(1200)
 right: Some(1105)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  Media File: N:\Media\Torrent\media-test\regression\audio\good\07 commence.mp3:
   Yaml File: test_files\audio\good\07 commence.mp3.yml:
thread 'tests::a_07_commence_mp3' panicked at symphonia-regression\src\lib.rs:118:13:
FAIL: Validation failed

---- tests::v_bunny_mp4 stdout ----
thread 'tests::v_bunny_mp4' panicked at symphonia-regression\src\lib.rs:259:13:
assertion `left == right` failed: track: 1, format.track.codec_param.v_extra_data[0].data length is different
  left: 100
 right: 43
  Media File: N:\Media\Torrent\media-test\regression\video\bunny.mp4:
   Yaml File: test_files\video\bunny.mp4.yml:
thread 'tests::v_bunny_mp4' panicked at symphonia-regression\src\lib.rs:118:13:
FAIL: Validation failed


failures:
    tests::a_07_commence_mp3
    tests::v_bunny_mp4

test result: FAILED. 9 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.07s

Feedback Requested
I’d love your feedback on:

• The proposed framework and its feasibility.
• Suggestions for sourcing and storing small, diverse, copyright-free media files.
• Potential improvements or additional features that could enhance regression testing for Symphonia.

[pdeljanov]

This is very interesting, and probably needed going forward.

The current testing strategy of mine involves running my personal media library through symphonia-check before cutting a release and comparing results against prior runs. I also test against the problematic files submitted to me. This has worked reasonably well, but is not reproducible at all due to copyright, size reasons, etc. It is also very tedious.

Note, we also have a similar issues regarding benchmarking.

I’d love your feedback on:

The proposed framework and its feasibility.

I don't see anything technically infeasible here.

The biggest challenge will be hosting and automation which are going to cost money. For now, it'd be possible for me to host a small set of copyright and royalty-free test vectors on my server, however, that will not scale long-term. Tests would also need to be manually run by developers on their local systems.

Suggestions for sourcing and storing small, diverse, copyright-free media files.

This is tough. The Internet Archive is one option, however, I do believe they re-encode files so we won't be getting a truly diverse set of samples. One benefit of using my personal library was that it was built over 2 decades and therefore contains many quirky files from long-dead encoders, etc.

Maybe using FFmpeg's test vectors is a good first start?

Potential improvements or additional features that could enhance regression testing for Symphonia.

For demuxer tests, a known-good YAML description like you have is sufficient. As for decoding, a known-good WAVE decoding could also accompany each audio test vector.

We should also add fuzzer generated test vectors that have previously broken Symphonia.