diff --git a/CHANGELOG.md b/CHANGELOG.md index 06cbfeb..a3597f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,20 @@ # Changelog +### v2.3.1-alpha + +* Now if a CD-ROM image contains any wrong sector MSF (like PSX libcript protection), then MSF cleanup is disabled because the process will not be lossles if not. +* Fixed a bug in lzma compression which makes the program to create bigger files. Was much more notizable in images with a lot of merged CDDA and data sectors (Medievil ECM file was 2gb+, which is much more than the original image). +* If there was an error processing the file, then the output file will be removed (added an option to keep it). + +### v2.3.0-alpha + +* Added FLAC compression using flaczlib class which helps to make it in a similar way to zlib +* Added FLAC repository as submodule and updated Makefile +* Fixed some memory leaks + ### v2.2.0-alpha -* Added lz4 HC compression using a lzlib4 class which helps to make it similar to zlib +* Added lz4 HC compression using a lzlib4 class which helps to make it in a similar way to zlib * Added lz4 repository as submódule and updated Makefile * Refactorized analisys, encodong and decoding processes * Fixed a bug in header write/read diff --git a/README.md b/README.md index c913c8c..6af21d6 100644 --- a/README.md +++ b/README.md @@ -20,9 +20,9 @@ This tests were made using the original ECM tool made by Neill Colett, this ECM | | Original Size | Original ECM | ECM Tool Reloaded | 7zip | Original ECM + 7z | ECM Tool Reloaded (Compressed) | |:----------:|:-------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:------------------------------:| -| FFVIII CD3 | 721.508.928 | 635.001.493 (12%) | 574.742.104 (20%) | 438.608.753 (39%) | 365.071.928 (49%) | 367.058.165 (49%) | -| Firebugs | 608.401.248 | 597.995.580 (2%) | 560.062.104 (8%) | 498.312.033 (18%) | 493.397.731 (19%) | 368.291.992 (39%) | -| Tekken 3 | 687.922.368 | 645.620.932 (6%) | 606.047.556 (12%) | 457.049.639 (34%) | 426.481.504 (38%) | 421.560.711 (39%) | +| FFVIII CD3 | 721.508.928 | 635.001.493 (12%) | 574.742.104 (20%) | 438.608.753 (39%) | 365.071.928 (49%) | 362.903.256 (50%) | +| Firebugs | 608.401.248 | 597.995.580 (2%) | 560.062.104 (8%) | 498.312.033 (18%) | 493.397.731 (19%) | 363.280.352 (40%) | +| Tekken 3 | 687.922.368 | 645.620.932 (6%) | 606.047.556 (12%) | 457.049.639 (34%) | 426.481.504 (38%) | 417.325.163 (39%) | As you can see, the ECM Tool Reloaded uncompressed size is improved a bit by removing more data and dummy sectors from the streams. The CDDA tracks in Firebugs have caused a poor reduction ratio in original ECM tool because it keeps all the sectors. Reloaded conversely removes the GAP data, so reduction ratio is improved a bit. @@ -68,4 +68,30 @@ Optional options: # Changelog -Changelog can be viewed in [CHANGELOG.md](CHANGELOG.md) file \ No newline at end of file +Changelog can be viewed in [CHANGELOG.md](CHANGELOG.md) file + +# FAQ + +## How this program works + +CDROM data is splitted in sectors, and those sectors are of 2352 bytes size. Depending of the sector type, it may contains Error Detection Code and Error Correction Code data, which will help to the CDROM drive to detect and recover a sector if it is damaged (for example by dust or even little scratches). This data is generated during the burn process and can be easily generated using an algorithm, and like it is not usefull to keep the data outside a CDROM, can be removed. On the decoding process the program will generate that data again and then the image file will be exactly like the original (checked using MD5, SHA1...). + +You can find more info in the [CDROM Rainbow books](https://en.wikipedia.org/wiki/Rainbow_Books) + +## Why I have created this "Reloaded" version + +The original program created by Neill Corlett is a very good tool... I was using it for a long time to store my CDROM images collection. The problem is that original ecm tool process the file without a clear block separation, storing the sectors info scattered in file. That makes the file unseekable or at least very hard to be seeked, and that is why my program was created. + +The ECM2 format keeps the sectors data in header, so just reading that data you can generate an index to seek in file very fast. Then you just will have to read the sector data depending of which type is, and regenerate the removed data easily. I think that this will be usefull for emulators and I am planning to do some tests in the future. + +ECM2 format also removes more data than original ECM format (depending of sector type, of course), like MSF if not libcrypt protection exists in game, sync, redundant flags, and zeroed sectors. This will help to save up to 8% of the image size. + +Finally, ECM2 format includes compression support which not only will allow to do everything in the same program, also will allow to compress every stream type with a different method, getting higher compression ratios in images that contains CDDA tracks thanks to the FLAC algorithm. The best example is the above Firebugs comparison in compression tests. + +## Can be used with Audio CD images + +Of course!... but the resulting file will be just the compressed stream if compression is used, and the whole data if not. That data will be packed in a ECM2 file format, which make it unplayable using an Audio Player program (even if FLAC compression is used). The best for that kind of images is to use an Audio CD ripper which allows you to create FLAC or another format files, which can be easily played on an Audio Player. + +## Why it still in Alpha + +I still working in this tool, so the ECM2 format can suffer changes. Also the tool is not extensive tested and there are bugs that must be fixed before remove the Alpha status. Just now I am perfoming several tests, and on those tests I have located two bugs (one of them was critical). \ No newline at end of file diff --git a/TODO b/TODO index 43495f9..4fa389a 100644 --- a/TODO +++ b/TODO @@ -6,6 +6,6 @@ * ~~Try to use mmap instead manual buffers to see if works faster.~~ Using FILE buffers. * ~~Add options to compress the stream directly instead to have to use external programs~~ For now zlib. * ~~Convert the program into a library to make it usable by another programs (for now is a class).~~ Doesn't worth. A class is enougth to be used in another programs. -* Try to ~~detect and~~ compress CDDA tracks using FLAC, APE, or similar. +* ~~Try to detect and compress CDDA tracks using FLAC, APE, or similar.~~ * Verify that all CD-ROM modes are working (For now only CDDA, Mode 2 and Mode 2 XA 1 were tested) -* Add a sumary about the sectors types processed and sizes \ No newline at end of file +* ~~Add a sumary about the sectors types processed and sizes~~ \ No newline at end of file diff --git a/ecmtool.cpp b/ecmtool.cpp index 175ef0a..0381bac 100644 --- a/ecmtool.cpp +++ b/ecmtool.cpp @@ -498,7 +498,7 @@ static ecmtool_return_code ecmify( // End Of TOC reached, so file should have be processed completly if (ftello(in) != in_total_size) { - printf("\n\nThere was an error processing the input file...\n"); + printf("\n\nThe input file was not fully readed...\n"); printfileerror(in, infilename); return_code = ECMTOOL_FILE_READ_ERROR; } @@ -535,15 +535,21 @@ static ecmtool_return_code ecmify( free(out_buffer); } // Delete the tocs data - delete[] sectors_toc; - delete[] streams_toc; - delete sTools; + if (sectors_toc) { + delete[] sectors_toc; + } + if (streams_toc) { + delete[] streams_toc; + } + if (sTools) { + delete sTools; + } if (!return_code) { printf("\n\nFinished!\n"); } else { - printf("\n\nThere was an error processing the input file: %d", return_code); + printf("\n\nThere was an error processing the input file: %d\n\n", return_code); // We will remove the file if something went wrong if (!options->keep_output) { if (remove(outfilename)) { @@ -726,16 +732,22 @@ static ecmtool_return_code unecmify( if (out_buffer) { free(out_buffer); } - delete[] sectors_toc; - delete[] streams_toc; - delete sTools; + if (sectors_toc) { + delete[] sectors_toc; + } + if (streams_toc) { + delete[] streams_toc; + } + if (sTools) { + delete sTools; + } // If something went wrong, inform the user and delete the output file if (!return_code) { printf("\n\nFinished!\n"); } else { - printf("\n\nThere was an error processing the input file: %d", return_code); + printf("\n\nThere was an error processing the input file: %d\n\n", return_code); // We will remove the file if something went wrong if (!options->keep_output) { if (remove(outfilename)) { diff --git a/ecmtool.h b/ecmtool.h index d81a368..d210693 100644 --- a/ecmtool.h +++ b/ecmtool.h @@ -2,7 +2,7 @@ // #define TITLE "ecmtool - Encoder/decoder for Error Code Modeler format, with advanced features" #define COPYR "Copyright (C) 2021 Daniel Carrasco" -#define VERSI "2.3.0-alpha" +#define VERSI "2.3.1-alpha" // // This program is free software: you can redistribute it and/or modify // it under the terms of the GNU General Public License as published by diff --git a/specifications.md b/specifications.md deleted file mode 100644 index d139dea..0000000 --- a/specifications.md +++ /dev/null @@ -1,104 +0,0 @@ -# ECM Reloaded specifications - -The ECM Reloaded file specifications are incompatible with the original ECM version. This version only works with CD-ROM images and manage the data in a different way. - -## File specification - -The ECM reloaded file specifications are the following. - -| Data type | Position | Description | -|-------------|---------------|-----------------------------------------------------------------------------------------------| -| Header | 0x00 - 0x03 | The file header consisting in the "ECM" text followed by a 0x01 character | -| Streams TOC | 0x04 - ?? | Information about the stream and their compression, ended by 0x00 | -| Sectors TOC | ?? - ?? | Information about the sectors and their types, ended by 0x00 | -| Data | ?? - ?? | The data copied from the original file used to regenerate it | -| EDC | ?? - ?? | The last 4 bytes of the file is the error detection code, used to verify the file | -------------------------------------------------------------------------------------------------------------------------------- - -### Header - -The header is just to identify the file. Uses the same first three bytes as the original ECM program, but the 4rd changes to 1 to indicate the file version. - -### Streams TOC - -The streams TOC contains all the information about the streams that were detected in the original file (audio and data), their end position in the original file and the compression used in this stream. This will be used to compress the data and the audio separately. The end positions is to verify that all bytes are readed. - -Every stream in the TOC will contains the following data. - -| Data type | Position | Description | -|-------------|-----------|----------------------------------------------------------------------------------------| -| Stream Info | 0x00 | This byte contains the info about the stream (type, compression) | -| Size | 0x01 - ?? | The size of the stream in the ecm file to detect when the program must stop reading | - -A 0x00 byte will be added at the end of the Streams TOC (At the end of all, not in every). The first bit of the Count bytes will be used to store the "continuation" byte, wich will tell the program to continue reading the next byte. This will limit the number stored in those bytes up to 128 instead the size of an uint8_t variable (256). - -#### First byte - -``` -Bits | 0 1 2 3 4 5 6 7 - | || || || | - --- ------------- ------------- --- - | Counter Comp Type - | - └-- Continuation bit -``` - -The first stream byte is composed by the first bit that is a continuation bit and will indicate if the program must use the next byte to the count calculation. Next three bytes are just the first count divider, the next three bytes are used to store the stream compression and the last bit will contain the stream type. - -#### Next Nth bytes - -``` -Bits | 0 1 2 3 4 5 6 7 - | || | - --- -------------------------------- - | Counter - | - └-- Continuation bit -``` - -As was explained in the above section, this byte will have he first bit as "continuation bit", and will indicate if the program must read the next byte to be used in the count calculation. The next 7 bits will be used to store the count divider. - -### Sectors TOC - -The sectors TOC is a list of kind of sectors detected in the original file, which will be used later to regenerate the original data based on its kind. - -| Data type | Position | Description | -|-----------------------|-----------|------------------------------------------------------------------------------------------------------------| -| Sector info and count | 0x00 | The sector type and part of the count (3 bits). The count divided by 8 will be stored here | -| Count | 0x01 - ?? | The rest of the count bytes. If the sectors count is bigger than 8, will be stored in the following bytes | - -The first bit of the Count bytes will be used to store the "continuation" byte, wich will tell the program to continue reading the next byte. This will limit the number stored in those bytes up to 128 instead the size of an uint8_t variable (256). This stream also ends by 0x00 to mark the EOD. - -#### First byte - -``` -Bits | 0 1 2 3 4 5 6 7 - | || || | - --- ------------- ----------------- - | Counter Sector type - | - └-- Continuation bit -``` - -First bit is 1 when the program must continue reading the next byte to be used in sectors count calculation. The last count byte will have this bit to 0. Next 3 bites are used to store the first part of the count, and last four bits are used to store the sector type. - -#### Next Nth bytes - -``` -Bits | 0 1 2 3 4 5 6 7 - | || | - --- -------------------------------- - | Counter - | - └-- Continuation bit -``` - -As was explained in the above section, this byte will have he first bit as "continuation bit", and will indicate if the program must read the next byte to be used in the count calculation. The next 7 bits will be used to store the count. - -### Data - -Contains the data copied from the original file, wich will depend of every sector type. - -### EDC - -Contains a 4 bytes data wich will be used to verify the restored file after extracting the ECM. \ No newline at end of file diff --git a/specifications.odt b/specifications.odt index 0a01b9a..d570953 100644 Binary files a/specifications.odt and b/specifications.odt differ diff --git a/specifications.pdf b/specifications.pdf new file mode 100644 index 0000000..46d45fd Binary files /dev/null and b/specifications.pdf differ