diff --git a/.gitattributes b/.gitattributes deleted file mode 100644 index 1ff0c423..00000000 --- a/.gitattributes +++ /dev/null @@ -1,63 +0,0 @@ -############################################################################### -# Set default behavior to automatically normalize line endings. -############################################################################### -* text=auto - -############################################################################### -# Set default behavior for command prompt diff. -# -# This is need for earlier builds of msysgit that does not have it on by -# default for csharp files. -# Note: This is only used by command line -############################################################################### -#*.cs diff=csharp - -############################################################################### -# Set the merge driver for project and solution files -# -# Merging from the command prompt will add diff markers to the files if there -# are conflicts (Merging from VS is not affected by the settings below, in VS -# the diff markers are never inserted). Diff markers may cause the following -# file extensions to fail to load in VS. An alternative would be to treat -# these files as binary and thus will always conflict and require user -# intervention with every merge. To do so, just uncomment the entries below -############################################################################### -#*.sln merge=binary -#*.csproj merge=binary -#*.vbproj merge=binary -#*.vcxproj merge=binary -#*.vcproj merge=binary -#*.dbproj merge=binary -#*.fsproj merge=binary -#*.lsproj merge=binary -#*.wixproj merge=binary -#*.modelproj merge=binary -#*.sqlproj merge=binary -#*.wwaproj merge=binary - -############################################################################### -# behavior for image files -# -# image files are treated as binary by default. -############################################################################### -#*.jpg binary -#*.png binary -#*.gif binary - -############################################################################### -# diff behavior for common document formats -# -# Convert binary document formats to text before diffing them. This feature -# is only available from the command line. Turn it on by uncommenting the -# entries below. -############################################################################### -#*.doc diff=astextplain -#*.DOC diff=astextplain -#*.docx diff=astextplain -#*.DOCX diff=astextplain -#*.dot diff=astextplain -#*.DOT diff=astextplain -#*.pdf diff=astextplain -#*.PDF diff=astextplain -#*.rtf diff=astextplain -#*.RTF diff=astextplain diff --git a/logo.png b/.github/logo.png similarity index 100% rename from logo.png rename to .github/logo.png diff --git a/README.md b/README.md index 611d4714..123b83ae 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ [![Build][build-badge]][build-url] ![code-progress-badge] [![Contributors][contributors-badge]][contributors-url] [![Discord Channel][discord-badge]][discord-url] [![Docs][docs-badge]][docs-url] [![Wiki][wiki-badge]][wiki-url] -[Sly 1 Decompilation Logo by Cooper941][docs-url] +[Sly 1 Decompilation Logo by Cooper941][docs-url] This is a work-in-progress decompilation of [*Sly Cooper and the Thievius Raccoonus*](https://en.wikipedia.org/wiki/Sly_Cooper_and_the_Thievius_Raccoonus) for the PlayStation 2. It builds the NTSC-U version of the game, `SCUS_971.98` (SHA1: `57dc305d`). @@ -63,7 +63,7 @@ Copy the file `SCUS_971.98` from your Sly 1 game disc to the `disc` directory of If it works, you will see this: ``` -[XXX/XXX] sha1sum checksum.sha1 +[XXX/XXX] sha1sum config/checksum.sha1 out/SCUS_971.98: OK ``` diff --git a/checksum.sha1 b/config/checksum.sha1 similarity index 100% rename from checksum.sha1 rename to config/checksum.sha1 diff --git a/config/readme.md b/config/readme.md new file mode 100644 index 00000000..b3c94cb9 --- /dev/null +++ b/config/readme.md @@ -0,0 +1,21 @@ +# Splat Config Files + +This directory contains config files for splat, the tool we use to split and disassemble the game binary. The config files are used to tell splat how to split the binary and what to name the output files. + +## Files in this directory +- `sly1.yaml` - Main config file for splat. Contains the offsets of each file split in the game binary. Creating new `.c` files involves modifying this file to create a `c` split and then running `python configure.py` to generate the new files. +- `symbol_addrs.txt` - Contains the addresses of symbols in the game binary. Used by splat to properly reference functions and variables in the game code when splitting the binary. +- `checksum.sha1` - SHA1 checksum of the original game binary. Used to verify the integrity of the binary after compilation. + +## Adding new splits +To add a new source code file to the project, you must add the split to `sly1.yaml`. Here is an example of some file splits: +```yaml + - [0x1f560, c, P2/brx] + - [0x1fe70, asm, P2/ac] + #- [0x, asm, P2/act] +``` +- The first line tells splat to split the binary at offset `0x1f560` and create a new file `P2/brx.c`. It will contain the data from the translation unit starting at `0x1f560` and ending at the next split offset. The `c` tells splat to create a `.c` file in `src/P2/`. +- The second line tells splat to split that the functions starting from offset `0x1fe70` are in the translation unit `P2/ac`, but not to create a new `.c` file. +- The third line is commented out and does nothing, because we know the next translation unit is `P2/act`, but we don't yet know the offset of the first function in that TU. + +When splat is run, it will create `.s` assembly files for each non-matching functions in each split in the `asm/nonmatchings/` directory. diff --git a/config/readme.txt b/config/readme.txt deleted file mode 100644 index 6a0e218c..00000000 --- a/config/readme.txt +++ /dev/null @@ -1 +0,0 @@ -This directory contains config files for Splat, which is used for Binary splitting. diff --git a/config/sly1.yaml b/config/sly1.yaml index faec49ac..ba221822 100644 --- a/config/sly1.yaml +++ b/config/sly1.yaml @@ -128,7 +128,7 @@ segments: - [0x529e0, c, P2/difficulty] - [0x53438, c, P2/dl] - - [0x5380C, asm, dl_padding] # Needed to align dmas + - [0x5380C, asm, P2/dl_padding] # Needed to align dmas - [0x53810, c, P2/dmas] - [0x53E28, asm, P2/dsp] #- [0x, asm, P2/dysh] diff --git a/configure.py b/configure.py index 725a1cd1..592fe7c0 100644 --- a/configure.py +++ b/configure.py @@ -1,5 +1,6 @@ """ -Configure the project for building. +Configures the project for building. Invokes splat to split the binary and +creates build files for ninja. """ #! /usr/bin/env python3 import argparse @@ -200,7 +201,7 @@ def build( ninja.build( ELF_PATH + ".ok", "sha1sum", - "checksum.sha1", + "config/checksum.sha1", implicit=[ELF_PATH], ) else: diff --git a/disc/readme.md b/disc/readme.md new file mode 100644 index 00000000..6b734e61 --- /dev/null +++ b/disc/readme.md @@ -0,0 +1,6 @@ +# Disc Files + +This directory is intentionally left empty. It is where you should place files you extract from your own copy of the game. The files you will need are: +- `SCUS_971.98` - The original executable of the game. This file is required to build the project. + +Extract your original ELF file and place it in this directory before building the project. diff --git a/disc/readme.txt b/disc/readme.txt deleted file mode 100644 index f6eea6cc..00000000 --- a/disc/readme.txt +++ /dev/null @@ -1 +0,0 @@ -To build the project, you need to extract the executable file SCUS_971.98 from your own copy of the game and put it in this directory. diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 6bad638c..e72d9daf 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -1,4 +1,6 @@ -# So you want to decompile Sly 1... +# Contributing Guide + +## So you want to decompile Sly 1... You've come to the right place! We welcome new contributors of all skill levels. Even if you've never tried game hacking, modding, or reverse-engineering, we have a dedicated community of modders and reverse-engineers to help you get started. Feel feel to join our [Discord server](https://discord.gg/2GSXcEzPJA) and ask for help in #sly-research. @@ -12,17 +14,18 @@ By following this guide, you will learn how to fork the repo on GitHub, choose a 2. [Find a function to match](#find-a-function-to-match) 3. [Match the function using Decomp.me](#match-the-function-using-decompme) 4. [Integrate the matched code](#integrate-the-matched-code) +5. [Make a Pull Request](#make-a-pull-request) 5. [Conclusion](#conclusion) -## Getting Started +## Getting started -If you haven't used Git before, follow the [Beginner's Guide](/docs/BEGINNERSGUIDE.md) to learn how to fork the repo, make changes, and submit a pull request. If you are already familiar with Git, continue on with this guide. +If you haven't used Git before, follow the [Beginner's Guide](/docs/BEGINNERSGUIDE.md) to learn how to fork the repo, make changes, and submit a pull request when you're done. If you're already familiar with Git, continue on with this guide. -Start by following the instructions in the [README](/README.md) to build the project. If you see the following in the terminal, you're ready to start contributing: +First, follow the instructions in the [README](/README.md) to build the project. If you see the following in the terminal, you're ready to start contributing: ``` -[XXX/XXX] sha1sum checksum.sha1 +[XXX/XXX] sha1sum config/checksum.sha1 out/SCUS_971.98: OK ``` @@ -41,14 +44,14 @@ First, find a function in the game that you want to match. There are a few ways Once you have chosen a function, you can use the website [decomp.me](https://decomp.me/) to match it. 1. Go the website and click "Start decomping". -2. Under "Preset", select "Sly Cooper and the Thievius Raccoonus". +2. Click "PS2", then under "Preset", select "Sly Cooper and the Thievius Raccoonus". 3. Under "Diff label", enter the name of the function. 4. Find the `.s` file corresponding to the function you want to match in the `asm/nonmatchings` directory. Copy the contents of the file into the "Target assembly" box. 5. Copy any functions, global variables, or data types used by the function from the headers in the `include` directory into the "Context" box. * E.g. For the function `void ResetClock(struct CLOCK* pclock, float t)`, you will need to include the definition of `CLOCK` struct, and the `TICK` data type since it is used in the CLOCK struct. * You should almost always copy the entirety of "types.h" to ensure you have all the necessary primitive types. 6. Click "Create scratch". -7. (Optional) Go to the "Options" tab and under "Debug information" select `-g3 (Macro expansions)`. This will show you which line numbers the source code coreespond to each line in the assembly. +7. **(Recommended)** Go to the "Options" tab and under "Debug information" select `-g3 (Macro expansions)`. This will show you which line numbers the source code correspond to each line in the assembly. Then start writing your code under the "Source code" tab. It will tell you what percent of the code matches the original. Tweak the code until it matches 100%. An example scratch can be found here: https://decomp.me/scratch/hmmyP @@ -60,12 +63,12 @@ Once the function matches 100%, follow these steps to integrate it into the proj 1. Replace the `INCLUDE_ASM` macro in the `.c` file with the matched function. * If the function is in a new file, **do not create a new file**. Creating new `.c` files is done by editing the `config/sly1.yaml` file to change the file split from `asm` to `c`, then running `python configure.py` to generate the new file. If you don't know how to do this, feel free to ask for help in the Discord server. 2. Check `config/symbol_addrs.txt` to see if the mangled name of the function is present. - * If it is not, add the **mangled name** of the function with it's address. The mangled name of the function can be found by searching the strings in the May 19 prototype elf, if it's the same as the release version. If you don't know how to do this, ask for help in the Discord server; Someone will be able to find it for you easily. - * The symbol_addrs.txt file is sorted alphabetically by filename, then by address within the file. + * If it is not, add the **mangled name** of the function with it's address. The mangled name of the function can be found in the debug symbols for the May 2002 prototype. If you don't know how to find it, ask for help in the Discord server; someone will be able to find it for you easily. + * The symbol_addrs.txt file is grouped alphabetically by filename, then sorted by address within the file. -The project should build and match. Here are some tips for troubleshooting common issues: -* Undefined reference errors usually mean the entry in the symbol_addrs.txt is wrong. Make sure the function name is mangled in symbol_addrs.txt and normal/un-mangled in the source code, and the mangled version matches the signature of the function. Also ensure that the address is correct in symbol_addrs.txt. -* Checksum failed means the compiled elf doesn't match the original. If it matches on decomp.me, it might be an issue with the compiler, so open an issue on GitHub or let someone know on Discord. +The project should build and match. Here are some common troubleshooting tips: +* `undefined reference error` usually means the entry in the symbol_addrs.txt is wrong. Make sure the function name is mangled in symbol_addrs.txt and unmangled in the source code, and the mangled version matches the signature of the function. Also ensure that the address is correct in symbol_addrs.txt. +* `checksum failed` means the compiled elf with your added code doesn't match the original. If it matches on decomp.me but not in the project, it might be an issue with the compiler, so open an issue on GitHub or let someone know on Discord. -## Code Review Process +## Make a pull request Once you add your code to the project and it builds + matches, you can open a pull request to merge your fork into the main branch of the project. A code reviewer will need to approve it before it can be merged into the main branch. We are a volunteer-driven project, so please be patient while we review your code. These are the main things we will look for in your code: - * It compiles without any errors. * The compiled elf matches the original elf. -If everything looks good, we will merge your pull request as soon as possible. If anything needs to be fixed, we will let you know. +If everything looks good, we will merge your pull request as soon as possible. If anything needs to be addressd, we will let you know. ## Conclusion diff --git a/reference/readme.txt b/reference/readme.md similarity index 74% rename from reference/readme.txt rename to reference/readme.md index 189f3677..728f254f 100644 --- a/reference/readme.txt +++ b/reference/readme.md @@ -1,8 +1,13 @@ -This directory contains reference files you can use to help with decompiling game functions. They are all source code files we wrote by hand before we used Splat to decompile and reassemble the game to a byte-matching executable. Because of this, none of the functions in this directory match, but we tried to make them as close as possible to the original game functions. +# Reference files -The include directory contains header files that are used by the source files in the src dir. Many of the short data structures/enums are correct and can be copied directly into the actual decomp. The longer ones are probably not correct (missing fields, wrong field types, etc), but they can be used as a reference. +This directory contains reference files you can use to help with decompiling game functions. They are all source code files we wrote by hand before we used Splat to decompile and reassemble the game to a byte-matching executable. -The src/P2 directory contains the source code for the game engine. The following is a list of all known source code file names from a prototype build of the game. Some of them are not present in the final game. +It is organized as follows: +- `include/` contains header files for the game engine Many of the very short structs/enums are correct. The longer ones are probably not correct (missing fields, wrong field types, etc), but they can be used as a reference. +- `src/P2/` contains source code files for the game engine. None of the code is matching, though at the time we tried to make them resemble the original code as closely as possible, so they may help you understand the logic and control flow of certain functions. + +## Source files +The following is a list of all known source code file names from a **prototype build** of the game. Some of them are not present in the final game, and the final game has some that are not present here. ac.c act.c actseg.c diff --git a/scripts/readme.md b/scripts/readme.md new file mode 100644 index 00000000..dc7ed682 --- /dev/null +++ b/scripts/readme.md @@ -0,0 +1,38 @@ +# Scripts + +This directory contains the following scripts used to setup and build the project. +- `build.sh` +- `check_progress.sh` +- `quickstart.sh` +- `setup_progd_linux.sh` + +There are a few others, but they are not used in the normal contributing workflow. + +## build.sh + +Runs a clean reconfigure (deletes build files and splits the binary), then builds the project. Will warn you if `disc/SCUS_971.98` is not present. + +## check_progress.sh + +Checks the progress of the project by comparing the number of bytes in the original game binary to the number of bytes that have been matched. The output will be in this format: + +``` +Category : DecompedSize / Total OfFolder% ( OfTotal%) +all : 10516 / 1143680 0.9195% ( 0.9195% / 100.0000%) + +sce : 0 / 116164 0.0000% ( 0.0000% / 10.1570%) +P2 : 10516 / 1005644 1.0457% ( 0.9195% / 87.9305%) +data : 0 / 21872 0.0000% ( 0.0000% / 1.9124%) +``` + +The value that we consider the progress of the project is row "`P2`", column "`OfFolder%`". This is the percentage of the game engine code that has been matched. + +## quickstart.sh + +Installs the necessary dependencies using pip/apt and sets up the build environment. You only need to run this script once. + +## setup_progd_linux.sh + +Installs the compiler needed to build the project on Linux. `quickstart.h` will run this script for you, so you don't need to run both. + +There is a similar script for Windows, but the compiler