Python-Based Generic Output Checking Autograder Framework for use with the Gradescope autograder.
Originally based on the General output-checking autograder example given by Gradescope.
- Download the latest release from the releases page or clone the repository
- Place testing drivers in the
tests/drivers
directory - Place any input files that the student's code will need to read or edit in the
tests/io_files
directory - Put unit tests in the
tests/test.py
file which can use the drivers and io_files - Zip up the autograder:
sh zipper.sh
(The name of the zip file can be changed in thezipper.sh
script) - Upload the autograder to Gradescope
- Upload your sample code to Gradescope and see if everything works as expected
Even though many files are mentioned, the key file to change is the test.py
file
setup.sh
- Run when the autograder docker image is built. This is where you should install any dependencies that your autograder needs and setup users. You could compile code here, but it's easier to handle errors and display them nicely if you compile in thetest.py
file. Also injects a checksum into the harness, preventingrun_autograder
from being overwritten by the student.run_autograder
- What Gradescope runs when the autograder is executed. This is typically used to start the Python test script.run_tests.py
- Finds all the unit tests within thetests
directory and runs them. It then logs the results to aresults.json
. This shouldn't need to be changed. Typically, it just runstest.py
requirements.txt
- A list of Python packages that are installed prior to the autograder running.zipper.sh
- A small script that zips up the autograder for upload to Gradescope.example_sample_code/samplecode.zip
- Sample code which should pass all the example test cases given in this framework
Directory | Use | Function |
---|---|---|
tests/drivers |
Store drivers | Copied into the source folder without giving read access to the "student" user |
tests/io_files |
Store input files | Copied into the source folder and granted read access to the "student" user |
test.py
- The Python script which compiles, runs, and observes drivers and the student's code. Examples are given in the file.utils
- A Python package that contains custom utilities for the autograder. This is where to define functions that are used multiple times in the autograder.
While security was not the primary focus during the development of this framework, several precautions have been implemented to mitigate common attack vectors. These measures include preventing attempts by students to tamper with critical system files, such as tarring the root directory, capturing hidden test cases, or overwriting the run_autograder
script to manipulate their scores.
Many of the utility functions in utils
leverage a restricted "student" user with limited permissions. Provided that the student's code is not executed with root privileges, the contents within the tests
directory should remain secure. However, further testing is recommended to ensure the robustness of these precautions.
Care should be taken when running student-provided makefiles without the "student" user, as they may be used to execute arbitrary commands on the system.
Security becomes increasingly critical as the number of students using the framework grows, making it more challenging to identify malicious submissions. For additional security considerations, Gradescope's best practices provide valuable guidelines. As an example, running a student's code as root while they have a malicious setup, such as this, may expose hidden test cases.
If you have a new feature you would like to add it to the framework, please ensure it meets the following 6 requirements:
-
Reliable
Everything built directly into the main framework must work consistently. This is meant to be a reliable baseplate for autograder developers.
We'll continue adding to the example_sample_code, so that we can test new features.
Experimental features can be kept on separate branches or forks. -
Modular
It must be easy to include or exclude any feature. Don't force it on the developer, but if they want to use this feature, it should be easy to get to. This aligns with breaking things into multiple files and using Python's packaging system to stay organized and promote reusability. -
Documented
How it works, when to use it, and what it does should be well documented. Add documentation to the README.md in the utils directory, so new users can find your feature and understand when to use it. -
Useful
This feature should address a real problem and either save time or effort for the developer or students. -
Minimal Footprint
Don't introduce unnecessary complexity. We want debugging student's submissions on the autograder to get easier, not harder. Autograders also need to build and run quickly. -
Backwards Compatibility
New features shouldn't break the old features. If an old feature needs to be removed, it should be replaced with a suitable alternative.
To contribute to this repository, follow one of the two workflows below:
- Clone the repository:
git clone <repository-url>
- Create a new branch for your changes:
git checkout -b <branch-name>
- Implement your changes.
- Commit your changes, ensuring all pre-commit hooks are satisfied:
git commit -m "Description of changes"
- Push your branch to the repository:
git push origin <branch-name>
- Create a pull request for review.
- Wait for approval and feedback.
- Once approved, merge your branch into the main branch.
- Delete your branch after merging to keep the repository clean:
git branch -d <branch-name>
- Celebrate!
- Fork the repository by clicking the "Fork" button on GitHub.
- Clone your forked repository:
git clone <your-fork-url>
- Create a new branch for your changes:
git checkout -b <branch-name>
- Implement your changes.
- Commit your changes, ensuring all pre-commit hooks are satisfied:
git commit -m "Description of changes"
- Push your changes to your fork:
git push origin <branch-name>
- Create a pull request from your fork to the original repository for review.
- Wait for approval and feedback.
- Once approved, your changes will be merged into the original repository.
- Keep your fork in sync with the original repository by pulling updates:
git remote add upstream <original-repo-url>
git fetch upstream
git merge upstream/main
By following either method, you can contribute effectively to the project.
Feel free to email me at ema8@clemson.edu, or if you are a Clemson student, message me on Teams to ask specific questions or set up a time to meet.
If you find any issues, please report them on the issues page of the repository. Please include as much information as possible so that the issue can be reproduced and fixed.
If you have any feature requests, also use the issues page of the repository. Please include as much information as possible so that the feature can be implemented as requested.