Skip to content

Commit

Permalink
Merge pull request #15 from Elan456/dev
Browse files Browse the repository at this point in the history 
Formatting and README updates
  • Loading branch information
@Elan456
Elan456 authored Sep 29, 2024
2 parents ecd61e7 + 6193a5e commit 73b0756
Show file tree
Hide file tree
Showing 3 changed files with 127 additions and 83 deletions.
124 changes: 74 additions & 50 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Clemson University Autograder Framework

Python-based generic output checking autograder framework for use with the
Python-Based Generic Output Checking Autograder Framework for use with the
[Gradescope](https://gradescope.com) autograder.

Originally based on the
Expand All @@ -9,29 +9,31 @@ given by Gradescope.

# Usage
1. Download the latest release from the [releases page](https://github.com/Elan456/cu-autograder-framework/releases)
or clone the repository.
2. Place testing drivers in the `tests/drivers` directory.
3. Place any input files that the student's code should read in the `tests/io_files` directory.
4. Put unit tests in the `tests/test.py` file which can use the drivers and io_files.
5. Zip up the autograder: `sh zipper.sh` (The name can be changed in the `zipper.sh` file)
6. Upload the autograder to Gradescope.
7. Upload your sample code to Gradescope and see if everything works as expected.
or clone the repository
2. Place testing drivers in the `tests/drivers` directory
3. Place any input files that the student's code will need to read or edit in the `tests/io_files` directory
4. Put unit tests in the `tests/test.py` file which can use the drivers and io_files
5. Zip up the autograder: `sh zipper.sh` (The name of the zip file can be changed in the `zipper.sh` script)
6. Upload the autograder to Gradescope
7. Upload your sample code to Gradescope and see if everything works as expected

## Files (and what they do)

Even though many files are mentioned, the two things that must be changed are the
`test.py` folder and adding drivers to the `drivers` directory.
Even though many files are mentioned, the key file to change is the `test.py` file

* `setup.sh` - Ran when the autograder docker image is built. This is where
* `setup.sh` - Run when the autograder docker image is built. This is where
you should install any dependencies that your autograder needs and setup
users. You could compile code here, but it's easier to handle errors and
display them nicely if you compile in the `test.py` file.
display them nicely if you compile in the `test.py` file. Also injects a
checksum into the harness, preventing `run_autograder` from being overwritten
by the student.
* `run_autograder` - What Gradescope runs when the autograder is executed.
This is typically used to start the Python test script.
* `run_tests.py` - Finds all the unit tests within the `tests` directory and
runs them. It then logs the results to a `results.json`
This is typically used to start the Python test script.
* `run_tests.py` - Finds all the unit tests within the `tests` directory
and runs them. It then logs the results to a `results.json`.
This shouldn't need to be changed. Typically, it just runs `test.py`
* `requirements.txt` - A list of Python packages that need to be installed
for the autograder to run.
* `requirements.txt` - A list of Python packages that are installed
prior to the autograder running.
* `zipper.sh` - A small script that zips up the autograder for upload to
Gradescope.
* `example_sample_code/samplecode.zip` - Sample code which should pass
Expand All @@ -42,57 +44,79 @@ all the example test cases given in this framework

| Directory | Use | Function |
|------------------|--------------------------|----------------------------------------------------------------------------------|
| `tests/drivers` | Store non-Python drivers | Copied into the `source` folder **without giving** read access to the "student" user |
| `tests/io_files` | Store input files | Copied into the `source` folder and **given** read access to the "student" user |
| `tests/drivers` | Store drivers | Copied into the `source` folder **without giving** read access to the "student" user |
| `tests/io_files` | Store input files | Copied into the `source` folder and **granted** read access to the "student" user |

* `test.py` - The Python script which compiles, runs, and observes drivers and
the student's code. Examples are given in the file.
* `utils` - A Python package that contains our custom utilities for the autograder.
* `utils` - A Python package that contains custom utilities for the autograder.
This is where to define functions that are used multiple times in the autograder.

# Security
This framework wasn't made with security as the main focus; however, precautions
were taken to prevent simple attacks such as students attempting to Tar the entire root
and capture hidden test cases.

Many of the utility functions in `utils` make use of the "student" user who has
limited permissions. As long as student's code is not run as root, everything
within the tests directory should be safe, but more testing is still needed.
# Security

While security was not the primary focus during the development of this framework, several precautions have been implemented to mitigate common attack vectors. These measures include preventing attempts by students to tamper with critical system files, such as tarring the root directory, capturing hidden test cases, or overwriting the `run_autograder` script to manipulate their scores.

Be careful running student's makefiles without the "student" user
as they can be used to run arbitrary commands on the system.
Many of the utility functions in `utils` leverage a restricted "student" user with limited permissions. Provided that the student's code is not executed with root privileges, the contents within the `tests` directory should remain secure. However, further testing is recommended to ensure the robustness of these precautions.

Security becomes important when there are more students because it becomes
more difficult to find malicious submissions.
[Gradescope's best practices](https://gradescope-autograders.readthedocs.io/en/latest/best_practices/)
mentions a few things to keep in mind.
For example, if you run the student's code as root while they have something
like [this](https://www.reddit.com/r/csMajors/comments/rlkf55/if_your_school_uses_gradescope_autograder_hidden/)
set up, they'll find all your hidden test cases.
Care should be taken when running student-provided makefiles without the "student" user, as they may be used to execute arbitrary commands on the system.

Security becomes increasingly critical as the number of students using the framework grows, making it more challenging to identify malicious submissions. For additional security considerations, [Gradescope's best practices](https://gradescope-autograders.readthedocs.io/en/latest/best_practices/) provide valuable guidelines. As an example, running a student's code as root while they have a malicious setup, such as [this](https://www.reddit.com/r/csMajors/comments/rlkf55/if_your_school_uses_gradescope_autograder_hidden/), may expose hidden test cases.

# Contributing

## Making Changes
1. Clone the repository
2. Create a new branch
3. Make your changes
4. Commit your changes and modify files to satisfy the pre-commit hooks
5. Create a pull request
6. Wait for approval
7. Merge your branch into main
8. Delete your branch
9. Celebrate!!

## Getting Help
Feel free to email me at [ema8@clemson.edu](ema8@clemson.edu), or if you are a Clemson student,

To contribute to this repository, follow one of the two workflows below:

### Option 1: Direct Contribution
1. Clone the repository:
`git clone <repository-url>`
2. Create a new branch for your changes:
`git checkout -b <branch-name>`
3. Implement your changes.
4. Commit your changes, ensuring all pre-commit hooks are satisfied:
`git commit -m "Description of changes"`
5. Push your branch to the repository:
`git push origin <branch-name>`
6. Create a pull request for review.
7. Wait for approval and feedback.
8. Once approved, merge your branch into the main branch.
9. Delete your branch after merging to keep the repository clean:
`git branch -d <branch-name>`
10. Celebrate!

### Option 2: Forking the Repository
1. Fork the repository by clicking the "Fork" button on GitHub.
2. Clone your forked repository:
`git clone <your-fork-url>`
3. Create a new branch for your changes:
`git checkout -b <branch-name>`
4. Implement your changes.
5. Commit your changes, ensuring all pre-commit hooks are satisfied:
`git commit -m "Description of changes"`
6. Push your changes to your fork:
`git push origin <branch-name>`
7. Create a pull request from your fork to the original repository for review.
8. Wait for approval and feedback.
9. Once approved, your changes will be merged into the original repository.
10. Keep your fork in sync with the original repository by pulling updates:
`git remote add upstream <original-repo-url>`
`git fetch upstream`
`git merge upstream/main`

By following either method, you can contribute effectively to the project.

# Getting Help
Feel free to email me at [ema8@clemson.edu](mailto:ema8@clemson.edu), or if you are a Clemson student,
message me on Teams to ask specific questions or set up a time to meet.

## Issues
# Issues
If you find any issues, please report them on the issues page
of the repository. Please include as much information as possible
so that the issue can be reproduced and fixed.

## Feature Requests
# Feature Requests
If you have any feature requests, also use the issues
page of the repository. Please include as much information as possible
so that the feature can be implemented as requested.
Expand Down
84 changes: 52 additions & 32 deletions tests/utils/parsing.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@ def parse_file(input_filename: str) -> TranslationUnit:
return index.parse(input_filename)


def find_entities(node: Cursor, include_calls: bool,
*entities: tuple[CursorKind, str]) -> list[Cursor, ...]:
def find_entities(
node: Cursor, include_calls: bool, *entities: tuple[CursorKind, str]
) -> list[Cursor, ...]:
"""
Find the desired entities as a list of Cursors from the supplied node
"""
Expand All @@ -32,8 +33,10 @@ def find_entities(node: Cursor, include_calls: bool,
current_entity = stack.pop()

for child in current_entity.get_children():
if any(child.kind == kind and child.spelling == name
for kind, name in entities):
if any(
child.kind == kind and child.spelling == name
for kind, name in entities
):
# add matching entries
found_entities.append(child)
else:
Expand All @@ -48,14 +51,17 @@ def find_entities(node: Cursor, include_calls: bool,

for child in current_entity.get_children():
# continue search only for children in same file
if str(child.location.file) == str(current_entity.
location.file):
if str(child.location.file) == str(
current_entity.location.file
):
# check if child is call expression to ref in same file
if (child.kind == CursorKind.CALL_EXPR and
child.referenced is not None and
str(child.referenced.location.
file) == str(current_entity.location.file)
and child.referenced not in found_entities):
if (
child.kind == CursorKind.CALL_EXPR
and child.referenced is not None
and str(child.referenced.location.file)
== str(current_entity.location.file)
and child.referenced not in found_entities
):
# add call expression's reference if true
found_entities.append(child.referenced)
stack.append(child.referenced)
Expand All @@ -81,9 +87,12 @@ def get_direct_include_offsets(tu: TranslationUnit) -> tuple[int, ...]:
Returns the offsets for the inclusion directives directly in the file
"""

return (x.location.offset for x in
filter(lambda x: x.depth == 1, # only direct includes
tu.get_includes()))
return (
x.location.offset
for x in filter(
lambda x: x.depth == 1, tu.get_includes() # only direct includes
)
)


def get_global_ranges(node: Cursor) -> list[tuple[int, int, str]]:
Expand All @@ -98,9 +107,11 @@ def get_global_ranges(node: Cursor) -> list[tuple[int, int, str]]:
return found_entities

for child in node.get_children():
if child.kind in (CursorKind.VAR_DECL, # global variables
CursorKind.USING_DIRECTIVE, # using namespace ...
CursorKind.USING_DECLARATION): # using ...
if child.kind in (
CursorKind.VAR_DECL, # global variables
CursorKind.USING_DIRECTIVE, # using namespace ...
CursorKind.USING_DECLARATION,
): # using ...
found_entities.append(child)

# store entity ranges for found requested entities
Expand All @@ -110,15 +121,18 @@ def get_global_ranges(node: Cursor) -> list[tuple[int, int, str]]:
return sorted(entity_ranges, key=lambda x: x[0])


def get_function_ranges(cursor: Cursor, include_calls: bool,
*function_names: str) -> list[tuple[int, int]]:
def get_function_ranges(
cursor: Cursor, include_calls: bool, *function_names: str
) -> list[tuple[int, int]]:
"""
Get the ranges for supplied functions
"""

found_functions = find_entities(cursor, include_calls,
*((CursorKind.FUNCTION_DECL, func)
for func in function_names))
found_functions = find_entities(
cursor,
include_calls,
*((CursorKind.FUNCTION_DECL, func) for func in function_names)
)

# store entity ranges for found requested functions
function_ranges = map(get_cursor_range, found_functions)
Expand All @@ -127,16 +141,21 @@ def get_function_ranges(cursor: Cursor, include_calls: bool,
return sorted(function_ranges, key=lambda x: x[0])


def extract_functions(input_filename: str, output_filename: str,
include_directives: bool, include_calls: bool,
*function_names: str) -> None:
def extract_functions(
input_filename: str,
output_filename: str,
include_directives: bool,
include_calls: bool,
*function_names: str
) -> None:
"""
Extract the desired function names from the input file into the output file
"""

unit = parse_file(input_filename)
function_ranges = get_function_ranges(unit.cursor, include_calls,
*function_names)
function_ranges = get_function_ranges(
unit.cursor, include_calls, *function_names
)

contents = []
with open(input_filename, "r") as input_file:
Expand All @@ -146,11 +165,11 @@ def extract_functions(input_filename: str, output_filename: str,
input_file.seek(offset)
contents.append("#include " + input_file.readline().strip())

for (offset, length) in get_global_ranges(unit.cursor):
for offset, length in get_global_ranges(unit.cursor):
input_file.seek(offset)
contents.append(input_file.read(length) + ";")

for (offset, length) in function_ranges:
for offset, length in function_ranges:
# add function to content to be written
input_file.seek(offset)
contents.append(input_file.read(length))
Expand All @@ -159,8 +178,9 @@ def extract_functions(input_filename: str, output_filename: str,
output_file.write("\n".join(contents))


def remove_functions(input_filename: str, output_filename: str,
*function_names: str) -> None:
def remove_functions(
input_filename: str, output_filename: str, *function_names: str
) -> None:
"""
Remove specified function names from the input file and place in the output
file
Expand All @@ -172,7 +192,7 @@ def remove_functions(input_filename: str, output_filename: str,
contents = []
with open(input_filename, "r") as input_file:
prev_pos = 0
for (offset, length) in function_ranges:
for offset, length in function_ranges:
# add file contents excluding function to remove
contents.append(input_file.read(offset - prev_pos))
input_file.seek(offset + length)
Expand Down
2 changes: 1 addition & 1 deletion zipper.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@
# SET NAME BELOW
name="autograder"

zip -r ../$name.zip . -x "*.git*"
zip -r ../$name.zip . -x "*.git*" -x "example_sample_code/*"

0 comments on commit 73b0756

Please sign in to comment.