Skip to content

Commit

Permalink
rebranding, documentation, and polishing
Browse files Browse the repository at this point in the history
  • Loading branch information
@Giantpizzahead
Giantpizzahead committed Jul 23, 2020
1 parent d8f7714 commit dfca98b
Show file tree
Hide file tree
Showing 21 changed files with 3,039 additions and 2,923 deletions.
65 changes: 27 additions & 38 deletions .github/ISSUE_TEMPLATE/bug_report.md
View file
Original file line number Diff line number Diff line change
@@ -1,38 +1,27 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''

---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]

**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
---
name: Bug report
about: Create a report to help improve JudgeLite
title: ''
labels: ''
assignees: ''

---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here.
40 changes: 20 additions & 20 deletions .github/ISSUE_TEMPLATE/feature_request.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''

---

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
---
name: Feature request
about: Suggest an idea for JudgeLite
title: ''
labels: ''
assignees: ''

---

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
121 changes: 77 additions & 44 deletions .github/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,44 +1,77 @@
# Submission Judge

<a href="https://github.com/Giantpizzahead/submission-judge/actions?query=workflow%3Abuild"><img alt="build" src="https://github.com/Giantpizzahead/submission-judge/workflows/build/badge.svg" /></a>
<a href="https://codecov.io/gh/Giantpizzahead/submission-judge"><img alt="codecov" src="https://codecov.io/gh/Giantpizzahead/submission-judge/branch/master/graph/badge.svg" /></a>
<a href="https://github.com/Giantpizzahead/submission-judge/blob/master/LICENSE"><img alt="license" src="https://img.shields.io/github/license/giantpizzahead/submission-judge" /></a>

A simple judge for checking the correctness of code. Designed to be used for simple competitive programming problems.

This is a personal project, and it is **still in EARLY DEVELOPMENT**.

Documentation is currently lacking (will be added later... whenever i feel like it lol), but most of the function / variable names should be self-explanatory.

Hard-coding things is bad, so most of the 'hard-coded' values are configurable via environment variables! Check the variables in **env_vars.py** to figure out what they do.

Also look at the sample problems in the **sample_problem_info** folder (check the README.txt in problem #1 as well) to figure out how problems are structured. The problem creation process will probably be abstracted into a web form at some point though.

## Demo
A live version of this Docker image can be found <a href="http://submission-judge.westus2.cloudapp.azure.com:8080/">**here**</a>. Feel free to mess around with it. If you're feeling up for a challenge, see if you can solve the 2nd sample problem.

The security of this system should be pretty good, so if you want to try and break the judge system, go ahead! (Of course, if you do manage to break it, please file an issue so I can fix it. Thanks!)

## Usage
The built docker image can be found as a package on Github (located <a href="https://github.com/Giantpizzahead/submission-judge/packages/318563">**here**</a>). To play around with the submission judge, first <a href="https://docs.docker.com/get-docker/">install docker</a>, then <a href="https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages#authenticating-to-github-packages">authenticate to GitHub Packages</a> with Docker. Next, run:
```commandline
sudo swapoff -a
```
The above command disables memory swapping on the machine, which is required for the submission judge to enforce memory limits. By default, this does not persist across reboots.

Finally, run the below command to start the Docker container (Warning: The container may temporarily change some settings to make sure judging is consistent; these changes also do not persist across reboots):
```commandline
docker run -p 8080:8080 --privileged docker.pkg.github.com/giantpizzahead/submission-judge/submission-judge:latest
```
(Yes, it requires privileged mode. rip <a href="https://github.com/ioi/isolate">isolate</a>)

The web server listens on **port 8080**. You'll be greeted by a testing interface to mess around with.

Andddd that's it! Go get that AC! (Legends say there might even be an AC\* verdict...)

## Licensing
nah who cares about licensing, just use it! :D

(This repo uses the MIT License)

&copy;Corona 2020 Giantpizzahead
# JudgeLite

<a href="https://github.com/Giantpizzahead/judgelite/actions?query=workflow%3Abuild"><img alt="build" src="https://github.com/Giantpizzahead/judgelite/workflows/build/badge.svg" /></a>
<a href="https://codecov.io/gh/Giantpizzahead/judgelite"><img alt="codecov" src="https://codecov.io/gh/Giantpizzahead/judgelite/branch/master/graph/badge.svg" /></a>
<a href="https://github.com/Giantpizzahead/judgelite/.github/LICENSE"><img alt="license" src="https://img.shields.io/github/license/giantpizzahead/judgelite" /></a>

A simple, easy to setup judge for checking the correctness of code.

## Features

* **Multiple test cases and subtasks!**
* Supports interactive problems (WIP)
* Variety of scoring methods (all-or-nothing, partials, bonus tests, etc.)
* Supports submissions in C++, Java, and Python 3
* Secure code compilation and execution using <a href="https://github.com/ioi/isolate">isolate</a>
* Extremely simple setup
* Easy-to-use submission API
* Simple administrative web interface (WIP)

## Demo

**http://judgelite.westus2.cloudapp.azure.com:8080**

A live version of JudgeLite can be found at the above link. Feel free to mess around with it. If you're feeling up for a challenge, see if you can solve the 2nd sample problem.

## Setup

Easy setup instructions for Linux and Windows are on <a href="https://github.com/Giantpizzahead/judgelite/wiki/Setup-Instructions">this wiki page</a>.

The web server listens on **port 8080**. You'll be greeted by a testing interface to mess around with.

Andddd that's it! Go get that AC! :)

## Usage

JudgeLite is designed to be used for competitive programming problems, although anything requiring automatic testing of source code will work.

A more detailed, easier to understand Wiki page explaining how to use the submission API will be created later. For now, here is a quick summary:

To submit code, send a **POST** request to **/api/submit** with the content type **multipart/form-data**. Include the following fields:

* **problem_id** - The ID of the problem that you're submitting a solution for.
* **type** - The language that your code is written in. Should be one of "java", "cpp", or "python".
* **code** - The file containing the submission's source code. Note that this is an actual *file*, not just a string representing the source code.

The returned status code will be 200 if the submission was successful, and 400 if the submission failed. The returned repsonse will be JSON. It will have exactly two of the following entries:

* **status** - Reports on whether or not the submission was successful. Will be either "success" or "fail".
* **error** - Only present if the submission failed. Presents a user-friendly message of why the submission failed.
* **job_id** - Only present if the submission succeeded. This is the unique ID value that was assigned to the submission. It is used to query the status of the submission (see below).

To get the status of a submission, send a **GET** request to **/api/status/<job_id>**, where job_id is the ID returned from the initial POST request. The returned status code will be 202 if the submission is still processing, and 200 if the submission has been fully evaluated (or an error occurred). The returned response will be JSON. It will have the following entries:

* **status** - Contains the status of the submission. Will be one of "queued", "judging", "done", "compile_error", or "internal_error".

The following entries may or may not be available, depending on the returned status:

* **score** - (judging, done) The current score of the submission for each subtask.
* **is_bonus** - (judging, done) Whether or not each subtask is a bonus subtask (useful for changing how bonus verdicts are displayed).
* **subtasks** - (judging, done) A list of subtasks. Each subtask has a list of test cases. Each test case is an array with 3 values: ['verdict', time, memory].
* **max_score** - (judging, done, compile_error) The maximum score that a submission could get on a problem, excluding bonus points.
* **final_score** - (done, compile_error) The final score that a submission recieved.
* **error** - (compile_error, internal_error) If status is compiler_error, the error message that the compiler generated. Else, the error code for the internal error.

A sample submission form / results page can be found on the demo website. You can use the code there as an example on how to use the API.

In order to use the submission API, you would first need to create some problems to submit to! Unfortunately, JudgeLite is currently in alpha, so the web interface for making problems isn't finished yet. The file structure for creating problems is pretty easy to create though. If you want to try making some problems, take a look at the sample problems in the **sample_problem_info** directory.

## License

JudgeLite is distributed under the MIT License. This means that you are **free to customize anything you want**, as long as you distribute your work under the MIT License as well.

## Credits

This repository is maintained by <a href="https://github.com/Giantpizzahead">Giantpizzahead</a>.

Special thanks to <a href="https://github.com/frodakcin">frodakcin</a> for helping out with JudgeLite's design!
57 changes: 30 additions & 27 deletions TODOs.md
View file
Original file line number Diff line number Diff line change
@@ -1,27 +1,30 @@
## TODOs
- Organize & document the code
- Polish up the submission API (add a username & secret key)
- Make sure to include documentation for the API
- Add problem statement support (markdown) to the problem info
- https://benweet.github.io/stackedit.js/
- Use a config file instead of env vars
- Take a look at docker config (https://medium.com/better-programming/about-using-docker-config-e967d4a74b83)
- Combine debug env vars into a single DEBUG integer
- Add a way to send a POST request to some other web server when a submission is done being evaluated (for easier integration)
- Keep track of past submissions, and make a way to view all past submissions in a table
- Great way to get started with databases! :D
- Redis would be a great choice since it's already part of the application (https://redis.io/ for docs)
- Add customizable Redis database mount point (to persist data between Docker updates)
- Add documentation for this, along with problem info mount point

### Less important TODOs
- Make a front-end web interface to create and edit problems
- Make sure generated YAML file uses strings as subtask names
- Make convenience function to generate correct outputs using a program
- Have it check the program's output if the .out file already exists (to manually check for correctness)
- Allow custom graders (other than diff, as a standalone binary)
- Add support for partial scoring (fractional scores for each test case allowed, instead of just right or wrong)

#### Other stuff
- See if there's some way to automate the 'swapoff -a' command (maybe replace with swap accounting?)
- Disk quota is not enabled, only fsize is. That means that you could bypass the max file size limit by creating tons of small files, overloading the system. No one would do this by accident though, so it's not something that is too important to fix (plus, who would even think to try this?).
## TODOs
- Organize & document the code
- Modify submission form so that it reloads the page on a submit (to prevent weird JS issues)
- Polish up the submission API (add a username & secret key)
- Make sure to include documentation for the API
- Add problem statement support (markdown) to the problem info
- https://benweet.github.io/stackedit.js/
- Use a config file instead of env vars
- Take a look at docker config (https://medium.com/better-programming/about-using-docker-config-e967d4a74b83)
- Combine debug env vars into a single DEBUG integer
- Add a way to send a POST request to some other web server when a submission is done being evaluated (for easier integration)
- Keep track of past submissions, and make a way to view all past submissions in a table
- Great way to get started with databases! :D
- Redis would be a great choice since it's already part of the application (https://redis.io/ for docs)
- Add customizable Redis database mount point (to persist data between Docker updates)
- Add documentation for this, along with problem info mount point

### Less important TODOs
- Make a front-end web interface to create and edit problems
- Make sure generated YAML file uses strings as subtask names
- Make convenience function to generate correct outputs using a program
- Have it check the program's output if the .out file already exists (to manually check for correctness)
- Allow custom graders (other than diff, as a standalone binary)
- Allow custom graders for interactive problems!
- Maybe make two modes: One where your program is directly linked to the submitted code via stdin and stdout, and one where your program's stdout is used (to allow for completely custom problem setup)
- Add support for partial scoring (fractional scores for each test case allowed, instead of just right or wrong)

#### Other stuff
- See if there's some way to automate the 'swapoff -a' command (maybe replace with swap accounting?)
- Disk quota is not enabled, only fsize is. That means that you could bypass the max file size limit by creating tons of small files, overloading the system. No one would do this by accident though, so it's not something that is too important to fix (plus, who would even think to try this?).
Loading

0 comments on commit dfca98b

Please sign in to comment.