Grading automation script for CptS 355

readme.md

Grading helper

Basic use

1. Make a new folder.
2. Into this folder, move/copy the following:
   • helper.ps1
   • submissions.zip (from Canvas)
   • Your grading program files, probably written using a unit test framework. This example will use check.py
3. Open a PowerShell terminal in this folder.
4. Run this command: helper.ps1 check.py

Advanced use

The script has some more advanced features made available with some command-line switches. Run Get-Help helper.ps1 -Detailed for more on those.

Testing Python with VS Code

VS Code has a testing interface built in. The Python extension provides a connection between it and any Python unit tests in your workspace.

You can install the Python extension by searching for ms-python.python in the Extensions sidebar panel.

To enable unittest unit tests for VS Code and correctly configure them for the workspace managed by the script, place this in your .vscode/settings.json file:

{
	"python.testing.unittestArgs": ["-v", "-s", "./workspace", "-p", "*_test.py"],
	"python.testing.unittestEnabled": true
}

and rename your grading test file to end with _test.py. (Or change the last argument in the python.testing.unittestArgs setting.)

Adding support for another language

Right now, the script has support for Python and Haskell.

To add support for other languages, you need to know (1) what the file extension is for that language, and (2) what the executable is that will do anything it needs to to run that file.

For example, Python has the .py file extension and the python executable. Haskell has .hs and runghc.

Knowing that, find the $program variable declaration in the script and add a line to that block, using the established pattern. (The -match operator is a regex check; PowerShell uses ` as an escape character.)

helper.ps1

#Requires -Version 7.0

<#
    Created by Duncan Gibson. This script is shared under the terms of the
    CC Attribution-NonCommercial-ShareAlike 4.0 International license, which
    may be found here: https://creativecommons.org/licenses/by-nc-sa/4.0/
#>

<#
    The PowerShell help system can be invoked to show you how to use this
    script by running `Get-Help helper.ps1`
#>

<#
.SYNOPSIS
    This script was designed to help automate the grading of homework
    assignments for Dr. Sakire Arslan Ay's CptS 355 class.

    It expects there to be a "submissions.zip" file from Canvas in the working
    directory. You can acquire one of these files by going to the assignment in
    Canvas and clicking "Download Submissions" in the panel on the right.

    It also expects you to have a file of unit tests that check your assignment.

.DESCRIPTION
    This script will unzip the submissions.zip file and group the contents by
    student. Then it will copy each student's files to a subdirectory named
    "workspace", one student at a time. Submitted .zip files are a special
    case; they will be unzipped and their contents added to the workspace.

    The test file and other provided files will likewise be copied to the
    workspace.

    Once all files for a student are in place, it will run the test file,
    display its outputs, and pause. If you need to re-run the tests for a
    student for some reason, open another terminal and run the test file in the
    workspace. Workspace files are safe to edit without affecting other
    students.

    The workspace is reset before the next student's submission is loaded.

    WARNING: This script does not make guarantees about the safety of submitted
    files. If you're feeling paranoid, you should probably use a VM.

.NOTES
    This script is currently configured to run Haskell and Python programs.
    More languages can be supported by adding a case for them in the
    configuration section of the script, right below the parameter declaration.

.LINK
    https://gist.github.com/legowerewolf/f34835c07acb06527358155e9ca66c4c
#>

[CmdletBinding()]
param (
    [Parameter(Mandatory)]
    [string]$TestFileName,

    [Parameter()]
    <#
    A list of files that we provide that should be in the workspace for
    tests to run.
    #>
    [string[]]$TesterProvidedFiles = @(),

    [Parameter()]
    <#
    A list of files that we expect to be in the submission. If a given file
    isn't detected, the grader will be prompted to check the workspace and
    correct the filename.

    This list should, at minimum, include files that are depended on by the
    main test file.
    #>
    [string[]]$ExpectedFiles = @()
)

#################### START CONFIGURATION ###################

<#
This specifies the program that gets run for the selected test file. You
shouldn't have to edit this unless Sakire uses a language other than Python or
Haskell.
#>
$program = {
    if ($TestFileName -match ".*`.hs") { return "runghc" }
    if ($TestFileName -match ".*`.py") { return "python" }
}.Invoke()

##################### END CONFIGURATION ####################

# helper function for the script's interface
function Start-Pause {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory)]
        [ValidateLength(1, 1)]
        [string]$ResumeChar,

        [Parameter(Mandatory)]
        [string]$Prompt
    )

    Write-Output $Prompt
    Start-Sleep -s 1
    $Host.UI.RawUI.FlushInputBuffer()
    while ($Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown').Character -notlike $ResumeChar) {}
}

# real start of the script

# verify and fix configuration
if ($TesterProvidedFiles.GetType().FullName -eq "System.String") { $TesterProvidedFiles = , $TesterProvidedFiles }
if ($ExpectedFiles.GetType().FullName -eq "System.String") {
    $ExpectedFiles = , $ExpectedFiles
}

# purge existing files
Remove-Item "./submissions/" -Recurse -ErrorAction SilentlyContinue
Remove-Item "./workspace/" -Recurse -ErrorAction SilentlyContinue

# prep files and folder structure
Expand-Archive "./submissions.zip" -DestinationPath "./submissions/"

# group files by student
$submissiongroups = Get-ChildItem "./submissions/" | Where-Object Name -NotLike "*test*" | ForEach-Object -Begin { $groups = @{} } -Process {
    $key = $_.Name.Split("_")[0]

    if ($groups.ContainsKey($key)) {
        $groups[$key] += $_
    }
    else {
        $groups.Add($key, @($_))
    }
} -End { $groups }

# loop over each student
foreach ($submission in ($submissiongroups.GetEnumerator() | Sort-Object -Property Name)) {
    # reset the workspace
    Remove-Item "./workspace/" -Recurse -Force -ErrorAction SilentlyContinue
    New-Item -Path "./workspace" -ItemType Directory >$null

    # copy our provided files to the workspace
    foreach ($file in $TesterProvidedFiles) {
        Copy-Item $file -Destination "./workspace/"
    }

    # copy the student's files to the workspace
    foreach ($file in $submission.Value) {
        $destname = $file.Name.Split("_")[-1]

        $destname = $destname -replace "-\d+\.", "."

        Copy-Item $file -Destination "./workspace/$destname"

        if ($destname -like "*.zip") {
            Expand-Archive "./workspace/$destname" -DestinationPath "./workspace/zipped/"

            $zip_contents = Get-ChildItem "./workspace/zipped/"

            if ($zip_contents.count -eq 1 -and $zip_contents[0].Mode -like "d*") {
                $zip_contents = $zip_contents[0] | Get-ChildItem
            }

            $zip_contents | ForEach-Object { Move-Item -Path $_.FullName -Destination ".\workspace\" }

            Remove-Item -Recurse .\workspace\zipped\
            Remove-Item "./workspace/$destname"
        }
    }

    # check that the submission has the expected filenames
    foreach ($expected_file_name in $ExpectedFiles) {
        if (-not (Test-Path "./workspace/$expected_file_name")) {
            Start-Pause -ResumeChar 't' -Prompt "Warning! No $expected_file_name detected! Correct this, then press [t] to continue."
        }
    }

    # copy the test file to the workspace
    Copy-Item $TestFileName -Destination "./workspace/"

    Write-Output "Loaded files for $($submission.Key)."

    # set current working directory to the workspace
    Push-Location
    Set-Location "./workspace"

    # run the test file
    Invoke-Expression "$program $TestFileName"

    # return to the original directory
    Pop-Location

    Write-Output "Above results are for $($submission.Key)."

    Start-Pause -ResumeChar 't' -Prompt (("=" * 20) + " Press [t] to continue to next student. " + ("=" * 20))
}