Testing solutions of stepic.org programming assignments

stepic.org testing scripts

Please read StepicTest.md carefully)

_clean.bat

@echo off

del test.exe
del test.log
del *.out
del *.err
https://gist.github.com/in4lio/413f4836beabd6ae3c6f#file-test-cpp

_test.bat

@echo off
SETLOCAL ENABLEDELAYEDEXPANSION

set /a failed=0

if exist test.py (
	set test=python -u test.py
) else (
	set test=test.exe
)
echo %date% %time%

echo %date% %time% >> test.log
echo.>>test.log

for %%i in (*.in) do (
	echo test ^< %%~nxi
	%test% < %%~nxi > %%~ni.out 2>%%~ni.err
	>nul findstr "^" %%~ni.err || del %%~ni.err
	fc /L %%~ni.out %%~ni.ok >> test.log 2>&1
	if !errorlevel! equ 0 (
		echo OK
	) else (
		echo ***** FAILED *****
		set /a failed=failed+1
	)
)
if %failed%==0 (
	echo DONE
) else (
	echo DONE, %failed% failed
)

echo.
echo ---------------------------------------- >> test.log

pause

StepicTest.md

How to use Windows batch script for testing solutions of stepic.org programming assignments

Version 0.5

I. Put the source code of the solution into test.cpp:

#include <iostream>
using namespace std;

int main()
{
	/* put your code here */
	int a, b;
	cin >> a >> b;
	if (a == 2) a = 3; /* bug! */
	cerr << "a = " << a << " b = " << b << endl; /* debug message */
	cout << (a + b);

	return 0;
}

or into test.py:

from __future__ import division
from math import *
import sys

inp = sys.stdin.read().split()
a = int(inp.pop(0))
b = int(inp.pop(0))

if a == 2: a = 3  # bug!
sys.stderr.write("a = {0} b = {1}\n".format(a, b))  # debug message
sys.stdout.write(str(a + b))

II. Create the files of input data *.in and the corresponding files of correct output data *.ok:

01.in:

1 2

01.ok:

3

02_fail.in:

2 2

02_fail.ok:

4

III. Compile test.cpp, if you're writing in C++, e.g.

g++ test.cpp -o test.exe

IV. Run _test.bat. Your solution will be applied to all files of input data and the results will be compared with correct output data:

15.03.2015 14:59:59,59
test < 01.in
OK
test < 02_fail.in
***** FAILED *****
DONE, 1 failed

V. See test.log (cp866) for details:

16.03.2015 18:55:13,43

Сравнение файлов 01.out и 01.OK
FC: различия не найдены

Сравнение файлов 02_fail.out и 02_FAIL.OK
***** 02_fail.out
5
***** 02_FAIL.OK
4
*****

----------------------------------------

VI. See debug messages in *.err files.

P.S.
If you get stuck and can't solve the problem without seeing the original test data, use the method shown into the last script in the gist.

test.cpp

#include <iostream>
using namespace std;

int main()
{
	/* put your code here */
	int a, b;
	cin >> a >> b;
	if (a == 2) a = 3; /* bug! */
	cerr << "a = " << a << " b = " << b << endl; /* debug message */
	cout << (a + b);

	return 0;
}

test.py

from __future__ import division
from math import *
import sys

inp = sys.stdin.read().split()
a = int(inp.pop(0))
b = int(inp.pop(0))

if a == 2: a = 3  # bug!
sys.stderr.write("a = {0} b = {1}\n".format(a, b))  # debug message
sys.stdout.write(str(a + b))

test_leak.py

from __future__ import division
from math import *
import sys

TEST = frozenset([
"4 2 1 2 3 2",
# ...

])

# input the test data
inp = sys.stdin.read().split()
inpst = ' '.join( inp )

# look for a new test data
if inpst not in TEST:
    sys.stderr.write( '"{}"\n'.format( inpst ))  # manually copy the test data from Feedback into TEST set
    raise Exception( "New test #{}".format( len( TEST ) + 1 ))

# parse input
a = int( inp.pop( 0 ))
b = int( inp.pop( 0 ))
# ...

# solve the problem...


# output the result
sys.stdout.write( "NONE" )