Create your own self-marking python tests

Tests.json expect reference guide

Each test that you write in your tests.json file is defined by a new object in the expect array:

Example 3: process property defines how data is tested

These are the properties that are currently supported for each test item in the expect array:

search:

The search property defines where the test will look to find what is looking for. It can have the following string values:

file: searches to see if a file exists, or if you specify a name property it will search the contents of that file

code: searches the content of the default file (mycode.py)

errors: searches through any error messages

output: searches through any program output

input: searches through any prompts displayed when the user is asked to input a value

globals: searches to see if a global variable exists, or if you specify a name property it will search the contents of that variable

Defining where to search
Example 4: searching for a file

Line 4 and 5 of example 4 (above) means search through the list of files to see if there’s one called text.txt

name:

The name property is only useful if the search property is either file or globals. It is used to specify the name of the file or global variable to be tested.

Example 5: searching file contents

Lines 4-6 of example 5 (above) means search through the file called text.txt and check if it contains just the text Hello

for:

The for property defines what the test will search for (within the data specified by the search property)

The for property can be an integer, float or boolean value but is usually a string which specifies what the test is looking for

Defining what to look for

match:

The match property defines the way the test will compare what it finds with what is specified by the for property. It can be any of the following values:

=: data specified by search property (when the test runs) must exactly match the data specified by the for property (when the test is defined).

>: data specified by search property must be greater than the data specified by the for property.

<: data specified by search property must be less than the data specified by the for property.

>=: data specified by search property must be greater than or equal to the data specified by the for property.

<=: data specified by search property must be less than or equal to the data specified by the for property.

contains: data specified by search property include the data specified by the for property.

Defining how to look

process:

The process property allows the data specified by the search property to be processed before it is compared with what you specify in the for property. If defined, the process property must be an array that can include any of the following strings:

ignoreblanks: any empty lines are ignored

count: the number of characters / lines / files is tested rather than the data itself. This is the same as length (see below)

length: as above, the number of characters / lines / files is tested rather than the data itself

lower: the data is converted to lowercase

upper: the data is converted to uppercase

lines: the data is split into separate lines (useful for combining with count to count the number of lines)

Defining how to process data before testing
Example 6: counting the number of error messages

Example 6 (above) shows how to use the process property to count the number of error messages so that the test will pass if there are no syntax or runtime errors.

description:

The description property lets you customise how each test is displayed in the test table.

Example 7: customising the test description

The test in Example 7 (above) behaves in exactly the same as example 6 but line 7 defines the text which describes that test in the testing table.

Multiple tests

You can define as many tests as you like by creating each one as a new item in the expect array in the tests.json file:

Example 8: defining multiple tests

Example 8 (above) defines three separate tests.

Test 1 (lines 3-8) counts all of the error messages and checks if there are none of them.

Test 2 (lines 10-13) searches the list of files to see if a file called output.txt exists

Test 3 (lines 15-19) searches the file called output.txt to see if it contains just the text ABCDEFH

The next page is a reference guide for how to use the provide array in tests.json in order to provide input data to each test as it runs.