README.md
1Testing NASM
2============
3We use [Travis CI](https://travis-ci.org/) service to execute NASM tests,
4which basically prepares the environment and runs our `nasm-t.py` script.
5
6The script scans a testing directory for `*.json` test descriptor files
7and runs test by descriptor content.
8
9Test engine
10-----------
11`nasm-t.py` script is a simple test engine written by Python3 language
12which allows either execute a single test or run them all in a sequence.
13
14A typical test case processed by the following steps:
15
16 - a test descriptor get parsed to figure out which arguments
17 are to be provided into the NASM command line;
18 - invoke the NASM with arguments;
19 - compare generated files with precompiled templates.
20
21`nasm-t.py` supports the following commands:
22
23 - `list`: to list all test cases
24 - `run`: to run test cases
25 - `update`: to update precompiled templates
26
27Use `nasm-t.py -h` command to get the detailed description of every option.
28
29Test descriptor file
30--------------------
31A descriptor file should provide enough information how to run the NASM
32itself and which output files or streams to compare with predefined ones.
33We use `JSON` format with the following fields:
34
35 - `description`: a short description of a test which is shown to
36 a user when tests are listed;
37 - `id`: descriptor internal name to use with `ref` field;
38 - `ref`: a reference to `id` from where settings should be
39 copied, it is convenient when say only `option` is different
40 while the rest of the fields are the same;
41 - `format`: NASM output format to use (`bin`,`elf` and etc);
42 - `source`: is a source file name to compile, this file must
43 be shipped together with descriptor file itself;
44 - `option`: an additional option passed to the command line;
45 - `update`: a trigger to skip updating targets when running
46 an update procedure;
47 - `target`: an array of targets which the test engine should
48 check once compilation finished:
49 - `stderr`: a file containing *stderr* stream output to check;
50 - `stdout`: a file containing *stdout* stream output to check;
51 - `output`: a file containing compiled result to check, in other
52 words it is a name passed as `-o` option to the compiler.
53
54Examples
55--------
56
57A simple test where no additional options are used, simply compile
58`absolute.asm` file with `bin` format for output, then compare
59produced `absolute.bin` file with precompiled `absolute.bin.dest`.
60
61```json
62{
63 "description": "Check absolute addressing",
64 "format": "bin",
65 "source": "absolute.asm",
66 "target": [
67 { "output": "absolute.bin" }
68 ]
69}
70```
71
72A slightly complex example: compile one source file with different optimization
73options and all results must be the same. To not write three descriptors
74we assign `id` to the first one and use `ref` term to copy settings.
75Also, it is expected that `stderr` stream will not be empty but carry some
76warnings to compare.
77
78```json
79[
80 {
81 "description": "Check 64-bit addressing (-Ox)",
82 "id": "addr64x",
83 "format": "bin",
84 "source": "addr64x.asm",
85 "option": "-Ox",
86 "target": [
87 { "output": "addr64x.bin" },
88 { "stderr": "addr64x.stderr" }
89 ]
90 },
91 {
92 "description": "Check 64-bit addressing (-O1)",
93 "ref": "addr64x",
94 "option": "-O1",
95 "update": "false"
96 },
97 {
98 "description": "Check 64-bit addressing (-O0)",
99 "ref": "addr64x",
100 "option": "-O0",
101 "update": "false"
102 }
103]
104```
105