1# Guide to YAML::Tiny testing
2
3YAML::Tiny tests use several components:
4
5* .t files
6* Test libraries in t/lib
7* YAML data files in t/data
8* TestML data files in t/tml-*
9
10The use of each is described below.
11
12## .t files
13
14The .t files are standard Perl test files.  They may use one or more of the
15test libraries in t/lib.  They may only use modules available in Perl 5.8.1 or
16later (and not subsequently deprecated), but they may use newer non-XS versions
17of those modules as necessary to avoid known bugs.
18
19Some .t files have complete inputs/outputs for their tests.  Others iterate
20over .tml files in the t/tml-* directories.
21
22A .t file should load Test::More and use `done_testing` at the end
23to so that new tests may be added without needing to also update a test plan.
24
25Currently, the convention is to name .t files matching the pattern
26qr/^\d\d_\w+\.t$/
27
28## Test libraries
29
30There are currently three test libraries in t/lib.  A .t file that uses one or
31more of them should put `use lib 't/lib';` at the top of the .t file.  Test
32libraries can assume that if they were loaded, that 't/lib' is already in @INC.
33
34The test libraries are:
35
36* SubtestCompat
37* TestML::Tiny
38* TestBridge
39* TestUtils
40
41The "SubtestCompat" library provides a limited emulation of
42Test::More::subtest that is reasonably compatible back to 0.88.  If using
43subtests, you must not set a plan in the subtest and you must use
44done_testing in the *.t file.
45
46The TestML::Tiny library contains functions for parsing and executing TestML
47tests with callbacks.  TestML is a data-driven testing language; TestML::Tiny
48implements a small subset of its features. See the section on TestML, below,
49for an example. Generally, bugs should be patched upstream on CPAN and then
50a new Test::Tiny CPAN release can be copied here and pod-stripped.
51
52The TestBridge library contains testing functions for use in .t files or to
53be passed to TestML::Tiny functions as callbacks.  Test functions should not
54include `done_testing`.  They should use `subtest` for any repetitive testing
55that loops over test cases.  Callback should check for the expected test
56points (see below) and skip a TML block if those points are not available.
57
58The TestUtils library contains utility functions.  Testing functions should
59not be added here (i.e. nothing that uses Test::More).
60
61## YAML data files in t/data
62
63Files in the t/data directory are intended to test how YAML files are loaded
64and decoded and typically need some custom test code to load the file and see
65if the result matches expectations (successful or not).
66
67If a real-world YAML file cannot be loaded due to character set encoding
68issues, it should be placed in this directory for testing.  If a real-world
69YAML file is ASCII or UTF-8 encoded, can be decoded successfully, but has
70problems in parsing, it should be reduced to the smallest sample of YAML that
71demonstrates the parsing problem and added to a .tml file for testing.  See
72below for more details.
73
74## TestML quick intro
75
76TestML data files are UTF-8 encoded files with a .tml suffix that contain one
77or more test "blocks".  Each block has a test label, and one or more 'test
78points', usually representing input and expected output, and possibly
79additional annotations or flags.
80
81Here is an example of a .tml file with a single block:
82
83    # This is a TestML block:    (this line is a comment)
84    === This is the test label
85    Lines until the first point are ignored
86
87    # This is a "block" style point. All non-comment lines until next point
88    # are the data for the 'yaml' point. The data ends with newline, and
89    # trailing blank lines are trimmed.
90    --- yaml
91    ---
92    foo: bar
93    # a comment
94    \# not a comment
95
96    # This is the second point; "inline" style. The data after the colon goes
97    # to end of line. Leading/trailing whitespace is trimmed.
98    --- perl: [ { foo => 'bar' } ]
99
100    # This is a point whose value is the empty string
101    --- a_flag
102
103    # This is the next block:
104    === Another test case
105
106The test label is provided on a line beginning with qr/^===/.  Test "points"
107are provided in sections beginning with qr/^--- +\w+/.  All flush-left comment
108lines are stripped.  Lines beginning with '\' are escaped.
109
110Different tests expect different test points in a .tml file, based on the
111specific test callback being used.
112
113Many .tml files have the points 'yaml' and 'perl' as in the example above.  The
114'yaml' point is a YAML document and the 'perl' point is a Data::Dumper Perl
115data structure.  The test checks whether the YAML parses into a data structure
116identical to the Perl one.  The 'a_flag' point is an annotation that the
117testing callback can use to affect the run of a given test.
118
119The semantics of points (including annotations) is specific to the callback
120functions used to process test blocks.
121
122# TestML data files in t/tml-*
123
124TestML data files are organized into three directories:
125
126* t/tml-spec — these test files are provided by the YAML spec maintainers and
127should not be modified except to skip testing features that YAML::Tiny does not
128support
129
130* t/tml-local — these test files are YAML::Tiny's own unit tests; generally new
131test cases for coverage or correctness should be added here; these are
132broken into subdirectories, described later
133
134* t/tml-world — these test files represent "real world" YAML and their
135corresponding expected Perl output
136
137Generally, if a "real world" problem can be isolated to a particular snippet of
138YAML, it's best to add it to a t/tml-local file (or create a new one).  If the
139problem can only be seen in the context of the entire YAML document, include it
140in t/tml-world.  If the problem relates to encoding, it should be put into
141t/data instead.
142
143# t/tml-local subdirectories
144
145The subdirectories in t/tml-local define four types of tests:
146
147* perl-to-yaml: test that perl data dump to an expected YAML string
148
149* yaml-roundtrip: test that a YAML string loads to an expected perl data
150  structure; also tests that the perl data can be dumped and loaded back;
151
152* dump-error: test that certain perl data trigger expected errors
153
154* load-error: test that certain YAML strings trigger expected errors
155
156All .tml files in a t/tml-local directory must have the TestML
157test points required by the corresponding test functions defined
158in the TestBridge library.
159
160Generally, files should be grouped by data type or feature so that
161related tests are kept together.
162