1# Guide to CPAN::Meta::YAML testing
2
3CPAN::Meta::YAML tests use several components:
4
5* .t files
6* Test libraries in t/lib
7* YAML data files in t/data
8* TestML data files in t/tml-*
9
10The use of each is described below.
11
12## .t files
13
14The .t files are standard Perl test files.  They may use one or more of the
15test libraries in t/lib.  They may only use modules available in Perl 5.8.1 or
16later (and not subsequently deprecated), but they may use newer non-XS versions
17of those modules as necessary to avoid known bugs.
18
19Some .t files have complete inputs/outputs for their tests.  Others iterate
20over .tml files in the t/tml-* directories.
21
22A .t file should load Test::More and either use `done_testing` or provide a
23test plan.  If tests iterate over external data, the use of `done_testing` is
24preferred so that external data can be updated with new tests without needing
25to also update a test plan.
26
27Currently, the convention is to name .t files matching the pattern
28qr/^\d\d_\w+\.t$/
29
30## Test libraries
31
32There are currently three test libraries in t/lib.  A .t file that uses one or
33more of them should put `use lib 't/lib';` at the top of the .t file.  Test
34libraries can assume that if they were loaded, that 't/lib' is already in @INC.
35
36The test libraries are:
37
38* TestML::Tiny
39* TestBridge
40* TestUtils
41
42The TestML::Tiny library contains functions for parsing and executing TestML
43tests with callbacks.  TestML is a data-driven testing language; TestML::Tiny
44implements a small subset of its features. See the section on TestML, below,
45for an example. Generally, bugs should be patched upstream on CPAN and then
46a new Test::Tiny CPAN release can be copied here and pod-stripped.
47
48The TestBridge library contains testing functions for use in .t files or to
49be passed to TestML::Tiny functions as callbacks.  Test functions should not
50include `done_testing`.  They should use `subtest` for any repetitive testing
51that loops over test cases.  Callback should check for the expected test
52points (see below) and skip a TML block if those points are not available.
53
54The TestUtils library contains utility functions.  Testing functions should
55not be added here (i.e. nothing that uses Test::More).
56
57## YAML data files in t/data
58
59Files in the t/data directory are intended to test how YAML files are loaded
60and decoded and typically need some custom test code to load the file and see
61if the result matches expectations (successful or not).
62
63If a real-world YAML file cannot be loaded due to character set encoding
64issues, it should be placed in this directory for testing.  If a real-world
65YAML file is ASCII or UTF-8 encoded, can be decoded successfully, but has
66problems in parsing, it should be reduced to the smallest sample of YAML that
67demonstrates the parsing problem and added to a .tml file for testing.  See
68below for more details.
69
70## TestML quick intro
71
72TestML data files are UTF-8 encoded files with a .tml suffix that contain one
73or more test "blocks".  Each block has a test label, and one or more 'test
74points', usually representing input and expected output, and possibly
75additional annotations or flags.
76
77Here is an example of a .tml file with a single block:
78
79    # This is a TestML block:    (this line is a comment)
80    === This is the test label
81    Lines until the first point are ignored
82
83    # This is a "block" style point. All non-comment lines until next point
84    # are the data for the 'yaml' point. The data ends with newline, and
85    # trailing blank lines are trimmed.
86    --- yaml
87    ---
88    foo: bar
89    # a comment
90    \# not a comment
91
92    # This is the second point; "inline" style. The data after the colon goes
93    # to end of line. Leading/trailing whitespace is trimmed.
94    --- perl: [ { foo => 'bar' } ]
95
96    # This is a point whose value is the empty string
97    --- a_flag
98
99    # This is the next block:
100    === Another test case
101
102The test label is provided on a line beginning with qr/^===/.  Test "points"
103are provided in sections beginning with qr/^--- +\w+/.  All flush-left comment
104lines are stripped.  Lines beginning with '\' are escaped.
105
106Different tests expect different test points in a .tml file, based on the
107specific test callback being used.
108
109Many .tml files have the points 'yaml' and 'perl' as in the example above.  The
110'yaml' point is a YAML document and the 'perl' point is a Data::Dumper Perl
111data structure.  The test checks whether the YAML parses into a data structure
112identical to the Perl one.  The 'a_flag' point is an annotation that the
113testing callback can use to affect the run of a given test.
114
115The semantics of points (including annotations) is specific to the callback
116functions used to process test blocks.
117
118# TestML data files in t/tml-*
119
120TestML data files are organized into three directories:
121
122* t/tml-spec — these test files are provided by the YAML spec maintainers and
123should not be modified except to skip testing features that CPAN::Meta::YAML does not
124support
125
126* t/tml-local — these test files are CPAN::Meta::YAML's own unit tests; generally new
127test cases for coverage or correctness should be added here; these are
128broken into subdirectories, described later
129
130* t/tml-world — these test files represent "real world" YAML and their
131corresponding expected Perl output
132
133Generally, if a "real world" problem can be isolated to a particular snippet of
134YAML, it's best to add it to a t/tml-local file (or create a new one).  If the
135problem can only be seen in the context of the entire YAML document, include it
136in t/tml-world.  If the problem relates to encoding, it should be put into
137t/data instead.
138
139# t/tml-local subdirectories
140
141The subdirectories in t/tml-local define four types of tests:
142
143* perl-to-yaml: test that perl data dump to an expected YAML string
144
145* yaml-roundtrip: test that a YAML string loads to an expected perl data
146  structure; also tests that the perl data can be dumped and loaded back;
147
148* dump-error: test that certain perl data trigger expected errors
149
150* load-error: test that certain YAML strings trigger expected errors
151
152All .tml files in a t/tml-local directory must have the TestML
153test points required by the corresponding test functions defined
154in the TestBridge library.
155
156Generally, files should be grouped by data type or feature so that
157related tests are kept together.
158