1This README file describes the files and directories related       -*- rst -*-
2to the Python test suite under the current 'test' directory.
3
4- dotest.py
5
6  Provides the test driver for the test suite.  To invoke it, cd to the 'test'
7  directory and issue the './dotest.py' command or './dotest.py -v' for more
8  verbose output.  '.dotest.py -h' prints out the help messge.
9
10  A specific naming pattern is followed by the .py script under the 'test'
11  directory in order to be recognized by 'dotest.py' test driver as a module
12  which implements a test case, namely, Test*.py.
13
14  Some example usages:
15
16  1. ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log0
17     This runs the test suite and directs the run log to a file.
18
19  2. LLDB_LOG=/tmp/lldb.log GDB_REMOTE_LOG=/tmp/gdb-remote.log ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log
20     This runs the test suite, with logging turned on for the lldb as well as
21     the process.gdb-remote channels and directs the run log to a file.
22
23- lldbtest.py
24
25  Provides an abstract base class of lldb test case named 'TestBase', which in
26  turn inherits from Python's unittest.TestCase.  The concrete subclass can
27  override lldbtest.TestBase in order to inherit the common behavior for
28  unittest.TestCase.setUp/tearDown implemented in this file.
29
30  To provide a test case, the concrete subclass provides methods whose names
31  start with the letters test.  For more details about the Python's unittest
32  framework, go to http://docs.python.org/library/unittest.html.
33
34  ./command_source/TestCommandSource.py provides a simple example of test case
35  which overrides lldbtest.TestBase to exercise the lldb's 'command source'
36  command.
37
38  The doc string provides more details about the setup required for running a
39  test case on its own.  To run the whole test suite, 'dotest.py' is all you
40  need to do.
41
42- subdirectories of 'test'
43
44  Most of them predate the introduction of the python test suite and contain
45  example C/C++/ObjC source files which get compiled into executables which are
46  to be exercised by the debugger.
47
48  For such subdirectory which has an associated Test*.py file, it was added as
49  part of the Python-based test suite to test lldb functionality.
50
51  Some of the subdirectories, for example, the 'help' subdirectory, do not have
52  C/C++/ObjC source files; they were created to house the Python test case which
53  does not involve lldb reading in an executable file at all.
54
55  The sample_test directory contains examples of both a full and an "inline"
56  testcase that run a process to a breakpoint and check a local variable.  These
57  are convenient starting points for adding new tests.
58
59- make directory
60
61  Contains Makefile.rules, which can be utilized by test cases to write Makefile
62  based rules to build binaries for the inferiors.
63
64  By default, the built executable name is a.out, which can be overwritten by
65  specifying your EXE make variable, via the Makefile under the specific test
66  directory or via supplying a Python dictionary to the build method in your
67  Python test script.  An example of the latter can be found in
68  test/lang/objc/radar-9691614/TestObjCMethodReturningBOOL.py, where:
69
70    def test_method_ret_BOOL(self):
71        """Test that objective-c method returning BOOL works correctly."""
72        d = {'EXE': self.exe_name}
73        self.build(dictionary=d)
74        self.setTearDownCleanup(dictionary=d)
75        ...
76
77    def setUp(self):
78        # Call super's setUp().
79        TestBase.setUp(self)
80        # We'll use the test method name as the exe_name.
81        self.exe_name = self.testMethodName
82        # Find the line number to break inside main().
83        self.main_source = "main.m"
84        self.line = line_number(self.main_source, '// Set breakpoint here.')
85
86  The exe names for the two test methods are equal to the test method names and
87  are therefore guaranteed different.
88
89- plugins directory
90
91  Contains platform specific plugin to build binaries with dsym/dwarf debugging
92  info.  Other platform specific functionalities may be added in the future.
93
94- unittest2 directory
95
96  Many new features were added to unittest in Python 2.7, including test
97  discovery. unittest2 allows you to use these features with earlier versions of
98  Python.
99
100  It currently has unittest2 0.5.1 from http://pypi.python.org/pypi/unittest2.
101  Version 0.5.1 of unittest2 has feature parity with unittest in Python 2.7
102  final. If you want to ensure that your tests run identically under unittest2
103  and unittest in Python 2.7 you should use unittest2 0.5.1.
104
105  Later versions of unittest2 include changes in unittest made in Python 3.2 and
106  onwards after the release of Python 2.7.
107
108- Profiling dotest.py runs
109
110  I used the following command line thingy to do the profiling on a SnowLeopard
111  machine:
112
113    $ DOTEST_PROFILE=YES DOTEST_SCRIPT_DIR=/Volumes/data/lldb/svn/trunk/test /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/cProfile.py -o my.profile ./dotest.py -v -w 2> ~/Developer/Log/lldbtest.log
114
115  After that, I used the pstats.py module to browse the statistics:
116
117    $ python /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/pstats.py my.profile
118
119- Writing test cases:
120
121  We strongly prefer writing test cases using the SB API's rather than
122  the runCmd & expect.  Unless you are actually testing some feature
123  of the command line, please don't write command based tests.  For
124  historical reasons there are plenty of examples of tests in the test
125  suite that use runCmd where they shouldn't, but don't copy them,
126  copy the plenty that do use the SB API's instead.
127
128  The reason for this is that our policy is that we will maintain
129  compatibility with the SB API's.  But we don't make any similar
130  guarantee about the details of command result format.  If your test
131  is using the command line, it is going to have to check against the
132  command result text, and you either end up writing your check
133  pattern by checking as little as possible so you won't be exposed to
134  random changes in the text; in which case you can end up missing
135  some failure, or you test too much and it means irrelevant changes
136  break your tests.
137
138  However, if you use the Python API's it is possible to check all the
139  results you want to check in a very explicit way, which makes the
140  tests much more robust.
141
142  Even if you are testing that a command-line command does some
143  specific thing, it is still better in general to use the SB API's to
144  drive to the point where you want to run the test, then use
145  SBInterpreter::HandleCommand to run the command.  You get the full
146  result text from the command in the command return object, and all
147  the part where you are driving the debugger to the point you want to
148  test will be more robust.
149
150  The sample_test directory contains a standard and an "inline" test
151  that are good starting points for writing a new test.
152
153- Attaching in test cases:
154
155  If you need to attach to inferiors in your tests, you must make sure
156  the inferior calls lldb_enable_attach(), before the debugger
157  attempts to attach. This function performs any platform-specific
158  processing needed to enable attaching to this process (e.g., on
159  Linux, we execute prctl(PR_SET_TRACER) syscall to disable
160  protections present in some Linux systems).
161