1.. Licensed to the Apache Software Foundation (ASF) under one
2   or more contributor license agreements.  See the NOTICE file
3   distributed with this work for additional information
4   regarding copyright ownership.  The ASF licenses this file
5   to you under the Apache License, Version 2.0 (the
6   "License"); you may not use this file except in compliance
7   with the License.  You may obtain a copy of the License at
8
9   http://www.apache.org/licenses/LICENSE-2.0
10
11   Unless required by applicable law or agreed to in writing,
12   software distributed under the License is distributed on an
13   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14   KIND, either express or implied.  See the License for the
15   specific language governing permissions and limitations
16   under the License.
17
18.. include:: ../../common.defs
19
20.. _blackbox-testing:
21
22|TS| Blackbox Testing
23********************************
24|TS| uses the Reusable Gold Testing System (`AuTest <https://bitbucket.org/autestsuite/reusable-gold-testing-system/src/master/>`_)
25for functional testing. The current layout is:
26
27::
28
29    gold_tests/ - contains all the tests that run on the Reusable Gold Testing System (AuTest)
30    tools/ - contains programs used to help with testing.
31
32Scripts
33========
34To ease the process of running Autest, there is *autest.sh* and *bootstrap.py*.
35
36**autest.sh** - This file is a simple wrapper that will call the Reusable Gold Testing System (Autest) program in a pipenv.
37If the pipenv is not setup, the script will prompt user the missing components.
38That will set up the Autest on most systems in a Python virtual environment.
39The wrapper adds some basic options to the command to point to the location of the tests.
40Run the script from the ``tests/`` directory followed by ``--ats-bin`` and the bin directory where ATS is located (e.g. ``~/ats/bin``)
41Use ``--help`` for more details on options for running Autest.
42
43**bootstrap.py** - This script will check for the necessary packages needed to create a pipenv that can run Autest.
44If any package is missing, the script will alert the user.
45If all packages are available, it will install a virtual environment using the provided Pipfile.
46
47
48Manual Setup
49=============
50To run autest manually, the recommended way is to follow these steps:
51
521. ``pipenv install``: create the virtual environment(only needed once).
532. ``pipenv shell``: enter a shell in the virtual environment(type ``exit`` to leave the shell).
543. ``cd gold_tests``: enter the directory containing the test files.
554. ``autest --ats-bin user_ats_bin``: run autest where user_ats_bin is the bin directory in the user's ats directory.
56
57
58Advanced Setup
59===============
60AuTest and the relevant tools can be install manually instead of using the wrapper script.
61By doing this, it is often easier to debug issues with the testing system, or the tests.
62There are two ways this can be done.
63
641. Run the bootstrap script then source the path with a ``source ./env-test/bin/activate`` command. At this point autest command should run without the wrapper script.
652. Make sure you install python 3.5 or better on your system. From there install these python packages (e.g. ``pip install``):
66    - hyper
67    - git+https://bitbucket.org/autestsuite/reusable-gold-testing-system.git
68    - `traffic-replay <https://bitbucket.org/autestsuite/trafficreplay/src/master/>`_ (This will automatically install `MicroDNS <https://bitbucket.org/autestsuite/microdns/src/master/>`_, `MicroServer <https://bitbucket.org/autestsuite/microserver/src/master/>`_, `TrafficReplayLibrary <https://bitbucket.org/autestsuite/trafficreplaylibrary/src/master/>`_, and dnslib as part of the dependencies.)
69
70Writing Tests
71==============
72When writing tests, please refer to the current `documentation <https://autestsuite.bitbucket.io>`_ for general use of the Autest system.
73
74
75Testing Environment
76--------------------
77The environment of the testing process will have a number of added environment variables to control |TS| running the in the sandbox location correctly.
78This can be used to easily setup other commands that should run under same environment.
79
80Autest Extensions
81------------------
82Autest allows the user to create extensions to help specialize and simplify test writing for a given application domain.
83
84TrafficServer
85~~~~~~~~~~~~~~
86
87For TrafficServer, we have defined the following functions and objects in ``tests/gold_tests/autest-site/trafficserver.test.ext``:
88
89- ``MakeATSProcess(obj, name, command='traffic_server', select_ports=True, enable_tls=False)``
90
91  - name - A name for this instance of ATS
92  - command - optional argument defining what process to use. Defaults to ``traffic_server``
93  - select_ports - have Autest automatically select a nonSSL port to use
94  - enable_tls - have Autest automatically select SSL port (``select_ports`` must be **True**)
95
96- ``CopyConfig(file, targetname=None, process=None)``
97
98  - file - name of the file to copy. Relative paths are relative from the test file location
99  - targetname - the name of the file when copied to the correct configuration location
100  - process - optional process object to use for getting path location to copy to. Only needed if the Setup object call is not in the scope of the process object created with the MakeATSProcess(...) API.
101
102This function copies a given configuration file to the location of the |TS| sandbox used in a test. Given a test might have more than on |TS| instance, it can be difficult to understand the correct location to copy to. This function will deal with the details correctly.
103
104When automatically selected, the following ports will be allocated for TS:
105
106- port
107- portv6
108- ssl_port
109- admin_port - this is set even if select_port is **False**
110
111A number of file objects are also defined to help test TrafficServer. Files that are currently defined are:
112
113- squid.log
114- error.log
115- diags.log
116- records.config
117- cache.config
118- hosting.config
119- ip_allow.yaml
120- logging.yaml
121- parent.config
122- plugin.config
123- remap.config
124- sni.yaml
125- socks.config
126- splitdns.config
127- ssl_multicert.config
128- storage.config
129- volume.config
130
131Example
132++++++++
133.. code-block:: python
134
135  ts1 = Test.MakeATSProcess("ts1",select_ports=False)
136  # uses the setup object in the scope of the process object
137  ts1.Setup.ts.CopyConfig('config/records_8090.config','records.config')
138
139
140Origin Server
141~~~~~~~~~~~~~~
142
143- ``Test.MakeOriginServer(name, port, s_port, ip, delay, ssl, lookup_key, clientcert, clientkey)``
144
145  - name - A name for this instance of origin server.
146  - port - option to specify the nonSSL port. If left unspecified, the port will be autoselected.
147  - s_port - option to specify the SSL port. If left unspecified, the port will be autoselected (SSL has to be True).
148  - ip - option to specify IP address. Defaults to ``127.0.0.1``.
149  - delay - option to have MicroServer delay for set amount of seconds before returning response. Defaults to ``0``.
150  - ssl - option to enable SSL
151  - lookup_key - option to change the unique identifier that MicroServer uses to identify each transaction. Defaults to ``PATH``.
152  - clientcert - path to cert used for SSL. Defaults to the included cert in ``tests/tools/microserver/ssl``.
153  - clientkey - path to key used for SSL. Same default as above.
154
155This function returns a AuTest process object that launches the python-based Microserver.
156Microserver is a mock server which responds to client http requests.
157Microserver needs to be setup for the tests that require an origin server behind ATS.
158The server reads a JSON-formatted data file that contains request headers and the corresponding response headers.
159Microserver responds with payload if the response header contains Content-Length or Transfer-Encoding specified.
160
161- ``Test.addResponse(filename, request_header, response_header)``
162
163  - filename - name of the file where the request header and response header will be written to in JSON format
164  - request_header - dictionary of request header
165  - response_header - dictionary of response header corresponding to the request header.
166
167This function adds the request header and response header to a file which is then read by the microserver to populate request-response map.
168The key-fields required for the header dictionary are 'headers', 'timestamp' and 'body'.
169
170Example
171++++++++
172.. code-block:: python
173
174  # create the origin server process
175  server=Test.MakeOriginServer("server")
176  # define the request header and the desired response header
177  request_header={"headers": "GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""}
178  # desired response form the origin server
179  response_header={"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""}
180  # addResponse adds the transaction to a file which is used by the server
181  server.addResponse("sessionlog.json", request_header, response_header)
182
183
184DNS
185~~~~
186
187- ``Test.MakeDNServer(name, filename, port, ip, rr, default)``
188
189  - name - A name for this instance of MicroDNS.
190  - filename - file containing zone information for MicroDNS to read from. Defaults to ``dns_file.json``
191  - port - option for the DNS port. Autoselected if left unspecified.
192  - ip - option for IP address. Defaults to ``127.0.0.1``
193  - rr - option to enable round robin IP. Defaults to ``False``
194  - default - option to specify a default IP response when MicroDNS can't find a domain:IP pair.
195
196- ``dns.addRecords(records, jsonFile)``
197
198  - records - a dictionary of domain:IP mappings in the form of ``{"domain A": [IP1, IP2], "domain B": [IP3, IP4]}``
199  - jsonFile - a JSON file containing domain:IP mappings
200
201The JSON file must take the form of
202
203.. code-block:: python
204
205  {
206    "mappings: [
207        {"domain A": [IP1, IP2]},
208        {"domain B": [IP3, IP4]}
209    ]
210  }
211
212
213Example
214++++++++
215.. code-block:: python
216
217  # If everything is mapped to 127.0.0.1
218  dns = Test.MakeDNServer("dns", default=['127.0.0.1'])
219  #------------------------------------------------------
220  # Using addRecords method
221  dns = Test.MakeDNServer("dns")
222
223  dns.addRecords(records={"foo.com.":["127.0.0.1", "127.0.1.1"]})
224  # AND/OR
225  dns.addRecords(jsonFile="zone.json") # where zone.json is in the format described above
226
227
228Condition Testing
229+++++++++++++++++
230    - ``Condition.HasCurlFeature(feature)``
231        This function tests Curl for possible features it has been compiled with. Consult Curl documentation for possible features.
232    - ``Condition.PluginExists(pluginname)``
233        This function tests for the existence of a certain plugin in ATS.
234    - ``Condition.HasATSFeature(feature)``
235        This function tests TrafficServer for possible features it has been compiled with. Current features you can test for are:
236
237        - TS_HAS_LIBZ
238        - TS_HAS_LZMA
239        - TS_HAS_JEMALLOC
240        - TS_HAS_TCMALLOC
241        - TS_HAS_IN6_IS_ADDR_UNSPECIFIED
242        - TS_HAS_BACKTRACE
243        - TS_HAS_PROFILER
244        - TS_USE_FAST_SDK
245        - TS_USE_DIAGS
246        - TS_USE_EPOLL
247        - TS_USE_KQUEUE
248        - TS_USE_PORT
249        - TS_USE_POSIX_CAP
250        - TS_USE_TPROXY
251        - TS_HAS_SO_MARK
252        - TS_HAS_IP_TOS
253        - TS_USE_HWLOC
254        - TS_USE_SET_RBIO
255        - TS_USE_QUIC
256        - TS_USE_LINUX_NATIVE_AIO
257        - TS_HAS_SO_PEERCRED
258        - TS_USE_REMOTE_UNWINDING
259        - TS_HAS_128BIT_CAS
260        - TS_HAS_TESTS
261        - TS_HAS_WCCP
262
263
264Examples:
265+++++++++
266.. code-block:: python
267
268  Test.SkipUnless(
269    Condition.HasATSFeature('TS_USE_LINUX_NATIVE_AIO'),
270  )
271
272  Test.SkipUnless(
273    Condition.HasCurlFeature('http2'),
274  )
275
276  Test.SkipUnless(
277    Condition.PluginExists('a-plugin.so'),
278  )
279