1.. Licensed to the Apache Software Foundation (ASF) under one 2 or more contributor license agreements. See the NOTICE file 3 distributed with this work for additional information 4 regarding copyright ownership. The ASF licenses this file 5 to you under the Apache License, Version 2.0 (the 6 "License"); you may not use this file except in compliance 7 with the License. You may obtain a copy of the License at 8 9 http://www.apache.org/licenses/LICENSE-2.0 10 11 Unless required by applicable law or agreed to in writing, 12 software distributed under the License is distributed on an 13 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 14 KIND, either express or implied. See the License for the 15 specific language governing permissions and limitations 16 under the License. 17 18.. include:: ../../common.defs 19 20.. _blackbox-testing: 21 22|TS| Blackbox Testing 23******************************** 24|TS| uses the Reusable Gold Testing System (`AuTest <https://bitbucket.org/autestsuite/reusable-gold-testing-system/src/master/>`_) 25for functional testing. The current layout is: 26 27:: 28 29 gold_tests/ - contains all the tests that run on the Reusable Gold Testing System (AuTest) 30 tools/ - contains programs used to help with testing. 31 32Scripts 33======== 34To ease the process of running Autest, there is *autest.sh* and *bootstrap.py*. 35 36**autest.sh** - This file is a simple wrapper that will call the Reusable Gold Testing System (Autest) program in a pipenv. 37If the pipenv is not setup, the script will prompt user the missing components. 38That will set up the Autest on most systems in a Python virtual environment. 39The wrapper adds some basic options to the command to point to the location of the tests. 40Run the script from the ``tests/`` directory followed by ``--ats-bin`` and the bin directory where ATS is located (e.g. ``~/ats/bin``) 41Use ``--help`` for more details on options for running Autest. 42 43**bootstrap.py** - This script will check for the necessary packages needed to create a pipenv that can run Autest. 44If any package is missing, the script will alert the user. 45If all packages are available, it will install a virtual environment using the provided Pipfile. 46 47 48Manual Setup 49============= 50To run autest manually, the recommended way is to follow these steps: 51 521. ``pipenv install``: create the virtual environment(only needed once). 532. ``pipenv shell``: enter a shell in the virtual environment(type ``exit`` to leave the shell). 543. ``cd gold_tests``: enter the directory containing the test files. 554. ``autest --ats-bin user_ats_bin``: run autest where user_ats_bin is the bin directory in the user's ats directory. 56 57 58Advanced Setup 59=============== 60AuTest and the relevant tools can be install manually instead of using the wrapper script. 61By doing this, it is often easier to debug issues with the testing system, or the tests. 62There are two ways this can be done. 63 641. Run the bootstrap script then source the path with a ``source ./env-test/bin/activate`` command. At this point autest command should run without the wrapper script. 652. Make sure you install python 3.5 or better on your system. From there install these python packages (e.g. ``pip install``): 66 - hyper 67 - git+https://bitbucket.org/autestsuite/reusable-gold-testing-system.git 68 - `traffic-replay <https://bitbucket.org/autestsuite/trafficreplay/src/master/>`_ (This will automatically install `MicroDNS <https://bitbucket.org/autestsuite/microdns/src/master/>`_, `MicroServer <https://bitbucket.org/autestsuite/microserver/src/master/>`_, `TrafficReplayLibrary <https://bitbucket.org/autestsuite/trafficreplaylibrary/src/master/>`_, and dnslib as part of the dependencies.) 69 70Writing Tests 71============== 72When writing tests, please refer to the current `documentation <https://autestsuite.bitbucket.io>`_ for general use of the Autest system. 73 74 75Testing Environment 76-------------------- 77The environment of the testing process will have a number of added environment variables to control |TS| running the in the sandbox location correctly. 78This can be used to easily setup other commands that should run under same environment. 79 80Autest Extensions 81------------------ 82Autest allows the user to create extensions to help specialize and simplify test writing for a given application domain. 83 84TrafficServer 85~~~~~~~~~~~~~~ 86 87For TrafficServer, we have defined the following functions and objects in ``tests/gold_tests/autest-site/trafficserver.test.ext``: 88 89- ``MakeATSProcess(obj, name, command='traffic_server', select_ports=True, enable_tls=False)`` 90 91 - name - A name for this instance of ATS 92 - command - optional argument defining what process to use. Defaults to ``traffic_server`` 93 - select_ports - have Autest automatically select a nonSSL port to use 94 - enable_tls - have Autest automatically select SSL port (``select_ports`` must be **True**) 95 96- ``CopyConfig(file, targetname=None, process=None)`` 97 98 - file - name of the file to copy. Relative paths are relative from the test file location 99 - targetname - the name of the file when copied to the correct configuration location 100 - process - optional process object to use for getting path location to copy to. Only needed if the Setup object call is not in the scope of the process object created with the MakeATSProcess(...) API. 101 102This function copies a given configuration file to the location of the |TS| sandbox used in a test. Given a test might have more than on |TS| instance, it can be difficult to understand the correct location to copy to. This function will deal with the details correctly. 103 104When automatically selected, the following ports will be allocated for TS: 105 106- port 107- portv6 108- ssl_port 109- admin_port - this is set even if select_port is **False** 110 111A number of file objects are also defined to help test TrafficServer. Files that are currently defined are: 112 113- squid.log 114- error.log 115- diags.log 116- records.config 117- cache.config 118- hosting.config 119- ip_allow.yaml 120- logging.yaml 121- parent.config 122- plugin.config 123- remap.config 124- sni.yaml 125- socks.config 126- splitdns.config 127- ssl_multicert.config 128- storage.config 129- volume.config 130 131Example 132++++++++ 133.. code-block:: python 134 135 ts1 = Test.MakeATSProcess("ts1",select_ports=False) 136 # uses the setup object in the scope of the process object 137 ts1.Setup.ts.CopyConfig('config/records_8090.config','records.config') 138 139 140Origin Server 141~~~~~~~~~~~~~~ 142 143- ``Test.MakeOriginServer(name, port, s_port, ip, delay, ssl, lookup_key, clientcert, clientkey)`` 144 145 - name - A name for this instance of origin server. 146 - port - option to specify the nonSSL port. If left unspecified, the port will be autoselected. 147 - s_port - option to specify the SSL port. If left unspecified, the port will be autoselected (SSL has to be True). 148 - ip - option to specify IP address. Defaults to ``127.0.0.1``. 149 - delay - option to have MicroServer delay for set amount of seconds before returning response. Defaults to ``0``. 150 - ssl - option to enable SSL 151 - lookup_key - option to change the unique identifier that MicroServer uses to identify each transaction. Defaults to ``PATH``. 152 - clientcert - path to cert used for SSL. Defaults to the included cert in ``tests/tools/microserver/ssl``. 153 - clientkey - path to key used for SSL. Same default as above. 154 155This function returns a AuTest process object that launches the python-based Microserver. 156Microserver is a mock server which responds to client http requests. 157Microserver needs to be setup for the tests that require an origin server behind ATS. 158The server reads a JSON-formatted data file that contains request headers and the corresponding response headers. 159Microserver responds with payload if the response header contains Content-Length or Transfer-Encoding specified. 160 161- ``Test.addResponse(filename, request_header, response_header)`` 162 163 - filename - name of the file where the request header and response header will be written to in JSON format 164 - request_header - dictionary of request header 165 - response_header - dictionary of response header corresponding to the request header. 166 167This function adds the request header and response header to a file which is then read by the microserver to populate request-response map. 168The key-fields required for the header dictionary are 'headers', 'timestamp' and 'body'. 169 170Example 171++++++++ 172.. code-block:: python 173 174 # create the origin server process 175 server=Test.MakeOriginServer("server") 176 # define the request header and the desired response header 177 request_header={"headers": "GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""} 178 # desired response form the origin server 179 response_header={"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""} 180 # addResponse adds the transaction to a file which is used by the server 181 server.addResponse("sessionlog.json", request_header, response_header) 182 183 184DNS 185~~~~ 186 187- ``Test.MakeDNServer(name, filename, port, ip, rr, default)`` 188 189 - name - A name for this instance of MicroDNS. 190 - filename - file containing zone information for MicroDNS to read from. Defaults to ``dns_file.json`` 191 - port - option for the DNS port. Autoselected if left unspecified. 192 - ip - option for IP address. Defaults to ``127.0.0.1`` 193 - rr - option to enable round robin IP. Defaults to ``False`` 194 - default - option to specify a default IP response when MicroDNS can't find a domain:IP pair. 195 196- ``dns.addRecords(records, jsonFile)`` 197 198 - records - a dictionary of domain:IP mappings in the form of ``{"domain A": [IP1, IP2], "domain B": [IP3, IP4]}`` 199 - jsonFile - a JSON file containing domain:IP mappings 200 201The JSON file must take the form of 202 203.. code-block:: python 204 205 { 206 "mappings: [ 207 {"domain A": [IP1, IP2]}, 208 {"domain B": [IP3, IP4]} 209 ] 210 } 211 212 213Example 214++++++++ 215.. code-block:: python 216 217 # If everything is mapped to 127.0.0.1 218 dns = Test.MakeDNServer("dns", default=['127.0.0.1']) 219 #------------------------------------------------------ 220 # Using addRecords method 221 dns = Test.MakeDNServer("dns") 222 223 dns.addRecords(records={"foo.com.":["127.0.0.1", "127.0.1.1"]}) 224 # AND/OR 225 dns.addRecords(jsonFile="zone.json") # where zone.json is in the format described above 226 227 228Condition Testing 229+++++++++++++++++ 230 - ``Condition.HasCurlFeature(feature)`` 231 This function tests Curl for possible features it has been compiled with. Consult Curl documentation for possible features. 232 - ``Condition.PluginExists(pluginname)`` 233 This function tests for the existence of a certain plugin in ATS. 234 - ``Condition.HasATSFeature(feature)`` 235 This function tests TrafficServer for possible features it has been compiled with. Current features you can test for are: 236 237 - TS_HAS_LIBZ 238 - TS_HAS_LZMA 239 - TS_HAS_JEMALLOC 240 - TS_HAS_TCMALLOC 241 - TS_HAS_IN6_IS_ADDR_UNSPECIFIED 242 - TS_HAS_BACKTRACE 243 - TS_HAS_PROFILER 244 - TS_USE_FAST_SDK 245 - TS_USE_DIAGS 246 - TS_USE_EPOLL 247 - TS_USE_KQUEUE 248 - TS_USE_PORT 249 - TS_USE_POSIX_CAP 250 - TS_USE_TPROXY 251 - TS_HAS_SO_MARK 252 - TS_HAS_IP_TOS 253 - TS_USE_HWLOC 254 - TS_USE_SET_RBIO 255 - TS_USE_QUIC 256 - TS_USE_LINUX_NATIVE_AIO 257 - TS_HAS_SO_PEERCRED 258 - TS_USE_REMOTE_UNWINDING 259 - TS_HAS_128BIT_CAS 260 - TS_HAS_TESTS 261 - TS_HAS_WCCP 262 263 264Examples: 265+++++++++ 266.. code-block:: python 267 268 Test.SkipUnless( 269 Condition.HasATSFeature('TS_USE_LINUX_NATIVE_AIO'), 270 ) 271 272 Test.SkipUnless( 273 Condition.HasCurlFeature('http2'), 274 ) 275 276 Test.SkipUnless( 277 Condition.PluginExists('a-plugin.so'), 278 ) 279