This directory contains test suites for the MariaDB server. To run
currently existing test cases, execute ./mysql-test-run in this directory.

Some tests are known to fail on some platforms or be otherwise unreliable.
The file "unstable-tests" contains the list of such tests along with
a comment for every test.
To exclude them from the test run, execute
  # ./mysql-test-run --skip-test-list=unstable-tests

In general you do not have to have to do "make install", and you can have
a co-existing MariaDB installation, the tests will not conflict with it.
To run the tests in a source directory, you must do "make" first.

In Red Hat distributions, you should run the script as user "mysql".
The user is created with nologin shell, so the best bet is something like
  # su -
  # cd /usr/share/mysql-test
  # su -s /bin/bash mysql -c "./mysql-test-run --skip-test-list=unstable-tests"

This will use the installed MariaDB executables, but will run a private
copy of the server process (using data files within /usr/share/mysql-test),
so you need not start the mysqld service beforehand.

You can omit --skip-test-list option if you want to check whether
the listed failures occur for you.

To clean up afterwards, remove the created "var" subdirectory, e.g.
  # su -s /bin/bash - mysql -c "rm -rf /usr/share/mysql-test/var"

If one or more tests fail on your system on reasons other than listed
in lists of unstable tests, please read the following manual section
for instructions on how to report the problem:

https://mariadb.com/kb/en/reporting-bugs

If you want to use an already running MySQL server for specific tests,
use the --extern option to mysql-test-run. Please note that in this mode,
you are expected to provide names of the tests to run.

For example, here is the command to run the "alias" and "analyze" tests
with an external server:

  # mysql-test-run --extern socket=/tmp/mysql.sock alias analyze

To match your setup, you might need to provide other relevant options.

With no test names on the command line, mysql-test-run will attempt
to execute the default set of tests, which will certainly fail, because
many tests cannot run with an external server (they need to control the
options with which the server is started, restart the server during
execution, etc.)

You can create your own test cases. To create a test case, create a new
file in the main subdirectory using a text editor. The file should have a .test
extension. For example:

  # xemacs t/test_case_name.test

In the file, put a set of SQL statements that create some tables,
load test data, and run some queries to manipulate it.

Your test should begin by dropping the tables you are going to create and
end by dropping them again. This ensures that you can run the test over
and over again.

If you are using mysqltest commands in your test case, you should create
the result file as follows:

  # mysql-test-run --record test_case_name

  or

  # mysqltest --record < t/test_case_name.test

If you only have a simple test case consisting of SQL statements and
comments, you can create the result file in one of the following ways:

  # mysql-test-run --record test_case_name

  # mysql test < t/test_case_name.test > r/test_case_name.result

  # mysqltest --record --database test --result-file=r/test_case_name.result < t/test_case_name.test

When this is done, take a look at r/test_case_name.result.
If the result is incorrect, you have found a bug. In this case, you should
edit the test result to the correct results so that we can verify that
the bug is corrected in future releases.

If you want to submit your test case you can send it
to maria-developers@lists.launchpad.net or attach it to a bug report on
https://mariadb.org/jira/.

If the test case is really big or if it contains 'not public' data,
then put your .test file and .result file(s) into a tar.gz archive,
add a README that explains the problem, ftp the archive to
ftp://ftp.askmonty.org/private and submit a report to
https://mariadb.org/jira about it.

The latest information about mysql-test-run can be found at:
https://mariadb.com/kb/en/mariadb/mysqltest/

If you want to create .rdiff files, check
https://mariadb.com/kb/en/mariadb/mysql-test-auxiliary-files/

To be able to see the level of coverage with the current test suite,
do the following:

  - Make sure gcov is installed
  - Compile the MySQL distribution with BUILD/compile-pentium64-gcov (if your
    machine does not have a pentium CPU, hack this script, or just live with
	the pentium-specific stuff)
  - In the mysql-test directory, run this command: ./mysql-test-run -gcov
  - To see the level of coverage for a given source file:
     grep -1 source_file_name ../mysql-test-gcov.msg
  - To see which lines are not yet covered, look at source_file_name.gcov in
    the source tree.  You can find this by doing something like:
    find source-directory -name "mysqld.cc.gcov"
    Then think hard about a test case that will cover those lines, and write
    one!

Overview
--------

The stress script is designed to perform testing of the MySQL server in
a multi-threaded environment.

All functionality regarding stress testing is implemented in the
mysql-stress-test.pl script.

The stress script allows:

 - To stress test the mysqltest binary test engine.
 - To stress test the regular test suite and any additional test suites
   (such as mysql-test-extra-5.0).
 - To specify files with lists of tests both for initialization of
   stress db and for further testing itself.
 - To define the number of threads to be concurrently used in testing.
 - To define limitations for the test run. such as the number of tests or
   loops for execution or duration of testing, delay between test
   executions, and so forth.
 - To get a readable log file that can be used for identification of
   errors that occur during testing.

There are two ways to run the mysql-stress-test.pl script:

 - For most cases, it is enough to use the options below for starting
   the stress test from the mysql-test-run wrapper. In this case, the
   server is run automatically, all preparation steps are performed,
   and after that the stress test is started.

 - In advanced case, you can run the mysql-stress-test.pl script directly.
   But this requires that you perform some preparation steps and to specify
   a bunch of options as well, so this invocation method may be a bit
   complicated.

Usage
-----

The following mysql-test-run options are specific to stress-testing:

--stress
  Enable stress mode

--stress-suite=<suite name>
  Test suite name to use in stress testing.  We assume that all suites
  are located in the mysql-test/suite directory.
  There is one special suite name - <main|default> that corresponds
  to the regular test suite located in the mysql-test directory.

--stress-threads=<number of threads>
  The number of threads to use in stress testing.

--stress-tests-file=<filename with list of tests>
  The file that contains the list of tests (without .test suffix) to use in
  stress testing. The default filename is stress_tests.txt and the default
  location of this file is suite/<suite name>/stress_tests.txt

--stress-init-file=<filename with list of tests>
  The file that contains list of tests (without .test suffix) to use in
  stress testing for initialization of the stress db. These tests will be
  executed only once before starting the test itself. The default filename
  is stress_init.txt and the default location of this file is
  suite/<suite name>/stress_init.txt

--stress-mode=<method which will be used for choosing tests from the list>
  Possible values are: random(default), seq

  There are two possible modes that affect the order of test selection
  from the list:
    - In random mode, tests are selected in random order
    - In seq mode, each thread executes tests in a loop one by one in
	  the order specified in the list file.

--stress-test-count=<number>
  Total number of tests that will be executed concurrently by all threads

--stress-loop-count=<number>
  Total number of loops in seq mode that will be executed concurrently
  by all threads

--stress-test-duration=<number>
  Duration of stress testing in seconds

Examples
--------

1. Example of a simple command line to start a stress test:

 mysql-test-run --stress alias

Runs a stress test with default values for number of threads and number
of tests, with test 'alias' from suite 'main'.

2. Using in stress testing tests from other suites:

 - mysql-test-run --stress --stress-threads=10 --stress-test-count=1000 \
                 --stress-suite=example  --stress-tests-file=testslist.txt

   Runs a stress test with 10 threads, executes 1000 tests by all
   threads, tests are used from suite 'example', the list of tests is
   taken from file 'testslist.txt'

 - mysql-test-run --stress --stress-threads=10 --stress-test-count=1000 \
                  --stress-suite=example sum_distinct

   Runs stress test with 10 threads, executes 1000 tests by all
   threads, tests are used from suite 'example', the list of tests
   contains only one test 'sum_distinct'

3. Debugging of issues found with stress test

 Right now, the stress test is not fully integrated in mysql-test-run
 and does not support the --gdb option. To debug issues found with the
 stress test, you must start the MySQL server separately under a debugger
 and then run the stress test like this:

 - mysql-test-run --extern --stress --stress-threads=10 \
                  --stress-test-count=1000 --stress-suite=example \
                  sum_distinct