xref: /minix/external/bsd/bind/dist/unit/atf-src/TODO (revision 00b67f09)

Things to do                                    Automated Testing Framework
===========================================================================


Last revised: November 30th, 2010


This document includes the list of things that need to be done in ATF that
are most requested by the users. This information used to be available in
an ad-hoc bug tracker but that proved to be a bad idea. I have collected
all worthy comments in here.

Please note that most work these days is going into Kyua (see
http://code.google.com/p/kyua/). The ideas listed here apply to the
components of ATF that have *not* been migrated to the new codebase yet.
For bug reports or ideas that apply to the components that already have
been migrated, please use the bug tracker in the URL above. Similarly,
whenever a component is migrated, the ideas in this file should be revised
and migrated to the new bug tracker where appropriate.


---------------------------------------------------------------------------
Add build-time checks to atf-sh

The 0.7 release introduced build-time tests to atf-c and atf-c++, but not
to atf-sh. Expose the functionality to the shell interface.

This will probably require writing an atf-build utility that exposes the C
code and can be called from the shell.

---------------------------------------------------------------------------
Revisit what to do when an Atffile lists a non-existent file

---------------------------------------------------------------------------
Add ATF_CHECK* versions to atf-c++ to support non-fatal tests

---------------------------------------------------------------------------
Implement race-condition tests

gcooper:

I would think that stress/negative tests would be of more value than race
condition tests (they're similar, but not exactly the same in my mind).

In particular,

1. Feed through as much data as possible to determine where reporting
   breaks down.
2. Feed through data quickly and terminate ASAP. The data should be
   captured. Terminate child applications with unexpected exit codes and
   signals (in particular, SIGCHLD, SIGPIPE, exit codes that terminate,
   etc).
3. Open up a file descriptor in the test application, don't close the file
   descriptor.
4. fork(2) a process; don't wait(2) for the application to complete.

There are other scenarios that could be exercised, but these are the ones
I could think of off the topic of my head.

--

jmmv:

1. The thing is: how do you express any of this in a portable/abstract
   interface? How do you express that a test case "receives data"? What
   does that exactly mean? I don't think the framework should care about
   this: each test should be free to decide where its data is and how to
   deal with it.

2. Ditto.

3. Not sure I understand your request, but testing for "unexpected exit
   codes" is already supported. See wiki:DesignXFail for the feature
   design details.

4. What's the problem with this case? The test case exits right away after
   terminating the execution of its body; any open file descriptors,
   leaked memory, etc. die with it.

5. forking and not waiting for a subprocess was a problem already
   addressed.

I kinda have an idea of what Antti means with "race condition tests", but
every time I have tried to describe my understanding of matters I seem to
be wrong. Would be nice to have a clear description of what this involves;
in particular, what are the expectations from the framework and how should
the feature be exposed.

As of now, what I understand by "race condition test" is: a test case that
exercises a race condition. The test case may finish without triggering
the race, in which case it just exists with a successful status.
Otherwise, if the race triggers, the test case gets stuck and times out.
The result should be reported as an "expected failure" different from
timeout.

--

pooka:

Yup. Plus some atf-wide mechanism for the operator to supply some kind of
guideline if the test should try to trigger the race for a second or for
an hour.

--

jmmv:

Alright. While mocking up some code for this, I think that your two
requests are complementary.

On the one hand, when you are talking about a "race condition" test you
really mean an "expected race condition" test. Correct? If so, we need to
extend the xfail mechanism to add one more case, which is to report any
failures as a race condition error and, if there is no failure, report the
test as successful.

On the other hand, the atf-wide mechanism to support how long the test
should run for can be thought as a "stress test" mechanism. I.e. run this
test for X time / iterations and report its results regularly without
involving xfail at all.

So, with this in mind:

* For a test that triggers an unfixed race condition, you set xfail to
  race mode and define the test as a stress test. Any failures are
  reported as expected failures.

* For a test that verifies a supposedly-fixed race condition, you do *not*
  set xfail to race mode, and only set the test to stress test. Any
  failures are reported as real failures.

These stress test cases implement a single iteration of the test and
atf-run is in charge of running the test several times, stopping on the
first failure.

Does that make sense?

---------------------------------------------------------------------------
Implement ATF_REQUIRE_ERRNO

pooka:

Most of the lines in tests against system functionality are:

if (syscall(args) == -1)
    atf_tc_fail_errno("flop")

Some shorthand would be helpful, like ATF_REQUIRE_ERRNO(syscall(args))
Also, a variant which allows arbitrary return value checks (e.g. "!= 0" or
"< 124" or "!= size") would be nice.

--

gcooper:

There's a problem with this request; not all functions fail in the same
way ... in particular compare the pthread family of functions (which
return errno) vs many native syscalls. Furthermore, compare some
fcntl-like syscalls vs other syscalls. One size fits all solutions may not
be a wise idea in this case, so I think that the problem statement needs
to be better defined, because the above request is too loose.

FWIW, there's also a TEST macro in LTP, which tests for non-zero status,
and sets an appropriate set of global variables for errnos and return
codes, respectively. It was a good idea, but has been mostly abandoned
because it's too difficult to define a success and failure in a universal
manner, so I think that we need to be careful with what's implemented in
ATF to not repeat the mistakes that others have made.

--

jmmv:

I think you've got a good point.

This was mostly intended to simplify the handling of the stupid errno
global variable. I think this is valuable to have, but maybe the
macro/function name should be different because _ERRNO can be confusing.
Probably something like an ATF_CHECK_LIBC / ATF_CHECK_PTHREAD approach
would be more flexible and simple.


===========================================================================
vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2