Commit 95ce14c3 authored by Brenda J. Butler's avatar Brenda J. Butler Committed by David S. Miller

tools: tc-testing: Update README and TODO

Signed-off-by: default avatarBrenda J. Butler <bjb@mojatatu.com>
Acked-by: default avatarLucas Bates <lucasb@mojatatu.com>
Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
parent c25e4736
......@@ -14,11 +14,11 @@ REQUIREMENTS
* The kernel must have network namespace support
* The kernel must have veth support available, as a veth pair is created
* The kernel must have veth support available, as a veth pair is created
prior to running the tests.
* All tc-related features must be built in or available as modules.
To check what is required in current setup run:
* All tc-related features being tested must be built in or available as
modules. To check what is required in current setup run:
./tdc.py -c
Note:
......@@ -44,10 +44,13 @@ using the -p option when running tdc:
RUNNING TDC
-----------
To use tdc, root privileges are required. tdc will not run otherwise.
To use tdc, root privileges are required. This is because the
commands being tested must be run as root. The code that enforces
execution by root uid has been moved into a plugin (see PLUGIN
ARCHITECTURE, below).
All tests are executed inside a network namespace to prevent conflicts
within the host.
If nsPlugin is linked, all tests are executed inside a network
namespace to prevent conflicts within the host.
Running tdc without any arguments will run all tests. Refer to the section
on command line arguments for more information, or run:
......@@ -59,6 +62,33 @@ output captured from the failing test will be printed immediately following
the failed test in the TAP output.
OVERVIEW OF TDC EXECUTION
-------------------------
One run of tests is considered a "test suite" (this will be refined in the
future). A test suite has one or more test cases in it.
A test case has four stages:
- setup
- execute
- verify
- teardown
The setup and teardown stages can run zero or more commands. The setup
stage does some setup if the test needs it. The teardown stage undoes
the setup and returns the system to a "neutral" state so any other test
can be run next. These two stages require any commands run to return
success, but do not otherwise verify the results.
The execute and verify stages each run one command. The execute stage
tests the return code against one or more acceptable values. The
verify stage checks the return code for success, and also compares
the stdout with a regular expression.
Each of the commands in any stage will run in a shell instance.
USER-DEFINED CONSTANTS
----------------------
......@@ -70,23 +100,132 @@ executed as part of the test. More will be added as test cases require.
Example:
$TC qdisc add dev $DEV1 ingress
The NAMES values are used to substitute into the commands in the test cases.
COMMAND LINE ARGUMENTS
----------------------
Run tdc.py -h to see the full list of available arguments.
-p PATH Specify the tc executable located at PATH to be used on this
test run
-c Show the available test case categories in this test file
-c CATEGORY Run only tests that belong to CATEGORY
-f FILE Read test cases from the JSON file named FILE
-l [CATEGORY] List all test cases in the JSON file. If CATEGORY is
specified, list test cases matching that category.
-s ID Show the test case matching ID
-e ID Execute the test case identified by ID
-i Generate unique ID numbers for test cases with no existing
ID number
usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]]
[-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v]
[-d DEVICE] [-n NS] [-V]
Linux TC unit tests
optional arguments:
-h, --help show this help message and exit
-p PATH, --path PATH The full path to the tc executable to use
-v, --verbose Show the commands that are being run
-d DEVICE, --device DEVICE
Execute the test case in flower category
selection:
select which test cases: files plus directories; filtered by categories
plus testids
-D DIR [DIR ...], --directory DIR [DIR ...]
Collect tests from the specified directory(ies)
(default [tc-tests])
-f FILE [FILE ...], --file FILE [FILE ...]
Run tests from the specified file(s)
-c [CATG [CATG ...]], --category [CATG [CATG ...]]
Run tests only from the specified category/ies, or if
no category/ies is/are specified, list known
categories.
-e ID [ID ...], --execute ID [ID ...]
Execute the specified test cases with specified IDs
action:
select action to perform on selected test cases
-l, --list List all test cases, or those only within the
specified category
-s, --show Display the selected test cases
-i, --id Generate ID numbers for new test cases
netns:
options for nsPlugin(run commands in net namespace)
-n NS, --namespace NS
Run commands in namespace NS
valgrind:
options for valgrindPlugin (run command under test under Valgrind)
-V, --valgrind Run commands under valgrind
PLUGIN ARCHITECTURE
-------------------
There is now a plugin architecture, and some of the functionality that
was in the tdc.py script has been moved into the plugins.
The plugins are in the directory plugin-lib. The are executed from
directory plugins. Put symbolic links from plugins to plugin-lib,
and name them according to the order you want them to run.
Example:
bjb@bee:~/work/tc-testing$ ls -l plugins
total 4
lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py
lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py
-rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py
The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
must be called "SubPlugin" so tdc can find them. They are
distinguished from each other in the python program by their module
name.
This base class supplies "hooks" to run extra functions. These hooks are as follows:
pre- and post-suite
pre- and post-case
pre- and post-execute stage
adjust-command (runs in all stages and receives the stage name)
The pre-suite hook receives the number of tests and an array of test ids.
This allows you to dump out the list of skipped tests in the event of a
failure during setup or teardown stage.
The pre-case hook receives the ordinal number and test id of the current test.
The adjust-command hook receives the stage id (see list below) and the
full command to be executed. This allows for last-minute adjustment
of the command.
The stages are identified by the following strings:
- pre (pre-suite)
- setup
- command
- verify
- teardown
- post (post-suite)
To write a plugin, you need to inherit from TdcPlugin in
TdcPlugin.py. To use the plugin, you have to put the
implementation file in plugin-lib, and add a symbolic link to it from
plugins. It will be detected at run time and invoked at the
appropriate times. There are a few examples in the plugin-lib
directory:
- rootPlugin.py:
implements the enforcement of running as root
- nsPlugin.py:
sets up a network namespace and runs all commands in that namespace
- valgrindPlugin.py
runs each command in the execute stage under valgrind,
and checks for leaks.
This plugin will output an extra test for each test in the test file,
one is the existing output as to whether the test passed or failed,
and the other is a test whether the command leaked memory or not.
(This one is a preliminary version, it may not work quite right yet,
but the overall template is there and it should only need tweaks.)
ACKNOWLEDGEMENTS
......
......@@ -5,6 +5,27 @@ tc Testing Suite To-Do list:
- Add support for multiple versions of tc to run successively
- Improve error messages when tdc aborts its run
- Improve error messages when tdc aborts its run. Partially done - still
need to better handle problems in pre- and post-suite.
- Allow tdc to write its results to file
- Use python logger module for debug/verbose output
- Allow tdc to write its results to file.
Maybe use python logger module for this too.
- A better implementation of the "hooks". Currently, every plugin
will attempt to run a function at every hook point. Could be
changed so that plugin __init__ methods will register functions to
be run in the various predefined times. Then if a plugin does not
require action at a specific point, no penalty will be paid for
trying to run a function that will do nothing.
- Proper exception handling - make an exception class and use it
- a TestCase class, for easier testcase handling, searching, comparison
- a TestSuite class
and a way to configure a test suite,
to automate running multiple "test suites" with different requirements
- super simple test case example using ls, touch, etc
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment