Added initial support of stress testing. Now it is possible to start stress...

Added initial support of stress testing. Now it is possible to start stress test from mysql-test-run script.
For details see README.stress 

mysql-test/README.stress

+
+Overview
+--------
+
+Stress script is designed to perform testsing of mysql server in 
+multi-thread environment. 
+
+Stress script allows:
+ 
+ - to use for stress testing mysqltest binary as test engine
+ - to use for stress testing both regular test suite and any
+   additional test suites (e.g. mysql-test-extra-5.0)
+ - to specify files with lists of tests both for initialization of
+   stress db and for further testing itself
+ - to define number of threads that will be concurrently used in testing
+ - to define limitations for test run. e.g. number of tests or loops
+   for execution or duration of testing, delay between test executions, etc.
+ - to get readable log file which can be used for identification of
+   errors arose during testing
+
+All functionality regarding stress testing was implemeted in 
+mysql-stress-test.pl script and there are two ways to run stress test:
+
+ - for most cases it is enough to use options below for starting of
+   stress test from mysql-test-run wrapper. In this case server will 
+   be run automatically, all preparation steps will be performed
+   and after that stress test will be started.
+
+ - in advanced case one can run mysql-stress-test.pl script directly. 
+   But it requires to perform some preparation steps and to specify a 
+   bunch of options as well so this way may look a bit complicate.
+
+Usage
+-----
+
+Below is list of stress test specific options for mysql-test-run:
+
+--stress 
+  Enable stress mode
+
+--stress-suite=<suite name>
+  Test suite name that will be used in stress testing.
+  We assume that all suites are located in mysql-test/suite directory
+  There is one special suite name - <main|default> that corresponds
+  to regular test suite located in mysql-test directory. 
+  
+--stress-threads=<number of threads>
+  Number of threads that will be used in stress testing
+  
+--stress-tests-file=<filename with list of tests>
+  Filename with list of tests(without .test suffix) that will be used in 
+  stress testing. Default filename is stress_tests.txt and default 
+  location of this file is suite/<suite name>/stress_tests.txt
+
+--stress-init-file=<filename with list of tests>
+  Filename with list of tests(without .test suffix) that will be used in 
+  stress testing for initialization of stress db. These tests will be 
+  executed only once before starting of test itself. Default filename 
+  is stress_init.txt and default location of this file is 
+  suite/<suite name>/stress_init.txt
+
+--stress-mode=<method which will be used for choosing tests from the list>
+  Possible values are: random(default), seq 
+
+  There are two possible modes which affect order of selecting of tests
+  from the list:
+    - in random mode tests will be selected in random order
+    - in seq mode each thread will execute tests in the loop one by one as
+      they specified in the list file.
+
+--stress-test-count= <number>
+  Total number of tests that will be executed concurrently by all threads
+
+--stress-loop-count= <number>
+  Total number of loops in seq mode that will be executed concurrently 
+  by all threads
+
+--stress-test-duration= <number>
+  Duration of stress testing in seconds
+
+Examples
+-------- 
+
+1. Example of simple command line to start stress test:
+
+ mysql-test-run --stress alias
+
+Runs stress test with default values for number of threads and number of tests,
+with test 'alias' from suite 'main'.
+
+2. Using in stress testing tests from other suites:
+
+ - mysql-test-run --stress --stress-threads=10 --stress-test-count=1000 \
+                 --stress-suite=example  --stress-tests-file=testslist.txt
+
+   Will run stress test with 10 threads, will execute 1000 tests by all 
+   threads, test will be used from suite 'example', list of test will be 
+   taken from file 'testslist.txt'
+
+ - mysql-test-run --stress --stress-threads=10 --stress-test-count=1000 \
+                  --stress-suite=example sum_distinct
+
+   Will run stress test with 10 threads, will execute 1000 tests by all 
+   threads, test will be used from suite 'example', list of test contains 
+   only one test 'sum_distinct'
+
+3. Debugging of issues found with stress test
+
+ Right now stress test is not fully integrated in mysql-test-run
+ and does not support --gdb option so to debug issue found with stress 
+ test you have to start separately mysql server under debuger and then 
+ run stress test as:
+ 
+ - mysql-test-run --extern --stress --stress-threads=10 \
+                  --stress-test-count=1000 --stress-suite=example \
+                  sum_distinct

mysql-test/mysql-test-run.sh

@@ -256,6 +256,16 @@ NDB_MGM_EXTRA_OPTS=
 NDB_MGMD_EXTRA_OPTS=
 NDBD_EXTRA_OPTS=
 
+STRESS_SUITE="main"
+STRESS_MODE="random"
+STRESS_THREADS=5
+STRESS_TEST_COUNT=20
+STRESS_LOOP_COUNT=""
+STRESS_TEST_DURATION=""
+STRESS_INIT_FILE=""
+STRESS_TEST_FILE=""
+STRESS_TEST=""
+
 while test $# -gt 0; do
   case "$1" in
     --embedded-server)
@@ -344,6 +354,35 @@ while test $# -gt 0; do
       DO_BENCH=1
       NO_SLAVE=1
       ;;
+    --stress)
+      DO_STRESS=1
+      NO_SLAVE=1
+      SKIP_SLAVE=1
+      ;;
+    --stress-suite=*)
+      STRESS_SUITE=`$ECHO "$1" | $SED -e "s;--stress-suite=;;"`
+      ;;
+    --stress-threads=*)
+      STRESS_THREADS=`$ECHO "$1" | $SED -e "s;--stress-threads=;;"`
+      ;;
+    --stress-test-file=*)
+      STRESS_TEST_FILE=`$ECHO "$1" | $SED -e "s;--stress-test-file=;;"`
+      ;;
+    --stress-init-file=*)
+      STRESS_INIT_FILE=`$ECHO "$1" | $SED -e "s;--stress-init-file=;;"`
+      ;; 
+    --stress-mode=*)
+      STRESS_MODE=`$ECHO "$1" | $SED -e "s;--stress-mode=;;"`
+      ;;
+    --stress-loop-count=*)
+      STRESS_LOOP_COUNT=`$ECHO "$1" | $SED -e "s;--stress-loop-count=;;"`
+      ;;
+    --stress-test-count=*)
+      STRESS_TEST_COUNT=`$ECHO "$1" | $SED -e "s;--stress-test-count=;;"`
+      ;;      
+    --stress-test-duration=*)
+      STRESS_TEST_DURATION=`$ECHO "$1" | $SED -e "s;--stress-test-duration=;;"`
+      ;;
     --big*)			# Actually --big-test
       EXTRA_MYSQL_TEST_OPT="$EXTRA_MYSQL_TEST_OPT $1" ;;
     --compress)
@@ -700,7 +739,7 @@ fi
 
 # If we should run all tests cases, we will use a local server for that
 
-if [ -z "$1" ]
+if [ -z "$1" -a $DO_STRESS -eq 0 ]
 then
    USE_RUNNING_SERVER=0
 fi
@@ -1189,7 +1228,7 @@ start_master()
   then
       CURR_MASTER_MYSQLD_TRACE="$EXTRA_MASTER_MYSQLD_TRACE$1"
   fi
-  if [ -z "$DO_BENCH" ]
+  if [ -z "$DO_BENCH" -a -z "$DO_STRESS"  ]
   then
     master_args="--no-defaults --log-bin=$MYSQL_TEST_DIR/var/log/master-bin$1 \
   	    --server-id=$id  \
@@ -1626,7 +1665,7 @@ run_testcase ()
      stop_master 1
      report_current_test $tname
      start_master
-     if [ -n "$USE_NDBCLUSTER" -a -z "$DO_BENCH" ] ; then
+     if [ -n "$USE_NDBCLUSTER" -a -z "$DO_BENCH" -a -z "$DO_STRESS" ] ; then
        start_master 1
      fi
      TZ=$MY_TZ; export TZ
@@ -1642,7 +1681,7 @@ run_testcase ()
        stop_master 1
        report_current_test $tname
        start_master
-       if [ -n "$USE_NDBCLUSTER"  -a -z "$DO_BENCH" ] ; then
+       if [ -n "$USE_NDBCLUSTER"  -a -z "$DO_BENCH" -a -z "$DO_STRESS" ] ; then
          start_master 1
        fi
      else
@@ -1763,6 +1802,120 @@ run_testcase ()
   fi
 }
 
+run_stress_test()
+{
+
+  STRESS_BASEDIR="$MYSQL_TEST_DIR/var/stress"
+
+  #Clean-up old stress test basedir
+  if [ -d $STRESS_BASEDIR ] ; then 
+    $RM -rf $STRESS_BASEDIR
+  fi
+  #Create stress test basedir 
+  mkdir $STRESS_BASEDIR
+
+  if [ "$STRESS_SUITE" != "main" -a "$STRESS_SUITE" != "default" ] ; then
+    STRESS_SUITE_DIR="$MYSQL_TEST_DIR/suite/$STRESS_SUITE"
+  else
+    STRESS_SUITE_DIR="$MYSQL_TEST_DIR"
+  fi
+
+  if [ -d "$STRESS_SUITE_DIR" ] ; then 
+    STRESS_SUITE_T_DIR="$STRESS_SUITE_DIR/t"
+    STRESS_SUITE_R_DIR="$STRESS_SUITE_DIR/r"
+    #FIXME: check that dirs above are exist
+  else
+    echo "Directory $STRESS_SUITE_DIR with test suite doesn't exists. Abort stress testing"
+    exit 1
+  fi
+  
+  if [ -n "$STRESS_TEST" ] ; then 
+    STRESS_TEST_FILE="$STRESS_BASEDIR/stress_tests.txt"
+    echo $STRESS_TEST > $STRESS_TEST_FILE
+  elif [ -n "$STRESS_TEST_FILE" ] ; then    
+    STRESS_TEST_FILE="$STRESS_SUITE_DIR/$STRESS_TEST_FILE"
+    if [ ! -f  "$STRESS_TEST_FILE" ] ; then 
+      echo "Specified file $STRESS_TEST_FILE with list of tests does not exist. Please ensure that file exists and has proper permissions"
+      exit 1
+    fi
+  else 
+    STRESS_TEST_FILE="$STRESS_SUITE_DIR/stress_tests.txt"
+    if [ ! -f  "$STRESS_TEST_FILE" ] ; then 
+      echo "Default file $STRESS_TEST_FILE with list of tests does not exist. Please use --stress-test-file option to specify custom one or you can just specify name of test for testing as last argument in command line"
+      exit 1
+    fi
+  fi
+
+  if [ -n "$STRESS_INIT_FILE" ] ; then 
+    STRESS_INIT_FILE="$STRESS_SUITE_DIR/$STRESS_INIT_FILE"
+    if [ ! -f  "$STRESS_INIT_FILE" ] ; then 
+      echo "Specified file $STRESS_INIT_FILE with list of tests doesn't exist. Please ensure that file exists and has proper permissions"
+      exit 1
+    fi
+  else
+    STRESS_INIT_FILE="$STRESS_SUITE_DIR/stress_init.txt"
+    #Check for default init file
+    if [ ! -f "$STRESS_INIT_FILE" ] ; then 
+      STRESS_INIT_FILE=""
+    fi
+  fi
+
+  if [ "$STRESS_MODE" != "random" -a "$STRESS_MODE" != "seq" ] ; then
+    echo "You specified wrong mode '$STRESS_MODE' for stress test. Correct values are 'random' or 'seq'"
+    exit 1
+  fi
+ 
+  STRESS_TEST_ARGS="--server-socket=$MASTER_MYSOCK \
+                    --server-user=$DBUSER \
+                    --server-database=$DB \
+                    --stress-suite-basedir=$MYSQL_TEST_DIR \
+                    --suite=$STRESS_SUITE \
+                    --stress-tests-file=$STRESS_TEST_FILE \
+                    --stress-basedir=$STRESS_BASEDIR \
+                    --server-logs-dir=$STRESS_BASEDIR \
+                    --stress-mode=$STRESS_MODE \
+                    --mysqltest=$BASEDIR/client/mysqltest \
+                    --threads=$STRESS_THREADS \
+                    --verbose \
+                    --cleanup \
+                    --log-error-details \
+                    --abort-on-error" 
+  
+  if [ -n "$STRESS_INIT_FILE" ] ; then 
+    STRESS_TEST_ARGS="$STRESS_TEST_ARGS --stress-init-file=$STRESS_INIT_FILE"
+  fi
+
+  if [ -n "$STRESS_LOOP_COUNT" ] ; then 
+    STRESS_TEST_ARGS="$STRESS_TEST_ARGS --loop-count=$STRESS_LOOP_COUNT"
+  fi
+
+  if [ -n "$STRESS_TEST_COUNT" ] ; then 
+    STRESS_TEST_ARGS="$STRESS_TEST_ARGS --test-count=$STRESS_TEST_COUNT"
+  fi
+
+  if [ -n "$STRESS_TEST_DURATION" ] ; then 
+    STRESS_TEST_ARGS="$STRESS_TEST_ARGS --test-duration=$STRESS_TEST_DURATION"
+  fi  
+
+  echo "Stress test related variables:"
+  echo "TESTS                - $1"
+  echo "STRESS               - $DO_STRESS"
+  echo "STRESS_SUITE         - $STRESS_SUITE"
+  echo "STRESS_TEST_FILE     - $STRESS_TEST_FILE"
+  echo "STRESS_INIT_FILE     - $STRESS_INIT_FILE"
+  echo "STRESS_THREADS       - $STRESS_THREADS"
+  echo "STRESS_MODE          - $STRESS_MODE"
+  echo "STRESS_TEST_COUNT    - $STRESS_TEST_COUNT"
+  echo "STRESS_LOOP_COUNT    - $STRESS_LOOP_COUNT"  
+  echo "STRESS_TEST_DURATION - $STRESS_TEST_DURATION"
+
+  #echo "$STRESS_TEST_ARGS";
+  #Run stress test 
+  $MYSQL_TEST_DIR/mysql-stress-test.pl $STRESS_TEST_ARGS
+
+
+}
+
 ######################################################################
 # Main script starts here
 ######################################################################
@@ -1877,6 +2030,32 @@ then
   exit
 fi
 
+#
+# Stress testing
+#
+if [ "$DO_STRESS" = 1 ] 
+then
+
+  if [ -n "$1" ] ; then
+    STRESS_TEST="$1";
+  fi                      
+ 
+  if [ $USE_RUNNING_SERVER -eq 0 ] ; then 
+    start_master
+  fi
+ 
+  run_stress_test
+
+  if [ $USE_RUNNING_SERVER -eq 0 ] ; then
+    mysql_stop
+    stop_manager
+  fi
+  
+  exit
+
+fi
+
+
 $ECHO
 if [ x$USE_TIMER = x1 ] ; then
 $ECHO "TEST                            RESULT        TIME (ms)"