New type of page in the page cache fixes.


storage/maria/ma_pagecache.c:
  Mark the page dirty if we store LSN on it.
  Symbolic representation of new page type added (for debugging output).
  Asserts added (unknown type can't be used if we write the page).

BitKeeper/etc/ignore:
  added storage/maria/maria_read_log
support-files/compiler_warnings.supp:
  Ignore function used when debugging (can be called from gdb)

into  mysql.com:/home/my/mysql-maria


include/maria.h:
  Auto merged
sql/handler.h:
  Auto merged
storage/maria/ha_maria.cc:
  Auto merged
storage/maria/ma_close.c:
  Auto merged
storage/maria/ma_loghandler.c:
  Auto merged
storage/maria/ma_open.c:
  Auto merged
storage/maria/ma_pagecache.c:
  Auto merged
storage/maria/ma_pagecache.h:
  Auto merged
storage/maria/maria_chk.c:
  Auto merged
storage/maria/ma_blockrec.c:
  Manual merge (No changes)
storage/maria/ma_check.c:
  Manual merge
storage/maria/ma_create.c:
  Manual merge
storage/maria/ma_delete_all.c:
  Manual merge
storage/maria/ma_init.c:
  Manual merge (no changes)
storage/maria/ma_test_all.sh:
  Manual merge
storage/maria/maria_def.h:
  Manual merge

Fixed maria_chk to repair BLOCK-ROW tables.
Added CREATE options ROW_FORMAT=PAGE & TRANSACTIONAL= 0|1
More DBUG information in a lot of functions
Some minor code cleanups
Enable handler errors earlier for better clear text error messages at handler startup / standalone usage.
Don't print NULL strings in my_create_with_symlink();  Fixes core dump when used with --debug


include/maria.h:
  Added extra variables needed for REPAIR with BLOCK records
include/my_base.h:
  Added argument for opening copy of maria table without a shared object
include/my_handler.h:
  Prototypes for my_handler_error_register() & my_handler_error_unregister()
include/pagecache.h:
  Added PAGECACHE_READ_UNKNOWN_PAGE
mysql-test/include/ps_conv.inc:
  Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
mysql-test/r/maria.result:
  Moved some things to maria-connect.test
  Updared results as REPAIR now works
  Added tests for creation option TRANSACTIONAL
mysql-test/r/ps_2myisam.result:
  Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
mysql-test/r/ps_3innodb.result:
  Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
mysql-test/r/ps_4heap.result:
  Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
mysql-test/r/ps_5merge.result:
  Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
mysql-test/r/ps_7ndb.result:
  Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
mysql-test/r/ps_maria.result:
  Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
mysql-test/t/maria.test:
  Moved some things to maria-connect.test
  Updared results as REPAIR now works
  Added tests for creation option TRANSACTIONAL
mysys/mf_iocache.c:
  More debugging
mysys/mf_tempfile.c:
  Added missing close()
mysys/my_error.c:
  init_glob_errs() is now done in my_init()
mysys/my_handler.c:
  Added functions to initialize handler error messages
mysys/my_init.c:
  Moevd init_glob_errs() here.
mysys/my_open.c:
  More comments
  More debugging
  Code cleanup (join multiple code paths) and indentation fixes. No change in logic.
mysys/my_symlink2.c:
  Don't print NULL strings
sql/handler.cc:
  Added printing of PAGE row type
  Moved out initializing of handler errors to allow handler to give better error messages at startup
sql/handler.h:
  ROW_TYPE_PAGES -> ROW_TYPE_PAGE
sql/lex.h:
  Added 'PAGE' and 'TRANSACTIONAL'
sql/mysqld.cc:
  Initialize handler error messages early to get better error messages from handler startup
sql/sql_show.cc:
  ROW_TYPE_PAGES -> ROW_TYPE_PAGE
sql/sql_table.cc:
  Removed not needed initializer
sql/sql_yacc.yy:
  Added CREATE options ROW_FORMAT=PAGE and TRANSACTIONAL=[0|1]
sql/table.cc:
  Store transactional flag in .frm
  More comments
sql-bench/example:
  Better example
sql/table.h:
  Added transactional table option
storage/maria/ha_maria.cc:
  More debug information
  Enable REPAIR
  Detect usage of TRANSACTIONAL table option
storage/maria/ma_bitmap.c:
  More comments (from Guilhem)
storage/maria/ma_blockrec.c:
  SANITY_CHECK -> SANITY_CHECKS (fixed typo)
  Write out pages on delete even if there is no rows. (Fixed problem with REPAIR)
  Removed some ASSERTS to runtime checks (for better REPAIR)
  Fixed bug when scanning rows
  More DBUG information
storage/maria/ma_check.c:
  Partial rewrite to allow REPAIR of BLOCK/PAGE format.
  Repair of BLOCK format rows is for now only done with 'maria_repair()' (= repair through key cache)
  The new logic to repair rows with BLOCK format is:
  
  - Create new, unrelated MARIA_HA of the table
  - Create new datafile and associate it with new handler
  - Reset all statistic information in new handler
  - Copy all data to new handler with normal write operations
  - Move state of new handler to old handler
  - Close new handler
  - Close data file in old handler
  - Rename old data file to new data file.
  - Reopen data file in old handler
storage/maria/ma_close.c:
  REmoved not needed block
storage/maria/ma_create.c:
  Swap arguments to _ma_initialize_data_file()
storage/maria/ma_delete_all.c:
  Split maria_delete_all_rows() to two functions to allow REPAIR to easily reset all status information.
storage/maria/ma_dynrec.c:
  Added checksum argument to _ma_rec_check (multi-thread fix)
storage/maria/ma_info.c:
  Indentation fix
storage/maria/ma_init.c:
  Register error message to get better error message on init and when using as standalone module.
storage/maria/ma_loghandler.c:
  Fixed typo that disabled some error detection by valgrind
storage/maria/ma_open.c:
  Added 'calc_check_checksum()'
  Don't log things during repair
  Added option HA_OPEN_COPY to allow one to open a Maria table with an independent share (required by REPAIR)
storage/maria/ma_pagecache.c:
  Fixed some compiler warnings
  Added support for PAGECACHE_READ_UNKNOWN_PAGE (used for scanning file without knowing page types)
storage/maria/ma_test_all.sh:
  More test of REPAIR
storage/maria/ma_update.c:
  Optimized checksum code
storage/maria/maria_chk.c:
  Use DBUG_SET_INITIAL() to get DBUG to work with --parallel-repair
  Ensure we always use maria_repair() for BLOCK format (for now)
  More DBUG information
storage/maria/maria_def.h:
  For now, always run with more checkings (SANITY_CHECKS)
  Added share->calc_check_checksum to be used with REPAIR / CHECK table.
  Swaped arguments to _ma_initialize_data_file()
storage/myisam/ft_stopwords.c:
  Added DBUG information
mysql-test/r/maria-connect.result:
  New BitKeeper file ``mysql-test/r/maria-connect.result''
mysql-test/t/maria-connect.test:
  New BitKeeper file ``mysql-test/t/maria-connect.test''

creating the data file, and sync this log, so that the table cannot be
used if log record didn't reach disk. The same way, we force the log
in DROP/RENAME TABLE. Also in REPAIR TABLE though logging in this
case is not polished.
Making DELETE FROM t <no WHERE> atomic: we log the record before
starting the operation, and will finish this op at Recovery if needed.


storage/maria/ma_check.c:
  comment. Force the log record for the log to have a complete history.
storage/maria/ma_create.c:
  better conformance to the text of WL#3239 "log CREATE TABLE in Maria":
  write the log record before creating the data file. This ensures
  that the log can be applied to an old backup in all circumstances.
  errpos=2 was wrong.
storage/maria/ma_delete_all.c:
  making DELETE FROM t <no WHERE> atomic: we log the record before
  starting the operation, and will finish the operation at Recovery
  if needed. Thus there is no need to force files to disk.
storage/maria/ma_delete_table.c:
  forcing the log before dropping a table, so that the log has the
  entire history.
storage/maria/ma_loghandler.c:
  LOGREC_REDO_DELETE_ALL needs to set trn's rec_lsn so that the log's
  low-water mark and Checkpoint retain this record until the 
  delete operation has finished.
storage/maria/ma_rename.c:
  force the log before renaming a table, so that the log has a complete
  history.

For this scenario: server crashes (could be because a table is
corrupted) and Recovery repeatedly crashes on this table. User repairs
it with maria_chk (as REPAIR TABLE is not possible), restarts the
server, Recovery runs: for Recovery to not apply old REDOs to this
repaired table (which would fail: rows have moved), maria_chk sets
create_rename_lsn to the max value. Later when the server opens
the table via ha_maria, it sets the LSN to the correct current value.


storage/maria/ma_check.c:
  using helper function
storage/maria/ma_create.c:
  A new helper function which stores the create_rename_lsn 
  into the table's header on disk when we cannot wait for this to happen
  naturally at a later _ma_state_info_write().
storage/maria/ma_delete_all.c:
  using helper function; so log_data now can be FILEID_STORE_SIZE.
storage/maria/ma_open.c:
  When opening a transactional table in the server, we discover
  if it has been repaired with maria_chk and if yes, give it a correct
  create_rename_lsn.
storage/maria/ma_rename.c:
  using helper function
storage/maria/maria_chk.c:
  By setting create_rename_lsn to the maximum possible LSN, maria_chk
  ensures that old REDOs are not applied to the new table it is
  going to produce.
storage/maria/maria_def.h:
  new helper function

comments; remember the UNDO's LSN for storing it in pages when
executing REDO's (to imitate what the runtime code does)


storage/maria/maria_read_log.c:
  comments; remember the UNDO's LSN for storing it in pages when
  executing REDO's (to imitate what the runtime code does)

fixes for build failures; copyrights; small bugfixes and comments


mysys/Makefile.am:
  missing .h breaks building from tarball
storage/maria/ma_loghandler.c:
  applying Serg's bugfix of trnman_new_trid() to translog_assign_id_to_share()
storage/maria/ma_loghandler.h:
  copyright
storage/maria/ma_loghandler_lsn.h:
  copyright
storage/maria/maria_read_log.c:
  fix for compiler warnings. Comments.
  Close tables when program ends.

storage/maria/maria_read_log.c:
  assertions to protect against future bugs (especially, to ensure
  that replaying DROP TABLE, if implemented, wouldn't leave open tables
  behind it)

- new program maria_read_log to display and apply log records
found in a Maria log (see file's revision comment)
- minor, misc fixes


storage/maria/Makefile.am:
  new program maria_read_log
storage/maria/ha_maria.cc:
  create control file if missing
storage/maria/ma_blockrec.c:
  0 -> LSN_IMPOSSIBLE; comments
storage/maria/ma_checkpoint.h:
  preparations for Checkpoint module
storage/maria/ma_close.c:
  comment
storage/maria/ma_control_file.c:
  renaming constants.
  Possibility to say "open control file but don't create it if it's
  missing" (used by maria_read_log which does not want to create
  anything)
storage/maria/ma_control_file.h:
  renaming constants
storage/maria/ma_create.c:
  I had duplicated "linkname" and "linkname_ptr", now I see it's not
  needed, reverting. Indeed those variables don't contain interesting
  information; fixing log record accordingly (the links are in
  ci->data/index_file_name). Storing keystart in log record is needed,
  to know at which size we must extend the file if we replay
  LOGREC_CREATE_TABLE.
storage/maria/ma_loghandler.c:
  some structures need to be known to maria_read_log.c, taking
  them to ma_loghandler.h
storage/maria/ma_loghandler.h:
  we have page_store, adding page_korr.
  translog_lock() made public, because Checkpoint will need it (to
  write to control file).
  Some structures moved from ma_loghandler.c because maria_read_log.c
  needs them (needs to know the execute-in-REDO-phase hooks of each
  record).
storage/maria/ma_loghandler_lsn.h:
  constants defined in ma_control_file.h serve everywhere,
  and they relate to LSNs, so putting them in ma_loghandler_lsn.h.
  Stronger constraints in LSN_VALID().
storage/maria/ma_pagecache.c:
  renaming constants
storage/maria/ma_recovery.h:
  copyright
storage/maria/ma_test1.c:
  new prototype
storage/maria/ma_test2.c:
  new prototype
storage/maria/trnman_public.h:
  double-inclusion safe
storage/maria/unittest/ma_control_file-t.c:
  constants renamed, new prototype
storage/maria/unittest/ma_test_loghandler-t.c:
  constants renamed, new prototype
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  constants renamed, new prototype
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  constants renamed, new prototype
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
  constants renamed, new prototype
storage/myisam/mi_close.c:
  comment
storage/maria/maria_read_log.c:
  program to read and print log records from a Maria transaction log,
  and optionally apply them to tables. Very basic, early version.
  Should serve as a base for Recovery's code. Designed to be idempotent.
  Create a log by running maria.test, then cd to var/master-data
  and run "maria_read_log --only-display" to see info about records;
  run "maria_read_log --display-and-apply" to also apply the records
  to tables (it's more interesting if you first wipe out the
  tables in var/master-data/test, to see how they get re-created).
  Only a few records are handled by now: LONG_TRANSACTION_ID,
  COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for
  REDO_INSERT_ROW_HEAD where I could use Monty's help (search for
  "Monty" in the file). Note: changes to the index pages, index's header
  and bitmap pages are not properly logged yet, so don't expect
  the program to work with that.

storage/maria/ma_loghandler.c:
  Structure and function to read loghandler file data added.
  Creation of new log when maria change version added.

- WL#3240 "log DROP TABLE in Maria"
- similarly, log RENAME TABLE, REPAIR/OPTIMIZE TABLE, and
DELETE no_WHERE_clause (== the DELETE which just truncates the files)
- create_rename_lsn added to MARIA_SHARE's state
- all these operations (except DROP TABLE) also update the table's
create_rename_lsn, which is needed for the correctness of
Recovery (see function comment of _ma_repair_write_log_record()
in ma_check.c)
- write a COMMIT record when transaction commits.
- don't log REDOs/UNDOs if this is an internal temporary table
like inside ALTER TABLE (I expect this to be a big win). There was
already no logging for user-created "CREATE TEMPORARY" tables.
- don't fsync files/directories if the table is not transactional
- in translog_write_record(), autogenerate a 2-byte-id for the table
and log the "id->name" pair (LOGREC_FILE_ID); log
LOGREC_LONG_TRANSACTION_ID; automatically store
the table's 2-byte-id in any log record.
- preparations for Checkpoint: translog_get_horizon(); pausing Checkpoint
when some dirty pages are unknown; capturing trn->rec_lsn,
trn->first_undo_lsn for Checkpoint and log's low-water-mark computing.
- assertions, comments.


storage/maria/Makefile.am:
  more files to build
storage/maria/ha_maria.cc:
  - logging a REPAIR log record if REPAIR/OPTIMIZE was successful.
  - ha_maria::data_file_type does not have to be set in every info()
  call, just do it once in open().
  - if caller said that transactionality can be disabled (like if
  caller is ALTER TABLE) i.e. thd->transaction.on==FALSE, then we
  temporarily disable transactionality of the table in external_lock();
  that will ensure that no REDOs/UNDOs are logged for this possibly
  massive write operation (they are not needed, as if any write fails,
  the table will be dropped). We re-enable in external_lock(F_UNLCK),
  which in ALTER TABLE happens before the tmp table replaces the original
  one (which is good, as thus the final table will have a REDO RENAME
  and a correct create_rename_lsn).
  - when we commit we also have to write a log record, so
  trnman_commit_trn() calls become ma_commit() calls
  - at end of engine's initialization, we are potentially entering a
  multi-threaded dangerous world (clients are going to be accepted)
  and so some assertions of mutex-owning become enforceable, for that
  we set maria_multi_threaded=TRUE (see ma_control_file.c)
storage/maria/ha_maria.h:
  new member ha_maria::save_transactional (see also ha_maria.cc)
storage/maria/ma_blockrec.c:
  - fixing comments according to discussion with Monty
  - if a table is transactional but temporarily non-transactional
  (like in ALTER TABLE), we need to give a sensible LSN to the pages
  (and, if we give 0, pagecache asserts).
  - translog_write_record() now takes care of storing the share's
  2-byte-id in the log record
storage/maria/ma_blockrec.h:
  fixing comment according to discussion with Monty
storage/maria/ma_check.c:
  When REPAIR/OPTIMIZE modify the data/index file, if this is a
  transactional table, they must sync it; if they remove files or rename
  files, they must sync the directory, so that everything is durable.
  This is just applying to REPAIR/OPTIMIZE the logic already implemented
  in CREATE/DROP/RENAME a few months ago.
  Adding a function to write a LOGREC_REPAIR_TABLE at end of
  REPAIR/OPTIMIZE (called only by ha_maria, not by maria_chk), and
  to update the table's create_rename_lsn.
storage/maria/ma_close.c:
  fix for a future bug
storage/maria/ma_control_file.c:
  ensuring that if Maria is running in multi-threaded mode, anybody
  wanting to write to the control file and update
  last_checkpoint_lsn/last_logno owns the log's lock.
storage/maria/ma_control_file.h:
  see ma_control_file.c
storage/maria/ma_create.c:
  when creating a table:
  - sync it and its directory only if this is a transactional table
  and there is a log (no point in syncing in maria_chk)
  - decouple the two uses of linkname/linkname_ptr (for index file and
  for data file) into more variables, as we need to know all links
  until the moment we write the LOGREC_CREATE_TABLE.
  - set share.data_file_type early so that _ma_initialize_data_file()
  knows it (Monty's bugfix so that a table always has at least a bitmap
  page when it is created; so data-file is not 0 bytes anymore).
  - log a LOGREC_CREATE_TABLE; it contains the bytes which we have
  just written to the index file's header. Update table's
  create_rename_lsn.
  - syncing of kfile had been bugified in a previous merge, correcting
  - syncing of dfile is now needed as it's not empty anymore
  - in _ma_initialize_data_file(), use share's block_size and not the
  global one. This is a gratuitous change, both variables are equal,
  just that I find it more future-proof to use share-bound variable
  rather than global one.
storage/maria/ma_delete_all.c:
  log a LOGREC_DELETE_ALL record when doing ma_delete_all_rows();
  update create_rename_lsn then.
storage/maria/ma_delete_table.c:
  - logging LOGREC_DROP_TABLE; knowing if this is needed, requires
  knowing if the table is transactional, which requires opening the
  table.
  - we need to sync directories only if the table is transactional
storage/maria/ma_extra.c:
  questions
storage/maria/ma_init.c:
  when maria_end() is called, engine is not multithreaded
storage/maria/ma_loghandler.c:
  - translog_inited has to be visible to ma_create() (see how it is used
  in ma_create())
  - checkpoint record will be a single record, not three
  - no REDO for TRUNCATE (TRUNCATE calls ma_create() internally so will
  log a REDO_CREATE)
  - adding REDO for DELETE no_WHERE_clause (fast DELETE of all rows by
  truncating the files), REPAIR.
  - MY_WAIT_IF_FULL to wait&retry if a log write hits a full disk
  - in translog_write_record(), if MARIA_SHARE does not yet have a
  2-byte-id, generate one for it and log LOGREC_FILE_ID; automatically
  store this short id into log records.
  - in translog_write_record(), if transaction has not logged its
  long trid, log LOGREC_LONG_TRANSACTION_ID.
  - For Checkpoint, we need to know the current end-of-log: adding
  translog_get_horizon().
  - For Control File, adding an assertion that the thread owns the
  log's lock (control file is protected by this lock)
storage/maria/ma_loghandler.h:
  Changes in log records (see ma_loghandler.c).
  new prototypes, new functions.
storage/maria/ma_loghandler_lsn.h:
  adding a type LSN_WITH_FLAGS especially for TRN::first_undo_lsn,
  where the most significant byte is used for flags.
storage/maria/ma_open.c:
  storing the create_rename_lsn in the index file's header (in the
  state, precisely) and retrieving it from there.
storage/maria/ma_pagecache.c:
  - my set_if_bigger was wrong, correcting it
  - if the first_in_switch list is not empty, it means that
  changed_blocks misses some dirty pages, so Checkpoint cannot run and
  needs to wait. A variable missing_blocks_in_changed_list is added to
  tell that (should it be named missing_blocks_in_changed_blocks?)
  - pagecache_collect_changed_blocks_with_lsn() now also tells the
  minimum rec_lsn (needed for low-water mark computation).
storage/maria/ma_pagecache.h:
  see ma_pagecache.c
storage/maria/ma_panic.c:
  comment
storage/maria/ma_range.c:
  comment
storage/maria/ma_rename.c:
  - logging LOGREC_RENAME_TABLE; knowing if this is needed, requires
  knowing if the table is transactional, which requires opening the
  table.
  - update create_rename_lsn
  - we need to sync directories only if the table is transactional
storage/maria/ma_static.c:
  comment
storage/maria/ma_test_all.sh:
  - tip for Valgrind-ing ma_test_all
  - do "export maria_path=somepath" before calling ma_test_all,
  if you want to run ma_test_all out of storage/maria (useful
  to have parallel runs, like one normal and one Valgrind, they
  must not use the same tables so need to run in different directories)
storage/maria/maria_def.h:
  - state now contains, in memory and on disk, the create_rename_lsn
  - share now contains a 2-byte-id
storage/maria/trnman.c:
  preparations for Checkpoint: capture trn->rec_lsn, trn->first_undo_lsn;
  minimum first_undo_lsn needed to know log's low-water-mark
storage/maria/trnman.h:
  using most significant byte of first_undo_lsn to hold miscellaneous
  flags, for now TRANSACTION_LOGGED_LONG_ID.
  dummy_transaction_object is already declared in ma_static.c.
storage/maria/trnman_public.h:
  dummy_transaction_object was declared in all files including
  trnman_public.h, while in fact it's a single object.
  new prototype
storage/maria/unittest/ma_test_loghandler-t.c:
  update for new prototype
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  update for new prototype
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  update for new prototype
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
  update for new prototype
storage/maria/ma_commit.c:
  function which wraps:
  - writing a LOGREC_COMMIT record (==commit on disk)
  - calling trnman_commit_trn() (=commit in memory)
storage/maria/ma_commit.h:
  new header file
.tree-is-private:
  this file is now needed to keep our tree private (don't push it
  to public trees). When 5.1 is merged into mysql-maria, we can abandon
  our maria-specific post-commit trigger; .tree_is_private will take
  care of keeping commit mails private. Don't push this file to public
  trees.

without the fix, only 896 rows were inserted into t2 in maria-big.test.


storage/maria/ma_blockrec.c:
  due to wrong test we were skipping some rows when scanning
mysql-test/r/maria-big.result:
  result for new test
mysql-test/t/maria-big.test:
  test for a bug where we missed some rows when scanning

storage/maria/unittest/ma_test_loghandler-t.c:
  Spaces at the end of the line removed.
  Parameters of translog_write_record() fixed.
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  Parameters of translog_write_record() fixed.

into  desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria-lsn


storage/maria/ma_loghandler.h:
  Auto merged
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  Auto merged
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
  Auto merged
storage/maria/ma_loghandler.c:
  merge
storage/maria/unittest/ma_test_loghandler-t.c:
  merge
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  merge

  relative LSN independed on "distance".
Added support for test record descriptors to avoid
  interfere woth real record descriptors.
Fixed descriptor of pseudofixed length record length,
  now it is length of record passed from/to client of
  the loghandler.


BitKeeper/etc/ignore:
  Added storage/maria/unittest/ma_test_loghandler_long-t-big to the ignore list
storage/maria/ma_init.c:
  Removed loghandler_init call because it is present in translog_init()
storage/maria/ma_loghandler.c:
  Fixed LSN codding to allow code all
    relative LSN independed on "distance".
  Added support for test record descriptors to avoid
    interfere woth real record descriptors.
  Fixed length of LOGREC_REDO_INSERT_ROW_TAIL.
  Fixed descriptor of pseudofixed length record length,
    now it is length of record passed from/to client of
    the loghandler.
storage/maria/ma_loghandler.h:
  Added support for test record descriptors to avoid
    interfere woth real record descriptors.
storage/maria/unittest/Makefile.am:
  Made new test for log with reference over 63 files.
  Layout fixed.
storage/maria/unittest/ma_test_loghandler-t.c:
  Added support for test record descriptors to avoid
    interfere woth real record descriptors.
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  Added support for test record descriptors to avoid
    interfere woth real record descriptors.
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  Added support for test record descriptors to avoid
    interfere woth real record descriptors.
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
  Added support for test record descriptors to avoid
    interfere woth real record descriptors.

into  gbichot3.local:/home/mysql_src/mysql-maria


storage/maria/ma_loghandler.c:
  Auto merged

specific
- adding TRN::first_undo_lsn, needed to know when a log can be deleted;
this variable must be set under log's mutex and that leads to setting
TRN::rec_lsn, TRN::undo_lsn and TRN::first_undo_lsn in a
inwrite_rec_hook; adding implementation of one hook for REDOs and one
for UNDOs. Thus translog_write_record() always uses TRN and so does
not need a short_id argument, can find it from TRN.
- Monty's patch for the last Valgrind error in the tree.
- Log handler's unit tests fail but Sanja says it's known


include/Makefile.am:
  pagecache.h moved and renamed
include/maria.h:
  pagecache.h moved and renamed
sql/handler.h:
  pagecache.h moved and renamed
storage/maria/Makefile.am:
  pagecache.h moved and renamed
storage/maria/ha_maria.cc:
  adding an assertion which sounds logical
storage/maria/ma_blockrec.c:
  trn->rec_lsn and trn->undo_lsn are now set via hooks inside the log
  record's writing; this allows to also set trn->first_undo_lsn
  needed to compute the log's low-water mark.
  The PAGERANGE_STORE_SIZE -> PAGE_STORE_SIZE is Monty's fix to a
  Valgrind error.
storage/maria/ma_loghandler.c:
  "tcb" renamed to "trn". Log handler now knows what is a transaction,
  and finds short_id from trn. trn's rec_lsn, undo_lsn, first_undo_lsn
  are now set by some inwrite_rec_hookS (one for REDOs, one for UNDOs).
  The HAVE_purify blocks are Monty's fix to a Valgrind error.
storage/maria/ma_loghandler.h:
  Log handler functions use TRN, that needs a forward declaration
storage/maria/ma_pagecache.c:
  pagecache.h was moved and renamed
storage/maria/ma_pagecache.h:
  pagecache.h was moved and renamed
storage/maria/ma_pagecaches.c:
  pagecache.h was moved and renamed
storage/maria/trnman.c:
  initializing some members of TRN.
storage/maria/trnman.h:
  TRN::first_undo_lsn needed for log's low-water mark calculation
  (which will serve to know which logs can be deleted)
storage/maria/unittest/ma_test_loghandler-t.c:
  translog_write_record() now needs a valid TRN
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  translog_write_record() now needs a valid TRN
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  translog_write_record() now needs a valid TRN
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
  translog_write_record() now needs a valid TRN
storage/maria/unittest/test_file.h:
  pagecache.h was moved and renamed

storage/maria/ma_loghandler.c:
  Variable length record descriptor fixed.
  Assignment of file number moved out of the loop.

Fixed bug in ma_dbug.c that gave valgrind warning (only relevant when using --debug)
Fixed bug in blob logging (Fixes valgrind warning)
maria_getint() -> maria_data_on_page()


mysys/safemalloc.c:
  Added debug function to print out where a piece of memory was allocated
sql/opt_range.cc:
  Remove DBUG_PRINT of unitailized memory
storage/maria/ma_blockrec.c:
  Fixed bug in blob logging
storage/maria/ma_check.c:
  Fixed compiler warning
storage/maria/ma_dbug.c:
  Added missed end++; Caused usage of unitialized memory for nullable keys that was not NULL
storage/maria/ma_delete.c:
  maria_getint() -> maria_data_on_page()
storage/maria/ma_init.c:
  Added header file to get rid of warning
storage/maria/ma_key.c:
  More debugging
storage/maria/ma_loghandler.c:
  Removed some wrong ';' to get rid of compiler errors when compiling without debugging
  Indentation fixes
  Removed not needed 'break's
  Fixed some compiler warnings
  Added code to detect logging of unitialized memory
storage/maria/ma_page.c:
  maria_getint() -> maria_data_on_page()
  Clear rest of index page before writing when used with valgrind
  (Fixes warning of writing pages with unitialized data)
storage/maria/ma_range.c:
  maria_getint() -> maria_data_on_page()
storage/maria/ma_rt_index.c:
  maria_getint() -> maria_data_on_page()
storage/maria/ma_rt_index.h:
  maria_getint() -> maria_data_on_page()
storage/maria/ma_rt_key.c:
  maria_getint() -> maria_data_on_page()
storage/maria/ma_rt_split.c:
  maria_getint() -> maria_data_on_page()
storage/maria/ma_search.c:
  maria_getint() -> maria_data_on_page()
storage/maria/ma_test1.c:
  Fixed compiler warning
storage/maria/ma_write.c:
  maria_getint() -> maria_data_on_page()
storage/maria/maria_chk.c:
  maria_getint() -> maria_data_on_page()
storage/maria/maria_def.h:
  maria_getint() -> maria_data_on_page()
storage/maria/unittest/ma_pagecache_consist.c:
  Fixed compiler warning
storage/maria/unittest/ma_pagecache_single.c:
  Fixed compiler warning
storage/maria/unittest/ma_test_loghandler-t.c:
  Fixed compiler warning
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  Fixed compiler warning
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  Fixed compiler warning
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
  Fixed compiler warning
storage/myisam/mi_dbug.c:
  Added missed end++; Caused usage of unitialized memory for nullable keys that was not NULL

- if table is temporary it's not crash-safe so we declare it
non-transactional (saves trnman calls, REDO/UNDO log writing,
and fixes the assertion failure at the first line of trnman_destroy()).


storage/maria/ha_maria.cc:
  if table is not transactional, no need to create a transaction:
  - it saves trnman calls (mutex locks etc)
  - it saves REDO and UNDO log writing
  - it closes a bug: if this is a temporary table, external_lock(F_RD|WRLCK)
  is not always paired with external_lock(F_UNLCK), which confuses the
  transaction logic in external_lock. As temp tables are not crash-safe
  and so not transactional in this Maria version, we skip transactions
  and de-confuse. Note that maria_lock_database(F_UNLCK) is
  properly called, so if the transaction logic moves from external_lock()
  to maria_lock_database() (probably TODO), transactional temp tables
  will be possible.
storage/maria/ma_create.c:
  temporary tables cannot be crash-safe as they are dropped at restart
storage/maria/maria_def.h:
  comment

make Maria support multiple calls to rnd_init() without an rnd_end()
call in between.


storage/maria/ma_blockrec.c:
  as explained in sql/handler.h, multiple calls to rnd_init() without
  a rnd_end() in between, are possible, and engine must be prepared to
  that. So in _ma_scan_init_block_record(), we allocate a buffer
  only if we have not yet one.

storage/maria/ha_maria.cc:
  initialize ha_maria::data_file_type when opening the table
  (it was not initialized, causing a Valgrind error)
storage/maria/ma_locking.c:
  if it's impossible, let's test it

allocated at startup).


storage/maria/ma_init.c:
  destroy transaction manager and log's pagecache when Maria shuts down
storage/maria/trnman.c:
  short_trid_to_active_trn!=NULL now serves to detect if
  trnman has initialized some objects and so if trnman_destroy()
  has objects to destroy. In other words, short_trid_to_active_trn
  serves as "trnman_inited" variable.
  trnman_destroy() is always called by maria_end(), but trnman_init() is
  not always called (for example in ma_test1), that's why
  trnman_destroy() cannot blindly destroy.

into  mysql.com:/home/my/mysql-maria


storage/maria/ha_maria.cc:
  Auto merged
storage/maria/ma_pagecache.c:
  Auto merged

Added maria_clone(), needed by future REPAIR code


storage/maria/unittest/ma_pagecache_consist.c:
  Change mode to -rw-rw-r--
storage/maria/unittest/lockman-t.c:
  Change mode to -rw-rw-r--
storage/maria/unittest/lockman1-t.c:
  Change mode to -rw-rw-r--
storage/maria/unittest/lockman2-t.c:
  Change mode to -rw-rw-r--
storage/maria/unittest/trnman-t.c:
  Change mode to -rw-rw-r--
include/maria.h:
  Added prototype for maria_clone (for future)
storage/maria/ha_maria.cc:
  Move filename to share structure
storage/maria/ma_blockrec.c:
  rec_lsn (first REDO LSN( is now given to the page cache on unpinning
  Removed impossible lock handling in get_head_or_tail_page()
  Changed calls ot translog_write_record() to remember rec_lsn
  Removed some logging in csse of not transactions
storage/maria/ma_delete.c:
  info->filename -> info->s->open_file_name
storage/maria/ma_loghandler.c:
  Indentation fixes
storage/maria/ma_open.c:
  Added maria_clone(), needed by future REPAIR code
storage/maria/ma_packrec.c:
  Fixed typo in comment
storage/maria/ma_pagecache.c:
  Added comment.
  Allow setting REC_LSN in case of read lock
storage/maria/ma_update.c:
  info->filename -> info->s->open_file_name
storage/maria/ma_write.c:
  info->filename -> info->s->open_file_name
storage/maria/maria_def.h:
  info->filename -> info->s->open_file_name
  Added have_rtree to simplify test in ma_clone()
storage/maria/maria_ftdump.c:
  info->filename -> info->s->open_file_name
storage/maria/maria_pack.c:
  info->filename -> info->s->open_file_name
storage/maria/trnman.h:
  Added rec_lsn

into  desktop.sanja.is.com.ua:/home/bell/mysql/bk/mysql-maria

storage/maria/ma_pagecache.c:
  - remove_reader() call is removed from unlock/unpin operations
    which uses direct link (because find_block() was not called)
  - patch which broke pagecache unittest is reverted

unittest/mysys/Makefile.am:
  build unit tests (they were lost in a bad merge)
unittest/mysys/my_atomic-t.c:
  update to the new lf_pinbox* API

into  mysql.com:/home/my/mysql-maria


storage/maria/ma_blockrec.c:
  Auto merged

storage/maria/ma_blockrec.c:
  Reset undo_lsn if not transactional table
  (Avoids assert() when checking LSN in pagecache code)
storage/maria/ma_create.c:
  Don't convert simple FIXED size tables to BLOCK format.
storage/maria/trnman.c:
  Reset undo_lsn

into  desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-pagecache


storage/maria/unittest/ma_pagecache_consist.c:
  Auto merged
include/pagecache.h:
  Merge
storage/maria/ma_pagecache.c:
  Merge
storage/maria/unittest/ma_pagecache_single.c:
  Merge

Some fixes from Sanja


BitKeeper/etc/ignore:
  added storage/maria/maria_log.*
include/pagecache.h:
  Always have enum PAGECACHE_EMPTY_PAGE available
  (Simpler code)
storage/maria/ma_bitmap.c:
  Reset 'debugging' bitmap when creating new one (fixes valgrind warning)
storage/maria/ma_blockrec.c:
  Removed duplicate (wrong) initialization
  Reset not initialized variable
storage/maria/ma_check.c:
  Use right page type
  (Patch from Sanja)
storage/maria/ma_init.c:
  Reset logging in maria_end()
  (Fixes memory leak)
storage/maria/ma_loghandler.c:
  Add missing copyright header
  Added checking of duplicate calls or calls without init to translog_destroy()
  Don't lock mutex before destroying them (not needed as you can't use a destroyed mutex anyway)
storage/maria/ma_pagecache.c:
  Added extra page type text
  Trivial indentation fixes
storage/maria/ma_test1.c:
  Added transaction setup
  (Patch from Sanja)
storage/maria/ma_test2.c:
  Added transaction setup
  (Patch from Sanja)

Added logging and pinning of pages to block format.
Integration of transaction manager, log handler.
Better page cache intergration
Split trnman.h into two files, so that we don't have to include my_atomic.h into C++ programs.
Renaming of structures, more comments, more debugging etc.
Fixed problem with small head block + long varchar.
Added extra argument to delete_record() and update_record() (needed for UNDO logging)
Small changes to interface of pagecache and log handler.
Change initialization of log_record_type_descriptors to not be depending on enum order.
Use array of LEX_STRING's to send data to log handler
Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.


include/lf.h:
  Interface fixes
  Rename of structures
  (Patch from Sergei via Sanja)
include/my_atomic.h:
  More comments
include/my_global.h:
  Added MY_ERRPTR
include/pagecache.h:
  Added undo LSN when unlocking pages
mysql-test/r/maria.result:
  Updated results
mysql-test/t/maria.test:
  Added autocommit around lock tables
  (Patch from Sanja)
mysys/lf_alloc-pin.c:
  Post-review fixes, simple optimizations
  More comments
  Struct slot renames
  Check amount of memory on stack
  (Patch from Sergei)
mysys/lf_dynarray.c:
  More comments
mysys/lf_hash.c:
  More comments
  After review fixes
  (Patch from Sergei)
storage/maria/ha_maria.cc:
  Split trnman.h into two files, so that we don't have to include my_atomic.h into the .cc program.
  (Temporary fix to avoid bug in gcc)
  Move out all deferencing of the transaction structure.
  Transaction manager integrated (Patch from Sergei)
storage/maria/ha_maria.h:
  Added prototype for start_stmt()
storage/maria/lockman.c:
  Function call rename
storage/maria/ma_bitmap.c:
  Mark deleted pages free from page cache
storage/maria/ma_blockrec.c:
  Offset -> rownr
  More debugging
  Fixed problem with small head block + long varchar
  Added logging of changed pages
  Added logging of undo (Including only loggging of changed fields in case of update)
  Added pinning/unpinning of all changed pages
  More comments
  Added free_full_pages() as the same code was used in several places.
  fill_rows_parts() renamed as fill_insert_undo_parts()
  offset -> rownr
  Added some optimization of not transactional tables
  _ma_update_block_record() has new parameter, as we need original row to do efficent undo for update
storage/maria/ma_blockrec.h:
  Added ROW_EXTENTS_ON_STACK
  Changed prototype for update and delete of row
storage/maria/ma_check.c:
  Added original row to delete_record() call
storage/maria/ma_control_file.h:
  Added ifdefs for C++
storage/maria/ma_delete.c:
  Added original row to delete_record() call
  (Needed for efficent undo logging)
storage/maria/ma_dynrec.c:
  Added extra argument to delete_record() and update_record()
  Removed not used variable
storage/maria/ma_init.c:
  Initialize log handler
storage/maria/ma_loghandler.c:
  Removed not used variable
  Change initialization of log_record_type_descriptors to not be depending on enum order
  Use array of LEX_STRING's to send data to log handler
storage/maria/ma_loghandler.h:
  New defines
  Use array of LEX_STRING's to send data to log handler
storage/maria/ma_open.c:
  Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.
  Store in MARIA_SHARE->page_type if pages will have up to date LSN's
storage/maria/ma_pagecache.c:
  Don't decrease number of readers when using pagecache_write()/pagecache_read()
  In pagecache_write() decrement request count if page was left pinned
  Added pagecache_delete_pages()
  Removed some casts
  Make trace output consistent with rest of code
  Simplify calling of DBUG_ASSERT(0)
  Only update LSN if the LSN is bigger than what's already on the page
  Added LSN parameter pagecache_unpin_page(), pagecache_unpin(), and pagecache_unlock()
  (Part of patch from Sanja)
storage/maria/ma_static.c:
  Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.
  Added default page cache
storage/maria/ma_statrec.c:
  Added extra argument to delete_record() and update_record()
storage/maria/ma_test1.c:
  Added option -T for transactions
storage/maria/ma_test2.c:
  Added option -T for transactions
storage/maria/ma_test_all.sh:
  Test with transactions
storage/maria/ma_update.c:
  Changed prototype for update of row
storage/maria/maria_def.h:
  Changed prototype for update & delete of row as block records need to access the old row
  Store in MARIA_SHARE->page_type if pages will have up to date LSN's
  Added MARIA_MAX_TREE_LEVELS to allow us to calculate the number of possible pinned pages we may need.
  Removed not used 'empty_bits_buffer'
  Added pointer to transaction object
  Added array for pinned pages
  Added log_row_parts array for logging of field data.
  Added MARIA_PINNED_PAGE to store pinned pages
storage/maria/trnman.c:
  Added accessor functions to transaction object
  Added missing DBUG_RETURN()
  More debugging
  More comments
  Changed // comment of code to #ifdef NOT_USED
  Transaction manager integrated.
  Post review fixes
  Part of patch originally from Sergei
storage/maria/trnman.h:
  Split trnman.h into two files, so that we don't have to include my_atomic.h into the .cc program.
  (Temporary fix to avoid bug in gcc)
storage/maria/unittest/ma_pagecache_single.c:
  Added missing argument
  Added SKIP_BIG_TESTS
  (Patch from Sanja)
storage/maria/unittest/ma_test_loghandler-t.c:
  Test logging with new LEX_STRING parameter
  (Patch from Sanja)
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  Test logging with new LEX_STRING parameter
  (Patch from Sanja)
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  Test logging with new LEX_STRING parameter
  (Patch from Sanja)
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
  Test logging with new LEX_STRING parameter
  (Patch from Sanja)
storage/maria/unittest/trnman-t.c:
  Stack overflow detection
  (Patch from Sergei)
unittest/unit.pl:
  Command-line options --big and --verbose
  (Patch from Sergei)
unittest/mytap/tap.c:
  Detect --big
  (Patch from Sergei)
unittest/mytap/tap.h:
  Skip_big_tests and SKIP_BIG_TESTS
  (Patch from Sergei)
storage/maria/trnman_public.h:
  New BitKeeper file ``storage/maria/trnman_public.h''

storage/maria/ma_bitmap.c:
  Set 'first_bitmap_with_space' to point at first page, if it's was not set before.
  (This is needed as we are not anymore reading the first bitmap into memory on startup)
  Fixed some bugs with full bitmaps and changing bitmaps that was unconvered while finding and fixing
  the above problem.

Read blocks through page cache in check_block_record()
Don't read first bitmap on ma_open()
Don't require that a files block_size is equal to maria_block_size, if page cache is not setup yet.
Changed ma_test1, ma_test2, maria_chk and maria_pack to always create a page cache.
The above fixes so that ma_test_all now works again



BitKeeper/etc/ignore:
  added storage/maria/unittest/ma_pagecache_consist_1k-t-big storage/maria/unittest/ma_pagecache_consist_1kHC-t-big storage/maria/unittest/ma_pagecache_consist_1kRD-t-big storage/maria/unittest/ma_pagecache_consist_1kWR-t-big storage/maria/unittest/ma_pagecache_consist_64k-t-big storage/maria/unittest/ma_pagecache_consist_64kHC-t-big storage/maria/unittest/ma_pagecache_consist_64kRD-t-big storage/maria/unittest/ma_pagecache_consist_64kWR-t-big storage/maria/unittest/ma_pagecache_single_64k-t-big
include/maria.h:
  Added MARIA_MIN_PAGE_CACHE_SIZE
include/pagecache.h:
  Filedescriptors should be of type File
storage/maria/ma_bitmap.c:
  After merge fixes
  Create dummy bitmap on startup (can't read first bitmap becasue page cache may not be set up yet)
storage/maria/ma_blockrec.c:
  After merge fixes
storage/maria/ma_check.c:
  Use page cache to read rows-in-block rows.
  Don't initialize page cache; It's now done in maria_chk
storage/maria/ma_dynrec.c:
  Trivial code reorganization
storage/maria/ma_open.c:
  Don't give error for conflicting block size if page cache is not initalized.
  (Needed for maria_chk to be able to work on tables with different page sizes)
  After merge fixes
storage/maria/ma_page.c:
  Fix compiler warning
  Remove net needed asserts (Guranteed by ma_create())
storage/maria/ma_pagecache.c:
  Allow one to create a page cache with just one block
  (For trivail scan of table)
  Trivial code simplication
storage/maria/ma_test1.c:
  Always create a page cache (Maria now requires a page cache to work)
storage/maria/ma_test2.c:
  Always create a page cache (Maria now requires a page cache to work)
storage/maria/maria_chk.c:
  Remove command line options --maria_block_size and --pagecache_block_size.
  Set the global maria_block_size from the data file. This allows maria_chk to work with tables of different block sizes.
  Simply DESCRIPT handling; Allows us to remove one indentation level in maria_chk().
  Always initialize page cache if we are doing check/repair.
  (Most of the patch is reindentation of the code)
storage/maria/maria_def.h:
  After merge fix
storage/maria/maria_pack.c:
  Set maria_block_size based on the files block_size.
  Initalize page cache (needed for getting rows-in-blocks to works)

into  mysql.com:/home/my/mysql-maria


include/maria.h:
  Auto merged
storage/maria/ha_maria.cc:
  Auto merged
storage/maria/ma_bitmap.c:
  Auto merged
storage/maria/ma_check.c:
  Auto merged
storage/maria/ma_dynrec.c:
  Auto merged
storage/maria/ma_info.c:
  Auto merged
storage/maria/ma_open.c:
  Auto merged
storage/maria/ma_packrec.c:
  Auto merged
storage/maria/ma_page.c:
  Auto merged
storage/maria/ma_test1.c:
  Auto merged
storage/maria/ma_test2.c:
  Auto merged
storage/maria/ma_write.c:
  Auto merged
storage/maria/maria_chk.c:
  Auto merged
storage/maria/maria_pack.c:
  Auto merged
storage/maria/ma_blockrec.c:
  Manual merge
storage/maria/ma_close.c:
  Manual merge
storage/maria/maria_def.h:
  Manual merge