- 05 Jan, 2017 5 commits
-
-
Marko Mäkelä authored
logs_empty_and_mark_files_at_shutdown(): Wait for the log_scrub_thread to exit.
-
Marko Mäkelä authored
Most of them are trivial, except for the thread_sync_t refactoring. We must not invoke memset() on non-POD objects. mtflush_work_initialized: Remove. Refer to mtflush_ctx != NULL instead. thread_sync_t::thread_sync_t(): Refactored from buf_mtflu_handler_init(). thread_sync_t::~thread_sync_t(): Refactored from buf_mtflu_io_thread_exit().
-
Marko Mäkelä authored
MariaDB Server is unnecessarily evaluating the arguments of DBUG_PRINT() macros when the label is not defined. The macro DBUG_LOG() for C++ operator<< output which was added for InnoDB diagnostics in MySQL 5.7 is missing from MariaDB. Unlike the MySQL 5.7 implementation, MariaDB will avoid allocating and initializing the output string when the label is not defined. Introduce DBUG_OUT("crypt") and DBUG_OUT("checksum") for some InnoDB diagnostics, replacing some use of ib::info().
-
Marko Mäkelä authored
Most conflicts are related to the MDEV-11638 InnoDB shutdown refactoring.
-
Marko Mäkelä authored
In the backport of Bug#24450908 UNDO LOG EXISTS AFTER SLOW SHUTDOWN from MySQL 5.7 to the MySQL 5.6 based MariaDB Server 10.1, we must use a mutex when HAVE_ATOMIC_BUILTINS is not defined. Also, correct a function comment. In MySQL 5.6 and MariaDB Server 10.1, also temporary InnoDB tables are redo-logged.
-
- 04 Jan, 2017 5 commits
-
-
Igor Babaev authored
1. The rows of a recursive CTE at some point may overflow the HEAP temporary table containing them. At this point the table is converted to a MyISAM temporary table and the new added rows are placed into this MyISAM table. A bug in the of select_union_recursive::send_data prevented the server from writing the row that caused the overflow into the temporary table used for the result of the iteration steps. This could lead, in particular,to a premature end of the iterations. 2. The method TABLE::insert_all_rows_into() that was used to copy all rows of one temporary table into another did not take into account that the destination temporary table must be converted to a MyISAM table at some point. This patch fixed this problem. It also renamed the method into TABLE::insert_all_rows_into_tmp_table() and added an extra parameter needed for the conversion.
-
Marko Mäkelä authored
encryption.innodb_scrub: Clean up. Make it also cover ROW_FORMAT=COMPRESSED, removing the need for encryption.innodb_scrub_compressed. Add a FIXME comment saying that we should create a secondary index, to demonstrate that also undo log pages get scrubbed. Currently that is not working! Also clean up encryption.innodb_scrub_background, but keep it disabled, because the background scrubbing does not work reliably. Fix both tests so that if something is not scrubbed, the test will be aborted, so that the data files will be preserved. Allow the tests to run on Windows as well.
-
Marko Mäkelä authored
InnoDB shutdown failed to properly take fil_crypt_thread() into account. The encryption threads were signalled to shut down together with other non-critical tasks. This could be much too early in case of slow shutdown, which could need minutes to complete the purge. Furthermore, InnoDB failed to wait for the fil_crypt_thread() to actually exit before proceeding to the final steps of shutdown, causing the race conditions. Furthermore, the log_scrub_thread() was shut down way too early. Also it should remain until the SRV_SHUTDOWN_FLUSH_PHASE. fil_crypt_threads_end(): Remove. This would cause the threads to be terminated way too early. srv_buf_dump_thread_active, srv_dict_stats_thread_active, lock_sys->timeout_thread_active, log_scrub_thread_active, srv_monitor_active, srv_error_monitor_active: Remove a race condition between startup and shutdown, by setting these in the startup thread that creates threads, not in each created thread. In this way, once the flag is cleared, it will remain cleared during shutdown. srv_n_fil_crypt_threads_started, fil_crypt_threads_event: Declare in global rather than static scope. log_scrub_event, srv_log_scrub_thread_active, log_scrub_thread(): Declare in static rather than global scope. Let these be created by log_init() and freed by log_shutdown(). rotate_thread_t::should_shutdown(): Do not shut down before the SRV_SHUTDOWN_FLUSH_PHASE. srv_any_background_threads_are_active(): Remove. These checks now exist in logs_empty_and_mark_files_at_shutdown(). logs_empty_and_mark_files_at_shutdown(): Shut down the threads in the proper order. Keep fil_crypt_thread() and log_scrub_thread() alive until SRV_SHUTDOWN_FLUSH_PHASE, and check that they actually terminate.
-
Marko Mäkelä authored
Port a bug fix from MySQL 5.7, so that all undo log pages will be freed during a slow shutdown. We cannot scrub pages that are left allocated. commit 173e171c6fb55f064eea278c76fbb28e2b1c757b Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com> Date: Fri Sep 9 18:01:27 2016 +0530 Bug #24450908 UNDO LOG EXISTS AFTER SLOW SHUTDOWN Problem: ======== 1) cached undo segment is not removed from rollback segment history (RSEG_HISTORY) during slow shutdown. In other words, If the segment is not completely free, we are failing to remove an entry from the history list. While starting the server, we traverse all rollback segment slots history list and make it as list of undo logs to be purged in purge queue. In that case, purge queue will never be empty after slow shutdown. 2) Freeing of undo log segment is linked with removing undo log header from history. Fix: ==== 1) Have separate logic of removing the undo log header from history list from rollback segment slots and remove it from rollback segment history even though it is not completely free. Reviewed-by: Debarun Banerjee <debarun.banerjee@oracle.com> Reviewed-by: Marko Mäkelä <marko.makela@oracle.com> RB:13672
-
Marko Mäkelä authored
-
- 03 Jan, 2017 11 commits
-
-
Marko Mäkelä authored
MariaDB Server 10.0.28 and 10.1.19 merged code from Percona XtraDB that introduced support for compressed columns. Much but not all of this code was disabled by placing #ifdef HAVE_PERCONA_COMPRESSED_COLUMNS around it. Among the unused but not disabled code is code to access some new system tables related to compressed columns. The creation of these system tables SYS_ZIP_DICT and SYS_ZIP_DICT_COLS would cause a crash in --innodb-read-only mode when upgrading from an earlier version to 10.0.28 or 10.1.19. Let us remove all the dead code related to compressed columns. Users who already upgraded to 10.0.28 and 10.1.19 will have the two above mentioned empty tables in their InnoDB system tablespace. Subsequent versions of MariaDB Server will completely ignore those tables.
-
Marko Mäkelä authored
fil_crypt_threads_cleanup(): Do nothing if nothing was initialized.
-
Marko Mäkelä authored
fil_crypt_threads_cleanup(): Do nothing if nothing was initialized.
-
Jan Lindström authored
10.1 is merged into 10.2 now. Two issues are left to fix: (1) encryption.innochecksum test (2) read_page0 vs page_0_crypt_read (1) innochecksum tool did not compile after merge because buf_page_is_corrupted uses fil_crypt_t that has been changed. extra/CMakeLists.txt: Added fil/fil0crypt.cc as dependency as we need to use fil_crypt_verify_checksum for encrypted pages. innochecksum.cc: If we think page is encrypted i.e. FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION != 0 we call fil_crypt_verify_checksum() function to compare calculated checksum to stored checksum calculated after encryption (this is stored on different offset i.e. FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION + 4). If checksum does not match we call normal buf_page_is_corrupted to compare calculated checksum to stored checksum. fil0crypt.cc: add #ifdef UNIV_INNOCHECKSUM to be able to compile this file for innochecksum tool. (2) read_page0 is not needed and thus removed.
-
Marko Mäkelä authored
after aborted InnoDB startup This bug was repeatable by starting MariaDB 10.2 with an invalid option, such as --innodb-flush-method=foo. It is not repeatable in MariaDB 10.1 in the same way, but the problem exists already there.
-
Marko Mäkelä authored
after aborted InnoDB startup This bug was repeatable by starting MariaDB 10.2 with an invalid option, such as --innodb-flush-method=foo. It is not repeatable in MariaDB 10.1 in the same way, but the problem exists already there.
-
Marko Mäkelä authored
The upper limit of innodb_spin_wait_delay was ~0UL. It does not make any sense to wait more than a few dozens of microseconds between attempts to acquire a busy mutex. Make the new upper limit 6000. ut_delay(6000) could correspond to several milliseconds even today.
-
Jan Lindström authored
MDEV-11705: InnoDB: Failing assertion: (&log_sys->mutex)->is_owned() if server started with innodb-scrub-log Problem was that log_scrub function did not take required log_sys mutex. Background: Unused space in log blocks are padded with MLOG_DUMMY_RECORD if innodb-scrub-log is enabled. As log files are written on circular fashion old log blocks can be reused later for new redo-log entries. Scrubbing pads unused space in log blocks to avoid visibility of the possible old redo-log contents. log_scrub(): Take log_sys mutex log_pad_current_log_block(): Increase srv_stats.n_log_scrubs if padding is done. srv0srv.cc: Set srv_stats.n_log_scrubs to export vars innodb_scrub_log ha_innodb.cc: Export innodb_scrub_log to global status.
-
Marko Mäkelä authored
The C preprocessor symbol WITH_NUMA is never defined. Instead, the symbol HAVE_LIBNUMA is used for checking if the feature is to be used. If cmake -DWITH_NUMA=OFF is specified, HAVE_LIBNUMA will not be defined at compilation time even if the library is available. If cmake -DWITH_NUMA=ON is specified but the library is not available at configuration time, the compilation will be aborted.
-
Sachin Setiya authored
This commit is for optimizing WSREP(thd) macro. #define WSREP(thd) \ (WSREP_ON && wsrep && (thd && thd->variables.wsrep_on)) In this we can safely remove wsrep and thd. We are not removing WSREP_ON because this will change WSREP(thd) behaviour. Patch Credit:- Nirbhay Choubay, Sergey Vojtovich
-
Sachin Setiya authored
Problem:- The condition that checks for node readiness is too strict as it does not allow SELECTs even if these selects do not access any tables. For example,if we run SELECT 1; OR SELECT @@max_allowed_packet; Solution:- We need not to report this error when all_tables(lex->query_tables) is NULL:
-
- 01 Jan, 2017 4 commits
-
-
Elena Stepanova authored
The patch fixes two test failures: - on slow builders, sometimes a connection attempt which should fail due to the exceeded number of thread_pool_max_threads actually succeeds; - on even slow builders, MTR sometimes cannot establish the initial connection, and check-testcase fails prior to the test start The problem with check-testcase was caused by connect-timeout=2 which was set for all clients in the test config file. On slow builders it might be not enough. There is no way to override it for the pre-test check, so it needed to be substantially increased or removed. The other problem was caused by a race condition between sleeps that the test performs in existing connections and the connect timeout for the connection attempt which was expected to fail. If sleeps finished before the connect-timeout was exceeded, it would allow the connection to succeed. To solve each problem without making the other one worse, connect-timeout should be configured dynamically during the test. Due to the nature of the test (all connections must be busy at the moment when we need to change the timeout, and cannot execute SET GLOBAL ...), it needs to be done independently from the server. The solution: - recognize 'connect_timeout' as a connection option in mysqltest's "connect" command; - remove connect-timeout from the test configuration file; - use the new connect_timeout option for those connections which are expected to fail; - re-arrange the test flow to allow running a huge SLEEP without affecting the test execution time (because it would be interrupted after the main test flow is finished). The test is still subject to false negatives, e.g. if the connection fails due to timeout rather than due to the exceeded number of allowed threads, or if the connection on extra port succeeds due to a race condition and not because the special logic for the extra port. But those false negatives have always been possible there on slow builders, they should not be critical because faster builders should catch such failures if they appear.
-
Sachin Setiya authored
Problem:- In replication if slave has extra persistent column then these column are not computed while applying write-set from master. Solution:- While applying row events from server, we will generate values for extra persistent columns.
-
Sachin Setiya authored
Problem:- In replication if slave has extra persistent column then these column are not computed while applying write-set from master. Solution:- While applying row events from server, we will generate values for extra persistent columns.
-
Sachin Setiya authored
Problem:- In replication if slave has extra persistent column then these column are not computed while applying write-set from master. Solution:- While applying row events from server, we will generate values for extra persistent columns.
-
- 30 Dec, 2016 10 commits
-
-
Marko Mäkelä authored
Deprecate the variable in MariaDB 10.2, saying that it will be removed in 10.3.
-
Marko Mäkelä authored
The InnoDB source code contains quite a few references to a closed-source hot backup tool which was originally called InnoDB Hot Backup (ibbackup) and later incorporated in MySQL Enterprise Backup. The open source backup tool XtraBackup uses the full database for recovery. So, the references to UNIV_HOTBACKUP are only cluttering the source code.
-
Marko Mäkelä authored
-
Marko Mäkelä authored
The configuration parameter innodb_use_fallocate, which is mapped to the variable srv_use_posix_fallocate, has no effect in MariaDB 10.2.2 or MariaDB 10.2.3. Thus the configuration parameter and the variable should be removed.
-
Marko Mäkelä authored
-
Marko Mäkelä authored
-
Marko Mäkelä authored
fil_space_t::recv_size: New member: recovered tablespace size in pages; 0 if no size change was read from the redo log, or if the size change was implemented. fil_space_set_recv_size(): New function for setting space->recv_size. innodb_data_file_size_debug: A debug parameter for setting the system tablespace size in recovery even when the redo log does not contain any size changes. It is hard to write a small test case that would cause the system tablespace to be extended at the critical moment. recv_parse_log_rec(): Note those tablespaces whose size is being changed by the redo log, by invoking fil_space_set_recv_size(). innobase_init(): Correct an error message, and do not require a larger innodb_buffer_pool_size when starting up with a smaller innodb_page_size. innobase_start_or_create_for_mysql(): Allow startup with any initial size of the ibdata1 file if the autoextend attribute is set. Require the minimum size of fixed-size system tablespaces to be 640 pages, not 10 megabytes. Implement innodb_data_file_size_debug. open_or_create_data_files(): Round the system tablespace size down to pages, not to full megabytes, (Our test truncates the system tablespace to more than 800 pages with innodb_page_size=4k. InnoDB should not imagine that it was truncated to 768 pages and then overwrite good pages in the tablespace.) fil_flush_low(): Refactored from fil_flush(). fil_space_extend_must_retry(): Refactored from fil_extend_space_to_desired_size(). fil_mutex_enter_and_prepare_for_io(): Extend the tablespace if fil_space_set_recv_size() was called. The test case has been successfully run with all the innodb_page_size values 4k, 8k, 16k, 32k, 64k.
-
Marko Mäkelä authored
-
Marko Mäkelä authored
-
Marko Mäkelä authored
-
- 29 Dec, 2016 5 commits
-
-
Marko Mäkelä authored
fsp_header_get_crypt_offset(): Remove. xdes_arr_size(): Remove. fsp_header_get_encryption_offset(): Make this an inline function. The correctness of this change was ensured with the following patch that ensures that the two functions returned the same value, only differing by FSP_HEADER_OFFSET (38 bytes): diff --git a/storage/innobase/fsp/fsp0fsp.cc b/storage/innobase/fsp/fsp0fsp.cc index f2a4c6bf218..e96c788b7df 100644 --- a/storage/innobase/fsp/fsp0fsp.cc +++ b/storage/innobase/fsp/fsp0fsp.cc @@ -850,6 +850,7 @@ fsp_parse_init_file_page( return(ptr); } +static ulint fsp_header_get_encryption_offset(const page_size_t&); /**********************************************************************//** Initializes the fsp system. */ void @@ -868,6 +869,31 @@ fsp_init(void) #endif /* Does nothing at the moment */ + + for (ulint sz = 4096; sz <= 65536; sz *= 2) { + ulint m; + if (sz <= 16384) { + for (ulint ph = 1024; ph <= sz; ph *= 2) { + const page_size_t ps(ph, sz, true); + ulint maria = fsp_header_get_crypt_offset(ps, &m), + oracle = fsp_header_get_encryption_offset(ps); + if (maria != oracle + 38) { + ib::error() << "zip size mismatch: " + << maria << "!=" << oracle + << "(" << ph <<","<<sz<<")" + << m; + } + } + } + const page_size_t p(sz, sz, false); + ulint maria = fsp_header_get_crypt_offset(p, &m), + oracle = fsp_header_get_encryption_offset(p); + if (maria != oracle + 38) { + ib::error() << "size mismatch: " + << maria << "!=" << oracle + << "(" <<sz<<")" << m; + } + } } /**********************************************************************//**
-
Marko Mäkelä authored
-
Sergei Golubchik authored
* some of these tests run just fine with InnoDB: -> s/have_xtradb/have_innodb/ * sys_var tests did basic tests for xtradb only variables -> remove them, they're useless anyway (sysvar_innodb does it better) * multi_update had innodb specific tests -> move to multi_update_innodb.test
-
Sergei Golubchik authored
-
Sergei Golubchik authored
-