InnoDB in MySQL 5.7 introduced two new parameters to the function
dict_hdr_get_new_id(), to allow redo logging to be disabled when
assigning identifiers to temporary tables or during the
backup-unfriendly TRUNCATE TABLE that was replaced in MariaDB
by MDEV-13564.

Now that MariaDB 10.4.0 removed the crash recovery code for the
backup-unfriendly TRUNCATE, we can revert dict_hdr_get_new_id()
to be used only for persistent data structures.

dict_table_assign_new_id(): Remove. This was a simple 2-line function
that was called from few places.

dict_table_open_on_id_low(): Declare in the only file where it
is called.

dict_sys_t::temp_id_hash: A separate lookup table for temporary tables.
Table names will be in the common dict_sys_t::table_hash.

dict_sys_t::get_temporary_table_id(): Assign a temporary table ID.

dict_sys_t::get_table(): Look up a persistent table.

dict_sys_t::get_temporary_table(): Look up a temporary table.

dict_sys_t::temp_table_id: The sequence of temporary table identifiers.
Starts from DICT_HDR_FIRST_ID, so that we can continue to simply compare
dict_table_t::id to a few constants for the persistent hard-coded
data dictionary tables.

undo_node_t::state: Distinguish temporary and persistent tables.

lock_check_dict_lock(), lock_get_table_id(): Assert that there cannot
be locks on temporary tables.

row_rec_to_index_entry_impl(): Assert that there cannot be metadata
records on temporary tables.

row_undo_ins_parse_undo_rec(): Distinguish temporary and persistent tables.
Move some assertions from the only caller. Return whether the table was
found.

row_undo_ins(): Add some assertions.

row_undo_mod_clust(), row_undo_mod(): Do not assign node->state.
Let row_undo() do that.

row_undo_mod_parse_undo_rec(): Distinguish temporary and persistent tables.
Move some assertions from the only caller. Return whether the table was
found.

row_undo_try_truncate(): Renamed and simplified from trx_roll_try_truncate().

row_undo_rec_get(): Replaces trx_roll_pop_top_rec_of_trx() and
trx_roll_pop_top_rec(). Fetch an undo log record, and assign undo->state
accordingly.

trx_undo_truncate_end(): Acquire the rseg->mutex only for the minimum
required duration, and release it between mini-transactions.

There was a race condition between ALTER TABLE and purge.

If a table turns out to be logically empty when instant ALTER TABLE
is executing, we will convert the table to the canonical format,
to avoid overhead during subsequent accesses, and to allow the
data file to be imported into older versions of MariaDB.

It could happen that at the time the table is logically empty,
there still exists an undo log record for updating the hidden
metadata record for an earlier instant ALTER TABLE operation.
If the table was converted to the canonical format before
purge processes this undo log record, the undo log record
could be referring to index fields that no longer exist,
causing a crash.

To prevent the race condition, we must delete the old undo log records.
We do this lazily by assigning a new table ID, so that the table lookup
for the old undo log records will fail.

dict_table_t::reassign_id(): Reassign the table_id to
effectively lazily delete old undo log records.

innobase_instant_try(): Invoke index->table->reassign_id() before
index->clear_instant_alter().

dict_index_t::clear_instant_alter(): Remove a loop that became
redundant in commit ae2004c6
and caused corruption of n_nullable when dropping the last column
which was not declared NOT NULL.

dict_index_t::clear_instant_alter(): Correctly move all fields
corresponding to instantly dropped columns to the end of the array.
This fixes a regression that was introduced in
commit 5aaee374.

commit 21a58840

If the engine wants to avoid rnd_pos() - force a temporary table
before a filesort. But don't do it if addon_fields are used.

We don't have many bits left, no need to add another InnoDB-specific flag.
Instead, we say that HA_REQUIRE_PRIMARY_KEY does not apply to SEQUENCE.
Meaning, if the engine declares HA_CAN_TABLES_WITHOUT_ROLLBACK (required
for SEQUENCE) it *must* support tables without a primary key.

recv_addr_trim(): Do not try to detach the hash bucket, because
the code for doing that does not always work.

recv_apply_hashed_log_recs(): Do not attempt to read pages for which
there exist no redo log records.

dict_index_t::clear_instant_alter(): Correctly sort the index fields
to correspond to the columns of the table.

Skip tests which expectedly fail when Aria is not used for temporary tables

Commit ff88e4bb broke the Windows build.
Clean up one more redundant #include to fix it.

lock_discard_page(): Traverse and discard the B-tree record locks
only if they exist. Else, discard the R-tree (spatial) index locks.

Compare against the correct constant 0. NOT_KILLED belongs to
enum killed_state, while THD_IS_NOT_KILLED is what we want.

  modified:   sql_yacc.yy
  modified:   sql_yacc_ora.yy

ib_counter_t::ib_counter_element_t: Avoid sizeof on a std::atomic type,
because it causes errors on some version of the Microsoft compiler.

Remove some redundant declarations of global variables. These declarations
can refer to undefined type names, because they occur before any #include
statement for the type definitions. On GNU/Linux, uint and ulong seem to
be defined in <sys/types.h> due to _DEFAULT_SOURCE and __USE_MISC
becoming defined in <features.h>.

dict_index_t::db_trx_id(): Return the position of DB_TRX_ID.
Only valid for the clustered index.

dict_index_t::db_roll_ptr(): Return the position of DB_ROLL_PTR.
Only valid for the clustered index.

dict_index_get_sys_col_pos(): Remove. This was performing unnecessarily
complex computations, which only made sense for DB_ROW_ID, which would
exist either as the first field in the clustered index or as the last
field in a secondary index (only when a DB_ROW_ID column is materialised).

row_sel_store_row_id_to_prebuilt(): Remove, and replace with simpler code.

row_upd_index_entry_sys_field(): Remove.

btr_cur_log_sys(): Replaces row_upd_write_sys_vals_to_log().

btr_cur_write_sys(): Write DB_TRX_ID,DB_ROLL_PTR to a data tuple.

row_import_set_sys_max_row_id(): Change the return type to void,
and access the first column (DB_ROW_ID) directly.

fts_get_max_doc_id(): Remove totally unnecessary call to rec_get_offsets().
In FTS_DOC_ID_INDEX, the FTS_DOC_ID is always stored in the first 8 bytes
of the record.

fts_get_doc_id_from_row(): Get the offsets from the caller.

row_ins_foreign_check_on_constraint(): Invoke rec_get_offsets()
for the call.

This caused ma_test_recovery.pl to fail

Other things:
- Fixed bug where "ma_test_recovert.pl --abort-on-error" didn't abort
  on error

dict_index_t::reconstruct_fields(): Correctly permute the fields
of the columns. The code was totally wrong in MDEV-15562.
It would only work when columns are added last or dropped,
but not when columns are permuted.

row_ins_clust_index_entry_low(): Do not attempt to read an AUTO_INCREMENT
column value from a metadata record, because it does not make any sense.
Moreover, the field offset would be off by one in case the AUTO_INCREMENT
column is not part of the PRIMARY KEY, because the MDEV-15562 metadata
record would contain an extra field at index->first_user_field().

On MariaDB Server 10.3 after MDEV-11369, we would unnecessarily read
a dummy AUTO_INCREMENT value from the metadata record, but that value
would always be written as NULL or 0, so there is no problem.