MDEV-15662 Instant DROP COLUMN or changing the order of columns

Allow ADD COLUMN anywhere in a table, not only adding as the
last column.

Allow instant DROP COLUMN and instant changing the order of columns.

The added columns will always be added last in clustered index records.
In new records, instantly dropped columns will be stored as NULL or
empty when possible.

Information about dropped and reordered columns will be written in
a metadata BLOB (mblob), which is stored before the first 'user' field
in the hidden metadata record at the start of the clustered index.
The presence of mblob is indicated by setting the delete-mark flag in
the metadata record.

The metadata BLOB stores the number of clustered index fields,
followed by an array of column information for each field.
For dropped columns, we store the NOT NULL flag, the fixed length,
and for variable-length columns, whether the maximum length exceeded
255 bytes. For non-dropped columns, we store the column position.

Unlike with MDEV-11369, when a table becomes empty, it cannot
be converted back to the canonical format. The reason for this is
that other threads may hold cached objects such as
row_prebuilt_t::ins_node that could refer to dropped or reordered
index fields.

For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC,
we must store the n_core_null_bytes in the root page, so that the
chain of node pointer records can be followed in order to reach the
leftmost leaf page where the metadata record is located.
If the mblob is present, we will zero-initialize the strings
"infimum" and "supremum" in the root page, and use the last byte of
"supremum" for storing the number of null bytes (which are allocated
but useless on node pointer pages). This is necessary for
btr_cur_instant_init_metadata() to be able to navigate to the mblob.

If the PRIMARY KEY contains any variable-length column and some
nullable columns were instantly dropped, the dict_index_t::n_nullable
in the data dictionary could be smaller than it actually is in the
non-leaf pages. Because of this, the non-leaf pages could use more
bytes for the null flags than the data dictionary expects, and we
could be reading the lengths of the variable-length columns from the
wrong offset, and thus reading the child page number from wrong place.
This is the result of two design mistakes that involve unnecessary
storage of data: First, it is nonsense to store any data fields for
the leftmost node pointer records, because the comparisons would be
resolved by the MIN_REC_FLAG alone. Second, there cannot be any null
fields in the clustered index node pointer fields, but we nevertheless
reserve space for all the null flags.

Limitations (future work):

MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists
MDEV-17468 Avoid table rebuild on operations on generated columns
MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large

btr_page_reorganize_low(): Preserve any metadata in the root page.
Call lock_move_reorganize_page() only after restoring the "infimum"
and "supremum" records, to avoid a memcmp() assertion failure.

dict_col_t::DROPPED: Magic value for dict_col_t::ind.

dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant().
Do not assert that the column was instantly added, because we
sometimes call this unconditionally for all columns.
Convert an instantly added column to a "core column". The old name
remove_instant() could be mistaken to refer to "instant DROP COLUMN".

dict_col_t::is_added(): Rename from dict_col_t::is_instant().

dtype_t::metadata_blob_init(): Initialize the mblob data type.

dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(),
upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits
refer to a metadata record.

dict_table_t::instant: Metadata about dropped or reordered columns.

dict_table_t::prepare_instant(): Prepare
ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE.
innobase_instant_try() will pass this to dict_table_t::instant_column().
On rollback, dict_table_t::rollback_instant() will be called.

dict_table_t::instant_column(): Renamed from instant_add_column().
Add the parameter col_map so that columns can be reordered.
Copy and adjust v_cols[] as well.

dict_table_t::find(): Find an old column based on a new column number.

dict_table_t::serialise_columns(), dict_table_t::deserialise_columns():
Convert the mblob.

dict_index_t::instant_metadata(): Create the metadata record
for instant ALTER TABLE. Invoke dict_table_t::serialise_columns().

dict_index_t::reconstruct_fields(): Invoked by
dict_table_t::deserialise_columns().

dict_index_t::clear_instant_alter(): Move the fields for the
dropped columns to the end, and sort the surviving index fields
in ascending order of column position.

ha_innobase::check_if_supported_inplace_alter(): Do not allow
adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists
due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.)

instant_alter_column_possible(): Add a parameter for InnoDB table,
to check for additional conditions, such as the maximum number of
index fields.

ha_innobase_inplace_ctx::first_alter_pos: The first column whose position
is affected by instant ADD, DROP, or changing the order of columns.

innobase_build_col_map(): Skip added virtual columns.

prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
Remove some unnecessary code. Note that the call to
innodb_base_col_setup() should be executed later.

commit_try_norebuild(): If ctx->is_instant(), let the virtual
columns be added or dropped by innobase_instant_try().

innobase_instant_try(): Fill in a zero default value for the
hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459).
If any columns were dropped or reordered (or added not last),
delete any SYS_COLUMNS records for the following columns, and
insert SYS_COLUMNS records for all subsequent stored columns as well
as for all virtual columns. If any virtual column is dropped, rewrite
all virtual column metadata. Use a shortcut only for adding
virtual columns. This is because innobase_drop_virtual_try()
assumes that the dropped virtual columns still exist in ctx->old_table.

innodb_update_cols(): Renamed from innodb_update_n_cols().

innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change
the return type to bool, and invoke my_error() when detecting an error.

innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS.
Refactored from innobase_add_one_virtual() and innobase_instant_add_col().

innobase_instant_add_col(): Replace the parameter dfield with type.

innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS
and all columns from SYS_VIRTUAL.

innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
the caller invoke innodb_update_cols().

innobase_rename_column_try(): Skip dropped columns.

commit_cache_norebuild(): Update table->fts->doc_col.

dict_mem_table_col_rename_low(): Skip dropped columns.

trx_undo_rec_get_partial_row(): Skip dropped columns.

trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly.

trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
Log metadata records consistently.
Apparently, the first fields of a clustered index may be updated
in an update_undo vector when the index is ID_IND of SYS_FOREIGN,
as part of renaming the table during ALTER TABLE. Normally, updates of
the PRIMARY KEY should be logged as delete-mark and an insert.

row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec():
Use trx_undo_metadata.

row_undo_mod_clust_low(): On metadata rollback, roll back the root page too.

row_undo_mod_clust(): Relax an assertion. The delete-mark flag was
repurposed for ALTER TABLE metadata records.

row_rec_to_index_entry_impl(): Add the template parameter mblob
and the optional parameter info_bits for specifying the desired new
info bits. For the metadata tuple, allow conversion between the original
format (ADD COLUMN only) and the generic format (with hidden BLOB).
Add the optional parameter "pad" to determine whether the tuple should
be padded to the index fields (on ALTER TABLE it should), or whether
it should remain at its original size (on rollback).

row_build_index_entry_low(): Clean up the code, removing
redundant variables and conditions. For instantly dropped columns,
generate a dummy value that is NULL, the empty string, or a
fixed length of NUL bytes, depending on the type of the dropped column.

row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY
of a record that contained a dropped column whose value was stored
externally, we will be inserting a dummy NULL or empty string value
to the field of the dropped column. The externally stored column would
eventually be dropped when purge removes the delete-marked record for
the old PRIMARY KEY value.

btr_index_rec_validate(): Recognize the metadata record.

btr_discard_only_page_on_level(): Preserve the generic instant
ALTER TABLE metadata.

btr_set_instant(): Replaces page_set_instant(). This sets a clustered
index root page to the appropriate format, or upgrades from
the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format.

btr_cur_instant_init_low(): Read and validate the metadata BLOB page
before reconstructing the dictionary information based on it.

btr_cur_instant_init_metadata(): Do not read any lengths from the
metadata record header before reading the BLOB. At this point, we
would not actually know how many nullable fields the metadata record
contains.

btr_cur_instant_root_init(): Initialize n_core_null_bytes in one
of two possible ways.

btr_cur_trim(): Handle the mblob record.

row_metadata_to_tuple(): Convert a metadata record to a data tuple,
based on the new info_bits of the metadata record.

btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed.
Invoke dtuple_convert_big_rec() for metadata records if the record is
too large, or if the mblob is not yet marked as externally stored.

btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
When the last user record is deleted, do not delete the
generic instant ALTER TABLE metadata record. Only delete
MDEV-11369 instant ADD COLUMN metadata records.

btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size.

btr_pcur_store_position(): Allow a logically empty page to contain
a metadata record for generic ALTER TABLE.

REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW.
This is for the old instant ADD COLUMN (MDEV-11369) only.

REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record,
with additional information for dropped or reordered columns.

rec_info_bits_valid(): Remove. The only case when this would fail
is when the record is the generic ALTER TABLE metadata record.

rec_is_alter_metadata(): Check if a record is the metadata record
for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function
must not be invoked on node pointer records, because the delete-mark
flag in those records may be set (it is garbage), and then a debug
assertion could fail because index->is_instant() does not necessarily
hold.

rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata
record (not more generic instant ALTER TABLE).

rec_get_converted_size_comp_prefix_low(): Assume that the metadata
field will be stored externally. In dtuple_convert_big_rec() during
the rec_get_converted_size() call, it would not be there yet.

rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple.

rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(),
rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>.
With mblob=true, process a record with a metadata BLOB.

rec_copy_prefix_to_buf(): Assert that no fields beyond the key and
system columns are being copied. Exclude the metadata BLOB field.

rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple
into a record.

row_upd_index_replace_metadata(): Apply an update vector to an
alter_metadata tuple.

row_log_allocate(): Replace dict_index_t::is_instant()
with a more appropriate condition that ignores dict_table_t::instant.
Only a table on which the MDEV-11369 ADD COLUMN was performed
can "lose its instantness" when it becomes empty. After
instant DROP COLUMN or reordering columns, we cannot simply
convert the table to the canonical format, because the data
dictionary cache and all possibly existing references to it
from other client connection threads would have to be adjusted.

row_quiesce_write_index_fields(): Do not crash when the table contains
an instantly dropped column.

Thanks to Thirunarayanan Balathandayuthapani for discussing the design
and implementing an initial prototype of this.
Thanks to Matthias Leich for testing.

mysql-test/suite/gcol/r/innodb_virtual_rebuild.result

@@ -7,31 +7,31 @@ ROW_FORMAT=REDUNDANT;
 INSERT INTO t4 SET i=1;
 ALTER TABLE t4 ADD INDEX(v), LOCK=NONE;
 ALTER TABLE t4 ADD COLUMN k INT, LOCK=NONE;
-ALTER TABLE t4 DROP k, LOCK=NONE;
 ERROR 0A000: LOCK=NONE is not supported. Reason: online rebuild with indexed virtual columns. Try LOCK=SHARED
-ALTER TABLE t4 DROP INDEX v, LOCK=NONE;
 ALTER TABLE t4 DROP k, LOCK=NONE;
+ERROR 42000: Can't DROP COLUMN `k`; check that it exists
+ALTER TABLE t4 DROP INDEX v, LOCK=NONE;
 INSERT INTO t3 SET i=1;
 ALTER TABLE t3 ADD INDEX(v), LOCK=NONE;
 ALTER TABLE t3 ADD COLUMN k INT, LOCK=NONE;
-ALTER TABLE t3 DROP k, LOCK=NONE;
 ERROR 0A000: LOCK=NONE is not supported. Reason: online rebuild with indexed virtual columns. Try LOCK=SHARED
-ALTER TABLE t3 DROP INDEX v, LOCK=NONE;
 ALTER TABLE t3 DROP k, LOCK=NONE;
+ERROR 42000: Can't DROP COLUMN `k`; check that it exists
+ALTER TABLE t3 DROP INDEX v, LOCK=NONE;
 INSERT INTO t2 SET i=1;
 ALTER TABLE t2 ADD INDEX(v), LOCK=NONE;
 ALTER TABLE t2 ADD COLUMN k INT, LOCK=NONE;
-ALTER TABLE t2 DROP k, LOCK=NONE;
 ERROR 0A000: LOCK=NONE is not supported. Reason: online rebuild with indexed virtual columns. Try LOCK=SHARED
-ALTER TABLE t2 DROP INDEX v, LOCK=NONE;
 ALTER TABLE t2 DROP k, LOCK=NONE;
+ERROR 42000: Can't DROP COLUMN `k`; check that it exists
+ALTER TABLE t2 DROP INDEX v, LOCK=NONE;
 INSERT INTO t1 SET i=1;
 ALTER TABLE t1 ADD INDEX(v), LOCK=NONE;
 ALTER TABLE t1 ADD COLUMN k INT, LOCK=NONE;
-ALTER TABLE t1 DROP k, LOCK=NONE;
 ERROR 0A000: LOCK=NONE is not supported. Reason: online rebuild with indexed virtual columns. Try LOCK=SHARED
-ALTER TABLE t1 DROP INDEX v, LOCK=NONE;
 ALTER TABLE t1 DROP k, LOCK=NONE;
+ERROR 42000: Can't DROP COLUMN `k`; check that it exists
+ALTER TABLE t1 DROP INDEX v, LOCK=NONE;
 connect  ddl,localhost,root,,test;
 connection default;
 connection ddl;

mysql-test/suite/gcol/t/innodb_virtual_rebuild.test

@@ -14,11 +14,12 @@ while ($n)
 {
 eval INSERT INTO t$n SET i=1;
 eval ALTER TABLE t$n ADD INDEX(v), LOCK=NONE;
-eval ALTER TABLE t$n ADD COLUMN k INT, LOCK=NONE;
+# MDEV-17468 FIXME: Fix this, and remove the 2 --error below.
 --error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
+eval ALTER TABLE t$n ADD COLUMN k INT, LOCK=NONE;
+--error ER_CANT_DROP_FIELD_OR_KEY
 eval ALTER TABLE t$n DROP k, LOCK=NONE;
 eval ALTER TABLE t$n DROP INDEX v, LOCK=NONE;
-eval ALTER TABLE t$n DROP k, LOCK=NONE;
 dec $n;
 }
 

mysql-test/suite/innodb/r/innodb-alter.result

@@ -449,6 +449,7 @@ FULLTEXT INDEX(t)
 ALTER TABLE tu ADD COLUMN c CHAR(1) NOT NULL FIRST, LOCK=NONE;
 ERROR 0A000: LOCK=NONE is not supported. Reason: InnoDB presently supports one FULLTEXT index creation at a time. Try LOCK=SHARED
 ALTER TABLE tu ADD COLUMN c CHAR(1) NOT NULL, LOCK=NONE;
+ERROR 0A000: LOCK=NONE is not supported. Reason: InnoDB presently supports one FULLTEXT index creation at a time. Try LOCK=SHARED
 DROP TABLE tu;
 CREATE TABLE tv (
 pk INT PRIMARY KEY, FTS_DOC_ID BIGINT UNSIGNED NOT NULL, t TEXT,
@@ -458,6 +459,7 @@ FULLTEXT INDEX(t)
 ALTER TABLE tv ADD COLUMN c CHAR(1) NOT NULL FIRST, LOCK=NONE;
 ERROR 0A000: LOCK=NONE is not supported. Reason: InnoDB presently supports one FULLTEXT index creation at a time. Try LOCK=SHARED
 ALTER TABLE tv ADD COLUMN c CHAR(1) NOT NULL, LOCK=NONE;
+ERROR 0A000: LOCK=NONE is not supported. Reason: InnoDB presently supports one FULLTEXT index creation at a time. Try LOCK=SHARED
 DROP TABLE tv;
 ALTER TABLE t1o CHANGE c1 dB_row_Id INT, ALGORITHM=COPY;
 ERROR 42000: Incorrect column name 'dB_row_Id'

mysql-test/suite/innodb/r/instant_alter.result

@@ -182,8 +182,8 @@ affected rows: 0
 info: Records: 0  Duplicates: 0  Warnings: 0
 INSERT INTO t1 SET id=9;
 ALTER TABLE t1 DROP c3;
-affected rows: 9
-info: Records: 9  Duplicates: 0  Warnings: 0
+affected rows: 0
+info: Records: 0  Duplicates: 0  Warnings: 0
 SHOW CREATE TABLE t1;
 Table	Create Table
 t1	CREATE TABLE `t1` (
@@ -486,6 +486,43 @@ DELETE FROM t1;
 COMMIT;
 InnoDB		0 transactions not purged
 DROP TABLE t1;
+CREATE TABLE t1 (a INT, b INT UNIQUE) ENGINE=InnoDB ROW_FORMAT=REDUNDANT;
+INSERT INTO t1 (a) VALUES (NULL), (NULL);
+ALTER TABLE t1 DROP a, ADD COLUMN a INT;
+DELETE FROM t1;
+BEGIN;
+INSERT INTO t1 SET a=NULL;
+ROLLBACK;
+DELETE FROM t1;
+DROP TABLE t1;
+CREATE TABLE t1 (a INT PRIMARY KEY, t VARCHAR(33101) NOT NULL) ENGINE=InnoDB ROW_FORMAT=REDUNDANT;
+INSERT INTO t1 VALUES(347,'');
+ALTER TABLE t1 DROP COLUMN t, ALGORITHM=INSTANT;
+SELECT * FROM t1;
+a
+347
+DROP TABLE t1;
+CREATE TABLE t1 (a INT) ENGINE=InnoDB ROW_FORMAT=REDUNDANT;
+INSERT INTO t1() VALUES();
+ALTER TABLE t1 ADD COLUMN b INT FIRST, ADD COLUMN c INT AFTER b;
+SELECT * FROM t1;
+b	c	a
+NULL	NULL	NULL
+DROP TABLE t1;
+CREATE TABLE t1 (t TEXT) ENGINE=InnoDB ROW_FORMAT=REDUNDANT;
+SET @t=REPEAT('x',@@innodb_page_size / 2);
+INSERT INTO t1 VALUES (@t),(@t),(@t),(@t),(@t),(@t),(NULL),(@t),(@t),(@t),(@t);
+ALTER TABLE t1 ADD COLUMN a INT FIRST;
+UPDATE t1 SET a = 0;
+DROP TABLE t1;
+CREATE TABLE t1 (t TEXT) ENGINE=InnoDB ROW_FORMAT=REDUNDANT;
+INSERT INTO t1 SET t = @x;
+ALTER TABLE t1 DROP COLUMN t, ADD COLUMN i INT NOT NULL DEFAULT 1;
+ALTER TABLE t1 ADD COLUMN t TEXT;
+SELECT * FROM t1;
+i	t
+1	NULL
+DROP TABLE t1;
 CREATE TABLE t1
 (id INT PRIMARY KEY, c2 INT UNIQUE,
 c3 POINT NOT NULL DEFAULT ST_GeomFromText('POINT(3 4)'),
@@ -614,8 +651,8 @@ affected rows: 0
 info: Records: 0  Duplicates: 0  Warnings: 0
 INSERT INTO t1 SET id=9;
 ALTER TABLE t1 DROP c3;
-affected rows: 9
-info: Records: 9  Duplicates: 0  Warnings: 0
+affected rows: 0
+info: Records: 0  Duplicates: 0  Warnings: 0
 SHOW CREATE TABLE t1;
 Table	Create Table
 t1	CREATE TABLE `t1` (
@@ -918,6 +955,43 @@ DELETE FROM t1;
 COMMIT;
 InnoDB		0 transactions not purged
 DROP TABLE t1;
+CREATE TABLE t1 (a INT, b INT UNIQUE) ENGINE=InnoDB ROW_FORMAT=COMPACT;
+INSERT INTO t1 (a) VALUES (NULL), (NULL);
+ALTER TABLE t1 DROP a, ADD COLUMN a INT;
+DELETE FROM t1;
+BEGIN;
+INSERT INTO t1 SET a=NULL;
+ROLLBACK;
+DELETE FROM t1;
+DROP TABLE t1;
+CREATE TABLE t1 (a INT PRIMARY KEY, t VARCHAR(33101) NOT NULL) ENGINE=InnoDB ROW_FORMAT=COMPACT;
+INSERT INTO t1 VALUES(347,'');
+ALTER TABLE t1 DROP COLUMN t, ALGORITHM=INSTANT;
+SELECT * FROM t1;
+a
+347
+DROP TABLE t1;
+CREATE TABLE t1 (a INT) ENGINE=InnoDB ROW_FORMAT=COMPACT;
+INSERT INTO t1() VALUES();
+ALTER TABLE t1 ADD COLUMN b INT FIRST, ADD COLUMN c INT AFTER b;
+SELECT * FROM t1;
+b	c	a
+NULL	NULL	NULL
+DROP TABLE t1;
+CREATE TABLE t1 (t TEXT) ENGINE=InnoDB ROW_FORMAT=COMPACT;
+SET @t=REPEAT('x',@@innodb_page_size / 2);
+INSERT INTO t1 VALUES (@t),(@t),(@t),(@t),(@t),(@t),(NULL),(@t),(@t),(@t),(@t);
+ALTER TABLE t1 ADD COLUMN a INT FIRST;
+UPDATE t1 SET a = 0;
+DROP TABLE t1;
+CREATE TABLE t1 (t TEXT) ENGINE=InnoDB ROW_FORMAT=COMPACT;
+INSERT INTO t1 SET t = @x;
+ALTER TABLE t1 DROP COLUMN t, ADD COLUMN i INT NOT NULL DEFAULT 1;
+ALTER TABLE t1 ADD COLUMN t TEXT;
+SELECT * FROM t1;
+i	t
+1	NULL
+DROP TABLE t1;
 CREATE TABLE t1
 (id INT PRIMARY KEY, c2 INT UNIQUE,
 c3 POINT NOT NULL DEFAULT ST_GeomFromText('POINT(3 4)'),
@@ -1046,8 +1120,8 @@ affected rows: 0
 info: Records: 0  Duplicates: 0  Warnings: 0
 INSERT INTO t1 SET id=9;
 ALTER TABLE t1 DROP c3;
-affected rows: 9
-info: Records: 9  Duplicates: 0  Warnings: 0
+affected rows: 0
+info: Records: 0  Duplicates: 0  Warnings: 0
 SHOW CREATE TABLE t1;
 Table	Create Table
 t1	CREATE TABLE `t1` (
@@ -1350,10 +1424,47 @@ DELETE FROM t1;
 COMMIT;
 InnoDB		0 transactions not purged
 DROP TABLE t1;
+CREATE TABLE t1 (a INT, b INT UNIQUE) ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
+INSERT INTO t1 (a) VALUES (NULL), (NULL);
+ALTER TABLE t1 DROP a, ADD COLUMN a INT;
+DELETE FROM t1;
+BEGIN;
+INSERT INTO t1 SET a=NULL;
+ROLLBACK;
+DELETE FROM t1;
+DROP TABLE t1;
+CREATE TABLE t1 (a INT PRIMARY KEY, t VARCHAR(33101) NOT NULL) ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
+INSERT INTO t1 VALUES(347,'');
+ALTER TABLE t1 DROP COLUMN t, ALGORITHM=INSTANT;
+SELECT * FROM t1;
+a
+347
+DROP TABLE t1;
+CREATE TABLE t1 (a INT) ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
+INSERT INTO t1() VALUES();
+ALTER TABLE t1 ADD COLUMN b INT FIRST, ADD COLUMN c INT AFTER b;
+SELECT * FROM t1;
+b	c	a
+NULL	NULL	NULL
+DROP TABLE t1;
+CREATE TABLE t1 (t TEXT) ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
+SET @t=REPEAT('x',@@innodb_page_size / 2);
+INSERT INTO t1 VALUES (@t),(@t),(@t),(@t),(@t),(@t),(NULL),(@t),(@t),(@t),(@t);
+ALTER TABLE t1 ADD COLUMN a INT FIRST;
+UPDATE t1 SET a = 0;
+DROP TABLE t1;
+CREATE TABLE t1 (t TEXT) ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
+INSERT INTO t1 SET t = @x;
+ALTER TABLE t1 DROP COLUMN t, ADD COLUMN i INT NOT NULL DEFAULT 1;
+ALTER TABLE t1 ADD COLUMN t TEXT;
+SELECT * FROM t1;
+i	t
+1	NULL
+DROP TABLE t1;
 disconnect analyze;
 SELECT variable_value-@old_instant instants
 FROM information_schema.global_status
 WHERE variable_name = 'innodb_instant_alter_column';
 instants
-51
+78
 SET GLOBAL innodb_purge_rseg_truncate_frequency= @saved_frequency;

mysql-test/suite/innodb/r/instant_alter_crash.result

@@ -29,10 +29,11 @@ BEGIN;
 DELETE FROM t1;
 ROLLBACK;
 InnoDB		0 transactions not purged
-INSERT INTO t2 VALUES (64,42,'De finibus bonorum'), (347,33101,' et malorum');
+INSERT INTO t2 VALUES
+(16,1551,'Omnium enim rerum'),(128,1571,' principia parva sunt');
 connect ddl, localhost, root;
 SET DEBUG_SYNC='innodb_alter_inplace_before_commit SIGNAL ddl WAIT_FOR ever';
-ALTER TABLE t2 ADD COLUMN (c4 TEXT NOT NULL DEFAULT ' et malorum');
+ALTER TABLE t2 DROP COLUMN c3, ADD COLUMN c5 TEXT DEFAULT 'naturam abhorrere';
 connection default;
 SET DEBUG_SYNC='now WAIT_FOR ddl';
 SET GLOBAL innodb_flush_log_at_trx_commit=1;
@@ -46,8 +47,35 @@ id	c2
 SELECT * FROM t2;
 id	c2	c3
 2	1	De finibus bonorum
-64	42	De finibus bonorum
-347	33101	 et malorum
+16	1551	Omnium enim rerum
+128	1571	 principia parva sunt
+BEGIN;
+INSERT INTO t1 SET id=1;
+DELETE FROM t2;
+ROLLBACK;
+InnoDB		0 transactions not purged
+INSERT INTO t2 VALUES (64,42,'De finibus bonorum'), (347,33101,' et malorum');
+connect ddl, localhost, root;
+ALTER TABLE t2 DROP COLUMN c3;
+SET DEBUG_SYNC='innodb_alter_inplace_before_commit SIGNAL ddl WAIT_FOR ever';
+ALTER TABLE t2 ADD COLUMN (c4 TEXT NOT NULL DEFAULT ' et malorum');
+connection default;
+SET DEBUG_SYNC='now WAIT_FOR ddl';
+SET GLOBAL innodb_flush_log_at_trx_commit=1;
+DELETE FROM t1;
+# Kill the server
+disconnect ddl;
+SET @saved_frequency= @@GLOBAL.innodb_purge_rseg_truncate_frequency;
+SET GLOBAL innodb_purge_rseg_truncate_frequency=1;
+SELECT * FROM t1;
+id	c2
+SELECT * FROM t2;
+id	c2
+2	1
+64	42
+16	1551
+128	1571
+347	33101
 BEGIN;
 INSERT INTO t1 SET id=1;
 DELETE FROM t2;
@@ -59,27 +87,38 @@ N_RECS=0; LEVEL=0
 header=0x010000030074 (id=0x696e66696d756d00)
 header=0x010008030000 (id=0x73757072656d756d00)
 t2 clustered index root page(type 18):
-N_RECS=4; LEVEL=0
-header=0x010000030088 (id=0x696e66696d756d00)
-header=0x1000100b00b9 (id=0x80000000,
+N_RECS=6; LEVEL=0
+header=0x01000003008f (id=0x0000000000000000)
+header=0x3000100c00d4 (id=0x80000000,
  DB_TRX_ID=0x000000000000,
  DB_ROLL_PTR=0x80000000000000,
+ BLOB=0x000000260000000000000008,
  c2=NULL(4 bytes),
  c3=0x44652066696e6962757320626f6e6f72756d)
-header=0x0000180900d8 (id=0x80000002,
+header=0x0000180900f4 (id=0x80000002,
  DB_TRX_ID=0x000000000000,
  DB_ROLL_PTR=0x80000000000000,
  c2=0x80000001)
-header=0x0000200900f8 (id=0x80000040,
+header=0x0000200b0124 (id=0x80000010,
+ DB_TRX_ID=0x000000000000,
+ DB_ROLL_PTR=0x80000000000000,
+ c2=0x8000060f,
+ c3=0x4f6d6e69756d20656e696d20726572756d)
+header=0x000028090144 (id=0x80000040,
  DB_TRX_ID=0x000000000000,
  DB_ROLL_PTR=0x80000000000000,
  c2=0x8000002a)
-header=0x0000280b0074 (id=0x8000015b,
+header=0x0000300b0179 (id=0x80000080,
+ DB_TRX_ID=0x000000000000,
+ DB_ROLL_PTR=0x80000000000000,
+ c2=0x80000623,
+ c3=0x207072696e63697069612070617276612073756e74)
+header=0x0000380b0074 (id=0x8000015b,
  DB_TRX_ID=0x000000000000,
  DB_ROLL_PTR=0x80000000000000,
  c2=0x8000814d,
  c3=0x206574206d616c6f72756d)
-header=0x050008030000 (id=0x73757072656d756d00)
+header=0x070008030000 (id=0x000000000000000100)
 UNLOCK TABLES;
 DELETE FROM t2;
 InnoDB		0 transactions not purged
@@ -96,7 +135,6 @@ Table	Create Table
 t2	CREATE TABLE `t2` (
   `id` int(11) NOT NULL,
   `c2` int(11) DEFAULT NULL,
-  `c3` text NOT NULL DEFAULT 'De finibus bonorum',
   PRIMARY KEY (`id`),
   UNIQUE KEY `c2` (`c2`)
 ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=REDUNDANT

mysql-test/suite/innodb/r/instant_alter_debug.result

@@ -197,7 +197,8 @@ DELETE FROM t1;
 connection ddl;
 SET DEBUG_SYNC='row_log_table_apply1_before SIGNAL copied WAIT_FOR logged';
 ALTER TABLE t1 FORCE;
-disconnect stop_purge;
+connection stop_purge;
+COMMIT;
 connection default;
 SET DEBUG_SYNC = 'now WAIT_FOR copied';
 InnoDB		1 transactions not purged
@@ -211,6 +212,29 @@ SELECT * FROM t1;
 a	b	c
 1	2	NULL
 2	3	4
+ALTER TABLE t1 DROP b, ALGORITHM=INSTANT;
+connection stop_purge;
+START TRANSACTION WITH CONSISTENT SNAPSHOT;
+connection default;
+DELETE FROM t1;
+connection ddl;
+SET DEBUG_SYNC='row_log_table_apply1_before SIGNAL copied WAIT_FOR logged';
+ALTER TABLE t1 ADD COLUMN b INT NOT NULL DEFAULT 2 AFTER a, FORCE;
+disconnect stop_purge;
+connection default;
+SET DEBUG_SYNC = 'now WAIT_FOR copied';
+InnoDB		1 transactions not purged
+INSERT INTO t1 SET a=1;
+INSERT INTO t1 SET a=2,c=4;
+SET DEBUG_SYNC = 'now SIGNAL logged';
+connection ddl;
+UPDATE t1 SET b = b + 1 WHERE a = 2;
+connection default;
+SET DEBUG_SYNC = RESET;
+SELECT * FROM t1;
+a	b	c
+1	2	NULL
+2	3	4
 #
 # MDEV-15872 Crash in online ALTER TABLE...ADD PRIMARY KEY
 # after instant ADD COLUMN ... NULL

mysql-test/suite/innodb/r/instant_alter_limit.result

+SET @old_instant=
+(SELECT variable_value FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column');
+CREATE TABLE t(a INT PRIMARY KEY, b INT, c INT, d INT, e INT)
+ENGINE=InnoDB;
+INSERT INTO t VALUES(1,2,3,4,5);
+SELECT * FROM t;
+b	d	a	c	e
+NULL	NULL	1	NULL	NULL
+ALTER TABLE t DROP b, DROP c, DROP d, DROP e,
+ADD COLUMN b INT, ALGORITHM=INSTANT;
+ERROR 0A000: ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=INPLACE
+ALTER TABLE t CHANGE COLUMN b beta INT AFTER a, ALGORITHM=INSTANT;
+ALTER TABLE t DROP e, DROP c, DROP d, ALGORITHM=INSTANT;
+SELECT * FROM t;
+a	beta
+1	NULL
+ALTER TABLE t DROP COLUMN beta, ALGORITHM=INSTANT;
+ALTER TABLE t ADD COLUMN b INT NOT NULL, ALGORITHM=INSTANT;
+ERROR 0A000: ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=INPLACE
+SELECT variable_value-@old_instant instants
+FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column';
+instants
+256
+ALTER TABLE t ADD COLUMN b INT NOT NULL;
+SELECT variable_value-@old_instant instants
+FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column';
+instants
+256
+SELECT * FROM t;
+a	b
+1	0
+DROP TABLE t;

mysql-test/suite/innodb/r/instant_alter_rollback.result

 FLUSH TABLES;
 #
 # MDEV-11369: Instant ADD COLUMN for InnoDB
+# MDEV-15562: Instant DROP COLUMN or changing the order of columns
 #
 connect to_be_killed, localhost, root;
+SET @old_instant=
+(SELECT variable_value FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column');
 CREATE TABLE empty (id INT PRIMARY KEY, c2 INT UNIQUE) ENGINE=InnoDB;
 CREATE TABLE once LIKE empty;
 CREATE TABLE twice LIKE empty;
+CREATE TABLE thrice LIKE empty;
 INSERT INTO once SET id=1,c2=1;
 INSERT INTO twice SET id=1,c2=1;
+INSERT INTO thrice SET id=1,c2=1;
 ALTER TABLE empty ADD COLUMN (d1 INT DEFAULT 15);
 ALTER TABLE once ADD COLUMN (d1 INT DEFAULT 20);
 ALTER TABLE twice ADD COLUMN (d1 INT DEFAULT 20);
+ALTER TABLE thrice ADD COLUMN (d1 INT DEFAULT 20);
 ALTER TABLE twice ADD COLUMN
 (d2 INT NOT NULL DEFAULT 10,
 d3 VARCHAR(15) NOT NULL DEFAULT 'var och en char');
+ALTER TABLE thrice ADD COLUMN
+(d2 INT NOT NULL DEFAULT 10,
+d3 TEXT NOT NULL DEFAULT 'con');
+ALTER TABLE thrice DROP c2, DROP d3, CHANGE d2 d3 INT NOT NULL FIRST;
+SELECT variable_value-@old_instant instants
+FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column';
+instants
+7
 BEGIN;
 INSERT INTO empty set id=0,c2=42;
 UPDATE once set c2=c2+1;
 UPDATE twice set c2=c2+1;
+UPDATE thrice set d3=d3+1;
 INSERT INTO twice SET id=2,c2=0,d3='';
+INSERT INTO thrice SET id=2,d3=0;
+DELETE FROM empty;
+DELETE FROM once;
+DELETE FROM twice;
+DELETE FROM thrice;
 connection default;
 SET GLOBAL innodb_flush_log_at_trx_commit=1;
 CREATE TABLE foo(a INT PRIMARY KEY) ENGINE=InnoDB;
@@ -37,4 +59,7 @@ id	c2	d1
 SELECT * FROM twice;
 id	c2	d1	d2	d3
 1	1	20	10	var och en char
-DROP TABLE empty, once, twice;
+SELECT * FROM thrice;
+d3	id	d1
+10	1	20
+DROP TABLE empty, once, twice, thrice;

mysql-test/suite/innodb/t/innodb-alter.test

@@ -206,7 +206,7 @@ CREATE TABLE tu (
 ) ENGINE=InnoDB;
 --error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
 ALTER TABLE tu ADD COLUMN c CHAR(1) NOT NULL FIRST, LOCK=NONE;
-# Instant ADD COLUMN (adding after the visible FTS_DOC_ID)
+--error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
 ALTER TABLE tu ADD COLUMN c CHAR(1) NOT NULL, LOCK=NONE;
 DROP TABLE tu;
 
@@ -217,7 +217,7 @@ CREATE TABLE tv (
 ) ENGINE=InnoDB;
 --error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
 ALTER TABLE tv ADD COLUMN c CHAR(1) NOT NULL FIRST, LOCK=NONE;
-# Instant ADD COLUMN (adding after the visible FTS_DOC_ID)
+--error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
 ALTER TABLE tv ADD COLUMN c CHAR(1) NOT NULL, LOCK=NONE;
 DROP TABLE tv;
 

mysql-test/suite/innodb/t/instant_alter.test

@@ -361,6 +361,42 @@ COMMIT;
 --source include/wait_all_purged.inc
 DROP TABLE t1;
 
+# MDEV-15562 Instant DROP/ADD/reorder columns
+
+eval CREATE TABLE t1 (a INT, b INT UNIQUE) $engine;
+INSERT INTO t1 (a) VALUES (NULL), (NULL);
+ALTER TABLE t1 DROP a, ADD COLUMN a INT;
+DELETE FROM t1;
+BEGIN;INSERT INTO t1 SET a=NULL;ROLLBACK;
+DELETE FROM t1;
+DROP TABLE t1;
+
+eval CREATE TABLE t1 (a INT PRIMARY KEY, t VARCHAR(33101) NOT NULL) $engine;
+INSERT INTO t1 VALUES(347,'');
+ALTER TABLE t1 DROP COLUMN t, ALGORITHM=INSTANT;
+SELECT * FROM t1;
+DROP TABLE t1;
+
+eval CREATE TABLE t1 (a INT) $engine;
+INSERT INTO t1() VALUES();
+ALTER TABLE t1 ADD COLUMN b INT FIRST, ADD COLUMN c INT AFTER b;
+SELECT * FROM t1;
+DROP TABLE t1;
+
+eval CREATE TABLE t1 (t TEXT) $engine;
+SET @t=REPEAT('x',@@innodb_page_size / 2);
+INSERT INTO t1 VALUES (@t),(@t),(@t),(@t),(@t),(@t),(NULL),(@t),(@t),(@t),(@t);
+ALTER TABLE t1 ADD COLUMN a INT FIRST;
+UPDATE t1 SET a = 0;
+DROP TABLE t1;
+
+eval CREATE TABLE t1 (t TEXT) $engine;
+INSERT INTO t1 SET t = @x;
+ALTER TABLE t1 DROP COLUMN t, ADD COLUMN i INT NOT NULL DEFAULT 1;
+ALTER TABLE t1 ADD COLUMN t TEXT;
+SELECT * FROM t1;
+DROP TABLE t1;
+
 dec $format;
 }
 disconnect analyze;

mysql-test/suite/innodb/t/instant_alter_crash.test

@@ -43,9 +43,38 @@ DELETE FROM t1;
 ROLLBACK;
 --source include/wait_all_purged.inc
 
+INSERT INTO t2 VALUES
+(16,1551,'Omnium enim rerum'),(128,1571,' principia parva sunt');
+
+connect ddl, localhost, root;
+SET DEBUG_SYNC='innodb_alter_inplace_before_commit SIGNAL ddl WAIT_FOR ever';
+--send
+ALTER TABLE t2 DROP COLUMN c3, ADD COLUMN c5 TEXT DEFAULT 'naturam abhorrere';
+
+connection default;
+SET DEBUG_SYNC='now WAIT_FOR ddl';
+SET GLOBAL innodb_flush_log_at_trx_commit=1;
+DELETE FROM t1;
+
+--source include/kill_mysqld.inc
+disconnect ddl;
+--source include/start_mysqld.inc
+
+SET @saved_frequency= @@GLOBAL.innodb_purge_rseg_truncate_frequency;
+SET GLOBAL innodb_purge_rseg_truncate_frequency=1;
+
+SELECT * FROM t1;
+SELECT * FROM t2;
+BEGIN;
+INSERT INTO t1 SET id=1;
+DELETE FROM t2;
+ROLLBACK;
+--source include/wait_all_purged.inc
+
 INSERT INTO t2 VALUES (64,42,'De finibus bonorum'), (347,33101,' et malorum');
 
 connect ddl, localhost, root;
+ALTER TABLE t2 DROP COLUMN c3;
 SET DEBUG_SYNC='innodb_alter_inplace_before_commit SIGNAL ddl WAIT_FOR ever';
 --send
 ALTER TABLE t2 ADD COLUMN (c4 TEXT NOT NULL DEFAULT ' et malorum');
@@ -96,16 +125,39 @@ for (my $offset= 0x65; $offset;
     my $n_fields= unpack("n", substr($page,$offset-4,2)) >> 1 & 0x3ff;
     my $start= 0;
     my $name;
-    for (my $i= 0; $i < $n_fields; $i++) {
-	my $end= unpack("C", substr($page, $offset-7-$i, 1));
-	print ",\n " if $i;
-	print "$fields[$i]=";
-	if ($end & 0x80) {
-	    print "NULL(", ($end & 0x7f) - $start, " bytes)"
-	} else {
-	    print "0x", unpack("H*", substr($page,$offset+$start,$end-$start))
+    if (unpack("C", substr($page,$offset-3,1)) & 1) {
+	for (my $i= 0; $i < $n_fields; $i++) {
+	    my $end= unpack("C", substr($page, $offset-7-$i, 1));
+	    print ",\n " if $i;
+	    print "$fields[$i]=";
+	    if ($end & 0x80) {
+		print "NULL(", ($end & 0x7f) - $start, " bytes)"
+	    } else {
+		print "0x", unpack("H*", substr($page,$offset+$start,$end-$start))
+	    }
+	    $start= $end & 0x7f;
+	}
+    } else {
+	for (my $i= 0; $i < $n_fields; $i++) {
+	    my $end= unpack("n", substr($page, $offset-8-2*$i, 2));
+	    print ",\n " if $i;
+	    if ($i > 2 && !(~unpack("C",substr($page,$offset-6,1)) & 0x30)) {
+		if ($i == 3) {
+		    print "BLOB=";
+		    $start += 8; # skip the space_id,page_number
+		} else {
+		    print "$fields[$i - 1]=";
+		}
+	    } else {
+		print "$fields[$i]=";
+	    }
+	    if ($end & 0x8000) {
+		print "NULL(", ($end & 0x7fff) - $start, " bytes)"
+	    } else {
+		print "0x", unpack("H*", substr($page,$offset+$start,($end-$start) & 0x3fff))
+	    }
+	    $start= $end & 0x3fff;
 	}
-	$start= $end & 0x7f;
     }
     print ")\n";
 }

mysql-test/suite/innodb/t/instant_alter_debug.test

@@ -222,7 +222,8 @@ connection ddl;
 SET DEBUG_SYNC='row_log_table_apply1_before SIGNAL copied WAIT_FOR logged';
 send ALTER TABLE t1 FORCE;
 
-disconnect stop_purge;
+connection stop_purge;
+COMMIT;
 
 connection default;
 SET DEBUG_SYNC = 'now WAIT_FOR copied';
@@ -235,6 +236,34 @@ SET DEBUG_SYNC = 'now SIGNAL logged';
 connection ddl;
 reap;
 
+connection default;
+SET DEBUG_SYNC = RESET;
+SELECT * FROM t1;
+ALTER TABLE t1 DROP b, ALGORITHM=INSTANT;
+connection stop_purge;
+START TRANSACTION WITH CONSISTENT SNAPSHOT;
+
+connection default;
+DELETE FROM t1;
+
+connection ddl;
+SET DEBUG_SYNC='row_log_table_apply1_before SIGNAL copied WAIT_FOR logged';
+send ALTER TABLE t1 ADD COLUMN b INT NOT NULL DEFAULT 2 AFTER a, FORCE;
+
+disconnect stop_purge;
+
+connection default;
+SET DEBUG_SYNC = 'now WAIT_FOR copied';
+let $wait_all_purged = 1;
+--source include/wait_all_purged.inc
+INSERT INTO t1 SET a=1;
+INSERT INTO t1 SET a=2,c=4;
+SET DEBUG_SYNC = 'now SIGNAL logged';
+
+connection ddl;
+reap;
+UPDATE t1 SET b = b + 1 WHERE a = 2;
+
 connection default;
 SET DEBUG_SYNC = RESET;
 SELECT * FROM t1;

mysql-test/suite/innodb/t/instant_alter_limit.test

+--source include/have_innodb.inc
+
+SET @old_instant=
+(SELECT variable_value FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column');
+
+CREATE TABLE t(a INT PRIMARY KEY, b INT, c INT, d INT, e INT)
+ENGINE=InnoDB;
+INSERT INTO t VALUES(1,2,3,4,5);
+--disable_query_log
+let $n=253;
+while ($n) {
+dec $n;
+ALTER TABLE t DROP b, DROP c, DROP d, DROP e,
+ADD COLUMN b INT FIRST, ADD COLUMN c INT, ADD COLUMN d INT AFTER b,
+ADD COLUMN e INT AFTER c, ALGORITHM=INSTANT;
+}
+--enable_query_log
+SELECT * FROM t;
+--error ER_ALTER_OPERATION_NOT_SUPPORTED
+ALTER TABLE t DROP b, DROP c, DROP d, DROP e,
+ADD COLUMN b INT, ALGORITHM=INSTANT;
+ALTER TABLE t CHANGE COLUMN b beta INT AFTER a, ALGORITHM=INSTANT;
+ALTER TABLE t DROP e, DROP c, DROP d, ALGORITHM=INSTANT;
+SELECT * FROM t;
+ALTER TABLE t DROP COLUMN beta, ALGORITHM=INSTANT;
+--error ER_ALTER_OPERATION_NOT_SUPPORTED
+ALTER TABLE t ADD COLUMN b INT NOT NULL, ALGORITHM=INSTANT;
+
+SELECT variable_value-@old_instant instants
+FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column';
+
+ALTER TABLE t ADD COLUMN b INT NOT NULL;
+
+SELECT variable_value-@old_instant instants
+FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column';
+
+SELECT * FROM t;
+
+DROP TABLE t;

mysql-test/suite/innodb/t/instant_alter_rollback.test

@@ -8,28 +8,49 @@ FLUSH TABLES;
 
 --echo #
 --echo # MDEV-11369: Instant ADD COLUMN for InnoDB
+--echo # MDEV-15562: Instant DROP COLUMN or changing the order of columns
 --echo #
 
 connect to_be_killed, localhost, root;
 
+SET @old_instant=
+(SELECT variable_value FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column');
 CREATE TABLE empty (id INT PRIMARY KEY, c2 INT UNIQUE) ENGINE=InnoDB;
 CREATE TABLE once LIKE empty;
 CREATE TABLE twice LIKE empty;
+CREATE TABLE thrice LIKE empty;
 INSERT INTO once SET id=1,c2=1;
 INSERT INTO twice SET id=1,c2=1;
+INSERT INTO thrice SET id=1,c2=1;
 ALTER TABLE empty ADD COLUMN (d1 INT DEFAULT 15);
 ALTER TABLE once ADD COLUMN (d1 INT DEFAULT 20);
 ALTER TABLE twice ADD COLUMN (d1 INT DEFAULT 20);
+ALTER TABLE thrice ADD COLUMN (d1 INT DEFAULT 20);
 ALTER TABLE twice ADD COLUMN
 (d2 INT NOT NULL DEFAULT 10,
  d3 VARCHAR(15) NOT NULL DEFAULT 'var och en char');
+ALTER TABLE thrice ADD COLUMN
+(d2 INT NOT NULL DEFAULT 10,
+ d3 TEXT NOT NULL DEFAULT 'con');
+ALTER TABLE thrice DROP c2, DROP d3, CHANGE d2 d3 INT NOT NULL FIRST;
+
+SELECT variable_value-@old_instant instants
+FROM information_schema.global_status
+WHERE variable_name = 'innodb_instant_alter_column';
 
 BEGIN;
 
 INSERT INTO empty set id=0,c2=42;
 UPDATE once set c2=c2+1;
 UPDATE twice set c2=c2+1;
+UPDATE thrice set d3=d3+1;
 INSERT INTO twice SET id=2,c2=0,d3='';
+INSERT INTO thrice SET id=2,d3=0;
+DELETE FROM empty;
+DELETE FROM once;
+DELETE FROM twice;
+DELETE FROM thrice;
 
 connection default;
 SET GLOBAL innodb_flush_log_at_trx_commit=1;
@@ -48,4 +69,5 @@ SET GLOBAL innodb_purge_rseg_truncate_frequency=@saved_frequency;
 SELECT * FROM empty;
 SELECT * FROM once;
 SELECT * FROM twice;
-DROP TABLE empty, once, twice;
+SELECT * FROM thrice;
+DROP TABLE empty, once, twice, thrice;

mysql-test/suite/innodb/t/instant_drop.test

+--source include/have_innodb.inc
+
+create table t1(f1 int not null, f2 int not null, f3 int not null)engine=innodb;
+insert into t1 values(1, 2, 3),(4, 5, 6);
+alter table t1 drop column f2, algorithm=instant;
+select * from t1;
+insert into t1 values(1,2);
+select * from t1;
+alter table t1 add column f4 int not null default 5, algorithm=instant;
+select * from t1;
+alter table t1 drop column f1, algorithm=instant;
+select * from t1;
+insert into t1 values(7, 9);
+select * from t1;
+alter table t1 add column f5 blob default repeat('aaa', 950), drop column f4, algorithm=instant;
+select * from t1;
+select f3 from t1;
+update t1 set f3 = 10 where f3 > 2;
+select * from t1;
+delete from t1 where f3 = 10;
+show create table t1;
+select f3 from t1;
+update t1 set f5 = 'world';
+select * from t1;
+drop table t1;
+
+create table t1(f1 int, f2 int not null, index idx(f2))engine=innodb;
+insert into t1 values(1, 2);
+alter table t1 drop column f1, add column f3 varchar(100) default 'thiru', algorithm=instant;
+select * from t1 force index (idx);
+alter table t1 drop column f3, algorithm=instant;
+select * from t1;
+begin;
+insert into t1 values(10);
+select * from t1;
+update t1 set f2 = 100;
+select * from t1;
+delete from t1 where f2 = 100;
+select * from t1;
+rollback;
+select * from t1;
+show create table t1;
+drop table t1;
+
+create table t1(f1 int, f2 int not null)engine=innodb;
+insert into t1 values(1, 2);
+alter table t1 drop column f2, algorithm=instant;
+insert into t1 values(NULL);
+select * from t1;
+drop table t1;
+
+create table t1(f1 int not null, f2 int not null)engine=innodb;
+insert into t1 values(1, 2);
+alter table t1 add column f5 int default 10, algorithm=instant;
+alter table t1 add column f3 int not null default 100, algorithm=instant;
+alter table t1 add column f4 int default 100, drop column f3, algorithm=instant;
+insert into t1 values(2, 3, 20, 100);
+select * from t1;
+drop table t1;
+
+create table t1(f1 int not null, f2 int not null) engine=innodb;
+insert into t1 values(1, 1);
+alter table t1 drop column f2, add column f3 int default 3, algorithm=instant;
+select * from t1;
+update t1 set f3 = 19;
+select * from t1;
+alter table t1 drop column f1, add column f5 int default 10, algorithm=instant;
+insert into t1 values(4, 10);
+select * from t1;
+
+create table t2(f1 int, f2 int not null) engine=innodb;
+insert into t2(f1, f2) values(1, 2);
+alter table t2 drop column f2, add column f4 varchar(100) default repeat('a', 20), add column f5 int default 10,  algorithm=instant;
+select * from t2;
+show create table t2;
+alter table t2 add column f6 char(100) default repeat('a', 99), algorithm=instant;
+
+create table t3(f1 int, f2 int not null)engine=innodb;
+insert into t3 values(1, 2);
+alter table t3 drop column f2, add column f3 int default 1, add column f4 int default 4, algorithm=instant;
+
+--source include/restart_mysqld.inc
+select * from t1;
+alter table t1 add column f6 int default 9,drop column f5, algorithm = instant;
+insert into t1 values(4, 9);
+alter table t1 force, algorithm=inplace;
+select * from t1;
+
+select * from t2;
+alter table t2 force, algorithm=inplace;
+select * from t2;
+show create table t2;
+
+select * from t3;
+alter table t3 add column f5 char(100) default repeat('a', 99), algorithm=instant;
+--source include/restart_mysqld.inc
+select * from t3;
+drop table t1,t2,t3;

mysql-test/suite/innodb_gis/r/alter_spatial_index.result

@@ -521,7 +521,7 @@ HEX(c8)
 
 ROLLBACK;
 ALTER TABLE tab add COLUMN c9 POINT NOT NULL AFTER c5, ALGORITHM = INPLACE, LOCK=NONE;
-ERROR 0A000: LOCK=NONE is not supported. Reason: Do not support online operation on table with GIS index. Try LOCK=SHARED
+ALTER TABLE tab DROP COLUMN c9, ALGORITHM=INSTANT;
 SHOW CREATE TABLE tab;
 Table	Create Table
 tab	CREATE TABLE `tab` (

mysql-test/suite/innodb_gis/t/alter_spatial_index.test

@@ -491,9 +491,8 @@ FROM tab LIMIT 1;
 SELECT HEX(c8) FROM tab;
 ROLLBACK;
 
-# not instant, not supported
---error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
 ALTER TABLE tab add COLUMN c9 POINT NOT NULL AFTER c5, ALGORITHM = INPLACE, LOCK=NONE;
+ALTER TABLE tab DROP COLUMN c9, ALGORITHM=INSTANT;
 
 SHOW CREATE TABLE tab;
 

mysql-test/suite/versioning/r/online.result

@@ -25,8 +25,7 @@ add s bigint unsigned as row start,
 add e bigint unsigned as row end,
 add period for system_time(s, e),
 add system versioning;
-alter table t drop column b, lock=none;
-ERROR 0A000: LOCK=NONE is not supported. Reason: Not implemented for system-versioned tables. Try LOCK=SHARED
+alter table t drop column b, algorithm=instant;
 alter table t add index idx(a), lock=none;
 alter table t drop column s, drop column e;
 alter table t drop system versioning, lock=none;

mysql-test/suite/versioning/t/online.test

@@ -32,8 +32,7 @@ alter table t
   add e bigint unsigned as row end,
   add period for system_time(s, e),
   add system versioning;
---error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
-alter table t drop column b, lock=none;
+alter table t drop column b, algorithm=instant;
 alter table t add index idx(a), lock=none;
 alter table t drop column s, drop column e;
 --error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON

storage/innobase/btr/btr0btr.cc

@@ -1660,11 +1660,6 @@ btr_page_reorganize_low(
 		goto func_exit;
 	}
 
-	if (!recovery && !dict_table_is_locking_disabled(index->table)) {
-		/* Update the record lock bitmaps */
-		lock_move_reorganize_page(block, temp_block);
-	}
-
 	data_size2 = page_get_data_size(page);
 	max_ins_size2 = page_get_max_insert_size_after_reorganize(page, 1);
 
@@ -1688,21 +1683,41 @@ btr_page_reorganize_low(
 		ut_ad(cursor->rec == page_get_infimum_rec(page));
 	}
 
-func_exit:
 #ifdef UNIV_ZIP_DEBUG
 	ut_a(!page_zip || page_zip_validate(page_zip, page, index));
 #endif /* UNIV_ZIP_DEBUG */
 
-	if (!recovery && page_is_root(temp_page)
-	    && fil_page_get_type(temp_page) == FIL_PAGE_TYPE_INSTANT) {
-		/* Preserve the PAGE_INSTANT information. */
-		ut_ad(!page_zip);
-		ut_ad(index->is_instant());
-		memcpy(FIL_PAGE_TYPE + page, FIL_PAGE_TYPE + temp_page, 2);
-		memcpy(PAGE_HEADER + PAGE_INSTANT + page,
-		       PAGE_HEADER + PAGE_INSTANT + temp_page, 2);
+	if (!recovery) {
+		if (page_is_root(temp_page)
+		    && fil_page_get_type(temp_page) == FIL_PAGE_TYPE_INSTANT) {
+			/* Preserve the PAGE_INSTANT information. */
+			ut_ad(!page_zip);
+			ut_ad(index->is_instant());
+			memcpy(FIL_PAGE_TYPE + page,
+			       FIL_PAGE_TYPE + temp_page, 2);
+			memcpy(PAGE_HEADER + PAGE_INSTANT + page,
+			       PAGE_HEADER + PAGE_INSTANT + temp_page, 2);
+			if (!index->table->instant) {
+			} else if (page_is_comp(page)) {
+				memcpy(PAGE_NEW_INFIMUM + page,
+				       PAGE_NEW_INFIMUM + temp_page, 8);
+				memcpy(PAGE_NEW_SUPREMUM + page,
+				       PAGE_NEW_SUPREMUM + temp_page, 8);
+			} else {
+				memcpy(PAGE_OLD_INFIMUM + page,
+				       PAGE_OLD_INFIMUM + temp_page, 8);
+				memcpy(PAGE_OLD_SUPREMUM + page,
+				       PAGE_OLD_SUPREMUM + temp_page, 8);
+			}
+		}
+
+		if (!dict_table_is_locking_disabled(index->table)) {
+			/* Update the record lock bitmaps */
+			lock_move_reorganize_page(block, temp_block);
+		}
 	}
 
+func_exit:
 	buf_block_free(temp_block);
 
 	/* Restore logging mode */
@@ -1748,6 +1763,14 @@ btr_page_reorganize_low(
 				 mach_read_from_2(PAGE_HEADER + PAGE_INSTANT
 						  + page),
 				 MLOG_2BYTES, mtr);
+		if (!index->table->instant) {
+		} else if (page_is_comp(page)) {
+			mlog_log_string(PAGE_NEW_INFIMUM + page, 8, mtr);
+			mlog_log_string(PAGE_NEW_SUPREMUM + page, 8, mtr);
+		} else {
+			mlog_log_string(PAGE_OLD_INFIMUM + page, 8, mtr);
+			mlog_log_string(PAGE_OLD_SUPREMUM + page, 8, mtr);
+		}
 	}
 
 	return(success);
@@ -1895,6 +1918,59 @@ btr_page_empty(
 	}
 }
 
+/** Write instant ALTER TABLE metadata to a root page.
+@param[in,out]	root	clustered index root page
+@param[in]	index	clustered index with instant ALTER TABLE
+@param[in,out]	mtr	mini-transaction */
+void btr_set_instant(buf_block_t* root, const dict_index_t& index, mtr_t* mtr)
+{
+	ut_ad(index.n_core_fields > 0);
+	ut_ad(index.n_core_fields < REC_MAX_N_FIELDS);
+	ut_ad(index.is_instant());
+	ut_ad(page_is_root(root->frame));
+
+	rec_t* infimum = page_get_infimum_rec(root->frame);
+	rec_t* supremum = page_get_supremum_rec(root->frame);
+	byte* page_type = root->frame + FIL_PAGE_TYPE;
+	uint16_t i = page_header_get_field(root->frame, PAGE_INSTANT);
+
+	switch (mach_read_from_2(page_type)) {
+	case FIL_PAGE_TYPE_INSTANT:
+		ut_ad(page_get_instant(root->frame) == index.n_core_fields);
+		if (memcmp(infimum, "infimum", 8)
+		    || memcmp(supremum, "supremum", 8)) {
+			ut_ad(index.table->instant);
+			ut_ad(!memcmp(infimum, field_ref_zero, 8));
+			ut_ad(!memcmp(supremum, field_ref_zero, 7));
+			ut_ad(supremum[7] == index.n_core_null_bytes);
+			return;
+		}
+		break;
+	default:
+		ut_ad(!"wrong page type");
+		/* fall through */
+	case FIL_PAGE_INDEX:
+		ut_ad(!page_is_comp(root->frame)
+		      || !page_get_instant(root->frame));
+		ut_ad(!memcmp(infimum, "infimum", 8));
+		ut_ad(!memcmp(supremum, "supremum", 8));
+		mlog_write_ulint(page_type, FIL_PAGE_TYPE_INSTANT,
+				 MLOG_2BYTES, mtr);
+		ut_ad(i <= PAGE_NO_DIRECTION);
+		i |= index.n_core_fields << 3;
+		mlog_write_ulint(PAGE_HEADER + PAGE_INSTANT + root->frame, i,
+				 MLOG_2BYTES, mtr);
+		break;
+	}
+
+	if (index.table->instant) {
+		mlog_memset(root, infimum - root->frame, 8, 0, mtr);
+		mlog_memset(root, supremum - root->frame, 7, 0, mtr);
+		mlog_write_ulint(&supremum[7], index.n_core_null_bytes,
+				 MLOG_1BYTE, mtr);
+	}
+}
+
 /*************************************************************//**
 Makes tree one level higher by splitting the root, and inserts
 the tuple. It is assumed that mtr contains an x-latch on the tree.
@@ -2080,11 +2156,7 @@ btr_root_raise_and_insert(
 
 	if (index->is_instant()) {
 		ut_ad(!root_page_zip);
-		byte* page_type = root_block->frame + FIL_PAGE_TYPE;
-		ut_ad(mach_read_from_2(page_type) == FIL_PAGE_INDEX);
-		mlog_write_ulint(page_type, FIL_PAGE_TYPE_INSTANT,
-				 MLOG_2BYTES, mtr);
-		page_set_instant(root_block->frame, index->n_core_fields, mtr);
+		btr_set_instant(root_block, *index, mtr);
 	}
 
 	/* Set the next node and previous node fields, although
@@ -3569,12 +3641,7 @@ btr_lift_page_up(
 
 	if (page_level == 0 && index->is_instant()) {
 		ut_ad(!father_page_zip);
-		byte* page_type = father_block->frame + FIL_PAGE_TYPE;
-		ut_ad(mach_read_from_2(page_type) == FIL_PAGE_INDEX);
-		mlog_write_ulint(page_type, FIL_PAGE_TYPE_INSTANT,
-				 MLOG_2BYTES, mtr);
-		page_set_instant(father_block->frame,
-				 index->n_core_fields, mtr);
+		btr_set_instant(father_block, *index, mtr);
 	}
 
 	page_level++;
@@ -4246,15 +4313,42 @@ btr_discard_only_page_on_level(
 	}
 #endif /* UNIV_BTR_DEBUG */
 
+	mem_heap_t* heap = NULL;
+	const rec_t* rec = NULL;
+	ulint* offsets = NULL;
+	if (index->table->instant) {
+		const rec_t* r = page_rec_get_next(page_get_infimum_rec(
+							   block->frame));
+		ut_ad(rec_is_metadata(r, *index) == index->is_instant());
+		if (rec_is_alter_metadata(r, *index)) {
+			heap = mem_heap_create(srv_page_size);
+			offsets = rec_get_offsets(r, index, NULL, true,
+						  ULINT_UNDEFINED, &heap);
+			rec = rec_copy(mem_heap_alloc(heap,
+						      rec_offs_size(offsets)),
+				       r, offsets);
+			rec_offs_make_valid(rec, index, true, offsets);
+		}
+	}
+
 	btr_page_empty(block, buf_block_get_page_zip(block), index, 0, mtr);
 	ut_ad(page_is_leaf(buf_block_get_frame(block)));
 	/* btr_page_empty() is supposed to zero-initialize the field. */
 	ut_ad(!page_get_instant(block->frame));
 
 	if (index->is_primary()) {
-		/* Concurrent access is prevented by the root_block->lock
-		X-latch, so this should be safe. */
-		index->remove_instant();
+		if (rec) {
+			DBUG_ASSERT(index->table->instant);
+			DBUG_ASSERT(rec_is_alter_metadata(rec, *index));
+			btr_set_instant(block, *index, mtr);
+			rec = page_cur_insert_rec_low(
+				page_get_infimum_rec(block->frame),
+				index, rec, offsets, mtr);
+			ut_ad(rec);
+			mem_heap_free(heap);
+		} else if (index->is_instant()) {
+			index->clear_instant_add();
+		}
 	} else if (!index->table->is_temporary()) {
 		/* We play it safe and reset the free bits for the root */
 		ibuf_reset_free_bits(block);
@@ -4678,14 +4772,32 @@ btr_index_rec_validate(
 		return(FALSE);
 	}
 
+	const bool is_alter_metadata = page_is_leaf(page)
+		&& !page_has_prev(page)
+		&& index->is_primary() && index->table->instant
+		&& rec == page_rec_get_next_const(page_get_infimum_rec(page));
+
+	if (is_alter_metadata
+	    && !rec_is_alter_metadata(rec, page_is_comp(page))) {
+		btr_index_rec_validate_report(page, rec, index);
+
+		ib::error() << "First record is not ALTER TABLE metadata";
+		return FALSE;
+	}
+
 	if (!page_is_comp(page)) {
 		const ulint n_rec_fields = rec_get_n_fields_old(rec);
 		if (n_rec_fields == DICT_FLD__SYS_INDEXES__MERGE_THRESHOLD
 		    && index->id == DICT_INDEXES_ID) {
 			/* A record for older SYS_INDEXES table
 			(missing merge_threshold column) is acceptable. */
+		} else if (is_alter_metadata) {
+			if (n_rec_fields != ulint(index->n_fields) + 1) {
+				goto n_field_mismatch;
+			}
 		} else if (n_rec_fields < index->n_core_fields
 			   || n_rec_fields > index->n_fields) {
+n_field_mismatch:
 			btr_index_rec_validate_report(page, rec, index);
 
 			ib::error() << "Has " << rec_get_n_fields_old(rec)
@@ -4704,15 +4816,28 @@ btr_index_rec_validate(
 
 	offsets = rec_get_offsets(rec, index, offsets, page_is_leaf(page),
 				  ULINT_UNDEFINED, &heap);
+	const dict_field_t* field = index->fields;
+	ut_ad(rec_offs_n_fields(offsets)
+	      == ulint(index->n_fields) + is_alter_metadata);
 
-	for (unsigned i = 0; i < index->n_fields; i++) {
-		dict_field_t*	field = dict_index_get_nth_field(index, i);
-		ulint		fixed_size = dict_col_get_fixed_size(
-						dict_field_get_col(field),
-						page_is_comp(page));
-
+	for (unsigned i = 0; i < rec_offs_n_fields(offsets); i++) {
 		rec_get_nth_field_offs(offsets, i, &len);
 
+		ulint fixed_size;
+
+		if (is_alter_metadata && i == index->first_user_field()) {
+			fixed_size = FIELD_REF_SIZE;
+			if (len != FIELD_REF_SIZE
+			    || !rec_offs_nth_extern(offsets, i)) {
+				goto len_mismatch;
+			}
+
+			continue;
+		} else {
+			fixed_size = dict_col_get_fixed_size(
+				field->col, page_is_comp(page));
+		}
+
 		/* Note that if fixed_size != 0, it equals the
 		length of a fixed-size column in the clustered index.
 		We should adjust it here.
@@ -4724,8 +4849,8 @@ btr_index_rec_validate(
 		    && (field->prefix_len
 			? len > field->prefix_len
 			: (fixed_size && len != fixed_size))) {
+len_mismatch:
 			btr_index_rec_validate_report(page, rec, index);
-
 			ib::error	error;
 
 			error << "Field " << i << " len is " << len
@@ -4743,6 +4868,8 @@ btr_index_rec_validate(
 			}
 			return(FALSE);
 		}
+
+		field++;
 	}
 
 #ifdef VIRTUAL_INDEX_DEBUG

storage/innobase/btr/btr0pcur.cc

@@ -151,13 +151,20 @@ btr_pcur_store_position(
 		rec = page_rec_get_prev(rec);
 
 		ut_ad(!page_rec_is_infimum(rec));
-		ut_ad(!rec_is_metadata(rec, index));
+		if (UNIV_UNLIKELY(rec_is_metadata(rec, *index))) {
+			ut_ad(index->table->instant);
+			ut_ad(page_get_n_recs(block->frame) == 1);
+			ut_ad(page_is_leaf(page));
+			ut_ad(page_get_page_no(page) == index->page);
+			cursor->rel_pos = BTR_PCUR_AFTER_LAST_IN_TREE;
+			return;
+		}
 
 		cursor->rel_pos = BTR_PCUR_AFTER;
 	} else if (page_rec_is_infimum_low(offs)) {
 		rec = page_rec_get_next(rec);
 
-		if (rec_is_metadata(rec, index)) {
+		if (rec_is_metadata(rec, *index)) {
 			rec = page_rec_get_next(rec);
 			ut_ad(!page_rec_is_supremum(rec));
 		}

storage/innobase/btr/btr0sea.cc

@@ -1190,7 +1190,7 @@ void btr_search_drop_page_hash_index(buf_block_t* block)
 
 	rec = page_get_infimum_rec(page);
 	rec = page_rec_get_next_low(rec, page_is_comp(page));
-	if (rec_is_metadata(rec, index)) {
+	if (rec_is_metadata(rec, *index)) {
 		rec = page_rec_get_next_low(rec, page_is_comp(page));
 	}
 
@@ -1398,7 +1398,7 @@ btr_search_build_page_hash_index(
 
 	rec = page_rec_get_next_const(page_get_infimum_rec(page));
 
-	if (rec_is_metadata(rec, index)) {
+	if (rec_is_metadata(rec, *index)) {
 		rec = page_rec_get_next_const(rec);
 		if (!--n_recs) return;
 	}
@@ -1862,7 +1862,7 @@ btr_search_update_hash_on_insert(btr_cur_t* cursor, rw_lock_t* ahi_latch)
 				     n_bytes, index->id);
 	}
 
-	if (!page_rec_is_infimum(rec) && !rec_is_metadata(rec, index)) {
+	if (!page_rec_is_infimum(rec) && !rec_is_metadata(rec, *index)) {
 		offsets = rec_get_offsets(
 			rec, index, offsets, true,
 			btr_search_get_n_fields(n_fields, n_bytes), &heap);

storage/innobase/data/data0data.cc

@@ -60,7 +60,12 @@ void dtuple_t::trim(const dict_index_t& index)
 	for (; i > index.n_core_fields; i--) {
 		const dfield_t* dfield = dtuple_get_nth_field(this, i - 1);
 		const dict_col_t* col = dict_index_get_nth_col(&index, i - 1);
-		ut_ad(col->is_instant());
+
+		if (col->is_dropped()) {
+			continue;
+		}
+
+		ut_ad(col->is_added());
 		ulint len = dfield_get_len(dfield);
 		if (len != col->def_val.len) {
 			break;
@@ -632,14 +637,23 @@ dtuple_convert_big_rec(
 	n_fields = 0;
 	ulint longest_i;
 
+	const bool mblob = entry->is_alter_metadata();
+	ut_ad(entry->n_fields >= index->first_user_field() + mblob);
+	ut_ad(entry->n_fields - mblob <= index->n_fields);
+
+	if (mblob) {
+		longest_i = index->first_user_field();
+		dfield = dtuple_get_nth_field(entry, longest_i);
+		local_len = BTR_EXTERN_FIELD_REF_SIZE;
+		goto ext_write;
+	}
+
 	if (!dict_table_has_atomic_blobs(index->table)) {
-		/* ROW_FORMAT=REDUNDANT or ROW_FORMAT=COMPACT:
-		store a 768-byte prefix locally */
+		/* up to MySQL 5.1: store a 768-byte prefix locally */
 		local_len = BTR_EXTERN_FIELD_REF_SIZE
 			+ DICT_ANTELOPE_MAX_INDEX_COL_LEN;
 	} else {
-		/* ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED:
-		do not store any BLOB prefix locally */
+		/* new-format table: do not store any BLOB prefix locally */
 		local_len = BTR_EXTERN_FIELD_REF_SIZE;
 	}
 
@@ -649,11 +663,10 @@ dtuple_convert_big_rec(
 				      dict_index_get_n_fields(index),
 				      dict_table_page_size(index->table))) {
 		longest_i = 0;
-
 		for (ulint i = index->first_user_field(), longest = 0;
-		     i < entry->n_fields; i++) {
+		     i + mblob < entry->n_fields; i++) {
 			ulint	savings;
-			dfield = dtuple_get_nth_field(entry, i);
+			dfield = dtuple_get_nth_field(entry, i + mblob);
 
 			const dict_field_t* ifield = dict_index_get_nth_field(
 				index, i);
@@ -711,8 +724,8 @@ dtuple_convert_big_rec(
 		We store the first bytes locally to the record. Then
 		we can calculate all ordering fields in all indexes
 		from locally stored data. */
-
 		dfield = dtuple_get_nth_field(entry, longest_i);
+ext_write:
 		local_prefix_len = local_len - BTR_EXTERN_FIELD_REF_SIZE;
 
 		vector->append(

storage/innobase/fts/fts0fts.cc

@@ -3733,7 +3733,7 @@ fts_get_max_doc_id(
 			goto func_exit;
 		}
 
-		ut_ad(!rec_is_metadata(rec, index));
+		ut_ad(!rec_is_metadata(rec, *index));
 		offsets = rec_get_offsets(
 			rec, index, offsets, true, ULINT_UNDEFINED, &heap);
 

storage/innobase/handler/ha_innodb.cc

@@ -9462,12 +9462,14 @@ ha_innobase::change_active_index(
 		}
 #endif
 	} else {
-		dtuple_set_n_fields(m_prebuilt->search_tuple,
-				    m_prebuilt->index->n_fields);
+		ulint n_fields = dict_index_get_n_unique_in_tree(
+			m_prebuilt->index);
+
+		dtuple_set_n_fields(m_prebuilt->search_tuple, n_fields);
 
 		dict_index_copy_types(
 			m_prebuilt->search_tuple, m_prebuilt->index,
-			m_prebuilt->index->n_fields);
+			n_fields);
 
 		/* If it's FTS query and FTS_DOC_ID exists FTS_DOC_ID field is
 		always added to read_set. */

storage/innobase/include/btr0btr.h

@@ -421,6 +421,12 @@ void
 btr_write_autoinc(dict_index_t* index, ib_uint64_t autoinc, bool reset = false)
 	MY_ATTRIBUTE((nonnull));
 
+/** Write instant ALTER TABLE metadata to a root page.
+@param[in,out]	root	clustered index root page
+@param[in]	index	clustered index with instant ALTER TABLE
+@param[in,out]	mtr	mini-transaction */
+void btr_set_instant(buf_block_t* root, const dict_index_t& index, mtr_t* mtr);
+
 /*************************************************************//**
 Makes tree one level higher by splitting the root, and inserts
 the tuple. It is assumed that mtr contains an x-latch on the tree.

storage/innobase/include/data0data.h

 /*****************************************************************************
 
 Copyright (c) 1994, 2016, Oracle and/or its affiliates. All Rights Reserved.
-Copyright (c) 2017, MariaDB Corporation.
+Copyright (c) 2017, 2018, MariaDB Corporation.
 
 This program is free software; you can redistribute it and/or modify it under
 the terms of the GNU General Public License as published by the Free Software
@@ -645,6 +645,33 @@ struct dtuple_t {
 	inserted or updated.
 	@param[in]	index	index possibly with instantly added columns */
 	void trim(const dict_index_t& index);
+
+	/**
+	@param info_bits	the info_bits of a data tuple
+	@return whether this is a hidden metadata record
+	for instant ADD COLUMN or ALTER TABLE */
+	static bool is_alter_metadata(ulint info_bits)
+	{
+		return UNIV_UNLIKELY(info_bits == REC_INFO_METADATA_ALTER);
+	}
+
+	/**
+	@param info_bits	the info_bits of a data tuple
+	@return whether this is a hidden metadata record
+	for instant ADD COLUMN or ALTER TABLE */
+	static bool is_metadata(ulint info_bits)
+	{
+		return UNIV_UNLIKELY((info_bits & ~REC_INFO_DELETED_FLAG)
+				     == REC_INFO_METADATA_ADD);
+	}
+
+	/** @return whether this is a hidden metadata record
+	for instant ALTER TABLE (not only ADD COLUMN) */
+	bool is_alter_metadata() const { return is_alter_metadata(info_bits); }
+
+	/** @return whether this is a hidden metadata record
+	for instant ADD COLUMN or ALTER TABLE */
+	bool is_metadata() const { return is_metadata(info_bits); }
 };
 
 /** A slot for a field in a big rec vector */

storage/innobase/include/data0type.h

@@ -554,11 +554,55 @@ struct dtype_t{
 	{
 		return (prtype & DATA_VERSIONED) == DATA_VERS_END;
 	}
+
+	/** Set the type of the BLOB in the hidden metadata record. */
+	void metadata_blob_init()
+	{
+		prtype = DATA_NOT_NULL;
+		mtype = DATA_BLOB;
+		len = 0;
+		mbminlen = 0;
+		mbmaxlen = 0;
+	}
 };
 
 /** The DB_TRX_ID,DB_ROLL_PTR values for "no history is available" */
 extern const byte reset_trx_id[DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN];
 
+/** Info bit denoting the predefined minimum record: this bit is set
+if and only if the record is the first user record on a non-leaf
+B-tree page that is the leftmost page on its level
+(PAGE_LEVEL is nonzero and FIL_PAGE_PREV is FIL_NULL). */
+#define REC_INFO_MIN_REC_FLAG	0x10UL
+/** The delete-mark flag in info bits */
+#define REC_INFO_DELETED_FLAG	0x20UL
+
+/** Record status values for ROW_FORMAT=COMPACT,DYNAMIC,COMPRESSED */
+enum rec_comp_status_t {
+	/** User record (PAGE_LEVEL=0, heap>=PAGE_HEAP_NO_USER_LOW) */
+	REC_STATUS_ORDINARY = 0,
+	/** Node pointer record (PAGE_LEVEL>=0, heap>=PAGE_HEAP_NO_USER_LOW) */
+	REC_STATUS_NODE_PTR = 1,
+	/** The page infimum pseudo-record (heap=PAGE_HEAP_NO_INFIMUM) */
+	REC_STATUS_INFIMUM = 2,
+	/** The page supremum pseudo-record (heap=PAGE_HEAP_NO_SUPREMUM) */
+	REC_STATUS_SUPREMUM = 3,
+	/** Clustered index record that has been inserted or updated
+	after instant ADD COLUMN (more than dict_index_t::n_core_fields) */
+	REC_STATUS_INSTANT = 4
+};
+
+/** The dtuple_t::info_bits of the hidden metadata of instant ADD COLUMN.
+@see rec_is_metadata()
+@see rec_is_alter_metadata() */
+static const byte REC_INFO_METADATA_ADD
+	= REC_INFO_MIN_REC_FLAG | REC_STATUS_INSTANT;
+
+/** The dtuple_t::info_bits of the hidden metadata of instant ALTER TABLE.
+@see rec_is_metadata() */
+static const byte REC_INFO_METADATA_ALTER
+	= REC_INFO_METADATA_ADD | REC_INFO_DELETED_FLAG;
+
 #include "data0type.ic"
 
 #endif

storage/innobase/include/dict0dict.ic

@@ -490,7 +490,8 @@ dict_table_get_nth_v_col(
 	ut_ad(table);
 	ut_ad(pos < table->n_v_def);
 	ut_ad(table->magic_n == DICT_TABLE_MAGIC_N);
-	ut_ad(!table->v_cols[pos].m_col.is_instant());
+	ut_ad(!table->v_cols[pos].m_col.is_added());
+	ut_ad(!table->v_cols[pos].m_col.is_dropped());
 	return &table->v_cols[pos];
 }
 

storage/innobase/include/page0cur.ic

@@ -280,6 +280,7 @@ page_cur_tuple_insert(
 	*offsets = rec_get_offsets(rec, index, *offsets,
 				   page_is_leaf(cursor->block->frame),
 				   ULINT_UNDEFINED, heap);
+	ut_ad(size == rec_offs_size(*offsets));
 
 	if (buf_block_get_page_zip(cursor->block)) {
 		rec = page_cur_insert_rec_zip(

storage/innobase/include/page0page.h

@@ -1025,13 +1025,6 @@ page_get_direction(const page_t* page)
 inline
 uint16_t
 page_get_instant(const page_t* page);
-/** Assign the PAGE_INSTANT field.
-@param[in,out]	page	clustered index root page
-@param[in]	n	original number of clustered index fields
-@param[in,out]	mtr	mini-transaction */
-inline
-void
-page_set_instant(page_t* page, unsigned n, mtr_t* mtr);
 
 /**********************************************************//**
 Create an uncompressed B-tree index page.

storage/innobase/include/page0page.ic

@@ -1098,24 +1098,6 @@ page_get_instant(const page_t* page)
 #endif /* UNIV_DEBUG */
 	return(i >> 3);
 }
-
-/** Assign the PAGE_INSTANT field.
-@param[in,out]	page	clustered index root page
-@param[in]	n	original number of clustered index fields
-@param[in,out]	mtr	mini-transaction */
-inline
-void
-page_set_instant(page_t* page, unsigned n, mtr_t* mtr)
-{
-	ut_ad(fil_page_get_type(page) == FIL_PAGE_TYPE_INSTANT);
-	ut_ad(n > 0);
-	ut_ad(n < REC_MAX_N_FIELDS);
-	uint16_t i = page_header_get_field(page, PAGE_INSTANT);
-	ut_ad(i <= PAGE_NO_DIRECTION);
-	i |= n << 3;
-	mlog_write_ulint(PAGE_HEADER + PAGE_INSTANT + page, i,
-			 MLOG_2BYTES, mtr);
-}
 #endif /* !UNIV_INNOCHECKSUM */
 
 #ifdef UNIV_MATERIALIZE

storage/innobase/include/page0size.h

@@ -35,7 +35,7 @@ Created Nov 14, 2013 Vasil Dimov
 /** A BLOB field reference full of zero, for use in assertions and
 tests.Initially, BLOB field references are set to zero, in
 dtuple_convert_big_rec(). */
-extern const byte field_ref_zero[FIELD_REF_SIZE];
+extern const byte field_ref_zero[UNIV_PAGE_SIZE_MAX];
 
 #define PAGE_SIZE_T_SIZE_BITS	17
 

storage/innobase/include/rem0rec.h

@@ -39,15 +39,6 @@ Created 5/30/1994 Heikki Tuuri
 #include <ostream>
 #include <sstream>
 
-/* Info bit denoting the predefined minimum record: this bit is set
-if and only if the record is the first user record on a non-leaf
-B-tree page that is the leftmost page on its level
-(PAGE_LEVEL is nonzero and FIL_PAGE_PREV is FIL_NULL). */
-#define REC_INFO_MIN_REC_FLAG	0x10UL
-/* The deleted flag in info bits */
-#define REC_INFO_DELETED_FLAG	0x20UL	/* when bit is set to 1, it means the
-					record has been delete marked */
-
 /* Number of extra bytes in an old-style record,
 in addition to the data and the offsets */
 #define REC_N_OLD_EXTRA_BYTES	6
@@ -55,26 +46,6 @@ in addition to the data and the offsets */
 in addition to the data and the offsets */
 #define REC_N_NEW_EXTRA_BYTES	5
 
-/** Record status values for ROW_FORMAT=COMPACT,DYNAMIC,COMPRESSED */
-enum rec_comp_status_t {
-	/** User record (PAGE_LEVEL=0, heap>=PAGE_HEAP_NO_USER_LOW) */
-	REC_STATUS_ORDINARY = 0,
-	/** Node pointer record (PAGE_LEVEL>=0, heap>=PAGE_HEAP_NO_USER_LOW) */
-	REC_STATUS_NODE_PTR = 1,
-	/** The page infimum pseudo-record (heap=PAGE_HEAP_NO_INFIMUM) */
-	REC_STATUS_INFIMUM = 2,
-	/** The page supremum pseudo-record (heap=PAGE_HEAP_NO_SUPREMUM) */
-	REC_STATUS_SUPREMUM = 3,
-	/** Clustered index record that has been inserted or updated
-	after instant ADD COLUMN (more than dict_index_t::n_core_fields) */
-	REC_STATUS_COLUMNS_ADDED = 4
-};
-
-/** The dtuple_t::info_bits of the metadata pseudo-record.
-@see rec_is_metadata() */
-static const byte REC_INFO_METADATA
-	= REC_INFO_MIN_REC_FLAG | REC_STATUS_COLUMNS_ADDED;
-
 #define REC_NEW_STATUS		3	/* This is single byte bit-field */
 #define REC_NEW_STATUS_MASK	0x7UL
 #define REC_NEW_STATUS_SHIFT	0
@@ -296,7 +267,7 @@ rec_comp_status_t
 rec_get_status(const rec_t* rec)
 {
 	byte bits = rec[-REC_NEW_STATUS] & REC_NEW_STATUS_MASK;
-	ut_ad(bits <= REC_STATUS_COLUMNS_ADDED);
+	ut_ad(bits <= REC_STATUS_INSTANT);
 	return static_cast<rec_comp_status_t>(bits);
 }
 
@@ -307,12 +278,12 @@ inline
 void
 rec_set_status(rec_t* rec, byte bits)
 {
-	ut_ad(bits <= REC_STATUS_COLUMNS_ADDED);
+	ut_ad(bits <= REC_STATUS_INSTANT);
 	rec[-REC_NEW_STATUS] = (rec[-REC_NEW_STATUS] & ~REC_NEW_STATUS_MASK)
 		| bits;
 }
 
-/** Get the length of added field count in a REC_STATUS_COLUMNS_ADDED record.
+/** Get the length of added field count in a REC_STATUS_INSTANT record.
 @param[in]	n_add_field	number of added fields, minus one
 @return	storage size of the field count, in bytes */
 inline unsigned rec_get_n_add_field_len(ulint n_add_field)
@@ -321,8 +292,8 @@ inline unsigned rec_get_n_add_field_len(ulint n_add_field)
 	return n_add_field < 0x80 ? 1 : 2;
 }
 
-/** Set the added field count in a REC_STATUS_COLUMNS_ADDED record.
-@param[in,out]	header	variable header of a REC_STATUS_COLUMNS_ADDED record
+/** Set the added field count in a REC_STATUS_INSTANT record.
+@param[in,out]	header	variable header of a REC_STATUS_INSTANT record
 @param[in]	n_add	number of added fields, minus 1
 @return	record header before the number of added fields */
 inline void rec_set_n_add_field(byte*& header, ulint n_add)
@@ -781,20 +752,89 @@ rec_offs_comp(const ulint* offsets)
 }
 
 /** Determine if the record is the metadata pseudo-record
-in the clustered index.
+in the clustered index for instant ADD COLUMN or ALTER TABLE.
+@param[in]	rec	leaf page record
+@param[in]	comp	0 if ROW_FORMAT=REDUNDANT, else nonzero
+@return	whether the record is the metadata pseudo-record */
+inline bool rec_is_metadata(const rec_t* rec, ulint comp)
+{
+	bool is = !!(rec_get_info_bits(rec, comp) & REC_INFO_MIN_REC_FLAG);
+	ut_ad(!is || !comp || rec_get_status(rec) == REC_STATUS_INSTANT);
+	return is;
+}
+
+/** Determine if the record is the metadata pseudo-record
+in the clustered index for instant ADD COLUMN or ALTER TABLE.
 @param[in]	rec	leaf page record
 @param[in]	index	index of the record
 @return	whether the record is the metadata pseudo-record */
-inline bool rec_is_metadata(const rec_t* rec, const dict_index_t* index)
+inline bool rec_is_metadata(const rec_t* rec, const dict_index_t& index)
 {
-	bool is = rec_get_info_bits(rec, dict_table_is_comp(index->table))
-		& REC_INFO_MIN_REC_FLAG;
-	ut_ad(!is || index->is_instant());
-	ut_ad(!is || !dict_table_is_comp(index->table)
-	      || rec_get_status(rec) == REC_STATUS_COLUMNS_ADDED);
+	bool is = rec_is_metadata(rec, dict_table_is_comp(index.table));
+	ut_ad(!is || index.is_instant());
 	return is;
 }
 
+/** Determine if the record is the metadata pseudo-record
+in the clustered index for instant ADD COLUMN (not other ALTER TABLE).
+@param[in]	rec	leaf page record
+@param[in]	comp	0 if ROW_FORMAT=REDUNDANT, else nonzero
+@return	whether the record is the metadata pseudo-record */
+inline bool rec_is_add_metadata(const rec_t* rec, ulint comp)
+{
+	bool is = rec_get_info_bits(rec, comp) == REC_INFO_MIN_REC_FLAG;
+	ut_ad(!is || !comp || rec_get_status(rec) == REC_STATUS_INSTANT);
+	return is;
+}
+
+/** Determine if the record is the metadata pseudo-record
+in the clustered index for instant ADD COLUMN (not other ALTER TABLE).
+@param[in]	rec	leaf page record
+@param[in]	index	index of the record
+@return	whether the record is the metadata pseudo-record */
+inline bool rec_is_add_metadata(const rec_t* rec, const dict_index_t& index)
+{
+	bool is = rec_is_add_metadata(rec, dict_table_is_comp(index.table));
+	ut_ad(!is || index.is_instant());
+	return is;
+}
+
+/** Determine if the record is the metadata pseudo-record
+in the clustered index for instant ALTER TABLE (not plain ADD COLUMN).
+@param[in]	rec	leaf page record
+@param[in]	comp	0 if ROW_FORMAT=REDUNDANT, else nonzero
+@return	whether the record is the ALTER TABLE metadata pseudo-record */
+inline bool rec_is_alter_metadata(const rec_t* rec, ulint comp)
+{
+	bool is = !(~rec_get_info_bits(rec, comp)
+		    & (REC_INFO_MIN_REC_FLAG | REC_INFO_DELETED_FLAG));
+	ut_ad(!is || rec_is_metadata(rec, comp));
+	return is;
+}
+
+/** Determine if the record is the metadata pseudo-record
+in the clustered index for instant ALTER TABLE (not plain ADD COLUMN).
+@param[in]	rec	leaf page record
+@param[in]	index	index of the record
+@return	whether the record is the ALTER TABLE metadata pseudo-record */
+inline bool rec_is_alter_metadata(const rec_t* rec, const dict_index_t& index)
+{
+	bool is = rec_is_alter_metadata(rec, dict_table_is_comp(index.table));
+	ut_ad(!is || index.is_dummy || index.is_instant());
+	return is;
+}
+
+/** Determine if a record is delete-marked (not a metadata pseudo-record).
+@param[in]	rec	record
+@param[in]	comp	nonzero if ROW_FORMAT!=REDUNDANT
+@return	whether the record is a delete-marked user record */
+inline bool rec_is_delete_marked(const rec_t* rec, ulint comp)
+{
+	return (rec_get_info_bits(rec, comp)
+		& (REC_INFO_MIN_REC_FLAG | REC_INFO_DELETED_FLAG))
+		== REC_INFO_DELETED_FLAG;
+}
+
 /** Get the nth field from an index.
 @param[in]	rec	index record
 @param[in]	index	index
@@ -812,6 +852,7 @@ rec_get_nth_cfield(
 	ulint*			len)
 {
 	ut_ad(rec_offs_validate(rec, index, offsets));
+
 	if (!rec_offs_nth_default(offsets, n)) {
 		return rec_get_nth_field(rec, offsets, n, len);
 	}
@@ -958,7 +999,7 @@ rec_copy(
 @param[in]	fields		data fields
 @param[in]	n_fields	number of data fields
 @param[out]	extra		record header size
-@param[in]	status		REC_STATUS_ORDINARY or REC_STATUS_COLUMNS_ADDED
+@param[in]	status		REC_STATUS_ORDINARY or REC_STATUS_INSTANT
 @return	total size, in bytes */
 ulint
 rec_get_converted_size_temp(
@@ -975,7 +1016,7 @@ rec_get_converted_size_temp(
 @param[in,out]	offsets	offsets to the fields; in: rec_offs_n_fields(offsets)
 @param[in]	n_core	number of core fields (index->n_core_fields)
 @param[in]	def_val	default values for non-core fields
-@param[in]	status	REC_STATUS_ORDINARY or REC_STATUS_COLUMNS_ADDED */
+@param[in]	status	REC_STATUS_ORDINARY or REC_STATUS_INSTANT */
 void
 rec_init_offsets_temp(
 	const rec_t*		rec,
@@ -1002,8 +1043,7 @@ rec_init_offsets_temp(
 @param[in]	index		clustered or secondary index
 @param[in]	fields		data fields
 @param[in]	n_fields	number of data fields
-@param[in]	status		REC_STATUS_ORDINARY or REC_STATUS_COLUMNS_ADDED
-*/
+@param[in]	status		REC_STATUS_ORDINARY or REC_STATUS_INSTANT */
 void
 rec_convert_dtuple_to_temp(
 	rec_t*			rec,
@@ -1066,21 +1106,20 @@ rec_get_converted_size_comp_prefix(
 	ulint			n_fields,/*!< in: number of data fields */
 	ulint*			extra)	/*!< out: extra size */
 	MY_ATTRIBUTE((warn_unused_result, nonnull(1,2)));
-/**********************************************************//**
-Determines the size of a data tuple in ROW_FORMAT=COMPACT.
+
+/** Determine the size of a record in ROW_FORMAT=COMPACT.
+@param[in]	index		record descriptor. dict_table_is_comp()
+				is assumed to hold, even if it doesn't
+@param[in]	tuple		logical record
+@param[out]	extra		extra size
 @return total size */
 ulint
 rec_get_converted_size_comp(
-/*========================*/
-	const dict_index_t*	index,	/*!< in: record descriptor;
-					dict_table_is_comp() is
-					assumed to hold, even if
-					it does not */
-	rec_comp_status_t	status,	/*!< in: status bits of the record */
-	const dfield_t*		fields,	/*!< in: array of data fields */
-	ulint			n_fields,/*!< in: number of data fields */
-	ulint*			extra)	/*!< out: extra size */
-	MY_ATTRIBUTE((nonnull(1,3)));
+	const dict_index_t*	index,
+	const dtuple_t*		tuple,
+	ulint*			extra)
+	MY_ATTRIBUTE((nonnull(1,2)));
+
 /**********************************************************//**
 The following function returns the size of a data tuple when converted to
 a physical record.

storage/innobase/include/row0row.h

@@ -77,6 +77,7 @@ row_get_rec_roll_ptr(
 #define ROW_BUILD_FOR_PURGE	1	/*!< build row for purge. */
 #define ROW_BUILD_FOR_UNDO	2	/*!< build row for undo. */
 #define ROW_BUILD_FOR_INSERT	3	/*!< build row for insert. */
+
 /*****************************************************************//**
 When an insert or purge to a table is performed, this function builds
 the entry to be inserted into or purged from an index on the table.
@@ -230,6 +231,26 @@ row_rec_to_index_entry(
 	mem_heap_t*		heap)	/*!< in: memory heap from which
 					the memory needed is allocated */
 	MY_ATTRIBUTE((warn_unused_result));
+
+/** Convert a metadata record to a data tuple.
+@param[in]	rec		metadata record
+@param[in]	index		clustered index after instant ALTER TABLE
+@param[in]	offsets		rec_get_offsets(rec)
+@param[out]	n_ext		number of externally stored fields
+@param[in,out]	heap		memory heap for allocations
+@param[in]	info_bits	the info_bits after an update
+@param[in]	pad		whether to pad to index->n_fields */
+dtuple_t*
+row_metadata_to_tuple(
+	const rec_t*		rec,
+	const dict_index_t*	index,
+	const ulint*		offsets,
+	ulint*			n_ext,
+	mem_heap_t*		heap,
+	ulint			info_bits,
+	bool			pad)
+	MY_ATTRIBUTE((nonnull,warn_unused_result));
+
 /*******************************************************************//**
 Builds from a secondary index record a row reference with which we can
 search the clustered index record.