Commit d51e4fe6 authored by NeilBrown's avatar NeilBrown

Merge branch 'cluster' into for-next

parents 47d68979 97f6cd39
The cluster MD is a shared-device RAID for a cluster.
1. On-disk format
Separate write-intent-bitmap are used for each cluster node.
The bitmaps record all writes that may have been started on that node,
and may not yet have finished. The on-disk layout is:
0 4k 8k 12k
-------------------------------------------------------------------
| idle | md super | bm super [0] + bits |
| bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
| bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
| bm bits [3, contd] | | |
During "normal" functioning we assume the filesystem ensures that only one
node writes to any given block at a time, so a write
request will
- set the appropriate bit (if not already set)
- commit the write to all mirrors
- schedule the bit to be cleared after a timeout.
Reads are just handled normally. It is up to the filesystem to
ensure one node doesn't read from a location where another node (or the same
node) is writing.
2. DLM Locks for management
There are two locks for managing the device:
2.1 Bitmap lock resource (bm_lockres)
The bm_lockres protects individual node bitmaps. They are named in the
form bitmap001 for node 1, bitmap002 for node and so on. When a node
joins the cluster, it acquires the lock in PW mode and it stays so
during the lifetime the node is part of the cluster. The lock resource
number is based on the slot number returned by the DLM subsystem. Since
DLM starts node count from one and bitmap slots start from zero, one is
subtracted from the DLM slot number to arrive at the bitmap slot number.
3. Communication
Each node has to communicate with other nodes when starting or ending
resync, and metadata superblock updates.
3.1 Message Types
There are 3 types, of messages which are passed
3.1.1 METADATA_UPDATED: informs other nodes that the metadata has been
updated, and the node must re-read the md superblock. This is performed
synchronously.
3.1.2 RESYNC: informs other nodes that a resync is initiated or ended
so that each node may suspend or resume the region.
3.2 Communication mechanism
The DLM LVB is used to communicate within nodes of the cluster. There
are three resources used for the purpose:
3.2.1 Token: The resource which protects the entire communication
system. The node having the token resource is allowed to
communicate.
3.2.2 Message: The lock resource which carries the data to
communicate.
3.2.3 Ack: The resource, acquiring which means the message has been
acknowledged by all nodes in the cluster. The BAST of the resource
is used to inform the receive node that a node wants to communicate.
The algorithm is:
1. receive status
sender receiver receiver
ACK:CR ACK:CR ACK:CR
2. sender get EX of TOKEN
sender get EX of MESSAGE
sender receiver receiver
TOKEN:EX ACK:CR ACK:CR
MESSAGE:EX
ACK:CR
Sender checks that it still needs to send a message. Messages received
or other events that happened while waiting for the TOKEN may have made
this message inappropriate or redundant.
3. sender write LVB.
sender down-convert MESSAGE from EX to CR
sender try to get EX of ACK
[ wait until all receiver has *processed* the MESSAGE ]
[ triggered by bast of ACK ]
receiver get CR of MESSAGE
receiver read LVB
receiver processes the message
[ wait finish ]
receiver release ACK
sender receiver receiver
TOKEN:EX MESSAGE:CR MESSAGE:CR
MESSAGE:CR
ACK:EX
4. triggered by grant of EX on ACK (indicating all receivers have processed
message)
sender down-convert ACK from EX to CR
sender release MESSAGE
sender release TOKEN
receiver upconvert to EX of MESSAGE
receiver get CR of ACK
receiver release MESSAGE
sender receiver receiver
ACK:CR ACK:CR ACK:CR
4. Handling Failures
4.1 Node Failure
When a node fails, the DLM informs the cluster with the slot. The node
starts a cluster recovery thread. The cluster recovery thread:
- acquires the bitmap<number> lock of the failed node
- opens the bitmap
- reads the bitmap of the failed node
- copies the set bitmap to local node
- cleans the bitmap of the failed node
- releases bitmap<number> lock of the failed node
- initiates resync of the bitmap on the current node
The resync process, is the regular md resync. However, in a clustered
environment when a resync is performed, it needs to tell other nodes
of the areas which are suspended. Before a resync starts, the node
send out RESYNC_START with the (lo,hi) range of the area which needs
to be suspended. Each node maintains a suspend_list, which contains
the list of ranges which are currently suspended. On receiving
RESYNC_START, the node adds the range to the suspend_list. Similarly,
when the node performing resync finishes, it send RESYNC_FINISHED
to other nodes and other nodes remove the corresponding entry from
the suspend_list.
A helper function, should_suspend() can be used to check if a particular
I/O range should be suspended or not.
4.2 Device Failure
Device failures are handled and communicated with the metadata update
routine.
5. Adding a new Device
For adding a new device, it is necessary that all nodes "see" the new device
to be added. For this, the following algorithm is used:
1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
ioctl(ADD_NEW_DISC with disc.state set to MD_DISK_CLUSTER_ADD)
2. Node 1 sends NEWDISK with uuid and slot number
3. Other nodes issue kobject_uevent_env with uuid and slot number
(Steps 4,5 could be a udev rule)
4. In userspace, the node searches for the disk, perhaps
using blkid -t SUB_UUID=""
5. Other nodes issue either of the following depending on whether the disk
was found:
ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
disc.number set to slot number)
ioctl(CLUSTERED_DISK_NACK)
6. Other nodes drop lock on no-new-devs (CR) if device is found
7. Node 1 attempts EX lock on no-new-devs
8. If node 1 gets the lock, it sends METADATA_UPDATED after unmarking the disk
as SpareLocal
9. If not (get no-new-dev lock), it fails the operation and sends METADATA_UPDATED
10. Other nodes get the information whether a disk is added or not
by the following METADATA_UPDATED.
...@@ -175,6 +175,22 @@ config MD_FAULTY ...@@ -175,6 +175,22 @@ config MD_FAULTY
In unsure, say N. In unsure, say N.
config MD_CLUSTER
tristate "Cluster Support for MD (EXPERIMENTAL)"
depends on BLK_DEV_MD
depends on DLM
default n
---help---
Clustering support for MD devices. This enables locking and
synchronization across multiple systems on the cluster, so all
nodes in the cluster can access the MD devices simultaneously.
This brings the redundancy (and uptime) of RAID levels across the
nodes of the cluster.
If unsure, say N.
source "drivers/md/bcache/Kconfig" source "drivers/md/bcache/Kconfig"
config BLK_DEV_DM_BUILTIN config BLK_DEV_DM_BUILTIN
......
...@@ -30,6 +30,7 @@ obj-$(CONFIG_MD_RAID10) += raid10.o ...@@ -30,6 +30,7 @@ obj-$(CONFIG_MD_RAID10) += raid10.o
obj-$(CONFIG_MD_RAID456) += raid456.o obj-$(CONFIG_MD_RAID456) += raid456.o
obj-$(CONFIG_MD_MULTIPATH) += multipath.o obj-$(CONFIG_MD_MULTIPATH) += multipath.o
obj-$(CONFIG_MD_FAULTY) += faulty.o obj-$(CONFIG_MD_FAULTY) += faulty.o
obj-$(CONFIG_MD_CLUSTER) += md-cluster.o
obj-$(CONFIG_BCACHE) += bcache/ obj-$(CONFIG_BCACHE) += bcache/
obj-$(CONFIG_BLK_DEV_MD) += md-mod.o obj-$(CONFIG_BLK_DEV_MD) += md-mod.o
obj-$(CONFIG_BLK_DEV_DM) += dm-mod.o obj-$(CONFIG_BLK_DEV_DM) += dm-mod.o
......
This diff is collapsed.
...@@ -130,8 +130,9 @@ typedef struct bitmap_super_s { ...@@ -130,8 +130,9 @@ typedef struct bitmap_super_s {
__le32 write_behind; /* 60 number of outstanding write-behind writes */ __le32 write_behind; /* 60 number of outstanding write-behind writes */
__le32 sectors_reserved; /* 64 number of 512-byte sectors that are __le32 sectors_reserved; /* 64 number of 512-byte sectors that are
* reserved for the bitmap. */ * reserved for the bitmap. */
__le32 nodes; /* 68 the maximum number of nodes in cluster. */
__u8 pad[256 - 68]; /* set to zero */ __u8 cluster_name[64]; /* 72 cluster name to which this md belongs */
__u8 pad[256 - 136]; /* set to zero */
} bitmap_super_t; } bitmap_super_t;
/* notes: /* notes:
...@@ -226,12 +227,13 @@ struct bitmap { ...@@ -226,12 +227,13 @@ struct bitmap {
wait_queue_head_t behind_wait; wait_queue_head_t behind_wait;
struct kernfs_node *sysfs_can_clear; struct kernfs_node *sysfs_can_clear;
int cluster_slot; /* Slot offset for clustered env */
}; };
/* the bitmap API */ /* the bitmap API */
/* these are used only by md/bitmap */ /* these are used only by md/bitmap */
int bitmap_create(struct mddev *mddev); struct bitmap *bitmap_create(struct mddev *mddev, int slot);
int bitmap_load(struct mddev *mddev); int bitmap_load(struct mddev *mddev);
void bitmap_flush(struct mddev *mddev); void bitmap_flush(struct mddev *mddev);
void bitmap_destroy(struct mddev *mddev); void bitmap_destroy(struct mddev *mddev);
...@@ -260,6 +262,8 @@ void bitmap_daemon_work(struct mddev *mddev); ...@@ -260,6 +262,8 @@ void bitmap_daemon_work(struct mddev *mddev);
int bitmap_resize(struct bitmap *bitmap, sector_t blocks, int bitmap_resize(struct bitmap *bitmap, sector_t blocks,
int chunksize, int init); int chunksize, int init);
int bitmap_copy_from_slot(struct mddev *mddev, int slot,
sector_t *lo, sector_t *hi, bool clear_bits);
#endif #endif
#endif #endif
This diff is collapsed.
#ifndef _MD_CLUSTER_H
#define _MD_CLUSTER_H
#include "md.h"
struct mddev;
struct md_rdev;
struct md_cluster_operations {
int (*join)(struct mddev *mddev, int nodes);
int (*leave)(struct mddev *mddev);
int (*slot_number)(struct mddev *mddev);
void (*resync_info_update)(struct mddev *mddev, sector_t lo, sector_t hi);
int (*resync_start)(struct mddev *mddev, sector_t lo, sector_t hi);
void (*resync_finish)(struct mddev *mddev);
int (*metadata_update_start)(struct mddev *mddev);
int (*metadata_update_finish)(struct mddev *mddev);
int (*metadata_update_cancel)(struct mddev *mddev);
int (*area_resyncing)(struct mddev *mddev, sector_t lo, sector_t hi);
int (*add_new_disk_start)(struct mddev *mddev, struct md_rdev *rdev);
int (*add_new_disk_finish)(struct mddev *mddev);
int (*new_disk_ack)(struct mddev *mddev, bool ack);
int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
int (*gather_bitmaps)(struct md_rdev *rdev);
};
#endif /* _MD_CLUSTER_H */
This diff is collapsed.
...@@ -23,6 +23,7 @@ ...@@ -23,6 +23,7 @@
#include <linux/timer.h> #include <linux/timer.h>
#include <linux/wait.h> #include <linux/wait.h>
#include <linux/workqueue.h> #include <linux/workqueue.h>
#include "md-cluster.h"
#define MaxSector (~(sector_t)0) #define MaxSector (~(sector_t)0)
...@@ -170,6 +171,10 @@ enum flag_bits { ...@@ -170,6 +171,10 @@ enum flag_bits {
* a want_replacement device with same * a want_replacement device with same
* raid_disk number. * raid_disk number.
*/ */
Candidate, /* For clustered environments only:
* This device is seen locally but not
* by the whole cluster
*/
}; };
#define BB_LEN_MASK (0x00000000000001FFULL) #define BB_LEN_MASK (0x00000000000001FFULL)
...@@ -202,6 +207,8 @@ extern int rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, int sectors, ...@@ -202,6 +207,8 @@ extern int rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
int is_new); int is_new);
extern void md_ack_all_badblocks(struct badblocks *bb); extern void md_ack_all_badblocks(struct badblocks *bb);
struct md_cluster_info;
struct mddev { struct mddev {
void *private; void *private;
struct md_personality *pers; struct md_personality *pers;
...@@ -430,6 +437,8 @@ struct mddev { ...@@ -430,6 +437,8 @@ struct mddev {
unsigned long daemon_sleep; /* how many jiffies between updates? */ unsigned long daemon_sleep; /* how many jiffies between updates? */
unsigned long max_write_behind; /* write-behind mode */ unsigned long max_write_behind; /* write-behind mode */
int external; int external;
int nodes; /* Maximum number of nodes in the cluster */
char cluster_name[64]; /* Name of the cluster */
} bitmap_info; } bitmap_info;
atomic_t max_corr_read_errors; /* max read retries */ atomic_t max_corr_read_errors; /* max read retries */
...@@ -448,6 +457,7 @@ struct mddev { ...@@ -448,6 +457,7 @@ struct mddev {
struct work_struct flush_work; struct work_struct flush_work;
struct work_struct event_work; /* used by dm to report failure event */ struct work_struct event_work; /* used by dm to report failure event */
void (*sync_super)(struct mddev *mddev, struct md_rdev *rdev); void (*sync_super)(struct mddev *mddev, struct md_rdev *rdev);
struct md_cluster_info *cluster_info;
}; };
static inline int __must_check mddev_lock(struct mddev *mddev) static inline int __must_check mddev_lock(struct mddev *mddev)
...@@ -608,6 +618,11 @@ static inline void safe_put_page(struct page *p) ...@@ -608,6 +618,11 @@ static inline void safe_put_page(struct page *p)
extern int register_md_personality(struct md_personality *p); extern int register_md_personality(struct md_personality *p);
extern int unregister_md_personality(struct md_personality *p); extern int unregister_md_personality(struct md_personality *p);
extern int register_md_cluster_operations(struct md_cluster_operations *ops,
struct module *module);
extern int unregister_md_cluster_operations(void);
extern int md_setup_cluster(struct mddev *mddev, int nodes);
extern void md_cluster_stop(struct mddev *mddev);
extern struct md_thread *md_register_thread( extern struct md_thread *md_register_thread(
void (*run)(struct md_thread *thread), void (*run)(struct md_thread *thread),
struct mddev *mddev, struct mddev *mddev,
...@@ -654,6 +669,10 @@ extern struct bio *bio_alloc_mddev(gfp_t gfp_mask, int nr_iovecs, ...@@ -654,6 +669,10 @@ extern struct bio *bio_alloc_mddev(gfp_t gfp_mask, int nr_iovecs,
struct mddev *mddev); struct mddev *mddev);
extern void md_unplug(struct blk_plug_cb *cb, bool from_schedule); extern void md_unplug(struct blk_plug_cb *cb, bool from_schedule);
extern void md_reload_sb(struct mddev *mddev);
extern void md_update_sb(struct mddev *mddev, int force);
extern void md_kick_rdev_from_array(struct md_rdev * rdev);
struct md_rdev *md_find_rdev_nr_rcu(struct mddev *mddev, int nr);
static inline int mddev_check_plugged(struct mddev *mddev) static inline int mddev_check_plugged(struct mddev *mddev)
{ {
return !!blk_check_plugged(md_unplug, mddev, return !!blk_check_plugged(md_unplug, mddev,
...@@ -669,4 +688,9 @@ static inline void rdev_dec_pending(struct md_rdev *rdev, struct mddev *mddev) ...@@ -669,4 +688,9 @@ static inline void rdev_dec_pending(struct md_rdev *rdev, struct mddev *mddev)
} }
} }
extern struct md_cluster_operations *md_cluster_ops;
static inline int mddev_is_clustered(struct mddev *mddev)
{
return mddev->cluster_info && mddev->bitmap_info.nodes > 1;
}
#endif /* _MD_MD_H */ #endif /* _MD_MD_H */
...@@ -539,7 +539,13 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect ...@@ -539,7 +539,13 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect
has_nonrot_disk = 0; has_nonrot_disk = 0;
choose_next_idle = 0; choose_next_idle = 0;
choose_first = (conf->mddev->recovery_cp < this_sector + sectors); if ((conf->mddev->recovery_cp < this_sector + sectors) ||
(mddev_is_clustered(conf->mddev) &&
md_cluster_ops->area_resyncing(conf->mddev, this_sector,
this_sector + sectors)))
choose_first = 1;
else
choose_first = 0;
for (disk = 0 ; disk < conf->raid_disks * 2 ; disk++) { for (disk = 0 ; disk < conf->raid_disks * 2 ; disk++) {
sector_t dist; sector_t dist;
...@@ -1102,8 +1108,10 @@ static void make_request(struct mddev *mddev, struct bio * bio) ...@@ -1102,8 +1108,10 @@ static void make_request(struct mddev *mddev, struct bio * bio)
md_write_start(mddev, bio); /* wait on superblock update early */ md_write_start(mddev, bio); /* wait on superblock update early */
if (bio_data_dir(bio) == WRITE && if (bio_data_dir(bio) == WRITE &&
bio_end_sector(bio) > mddev->suspend_lo && ((bio_end_sector(bio) > mddev->suspend_lo &&
bio->bi_iter.bi_sector < mddev->suspend_hi) { bio->bi_iter.bi_sector < mddev->suspend_hi) ||
(mddev_is_clustered(mddev) &&
md_cluster_ops->area_resyncing(mddev, bio->bi_iter.bi_sector, bio_end_sector(bio))))) {
/* As the suspend_* range is controlled by /* As the suspend_* range is controlled by
* userspace, we want an interruptible * userspace, we want an interruptible
* wait. * wait.
...@@ -1114,7 +1122,10 @@ static void make_request(struct mddev *mddev, struct bio * bio) ...@@ -1114,7 +1122,10 @@ static void make_request(struct mddev *mddev, struct bio * bio)
prepare_to_wait(&conf->wait_barrier, prepare_to_wait(&conf->wait_barrier,
&w, TASK_INTERRUPTIBLE); &w, TASK_INTERRUPTIBLE);
if (bio_end_sector(bio) <= mddev->suspend_lo || if (bio_end_sector(bio) <= mddev->suspend_lo ||
bio->bi_iter.bi_sector >= mddev->suspend_hi) bio->bi_iter.bi_sector >= mddev->suspend_hi ||
(mddev_is_clustered(mddev) &&
!md_cluster_ops->area_resyncing(mddev,
bio->bi_iter.bi_sector, bio_end_sector(bio))))
break; break;
schedule(); schedule();
} }
...@@ -1561,6 +1572,7 @@ static int raid1_spare_active(struct mddev *mddev) ...@@ -1561,6 +1572,7 @@ static int raid1_spare_active(struct mddev *mddev)
struct md_rdev *rdev = conf->mirrors[i].rdev; struct md_rdev *rdev = conf->mirrors[i].rdev;
struct md_rdev *repl = conf->mirrors[conf->raid_disks + i].rdev; struct md_rdev *repl = conf->mirrors[conf->raid_disks + i].rdev;
if (repl if (repl
&& !test_bit(Candidate, &repl->flags)
&& repl->recovery_offset == MaxSector && repl->recovery_offset == MaxSector
&& !test_bit(Faulty, &repl->flags) && !test_bit(Faulty, &repl->flags)
&& !test_and_set_bit(In_sync, &repl->flags)) { && !test_and_set_bit(In_sync, &repl->flags)) {
......
...@@ -78,6 +78,12 @@ ...@@ -78,6 +78,12 @@
#define MD_DISK_ACTIVE 1 /* disk is running or spare disk */ #define MD_DISK_ACTIVE 1 /* disk is running or spare disk */
#define MD_DISK_SYNC 2 /* disk is in sync with the raid set */ #define MD_DISK_SYNC 2 /* disk is in sync with the raid set */
#define MD_DISK_REMOVED 3 /* disk is in sync with the raid set */ #define MD_DISK_REMOVED 3 /* disk is in sync with the raid set */
#define MD_DISK_CLUSTER_ADD 4 /* Initiate a disk add across the cluster
* For clustered enviroments only.
*/
#define MD_DISK_CANDIDATE 5 /* disk is added as spare (local) until confirmed
* For clustered enviroments only.
*/
#define MD_DISK_WRITEMOSTLY 9 /* disk is "write-mostly" is RAID1 config. #define MD_DISK_WRITEMOSTLY 9 /* disk is "write-mostly" is RAID1 config.
* read requests will only be sent here in * read requests will only be sent here in
...@@ -101,6 +107,7 @@ typedef struct mdp_device_descriptor_s { ...@@ -101,6 +107,7 @@ typedef struct mdp_device_descriptor_s {
#define MD_SB_CLEAN 0 #define MD_SB_CLEAN 0
#define MD_SB_ERRORS 1 #define MD_SB_ERRORS 1
#define MD_SB_CLUSTERED 5 /* MD is clustered */
#define MD_SB_BITMAP_PRESENT 8 /* bitmap may be present nearby */ #define MD_SB_BITMAP_PRESENT 8 /* bitmap may be present nearby */
/* /*
......
...@@ -62,6 +62,7 @@ ...@@ -62,6 +62,7 @@
#define STOP_ARRAY _IO (MD_MAJOR, 0x32) #define STOP_ARRAY _IO (MD_MAJOR, 0x32)
#define STOP_ARRAY_RO _IO (MD_MAJOR, 0x33) #define STOP_ARRAY_RO _IO (MD_MAJOR, 0x33)
#define RESTART_ARRAY_RW _IO (MD_MAJOR, 0x34) #define RESTART_ARRAY_RW _IO (MD_MAJOR, 0x34)
#define CLUSTERED_DISK_NACK _IO (MD_MAJOR, 0x35)
/* 63 partitions with the alternate major number (mdp) */ /* 63 partitions with the alternate major number (mdp) */
#define MdpMinorShift 6 #define MdpMinorShift 6
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment