| blob | cd2cb2fc85ead940394873d941b24a7858ea3336 |
1 dm-raid
2 =======
4 The device-mapper RAID (dm-raid) target provides a bridge from DM to MD.
5 It allows the MD RAID drivers to be accessed using a device-mapper
6 interface.
9 Mapping Table Interface
10 -----------------------
11 The target is named "raid" and it accepts the following parameters:
13 <raid_type> <#raid_params> <raid_params> \
14 <#raid_devs> <metadata_dev0> <dev0> [.. <metadata_devN> <devN>]
16 <raid_type>:
17 raid0 RAID0 striping (no resilience)
18 raid1 RAID1 mirroring
19 raid4 RAID4 with dedicated last parity disk
20 raid5_n RAID5 with dedicated last parity disk supporting takeover
21 Same as raid4
22 -Transitory layout
23 raid5_la RAID5 left asymmetric
24 - rotating parity 0 with data continuation
25 raid5_ra RAID5 right asymmetric
26 - rotating parity N with data continuation
27 raid5_ls RAID5 left symmetric
28 - rotating parity 0 with data restart
29 raid5_rs RAID5 right symmetric
30 - rotating parity N with data restart
31 raid6_zr RAID6 zero restart
32 - rotating parity zero (left-to-right) with data restart
33 raid6_nr RAID6 N restart
34 - rotating parity N (right-to-left) with data restart
35 raid6_nc RAID6 N continue
36 - rotating parity N (right-to-left) with data continuation
37 raid6_n_6 RAID6 with dedicate parity disks
38 - parity and Q-syndrome on the last 2 disks;
39 layout for takeover from/to raid4/raid5_n
40 raid6_la_6 Same as "raid_la" plus dedicated last Q-syndrome disk
41 - layout for takeover from raid5_la from/to raid6
42 raid6_ra_6 Same as "raid5_ra" dedicated last Q-syndrome disk
43 - layout for takeover from raid5_ra from/to raid6
44 raid6_ls_6 Same as "raid5_ls" dedicated last Q-syndrome disk
45 - layout for takeover from raid5_ls from/to raid6
46 raid6_rs_6 Same as "raid5_rs" dedicated last Q-syndrome disk
47 - layout for takeover from raid5_rs from/to raid6
48 raid10 Various RAID10 inspired algorithms chosen by additional params
49 (see raid10_format and raid10_copies below)
50 - RAID10: Striped Mirrors (aka 'Striping on top of mirrors')
51 - RAID1E: Integrated Adjacent Stripe Mirroring
52 - RAID1E: Integrated Offset Stripe Mirroring
53 - and other similar RAID10 variants
55 Reference: Chapter 4 of
56 http://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf
58 <#raid_params>: The number of parameters that follow.
60 <raid_params> consists of
61 Mandatory parameters:
62 <chunk_size>: Chunk size in sectors. This parameter is often known as
63 "stripe size". It is the only mandatory parameter and
64 is placed first.
66 followed by optional parameters (in any order):
67 [sync|nosync] Force or prevent RAID initialization.
69 [rebuild <idx>] Rebuild drive number 'idx' (first drive is 0).
71 [daemon_sleep <ms>]
72 Interval between runs of the bitmap daemon that
73 clear bits. A longer interval means less bitmap I/O but
74 resyncing after a failure is likely to take longer.
76 [min_recovery_rate <kB/sec/disk>] Throttle RAID initialization
77 [max_recovery_rate <kB/sec/disk>] Throttle RAID initialization
78 [write_mostly <idx>] Mark drive index 'idx' write-mostly.
79 [max_write_behind <sectors>] See '--write-behind=' (man mdadm)
80 [stripe_cache <sectors>] Stripe cache size (RAID 4/5/6 only)
81 [region_size <sectors>]
82 The region_size multiplied by the number of regions is the
83 logical size of the array. The bitmap records the device
84 synchronisation state for each region.
86 [raid10_copies <# copies>]
87 [raid10_format <near|far|offset>]
88 These two options are used to alter the default layout of
89 a RAID10 configuration. The number of copies is can be
90 specified, but the default is 2. There are also three
91 variations to how the copies are laid down - the default
92 is "near". Near copies are what most people think of with
93 respect to mirroring. If these options are left unspecified,
94 or 'raid10_copies 2' and/or 'raid10_format near' are given,
95 then the layouts for 2, 3 and 4 devices are:
96 2 drives 3 drives 4 drives
97 -------- ---------- --------------
98 A1 A1 A1 A1 A2 A1 A1 A2 A2
99 A2 A2 A2 A3 A3 A3 A3 A4 A4
100 A3 A3 A4 A4 A5 A5 A5 A6 A6
101 A4 A4 A5 A6 A6 A7 A7 A8 A8
102 .. .. .. .. .. .. .. .. ..
103 The 2-device layout is equivalent 2-way RAID1. The 4-device
104 layout is what a traditional RAID10 would look like. The
105 3-device layout is what might be called a 'RAID1E - Integrated
106 Adjacent Stripe Mirroring'.
108 If 'raid10_copies 2' and 'raid10_format far', then the layouts
109 for 2, 3 and 4 devices are:
110 2 drives 3 drives 4 drives
111 -------- -------------- --------------------
112 A1 A2 A1 A2 A3 A1 A2 A3 A4
113 A3 A4 A4 A5 A6 A5 A6 A7 A8
114 A5 A6 A7 A8 A9 A9 A10 A11 A12
115 .. .. .. .. .. .. .. .. ..
116 A2 A1 A3 A1 A2 A2 A1 A4 A3
117 A4 A3 A6 A4 A5 A6 A5 A8 A7
118 A6 A5 A9 A7 A8 A10 A9 A12 A11
119 .. .. .. .. .. .. .. .. ..
121 If 'raid10_copies 2' and 'raid10_format offset', then the
122 layouts for 2, 3 and 4 devices are:
123 2 drives 3 drives 4 drives
124 -------- ------------ -----------------
125 A1 A2 A1 A2 A3 A1 A2 A3 A4
126 A2 A1 A3 A1 A2 A2 A1 A4 A3
127 A3 A4 A4 A5 A6 A5 A6 A7 A8
128 A4 A3 A6 A4 A5 A6 A5 A8 A7
129 A5 A6 A7 A8 A9 A9 A10 A11 A12
130 A6 A5 A9 A7 A8 A10 A9 A12 A11
131 .. .. .. .. .. .. .. .. ..
132 Here we see layouts closely akin to 'RAID1E - Integrated
133 Offset Stripe Mirroring'.
135 [delta_disks <N>]
136 The delta_disks option value (-251 < N < +251) triggers
137 device removal (negative value) or device addition (positive
138 value) to any reshape supporting raid levels 4/5/6 and 10.
139 RAID levels 4/5/6 allow for addition of devices (metadata
140 and data device tuple), raid10_near and raid10_offset only
141 allow for device addition. raid10_far does not support any
142 reshaping at all.
143 A minimum of devices have to be kept to enforce resilience,
144 which is 3 devices for raid4/5 and 4 devices for raid6.
146 [data_offset <sectors>]
147 This option value defines the offset into each data device
148 where the data starts. This is used to provide out-of-place
149 reshaping space to avoid writing over data whilst
150 changing the layout of stripes, hence an interruption/crash
151 may happen at any time without the risk of losing data.
152 E.g. when adding devices to an existing raid set during
153 forward reshaping, the out-of-place space will be allocated
154 at the beginning of each raid device. The kernel raid4/5/6/10
155 MD personalities supporting such device addition will read the data from
156 the existing first stripes (those with smaller number of stripes)
157 starting at data_offset to fill up a new stripe with the larger
158 number of stripes, calculate the redundancy blocks (CRC/Q-syndrome)
159 and write that new stripe to offset 0. Same will be applied to all
160 N-1 other new stripes. This out-of-place scheme is used to change
161 the RAID type (i.e. the allocation algorithm) as well, e.g.
162 changing from raid5_ls to raid5_n.
164 [journal_dev <dev>]
165 This option adds a journal device to raid4/5/6 raid sets and
166 uses it to close the 'write hole' caused by the non-atomic updates
167 to the component devices which can cause data loss during recovery.
168 The journal device is used as writethrough thus causing writes to
169 be throttled versus non-journaled raid4/5/6 sets.
170 Takeover/reshape is not possible with a raid4/5/6 journal device;
171 it has to be deconfigured before requesting these.
173 <#raid_devs>: The number of devices composing the array.
174 Each device consists of two entries. The first is the device
175 containing the metadata (if any); the second is the one containing the
176 data. A Maximum of 64 metadata/data device entries are supported
177 up to target version 1.8.0.
178 1.9.0 supports up to 253 which is enforced by the used MD kernel runtime.
180 If a drive has failed or is missing at creation time, a '-' can be
181 given for both the metadata and data drives for a given position.
184 Example Tables
185 --------------
186 # RAID4 - 4 data drives, 1 parity (no metadata devices)
187 # No metadata devices specified to hold superblock/bitmap info
188 # Chunk size of 1MiB
189 # (Lines separated for easy reading)
191 0 1960893648 raid \
192 raid4 1 2048 \
193 5 - 8:17 - 8:33 - 8:49 - 8:65 - 8:81
195 # RAID4 - 4 data drives, 1 parity (with metadata devices)
196 # Chunk size of 1MiB, force RAID initialization,
197 # min recovery rate at 20 kiB/sec/disk
199 0 1960893648 raid \
200 raid4 4 2048 sync min_recovery_rate 20 \
201 5 8:17 8:18 8:33 8:34 8:49 8:50 8:65 8:66 8:81 8:82
204 Status Output
205 -------------
206 'dmsetup table' displays the table used to construct the mapping.
207 The optional parameters are always printed in the order listed
208 above with "sync" or "nosync" always output ahead of the other
209 arguments, regardless of the order used when originally loading the table.
210 Arguments that can be repeated are ordered by value.
213 'dmsetup status' yields information on the state and health of the array.
214 The output is as follows (normally a single line, but expanded here for
215 clarity):
216 1: <s> <l> raid \
217 2: <raid_type> <#devices> <health_chars> \
218 3: <sync_ratio> <sync_action> <mismatch_cnt>
220 Line 1 is the standard output produced by device-mapper.
221 Line 2 & 3 are produced by the raid target and are best explained by example:
222 0 1960893648 raid raid4 5 AAAAA 2/490221568 init 0
223 Here we can see the RAID type is raid4, there are 5 devices - all of
224 which are 'A'live, and the array is 2/490221568 complete with its initial
225 recovery. Here is a fuller description of the individual fields:
226 <raid_type> Same as the <raid_type> used to create the array.
227 <health_chars> One char for each device, indicating: 'A' = alive and
228 in-sync, 'a' = alive but not in-sync, 'D' = dead/failed.
229 <sync_ratio> The ratio indicating how much of the array has undergone
230 the process described by 'sync_action'. If the
231 'sync_action' is "check" or "repair", then the process
232 of "resync" or "recover" can be considered complete.
233 <sync_action> One of the following possible states:
234 idle - No synchronization action is being performed.
235 frozen - The current action has been halted.
236 resync - Array is undergoing its initial synchronization
237 or is resynchronizing after an unclean shutdown
238 (possibly aided by a bitmap).
239 recover - A device in the array is being rebuilt or
240 replaced.
241 check - A user-initiated full check of the array is
242 being performed. All blocks are read and
243 checked for consistency. The number of
244 discrepancies found are recorded in
245 <mismatch_cnt>. No changes are made to the
246 array by this action.
247 repair - The same as "check", but discrepancies are
248 corrected.
249 reshape - The array is undergoing a reshape.
250 <mismatch_cnt> The number of discrepancies found between mirror copies
251 in RAID1/10 or wrong parity values found in RAID4/5/6.
252 This value is valid only after a "check" of the array
253 is performed. A healthy array has a 'mismatch_cnt' of 0.
254 <data_offset> The current data offset to the start of the user data on
255 each component device of a raid set (see the respective
256 raid parameter to support out-of-place reshaping).
257 <journal_char> 'A' - active raid4/5/6 journal device.
258 'D' - dead journal device.
259 '-' - no journal device.
262 Message Interface
263 -----------------
264 The dm-raid target will accept certain actions through the 'message' interface.
265 ('man dmsetup' for more information on the message interface.) These actions
266 include:
267 "idle" - Halt the current sync action.
268 "frozen" - Freeze the current sync action.
269 "resync" - Initiate/continue a resync.
270 "recover"- Initiate/continue a recover process.
271 "check" - Initiate a check (i.e. a "scrub") of the array.
272 "repair" - Initiate a repair of the array.
275 Discard Support
276 ---------------
277 The implementation of discard support among hardware vendors varies.
278 When a block is discarded, some storage devices will return zeroes when
279 the block is read. These devices set the 'discard_zeroes_data'
280 attribute. Other devices will return random data. Confusingly, some
281 devices that advertise 'discard_zeroes_data' will not reliably return
282 zeroes when discarded blocks are read! Since RAID 4/5/6 uses blocks
283 from a number of devices to calculate parity blocks and (for performance
284 reasons) relies on 'discard_zeroes_data' being reliable, it is important
285 that the devices be consistent. Blocks may be discarded in the middle
286 of a RAID 4/5/6 stripe and if subsequent read results are not
287 consistent, the parity blocks may be calculated differently at any time;
288 making the parity blocks useless for redundancy. It is important to
289 understand how your hardware behaves with discards if you are going to
290 enable discards with RAID 4/5/6.
292 Since the behavior of storage devices is unreliable in this respect,
293 even when reporting 'discard_zeroes_data', by default RAID 4/5/6
294 discard support is disabled -- this ensures data integrity at the
295 expense of losing some performance.
297 Storage devices that properly support 'discard_zeroes_data' are
298 increasingly whitelisted in the kernel and can thus be trusted.
300 For trusted devices, the following dm-raid module parameter can be set
301 to safely enable discard support for RAID 4/5/6:
302 'devices_handle_discards_safely'
305 Version History
306 ---------------
307 1.0.0 Initial version. Support for RAID 4/5/6
308 1.1.0 Added support for RAID 1
309 1.2.0 Handle creation of arrays that contain failed devices.
310 1.3.0 Added support for RAID 10
311 1.3.1 Allow device replacement/rebuild for RAID 10
312 1.3.2 Fix/improve redundancy checking for RAID10
313 1.4.0 Non-functional change. Removes arg from mapping function.
314 1.4.1 RAID10 fix redundancy validation checks (commit 55ebbb5).
315 1.4.2 Add RAID10 "far" and "offset" algorithm support.
316 1.5.0 Add message interface to allow manipulation of the sync_action.
317 New status (STATUSTYPE_INFO) fields: sync_action and mismatch_cnt.
318 1.5.1 Add ability to restore transiently failed devices on resume.
319 1.5.2 'mismatch_cnt' is zero unless [last_]sync_action is "check".
320 1.6.0 Add discard support (and devices_handle_discard_safely module param).
321 1.7.0 Add support for MD RAID0 mappings.
322 1.8.0 Explicitly check for compatible flags in the superblock metadata
323 and reject to start the raid set if any are set by a newer
324 target version, thus avoiding data corruption on a raid set
325 with a reshape in progress.
326 1.9.0 Add support for RAID level takeover/reshape/region size
327 and set size reduction.
328 1.9.1 Fix activation of existing RAID 4/10 mapped devices
329 1.9.2 Don't emit '- -' on the status table line in case the constructor
330 fails reading a superblock. Correctly emit 'maj:min1 maj:min2' and
331 'D' on the status line. If '- -' is passed into the constructor, emit
332 '- -' on the table line and '-' as the status line health character.
333 1.10.0 Add support for raid4/5/6 journal device