| blob | ddf8947d54889ef99d200e03bd7e7ec9e631aeec |
1 ..
2 Copyright 2019 John Snow <jsnow@redhat.com> and Red Hat, Inc.
3 All rights reserved.
5 This file is licensed via The FreeBSD Documentation License, the full
6 text of which is included at the end of this document.
8 ====================================
9 Dirty Bitmaps and Incremental Backup
10 ====================================
12 Dirty Bitmaps are in-memory objects that track writes to block devices. They
13 can be used in conjunction with various block job operations to perform
14 incremental or differential backup regimens.
16 This document explains the conceptual mechanisms, as well as up-to-date,
17 complete and comprehensive documentation on the API to manipulate them.
18 (Hopefully, the "why", "what", and "how".)
20 The intended audience for this document is developers who are adding QEMU
21 backup features to management applications, or power users who run and
22 administer QEMU directly via QMP.
24 .. contents::
26 Overview
27 --------
29 Bitmaps are bit vectors where each '1' bit in the vector indicates a modified
30 ("dirty") segment of the corresponding block device. The size of the segment
31 that is tracked is the granularity of the bitmap. If the granularity of a
32 bitmap is 64K, each '1' bit means that a 64K region as a whole may have
33 changed in some way, possibly by as little as one byte.
35 Smaller granularities mean more accurate tracking of modified disk data, but
36 requires more computational overhead and larger bitmap sizes. Larger
37 granularities mean smaller bitmap sizes, but less targeted backups.
39 The size of a bitmap (in bytes) can be computed as such:
40 ``size`` = ceil(ceil(``image_size`` / ``granularity``) / 8)
42 e.g. the size of a 64KiB granularity bitmap on a 2TiB image is:
43 ``size`` = ((2147483648K / 64K) / 8)
44 = 4194304B = 4MiB.
46 QEMU uses these bitmaps when making incremental backups to know which sections
47 of the file to copy out. They are not enabled by default and must be
48 explicitly added in order to begin tracking writes.
50 Bitmaps can be created at any time and can be attached to any arbitrary block
51 node in the storage graph, but are most useful conceptually when attached to
52 the root node attached to the guest's storage device model.
54 That is to say: It's likely most useful to track the guest's writes to disk,
55 but you could theoretically track things like qcow2 metadata changes by
56 attaching the bitmap elsewhere in the storage graph. This is beyond the scope
57 of this document.
59 QEMU supports persisting these bitmaps to disk via the qcow2 image format.
60 Bitmaps which are stored or loaded in this way are called "persistent",
61 whereas bitmaps that are not are called "transient".
63 QEMU also supports the migration of both transient bitmaps (tracking any
64 arbitrary image format) or persistent bitmaps (qcow2) via live migration.
66 Supported Image Formats
67 -----------------------
69 QEMU supports all documented features below on the qcow2 image format.
71 However, qcow2 is only strictly necessary for the persistence feature, which
72 writes bitmap data to disk upon close. If persistence is not required for a
73 specific use case, all bitmap features excepting persistence are available for
74 any arbitrary image format.
76 For example, Dirty Bitmaps can be combined with the 'raw' image format, but
77 any changes to the bitmap will be discarded upon exit.
79 .. warning:: Transient bitmaps will not be saved on QEMU exit! Persistent
80 bitmaps are available only on qcow2 images.
82 Dirty Bitmap Names
83 ------------------
85 Bitmap objects need a method to reference them in the API. All API-created and
86 managed bitmaps have a human-readable name chosen by the user at creation
87 time.
89 - A bitmap's name is unique to the node, but bitmaps attached to different
90 nodes can share the same name. Therefore, all bitmaps are addressed via
91 their (node, name) pair.
93 - The name of a user-created bitmap cannot be empty ("").
95 - Transient bitmaps can have JSON unicode names that are effectively not
96 length limited. (QMP protocol may restrict messages to less than 64MiB.)
98 - Persistent storage formats may impose their own requirements on bitmap names
99 and namespaces. Presently, only qcow2 supports persistent bitmaps. See
100 docs/interop/qcow2.txt for more details on restrictions. Notably:
102 - qcow2 bitmap names are limited to between 1 and 1023 bytes long.
104 - No two bitmaps saved to the same qcow2 file may share the same name.
106 - QEMU occasionally uses bitmaps for internal use which have no name. They are
107 hidden from API query calls, cannot be manipulated by the external API, are
108 never persistent, nor ever migrated.
110 Bitmap Status
111 -------------
113 Dirty Bitmap objects can be queried with the QMP command `query-block
114 <qemu-qmp-ref.html#index-query_002dblock>`_, and are visible via the
115 `BlockDirtyInfo <qemu-qmp-ref.html#index-BlockDirtyInfo>`_ QAPI structure.
117 This struct shows the name, granularity, and dirty byte count for each bitmap.
118 Additionally, it shows several boolean status indicators:
120 - ``recording``: This bitmap is recording writes.
121 - ``busy``: This bitmap is in-use by an operation.
122 - ``persistent``: This bitmap is a persistent type.
123 - ``inconsistent``: This bitmap is corrupted and cannot be used.
125 The ``+busy`` status prohibits you from deleting, clearing, or otherwise
126 modifying a bitmap, and happens when the bitmap is being used for a backup
127 operation or is in the process of being loaded from a migration. Many of the
128 commands documented below will refuse to work on such bitmaps.
130 The ``+inconsistent`` status similarly prohibits almost all operations,
131 notably allowing only the ``block-dirty-bitmap-remove`` operation.
133 There is also a deprecated ``status`` field of type `DirtyBitmapStatus
134 <qemu-qmp-ref.html#index-DirtyBitmapStatus>`_. A bitmap historically had
135 five visible states:
137 #. ``Frozen``: This bitmap is currently in-use by an operation and is
138 immutable. It can't be deleted, renamed, reset, etc.
140 (This is now ``+busy``.)
142 #. ``Disabled``: This bitmap is not recording new writes.
144 (This is now ``-recording -busy``.)
146 #. ``Active``: This bitmap is recording new writes.
148 (This is now ``+recording -busy``.)
150 #. ``Locked``: This bitmap is in-use by an operation, and is immutable.
151 The difference from "Frozen" was primarily implementation details.
153 (This is now ``+busy``.)
155 #. ``Inconsistent``: This persistent bitmap was not saved to disk
156 correctly, and can no longer be used. It remains in memory to serve as
157 an indicator of failure.
159 (This is now ``+inconsistent``.)
161 These states are directly replaced by the status indicators and should not be
162 used. The difference between ``Frozen`` and ``Locked`` is an implementation
163 detail and should not be relevant to external users.
165 Basic QMP Usage
166 ---------------
168 The primary interface to manipulating bitmap objects is via the QMP
169 interface. If you are not familiar, see the :doc:`qmp-spec` for the
170 protocol, and :doc:`qemu-qmp-ref` for a full reference of all QMP
171 commands.
173 Supported Commands
174 ~~~~~~~~~~~~~~~~~~
176 There are six primary bitmap-management API commands:
178 - ``block-dirty-bitmap-add``
179 - ``block-dirty-bitmap-remove``
180 - ``block-dirty-bitmap-clear``
181 - ``block-dirty-bitmap-disable``
182 - ``block-dirty-bitmap-enable``
183 - ``block-dirty-bitmap-merge``
185 And one related query command:
187 - ``query-block``
189 Creation: block-dirty-bitmap-add
190 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
192 `block-dirty-bitmap-add
193 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dadd>`_:
195 Creates a new bitmap that tracks writes to the specified node. granularity,
196 persistence, and recording state can be adjusted at creation time.
198 .. admonition:: Example
200 to create a new, actively recording persistent bitmap:
202 .. code-block:: QMP
204 -> { "execute": "block-dirty-bitmap-add",
205 "arguments": {
206 "node": "drive0",
207 "name": "bitmap0",
208 "persistent": true,
209 }
210 }
212 <- { "return": {} }
214 - This bitmap will have a default granularity that matches the cluster size of
215 its associated drive, if available, clamped to between [4KiB, 64KiB]. The
216 current default for qcow2 is 64KiB.
218 .. admonition:: Example
220 To create a new, disabled (``-recording``), transient bitmap that tracks
221 changes in 32KiB segments:
223 .. code-block:: QMP
225 -> { "execute": "block-dirty-bitmap-add",
226 "arguments": {
227 "node": "drive0",
228 "name": "bitmap1",
229 "granularity": 32768,
230 "disabled": true
231 }
232 }
234 <- { "return": {} }
236 Deletion: block-dirty-bitmap-remove
237 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
239 `block-dirty-bitmap-remove
240 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dremove>`_:
242 Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed.
244 - Deleting a bitmap does not impact any other bitmaps attached to the same
245 node, nor does it affect any backups already created from this bitmap or
246 node.
248 - Because bitmaps are only unique to the node to which they are attached, you
249 must specify the node/drive name here, too.
251 - Deleting a persistent bitmap will remove it from the qcow2 file.
253 .. admonition:: Example
255 Remove a bitmap named ``bitmap0`` from node ``drive0``:
257 .. code-block:: QMP
259 -> { "execute": "block-dirty-bitmap-remove",
260 "arguments": {
261 "node": "drive0",
262 "name": "bitmap0"
263 }
264 }
266 <- { "return": {} }
268 Resetting: block-dirty-bitmap-clear
269 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
271 `block-dirty-bitmap-clear
272 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dclear>`_:
274 Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared.
276 - An incremental backup created from an empty bitmap will copy no data, as if
277 nothing has changed.
279 .. admonition:: Example
281 Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``:
283 .. code-block:: QMP
285 -> { "execute": "block-dirty-bitmap-clear",
286 "arguments": {
287 "node": "drive0",
288 "name": "bitmap0"
289 }
290 }
292 <- { "return": {} }
294 Enabling: block-dirty-bitmap-enable
295 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
297 `block-dirty-bitmap-enable
298 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002denable>`_:
300 "Enables" a bitmap, setting the ``recording`` bit to true, causing writes to
301 begin being recorded. ``+busy`` bitmaps cannot be enabled.
303 - Bitmaps default to being enabled when created, unless configured otherwise.
305 - Persistent enabled bitmaps will remember their ``+recording`` status on
306 load.
308 .. admonition:: Example
310 To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``:
312 .. code-block:: QMP
314 -> { "execute": "block-dirty-bitmap-enable",
315 "arguments": {
316 "node": "drive0",
317 "name": "bitmap0"
318 }
319 }
321 <- { "return": {} }
323 Enabling: block-dirty-bitmap-disable
324 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
326 `block-dirty-bitmap-disable
327 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002ddisable>`_:
329 "Disables" a bitmap, setting the ``recording`` bit to false, causing further
330 writes to begin being ignored. ``+busy`` bitmaps cannot be disabled.
332 .. warning::
334 This is potentially dangerous: QEMU makes no effort to stop any writes if
335 there are disabled bitmaps on a node, and will not mark any disabled bitmaps
336 as ``+inconsistent`` if any such writes do happen. Backups made from such
337 bitmaps will not be able to be used to reconstruct a coherent image.
339 - Disabling a bitmap may be useful for examining which sectors of a disk
340 changed during a specific time period, or for explicit management of
341 differential backup windows.
343 - Persistent disabled bitmaps will remember their ``-recording`` status on
344 load.
346 .. admonition:: Example
348 To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``:
350 .. code-block:: QMP
352 -> { "execute": "block-dirty-bitmap-disable",
353 "arguments": {
354 "node": "drive0",
355 "name": "bitmap0"
356 }
357 }
359 <- { "return": {} }
361 Merging, Copying: block-dirty-bitmap-merge
362 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
364 `block-dirty-bitmap-merge
365 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dmerge>`_:
367 Merges one or more bitmaps into a target bitmap. For any segment that is dirty
368 in any one source bitmap, the target bitmap will mark that segment dirty.
370 - Merge takes one or more bitmaps as a source and merges them together into a
371 single destination, such that any segment marked as dirty in any source
372 bitmap(s) will be marked dirty in the destination bitmap.
374 - Merge does not create the destination bitmap if it does not exist. A blank
375 bitmap can be created beforehand to achieve the same effect.
377 - The destination is not cleared prior to merge, so subsequent merge
378 operations will continue to cumulatively mark more segments as dirty.
380 - If the merge operation should fail, the destination bitmap is guaranteed to
381 be unmodified. The operation may fail if the source or destination bitmaps
382 are busy, or have different granularities.
384 - Bitmaps can only be merged on the same node. There is only one "node"
385 argument, so all bitmaps must be attached to that same node.
387 - Copy can be achieved by merging from a single source to an empty
388 destination.
390 .. admonition:: Example
392 Merge the data from ``bitmap0`` into the bitmap ``new_bitmap`` on node
393 ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves
394 a copy.
396 .. code-block:: QMP
398 -> { "execute": "block-dirty-bitmap-merge",
399 "arguments": {
400 "node": "drive0",
401 "target": "new_bitmap",
402 "bitmaps": [ "bitmap0" ]
403 }
404 }
406 <- { "return": {} }
408 Querying: query-block
409 ~~~~~~~~~~~~~~~~~~~~~
411 `query-block
412 <qemu-qmp-ref.html#index-query_002dblock>`_:
414 Not strictly a bitmaps command, but will return information about any bitmaps
415 attached to nodes serving as the root for guest devices.
417 - The "inconsistent" bit will not appear when it is false, appearing only when
418 the value is true to indicate there is a problem.
420 .. admonition:: Example
422 Query the block sub-system of QEMU. The following json has trimmed irrelevant
423 keys from the response to highlight only the bitmap-relevant portions of the
424 API. This result highlights a bitmap ``bitmap0`` attached to the root node of
425 device ``drive0``.
427 .. code-block:: QMP
429 -> {
430 "execute": "query-block",
431 "arguments": {}
432 }
434 <- {
435 "return": [ {
436 "dirty-bitmaps": [ {
437 "status": "active",
438 "count": 0,
439 "busy": false,
440 "name": "bitmap0",
441 "persistent": false,
442 "recording": true,
443 "granularity": 65536
444 } ],
445 "device": "drive0",
446 } ]
447 }
449 Bitmap Persistence
450 ------------------
452 As outlined in `Supported Image Formats`_, QEMU can persist bitmaps to qcow2
453 files. Demonstrated in `Creation: block-dirty-bitmap-add`_, passing
454 ``persistent: true`` to ``block-dirty-bitmap-add`` will persist that bitmap to
455 disk.
457 Persistent bitmaps will be automatically loaded into memory upon load, and
458 will be written back to disk upon close. Their usage should be mostly
459 transparent.
461 However, if QEMU does not get a chance to close the file cleanly, the bitmap
462 will be marked as ``+inconsistent`` at next load and considered unsafe to use
463 for any operation. At this point, the only valid operation on such bitmaps is
464 ``block-dirty-bitmap-remove``.
466 Losing a bitmap in this way does not invalidate any existing backups that have
467 been made from this bitmap, but no further backups will be able to be issued
468 for this chain.
470 Transactions
471 ------------
473 Transactions are a QMP feature that allows you to submit multiple QMP commands
474 at once, being guaranteed that they will all succeed or fail atomically,
475 together. The interaction of bitmaps and transactions are demonstrated below.
477 See `transaction <qemu-qmp.ref.html#index-transaction>`_ in the QMP reference
478 for more details.
480 Justification
481 ~~~~~~~~~~~~~
483 Bitmaps can generally be modified at any time, but certain operations often
484 only make sense when paired directly with other commands. When a VM is paused,
485 it's easy to ensure that no guest writes occur between individual QMP
486 commands. When a VM is running, this is difficult to accomplish with
487 individual QMP commands that may allow guest writes to occur between each
488 command.
490 For example, using only individual QMP commands, we could:
492 #. Boot the VM in a paused state.
493 #. Create a full drive backup of drive0.
494 #. Create a new bitmap attached to drive0, confident that nothing has been
495 written to drive0 in the meantime.
496 #. Resume execution of the VM.
497 #. At a later point, issue incremental backups from ``bitmap0``.
499 At this point, the bitmap and drive backup would be correctly in sync, and
500 incremental backups made from this point forward would be correctly aligned to
501 the full drive backup.
503 This is not particularly useful if we decide we want to start incremental
504 backups after the VM has been running for a while, for which we would want to
505 perform actions such as the following:
507 #. Boot the VM and begin execution.
508 #. Using a single transaction, perform the following operations:
510 - Create ``bitmap0``.
511 - Create a full drive backup of ``drive0``.
513 #. At a later point, issue incremental backups from ``bitmap0``.
515 .. note:: As a consideration, if ``bitmap0`` is created prior to the full
516 drive backup, incremental backups can still be authored from this
517 bitmap, but they will copy extra segments reflecting writes that
518 occurred prior to the backup operation. Transactions allow us to
519 narrow critical points in time to reduce waste, or, in the other
520 direction, to ensure that no segments are omitted.
522 Supported Bitmap Transactions
523 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
525 - ``block-dirty-bitmap-add``
526 - ``block-dirty-bitmap-clear``
527 - ``block-dirty-bitmap-enable``
528 - ``block-dirty-bitmap-disable``
529 - ``block-dirty-bitmap-merge``
531 The usages for these commands are identical to their respective QMP commands,
532 but see the sections below for concrete examples.
534 Incremental Backups - Push Model
535 --------------------------------
537 Incremental backups are simply partial disk images that can be combined with
538 other partial disk images on top of a base image to reconstruct a full backup
539 from the point in time at which the incremental backup was issued.
541 The "Push Model" here references the fact that QEMU is "pushing" the modified
542 blocks out to a destination. We will be using the `blockdev-backup
543 <qemu-qmp-ref.html#index-blockdev_002dbackup>`_ QMP command to create both
544 full and incremental backups.
546 The command is a background job, which has its own QMP API for querying and
547 management documented in `Background jobs
548 <qemu-qmp-ref.html#Background-jobs>`_.
550 Example: New Incremental Backup Anchor Point
551 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
553 As outlined in the Transactions - `Justification`_ section, perhaps we want to
554 create a new incremental backup chain attached to a drive.
556 This example creates a new, full backup of "drive0" and accompanies it with a
557 new, empty bitmap that records writes from this point in time forward.
559 The target can be created with the help of `blockdev-add
560 <qemu-qmp-ref.html#index-blockdev_002dadd>`_ or `blockdev-create
561 <qemu-qmp-ref.html#index-blockdev_002dcreate>`_ command.
563 .. note:: Any new writes that happen after this command is issued, even while
564 the backup job runs, will be written locally and not to the backup
565 destination. These writes will be recorded in the bitmap
566 accordingly.
568 .. code-block:: QMP
570 -> {
571 "execute": "transaction",
572 "arguments": {
573 "actions": [
574 {
575 "type": "block-dirty-bitmap-add",
576 "data": {
577 "node": "drive0",
578 "name": "bitmap0"
579 }
580 },
581 {
582 "type": "blockdev-backup",
583 "data": {
584 "device": "drive0",
585 "target": "target0",
586 "sync": "full"
587 }
588 }
589 ]
590 }
591 }
593 <- { "return": {} }
595 <- {
596 "timestamp": {
597 "seconds": 1555436945,
598 "microseconds": 179620
599 },
600 "data": {
601 "status": "created",
602 "id": "drive0"
603 },
604 "event": "JOB_STATUS_CHANGE"
605 }
607 ...
609 <- {
610 "timestamp": {...},
611 "data": {
612 "device": "drive0",
613 "type": "backup",
614 "speed": 0,
615 "len": 68719476736,
616 "offset": 68719476736
617 },
618 "event": "BLOCK_JOB_COMPLETED"
619 }
621 <- {
622 "timestamp": {...},
623 "data": {
624 "status": "concluded",
625 "id": "drive0"
626 },
627 "event": "JOB_STATUS_CHANGE"
628 }
630 <- {
631 "timestamp": {...},
632 "data": {
633 "status": "null",
634 "id": "drive0"
635 },
636 "event": "JOB_STATUS_CHANGE"
637 }
639 A full explanation of the job transition semantics and the JOB_STATUS_CHANGE
640 event are beyond the scope of this document and will be omitted in all
641 subsequent examples; above, several more events have been omitted for brevity.
643 .. note:: Subsequent examples will omit all events except BLOCK_JOB_COMPLETED
644 except where necessary to illustrate workflow differences.
646 Omitted events and json objects will be represented by ellipses:
647 ``...``
649 Example: Resetting an Incremental Backup Anchor Point
650 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
652 If we want to start a new backup chain with an existing bitmap, we can also
653 use a transaction to reset the bitmap while making a new full backup:
655 .. code-block:: QMP
657 -> {
658 "execute": "transaction",
659 "arguments": {
660 "actions": [
661 {
662 "type": "block-dirty-bitmap-clear",
663 "data": {
664 "node": "drive0",
665 "name": "bitmap0"
666 }
667 },
668 {
669 "type": "blockdev-backup",
670 "data": {
671 "device": "drive0",
672 "target": "target0",
673 "sync": "full"
674 }
675 }
676 ]
677 }
678 }
680 <- { "return": {} }
682 ...
684 <- {
685 "timestamp": {...},
686 "data": {
687 "device": "drive0",
688 "type": "backup",
689 "speed": 0,
690 "len": 68719476736,
691 "offset": 68719476736
692 },
693 "event": "BLOCK_JOB_COMPLETED"
694 }
696 ...
698 The result of this example is identical to the first, but we clear an existing
699 bitmap instead of adding a new one.
701 .. tip:: In both of these examples, "bitmap0" is tied conceptually to the
702 creation of new, full backups. This relationship is not saved or
703 remembered by QEMU; it is up to the operator or management layer to
704 remember which bitmaps are associated with which backups.
706 Example: First Incremental Backup
707 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
709 #. Create a full backup and sync it to a dirty bitmap using any method:
711 - Either of the two live backup method demonstrated above,
712 - Using QMP commands with the VM paused as in the `Justification`_ section,
713 or
714 - With the VM offline, manually copy the image and start the VM in a paused
715 state, careful to add a new bitmap before the VM begins execution.
717 Whichever method is chosen, let's assume that at the end of this step:
719 - The full backup is named ``drive0.full.qcow2``.
720 - The bitmap we created is named ``bitmap0``, attached to ``drive0``.
722 #. Create a destination image for the incremental backup that utilizes the
723 full backup as a backing image.
725 - Let's assume the new incremental image is named ``drive0.inc0.qcow2``:
727 .. code:: bash
729 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
730 -b drive0.full.qcow2 -F qcow2
732 #. Add target block node:
734 .. code-block:: QMP
736 -> {
737 "execute": "blockdev-add",
738 "arguments": {
739 "node-name": "target0",
740 "driver": "qcow2",
741 "file": {
742 "driver": "file",
743 "filename": "drive0.inc0.qcow2"
744 }
745 }
746 }
748 <- { "return": {} }
750 #. Issue an incremental backup command:
752 .. code-block:: QMP
754 -> {
755 "execute": "blockdev-backup",
756 "arguments": {
757 "device": "drive0",
758 "bitmap": "bitmap0",
759 "target": "target0",
760 "sync": "incremental"
761 }
762 }
764 <- { "return": {} }
766 ...
768 <- {
769 "timestamp": {...},
770 "data": {
771 "device": "drive0",
772 "type": "backup",
773 "speed": 0,
774 "len": 68719476736,
775 "offset": 68719476736
776 },
777 "event": "BLOCK_JOB_COMPLETED"
778 }
780 ...
782 This copies any blocks modified since the full backup was created into the
783 ``drive0.inc0.qcow2`` file. During the operation, ``bitmap0`` is marked
784 ``+busy``. If the operation is successful, ``bitmap0`` will be cleared to
785 reflect the "incremental" backup regimen, which only copies out new changes
786 from each incremental backup.
788 .. note:: Any new writes that occur after the backup operation starts do not
789 get copied to the destination. The backup's "point in time" is when
790 the backup starts, not when it ends. These writes are recorded in a
791 special bitmap that gets re-added to bitmap0 when the backup ends so
792 that the next incremental backup can copy them out.
794 Example: Second Incremental Backup
795 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
797 #. Create a new destination image for the incremental backup that points to
798 the previous one, e.g.: ``drive0.inc1.qcow2``
800 .. code:: bash
802 $ qemu-img create -f qcow2 drive0.inc1.qcow2 \
803 -b drive0.inc0.qcow2 -F qcow2
805 #. Add target block node:
807 .. code-block:: QMP
809 -> {
810 "execute": "blockdev-add",
811 "arguments": {
812 "node-name": "target0",
813 "driver": "qcow2",
814 "file": {
815 "driver": "file",
816 "filename": "drive0.inc1.qcow2"
817 }
818 }
819 }
821 <- { "return": {} }
823 #. Issue a new incremental backup command. The only difference here is that we
824 have changed the target image below.
826 .. code-block:: QMP
828 -> {
829 "execute": "blockdev-backup",
830 "arguments": {
831 "device": "drive0",
832 "bitmap": "bitmap0",
833 "target": "target0",
834 "sync": "incremental"
835 }
836 }
838 <- { "return": {} }
840 ...
842 <- {
843 "timestamp": {...},
844 "data": {
845 "device": "drive0",
846 "type": "backup",
847 "speed": 0,
848 "len": 68719476736,
849 "offset": 68719476736
850 },
851 "event": "BLOCK_JOB_COMPLETED"
852 }
854 ...
856 Because the first incremental backup from the previous example completed
857 successfully, ``bitmap0`` was synchronized with ``drive0.inc0.qcow2``. Here,
858 we use ``bitmap0`` again to create a new incremental backup that targets the
859 previous one, creating a chain of three images:
861 .. admonition:: Diagram
863 .. code:: text
865 +-------------------+ +-------------------+ +-------------------+
866 | drive0.full.qcow2 |<--| drive0.inc0.qcow2 |<--| drive0.inc1.qcow2 |
867 +-------------------+ +-------------------+ +-------------------+
869 Each new incremental backup re-synchronizes the bitmap to the latest backup
870 authored, allowing a user to continue to "consume" it to create new backups on
871 top of an existing chain.
873 In the above diagram, neither drive0.inc1.qcow2 nor drive0.inc0.qcow2 are
874 complete images by themselves, but rely on their backing chain to reconstruct
875 a full image. The dependency terminates with each full backup.
877 Each backup in this chain remains independent, and is unchanged by new entries
878 made later in the chain. For instance, drive0.inc0.qcow2 remains a perfectly
879 valid backup of the disk as it was when that backup was issued.
881 Example: Incremental Push Backups without Backing Files
882 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
884 Backup images are best kept off-site, so we often will not have the preceding
885 backups in a chain available to link against. This is not a problem at backup
886 time; we simply do not set the backing image when creating the destination
887 image:
889 #. Create a new destination image with no backing file set. We will need to
890 specify the size of the base image, because the backing file isn't
891 available for QEMU to use to determine it.
893 .. code:: bash
895 $ qemu-img create -f qcow2 drive0.inc2.qcow2 64G
897 .. note:: Alternatively, you can omit ``mode: "existing"`` from the push
898 backup commands to have QEMU create an image without a backing
899 file for you, but you lose control over format options like
900 compatibility and preallocation presets.
902 #. Add target block node:
904 .. code-block:: QMP
906 -> {
907 "execute": "blockdev-add",
908 "arguments": {
909 "node-name": "target0",
910 "driver": "qcow2",
911 "file": {
912 "driver": "file",
913 "filename": "drive0.inc2.qcow2"
914 }
915 }
916 }
918 <- { "return": {} }
920 #. Issue a new incremental backup command. Apart from the new destination
921 image, there is no difference from the last two examples.
923 .. code-block:: QMP
925 -> {
926 "execute": "blockdev-backup",
927 "arguments": {
928 "device": "drive0",
929 "bitmap": "bitmap0",
930 "target": "target0",
931 "sync": "incremental"
932 }
933 }
935 <- { "return": {} }
937 ...
939 <- {
940 "timestamp": {...},
941 "data": {
942 "device": "drive0",
943 "type": "backup",
944 "speed": 0,
945 "len": 68719476736,
946 "offset": 68719476736
947 },
948 "event": "BLOCK_JOB_COMPLETED"
949 }
951 ...
953 The only difference from the perspective of the user is that you will need to
954 set the backing image when attempting to restore the backup:
956 .. code:: bash
958 $ qemu-img rebase drive0.inc2.qcow2 \
959 -u -b drive0.inc1.qcow2
961 This uses the "unsafe" rebase mode to simply set the backing file to a file
962 that isn't present.
964 It is also possible to use ``--image-opts`` to specify the entire backing
965 chain by hand as an ephemeral property at runtime, but that is beyond the
966 scope of this document.
968 Example: Multi-drive Incremental Backup
969 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
971 Assume we have a VM with two drives, "drive0" and "drive1" and we wish to back
972 both of them up such that the two backups represent the same crash-consistent
973 point in time.
975 #. For each drive, create an empty image:
977 .. code:: bash
979 $ qemu-img create -f qcow2 drive0.full.qcow2 64G
980 $ qemu-img create -f qcow2 drive1.full.qcow2 64G
982 #. Add target block nodes:
984 .. code-block:: QMP
986 -> {
987 "execute": "blockdev-add",
988 "arguments": {
989 "node-name": "target0",
990 "driver": "qcow2",
991 "file": {
992 "driver": "file",
993 "filename": "drive0.full.qcow2"
994 }
995 }
996 }
998 <- { "return": {} }
1000 -> {
1001 "execute": "blockdev-add",
1002 "arguments": {
1003 "node-name": "target1",
1004 "driver": "qcow2",
1005 "file": {
1006 "driver": "file",
1007 "filename": "drive1.full.qcow2"
1008 }
1009 }
1010 }
1012 <- { "return": {} }
1014 #. Create a full (anchor) backup for each drive, with accompanying bitmaps:
1016 .. code-block:: QMP
1018 -> {
1019 "execute": "transaction",
1020 "arguments": {
1021 "actions": [
1022 {
1023 "type": "block-dirty-bitmap-add",
1024 "data": {
1025 "node": "drive0",
1026 "name": "bitmap0"
1027 }
1028 },
1029 {
1030 "type": "block-dirty-bitmap-add",
1031 "data": {
1032 "node": "drive1",
1033 "name": "bitmap0"
1034 }
1035 },
1036 {
1037 "type": "blockdev-backup",
1038 "data": {
1039 "device": "drive0",
1040 "target": "target0",
1041 "sync": "full"
1042 }
1043 },
1044 {
1045 "type": "blockdev-backup",
1046 "data": {
1047 "device": "drive1",
1048 "target": "target1",
1049 "sync": "full"
1050 }
1051 }
1052 ]
1053 }
1054 }
1056 <- { "return": {} }
1058 ...
1060 <- {
1061 "timestamp": {...},
1062 "data": {
1063 "device": "drive0",
1064 "type": "backup",
1065 "speed": 0,
1066 "len": 68719476736,
1067 "offset": 68719476736
1068 },
1069 "event": "BLOCK_JOB_COMPLETED"
1070 }
1072 ...
1074 <- {
1075 "timestamp": {...},
1076 "data": {
1077 "device": "drive1",
1078 "type": "backup",
1079 "speed": 0,
1080 "len": 68719476736,
1081 "offset": 68719476736
1082 },
1083 "event": "BLOCK_JOB_COMPLETED"
1084 }
1086 ...
1088 #. Later, create new destination images for each of the incremental backups
1089 that point to their respective full backups:
1091 .. code:: bash
1093 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
1094 -b drive0.full.qcow2 -F qcow2
1095 $ qemu-img create -f qcow2 drive1.inc0.qcow2 \
1096 -b drive1.full.qcow2 -F qcow2
1098 #. Add target block nodes:
1100 .. code-block:: QMP
1102 -> {
1103 "execute": "blockdev-add",
1104 "arguments": {
1105 "node-name": "target0",
1106 "driver": "qcow2",
1107 "file": {
1108 "driver": "file",
1109 "filename": "drive0.inc0.qcow2"
1110 }
1111 }
1112 }
1114 <- { "return": {} }
1116 -> {
1117 "execute": "blockdev-add",
1118 "arguments": {
1119 "node-name": "target1",
1120 "driver": "qcow2",
1121 "file": {
1122 "driver": "file",
1123 "filename": "drive1.inc0.qcow2"
1124 }
1125 }
1126 }
1128 <- { "return": {} }
1130 #. Issue a multi-drive incremental push backup transaction:
1132 .. code-block:: QMP
1134 -> {
1135 "execute": "transaction",
1136 "arguments": {
1137 "actions": [
1138 {
1139 "type": "blockev-backup",
1140 "data": {
1141 "device": "drive0",
1142 "bitmap": "bitmap0",
1143 "sync": "incremental",
1144 "target": "target0"
1145 }
1146 },
1147 {
1148 "type": "blockdev-backup",
1149 "data": {
1150 "device": "drive1",
1151 "bitmap": "bitmap0",
1152 "sync": "incremental",
1153 "target": "target1"
1154 }
1155 },
1156 ]
1157 }
1158 }
1160 <- { "return": {} }
1162 ...
1164 <- {
1165 "timestamp": {...},
1166 "data": {
1167 "device": "drive0",
1168 "type": "backup",
1169 "speed": 0,
1170 "len": 68719476736,
1171 "offset": 68719476736
1172 },
1173 "event": "BLOCK_JOB_COMPLETED"
1174 }
1176 ...
1178 <- {
1179 "timestamp": {...},
1180 "data": {
1181 "device": "drive1",
1182 "type": "backup",
1183 "speed": 0,
1184 "len": 68719476736,
1185 "offset": 68719476736
1186 },
1187 "event": "BLOCK_JOB_COMPLETED"
1188 }
1190 ...
1192 Push Backup Errors & Recovery
1193 -----------------------------
1195 In the event of an error that occurs after a push backup job is successfully
1196 launched, either by an individual QMP command or a QMP transaction, the user
1197 will receive a ``BLOCK_JOB_COMPLETE`` event with a failure message,
1198 accompanied by a ``BLOCK_JOB_ERROR`` event.
1200 In the case of a job being cancelled, the user will receive a
1201 ``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR
1202 events.
1204 In either failure case, the bitmap used for the failed operation is not
1205 cleared. It will contain all of the dirty bits it did at the start of the
1206 operation, plus any new bits that got marked during the operation.
1208 Effectively, the "point in time" that a bitmap is recording differences
1209 against is kept at the issuance of the last successful incremental backup,
1210 instead of being moved forward to the start of this now-failed backup.
1212 Once the underlying problem is addressed (e.g. more storage space is allocated
1213 on the destination), the incremental backup command can be retried with the
1214 same bitmap.
1216 Example: Individual Failures
1217 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1219 Incremental Push Backup jobs that fail individually behave simply as
1220 described above. This example demonstrates the single-job failure case:
1222 #. Create a target image:
1224 .. code:: bash
1226 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
1227 -b drive0.full.qcow2 -F qcow2
1229 #. Add target block node:
1231 .. code-block:: QMP
1233 -> {
1234 "execute": "blockdev-add",
1235 "arguments": {
1236 "node-name": "target0",
1237 "driver": "qcow2",
1238 "file": {
1239 "driver": "file",
1240 "filename": "drive0.inc0.qcow2"
1241 }
1242 }
1243 }
1245 <- { "return": {} }
1247 #. Attempt to create an incremental backup via QMP:
1249 .. code-block:: QMP
1251 -> {
1252 "execute": "blockdev-backup",
1253 "arguments": {
1254 "device": "drive0",
1255 "bitmap": "bitmap0",
1256 "target": "target0",
1257 "sync": "incremental"
1258 }
1259 }
1261 <- { "return": {} }
1263 #. Receive a pair of events indicating failure:
1265 .. code-block:: QMP
1267 <- {
1268 "timestamp": {...},
1269 "data": {
1270 "device": "drive0",
1271 "action": "report",
1272 "operation": "write"
1273 },
1274 "event": "BLOCK_JOB_ERROR"
1275 }
1277 <- {
1278 "timestamp": {...},
1279 "data": {
1280 "speed": 0,
1281 "offset": 0,
1282 "len": 67108864,
1283 "error": "No space left on device",
1284 "device": "drive0",
1285 "type": "backup"
1286 },
1287 "event": "BLOCK_JOB_COMPLETED"
1288 }
1290 #. Remove target node:
1292 .. code-block:: QMP
1294 -> {
1295 "execute": "blockdev-del",
1296 "arguments": {
1297 "node-name": "target0",
1298 }
1299 }
1301 <- { "return": {} }
1303 #. Delete the failed image, and re-create it.
1305 .. code:: bash
1307 $ rm drive0.inc0.qcow2
1308 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
1309 -b drive0.full.qcow2 -F qcow2
1311 #. Add target block node:
1313 .. code-block:: QMP
1315 -> {
1316 "execute": "blockdev-add",
1317 "arguments": {
1318 "node-name": "target0",
1319 "driver": "qcow2",
1320 "file": {
1321 "driver": "file",
1322 "filename": "drive0.inc0.qcow2"
1323 }
1324 }
1325 }
1327 <- { "return": {} }
1329 #. Retry the command after fixing the underlying problem, such as
1330 freeing up space on the backup volume:
1332 .. code-block:: QMP
1334 -> {
1335 "execute": "blockdev-backup",
1336 "arguments": {
1337 "device": "drive0",
1338 "bitmap": "bitmap0",
1339 "target": "target0",
1340 "sync": "incremental"
1341 }
1342 }
1344 <- { "return": {} }
1346 #. Receive confirmation that the job completed successfully:
1348 .. code-block:: QMP
1350 <- {
1351 "timestamp": {...},
1352 "data": {
1353 "device": "drive0",
1354 "type": "backup",
1355 "speed": 0,
1356 "len": 67108864,
1357 "offset": 67108864
1358 },
1359 "event": "BLOCK_JOB_COMPLETED"
1360 }
1362 Example: Partial Transactional Failures
1363 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1365 QMP commands like `blockdev-backup
1366 <qemu-qmp-ref.html#index-blockdev_002dbackup>`_
1367 conceptually only start a job, and so transactions containing these commands
1368 may succeed even if the job it created later fails. This might have surprising
1369 interactions with notions of how a "transaction" ought to behave.
1371 This distinction means that on occasion, a transaction containing such job
1372 launching commands may appear to succeed and return success, but later
1373 individual jobs associated with the transaction may fail. It is possible that
1374 a management application may have to deal with a partial backup failure after
1375 a "successful" transaction.
1377 If multiple backup jobs are specified in a single transaction, if one of those
1378 jobs fails, it will not interact with the other backup jobs in any way by
1379 default. The job(s) that succeeded will clear the dirty bitmap associated with
1380 the operation, but the job(s) that failed will not. It is therefore not safe
1381 to delete any incremental backups that were created successfully in this
1382 scenario, even though others failed.
1384 This example illustrates a transaction with two backup jobs, where one fails
1385 and one succeeds:
1387 #. Issue the transaction to start a backup of both drives.
1389 .. code-block:: QMP
1391 -> {
1392 "execute": "transaction",
1393 "arguments": {
1394 "actions": [
1395 {
1396 "type": "blockdev-backup",
1397 "data": {
1398 "device": "drive0",
1399 "bitmap": "bitmap0",
1400 "sync": "incremental",
1401 "target": "target0"
1402 }
1403 },
1404 {
1405 "type": "blockdev-backup",
1406 "data": {
1407 "device": "drive1",
1408 "bitmap": "bitmap0",
1409 "sync": "incremental",
1410 "target": "target1"
1411 }
1412 }]
1413 }
1414 }
1416 #. Receive notice that the Transaction was accepted, and jobs were
1417 launched:
1419 .. code-block:: QMP
1421 <- { "return": {} }
1423 #. Receive notice that the first job has completed:
1425 .. code-block:: QMP
1427 <- {
1428 "timestamp": {...},
1429 "data": {
1430 "device": "drive0",
1431 "type": "backup",
1432 "speed": 0,
1433 "len": 67108864,
1434 "offset": 67108864
1435 },
1436 "event": "BLOCK_JOB_COMPLETED"
1437 }
1439 #. Receive notice that the second job has failed:
1441 .. code-block:: QMP
1443 <- {
1444 "timestamp": {...},
1445 "data": {
1446 "device": "drive1",
1447 "action": "report",
1448 "operation": "read"
1449 },
1450 "event": "BLOCK_JOB_ERROR"
1451 }
1453 ...
1455 <- {
1456 "timestamp": {...},
1457 "data": {
1458 "speed": 0,
1459 "offset": 0,
1460 "len": 67108864,
1461 "error": "Input/output error",
1462 "device": "drive1",
1463 "type": "backup"
1464 },
1465 "event": "BLOCK_JOB_COMPLETED"
1466 }
1468 At the conclusion of the above example, ``drive0.inc0.qcow2`` is valid and
1469 must be kept, but ``drive1.inc0.qcow2`` is incomplete and should be
1470 deleted. If a VM-wide incremental backup of all drives at a point-in-time is
1471 to be made, new backups for both drives will need to be made, taking into
1472 account that a new incremental backup for drive0 needs to be based on top of
1473 ``drive0.inc0.qcow2``.
1475 For this example, an incremental backup for ``drive0`` was created, but not
1476 for ``drive1``. The last VM-wide crash-consistent backup that is available in
1477 this case is the full backup:
1479 .. code:: text
1481 [drive0.full.qcow2] <-- [drive0.inc0.qcow2]
1482 [drive1.full.qcow2]
1484 To repair this, issue a new incremental backup across both drives. The result
1485 will be backup chains that resemble the following:
1487 .. code:: text
1489 [drive0.full.qcow2] <-- [drive0.inc0.qcow2] <-- [drive0.inc1.qcow2]
1490 [drive1.full.qcow2] <-------------------------- [drive1.inc1.qcow2]
1492 Example: Grouped Completion Mode
1493 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1495 While jobs launched by transactions normally complete or fail individually,
1496 it's possible to instruct them to complete or fail together as a group. QMP
1497 transactions take an optional properties structure that can affect the
1498 behavior of the transaction.
1500 The ``completion-mode`` transaction property can be either ``individual``
1501 which is the default legacy behavior described above, or ``grouped``, detailed
1502 below.
1504 In ``grouped`` completion mode, no jobs will report success until all jobs are
1505 ready to report success. If any job fails, all other jobs will be cancelled.
1507 Regardless of if a participating incremental backup job failed or was
1508 cancelled, their associated bitmaps will all be held at their existing
1509 points-in-time, as in individual failure cases.
1511 Here's the same multi-drive backup scenario from `Example: Partial
1512 Transactional Failures`_, but with the ``grouped`` completion-mode property
1513 applied:
1515 #. Issue the multi-drive incremental backup transaction:
1517 .. code-block:: QMP
1519 -> {
1520 "execute": "transaction",
1521 "arguments": {
1522 "properties": {
1523 "completion-mode": "grouped"
1524 },
1525 "actions": [
1526 {
1527 "type": "blockdev-backup",
1528 "data": {
1529 "device": "drive0",
1530 "bitmap": "bitmap0",
1531 "sync": "incremental",
1532 "target": "target0"
1533 }
1534 },
1535 {
1536 "type": "blockdev-backup",
1537 "data": {
1538 "device": "drive1",
1539 "bitmap": "bitmap0",
1540 "sync": "incremental",
1541 "target": "target1"
1542 }
1543 }]
1544 }
1545 }
1547 #. Receive notice that the Transaction was accepted, and jobs were launched:
1549 .. code-block:: QMP
1551 <- { "return": {} }
1553 #. Receive notification that the backup job for ``drive1`` has failed:
1555 .. code-block:: QMP
1557 <- {
1558 "timestamp": {...},
1559 "data": {
1560 "device": "drive1",
1561 "action": "report",
1562 "operation": "read"
1563 },
1564 "event": "BLOCK_JOB_ERROR"
1565 }
1567 <- {
1568 "timestamp": {...},
1569 "data": {
1570 "speed": 0,
1571 "offset": 0,
1572 "len": 67108864,
1573 "error": "Input/output error",
1574 "device": "drive1",
1575 "type": "backup"
1576 },
1577 "event": "BLOCK_JOB_COMPLETED"
1578 }
1580 #. Receive notification that the job for ``drive0`` has been cancelled:
1582 .. code-block:: QMP
1584 <- {
1585 "timestamp": {...},
1586 "data": {
1587 "device": "drive0",
1588 "type": "backup",
1589 "speed": 0,
1590 "len": 67108864,
1591 "offset": 16777216
1592 },
1593 "event": "BLOCK_JOB_CANCELLED"
1594 }
1596 At the conclusion of *this* example, both jobs have been aborted due to a
1597 failure. Both destination images should be deleted and are no longer of use.
1599 The transaction as a whole can simply be re-issued at a later time.
1601 .. raw:: html
1603 <!--
1604 The FreeBSD Documentation License
1606 Redistribution and use in source (ReST) and 'compiled' forms (SGML, HTML,
1607 PDF, PostScript, RTF and so forth) with or without modification, are
1608 permitted provided that the following conditions are met:
1610 Redistributions of source code (ReST) must retain the above copyright notice,
1611 this list of conditions and the following disclaimer of this file unmodified.
1613 Redistributions in compiled form (transformed to other DTDs, converted to
1614 PDF, PostScript, RTF and other formats) must reproduce the above copyright
1615 notice, this list of conditions and the following disclaimer in the
1616 documentation and/or other materials provided with the distribution.
1618 THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
1619 IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
1620 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
1621 ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
1622 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
1623 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
1624 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
1625 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
1626 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
1627 ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF
1628 THE POSSIBILITY OF SUCH DAMAGE.
1629 -->