| blob | 0455a78486d5ecf8a479e6e5dd61e7f7ca3b85fa |
1 ======================
2 Firmware-Assisted Dump
3 ======================
5 July 2011
7 The goal of firmware-assisted dump is to enable the dump of
8 a crashed system, and to do so from a fully-reset system, and
9 to minimize the total elapsed time until the system is back
10 in production use.
12 - Firmware-Assisted Dump (FADump) infrastructure is intended to replace
13 the existing phyp assisted dump.
14 - Fadump uses the same firmware interfaces and memory reservation model
15 as phyp assisted dump.
16 - Unlike phyp dump, FADump exports the memory dump through /proc/vmcore
17 in the ELF format in the same way as kdump. This helps us reuse the
18 kdump infrastructure for dump capture and filtering.
19 - Unlike phyp dump, userspace tool does not need to refer any sysfs
20 interface while reading /proc/vmcore.
21 - Unlike phyp dump, FADump allows user to release all the memory reserved
22 for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem.
23 - Once enabled through kernel boot parameter, FADump can be
24 started/stopped through /sys/kernel/fadump_registered interface (see
25 sysfs files section below) and can be easily integrated with kdump
26 service start/stop init scripts.
28 Comparing with kdump or other strategies, firmware-assisted
29 dump offers several strong, practical advantages:
31 - Unlike kdump, the system has been reset, and loaded
32 with a fresh copy of the kernel. In particular,
33 PCI and I/O devices have been reinitialized and are
34 in a clean, consistent state.
35 - Once the dump is copied out, the memory that held the dump
36 is immediately available to the running kernel. And therefore,
37 unlike kdump, FADump doesn't need a 2nd reboot to get back
38 the system to the production configuration.
40 The above can only be accomplished by coordination with,
41 and assistance from the Power firmware. The procedure is
42 as follows:
44 - The first kernel registers the sections of memory with the
45 Power firmware for dump preservation during OS initialization.
46 These registered sections of memory are reserved by the first
47 kernel during early boot.
49 - When system crashes, the Power firmware will copy the registered
50 low memory regions (boot memory) from source to destination area.
51 It will also save hardware PTE's.
53 NOTE:
54 The term 'boot memory' means size of the low memory chunk
55 that is required for a kernel to boot successfully when
56 booted with restricted memory. By default, the boot memory
57 size will be the larger of 5% of system RAM or 256MB.
58 Alternatively, user can also specify boot memory size
59 through boot parameter 'crashkernel=' which will override
60 the default calculated size. Use this option if default
61 boot memory size is not sufficient for second kernel to
62 boot successfully. For syntax of crashkernel= parameter,
63 refer to Documentation/admin-guide/kdump/kdump.rst. If any
64 offset is provided in crashkernel= parameter, it will be
65 ignored as FADump uses a predefined offset to reserve memory
66 for boot memory dump preservation in case of a crash.
68 - After the low memory (boot memory) area has been saved, the
69 firmware will reset PCI and other hardware state. It will
70 *not* clear the RAM. It will then launch the bootloader, as
71 normal.
73 - The freshly booted kernel will notice that there is a new node
74 (rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot
75 on OPAL platform) in the device tree, indicating that
76 there is crash data available from a previous boot. During
77 the early boot OS will reserve rest of the memory above
78 boot memory size effectively booting with restricted memory
79 size. This will make sure that this kernel (also, referred
80 to as second kernel or capture kernel) will not touch any
81 of the dump memory area.
83 - User-space tools will read /proc/vmcore to obtain the contents
84 of memory, which holds the previous crashed kernel dump in ELF
85 format. The userspace tools may copy this info to disk, or
86 network, nas, san, iscsi, etc. as desired.
88 - Once the userspace tool is done saving dump, it will echo
89 '1' to /sys/kernel/fadump_release_mem to release the reserved
90 memory back to general use, except the memory required for
91 next firmware-assisted dump registration.
93 e.g.::
95 # echo 1 > /sys/kernel/fadump_release_mem
97 Please note that the firmware-assisted dump feature
98 is only available on POWER6 and above systems on pSeries
99 (PowerVM) platform and POWER9 and above systems with OP940
100 or later firmware versions on PowerNV (OPAL) platform.
101 Note that, OPAL firmware exports ibm,opal/dump node when
102 FADump is supported on PowerNV platform.
104 On OPAL based machines, system first boots into an intermittent
105 kernel (referred to as petitboot kernel) before booting into the
106 capture kernel. This kernel would have minimal kernel and/or
107 userspace support to process crash data. Such kernel needs to
108 preserve previously crash'ed kernel's memory for the subsequent
109 capture kernel boot to process this crash data. Kernel config
110 option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel
111 to ensure that crash data is preserved to process later.
113 -- On OPAL based machines (PowerNV), if the kernel is build with
114 CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also
115 exported as /sys/firmware/opal/core file. This procfs file is
116 helpful in debugging OPAL crashes with GDB. The kernel memory
117 used for exporting this procfs file can be released by echo'ing
118 '1' to /sys/kernel/fadump_release_opalcore node.
120 e.g.
121 # echo 1 > /sys/kernel/fadump_release_opalcore
123 Implementation details:
124 -----------------------
126 During boot, a check is made to see if firmware supports
127 this feature on that particular machine. If it does, then
128 we check to see if an active dump is waiting for us. If yes
129 then everything but boot memory size of RAM is reserved during
130 early boot (See Fig. 2). This area is released once we finish
131 collecting the dump from user land scripts (e.g. kdump scripts)
132 that are run. If there is dump data, then the
133 /sys/kernel/fadump_release_mem file is created, and the reserved
134 memory is held.
136 If there is no waiting dump data, then only the memory required to
137 hold CPU state, HPTE region, boot memory dump, FADump header and
138 elfcore header, is usually reserved at an offset greater than boot
139 memory size (see Fig. 1). This area is *not* released: this region
140 will be kept permanently reserved, so that it can act as a receptacle
141 for a copy of the boot memory content in addition to CPU state and
142 HPTE region, in the case a crash does occur.
144 Since this reserved memory area is used only after the system crash,
145 there is no point in blocking this significant chunk of memory from
146 production kernel. Hence, the implementation uses the Linux kernel's
147 Contiguous Memory Allocator (CMA) for memory reservation if CMA is
148 configured for kernel. With CMA reservation this memory will be
149 available for applications to use it, while kernel is prevented from
150 using it. With this FADump will still be able to capture all of the
151 kernel memory and most of the user space memory except the user pages
152 that were present in CMA region::
154 o Memory Reservation during first kernel
156 Low memory Top of memory
157 0 boot memory size |<--- Reserved dump area --->| |
158 | | | Permanent Reservation | |
159 V V | | V
160 +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
161 | | |///|////| DUMP | HDR | ELF |////| |
162 +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
163 | ^ ^ ^ ^ ^
164 | | | | | |
165 \ CPU HPTE / | |
166 ------------------------------ | |
167 Boot memory content gets transferred | |
168 to reserved area by firmware at the | |
169 time of crash. | |
170 FADump Header |
171 (meta area) |
172 |
173 |
174 Metadata: This area holds a metadata struture whose
175 address is registered with f/w and retrieved in the
176 second kernel after crash, on platforms that support
177 tags (OPAL). Having such structure with info needed
178 to process the crashdump eases dump capture process.
180 Fig. 1
183 o Memory Reservation during second kernel after crash
185 Low memory Top of memory
186 0 boot memory size |
187 | |<------------ Crash preserved area ------------>|
188 V V |<--- Reserved dump area --->| |
189 +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
190 | | |///|////| DUMP | HDR | ELF |////| |
191 +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
192 | |
193 V V
194 Used by second /proc/vmcore
195 kernel to boot
197 +---+
198 |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above
199 +---+ figures are not always present. For example, OPAL platform
200 does not have CPU & HPTE regions while Metadata region is
201 not supported on pSeries currently.
203 Fig. 2
206 Currently the dump will be copied from /proc/vmcore to a new file upon
207 user intervention. The dump data available through /proc/vmcore will be
208 in ELF format. Hence the existing kdump infrastructure (kdump scripts)
209 to save the dump works fine with minor modifications. KDump scripts on
210 major Distro releases have already been modified to work seemlessly (no
211 user intervention in saving the dump) when FADump is used, instead of
212 KDump, as dump mechanism.
214 The tools to examine the dump will be same as the ones
215 used for kdump.
217 How to enable firmware-assisted dump (FADump):
218 ----------------------------------------------
220 1. Set config option CONFIG_FA_DUMP=y and build kernel.
221 2. Boot into linux kernel with 'fadump=on' kernel cmdline option.
222 By default, FADump reserved memory will be initialized as CMA area.
223 Alternatively, user can boot linux kernel with 'fadump=nocma' to
224 prevent FADump to use CMA.
225 3. Optionally, user can also set 'crashkernel=' kernel cmdline
226 to specify size of the memory to reserve for boot memory dump
227 preservation.
229 NOTE:
230 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead
231 use 'crashkernel=' to specify size of the memory to reserve
232 for boot memory dump preservation.
233 2. If firmware-assisted dump fails to reserve memory then it
234 will fallback to existing kdump mechanism if 'crashkernel='
235 option is set at kernel cmdline.
236 3. if user wants to capture all of user space memory and ok with
237 reserved memory not available to production system, then
238 'fadump=nocma' kernel parameter can be used to fallback to
239 old behaviour.
241 Sysfs/debugfs files:
242 --------------------
244 Firmware-assisted dump feature uses sysfs file system to hold
245 the control files and debugfs file to display memory reserved region.
247 Here is the list of files under kernel sysfs:
249 /sys/kernel/fadump_enabled
250 This is used to display the FADump status.
252 - 0 = FADump is disabled
253 - 1 = FADump is enabled
255 This interface can be used by kdump init scripts to identify if
256 FADump is enabled in the kernel and act accordingly.
258 /sys/kernel/fadump_registered
259 This is used to display the FADump registration status as well
260 as to control (start/stop) the FADump registration.
262 - 0 = FADump is not registered.
263 - 1 = FADump is registered and ready to handle system crash.
265 To register FADump echo 1 > /sys/kernel/fadump_registered and
266 echo 0 > /sys/kernel/fadump_registered for un-register and stop the
267 FADump. Once the FADump is un-registered, the system crash will not
268 be handled and vmcore will not be captured. This interface can be
269 easily integrated with kdump service start/stop.
271 /sys/kernel/fadump_release_mem
272 This file is available only when FADump is active during
273 second kernel. This is used to release the reserved memory
274 region that are held for saving crash dump. To release the
275 reserved memory echo 1 to it::
277 echo 1 > /sys/kernel/fadump_release_mem
279 After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region
280 file will change to reflect the new memory reservations.
282 The existing userspace tools (kdump infrastructure) can be easily
283 enhanced to use this interface to release the memory reserved for
284 dump and continue without 2nd reboot.
286 /sys/kernel/fadump_release_opalcore
288 This file is available only on OPAL based machines when FADump is
289 active during capture kernel. This is used to release the memory
290 used by the kernel to export /sys/firmware/opal/core file. To
291 release this memory, echo '1' to it:
293 echo 1 > /sys/kernel/fadump_release_opalcore
295 Here is the list of files under powerpc debugfs:
296 (Assuming debugfs is mounted on /sys/kernel/debug directory.)
298 /sys/kernel/debug/powerpc/fadump_region
299 This file shows the reserved memory regions if FADump is
300 enabled otherwise this file is empty. The output format
301 is::
303 <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size>
305 and for kernel DUMP region is:
307 DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes
309 e.g.
310 Contents when FADump is registered during first kernel::
312 # cat /sys/kernel/debug/powerpc/fadump_region
313 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0
314 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0
315 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0
317 Contents when FADump is active during second kernel::
319 # cat /sys/kernel/debug/powerpc/fadump_region
320 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020
321 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000
322 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000
323 : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000
326 NOTE:
327 Please refer to Documentation/filesystems/debugfs.txt on
328 how to mount the debugfs filesystem.
331 TODO:
332 -----
333 - Need to come up with the better approach to find out more
334 accurate boot memory size that is required for a kernel to
335 boot successfully when booted with restricted memory.
336 - The FADump implementation introduces a FADump crash info structure
337 in the scratch area before the ELF core header. The idea of introducing
338 this structure is to pass some important crash info data to the second
339 kernel which will help second kernel to populate ELF core header with
340 correct data before it gets exported through /proc/vmcore. The current
341 design implementation does not address a possibility of introducing
342 additional fields (in future) to this structure without affecting
343 compatibility. Need to come up with the better approach to address this.
345 The possible approaches are:
347 1. Introduce version field for version tracking, bump up the version
348 whenever a new field is added to the structure in future. The version
349 field can be used to find out what fields are valid for the current
350 version of the structure.
351 2. Reserve the area of predefined size (say PAGE_SIZE) for this
352 structure and have unused area as reserved (initialized to zero)
353 for future field additions.
355 The advantage of approach 1 over 2 is we don't need to reserve extra space.
357 Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
359 This document is based on the original documentation written for phyp
361 assisted dump by Linas Vepstas and Manish Ahuja.