| blob | 75536bc5bea670ea2e1086d8264b3fcff64112ac |
1 =============================
2 User Guide for AMDGPU Backend
3 =============================
5 .. contents::
6 :local:
8 .. toctree::
9 :hidden:
11 AMDGPU/AMDGPUAsmGFX7
12 AMDGPU/AMDGPUAsmGFX8
13 AMDGPU/AMDGPUAsmGFX9
14 AMDGPU/AMDGPUAsmGFX900
15 AMDGPU/AMDGPUAsmGFX904
16 AMDGPU/AMDGPUAsmGFX906
17 AMDGPU/AMDGPUAsmGFX908
18 AMDGPU/AMDGPUAsmGFX90a
19 AMDGPU/AMDGPUAsmGFX940
20 AMDGPU/AMDGPUAsmGFX10
21 AMDGPU/AMDGPUAsmGFX1011
22 AMDGPU/AMDGPUAsmGFX1013
23 AMDGPU/AMDGPUAsmGFX1030
24 AMDGPU/AMDGPUAsmGFX11
25 AMDGPUModifierSyntax
26 AMDGPUOperandSyntax
27 AMDGPUInstructionSyntax
28 AMDGPUInstructionNotation
29 AMDGPUDwarfExtensionsForHeterogeneousDebugging
30 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack
32 Introduction
33 ============
35 The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the
36 R600 family up until the current GCN families. It lives in the
37 ``llvm/lib/Target/AMDGPU`` directory.
39 LLVM
40 ====
42 .. _amdgpu-target-triples:
44 Target Triples
45 --------------
47 Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>``
48 to specify the target triple:
50 .. table:: AMDGPU Architectures
51 :name: amdgpu-architecture-table
53 ============ ==============================================================
54 Architecture Description
55 ============ ==============================================================
56 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders.
57 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders.
58 ============ ==============================================================
60 .. table:: AMDGPU Vendors
61 :name: amdgpu-vendor-table
63 ============ ==============================================================
64 Vendor Description
65 ============ ==============================================================
66 ``amd`` Can be used for all AMD GPU usage.
67 ``mesa3d`` Can be used if the OS is ``mesa3d``.
68 ============ ==============================================================
70 .. table:: AMDGPU Operating Systems
71 :name: amdgpu-os
73 ============== ============================================================
74 OS Description
75 ============== ============================================================
76 *<empty>* Defaults to the *unknown* OS.
77 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes
78 such as:
80 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa*
81 loader on Linux. See *AMD ROCm Platform Release Notes*
82 [AMD-ROCm-Release-Notes]_ for supported hardware and
83 software.
84 - AMD's PAL runtime using the *pal-amdhsa* loader on
85 Windows.
87 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL
88 runtime using the *pal-amdpal* loader on Windows and Linux
89 Pro.
90 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa
91 3D runtime using the *mesa-mesa3d* loader on Linux.
92 ============== ============================================================
94 .. table:: AMDGPU Environments
95 :name: amdgpu-environment-table
97 ============ ==============================================================
98 Environment Description
99 ============ ==============================================================
100 *<empty>* Default.
101 ============ ==============================================================
103 .. _amdgpu-processors:
105 Processors
106 ----------
108 Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to
109 specify the AMDGPU processor together with optional target features. See
110 :ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target
111 specific information.
113 Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions:
115 * ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`).
118 .. table:: AMDGPU Processors
119 :name: amdgpu-processor-table
121 =========== =============== ============ ===== ================= =============== =============== ======================
122 Processor Alternative Target dGPU/ Target Target OS Support Example
123 Processor Triple APU Features Properties *(see* Products
124 Architecture Supported `amdgpu-os`_
125 *and
126 corresponding
127 runtime release
128 notes for
129 current
130 information and
131 level of
132 support)*
133 =========== =============== ============ ===== ================= =============== =============== ======================
134 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_
135 -----------------------------------------------------------------------------------------------------------------------
136 ``r600`` ``r600`` dGPU - Does not
137 support
138 generic
139 address
140 space
141 ``r630`` ``r600`` dGPU - Does not
142 support
143 generic
144 address
145 space
146 ``rs880`` ``r600`` dGPU - Does not
147 support
148 generic
149 address
150 space
151 ``rv670`` ``r600`` dGPU - Does not
152 support
153 generic
154 address
155 space
156 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_
157 -----------------------------------------------------------------------------------------------------------------------
158 ``rv710`` ``r600`` dGPU - Does not
159 support
160 generic
161 address
162 space
163 ``rv730`` ``r600`` dGPU - Does not
164 support
165 generic
166 address
167 space
168 ``rv770`` ``r600`` dGPU - Does not
169 support
170 generic
171 address
172 space
173 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_
174 -----------------------------------------------------------------------------------------------------------------------
175 ``cedar`` ``r600`` dGPU - Does not
176 support
177 generic
178 address
179 space
180 ``cypress`` ``r600`` dGPU - Does not
181 support
182 generic
183 address
184 space
185 ``juniper`` ``r600`` dGPU - Does not
186 support
187 generic
188 address
189 space
190 ``redwood`` ``r600`` dGPU - Does not
191 support
192 generic
193 address
194 space
195 ``sumo`` ``r600`` dGPU - Does not
196 support
197 generic
198 address
199 space
200 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_
201 -----------------------------------------------------------------------------------------------------------------------
202 ``barts`` ``r600`` dGPU - Does not
203 support
204 generic
205 address
206 space
207 ``caicos`` ``r600`` dGPU - Does not
208 support
209 generic
210 address
211 space
212 ``cayman`` ``r600`` dGPU - Does not
213 support
214 generic
215 address
216 space
217 ``turks`` ``r600`` dGPU - Does not
218 support
219 generic
220 address
221 space
222 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_
223 -----------------------------------------------------------------------------------------------------------------------
224 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal*
225 support
226 generic
227 address
228 space
229 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal*
230 - ``verde`` support
231 generic
232 address
233 space
234 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal*
235 - ``oland`` support
236 generic
237 address
238 space
239 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_
240 -----------------------------------------------------------------------------------------------------------------------
241 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000
242 flat - *pal-amdhsa* - A6 Pro-7050B
243 scratch - *pal-amdpal* - A8-7100
244 - A8 Pro-7150B
245 - A10-7300
246 - A10 Pro-7350B
247 - FX-7500
248 - A8-7200P
249 - A10-7400P
250 - FX-7600P
251 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100
252 flat - *pal-amdhsa* - FirePro W9100
253 scratch - *pal-amdpal* - FirePro S9150
254 - FirePro S9170
255 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290
256 flat - *pal-amdhsa* - Radeon R9 290x
257 scratch - *pal-amdpal* - Radeon R390
258 - Radeon R390x
259 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100
260 - ``mullins`` flat - *pal-amdpal* - E1-2200
261 scratch - E1-2500
262 - E2-3000
263 - E2-3800
264 - A4-5000
265 - A4-5100
266 - A6-5200
267 - A4 Pro-3340B
268 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790
269 flat - *pal-amdpal* - Radeon HD 8770
270 scratch - R7 260
271 - R7 260X
272 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA*
273 flat - *pal-amdpal*
274 scratch .. TODO::
276 Add product
277 names.
279 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_
280 -----------------------------------------------------------------------------------------------------------------------
281 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P
282 flat - *pal-amdhsa* - Pro A6-8500B
283 scratch - *pal-amdpal* - A8-8600P
284 - Pro A8-8600B
285 - FX-8800P
286 - Pro A12-8800B
287 - A10-8700P
288 - Pro A10-8700B
289 - A10-8780P
290 - A10-9600P
291 - A10-9630P
292 - A12-9700P
293 - A12-9730P
294 - FX-9800P
295 - FX-9830P
296 - E2-9010
297 - A6-9210
298 - A9-9410
299 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285
300 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380
301 scratch - *pal-amdpal* - Radeon R9 385
302 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano
303 - *pal-amdhsa* - Radeon R9 Fury
304 - *pal-amdpal* - Radeon R9 FuryX
305 - Radeon Pro Duo
306 - FirePro S9300x2
307 - Radeon Instinct MI8
308 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470
309 flat - *pal-amdhsa* - Radeon RX 480
310 scratch - *pal-amdpal* - Radeon Instinct MI6
311 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460
312 flat - *pal-amdhsa*
313 scratch - *pal-amdpal*
314 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150
315 flat - *pal-amdhsa* - FirePro S7100
316 scratch - *pal-amdpal* - FirePro W7100
317 - Mobile FirePro
318 M7170
319 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA*
320 flat - *pal-amdhsa*
321 scratch - *pal-amdpal* .. TODO::
323 Add product
324 names.
326 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ [AMD-GCN-GFX90A-CDNA2]_ [AMD-GCN-GFX940-GFX942-CDNA3]_
327 -----------------------------------------------------------------------------------------------------------------------
328 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega
329 flat - *pal-amdhsa* Frontier Edition
330 scratch - *pal-amdpal* - Radeon RX Vega 56
331 - Radeon RX Vega 64
332 - Radeon RX Vega 64
333 Liquid
334 - Radeon Instinct MI25
335 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G
336 flat - *pal-amdhsa* - Ryzen 5 2400G
337 scratch - *pal-amdpal*
338 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA*
339 - *pal-amdhsa*
340 - *pal-amdpal* .. TODO::
342 Add product
343 names.
345 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50
346 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60
347 scratch - *pal-amdpal* - Radeon VII
348 - Radeon Pro VII
349 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator
350 - xnack - Absolute
351 flat
352 scratch
353 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA*
354 flat
355 scratch .. TODO::
357 Add product
358 names.
360 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - AMD Instinct MI210 Accelerator
361 - tgsplit flat - *rocm-amdhsa* - AMD Instinct MI250 Accelerator
362 - xnack scratch - *rocm-amdhsa* - AMD Instinct MI250X Accelerator
363 - kernarg preload - Packed
364 work-item
365 IDs
367 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G
368 flat - Ryzen 7 4700GE
369 scratch - Ryzen 5 4600G
370 - Ryzen 5 4600GE
371 - Ryzen 3 4300G
372 - Ryzen 3 4300GE
373 - Ryzen Pro 4000G
374 - Ryzen 7 Pro 4700G
375 - Ryzen 7 Pro 4750GE
376 - Ryzen 5 Pro 4650G
377 - Ryzen 5 Pro 4650GE
378 - Ryzen 3 Pro 4350G
379 - Ryzen 3 Pro 4350GE
381 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA*
382 - tgsplit flat
383 - xnack scratch .. TODO::
384 - kernarg preload - Packed
385 work-item Add product
386 IDs names.
388 ``gfx941`` ``amdgcn`` dGPU - sramecc - Architected *TBA*
389 - tgsplit flat
390 - xnack scratch .. TODO::
391 - kernarg preload - Packed
392 work-item Add product
393 IDs names.
395 ``gfx942`` ``amdgcn`` dGPU - sramecc - Architected *TBA*
396 - tgsplit flat
397 - xnack scratch .. TODO::
398 - kernarg preload - Packed
399 work-item Add product
400 IDs names.
402 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_
403 -----------------------------------------------------------------------------------------------------------------------
404 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700
405 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT
406 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT
407 - Radeon Pro 5600M
408 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520
409 - wavefrontsize64 - Absolute - *pal-amdhsa*
410 - xnack flat - *pal-amdpal*
411 scratch
412 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500
413 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT
414 - xnack scratch - *pal-amdpal*
415 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA*
416 - wavefrontsize64 flat - *pal-amdhsa*
417 - xnack scratch - *pal-amdpal* .. TODO::
419 Add product
420 names.
422 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_
423 -----------------------------------------------------------------------------------------------------------------------
424 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800
425 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT
426 scratch - *pal-amdpal* - Radeon RX 6900 XT
427 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT
428 - wavefrontsize64 flat - *pal-amdhsa*
429 scratch - *pal-amdpal*
430 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA*
431 - wavefrontsize64 flat - *pal-amdhsa*
432 scratch - *pal-amdpal* .. TODO::
434 Add product
435 names.
437 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA*
438 - wavefrontsize64 flat
439 scratch .. TODO::
441 Add product
442 names.
443 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA*
444 - wavefrontsize64 flat
445 scratch .. TODO::
447 Add product
448 names.
450 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA*
451 - wavefrontsize64 flat
452 scratch .. TODO::
453 Add product
454 names.
456 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA*
457 - wavefrontsize64 flat
458 scratch .. TODO::
460 Add product
461 names.
463 **GCN GFX11 (RDNA 3)** [AMD-GCN-GFX11-RDNA3]_
464 -----------------------------------------------------------------------------------------------------------------------
465 ``gfx1100`` ``amdgcn`` dGPU - cumode - Architected - *pal-amdpal* *TBA*
466 - wavefrontsize64 flat
467 scratch .. TODO::
468 - Packed
469 work-item Add product
470 IDs names.
472 ``gfx1101`` ``amdgcn`` dGPU - cumode - Architected *TBA*
473 - wavefrontsize64 flat
474 scratch .. TODO::
475 - Packed
476 work-item Add product
477 IDs names.
479 ``gfx1102`` ``amdgcn`` dGPU - cumode - Architected *TBA*
480 - wavefrontsize64 flat
481 scratch .. TODO::
482 - Packed
483 work-item Add product
484 IDs names.
486 ``gfx1103`` ``amdgcn`` APU - cumode - Architected *TBA*
487 - wavefrontsize64 flat
488 scratch .. TODO::
489 - Packed
490 work-item Add product
491 IDs names.
493 ``gfx1150`` ``amdgcn`` APU - cumode - Architected *TBA*
494 - wavefrontsize64 flat
495 scratch .. TODO::
496 - Packed
497 work-item Add product
498 IDs names.
500 ``gfx1151`` ``amdgcn`` APU - cumode - Architected *TBA*
501 - wavefrontsize64 flat
502 scratch .. TODO::
503 - Packed
504 work-item Add product
505 IDs names.
507 ``gfx1200`` ``amdgcn`` dGPU - cumode - Architected *TBA*
508 - wavefrontsize64 flat
509 scratch .. TODO::
510 - Packed
511 work-item Add product
512 IDs names.
514 ``gfx1201`` ``amdgcn`` dGPU - cumode - Architected *TBA*
515 - wavefrontsize64 flat
516 scratch .. TODO::
517 - Packed
518 work-item Add product
519 IDs names.
521 =========== =============== ============ ===== ================= =============== =============== ======================
523 Generic processors allow execution of a single code object on any of the processors that
524 it supports. Such code objects may not perform as well as those for the non-generic processors.
526 Generic processors are only available on code object V6 and above (see :ref:`amdgpu-elf-code-object`).
528 Generic processor code objects are versioned. See :ref:`amdgpu-generic-processor-versioning` for more information on how versioning works.
530 .. table:: AMDGPU Generic Processors
531 :name: amdgpu-generic-processor-table
533 ==================== ============== ================= ================== ================= =================================
534 Processor Target Supported Target Features Target Properties Target Restrictions
535 Triple Processors Supported
536 Architecture
538 ==================== ============== ================= ================== ================= =================================
539 ``gfx9-generic`` ``amdgcn`` - ``gfx900`` - xnack - Absolute flat - ``v_mad_mix`` instructions
540 - ``gfx902`` scratch are not available on
541 - ``gfx904`` ``gfx900``, ``gfx902``,
542 - ``gfx906`` ``gfx909``, ``gfx90c``
543 - ``gfx909`` - ``v_fma_mix`` instructions
544 - ``gfx90c`` are not available on ``gfx904``
545 - sramecc is not available on
546 ``gfx906``
547 - The following instructions
548 are not available on ``gfx906``:
550 - ``v_fmac_f32``
551 - ``v_xnor_b32``
552 - ``v_dot4_i32_i8``
553 - ``v_dot8_i32_i4``
554 - ``v_dot2_i32_i16``
555 - ``v_dot2_u32_u16``
556 - ``v_dot4_u32_u8``
557 - ``v_dot8_u32_u4``
558 - ``v_dot2_f32_f16``
561 ``gfx10-1-generic`` ``amdgcn`` - ``gfx1010`` - xnack - Absolute flat - The following instructions are
562 - ``gfx1011`` - wavefrontsize64 scratch not available on ``gfx1011``
563 - ``gfx1012`` - cumode and ``gfx1012``
564 - ``gfx1013``
565 - ``v_dot4_i32_i8``
566 - ``v_dot8_i32_i4``
567 - ``v_dot2_i32_i16``
568 - ``v_dot2_u32_u16``
569 - ``v_dot2c_f32_f16``
570 - ``v_dot4c_i32_i8``
571 - ``v_dot4_u32_u8``
572 - ``v_dot8_u32_u4``
573 - ``v_dot2_f32_f16``
575 - BVH Ray Tracing instructions
576 are not available on
577 ``gfx1013``
580 ``gfx10-3-generic`` ``amdgcn`` - ``gfx1030`` - wavefrontsize64 - Absolute flat No restrictions.
581 - ``gfx1031`` - cumode scratch
582 - ``gfx1032``
583 - ``gfx1033``
584 - ``gfx1034``
585 - ``gfx1035``
586 - ``gfx1036``
589 ``gfx11-generic`` ``amdgcn`` - ``gfx1100`` - wavefrontsize64 - Architected Various codegen pessimizations
590 - ``gfx1101`` - cumode flat scratch are applied to work around some
591 - ``gfx1102`` - Packed hazards specific to some targets
592 - ``gfx1103`` work-item within this family.
593 - ``gfx1150`` IDs
594 - ``gfx1151`` Not all VGPRs can be used on:
596 - ``gfx1100``
597 - ``gfx1101``
598 - ``gfx1151``
600 SALU floating point instructions
601 and single-use VGPR hint
602 instructions are not available
603 on:
605 - ``gfx1150``
606 - ``gfx1151``
608 SGPRs are not supported for src1
609 in dpp instructions for:
611 - ``gfx1150``
612 - ``gfx1151``
613 ==================== ============== ================= ================== ================= =================================
615 .. _amdgpu-generic-processor-versioning:
617 Generic Processor Versioning
618 ----------------------------
620 Generic processor (see :ref:`amdgpu-generic-processor-table`) code objects are versioned (see :ref:`amdgpu-elf-header-e_flags-table-v6-onwards`) between 1 and 255.
621 The version of non-generic code objects is always set to 0.
623 For a generic code object, adding a new supported processor may require the code generated for the generic target to be changed
624 so it can continue to execute on the previously supported processors as well as on the new one.
625 When this happens, the generic code object version number is incremented at the same time as the generic target is updated.
627 Each supported processor of a generic target is mapped to the version it was introduced in.
628 A generic code object can execute on a supported processor if the version of the code object being loaded is
629 greater than or equal to the version in which the processor was added to the generic target.
631 .. _amdgpu-target-features:
633 Target Features
634 ---------------
636 Target features control how code is generated to support certain
637 processor specific features. Not all target features are supported by
638 all processors. The runtime must ensure that the features supported by
639 the device used to execute the code match the features enabled when
640 generating the code. A mismatch of features may result in incorrect
641 execution, or a reduction in performance.
643 The target features supported by each processor is listed in
644 :ref:`amdgpu-processors`.
646 Target features are controlled by exactly one of the following Clang
647 options:
649 ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>``
651 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as
652 optional components of the target ID. If omitted, the target feature has the
653 ``any`` value. See :ref:`amdgpu-target-id`.
655 ``-m[no-]<target-feature>``
657 Target features not specified by the target ID are specified using a
658 separate option. These target features can have an ``on`` or ``off``
659 value. ``on`` is specified by omitting the ``no-`` prefix, and
660 ``off`` is specified by including the ``no-`` prefix. The default
661 if not specified is ``off``.
663 For example:
665 ``-mcpu=gfx908:xnack+``
666 Enable the ``xnack`` feature.
667 ``-mcpu=gfx908:xnack-``
668 Disable the ``xnack`` feature.
669 ``-mcumode``
670 Enable the ``cumode`` feature.
671 ``-mno-cumode``
672 Disable the ``cumode`` feature.
674 .. table:: AMDGPU Target Features
675 :name: amdgpu-target-features-table
677 =============== ============================ ==================================================
678 Target Feature Clang Option to Control Description
679 Name
680 =============== ============================ ==================================================
681 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used
682 when generating code for kernels. When disabled
683 native WGP wavefront execution mode is used,
684 when enabled CU wavefront execution mode is used
685 (see :ref:`amdgpu-amdhsa-memory-model`).
687 sramecc - ``-mcpu`` If specified, generate code that can only be
688 - ``--offload-arch`` loaded and executed in a process that has a
689 matching setting for SRAMECC.
691 If not specified for code object V2 to V3, generate
692 code that can be loaded and executed in a process
693 with SRAMECC enabled.
695 If not specified for code object V4 or above, generate
696 code that can be loaded and executed in a process
697 with either setting of SRAMECC.
699 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes
700 work-groups are launched in threadgroup split mode.
701 When enabled the waves of a work-group may be
702 launched in different CUs.
704 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when
705 generating code for kernels. When disabled
706 native wavefront size 32 is used, when enabled
707 wavefront size 64 is used.
709 xnack - ``-mcpu`` If specified, generate code that can only be
710 - ``--offload-arch`` loaded and executed in a process that has a
711 matching setting for XNACK replay.
713 If not specified for code object V2 to V3, generate
714 code that can be loaded and executed in a process
715 with XNACK replay enabled.
717 If not specified for code object V4 or above, generate
718 code that can be loaded and executed in a process
719 with either setting of XNACK replay.
721 XNACK replay can be used for demand paging and
722 page migration. If enabled in the device, then if
723 a page fault occurs the code may execute
724 incorrectly unless generated with XNACK replay
725 enabled, or generated for code object V4 or above without
726 specifying XNACK replay. Executing code that was
727 generated with XNACK replay enabled, or generated
728 for code object V4 or above without specifying XNACK replay,
729 on a device that does not have XNACK replay
730 enabled will execute correctly but may be less
731 performant than code generated for XNACK replay
732 disabled.
733 =============== ============================ ==================================================
735 .. _amdgpu-target-id:
737 Target ID
738 ---------
740 AMDGPU supports target IDs. See `Clang Offload Bundler
741 <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general
742 description. The AMDGPU target specific information is:
744 **processor**
745 Is an AMDGPU processor or alternative processor name specified in
746 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both
747 the primary processor and alternative processor names. The canonical form
748 target ID only allow the primary processor name.
750 **target-feature**
751 Is a target feature name specified in :ref:`amdgpu-target-features-table` that
752 is supported by the processor. The target features supported by each processor
753 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in
754 a target ID are marked as being controlled by ``-mcpu`` and
755 ``--offload-arch``. Each target feature must appear at most once in a target
756 ID. The non-canonical form target ID allows the target features to be
757 specified in any order. The canonical form target ID requires the target
758 features to be specified in alphabetic order.
760 .. _amdgpu-target-id-v2-v3:
762 Code Object V2 to V3 Target ID
763 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
765 The target ID syntax for code object V2 to V3 is the same as defined in `Clang
766 Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except
767 when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler
768 directive and the bundle entry ID. In those cases it has the following BNF
769 syntax:
771 .. code::
773 <target-id> ::== <processor> ( "+" <target-feature> )*
775 Where a target feature is omitted if *Off* and present if *On* or *Any*.
777 .. note::
779 The code object V2 to V3 cannot represent *Any* and treats it the same as
780 *On*.
782 .. _amdgpu-embedding-bundled-objects:
784 Embedding Bundled Code Objects
785 ------------------------------
787 AMDGPU supports the HIP and OpenMP languages that perform code object embedding
788 as described in `Clang Offload Bundler
789 <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_.
791 .. note::
793 The target ID syntax used for code object V2 to V3 for a bundle entry ID
794 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`.
796 .. _amdgpu-address-spaces:
798 Address Spaces
799 --------------
801 The AMDGPU architecture supports a number of memory address spaces. The address
802 space names use the OpenCL standard names, with some additions.
804 The AMDGPU address spaces correspond to target architecture specific LLVM
805 address space numbers used in LLVM IR.
807 The AMDGPU address spaces are described in
808 :ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are
809 supported for the ``amdgcn`` target.
811 .. table:: AMDGPU Address Spaces
812 :name: amdgpu-address-spaces-table
814 ===================================== =============== =========== ================ ======= ============================
815 .. 64-Bit Process Address Space
816 ------------------------------------- --------------- ----------- ---------------- ------------------------------------
817 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value
818 Space Number Name Name Size
819 ===================================== =============== =========== ================ ======= ============================
820 Generic 0 flat flat 64 0x0000000000000000
821 Global 1 global global 64 0x0000000000000000
822 Region 2 N/A GDS 32 *not implemented for AMDHSA*
823 Local 3 group LDS 32 0xFFFFFFFF
824 Constant 4 constant *same as global* 64 0x0000000000000000
825 Private 5 private scratch 32 0xFFFFFFFF
826 Constant 32-bit 6 *TODO* 0x00000000
827 Buffer Fat Pointer 7 N/A N/A 160 0
828 Buffer Resource 8 N/A V# 128 0x00000000000000000000000000000000
829 Buffer Strided Pointer (experimental) 9 *TODO*
830 Streamout Registers 128 N/A GS_REGS
831 ===================================== =============== =========== ================ ======= ============================
833 **Generic**
834 The generic address space is supported unless the *Target Properties* column
835 of :ref:`amdgpu-processor-table` specifies *Does not support generic address
836 space*.
838 The generic address space uses the hardware flat address support for two fixed
839 ranges of virtual addresses (the private and local apertures), that are
840 outside the range of addressable global memory, to map from a flat address to
841 a private or local address. This uses FLAT instructions that can take a flat
842 address and access global, private (scratch), and group (LDS) memory depending
843 on if the address is within one of the aperture ranges.
845 Flat access to scratch requires hardware aperture setup and setup in the
846 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat
847 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register
848 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`).
850 To convert between a private or group address space address (termed a segment
851 address) and a flat address the base address of the corresponding aperture
852 can be used. For GFX7-GFX8 these are available in the
853 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with
854 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For
855 GFX9-GFX11 the aperture base addresses are directly available as inline
856 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``.
857 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is
858 aligned to 2^32 which makes it easier to convert from flat to segment or
859 segment to flat.
861 A global address space address has the same value when used as a flat address
862 so no conversion is needed.
864 **Global and Constant**
865 The global and constant address spaces both use global virtual addresses,
866 which are the same virtual address space used by the CPU. However, some
867 virtual addresses may only be accessible to the CPU, some only accessible
868 by the GPU, and some by both.
870 Using the constant address space indicates that the data will not change
871 during the execution of the kernel. This allows scalar read instructions to
872 be used. As the constant address space could only be modified on the host
873 side, a generic pointer loaded from the constant address space is safe to be
874 assumed as a global pointer since only the device global memory is visible
875 and managed on the host side. The vector and scalar L1 caches are invalidated
876 of volatile data before each kernel dispatch execution to allow constant
877 memory to change values between kernel dispatches.
879 **Region**
880 The region address space uses the hardware Global Data Store (GDS). All
881 wavefronts executing on the same device will access the same memory for any
882 given region address. However, the same region address accessed by wavefronts
883 executing on different devices will access different memory. It is higher
884 performance than global memory. It is allocated by the runtime. The data
885 store (DS) instructions can be used to access it.
887 **Local**
888 The local address space uses the hardware Local Data Store (LDS) which is
889 automatically allocated when the hardware creates the wavefronts of a
890 work-group, and freed when all the wavefronts of a work-group have
891 terminated. All wavefronts belonging to the same work-group will access the
892 same memory for any given local address. However, the same local address
893 accessed by wavefronts belonging to different work-groups will access
894 different memory. It is higher performance than global memory. The data store
895 (DS) instructions can be used to access it.
897 **Private**
898 The private address space uses the hardware scratch memory support which
899 automatically allocates memory when it creates a wavefront and frees it when
900 a wavefronts terminates. The memory accessed by a lane of a wavefront for any
901 given private address will be different to the memory accessed by another lane
902 of the same or different wavefront for the same private address.
904 If a kernel dispatch uses scratch, then the hardware allocates memory from a
905 pool of backing memory allocated by the runtime for each wavefront. The lanes
906 of the wavefront access this using dword (4 byte) interleaving. The mapping
907 used from private address to backing memory address is:
909 ``wavefront-scratch-base +
910 ((private-address / 4) * wavefront-size * 4) +
911 (wavefront-lane-id * 4) + (private-address % 4)``
913 If each lane of a wavefront accesses the same private address, the
914 interleaving results in adjacent dwords being accessed and hence requires
915 fewer cache lines to be fetched.
917 There are different ways that the wavefront scratch base address is
918 determined by a wavefront (see
919 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
921 Scratch memory can be accessed in an interleaved manner using buffer
922 instructions with the scratch buffer descriptor and per wavefront scratch
923 offset, by the scratch instructions, or by flat instructions. Multi-dword
924 access is not supported except by flat and scratch instructions in
925 GFX9-GFX11.
927 Code that manipulates the stack values in other lanes of a wavefront,
928 such as by ``addrspacecast``-ing stack pointers to generic ones and taking offsets
929 that reach other lanes or by explicitly constructing the scratch buffer descriptor,
930 triggers undefined behavior when it modifies the scratch values of other lanes.
931 The compiler may assume that such modifications do not occur.
932 When using code object V5 ``LIBOMPTARGET_STACK_SIZE`` may be used to provide the
933 private segment size in bytes, for cases where a dynamic stack is used.
935 **Constant 32-bit**
936 *TODO*
938 **Buffer Fat Pointer**
939 The buffer fat pointer is an experimental address space that is currently
940 unsupported in the backend. It exposes a non-integral pointer that is in
941 the future intended to support the modelling of 128-bit buffer descriptors
942 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit
943 *pointer*), allowing normal LLVM load/store/atomic operations to be used to
944 model the buffer descriptors used heavily in graphics workloads targeting
945 the backend.
947 The buffer descriptor used to construct a buffer fat pointer must be *raw*:
948 the stride must be 0, the "add tid" flag must be 0, the swizzle enable bits
949 must be off, and the extent must be measured in bytes. (On subtargets where
950 bounds checking may be disabled, buffer fat pointers may choose to enable
951 it or not).
953 **Buffer Resource**
954 The buffer resource pointer, in address space 8, is the newer form
955 for representing buffer descriptors in AMDGPU IR, replacing their
956 previous representation as `<4 x i32>`. It is a non-integral pointer
957 that represents a 128-bit buffer descriptor resource (`V#`).
959 Since, in general, a buffer resource supports complex addressing modes that cannot
960 be easily represented in LLVM (such as implicit swizzled access to structured
961 buffers), it is **illegal** to perform non-trivial address computations, such as
962 ``getelementptr`` operations, on buffer resources. They may be passed to
963 AMDGPU buffer intrinsics, and they may be converted to and from ``i128``.
965 Casting a buffer resource to a buffer fat pointer is permitted and adds an offset
966 of 0.
968 Buffer resources can be created from 64-bit pointers (which should be either
969 generic or global) using the `llvm.amdgcn.make.buffer.rsrc` intrinsic, which
970 takes the pointer, which becomes the base of the resource,
971 the 16-bit stride (and swzizzle control) field stored in bits `63:48` of a `V#`,
972 the 32-bit NumRecords/extent field (bits `95:64`), and the 32-bit flags field
973 (bits `127:96`). The specific interpretation of these fields varies by the
974 target architecture and is detailed in the ISA descriptions.
976 **Buffer Strided Pointer**
977 The buffer index pointer is an experimental address space. It represents
978 a 128-bit buffer descriptor and a 32-bit offset, like the **Buffer Fat
979 Pointer**. Additionally, it contains an index into the buffer, which
980 allows the direct addressing of structured elements. These components appear
981 in that order, i.e., the descriptor comes first, then the 32-bit offset
982 followed by the 32-bit index.
984 The bits in the buffer descriptor must meet the following requirements:
985 the stride is the size of a structured element, the "add tid" flag must be 0,
986 and the swizzle enable bits must be off.
988 **Streamout Registers**
989 Dedicated registers used by the GS NGG Streamout Instructions. The register
990 file is modelled as a memory in a distinct address space because it is indexed
991 by an address-like offset in place of named registers, and because register
992 accesses affect LGKMcnt. This is an internal address space used only by the
993 compiler. Do not use this address space for IR pointers.
995 .. _amdgpu-memory-scopes:
997 Memory Scopes
998 -------------
1000 This section provides LLVM memory synchronization scopes supported by the AMDGPU
1001 backend memory model when the target triple OS is ``amdhsa`` (see
1002 :ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`).
1004 The memory model supported is based on the HSA memory model [HSA]_ which is
1005 based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before
1006 relation is transitive over the synchronizes-with relation independent of scope
1007 and synchronizes-with allows the memory scope instances to be inclusive (see
1008 table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`).
1010 This is different to the OpenCL [OpenCL]_ memory model which does not have scope
1011 inclusion and requires the memory scopes to exactly match. However, this
1012 is conservatively correct for OpenCL.
1014 .. table:: AMDHSA LLVM Sync Scopes
1015 :name: amdgpu-amdhsa-llvm-sync-scopes-table
1017 ======================= ===================================================
1018 LLVM Sync Scope Description
1019 ======================= ===================================================
1020 *none* The default: ``system``.
1022 Synchronizes with, and participates in modification
1023 and seq_cst total orderings with, other operations
1024 (except image operations) for all address spaces
1025 (except private, or generic that accesses private)
1026 provided the other operation's sync scope is:
1028 - ``system``.
1029 - ``agent`` and executed by a thread on the same
1030 agent.
1031 - ``workgroup`` and executed by a thread in the
1032 same work-group.
1033 - ``wavefront`` and executed by a thread in the
1034 same wavefront.
1036 ``agent`` Synchronizes with, and participates in modification
1037 and seq_cst total orderings with, other operations
1038 (except image operations) for all address spaces
1039 (except private, or generic that accesses private)
1040 provided the other operation's sync scope is:
1042 - ``system`` or ``agent`` and executed by a thread
1043 on the same agent.
1044 - ``workgroup`` and executed by a thread in the
1045 same work-group.
1046 - ``wavefront`` and executed by a thread in the
1047 same wavefront.
1049 ``workgroup`` Synchronizes with, and participates in modification
1050 and seq_cst total orderings with, other operations
1051 (except image operations) for all address spaces
1052 (except private, or generic that accesses private)
1053 provided the other operation's sync scope is:
1055 - ``system``, ``agent`` or ``workgroup`` and
1056 executed by a thread in the same work-group.
1057 - ``wavefront`` and executed by a thread in the
1058 same wavefront.
1060 ``wavefront`` Synchronizes with, and participates in modification
1061 and seq_cst total orderings with, other operations
1062 (except image operations) for all address spaces
1063 (except private, or generic that accesses private)
1064 provided the other operation's sync scope is:
1066 - ``system``, ``agent``, ``workgroup`` or
1067 ``wavefront`` and executed by a thread in the
1068 same wavefront.
1070 ``singlethread`` Only synchronizes with and participates in
1071 modification and seq_cst total orderings with,
1072 other operations (except image operations) running
1073 in the same thread for all address spaces (for
1074 example, in signal handlers).
1076 ``one-as`` Same as ``system`` but only synchronizes with other
1077 operations within the same address space.
1079 ``agent-one-as`` Same as ``agent`` but only synchronizes with other
1080 operations within the same address space.
1082 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with
1083 other operations within the same address space.
1085 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with
1086 other operations within the same address space.
1088 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with
1089 other operations within the same address space.
1090 ======================= ===================================================
1092 LLVM IR Intrinsics
1093 ------------------
1095 The AMDGPU backend implements the following LLVM IR intrinsics.
1097 *This section is WIP.*
1099 .. table:: AMDGPU LLVM IR Intrinsics
1100 :name: amdgpu-llvm-ir-intrinsics-table
1102 ============================================== ==========================================================
1103 LLVM Intrinsic Description
1104 ============================================== ==========================================================
1105 llvm.amdgcn.sqrt Provides direct access to v_sqrt_f64, v_sqrt_f32 and v_sqrt_f16
1106 (on targets with half support). Performs sqrt function.
1108 llvm.amdgcn.log Provides direct access to v_log_f32 and v_log_f16
1109 (on targets with half support). Performs log2 function.
1111 llvm.amdgcn.exp2 Provides direct access to v_exp_f32 and v_exp_f16
1112 (on targets with half support). Performs exp2 function.
1114 :ref:`llvm.frexp <int_frexp>` Implemented for half, float and double.
1116 :ref:`llvm.log2 <int_log2>` Implemented for float and half (and vectors of float or
1117 half). Not implemented for double. Hardware provides
1118 1ULP accuracy for float, and 0.51ULP for half. Float
1119 instruction does not natively support denormal
1120 inputs.
1122 :ref:`llvm.sqrt <int_sqrt>` Implemented for double, float and half (and vectors).
1124 :ref:`llvm.log <int_log>` Implemented for float and half (and vectors).
1126 :ref:`llvm.exp <int_exp>` Implemented for float and half (and vectors).
1128 :ref:`llvm.log10 <int_log10>` Implemented for float and half (and vectors).
1130 :ref:`llvm.exp2 <int_exp2>` Implemented for float and half (and vectors of float or
1131 half). Not implemented for double. Hardware provides
1132 1ULP accuracy for float, and 0.51ULP for half. Float
1133 instruction does not natively support denormal
1134 inputs.
1136 :ref:`llvm.stacksave.p5 <int_stacksave>` Implemented, must use the alloca address space.
1137 :ref:`llvm.stackrestore.p5 <int_stackrestore>` Implemented, must use the alloca address space.
1139 :ref:`llvm.get.fpmode.i32 <int_get_fpmode>` The natural floating-point mode type is i32. This
1140 implemented by extracting relevant bits out of the MODE
1141 register with s_getreg_b32. The first 10 bits are the
1142 core floating-point mode. Bits 12:18 are the exception
1143 mask. On gfx9+, bit 23 is FP16_OVFL. Bitfields not
1144 relevant to floating-point instructions are 0s.
1146 :ref:`llvm.get.rounding<int_get_rounding>` AMDGPU supports two separately controllable rounding
1147 modes depending on the floating-point type. One
1148 controls float, and the other controls both double and
1149 half operations. If both modes are the same, returns
1150 one of the standard return values. If the modes are
1151 different, returns one of :ref:`12 extended values
1152 <amdgpu-rounding-mode-enumeration-values-table>`
1153 describing the two modes.
1155 To nearest, ties away from zero is not a supported
1156 mode. The raw rounding mode values in the MODE
1157 register do not exactly match the FLT_ROUNDS values,
1158 so a conversion is performed.
1160 :ref:`llvm.set.rounding<int_set_rounding>` Input value expected to be one of the valid results
1161 from '``llvm.get.rounding``'. Rounding mode is
1162 undefined if not passed a valid input. This should be
1163 a wave uniform value. In case of a divergent input
1164 value, the first active lane's value will be used.
1166 :ref:`llvm.get.fpenv<int_get_fpenv>` Returns the current value of the AMDGPU floating point environment.
1167 This stores information related to the current rounding mode,
1168 denormalization mode, enabled traps, and floating point exceptions.
1169 The format is a 64-bit concatenation of the MODE and TRAPSTS registers.
1171 :ref:`llvm.set.fpenv<int_set_fpenv>` Sets the floating point environment to the specifies state.
1173 llvm.amdgcn.wave.reduce.umin Performs an arithmetic unsigned min reduction on the unsigned values
1174 provided by each lane in the wavefront.
1175 Intrinsic takes a hint for reduction strategy using second operand
1176 0: Target default preference,
1177 1: `Iterative strategy`, and
1178 2: `DPP`.
1179 If target does not support the DPP operations (e.g. gfx6/7),
1180 reduction will be performed using default iterative strategy.
1181 Intrinsic is currently only implemented for i32.
1183 llvm.amdgcn.wave.reduce.umax Performs an arithmetic unsigned max reduction on the unsigned values
1184 provided by each lane in the wavefront.
1185 Intrinsic takes a hint for reduction strategy using second operand
1186 0: Target default preference,
1187 1: `Iterative strategy`, and
1188 2: `DPP`.
1189 If target does not support the DPP operations (e.g. gfx6/7),
1190 reduction will be performed using default iterative strategy.
1191 Intrinsic is currently only implemented for i32.
1193 llvm.amdgcn.udot2 Provides direct access to v_dot2_u32_u16 across targets which
1194 support such instructions. This performs unsigned dot product
1195 with two v2i16 operands, summed with the third i32 operand. The
1196 i1 fourth operand is used to clamp the output.
1198 llvm.amdgcn.udot4 Provides direct access to v_dot4_u32_u8 across targets which
1199 support such instructions. This performs unsigned dot product
1200 with two i32 operands (holding a vector of 4 8bit values), summed
1201 with the third i32 operand. The i1 fourth operand is used to clamp
1202 the output.
1204 llvm.amdgcn.udot8 Provides direct access to v_dot8_u32_u4 across targets which
1205 support such instructions. This performs unsigned dot product
1206 with two i32 operands (holding a vector of 8 4bit values), summed
1207 with the third i32 operand. The i1 fourth operand is used to clamp
1208 the output.
1210 llvm.amdgcn.sdot2 Provides direct access to v_dot2_i32_i16 across targets which
1211 support such instructions. This performs signed dot product
1212 with two v2i16 operands, summed with the third i32 operand. The
1213 i1 fourth operand is used to clamp the output.
1214 When applicable (e.g. no clamping), this is lowered into
1215 v_dot2c_i32_i16 for targets which support it.
1217 llvm.amdgcn.sdot4 Provides direct access to v_dot4_i32_i8 across targets which
1218 support such instructions. This performs signed dot product
1219 with two i32 operands (holding a vector of 4 8bit values), summed
1220 with the third i32 operand. The i1 fourth operand is used to clamp
1221 the output.
1222 When applicable (i.e. no clamping / operand modifiers), this is lowered
1223 into v_dot4c_i32_i8 for targets which support it.
1224 RDNA3 does not offer v_dot4_i32_i8, and rather offers
1225 v_dot4_i32_iu8 which has operands to hold the signedness of the
1226 vector operands. Thus, this intrinsic lowers to the signed version
1227 of this instruction for gfx11 targets.
1229 llvm.amdgcn.sdot8 Provides direct access to v_dot8_u32_u4 across targets which
1230 support such instructions. This performs signed dot product
1231 with two i32 operands (holding a vector of 8 4bit values), summed
1232 with the third i32 operand. The i1 fourth operand is used to clamp
1233 the output.
1234 When applicable (i.e. no clamping / operand modifiers), this is lowered
1235 into v_dot8c_i32_i4 for targets which support it.
1236 RDNA3 does not offer v_dot8_i32_i4, and rather offers
1237 v_dot4_i32_iu4 which has operands to hold the signedness of the
1238 vector operands. Thus, this intrinsic lowers to the signed version
1239 of this instruction for gfx11 targets.
1241 llvm.amdgcn.sudot4 Provides direct access to v_dot4_i32_iu8 on gfx11 targets. This performs
1242 dot product with two i32 operands (holding a vector of 4 8bit values), summed
1243 with the fifth i32 operand. The i1 sixth operand is used to clamp
1244 the output. The i1s preceding the vector operands decide the signedness.
1246 llvm.amdgcn.sudot8 Provides direct access to v_dot8_i32_iu4 on gfx11 targets. This performs
1247 dot product with two i32 operands (holding a vector of 8 4bit values), summed
1248 with the fifth i32 operand. The i1 sixth operand is used to clamp
1249 the output. The i1s preceding the vector operands decide the signedness.
1251 llvm.amdgcn.sched_barrier Controls the types of instructions that may be allowed to cross the intrinsic
1252 during instruction scheduling. The parameter is a mask for the instruction types
1253 that can cross the intrinsic.
1255 - 0x0000: No instructions may be scheduled across sched_barrier.
1256 - 0x0001: All, non-memory, non-side-effect producing instructions may be
1257 scheduled across sched_barrier, *i.e.* allow ALU instructions to pass.
1258 - 0x0002: VALU instructions may be scheduled across sched_barrier.
1259 - 0x0004: SALU instructions may be scheduled across sched_barrier.
1260 - 0x0008: MFMA/WMMA instructions may be scheduled across sched_barrier.
1261 - 0x0010: All VMEM instructions may be scheduled across sched_barrier.
1262 - 0x0020: VMEM read instructions may be scheduled across sched_barrier.
1263 - 0x0040: VMEM write instructions may be scheduled across sched_barrier.
1264 - 0x0080: All DS instructions may be scheduled across sched_barrier.
1265 - 0x0100: All DS read instructions may be scheduled accoss sched_barrier.
1266 - 0x0200: All DS write instructions may be scheduled across sched_barrier.
1267 - 0x0400: All Transcendental (e.g. V_EXP) instructions may be scheduled across sched_barrier.
1269 llvm.amdgcn.sched_group_barrier Creates schedule groups with specific properties to create custom scheduling
1270 pipelines. The ordering between groups is enforced by the instruction scheduler.
1271 The intrinsic applies to the code that preceeds the intrinsic. The intrinsic
1272 takes three values that control the behavior of the schedule groups.
1274 - Mask : Classify instruction groups using the llvm.amdgcn.sched_barrier mask values.
1275 - Size : The number of instructions that are in the group.
1276 - SyncID : Order is enforced between groups with matching values.
1278 The mask can include multiple instruction types. It is undefined behavior to set
1279 values beyond the range of valid masks.
1281 Combining multiple sched_group_barrier intrinsics enables an ordering of specific
1282 instruction types during instruction scheduling. For example, the following enforces
1283 a sequence of 1 VMEM read, followed by 1 VALU instruction, followed by 5 MFMA
1284 instructions.
1286 | ``// 1 VMEM read``
1287 | ``__builtin_amdgcn_sched_group_barrier(32, 1, 0)``
1288 | ``// 1 VALU``
1289 | ``__builtin_amdgcn_sched_group_barrier(2, 1, 0)``
1290 | ``// 5 MFMA``
1291 | ``__builtin_amdgcn_sched_group_barrier(8, 5, 0)``
1293 llvm.amdgcn.iglp_opt An **experimental** intrinsic for instruction group level parallelism. The intrinsic
1294 implements predefined intruction scheduling orderings. The intrinsic applies to the
1295 surrounding scheduling region. The intrinsic takes a value that specifies the
1296 strategy. The compiler implements two strategies.
1298 0. Interleave DS and MFMA instructions for small GEMM kernels.
1299 1. Interleave DS and MFMA instructions for single wave small GEMM kernels.
1301 Only one iglp_opt intrinsic may be used in a scheduling region. The iglp_opt intrinsic
1302 cannot be combined with sched_barrier or sched_group_barrier.
1304 The iglp_opt strategy implementations are subject to change.
1306 llvm.amdgcn.atomic.cond.sub.u32 Provides direct access to flat_atomic_cond_sub_u32, global_atomic_cond_sub_u32
1307 and ds_cond_sub_u32 based on address space on gfx12 targets. This
1308 performs subtraction only if the memory value is greater than or
1309 equal to the data value.
1311 llvm.amdgcn.s.getpc Provides access to the s_getpc_b64 instruction, but with the return value
1312 sign-extended from the width of the underlying PC hardware register even on
1313 processors where the s_getpc_b64 instruction returns a zero-extended value.
1315 ============================================== ==========================================================
1317 .. TODO::
1319 List AMDGPU intrinsics.
1321 .. _amdgpu_metadata:
1323 LLVM IR Metadata
1324 ================
1326 The AMDGPU backend implements the following target custom LLVM IR
1327 metadata.
1329 .. _amdgpu_last_use:
1331 '``amdgpu.last.use``' Metadata
1332 ------------------------------
1334 Sets TH_LOAD_LU temporal hint on load instructions that support it.
1335 Takes priority over nontemporal hint (TH_LOAD_NT). This takes no
1336 arguments.
1338 .. code-block:: llvm
1340 %val = load i32, ptr %in, align 4, !amdgpu.last.use !{}
1343 LLVM IR Attributes
1344 ==================
1346 The AMDGPU backend supports the following LLVM IR attributes.
1348 .. table:: AMDGPU LLVM IR Attributes
1349 :name: amdgpu-llvm-ir-attributes-table
1351 ======================================= ==========================================================
1352 LLVM Attribute Description
1353 ======================================= ==========================================================
1354 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that
1355 will be specified when the kernel is dispatched. Generated
1356 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_.
1357 The IR implied default value is 1,1024. Clang may emit this attribute
1358 with more restrictive bounds depending on language defaults.
1359 If the actual block or workgroup size exceeds the limit at any point during
1360 the execution, the behavior is undefined. For example, even if there is
1361 only one active thread but the thread local id exceeds the limit, the
1362 behavior is undefined.
1364 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel
1365 argument block size for the implicit arguments. This
1366 varies by OS and language (for OpenCL see
1367 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`).
1368 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by
1369 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_.
1370 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the
1371 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_.
1372 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per
1373 execution unit. Generated by the ``amdgpu_waves_per_eu``
1374 CLANG attribute [CLANG-ATTR]_. This is an optimization hint,
1375 and the backend may not be able to satisfy the request. If
1376 the specified range is incompatible with the function's
1377 "amdgpu-flat-work-group-size" value, the implied occupancy
1378 bounds by the workgroup size takes precedence.
1380 "amdgpu-ieee" true/false. GFX6-GFX11 Only
1381 Specify whether the function expects the IEEE field of the
1382 mode register to be set on entry. Overrides the default for
1383 the calling convention.
1384 "amdgpu-dx10-clamp" true/false. GFX6-GFX11 Only
1385 Specify whether the function expects the DX10_CLAMP field of
1386 the mode register to be set on entry. Overrides the default
1387 for the calling convention.
1389 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the
1390 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this
1391 attribute, or reached through a call site marked with this attribute,
1392 the value returned by the intrinsic is undefined. The backend can
1393 generally infer this during code generation, so typically there is no
1394 benefit to frontends marking functions with this.
1396 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the
1397 llvm.amdgcn.workitem.id.y intrinsic.
1399 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the
1400 llvm.amdgcn.workitem.id.z intrinsic.
1402 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the
1403 llvm.amdgcn.workgroup.id.x intrinsic.
1405 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the
1406 llvm.amdgcn.workgroup.id.y intrinsic.
1408 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the
1409 llvm.amdgcn.workgroup.id.z intrinsic.
1411 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the
1412 llvm.amdgcn.dispatch.ptr intrinsic.
1414 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the
1415 llvm.amdgcn.implicitarg.ptr intrinsic.
1417 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the
1418 llvm.amdgcn.dispatch.id intrinsic.
1420 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the
1421 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint
1422 attributes, the queue pointer may be required in situations where the
1423 intrinsic call does not directly appear in the program. Some subtargets
1424 require the queue pointer for to handle some addrspacecasts, as well
1425 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and
1426 llvm.debug intrinsics.
1428 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit
1429 kernel argument that holds the pointer to the hostcall buffer. If this
1430 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.
1432 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit
1433 kernel argument that holds the pointer to an initialized memory buffer
1434 that conforms to the requirements of the malloc/free device library V1
1435 version implementation. If this attribute is absent, then the
1436 amdgpu-no-implicitarg-ptr is also removed.
1438 "amdgpu-no-multigrid-sync-arg" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit
1439 kernel argument that holds the multigrid synchronization pointer. If this
1440 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.
1442 "amdgpu-no-default-queue" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit
1443 kernel argument that holds the default queue pointer. If this
1444 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.
1446 "amdgpu-no-completion-action" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit
1447 kernel argument that holds the completion action pointer. If this
1448 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.
1450 "amdgpu-lds-size"="min[,max]" Min is the minimum number of bytes that will be allocated in the Local
1451 Data Store at address zero. Variables are allocated within this frame
1452 using absolute symbol metadata, primarily by the AMDGPULowerModuleLDS
1453 pass. Optional max is the maximum number of bytes that will be allocated.
1454 Note that min==max indicates that no further variables can be added to
1455 the frame. This is an internal detail of how LDS variables are lowered,
1456 language front ends should not set this attribute.
1458 "amdgpu-gds-size" Bytes expected to be allocated at the start of GDS memory at entry.
1460 "amdgpu-git-ptr-high" The hard-wired high half of the address of the global information table
1461 for AMDPAL OS type. 0xffffffff represents no hard-wired high half, since
1462 current hardware only allows a 16 bit value.
1464 "amdgpu-32bit-address-high-bits" Assumed high 32-bits for 32-bit address spaces which are really truncated
1465 64-bit addresses (i.e., addrspace(6))
1467 "amdgpu-color-export" Indicates shader exports color information if set to 1.
1468 Defaults to 1 for :ref:`amdgpu_ps <amdgpu-cc>`, and 0 for other calling
1469 conventions. Determines the necessity and type of null exports when a shader
1470 terminates early by killing lanes.
1472 "amdgpu-depth-export" Indicates shader exports depth information if set to 1. Determines the
1473 necessity and type of null exports when a shader terminates early by killing
1474 lanes. A depth-only shader will export to depth channel when no null export
1475 target is available (GFX11+).
1477 "InitialPSInputAddr" Set the initial value of the `spi_ps_input_addr` register for
1478 :ref:`amdgpu_ps <amdgpu-cc>` shaders. Any bits enabled by this value will
1479 be enabled in the final register value.
1481 "amdgpu-wave-priority-threshold" VALU instruction count threshold for adjusting wave priority. If exceeded,
1482 temporarily raise the wave priority at the start of the shader function
1483 until its last VMEM instructions to allow younger waves to issue their VMEM
1484 instructions as well.
1486 "amdgpu-memory-bound" Set internally by backend
1488 "amdgpu-wave-limiter" Set internally by backend
1490 "amdgpu-unroll-threshold" Set base cost threshold preference for loop unrolling within this function,
1491 default is 300. Actual threshold may be varied by per-loop metadata or
1492 reduced by heuristics.
1494 "amdgpu-max-num-workgroups"="x,y,z" Specify the maximum number of work groups for the kernel dispatch in the
1495 X, Y, and Z dimensions. Generated by the ``amdgpu_max_num_work_groups``
1496 CLANG attribute [CLANG-ATTR]_. Clang only emits this attribute when all
1497 the three numbers are >= 1.
1499 "amdgpu-no-agpr" Indicates the function will not require allocating AGPRs. This is only
1500 relevant on subtargets with AGPRs. The behavior is undefined if a
1501 function which requires AGPRs is reached through any function marked
1502 with this attribute.
1504 ======================================= ==========================================================
1506 Calling Conventions
1507 ===================
1509 The AMDGPU backend supports the following calling conventions:
1511 .. table:: AMDGPU Calling Conventions
1512 :name: amdgpu-cc
1514 =============================== ==========================================================
1515 Calling Convention Description
1516 =============================== ==========================================================
1517 ``ccc`` The C calling convention. Used by default.
1518 See :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions`
1519 for more details.
1521 ``fastcc`` The fast calling convention. Mostly the same as the ``ccc``.
1523 ``coldcc`` The cold calling convention. Mostly the same as the ``ccc``.
1525 ``amdgpu_cs`` Used for Mesa/AMDPAL compute shaders.
1526 ..TODO::
1527 Describe.
1529 ``amdgpu_cs_chain`` Similar to ``amdgpu_cs``, with differences described below.
1531 Functions with this calling convention cannot be called directly. They must
1532 instead be launched via the ``llvm.amdgcn.cs.chain`` intrinsic.
1534 Arguments are passed in SGPRs, starting at s0, if they have the ``inreg``
1535 attribute, and in VGPRs otherwise, starting at v8. Using more SGPRs or VGPRs
1536 than available in the subtarget is not allowed. On subtargets that use
1537 a scratch buffer descriptor (as opposed to ``scratch_{load,store}_*`` instructions),
1538 the scratch buffer descriptor is passed in s[48:51]. This limits the
1539 SGPR / ``inreg`` arguments to the equivalent of 48 dwords; using more
1540 than that is not allowed.
1542 The return type must be void.
1543 Varargs, sret, byval, byref, inalloca, preallocated are not supported.
1545 Values in scalar registers as well as v0-v7 are not preserved. Values in
1546 VGPRs starting at v8 are not preserved for the active lanes, but must be
1547 saved by the callee for inactive lanes when using WWM.
1549 Wave scratch is "empty" at function boundaries. There is no stack pointer input
1550 or output value, but functions are free to use scratch starting from an initial
1551 stack pointer. Calls to ``amdgpu_gfx`` functions are allowed and behave like they
1552 do in ``amdgpu_cs`` functions.
1554 All counters (``lgkmcnt``, ``vmcnt``, ``storecnt``, etc.) are presumed in an
1555 unknown state at function entry.
1557 A function may have multiple exits (e.g. one chain exit and one plain ``ret void``
1558 for when the wave ends), but all ``llvm.amdgcn.cs.chain`` exits must be in
1559 uniform control flow.
1561 ``amdgpu_cs_chain_preserve`` Same as ``amdgpu_cs_chain``, but active lanes for VGPRs starting at v8 are preserved.
1562 Calls to ``amdgpu_gfx`` functions are not allowed, and any calls to ``llvm.amdgcn.cs.chain``
1563 must not pass more VGPR arguments than the caller's VGPR function parameters.
1565 ``amdgpu_es`` Used for AMDPAL shader stage before geometry shader if geometry is in
1566 use. So either the domain (= tessellation evaluation) shader if
1567 tessellation is in use, or otherwise the vertex shader.
1568 ..TODO::
1569 Describe.
1571 ``amdgpu_gfx`` Used for AMD graphics targets. Functions with this calling convention
1572 cannot be used as entry points.
1573 ..TODO::
1574 Describe.
1576 ``amdgpu_gs`` Used for Mesa/AMDPAL geometry shaders.
1577 ..TODO::
1578 Describe.
1580 ``amdgpu_hs`` Used for Mesa/AMDPAL hull shaders (= tessellation control shaders).
1581 ..TODO::
1582 Describe.
1584 ``amdgpu_kernel`` See :ref:`amdgpu-amdhsa-function-call-convention-kernel-functions`
1586 ``amdgpu_ls`` Used for AMDPAL vertex shader if tessellation is in use.
1587 ..TODO::
1588 Describe.
1590 ``amdgpu_ps`` Used for Mesa/AMDPAL pixel shaders.
1591 ..TODO::
1592 Describe.
1594 ``amdgpu_vs`` Used for Mesa/AMDPAL last shader stage before rasterization (vertex
1595 shader if tessellation and geometry are not in use, or otherwise
1596 copy shader if one is needed).
1597 ..TODO::
1598 Describe.
1600 =============================== ==========================================================
1602 AMDGPU MCExpr
1603 -------------
1605 As part of the AMDGPU MC layer, AMDGPU provides the following target specific
1606 ``MCExpr``\s.
1608 .. table:: AMDGPU MCExpr types:
1609 :name: amdgpu-mcexpr-table
1611 =================== ================= ========================================================
1612 MCExpr Operands Return value
1613 =================== ================= ========================================================
1614 ``max(arg, ...)`` 1 or more Variadic signed operation that returns the maximum
1615 value of all its arguments.
1617 ``or(arg, ...)`` 1 or more Variadic signed operation that returns the bitwise-or
1618 result of all its arguments.
1620 =================== ================= ========================================================
1622 .. _amdgpu-elf-code-object:
1624 ELF Code Object
1625 ===============
1627 The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that
1628 can be linked by ``lld`` to produce a standard ELF shared code object which can
1629 be loaded and executed on an AMDGPU target.
1631 .. _amdgpu-elf-header:
1633 Header
1634 ------
1636 The AMDGPU backend uses the following ELF header:
1638 .. table:: AMDGPU ELF Header
1639 :name: amdgpu-elf-header-table
1641 ========================== ===============================
1642 Field Value
1643 ========================== ===============================
1644 ``e_ident[EI_CLASS]`` ``ELFCLASS64``
1645 ``e_ident[EI_DATA]`` ``ELFDATA2LSB``
1646 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE``
1647 - ``ELFOSABI_AMDGPU_HSA``
1648 - ``ELFOSABI_AMDGPU_PAL``
1649 - ``ELFOSABI_AMDGPU_MESA3D``
1650 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2``
1651 - ``ELFABIVERSION_AMDGPU_HSA_V3``
1652 - ``ELFABIVERSION_AMDGPU_HSA_V4``
1653 - ``ELFABIVERSION_AMDGPU_HSA_V5``
1654 - ``ELFABIVERSION_AMDGPU_HSA_V6``
1655 - ``ELFABIVERSION_AMDGPU_PAL``
1656 - ``ELFABIVERSION_AMDGPU_MESA3D``
1657 ``e_type`` - ``ET_REL``
1658 - ``ET_DYN``
1659 ``e_machine`` ``EM_AMDGPU``
1660 ``e_entry`` 0
1661 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`,
1662 :ref:`amdgpu-elf-header-e_flags-table-v3`,
1663 :ref:`amdgpu-elf-header-e_flags-table-v4-v5`,
1664 and :ref:`amdgpu-elf-header-e_flags-table-v6-onwards`
1665 ========================== ===============================
1667 ..
1669 .. table:: AMDGPU ELF Header Enumeration Values
1670 :name: amdgpu-elf-header-enumeration-values-table
1672 =============================== =====
1673 Name Value
1674 =============================== =====
1675 ``EM_AMDGPU`` 224
1676 ``ELFOSABI_NONE`` 0
1677 ``ELFOSABI_AMDGPU_HSA`` 64
1678 ``ELFOSABI_AMDGPU_PAL`` 65
1679 ``ELFOSABI_AMDGPU_MESA3D`` 66
1680 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0
1681 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1
1682 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2
1683 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3
1684 ``ELFABIVERSION_AMDGPU_HSA_V6`` 4
1685 ``ELFABIVERSION_AMDGPU_PAL`` 0
1686 ``ELFABIVERSION_AMDGPU_MESA3D`` 0
1687 =============================== =====
1689 ``e_ident[EI_CLASS]``
1690 The ELF class is:
1692 * ``ELFCLASS32`` for ``r600`` architecture.
1694 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit
1695 process address space applications.
1697 ``e_ident[EI_DATA]``
1698 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering.
1700 ``e_ident[EI_OSABI]``
1701 One of the following AMDGPU target architecture specific OS ABIs
1702 (see :ref:`amdgpu-os`):
1704 * ``ELFOSABI_NONE`` for *unknown* OS.
1706 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS.
1708 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS.
1710 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS.
1712 ``e_ident[EI_ABIVERSION]``
1713 The ABI version of the AMDGPU target architecture specific OS ABI to which the code
1714 object conforms:
1716 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA
1717 runtime ABI for code object V2. Can no longer be emitted by this version of LLVM.
1719 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA
1720 runtime ABI for code object V3. Can no longer be emitted by this version of LLVM.
1722 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA
1723 runtime ABI for code object V4. Specify using the Clang option
1724 ``-mcode-object-version=4``.
1726 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA
1727 runtime ABI for code object V5. Specify using the Clang option
1728 ``-mcode-object-version=5``. This is the default code object
1729 version if not specified.
1731 * ``ELFABIVERSION_AMDGPU_HSA_V6`` is used to specify the version of AMD HSA
1732 runtime ABI for code object V6. Specify using the Clang option
1733 ``-mcode-object-version=6``.
1735 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL
1736 runtime ABI.
1738 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA
1739 3D runtime ABI.
1741 ``e_type``
1742 Can be one of the following values:
1745 ``ET_REL``
1746 The type produced by the AMDGPU backend compiler as it is relocatable code
1747 object.
1749 ``ET_DYN``
1750 The type produced by the linker as it is a shared code object.
1752 The AMD HSA runtime loader requires a ``ET_DYN`` code object.
1754 ``e_machine``
1755 The value ``EM_AMDGPU`` is used for the machine for all processors supported
1756 by the ``r600`` and ``amdgcn`` architectures (see
1757 :ref:`amdgpu-processor-table`). The specific processor is specified in the
1758 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see
1759 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the
1760 ``e_flags`` for code object V3 and above (see
1761 :ref:`amdgpu-elf-header-e_flags-table-v3`,
1762 :ref:`amdgpu-elf-header-e_flags-table-v4-v5` and
1763 :ref:`amdgpu-elf-header-e_flags-table-v6-onwards`).
1765 ``e_entry``
1766 The entry point is 0 as the entry points for individual kernels must be
1767 selected in order to invoke them through AQL packets.
1769 ``e_flags``
1770 The AMDGPU backend uses the following ELF header flags:
1772 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2
1773 :name: amdgpu-elf-header-e_flags-v2-table
1775 ===================================== ===== =============================
1776 Name Value Description
1777 ===================================== ===== =============================
1778 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack``
1779 target feature is
1780 enabled for all code
1781 contained in the code object.
1782 If the processor
1783 does not support the
1784 ``xnack`` target
1785 feature then must
1786 be 0.
1787 See
1788 :ref:`amdgpu-target-features`.
1789 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap
1790 handler is enabled for all
1791 code contained in the code
1792 object. If the processor
1793 does not support a trap
1794 handler then must be 0.
1795 See
1796 :ref:`amdgpu-target-features`.
1797 ===================================== ===== =============================
1799 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3
1800 :name: amdgpu-elf-header-e_flags-table-v3
1802 ================================= ===== =============================
1803 Name Value Description
1804 ================================= ===== =============================
1805 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection
1806 mask for
1807 ``EF_AMDGPU_MACH_xxx`` values
1808 defined in
1809 :ref:`amdgpu-ef-amdgpu-mach-table`.
1810 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack``
1811 target feature is
1812 enabled for all code
1813 contained in the code object.
1814 If the processor
1815 does not support the
1816 ``xnack`` target
1817 feature then must
1818 be 0.
1819 See
1820 :ref:`amdgpu-target-features`.
1821 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc``
1822 target feature is
1823 enabled for all code
1824 contained in the code object.
1825 If the processor
1826 does not support the
1827 ``sramecc`` target
1828 feature then must
1829 be 0.
1830 See
1831 :ref:`amdgpu-target-features`.
1832 ================================= ===== =============================
1834 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and V5
1835 :name: amdgpu-elf-header-e_flags-table-v4-v5
1837 ============================================ ===== ===================================
1838 Name Value Description
1839 ============================================ ===== ===================================
1840 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection
1841 mask for
1842 ``EF_AMDGPU_MACH_xxx`` values
1843 defined in
1844 :ref:`amdgpu-ef-amdgpu-mach-table`.
1845 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for
1846 ``EF_AMDGPU_FEATURE_XNACK_*_V4``
1847 values.
1848 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsupported.
1849 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value.
1850 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled.
1851 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled.
1852 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for
1853 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4``
1854 values.
1855 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsupported.
1856 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value.
1857 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled,
1858 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled.
1859 ============================================ ===== ===================================
1861 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V6 and After
1862 :name: amdgpu-elf-header-e_flags-table-v6-onwards
1864 ============================================ ========== =========================================
1865 Name Value Description
1866 ============================================ ========== =========================================
1867 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection
1868 mask for
1869 ``EF_AMDGPU_MACH_xxx`` values
1870 defined in
1871 :ref:`amdgpu-ef-amdgpu-mach-table`.
1872 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for
1873 ``EF_AMDGPU_FEATURE_XNACK_*_V4``
1874 values.
1875 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsupported.
1876 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value.
1877 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled.
1878 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled.
1879 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for
1880 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4``
1881 values.
1882 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsupported.
1883 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value.
1884 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled,
1885 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled.
1886 ``EF_AMDGPU_GENERIC_VERSION_V`` 0xff000000 Generic code object version selection
1887 mask. This is a value between 1 and 255,
1888 stored in the most significant byte
1889 of EFLAGS.
1890 See :ref:`amdgpu-generic-processor-versioning`
1891 ============================================ ========== =========================================
1893 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values
1894 :name: amdgpu-ef-amdgpu-mach-table
1896 ========================================== ========== =============================
1897 Name Value Description (see
1898 :ref:`amdgpu-processor-table`)
1899 ========================================== ========== =============================
1900 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified*
1901 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600``
1902 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630``
1903 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880``
1904 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670``
1905 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710``
1906 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730``
1907 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770``
1908 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar``
1909 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress``
1910 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper``
1911 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood``
1912 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo``
1913 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts``
1914 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos``
1915 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman``
1916 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks``
1917 *reserved* 0x011 - Reserved for ``r600``
1918 0x01f architecture processors.
1919 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600``
1920 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601``
1921 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700``
1922 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701``
1923 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702``
1924 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703``
1925 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704``
1926 *reserved* 0x027 Reserved.
1927 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801``
1928 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802``
1929 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803``
1930 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810``
1931 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900``
1932 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902``
1933 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904``
1934 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906``
1935 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908``
1936 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909``
1937 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c``
1938 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010``
1939 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011``
1940 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012``
1941 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030``
1942 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031``
1943 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032``
1944 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033``
1945 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602``
1946 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705``
1947 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805``
1948 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035``
1949 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034``
1950 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a``
1951 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940``
1952 ``EF_AMDGPU_MACH_AMDGCN_GFX1100`` 0x041 ``gfx1100``
1953 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013``
1954 ``EF_AMDGPU_MACH_AMDGCN_GFX1150`` 0x043 ``gfx1150``
1955 ``EF_AMDGPU_MACH_AMDGCN_GFX1103`` 0x044 ``gfx1103``
1956 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036``
1957 ``EF_AMDGPU_MACH_AMDGCN_GFX1101`` 0x046 ``gfx1101``
1958 ``EF_AMDGPU_MACH_AMDGCN_GFX1102`` 0x047 ``gfx1102``
1959 ``EF_AMDGPU_MACH_AMDGCN_GFX1200`` 0x048 ``gfx1200``
1960 *reserved* 0x049 Reserved.
1961 ``EF_AMDGPU_MACH_AMDGCN_GFX1151`` 0x04a ``gfx1151``
1962 ``EF_AMDGPU_MACH_AMDGCN_GFX941`` 0x04b ``gfx941``
1963 ``EF_AMDGPU_MACH_AMDGCN_GFX942`` 0x04c ``gfx942``
1964 *reserved* 0x04d Reserved.
1965 ``EF_AMDGPU_MACH_AMDGCN_GFX1201`` 0x04e ``gfx1201``
1966 *reserved* 0x04f Reserved.
1967 *reserved* 0x050 Reserved.
1968 ``EF_AMDGPU_MACH_AMDGCN_GFX9_GENERIC`` 0x051 ``gfx9-generic``
1969 ``EF_AMDGPU_MACH_AMDGCN_GFX10_1_GENERIC`` 0x052 ``gfx10-1-generic``
1970 ``EF_AMDGPU_MACH_AMDGCN_GFX10_3_GENERIC`` 0x053 ``gfx10-3-generic``
1971 ``EF_AMDGPU_MACH_AMDGCN_GFX11_GENERIC`` 0x054 ``gfx11-generic``
1972 *reserved* 0x055 Reserved.
1973 ========================================== ========== =============================
1975 Sections
1976 --------
1978 An AMDGPU target ELF code object has the standard ELF sections which include:
1980 .. table:: AMDGPU ELF Sections
1981 :name: amdgpu-elf-sections-table
1983 ================== ================ =================================
1984 Name Type Attributes
1985 ================== ================ =================================
1986 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE``
1987 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE``
1988 ``.debug_``\ *\** ``SHT_PROGBITS`` *none*
1989 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC``
1990 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC``
1991 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC``
1992 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE``
1993 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC``
1994 ``.note`` ``SHT_NOTE`` *none*
1995 ``.rela``\ *name* ``SHT_RELA`` *none*
1996 ``.rela.dyn`` ``SHT_RELA`` *none*
1997 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC``
1998 ``.shstrtab`` ``SHT_STRTAB`` *none*
1999 ``.strtab`` ``SHT_STRTAB`` *none*
2000 ``.symtab`` ``SHT_SYMTAB`` *none*
2001 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR``
2002 ================== ================ =================================
2004 These sections have their standard meanings (see [ELF]_) and are only generated
2005 if needed.
2007 ``.debug``\ *\**
2008 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for
2009 information on the DWARF produced by the AMDGPU backend.
2011 ``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash``
2012 The standard sections used by a dynamic loader.
2014 ``.note``
2015 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU
2016 backend.
2018 ``.rela``\ *name*, ``.rela.dyn``
2019 For relocatable code objects, *name* is the name of the section that the
2020 relocation records apply. For example, ``.rela.text`` is the section name for
2021 relocation records associated with the ``.text`` section.
2023 For linked shared code objects, ``.rela.dyn`` contains all the relocation
2024 records from each of the relocatable code object's ``.rela``\ *name* sections.
2026 See :ref:`amdgpu-relocation-records` for the relocation records supported by
2027 the AMDGPU backend.
2029 ``.text``
2030 The executable machine code for the kernels and functions they call. Generated
2031 as position independent code. See :ref:`amdgpu-code-conventions` for
2032 information on conventions used in the isa generation.
2034 .. _amdgpu-note-records:
2036 Note Records
2037 ------------
2039 The AMDGPU backend code object contains ELF note records in the ``.note``
2040 section. The set of generated notes and their semantics depend on the code
2041 object version; see :ref:`amdgpu-note-records-v2` and
2042 :ref:`amdgpu-note-records-v3-onwards`.
2044 As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding
2045 must be generated after the ``name`` field to ensure the ``desc`` field is 4
2046 byte aligned. In addition, minimal zero-byte padding must be generated to
2047 ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign``
2048 field of the ``.note`` section must be at least 4 to indicate at least 8 byte
2049 alignment.
2051 .. _amdgpu-note-records-v2:
2053 Code Object V2 Note Records
2054 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
2056 .. warning::
2057 Code object V2 generation is no longer supported by this version of LLVM.
2059 The AMDGPU backend code object uses the following ELF note record in the
2060 ``.note`` section when compiling for code object V2.
2062 The note record vendor field is "AMD".
2064 Additional note records may be present, but any which are not documented here
2065 are deprecated and should not be used.
2067 .. table:: AMDGPU Code Object V2 ELF Note Records
2068 :name: amdgpu-elf-note-records-v2-table
2070 ===== ===================================== ======================================
2071 Name Type Description
2072 ===== ===================================== ======================================
2073 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version.
2074 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL
2075 Finalizer and not the LLVM compiler.
2076 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version.
2077 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in
2078 YAML [YAML]_ textual format.
2079 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name.
2080 ===== ===================================== ======================================
2082 ..
2084 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values
2085 :name: amdgpu-elf-note-record-enumeration-values-v2-table
2087 ===================================== =====
2088 Name Value
2089 ===================================== =====
2090 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1
2091 ``NT_AMD_HSA_HSAIL`` 2
2092 ``NT_AMD_HSA_ISA_VERSION`` 3
2093 *reserved* 4-9
2094 ``NT_AMD_HSA_METADATA`` 10
2095 ``NT_AMD_HSA_ISA_NAME`` 11
2096 ===================================== =====
2098 ``NT_AMD_HSA_CODE_OBJECT_VERSION``
2099 Specifies the code object version number. The description field has the
2100 following layout:
2102 .. code:: c
2104 struct amdgpu_hsa_note_code_object_version_s {
2105 uint32_t major_version;
2106 uint32_t minor_version;
2107 };
2109 The ``major_version`` has a value less than or equal to 2.
2111 ``NT_AMD_HSA_HSAIL``
2112 Specifies the HSAIL properties used by the HSAIL Finalizer. The description
2113 field has the following layout:
2115 .. code:: c
2117 struct amdgpu_hsa_note_hsail_s {
2118 uint32_t hsail_major_version;
2119 uint32_t hsail_minor_version;
2120 uint8_t profile;
2121 uint8_t machine_model;
2122 uint8_t default_float_round;
2123 };
2125 ``NT_AMD_HSA_ISA_VERSION``
2126 Specifies the target ISA version. The description field has the following layout:
2128 .. code:: c
2130 struct amdgpu_hsa_note_isa_s {
2131 uint16_t vendor_name_size;
2132 uint16_t architecture_name_size;
2133 uint32_t major;
2134 uint32_t minor;
2135 uint32_t stepping;
2136 char vendor_and_architecture_name[1];
2137 };
2139 ``vendor_name_size`` and ``architecture_name_size`` are the length of the
2140 vendor and architecture names respectively, including the NUL character.
2142 ``vendor_and_architecture_name`` contains the NUL terminates string for the
2143 vendor, immediately followed by the NUL terminated string for the
2144 architecture.
2146 This note record is used by the HSA runtime loader.
2148 Code object V2 only supports a limited number of processors and has fixed
2149 settings for target features. See
2150 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of
2151 processors and the corresponding target ID. In the table the note record ISA
2152 name is a concatenation of the vendor name, architecture name, major, minor,
2153 and stepping separated by a ":".
2155 The target ID column shows the processor name and fixed target features used
2156 by the LLVM compiler. The LLVM compiler does not generate a
2157 ``NT_AMD_HSA_HSAIL`` note record.
2159 A code object generated by the Finalizer also uses code object V2 and always
2160 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and
2161 ``sramecc`` target feature is as shown in
2162 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack``
2163 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags``
2164 bit.
2166 ``NT_AMD_HSA_ISA_NAME``
2167 Specifies the target ISA name as a non-NUL terminated string.
2169 This note record is not used by the HSA runtime loader.
2171 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object
2172 V2's limited support of processors and fixed settings for target features.
2174 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping
2175 from the string to the corresponding target ID. If the ``xnack`` target
2176 feature is supported and enabled, the string produced by the LLVM compiler
2177 will may have a ``+xnack`` appended. The Finlizer did not do the appending and
2178 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit.
2180 ``NT_AMD_HSA_METADATA``
2181 Specifies extensible metadata associated with the code objects executed on HSA
2182 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the
2183 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See
2184 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object
2185 metadata string.
2187 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings
2188 :name: amdgpu-elf-note-record-supported_processors-v2-table
2190 ===================== ==========================
2191 Note Record ISA Name Target ID
2192 ===================== ==========================
2193 ``AMD:AMDGPU:6:0:0`` ``gfx600``
2194 ``AMD:AMDGPU:6:0:1`` ``gfx601``
2195 ``AMD:AMDGPU:6:0:2`` ``gfx602``
2196 ``AMD:AMDGPU:7:0:0`` ``gfx700``
2197 ``AMD:AMDGPU:7:0:1`` ``gfx701``
2198 ``AMD:AMDGPU:7:0:2`` ``gfx702``
2199 ``AMD:AMDGPU:7:0:3`` ``gfx703``
2200 ``AMD:AMDGPU:7:0:4`` ``gfx704``
2201 ``AMD:AMDGPU:7:0:5`` ``gfx705``
2202 ``AMD:AMDGPU:8:0:0`` ``gfx802``
2203 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+``
2204 ``AMD:AMDGPU:8:0:2`` ``gfx802``
2205 ``AMD:AMDGPU:8:0:3`` ``gfx803``
2206 ``AMD:AMDGPU:8:0:4`` ``gfx803``
2207 ``AMD:AMDGPU:8:0:5`` ``gfx805``
2208 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+``
2209 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-``
2210 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+``
2211 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-``
2212 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+``
2213 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-``
2214 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+``
2215 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-``
2216 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+``
2217 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-``
2218 ===================== ==========================
2220 .. _amdgpu-note-records-v3-onwards:
2222 Code Object V3 and Above Note Records
2223 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2225 The AMDGPU backend code object uses the following ELF note record in the
2226 ``.note`` section when compiling for code object V3 and above.
2228 The note record vendor field is "AMDGPU".
2230 Additional note records may be present, but any which are not documented here
2231 are deprecated and should not be used.
2233 .. table:: AMDGPU Code Object V3 and Above ELF Note Records
2234 :name: amdgpu-elf-note-records-table-v3-onwards
2236 ======== ============================== ======================================
2237 Name Type Description
2238 ======== ============================== ======================================
2239 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_
2240 binary format.
2241 ======== ============================== ======================================
2243 ..
2245 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values
2246 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards
2248 ============================== =====
2249 Name Value
2250 ============================== =====
2251 *reserved* 0-31
2252 ``NT_AMDGPU_METADATA`` 32
2253 ============================== =====
2255 ``NT_AMDGPU_METADATA``
2256 Specifies extensible metadata associated with an AMDGPU code object. It is
2257 encoded as a map in the Message Pack [MsgPack]_ binary data format. See
2258 :ref:`amdgpu-amdhsa-code-object-metadata-v3`,
2259 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and
2260 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the
2261 ``amdhsa`` OS.
2263 .. _amdgpu-symbols:
2265 Symbols
2266 -------
2268 Symbols include the following:
2270 .. table:: AMDGPU ELF Symbols
2271 :name: amdgpu-elf-symbols-table
2273 ===================== ================== ================ ==================
2274 Name Type Section Description
2275 ===================== ================== ================ ==================
2276 *link-name* ``STT_OBJECT`` - ``.data`` Global variable
2277 - ``.rodata``
2278 - ``.bss``
2279 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor
2280 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point
2281 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS
2282 ===================== ================== ================ ==================
2284 Global variable
2285 Global variables both used and defined by the compilation unit.
2287 If the symbol is defined in the compilation unit then it is allocated in the
2288 appropriate section according to if it has initialized data or is readonly.
2290 If the symbol is external then its section is ``STN_UNDEF`` and the loader
2291 will resolve relocations using the definition provided by another code object
2292 or explicitly defined by the runtime.
2294 If the symbol resides in local/group memory (LDS) then its section is the
2295 special processor specific section name ``SHN_AMDGPU_LDS``, and the
2296 ``st_value`` field describes alignment requirements as it does for common
2297 symbols.
2299 .. TODO::
2301 Add description of linked shared object symbols. Seems undefined symbols
2302 are marked as STT_NOTYPE.
2304 Kernel descriptor
2305 Every HSA kernel has an associated kernel descriptor. It is the address of the
2306 kernel descriptor that is used in the AQL dispatch packet used to invoke the
2307 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is
2308 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`.
2310 Kernel entry point
2311 Every HSA kernel also has a symbol for its machine code entry point.
2313 .. _amdgpu-relocation-records:
2315 Relocation Records
2316 ------------------
2318 The AMDGPU backend generates ``Elf64_Rela`` relocation records for
2319 AMDHSA or ``Elf64_Rel`` relocation records for Mesa/AMDPAL. Supported
2320 relocatable fields are:
2322 ``word32``
2323 This specifies a 32-bit field occupying 4 bytes with arbitrary byte
2324 alignment. These values use the same byte order as other word values in the
2325 AMDGPU architecture.
2327 ``word64``
2328 This specifies a 64-bit field occupying 8 bytes with arbitrary byte
2329 alignment. These values use the same byte order as other word values in the
2330 AMDGPU architecture.
2332 Following notations are used for specifying relocation calculations:
2334 **A**
2335 Represents the addend used to compute the value of the relocatable field. If
2336 the addend field is smaller than 64 bits then it is zero-extended to 64 bits
2337 for use in the calculations below. (In practice this only affects ``_HI``
2338 relocation types on Mesa/AMDPAL, where the addend comes from the 32-bit field
2339 but the result of the calculation depends on the high part of the full 64-bit
2340 address.)
2342 **G**
2343 Represents the offset into the global offset table at which the relocation
2344 entry's symbol will reside during execution.
2346 **GOT**
2347 Represents the address of the global offset table.
2349 **P**
2350 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``)
2351 of the storage unit being relocated (computed using ``r_offset``).
2353 **S**
2354 Represents the value of the symbol whose index resides in the relocation
2355 entry. Relocations not using this must specify a symbol index of
2356 ``STN_UNDEF``.
2358 **B**
2359 Represents the base address of a loaded executable or shared object which is
2360 the difference between the ELF address and the actual load address.
2361 Relocations using this are only valid in executable or shared objects.
2363 The following relocation types are supported:
2365 .. table:: AMDGPU ELF Relocation Records
2366 :name: amdgpu-elf-relocation-records-table
2368 ========================== ======= ===== ========== ==============================
2369 Relocation Type Kind Value Field Calculation
2370 ========================== ======= ===== ========== ==============================
2371 ``R_AMDGPU_NONE`` 0 *none* *none*
2372 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF
2373 Dynamic
2374 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32
2375 Dynamic
2376 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A
2377 Dynamic
2378 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P
2379 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P
2380 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A
2381 Dynamic
2382 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P
2383 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF
2384 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32
2385 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF
2386 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32
2387 *reserved* 12
2388 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A
2389 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4
2390 ========================== ======= ===== ========== ==============================
2392 ``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by
2393 the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``.
2395 There is no current OS loader support for 32-bit programs and so
2396 ``R_AMDGPU_ABS32`` is not used.
2398 .. _amdgpu-loaded-code-object-path-uniform-resource-identifier:
2400 Loaded Code Object Path Uniform Resource Identifier (URI)
2401 ---------------------------------------------------------
2403 The AMD GPU code object loader represents the path of the ELF shared object from
2404 which the code object was loaded as a textual Uniform Resource Identifier (URI).
2405 Note that the code object is the in memory loaded relocated form of the ELF
2406 shared object. Multiple code objects may be loaded at different memory
2407 addresses in the same process from the same ELF shared object.
2409 The loaded code object path URI syntax is defined by the following BNF syntax:
2411 .. code::
2413 code_object_uri ::== file_uri | memory_uri
2414 file_uri ::== "file://" file_path [ range_specifier ]
2415 memory_uri ::== "memory://" process_id range_specifier
2416 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number
2417 file_path ::== URI_ENCODED_OS_FILE_PATH
2418 process_id ::== DECIMAL_NUMBER
2419 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER
2421 **number**
2422 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X",
2423 and octal values by "0".
2425 **file_path**
2426 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding,
2427 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is
2428 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in
2429 the path are separated by "/".
2431 **offset**
2432 Is a 0-based byte offset to the start of the code object. For a file URI, it
2433 is from the start of the file specified by the ``file_path``, and if omitted
2434 defaults to 0. For a memory URI, it is the memory address and is required.
2436 **size**
2437 Is the number of bytes in the code object. For a file URI, if omitted it
2438 defaults to the size of the file. It is required for a memory URI.
2440 **process_id**
2441 Is the identity of the process owning the memory. For Linux it is the C
2442 unsigned integral decimal literal for the process ID (PID).
2444 For example:
2446 .. code::
2448 file:///dir1/dir2/file1
2449 file:///dir3/dir4/file2#offset=0x2000&size=3000
2450 memory://1234#offset=0x20000&size=3000
2452 .. _amdgpu-dwarf-debug-information:
2454 DWARF Debug Information
2455 =======================
2457 .. warning::
2459 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that
2460 is not currently fully implemented and is subject to change.
2462 AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see
2463 :ref:`amdgpu-elf-code-object`) which contain information that maps the code
2464 object executable code and data to the source language constructs. It can be
2465 used by tools such as debuggers and profilers. It uses features defined in
2466 :doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in
2467 DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension.
2469 This section defines the AMDGPU target architecture specific DWARF mappings.
2471 .. _amdgpu-dwarf-register-identifier:
2473 Register Identifier
2474 -------------------
2476 This section defines the AMDGPU target architecture register numbers used in
2477 DWARF operation expressions (see DWARF Version 5 section 2.5 and
2478 :ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information
2479 instructions (see DWARF Version 5 section 6.4 and
2480 :ref:`amdgpu-dwarf-call-frame-information`).
2482 A single code object can contain code for kernels that have different wavefront
2483 sizes. The vector registers and some scalar registers are based on the wavefront
2484 size. AMDGPU defines distinct DWARF registers for each wavefront size. This
2485 simplifies the consumer of the DWARF so that each register has a fixed size,
2486 rather than being dynamic according to the wavefront size mode. Similarly,
2487 distinct DWARF registers are defined for those registers that vary in size
2488 according to the process address size. This allows a consumer to treat a
2489 specific AMDGPU processor as a single architecture regardless of how it is
2490 configured at run time. The compiler explicitly specifies the DWARF registers
2491 that match the mode in which the code it is generating will be executed.
2493 DWARF registers are encoded as numbers, which are mapped to architecture
2494 registers. The mapping for AMDGPU is defined in
2495 :ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same
2496 mapping.
2498 .. table:: AMDGPU DWARF Register Mapping
2499 :name: amdgpu-dwarf-register-mapping-table
2501 ============== ================= ======== ==================================
2502 DWARF Register AMDGPU Register Bit Size Description
2503 ============== ================= ======== ==================================
2504 0 PC_32 32 Program Counter (PC) when
2505 executing in a 32-bit process
2506 address space. Used in the CFI to
2507 describe the PC of the calling
2508 frame.
2509 1 EXEC_MASK_32 32 Execution Mask Register when
2510 executing in wavefront 32 mode.
2511 2-15 *Reserved* *Reserved for highly accessed
2512 registers using DWARF shortcut.*
2513 16 PC_64 64 Program Counter (PC) when
2514 executing in a 64-bit process
2515 address space. Used in the CFI to
2516 describe the PC of the calling
2517 frame.
2518 17 EXEC_MASK_64 64 Execution Mask Register when
2519 executing in wavefront 64 mode.
2520 18-31 *Reserved* *Reserved for highly accessed
2521 registers using DWARF shortcut.*
2522 32-95 SGPR0-SGPR63 32 Scalar General Purpose
2523 Registers.
2524 96-127 *Reserved* *Reserved for frequently accessed
2525 registers using DWARF 1-byte ULEB.*
2526 128 STATUS 32 Status Register.
2527 129-511 *Reserved* *Reserved for future Scalar
2528 Architectural Registers.*
2529 512 VCC_32 32 Vector Condition Code Register
2530 when executing in wavefront 32
2531 mode.
2532 513-767 *Reserved* *Reserved for future Vector
2533 Architectural Registers when
2534 executing in wavefront 32 mode.*
2535 768 VCC_64 64 Vector Condition Code Register
2536 when executing in wavefront 64
2537 mode.
2538 769-1023 *Reserved* *Reserved for future Vector
2539 Architectural Registers when
2540 executing in wavefront 64 mode.*
2541 1024-1087 *Reserved* *Reserved for padding.*
2542 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers.
2543 1130-1535 *Reserved* *Reserved for future Scalar
2544 General Purpose Registers.*
2545 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers
2546 when executing in wavefront 32
2547 mode.
2548 1792-2047 *Reserved* *Reserved for future Vector
2549 General Purpose Registers when
2550 executing in wavefront 32 mode.*
2551 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers
2552 when executing in wavefront 32
2553 mode.
2554 2304-2559 *Reserved* *Reserved for future Vector
2555 Accumulation Registers when
2556 executing in wavefront 32 mode.*
2557 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers
2558 when executing in wavefront 64
2559 mode.
2560 2816-3071 *Reserved* *Reserved for future Vector
2561 General Purpose Registers when
2562 executing in wavefront 64 mode.*
2563 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers
2564 when executing in wavefront 64
2565 mode.
2566 3328-3583 *Reserved* *Reserved for future Vector
2567 Accumulation Registers when
2568 executing in wavefront 64 mode.*
2569 ============== ================= ======== ==================================
2571 The vector registers are represented as the full size for the wavefront. They
2572 are organized as consecutive dwords (32-bits), one per lane, with the dword at
2573 the least significant bit position corresponding to lane 0 and so forth. DWARF
2574 location expressions involving the ``DW_OP_LLVM_offset`` and
2575 ``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector
2576 register corresponding to the lane that is executing the current thread of
2577 execution in languages that are implemented using a SIMD or SIMT execution
2578 model.
2580 If the wavefront size is 32 lanes then the wavefront 32 mode register
2581 definitions are used. If the wavefront size is 64 lanes then the wavefront 64
2582 mode register definitions are used. Some AMDGPU targets support executing in
2583 both wavefront 32 and wavefront 64 mode. The register definitions corresponding
2584 to the wavefront mode of the generated code will be used.
2586 If code is generated to execute in a 32-bit process address space, then the
2587 32-bit process address space register definitions are used. If code is generated
2588 to execute in a 64-bit process address space, then the 64-bit process address
2589 space register definitions are used. The ``amdgcn`` target only supports the
2590 64-bit process address space.
2592 .. _amdgpu-dwarf-memory-space-identifier:
2594 Memory Space Identifier
2595 -----------------------
2597 The DWARF memory space represents the source language memory space. See DWARF
2598 Version 5 section 2.12 which is updated by the *DWARF Extensions For
2599 Heterogeneous Debugging* section :ref:`amdgpu-dwarf-memory-spaces`.
2601 The DWARF memory space mapping used for AMDGPU is defined in
2602 :ref:`amdgpu-dwarf-memory-space-mapping-table`.
2604 .. table:: AMDGPU DWARF Memory Space Mapping
2605 :name: amdgpu-dwarf-memory-space-mapping-table
2607 =========================== ====== =================
2608 DWARF AMDGPU
2609 ---------------------------------- -----------------
2610 Memory Space Name Value Memory Space
2611 =========================== ====== =================
2612 ``DW_MSPACE_LLVM_none`` 0x0000 Generic (Flat)
2613 ``DW_MSPACE_LLVM_global`` 0x0001 Global
2614 ``DW_MSPACE_LLVM_constant`` 0x0002 Global
2615 ``DW_MSPACE_LLVM_group`` 0x0003 Local (group/LDS)
2616 ``DW_MSPACE_LLVM_private`` 0x0004 Private (Scratch)
2617 ``DW_MSPACE_AMDGPU_region`` 0x8000 Region (GDS)
2618 =========================== ====== =================
2620 The DWARF memory space values defined in the *DWARF Extensions For Heterogeneous
2621 Debugging* section :ref:`amdgpu-dwarf-memory-spaces` are used.
2623 In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is
2624 available for use for the AMD extension for access to the hardware GDS memory
2625 which is scratchpad memory allocated per device.
2627 For AMDGPU if no ``DW_AT_LLVM_memory_space`` attribute is present, then the
2628 default memory space of ``DW_MSPACE_LLVM_none`` is used.
2630 See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU
2631 mapping of DWARF memory spaces to DWARF address spaces, including address size
2632 and NULL value.
2634 .. _amdgpu-dwarf-address-space-identifier:
2636 Address Space Identifier
2637 ------------------------
2639 DWARF address spaces correspond to target architecture specific linear
2640 addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions
2641 For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-address-spaces`.
2643 The DWARF address space mapping used for AMDGPU is defined in
2644 :ref:`amdgpu-dwarf-address-space-mapping-table`.
2646 .. table:: AMDGPU DWARF Address Space Mapping
2647 :name: amdgpu-dwarf-address-space-mapping-table
2649 ======================================= ===== ======= ======== ===================== =======================
2650 DWARF AMDGPU Notes
2651 --------------------------------------- ----- ---------------- --------------------- -----------------------
2652 Address Space Name Value Address Bit Size LLVM IR Address Space
2653 --------------------------------------- ----- ------- -------- --------------------- -----------------------
2654 .. 64-bit 32-bit
2655 process process
2656 address address
2657 space space
2658 ======================================= ===== ======= ======== ===================== =======================
2659 ``DW_ASPACE_LLVM_none`` 0x00 64 32 Global *default address space*
2660 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat)
2661 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS)
2662 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS)
2663 *Reserved* 0x04
2664 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane*
2665 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront*
2666 ======================================= ===== ======= ======== ===================== =======================
2668 See :ref:`amdgpu-address-spaces` for information on the AMDGPU LLVM IR address
2669 spaces including address size and NULL value.
2671 The ``DW_ASPACE_LLVM_none`` address space is the default target architecture
2672 address space used in DWARF operations that do not specify an address space. It
2673 therefore has to map to the global address space so that the ``DW_OP_addr*`` and
2674 related operations can refer to addresses in the program code.
2676 The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to
2677 specify the flat address space. If the address corresponds to an address in the
2678 local address space, then it corresponds to the wavefront that is executing the
2679 focused thread of execution. If the address corresponds to an address in the
2680 private address space, then it corresponds to the lane that is executing the
2681 focused thread of execution for languages that are implemented using a SIMD or
2682 SIMT execution model.
2684 .. note::
2686 CUDA-like languages such as HIP that do not have address spaces in the
2687 language type system, but do allow variables to be allocated in different
2688 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic``
2689 address space in the DWARF expression operations as the default address space
2690 is the global address space.
2692 The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to
2693 specify the local address space corresponding to the wavefront that is executing
2694 the focused thread of execution.
2696 The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions
2697 to specify the private address space corresponding to the lane that is executing
2698 the focused thread of execution for languages that are implemented using a SIMD
2699 or SIMT execution model.
2701 The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions
2702 to specify the unswizzled private address space corresponding to the wavefront
2703 that is executing the focused thread of execution. The wavefront view of private
2704 memory is the per wavefront unswizzled backing memory layout defined in
2705 :ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first
2706 location for the backing memory of the wavefront (namely the address is not
2707 offset by ``wavefront-scratch-base``). The following formula can be used to
2708 convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a
2709 ``DW_ASPACE_AMDGPU_private_wave`` address:
2711 ::
2713 private-address-wavefront =
2714 ((private-address-lane / 4) * wavefront-size * 4) +
2715 (wavefront-lane-id * 4) + (private-address-lane % 4)
2717 If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start
2718 of the dwords for each lane starting with lane 0 is required, then this
2719 simplifies to:
2721 ::
2723 private-address-wavefront =
2724 private-address-lane * wavefront-size
2726 A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a
2727 complete spilled vector register back into a complete vector register in the
2728 CFI. The frame pointer can be a private lane address which is dword aligned,
2729 which can be shifted to multiply by the wavefront size, and then used to form a
2730 private wavefront address that gives a location for a contiguous set of dwords,
2731 one per lane, where the vector register dwords are spilled. The compiler knows
2732 the wavefront size since it generates the code. Note that the type of the
2733 address may have to be converted as the size of a
2734 ``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a
2735 ``DW_ASPACE_AMDGPU_private_wave`` address.
2737 .. _amdgpu-dwarf-lane-identifier:
2739 Lane identifier
2740 ---------------
2742 DWARF lane identifies specify a target architecture lane position for hardware
2743 that executes in a SIMD or SIMT manner, and on which a source language maps its
2744 threads of execution onto those lanes. The DWARF lane identifier is pushed by
2745 the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5
2746 section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging*
2747 section :ref:`amdgpu-dwarf-operation-expressions`.
2749 For AMDGPU, the lane identifier corresponds to the hardware lane ID of a
2750 wavefront. It is numbered from 0 to the wavefront size minus 1.
2752 Operation Expressions
2753 ---------------------
2755 DWARF expressions are used to compute program values and the locations of
2756 program objects. See DWARF Version 5 section 2.5 and
2757 :ref:`amdgpu-dwarf-operation-expressions`.
2759 DWARF location descriptions describe how to access storage which includes memory
2760 and registers. When accessing storage on AMDGPU, bytes are ordered with least
2761 significant bytes first, and bits are ordered within bytes with least
2762 significant bits first.
2764 For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe
2765 unwinding vector registers that are spilled under the execution mask to memory:
2766 the zero-single location description is the vector register, and the one-single
2767 location description is the spilled memory location description. The
2768 ``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the
2769 memory location description.
2771 In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the
2772 ``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is
2773 controlled by the execution mask. An undefined location description together
2774 with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry
2775 to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example.
2777 Debugger Information Entry Attributes
2778 -------------------------------------
2780 This section describes how certain debugger information entry attributes are
2781 used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1
2782 which are updated by *DWARF Extensions For Heterogeneous Debugging* section
2783 :ref:`amdgpu-dwarf-low-level-information` and
2784 :ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`.
2786 .. _amdgpu-dwarf-dw-at-llvm-lane-pc:
2788 ``DW_AT_LLVM_lane_pc``
2789 ~~~~~~~~~~~~~~~~~~~~~~
2791 For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program
2792 location of the separate lanes of a SIMT thread.
2794 If the lane is an active lane then this will be the same as the current program
2795 location.
2797 If the lane is inactive, but was active on entry to the subprogram, then this is
2798 the program location in the subprogram at which execution of the lane is
2799 conceptual positioned.
2801 If the lane was not active on entry to the subprogram, then this will be the
2802 undefined location. A client debugger can check if the lane is part of a valid
2803 work-group by checking that the lane is in the range of the associated
2804 work-group within the grid, accounting for partial work-groups. If it is not,
2805 then the debugger can omit any information for the lane. Otherwise, the debugger
2806 may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the
2807 calling subprogram until it finds a non-undefined location. Conceptually the
2808 lane only has the call frames that it has a non-undefined
2809 ``DW_AT_LLVM_lane_pc``.
2811 The following example illustrates how the AMDGPU backend can generate a DWARF
2812 location list expression for the nested ``IF/THEN/ELSE`` structures of the
2813 following subprogram pseudo code for a target with 64 lanes per wavefront.
2815 .. code::
2816 :number-lines:
2818 SUBPROGRAM X
2819 BEGIN
2820 a;
2821 IF (c1) THEN
2822 b;
2823 IF (c2) THEN
2824 c;
2825 ELSE
2826 d;
2827 ENDIF
2828 e;
2829 ELSE
2830 f;
2831 ENDIF
2832 g;
2833 END
2835 The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the
2836 execution mask (``EXEC``) to linearize the control flow. The condition is
2837 evaluated to make a mask of the lanes for which the condition evaluates to true.
2838 First the ``THEN`` region is executed by setting the ``EXEC`` mask to the
2839 logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the
2840 ``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of
2841 the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE``
2842 region the ``EXEC`` mask is restored to the value it had at the beginning of the
2843 region. This is shown below. Other approaches are possible, but the basic
2844 concept is the same.
2846 .. code::
2847 :number-lines:
2849 $lex_start:
2850 a;
2851 %1 = EXEC
2852 %2 = c1
2853 $lex_1_start:
2854 EXEC = %1 & %2
2855 $if_1_then:
2856 b;
2857 %3 = EXEC
2858 %4 = c2
2859 $lex_1_1_start:
2860 EXEC = %3 & %4
2861 $lex_1_1_then:
2862 c;
2863 EXEC = ~EXEC & %3
2864 $lex_1_1_else:
2865 d;
2866 EXEC = %3
2867 $lex_1_1_end:
2868 e;
2869 EXEC = ~EXEC & %1
2870 $lex_1_else:
2871 f;
2872 EXEC = %1
2873 $lex_1_end:
2874 g;
2875 $lex_end:
2877 To create the DWARF location list expression that defines the location
2878 description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE``
2879 pseudo instruction can be used to annotate the linearized control flow. This can
2880 be done by defining an artificial variable for the lane PC. The DWARF location
2881 list expression created for it is used as the value of the
2882 ``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry.
2884 A DWARF procedure is defined for each well nested structured control flow region
2885 which provides the conceptual lane program location for a lane if it is not
2886 active (namely it is divergent). The DWARF operation expression for each region
2887 conceptually inherits the value of the immediately enclosing region and modifies
2888 it according to the semantics of the region.
2890 For an ``IF/THEN/ELSE`` region the divergent program location is at the start of
2891 the region for the ``THEN`` region since it is executed first. For the ``ELSE``
2892 region the divergent program location is at the end of the ``IF/THEN/ELSE``
2893 region since the ``THEN`` region has completed.
2895 The lane PC artificial variable is assigned at each region transition. It uses
2896 the immediately enclosing region's DWARF procedure to compute the program
2897 location for each lane assuming they are divergent, and then modifies the result
2898 by inserting the current program location for each lane that the ``EXEC`` mask
2899 indicates is active.
2901 By having separate DWARF procedures for each region, they can be reused to
2902 define the value for any nested region. This reduces the total size of the DWARF
2903 operation expressions.
2905 The following provides an example using pseudo LLVM MIR.
2907 .. code::
2908 :number-lines:
2910 $lex_start:
2911 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[
2912 DW_AT_name = "__uint64";
2913 DW_AT_byte_size = 8;
2914 DW_AT_encoding = DW_ATE_unsigned;
2915 ];
2916 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[
2917 DW_AT_name = "__active_lane_pc";
2918 DW_AT_location = [
2919 DW_OP_regx PC;
2920 DW_OP_LLVM_extend 64, 64;
2921 DW_OP_regval_type EXEC, %uint_64;
2922 DW_OP_LLVM_select_bit_piece 64, 64;
2923 ];
2924 ];
2925 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[
2926 DW_AT_name = "__divergent_lane_pc";
2927 DW_AT_location = [
2928 DW_OP_LLVM_undefined;
2929 DW_OP_LLVM_extend 64, 64;
2930 ];
2931 ];
2932 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[
2933 DW_OP_call_ref %__divergent_lane_pc;
2934 DW_OP_call_ref %__active_lane_pc;
2935 ];
2936 a;
2937 %1 = EXEC;
2938 DBG_VALUE %1, $noreg, %__lex_1_save_exec;
2939 %2 = c1;
2940 $lex_1_start:
2941 EXEC = %1 & %2;
2942 $lex_1_then:
2943 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[
2944 DW_AT_name = "__divergent_lane_pc_1_then";
2945 DW_AT_location = DIExpression[
2946 DW_OP_call_ref %__divergent_lane_pc;
2947 DW_OP_addrx &lex_1_start;
2948 DW_OP_stack_value;
2949 DW_OP_LLVM_extend 64, 64;
2950 DW_OP_call_ref %__lex_1_save_exec;
2951 DW_OP_deref_type 64, %__uint_64;
2952 DW_OP_LLVM_select_bit_piece 64, 64;
2953 ];
2954 ];
2955 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[
2956 DW_OP_call_ref %__divergent_lane_pc_1_then;
2957 DW_OP_call_ref %__active_lane_pc;
2958 ];
2959 b;
2960 %3 = EXEC;
2961 DBG_VALUE %3, %__lex_1_1_save_exec;
2962 %4 = c2;
2963 $lex_1_1_start:
2964 EXEC = %3 & %4;
2965 $lex_1_1_then:
2966 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[
2967 DW_AT_name = "__divergent_lane_pc_1_1_then";
2968 DW_AT_location = DIExpression[
2969 DW_OP_call_ref %__divergent_lane_pc_1_then;
2970 DW_OP_addrx &lex_1_1_start;
2971 DW_OP_stack_value;
2972 DW_OP_LLVM_extend 64, 64;
2973 DW_OP_call_ref %__lex_1_1_save_exec;
2974 DW_OP_deref_type 64, %__uint_64;
2975 DW_OP_LLVM_select_bit_piece 64, 64;
2976 ];
2977 ];
2978 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[
2979 DW_OP_call_ref %__divergent_lane_pc_1_1_then;
2980 DW_OP_call_ref %__active_lane_pc;
2981 ];
2982 c;
2983 EXEC = ~EXEC & %3;
2984 $lex_1_1_else:
2985 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[
2986 DW_AT_name = "__divergent_lane_pc_1_1_else";
2987 DW_AT_location = DIExpression[
2988 DW_OP_call_ref %__divergent_lane_pc_1_then;
2989 DW_OP_addrx &lex_1_1_end;
2990 DW_OP_stack_value;
2991 DW_OP_LLVM_extend 64, 64;
2992 DW_OP_call_ref %__lex_1_1_save_exec;
2993 DW_OP_deref_type 64, %__uint_64;
2994 DW_OP_LLVM_select_bit_piece 64, 64;
2995 ];
2996 ];
2997 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[
2998 DW_OP_call_ref %__divergent_lane_pc_1_1_else;
2999 DW_OP_call_ref %__active_lane_pc;
3000 ];
3001 d;
3002 EXEC = %3;
3003 $lex_1_1_end:
3004 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[
3005 DW_OP_call_ref %__divergent_lane_pc;
3006 DW_OP_call_ref %__active_lane_pc;
3007 ];
3008 e;
3009 EXEC = ~EXEC & %1;
3010 $lex_1_else:
3011 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[
3012 DW_AT_name = "__divergent_lane_pc_1_else";
3013 DW_AT_location = DIExpression[
3014 DW_OP_call_ref %__divergent_lane_pc;
3015 DW_OP_addrx &lex_1_end;
3016 DW_OP_stack_value;
3017 DW_OP_LLVM_extend 64, 64;
3018 DW_OP_call_ref %__lex_1_save_exec;
3019 DW_OP_deref_type 64, %__uint_64;
3020 DW_OP_LLVM_select_bit_piece 64, 64;
3021 ];
3022 ];
3023 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[
3024 DW_OP_call_ref %__divergent_lane_pc_1_else;
3025 DW_OP_call_ref %__active_lane_pc;
3026 ];
3027 f;
3028 EXEC = %1;
3029 $lex_1_end:
3030 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[
3031 DW_OP_call_ref %__divergent_lane_pc;
3032 DW_OP_call_ref %__active_lane_pc;
3033 ];
3034 g;
3035 $lex_end:
3037 The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements
3038 that are active, with the current program location.
3040 Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for
3041 the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo
3042 instruction, location list entries will be created that describe where the
3043 artificial variables are allocated at any given program location. The compiler
3044 may allocate them to registers or spill them to memory.
3046 The DWARF procedures for each region use the values of the saved execution mask
3047 artificial variables to only update the lanes that are active on entry to the
3048 region. All other lanes retain the value of the enclosing region where they were
3049 last active. If they were not active on entry to the subprogram, then will have
3050 the undefined location description.
3052 Other structured control flow regions can be handled similarly. For example,
3053 loops would set the divergent program location for the region at the end of the
3054 loop. Any lanes active will be in the loop, and any lanes not active must have
3055 exited the loop.
3057 An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of
3058 ``IF/THEN/ELSE`` regions.
3060 The DWARF procedures can use the active lane artificial variable described in
3061 :ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual
3062 ``EXEC`` mask in order to support whole or quad wavefront mode.
3064 .. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane:
3066 ``DW_AT_LLVM_active_lane``
3067 ~~~~~~~~~~~~~~~~~~~~~~~~~~
3069 The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information
3070 entry is used to specify the lanes that are conceptually active for a SIMT
3071 thread.
3073 The execution mask may be modified to implement whole or quad wavefront mode
3074 operations. For example, all lanes may need to temporarily be made active to
3075 execute a whole wavefront operation. Such regions would save the ``EXEC`` mask,
3076 update it to enable the necessary lanes, perform the operations, and then
3077 restore the ``EXEC`` mask from the saved value. While executing the whole
3078 wavefront region, the conceptual execution mask is the saved value, not the
3079 ``EXEC`` value.
3081 This is handled by defining an artificial variable for the active lane mask. The
3082 active lane mask artificial variable would be the actual ``EXEC`` mask for
3083 normal regions, and the saved execution mask for regions where the mask is
3084 temporarily updated. The location list expression created for this artificial
3085 variable is used to define the value of the ``DW_AT_LLVM_active_lane``
3086 attribute.
3088 ``DW_AT_LLVM_augmentation``
3089 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
3091 For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit
3092 debugger information entry has the following value for the augmentation string:
3094 ::
3096 [amdgpu:v0.0]
3098 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU
3099 extensions used in the DWARF of the compilation unit. The version number
3100 conforms to [SEMVER]_.
3102 Call Frame Information
3103 ----------------------
3105 DWARF Call Frame Information (CFI) describes how a consumer can virtually
3106 *unwind* call frames in a running process or core dump. See DWARF Version 5
3107 section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`.
3109 For AMDGPU, the Common Information Entry (CIE) fields have the following values:
3111 1. ``augmentation`` string contains the following null-terminated UTF-8 string:
3113 ::
3115 [amd:v0.0]
3117 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU
3118 extensions used in this CIE or to the FDEs that use it. The version number
3119 conforms to [SEMVER]_.
3121 2. ``address_size`` for the ``Global`` address space is defined in
3122 :ref:`amdgpu-dwarf-address-space-identifier`.
3124 3. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector.
3126 4. ``code_alignment_factor`` is 4 bytes.
3128 .. TODO::
3130 Add to :ref:`amdgpu-processor-table` table.
3132 5. ``data_alignment_factor`` is 4 bytes.
3134 .. TODO::
3136 Add to :ref:`amdgpu-processor-table` table.
3138 6. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64``
3139 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`.
3141 7. ``initial_instructions`` Since a subprogram X with fewer registers can be
3142 called from subprogram Y that has more allocated, X will not change any of
3143 the extra registers as it cannot access them. Therefore, the default rule
3144 for all columns is ``same value``.
3146 For AMDGPU the register number follows the numbering defined in
3147 :ref:`amdgpu-dwarf-register-identifier`.
3149 For AMDGPU the instructions are variable size. A consumer can subtract 1 from
3150 the return address to get the address of a byte within the call site
3151 instructions. See DWARF Version 5 section 6.4.4.
3153 Accelerated Access
3154 ------------------
3156 See DWARF Version 5 section 6.1.
3158 Lookup By Name Section Header
3159 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3161 See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`.
3163 For AMDGPU the lookup by name section header table:
3165 ``augmentation_string_size`` (uword)
3167 Set to the length of the ``augmentation_string`` value which is always a
3168 multiple of 4.
3170 ``augmentation_string`` (sequence of UTF-8 characters)
3172 Contains the following UTF-8 string null padded to a multiple of 4 bytes:
3174 ::
3176 [amdgpu:v0.0]
3178 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU
3179 extensions used in the DWARF of this index. The version number conforms to
3180 [SEMVER]_.
3182 .. note::
3184 This is different to the DWARF Version 5 definition that requires the first
3185 4 characters to be the vendor ID. But this is consistent with the other
3186 augmentation strings and does allow multiple vendor contributions. However,
3187 backwards compatibility may be more desirable.
3189 Lookup By Address Section Header
3190 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3192 See DWARF Version 5 section 6.1.2.
3194 For AMDGPU the lookup by address section header table:
3196 ``address_size`` (ubyte)
3198 Match the address size for the ``Global`` address space defined in
3199 :ref:`amdgpu-dwarf-address-space-identifier`.
3201 ``segment_selector_size`` (ubyte)
3203 AMDGPU does not use a segment selector so this is 0. The entries in the
3204 ``.debug_aranges`` do not have a segment selector.
3206 Line Number Information
3207 -----------------------
3209 See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`.
3211 AMDGPU does not use the ``isa`` state machine registers and always sets it to 0.
3212 The instruction set must be obtained from the ELF file header ``e_flags`` field
3213 in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header
3214 <amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2.
3216 .. TODO::
3218 Should the ``isa`` state machine register be used to indicate if the code is
3219 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA?
3221 For AMDGPU the line number program header fields have the following values (see
3222 DWARF Version 5 section 6.2.4):
3224 ``address_size`` (ubyte)
3225 Matches the address size for the ``Global`` address space defined in
3226 :ref:`amdgpu-dwarf-address-space-identifier`.
3228 ``segment_selector_size`` (ubyte)
3229 AMDGPU does not use a segment selector so this is 0.
3231 ``minimum_instruction_length`` (ubyte)
3232 For GFX9-GFX11 this is 4.
3234 ``maximum_operations_per_instruction`` (ubyte)
3235 For GFX9-GFX11 this is 1.
3237 Source text for online-compiled programs (for example, those compiled by the
3238 OpenCL language runtime) may be embedded into the DWARF Version 5 line table.
3239 See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For
3240 Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source
3241 <amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`.
3243 The Clang option used to control source embedding in AMDGPU is defined in
3244 :ref:`amdgpu-clang-debug-options-table`.
3246 .. table:: AMDGPU Clang Debug Options
3247 :name: amdgpu-clang-debug-options-table
3249 ==================== ==================================================
3250 Debug Flag Description
3251 ==================== ==================================================
3252 -g[no-]embed-source Enable/disable embedding source text in DWARF
3253 debug sections. Useful for environments where
3254 source cannot be written to disk, such as
3255 when performing online compilation.
3256 ==================== ==================================================
3258 For example:
3260 ``-gembed-source``
3261 Enable the embedded source.
3263 ``-gno-embed-source``
3264 Disable the embedded source.
3266 32-Bit and 64-Bit DWARF Formats
3267 -------------------------------
3269 See DWARF Version 5 section 7.4 and
3270 :ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`.
3272 For AMDGPU:
3274 * For the ``amdgcn`` target architecture only the 64-bit process address space
3275 is supported.
3277 * The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates
3278 the 32-bit DWARF format.
3280 Unit Headers
3281 ------------
3283 For AMDGPU the following values apply for each of the unit headers described in
3284 DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3:
3286 ``address_size`` (ubyte)
3287 Matches the address size for the ``Global`` address space defined in
3288 :ref:`amdgpu-dwarf-address-space-identifier`.
3290 .. _amdgpu-code-conventions:
3292 Code Conventions
3293 ================
3295 This section provides code conventions used for each supported target triple OS
3296 (see :ref:`amdgpu-target-triples`).
3298 AMDHSA
3299 ------
3301 This section provides code conventions used when the target triple OS is
3302 ``amdhsa`` (see :ref:`amdgpu-target-triples`).
3304 .. _amdgpu-amdhsa-code-object-metadata:
3306 Code Object Metadata
3307 ~~~~~~~~~~~~~~~~~~~~
3309 The code object metadata specifies extensible metadata associated with the code
3310 objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The
3311 encoding and semantics of this metadata depends on the code object version; see
3312 :ref:`amdgpu-amdhsa-code-object-metadata-v2`,
3313 :ref:`amdgpu-amdhsa-code-object-metadata-v3`,
3314 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and
3315 :ref:`amdgpu-amdhsa-code-object-metadata-v5`.
3317 Code object metadata is specified in a note record (see
3318 :ref:`amdgpu-note-records`) and is required when the target triple OS is
3319 ``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum
3320 information necessary to support the HSA compatible runtime kernel queries. For
3321 example, the segment sizes needed in a dispatch packet. In addition, a
3322 high-level language runtime may require other information to be included. For
3323 example, the AMD OpenCL runtime records kernel argument information.
3325 .. _amdgpu-amdhsa-code-object-metadata-v2:
3327 Code Object V2 Metadata
3328 +++++++++++++++++++++++
3330 .. warning::
3331 Code object V2 generation is no longer supported by this version of LLVM.
3333 Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record
3334 (see :ref:`amdgpu-note-records-v2`).
3336 The metadata is specified as a YAML formatted string (see [YAML]_ and
3337 :doc:`YamlIO`).
3339 .. TODO::
3341 Is the string null terminated? It probably should not if YAML allows it to
3342 contain null characters, otherwise it should be.
3344 The metadata is represented as a single YAML document comprised of the mapping
3345 defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and
3346 referenced tables.
3348 For boolean values, the string values of ``false`` and ``true`` are used for
3349 false and true respectively.
3351 Additional information can be added to the mappings. To avoid conflicts, any
3352 non-AMD key names should be prefixed by "*vendor-name*.".
3354 .. table:: AMDHSA Code Object V2 Metadata Map
3355 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table
3357 ========== ============== ========= =======================================
3358 String Key Value Type Required? Description
3359 ========== ============== ========= =======================================
3360 "Version" sequence of Required - The first integer is the major
3361 2 integers version. Currently 1.
3362 - The second integer is the minor
3363 version. Currently 0.
3364 "Printf" sequence of Each string is encoded information
3365 strings about a printf function call. The
3366 encoded information is organized as
3367 fields separated by colon (':'):
3369 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString``
3371 where:
3373 ``ID``
3374 A 32-bit integer as a unique id for
3375 each printf function call
3377 ``N``
3378 A 32-bit integer equal to the number
3379 of arguments of printf function call
3380 minus 1
3382 ``S[i]`` (where i = 0, 1, ... , N-1)
3383 32-bit integers for the size in bytes
3384 of the i-th FormatString argument of
3385 the printf function call
3387 FormatString
3388 The format string passed to the
3389 printf function call.
3390 "Kernels" sequence of Required Sequence of the mappings for each
3391 mapping kernel in the code object. See
3392 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table`
3393 for the definition of the mapping.
3394 ========== ============== ========= =======================================
3396 ..
3398 .. table:: AMDHSA Code Object V2 Kernel Metadata Map
3399 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table
3401 ================= ============== ========= ================================
3402 String Key Value Type Required? Description
3403 ================= ============== ========= ================================
3404 "Name" string Required Source name of the kernel.
3405 "SymbolName" string Required Name of the kernel
3406 descriptor ELF symbol.
3407 "Language" string Source language of the kernel.
3408 Values include:
3410 - "OpenCL C"
3411 - "OpenCL C++"
3412 - "HCC"
3413 - "OpenMP"
3415 "LanguageVersion" sequence of - The first integer is the major
3416 2 integers version.
3417 - The second integer is the
3418 minor version.
3419 "Attrs" mapping Mapping of kernel attributes.
3420 See
3421 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table`
3422 for the mapping definition.
3423 "Args" sequence of Sequence of mappings of the
3424 mapping kernel arguments. See
3425 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table`
3426 for the definition of the mapping.
3427 "CodeProps" mapping Mapping of properties related to
3428 the kernel code. See
3429 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table`
3430 for the mapping definition.
3431 ================= ============== ========= ================================
3433 ..
3435 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map
3436 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table
3438 =================== ============== ========= ==============================
3439 String Key Value Type Required? Description
3440 =================== ============== ========= ==============================
3441 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values
3442 3 integers must be >=1 and the dispatch
3443 work-group size X, Y, Z must
3444 correspond to the specified
3445 values. Defaults to 0, 0, 0.
3447 Corresponds to the OpenCL
3448 ``reqd_work_group_size``
3449 attribute.
3450 "WorkGroupSizeHint" sequence of The dispatch work-group size
3451 3 integers X, Y, Z is likely to be the
3452 specified values.
3454 Corresponds to the OpenCL
3455 ``work_group_size_hint``
3456 attribute.
3457 "VecTypeHint" string The name of a scalar or vector
3458 type.
3460 Corresponds to the OpenCL
3461 ``vec_type_hint`` attribute.
3463 "RuntimeHandle" string The external symbol name
3464 associated with a kernel.
3465 OpenCL runtime allocates a
3466 global buffer for the symbol
3467 and saves the kernel's address
3468 to it, which is used for
3469 device side enqueueing. Only
3470 available for device side
3471 enqueued kernels.
3472 =================== ============== ========= ==============================
3474 ..
3476 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map
3477 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table
3479 ================= ============== ========= ================================
3480 String Key Value Type Required? Description
3481 ================= ============== ========= ================================
3482 "Name" string Kernel argument name.
3483 "TypeName" string Kernel argument type name.
3484 "Size" integer Required Kernel argument size in bytes.
3485 "Align" integer Required Kernel argument alignment in
3486 bytes. Must be a power of two.
3487 "ValueKind" string Required Kernel argument kind that
3488 specifies how to set up the
3489 corresponding argument.
3490 Values include:
3492 "ByValue"
3493 The argument is copied
3494 directly into the kernarg.
3496 "GlobalBuffer"
3497 A global address space pointer
3498 to the buffer data is passed
3499 in the kernarg.
3501 "DynamicSharedPointer"
3502 A group address space pointer
3503 to dynamically allocated LDS
3504 is passed in the kernarg.
3506 "Sampler"
3507 A global address space
3508 pointer to a S# is passed in
3509 the kernarg.
3511 "Image"
3512 A global address space
3513 pointer to a T# is passed in
3514 the kernarg.
3516 "Pipe"
3517 A global address space pointer
3518 to an OpenCL pipe is passed in
3519 the kernarg.
3521 "Queue"
3522 A global address space pointer
3523 to an OpenCL device enqueue
3524 queue is passed in the
3525 kernarg.
3527 "HiddenGlobalOffsetX"
3528 The OpenCL grid dispatch
3529 global offset for the X
3530 dimension is passed in the
3531 kernarg.
3533 "HiddenGlobalOffsetY"
3534 The OpenCL grid dispatch
3535 global offset for the Y
3536 dimension is passed in the
3537 kernarg.
3539 "HiddenGlobalOffsetZ"
3540 The OpenCL grid dispatch
3541 global offset for the Z
3542 dimension is passed in the
3543 kernarg.
3545 "HiddenNone"
3546 An argument that is not used
3547 by the kernel. Space needs to
3548 be left for it, but it does
3549 not need to be set up.
3551 "HiddenPrintfBuffer"
3552 A global address space pointer
3553 to the runtime printf buffer
3554 is passed in kernarg. Mutually
3555 exclusive with
3556 "HiddenHostcallBuffer".
3558 "HiddenHostcallBuffer"
3559 A global address space pointer
3560 to the runtime hostcall buffer
3561 is passed in kernarg. Mutually
3562 exclusive with
3563 "HiddenPrintfBuffer".
3565 "HiddenDefaultQueue"
3566 A global address space pointer
3567 to the OpenCL device enqueue
3568 queue that should be used by
3569 the kernel by default is
3570 passed in the kernarg.
3572 "HiddenCompletionAction"
3573 A global address space pointer
3574 to help link enqueued kernels into
3575 the ancestor tree for determining
3576 when the parent kernel has finished.
3578 "HiddenMultiGridSyncArg"
3579 A global address space pointer for
3580 multi-grid synchronization is
3581 passed in the kernarg.
3583 "ValueType" string Unused and deprecated. This should no longer
3584 be emitted, but is accepted for compatibility.
3587 "PointeeAlign" integer Alignment in bytes of pointee
3588 type for pointer type kernel
3589 argument. Must be a power
3590 of 2. Only present if
3591 "ValueKind" is
3592 "DynamicSharedPointer".
3593 "AddrSpaceQual" string Kernel argument address space
3594 qualifier. Only present if
3595 "ValueKind" is "GlobalBuffer" or
3596 "DynamicSharedPointer". Values
3597 are:
3599 - "Private"
3600 - "Global"
3601 - "Constant"
3602 - "Local"
3603 - "Generic"
3604 - "Region"
3606 .. TODO::
3608 Is GlobalBuffer only Global
3609 or Constant? Is
3610 DynamicSharedPointer always
3611 Local? Can HCC allow Generic?
3612 How can Private or Region
3613 ever happen?
3615 "AccQual" string Kernel argument access
3616 qualifier. Only present if
3617 "ValueKind" is "Image" or
3618 "Pipe". Values
3619 are:
3621 - "ReadOnly"
3622 - "WriteOnly"
3623 - "ReadWrite"
3625 .. TODO::
3627 Does this apply to
3628 GlobalBuffer?
3630 "ActualAccQual" string The actual memory accesses
3631 performed by the kernel on the
3632 kernel argument. Only present if
3633 "ValueKind" is "GlobalBuffer",
3634 "Image", or "Pipe". This may be
3635 more restrictive than indicated
3636 by "AccQual" to reflect what the
3637 kernel actual does. If not
3638 present then the runtime must
3639 assume what is implied by
3640 "AccQual" and "IsConst". Values
3641 are:
3643 - "ReadOnly"
3644 - "WriteOnly"
3645 - "ReadWrite"
3647 "IsConst" boolean Indicates if the kernel argument
3648 is const qualified. Only present
3649 if "ValueKind" is
3650 "GlobalBuffer".
3652 "IsRestrict" boolean Indicates if the kernel argument
3653 is restrict qualified. Only
3654 present if "ValueKind" is
3655 "GlobalBuffer".
3657 "IsVolatile" boolean Indicates if the kernel argument
3658 is volatile qualified. Only
3659 present if "ValueKind" is
3660 "GlobalBuffer".
3662 "IsPipe" boolean Indicates if the kernel argument
3663 is pipe qualified. Only present
3664 if "ValueKind" is "Pipe".
3666 .. TODO::
3668 Can GlobalBuffer be pipe
3669 qualified?
3671 ================= ============== ========= ================================
3673 ..
3675 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map
3676 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table
3678 ============================ ============== ========= =====================
3679 String Key Value Type Required? Description
3680 ============================ ============== ========= =====================
3681 "KernargSegmentSize" integer Required The size in bytes of
3682 the kernarg segment
3683 that holds the values
3684 of the arguments to
3685 the kernel.
3686 "GroupSegmentFixedSize" integer Required The amount of group
3687 segment memory
3688 required by a
3689 work-group in
3690 bytes. This does not
3691 include any
3692 dynamically allocated
3693 group segment memory
3694 that may be added
3695 when the kernel is
3696 dispatched.
3697 "PrivateSegmentFixedSize" integer Required The amount of fixed
3698 private address space
3699 memory required for a
3700 work-item in
3701 bytes. If the kernel
3702 uses a dynamic call
3703 stack then additional
3704 space must be added
3705 to this value for the
3706 call stack.
3707 "KernargSegmentAlign" integer Required The maximum byte
3708 alignment of
3709 arguments in the
3710 kernarg segment. Must
3711 be a power of 2.
3712 "WavefrontSize" integer Required Wavefront size. Must
3713 be a power of 2.
3714 "NumSGPRs" integer Required Number of scalar
3715 registers used by a
3716 wavefront for
3717 GFX6-GFX11. This
3718 includes the special
3719 SGPRs for VCC, Flat
3720 Scratch (GFX7-GFX10)
3721 and XNACK (for
3722 GFX8-GFX10). It does
3723 not include the 16
3724 SGPR added if a trap
3725 handler is
3726 enabled. It is not
3727 rounded up to the
3728 allocation
3729 granularity.
3730 "NumVGPRs" integer Required Number of vector
3731 registers used by
3732 each work-item for
3733 GFX6-GFX11
3734 "MaxFlatWorkGroupSize" integer Required Maximum flat
3735 work-group size
3736 supported by the
3737 kernel in work-items.
3738 Must be >=1 and
3739 consistent with
3740 ReqdWorkGroupSize if
3741 not 0, 0, 0.
3742 "NumSpilledSGPRs" integer Number of stores from
3743 a scalar register to
3744 a register allocator
3745 created spill
3746 location.
3747 "NumSpilledVGPRs" integer Number of stores from
3748 a vector register to
3749 a register allocator
3750 created spill
3751 location.
3752 ============================ ============== ========= =====================
3754 .. _amdgpu-amdhsa-code-object-metadata-v3:
3756 Code Object V3 Metadata
3757 +++++++++++++++++++++++
3759 .. warning::
3760 Code object V3 generation is no longer supported by this version of LLVM.
3762 Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note
3763 record (see :ref:`amdgpu-note-records-v3-onwards`).
3765 The metadata is represented as Message Pack formatted binary data (see
3766 [MsgPack]_). The top level is a Message Pack map that includes the
3767 keys defined in table
3768 :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced
3769 tables.
3771 Additional information can be added to the maps. To avoid conflicts,
3772 any key names should be prefixed by "*vendor-name*." where
3773 ``vendor-name`` can be the name of the vendor and specific vendor
3774 tool that generates the information. The prefix is abbreviated to
3775 simply "." when it appears within a map that has been added by the
3776 same *vendor-name*.
3778 .. table:: AMDHSA Code Object V3 Metadata Map
3779 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3
3781 ================= ============== ========= =======================================
3782 String Key Value Type Required? Description
3783 ================= ============== ========= =======================================
3784 "amdhsa.version" sequence of Required - The first integer is the major
3785 2 integers version. Currently 1.
3786 - The second integer is the minor
3787 version. Currently 0.
3788 "amdhsa.printf" sequence of Each string is encoded information
3789 strings about a printf function call. The
3790 encoded information is organized as
3791 fields separated by colon (':'):
3793 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString``
3795 where:
3797 ``ID``
3798 A 32-bit integer as a unique id for
3799 each printf function call
3801 ``N``
3802 A 32-bit integer equal to the number
3803 of arguments of printf function call
3804 minus 1
3806 ``S[i]`` (where i = 0, 1, ... , N-1)
3807 32-bit integers for the size in bytes
3808 of the i-th FormatString argument of
3809 the printf function call
3811 FormatString
3812 The format string passed to the
3813 printf function call.
3814 "amdhsa.kernels" sequence of Required Sequence of the maps for each
3815 map kernel in the code object. See
3816 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3`
3817 for the definition of the keys included
3818 in that map.
3819 ================= ============== ========= =======================================
3821 ..
3823 .. table:: AMDHSA Code Object V3 Kernel Metadata Map
3824 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3
3826 =================================== ============== ========= ================================
3827 String Key Value Type Required? Description
3828 =================================== ============== ========= ================================
3829 ".name" string Required Source name of the kernel.
3830 ".symbol" string Required Name of the kernel
3831 descriptor ELF symbol.
3832 ".language" string Source language of the kernel.
3833 Values include:
3835 - "OpenCL C"
3836 - "OpenCL C++"
3837 - "HCC"
3838 - "HIP"
3839 - "OpenMP"
3840 - "Assembler"
3842 ".language_version" sequence of - The first integer is the major
3843 2 integers version.
3844 - The second integer is the
3845 minor version.
3846 ".args" sequence of Sequence of maps of the
3847 map kernel arguments. See
3848 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`
3849 for the definition of the keys
3850 included in that map.
3851 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values
3852 3 integers must be >=1 and the dispatch
3853 work-group size X, Y, Z must
3854 correspond to the specified
3855 values. Defaults to 0, 0, 0.
3857 Corresponds to the OpenCL
3858 ``reqd_work_group_size``
3859 attribute.
3860 ".workgroup_size_hint" sequence of The dispatch work-group size
3861 3 integers X, Y, Z is likely to be the
3862 specified values.
3864 Corresponds to the OpenCL
3865 ``work_group_size_hint``
3866 attribute.
3867 ".vec_type_hint" string The name of a scalar or vector
3868 type.
3870 Corresponds to the OpenCL
3871 ``vec_type_hint`` attribute.
3873 ".device_enqueue_symbol" string The external symbol name
3874 associated with a kernel.
3875 OpenCL runtime allocates a
3876 global buffer for the symbol
3877 and saves the kernel's address
3878 to it, which is used for
3879 device side enqueueing. Only
3880 available for device side
3881 enqueued kernels.
3882 ".kernarg_segment_size" integer Required The size in bytes of
3883 the kernarg segment
3884 that holds the values
3885 of the arguments to
3886 the kernel.
3887 ".group_segment_fixed_size" integer Required The amount of group
3888 segment memory
3889 required by a
3890 work-group in
3891 bytes. This does not
3892 include any
3893 dynamically allocated
3894 group segment memory
3895 that may be added
3896 when the kernel is
3897 dispatched.
3898 ".private_segment_fixed_size" integer Required The amount of fixed
3899 private address space
3900 memory required for a
3901 work-item in
3902 bytes. If the kernel
3903 uses a dynamic call
3904 stack then additional
3905 space must be added
3906 to this value for the
3907 call stack.
3908 ".kernarg_segment_align" integer Required The maximum byte
3909 alignment of
3910 arguments in the
3911 kernarg segment. Must
3912 be a power of 2.
3913 ".wavefront_size" integer Required Wavefront size. Must
3914 be a power of 2.
3915 ".sgpr_count" integer Required Number of scalar
3916 registers required by a
3917 wavefront for
3918 GFX6-GFX9. A register
3919 is required if it is
3920 used explicitly, or
3921 if a higher numbered
3922 register is used
3923 explicitly. This
3924 includes the special
3925 SGPRs for VCC, Flat
3926 Scratch (GFX7-GFX9)
3927 and XNACK (for
3928 GFX8-GFX9). It does
3929 not include the 16
3930 SGPR added if a trap
3931 handler is
3932 enabled. It is not
3933 rounded up to the
3934 allocation
3935 granularity.
3936 ".vgpr_count" integer Required Number of vector
3937 registers required by
3938 each work-item for
3939 GFX6-GFX9. A register
3940 is required if it is
3941 used explicitly, or
3942 if a higher numbered
3943 register is used
3944 explicitly.
3945 ".agpr_count" integer Required Number of accumulator
3946 registers required by
3947 each work-item for
3948 GFX90A, GFX908.
3949 ".max_flat_workgroup_size" integer Required Maximum flat
3950 work-group size
3951 supported by the
3952 kernel in work-items.
3953 Must be >=1 and
3954 consistent with
3955 ReqdWorkGroupSize if
3956 not 0, 0, 0.
3957 ".sgpr_spill_count" integer Number of stores from
3958 a scalar register to
3959 a register allocator
3960 created spill
3961 location.
3962 ".vgpr_spill_count" integer Number of stores from
3963 a vector register to
3964 a register allocator
3965 created spill
3966 location.
3967 ".kind" string The kind of the kernel
3968 with the following
3969 values:
3971 "normal"
3972 Regular kernels.
3974 "init"
3975 These kernels must be
3976 invoked after loading
3977 the containing code
3978 object and must
3979 complete before any
3980 normal and fini
3981 kernels in the same
3982 code object are
3983 invoked.
3985 "fini"
3986 These kernels must be
3987 invoked before
3988 unloading the
3989 containing code object
3990 and after all init and
3991 normal kernels in the
3992 same code object have
3993 been invoked and
3994 completed.
3996 If omitted, "normal" is
3997 assumed.
3998 ".max_num_work_groups_{x,y,z}" integer The max number of
3999 launched work-groups
4000 in the X, Y, and Z
4001 dimensions. Each number
4002 must be >=1.
4003 =================================== ============== ========= ================================
4005 ..
4007 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map
4008 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3
4010 ====================== ============== ========= ================================
4011 String Key Value Type Required? Description
4012 ====================== ============== ========= ================================
4013 ".name" string Kernel argument name.
4014 ".type_name" string Kernel argument type name.
4015 ".size" integer Required Kernel argument size in bytes.
4016 ".offset" integer Required Kernel argument offset in
4017 bytes. The offset must be a
4018 multiple of the alignment
4019 required by the argument.
4020 ".value_kind" string Required Kernel argument kind that
4021 specifies how to set up the
4022 corresponding argument.
4023 Values include:
4025 "by_value"
4026 The argument is copied
4027 directly into the kernarg.
4029 "global_buffer"
4030 A global address space pointer
4031 to the buffer data is passed
4032 in the kernarg.
4034 "dynamic_shared_pointer"
4035 A group address space pointer
4036 to dynamically allocated LDS
4037 is passed in the kernarg.
4039 "sampler"
4040 A global address space
4041 pointer to a S# is passed in
4042 the kernarg.
4044 "image"
4045 A global address space
4046 pointer to a T# is passed in
4047 the kernarg.
4049 "pipe"
4050 A global address space pointer
4051 to an OpenCL pipe is passed in
4052 the kernarg.
4054 "queue"
4055 A global address space pointer
4056 to an OpenCL device enqueue
4057 queue is passed in the
4058 kernarg.
4060 "hidden_global_offset_x"
4061 The OpenCL grid dispatch
4062 global offset for the X
4063 dimension is passed in the
4064 kernarg.
4066 "hidden_global_offset_y"
4067 The OpenCL grid dispatch
4068 global offset for the Y
4069 dimension is passed in the
4070 kernarg.
4072 "hidden_global_offset_z"
4073 The OpenCL grid dispatch
4074 global offset for the Z
4075 dimension is passed in the
4076 kernarg.
4078 "hidden_none"
4079 An argument that is not used
4080 by the kernel. Space needs to
4081 be left for it, but it does
4082 not need to be set up.
4084 "hidden_printf_buffer"
4085 A global address space pointer
4086 to the runtime printf buffer
4087 is passed in kernarg. Mutually
4088 exclusive with
4089 "hidden_hostcall_buffer"
4090 before Code Object V5.
4092 "hidden_hostcall_buffer"
4093 A global address space pointer
4094 to the runtime hostcall buffer
4095 is passed in kernarg. Mutually
4096 exclusive with
4097 "hidden_printf_buffer"
4098 before Code Object V5.
4100 "hidden_default_queue"
4101 A global address space pointer
4102 to the OpenCL device enqueue
4103 queue that should be used by
4104 the kernel by default is
4105 passed in the kernarg.
4107 "hidden_completion_action"
4108 A global address space pointer
4109 to help link enqueued kernels into
4110 the ancestor tree for determining
4111 when the parent kernel has finished.
4113 "hidden_multigrid_sync_arg"
4114 A global address space pointer for
4115 multi-grid synchronization is
4116 passed in the kernarg.
4118 ".value_type" string Unused and deprecated. This should no longer
4119 be emitted, but is accepted for compatibility.
4121 ".pointee_align" integer Alignment in bytes of pointee
4122 type for pointer type kernel
4123 argument. Must be a power
4124 of 2. Only present if
4125 ".value_kind" is
4126 "dynamic_shared_pointer".
4127 ".address_space" string Kernel argument address space
4128 qualifier. Only present if
4129 ".value_kind" is "global_buffer" or
4130 "dynamic_shared_pointer". Values
4131 are:
4133 - "private"
4134 - "global"
4135 - "constant"
4136 - "local"
4137 - "generic"
4138 - "region"
4140 .. TODO::
4142 Is "global_buffer" only "global"
4143 or "constant"? Is
4144 "dynamic_shared_pointer" always
4145 "local"? Can HCC allow "generic"?
4146 How can "private" or "region"
4147 ever happen?
4149 ".access" string Kernel argument access
4150 qualifier. Only present if
4151 ".value_kind" is "image" or
4152 "pipe". Values
4153 are:
4155 - "read_only"
4156 - "write_only"
4157 - "read_write"
4159 .. TODO::
4161 Does this apply to
4162 "global_buffer"?
4164 ".actual_access" string The actual memory accesses
4165 performed by the kernel on the
4166 kernel argument. Only present if
4167 ".value_kind" is "global_buffer",
4168 "image", or "pipe". This may be
4169 more restrictive than indicated
4170 by ".access" to reflect what the
4171 kernel actual does. If not
4172 present then the runtime must
4173 assume what is implied by
4174 ".access" and ".is_const" . Values
4175 are:
4177 - "read_only"
4178 - "write_only"
4179 - "read_write"
4181 ".is_const" boolean Indicates if the kernel argument
4182 is const qualified. Only present
4183 if ".value_kind" is
4184 "global_buffer".
4186 ".is_restrict" boolean Indicates if the kernel argument
4187 is restrict qualified. Only
4188 present if ".value_kind" is
4189 "global_buffer".
4191 ".is_volatile" boolean Indicates if the kernel argument
4192 is volatile qualified. Only
4193 present if ".value_kind" is
4194 "global_buffer".
4196 ".is_pipe" boolean Indicates if the kernel argument
4197 is pipe qualified. Only present
4198 if ".value_kind" is "pipe".
4200 .. TODO::
4202 Can "global_buffer" be pipe
4203 qualified?
4205 ====================== ============== ========= ================================
4207 .. _amdgpu-amdhsa-code-object-metadata-v4:
4209 Code Object V4 Metadata
4210 +++++++++++++++++++++++
4212 . warning::
4213 Code object V4 is not the default code object version emitted by this version
4214 of LLVM.
4216 Code object V4 metadata is the same as
4217 :ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions
4218 defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`.
4220 .. table:: AMDHSA Code Object V4 Metadata Map Changes
4221 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4
4223 ================= ============== ========= =======================================
4224 String Key Value Type Required? Description
4225 ================= ============== ========= =======================================
4226 "amdhsa.version" sequence of Required - The first integer is the major
4227 2 integers version. Currently 1.
4228 - The second integer is the minor
4229 version. Currently 1.
4230 "amdhsa.target" string Required The target name of the code using the syntax:
4232 .. code::
4234 <target-triple> [ "-" <target-id> ]
4236 A canonical target ID must be
4237 used. See :ref:`amdgpu-target-triples`
4238 and :ref:`amdgpu-target-id`.
4239 ================= ============== ========= =======================================
4241 .. _amdgpu-amdhsa-code-object-metadata-v5:
4243 Code Object V5 Metadata
4244 +++++++++++++++++++++++
4246 Code object V5 metadata is the same as
4247 :ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table
4248 :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5`, table
4249 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v5` and table
4250 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`.
4252 .. table:: AMDHSA Code Object V5 Metadata Map Changes
4253 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5
4255 ================= ============== ========= =======================================
4256 String Key Value Type Required? Description
4257 ================= ============== ========= =======================================
4258 "amdhsa.version" sequence of Required - The first integer is the major
4259 2 integers version. Currently 1.
4260 - The second integer is the minor
4261 version. Currently 2.
4262 ================= ============== ========= =======================================
4264 ..
4266 .. table:: AMDHSA Code Object V5 Kernel Metadata Map Additions
4267 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v5
4269 ============================= ============= ========== =======================================
4270 String Key Value Type Required? Description
4271 ============================= ============= ========== =======================================
4272 ".uses_dynamic_stack" boolean Indicates if the generated machine code
4273 is using a dynamically sized stack.
4274 ".workgroup_processor_mode" boolean (GFX10+) Controls ENABLE_WGP_MODE in
4275 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
4276 ============================= ============= ========== =======================================
4278 ..
4280 .. table:: AMDHSA Code Object V5 Kernel Attribute Metadata Map
4281 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v5-table
4283 =========================== ============== ========= ==============================
4284 String Key Value Type Required? Description
4285 =========================== ============== ========= ==============================
4286 ".uniform_work_group_size" integer Indicates if the kernel
4287 requires that each dimension
4288 of global size is a multiple
4289 of corresponding dimension of
4290 work-group size. Value of 1
4291 implies true and value of 0
4292 implies false. Metadata is
4293 only emitted when value is 1.
4294 =========================== ============== ========= ==============================
4296 ..
4298 ..
4300 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes
4301 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5
4303 ====================== ============== ========= ================================
4304 String Key Value Type Required? Description
4305 ====================== ============== ========= ================================
4306 ".value_kind" string Required Kernel argument kind that
4307 specifies how to set up the
4308 corresponding argument.
4309 Values include:
4310 the same as code object V3 metadata
4311 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`)
4312 with the following additions:
4314 "hidden_block_count_x"
4315 The grid dispatch work-group count for the X dimension
4316 is passed in the kernarg. Some languages, such as OpenCL,
4317 support a last work-group in each dimension being partial.
4318 This count only includes the non-partial work-group count.
4319 This is not the same as the value in the AQL dispatch packet,
4320 which has the grid size in work-items.
4322 "hidden_block_count_y"
4323 The grid dispatch work-group count for the Y dimension
4324 is passed in the kernarg. Some languages, such as OpenCL,
4325 support a last work-group in each dimension being partial.
4326 This count only includes the non-partial work-group count.
4327 This is not the same as the value in the AQL dispatch packet,
4328 which has the grid size in work-items. If the grid dimensionality
4329 is 1, then must be 1.
4331 "hidden_block_count_z"
4332 The grid dispatch work-group count for the Z dimension
4333 is passed in the kernarg. Some languages, such as OpenCL,
4334 support a last work-group in each dimension being partial.
4335 This count only includes the non-partial work-group count.
4336 This is not the same as the value in the AQL dispatch packet,
4337 which has the grid size in work-items. If the grid dimensionality
4338 is 1 or 2, then must be 1.
4340 "hidden_group_size_x"
4341 The grid dispatch work-group size for the X dimension is
4342 passed in the kernarg. This size only applies to the
4343 non-partial work-groups. This is the same value as the AQL
4344 dispatch packet work-group size.
4346 "hidden_group_size_y"
4347 The grid dispatch work-group size for the Y dimension is
4348 passed in the kernarg. This size only applies to the
4349 non-partial work-groups. This is the same value as the AQL
4350 dispatch packet work-group size. If the grid dimensionality
4351 is 1, then must be 1.
4353 "hidden_group_size_z"
4354 The grid dispatch work-group size for the Z dimension is
4355 passed in the kernarg. This size only applies to the
4356 non-partial work-groups. This is the same value as the AQL
4357 dispatch packet work-group size. If the grid dimensionality
4358 is 1 or 2, then must be 1.
4360 "hidden_remainder_x"
4361 The grid dispatch work group size of the partial work group
4362 of the X dimension, if it exists. Must be zero if a partial
4363 work group does not exist in the X dimension.
4365 "hidden_remainder_y"
4366 The grid dispatch work group size of the partial work group
4367 of the Y dimension, if it exists. Must be zero if a partial
4368 work group does not exist in the Y dimension.
4370 "hidden_remainder_z"
4371 The grid dispatch work group size of the partial work group
4372 of the Z dimension, if it exists. Must be zero if a partial
4373 work group does not exist in the Z dimension.
4375 "hidden_grid_dims"
4376 The grid dispatch dimensionality. This is the same value
4377 as the AQL dispatch packet dimensionality. Must be a value
4378 between 1 and 3.
4380 "hidden_heap_v1"
4381 A global address space pointer to an initialized memory
4382 buffer that conforms to the requirements of the malloc/free
4383 device library V1 version implementation.
4385 "hidden_dynamic_lds_size"
4386 Size of the dynamically allocated LDS memory is passed in the kernarg.
4388 "hidden_private_base"
4389 The high 32 bits of the flat addressing private aperture base.
4390 Only used by GFX8 to allow conversion between private segment
4391 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.
4393 "hidden_shared_base"
4394 The high 32 bits of the flat addressing shared aperture base.
4395 Only used by GFX8 to allow conversion between shared segment
4396 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.
4398 "hidden_queue_ptr"
4399 A global memory address space pointer to the ROCm runtime
4400 ``struct amd_queue_t`` structure for the HSA queue of the
4401 associated dispatch AQL packet. It is only required for pre-GFX9
4402 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`).
4404 ====================== ============== ========= ================================
4406 ..
4408 Kernel Dispatch
4409 ~~~~~~~~~~~~~~~
4411 The HSA architected queuing language (AQL) defines a user space memory interface
4412 that can be used to control the dispatch of kernels, in an agent independent
4413 way. An agent can have zero or more AQL queues created for it using an HSA
4414 compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which
4415 are 64 bytes) can be placed. See the *HSA Platform System Architecture
4416 Specification* [HSA]_ for the AQL queue mechanics and packet layouts.
4418 The packet processor of a kernel agent is responsible for detecting and
4419 dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the
4420 packet processor is implemented by the hardware command processor (CP),
4421 asynchronous dispatch controller (ADC) and shader processor input controller
4422 (SPI).
4424 An HSA compatible runtime can be used to allocate an AQL queue object. It uses
4425 the kernel mode driver to initialize and register the AQL queue with CP.
4427 To dispatch a kernel the following actions are performed. This can occur in the
4428 CPU host program, or from an HSA kernel executing on a GPU.
4430 1. A pointer to an AQL queue for the kernel agent on which the kernel is to be
4431 executed is obtained.
4432 2. A pointer to the kernel descriptor (see
4433 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained.
4434 It must be for a kernel that is contained in a code object that was loaded
4435 by an HSA compatible runtime on the kernel agent with which the AQL queue is
4436 associated.
4437 3. Space is allocated for the kernel arguments using the HSA compatible runtime
4438 allocator for a memory region with the kernarg property for the kernel agent
4439 that will execute the kernel. It must be at least 16-byte aligned.
4440 4. Kernel argument values are assigned to the kernel argument memory
4441 allocation. The layout is defined in the *HSA Programmer's Language
4442 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the
4443 kernel argument memory in the same way constant memory is accessed. (Note
4444 that the HSA specification allows an implementation to copy the kernel
4445 argument contents to another location that is accessed by the kernel.)
4446 5. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible
4447 runtime api uses 64-bit atomic operations to reserve space in the AQL queue
4448 for the packet. The packet must be set up, and the final write must use an
4449 atomic store release to set the packet kind to ensure the packet contents are
4450 visible to the kernel agent. AQL defines a doorbell signal mechanism to
4451 notify the kernel agent that the AQL queue has been updated. These rules, and
4452 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA
4453 System Architecture Specification* [HSA]_.
4454 6. A kernel dispatch packet includes information about the actual dispatch,
4455 such as grid and work-group size, together with information from the code
4456 object about the kernel, such as segment sizes. The HSA compatible runtime
4457 queries on the kernel symbol can be used to obtain the code object values
4458 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`.
4459 7. CP executes micro-code and is responsible for detecting and setting up the
4460 GPU to execute the wavefronts of a kernel dispatch.
4461 8. CP ensures that when the a wavefront starts executing the kernel machine
4462 code, the scalar general purpose registers (SGPR) and vector general purpose
4463 registers (VGPR) are set up as required by the machine code. The required
4464 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial
4465 register state is defined in
4466 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`.
4467 9. The prolog of the kernel machine code (see
4468 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary
4469 before continuing executing the machine code that corresponds to the kernel.
4470 10. When the kernel dispatch has completed execution, CP signals the completion
4471 signal specified in the kernel dispatch packet if not 0.
4473 .. _amdgpu-amdhsa-memory-spaces:
4475 Memory Spaces
4476 ~~~~~~~~~~~~~
4478 The memory space properties are:
4480 .. table:: AMDHSA Memory Spaces
4481 :name: amdgpu-amdhsa-memory-spaces-table
4483 ================= =========== ======== ======= ==================
4484 Memory Space Name HSA Segment Hardware Address NULL Value
4485 Name Name Size
4486 ================= =========== ======== ======= ==================
4487 Private private scratch 32 0x00000000
4488 Local group LDS 32 0xFFFFFFFF
4489 Global global global 64 0x0000000000000000
4490 Constant constant *same as 64 0x0000000000000000
4491 global*
4492 Generic flat flat 64 0x0000000000000000
4493 Region N/A GDS 32 *not implemented
4494 for AMDHSA*
4495 ================= =========== ======== ======= ==================
4497 The global and constant memory spaces both use global virtual addresses, which
4498 are the same virtual address space used by the CPU. However, some virtual
4499 addresses may only be accessible to the CPU, some only accessible by the GPU,
4500 and some by both.
4502 Using the constant memory space indicates that the data will not change during
4503 the execution of the kernel. This allows scalar read instructions to be
4504 used. The vector and scalar L1 caches are invalidated of volatile data before
4505 each kernel dispatch execution to allow constant memory to change values between
4506 kernel dispatches.
4508 The local memory space uses the hardware Local Data Store (LDS) which is
4509 automatically allocated when the hardware creates work-groups of wavefronts, and
4510 freed when all the wavefronts of a work-group have terminated. The data store
4511 (DS) instructions can be used to access it.
4513 The private memory space uses the hardware scratch memory support. If the kernel
4514 uses scratch, then the hardware allocates memory that is accessed using
4515 wavefront lane dword (4 byte) interleaving. The mapping used from private
4516 address to physical address is:
4518 ``wavefront-scratch-base +
4519 (private-address * wavefront-size * 4) +
4520 (wavefront-lane-id * 4)``
4522 There are different ways that the wavefront scratch base address is determined
4523 by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This
4524 memory can be accessed in an interleaved manner using buffer instruction with
4525 the scratch buffer descriptor and per wavefront scratch offset, by the scratch
4526 instructions, or by flat instructions. If each lane of a wavefront accesses the
4527 same private address, the interleaving results in adjacent dwords being accessed
4528 and hence requires fewer cache lines to be fetched. Multi-dword access is not
4529 supported except by flat and scratch instructions in GFX9-GFX11.
4531 The generic address space uses the hardware flat address support available in
4532 GFX7-GFX11. This uses two fixed ranges of virtual addresses (the private and
4533 local apertures), that are outside the range of addressible global memory, to
4534 map from a flat address to a private or local address.
4536 FLAT instructions can take a flat address and access global, private (scratch)
4537 and group (LDS) memory depending on if the address is within one of the
4538 aperture ranges. Flat access to scratch requires hardware aperture setup and
4539 setup in the kernel prologue (see
4540 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires
4541 hardware aperture setup and M0 (GFX7-GFX8) register setup (see
4542 :ref:`amdgpu-amdhsa-kernel-prolog-m0`).
4544 To convert between a segment address and a flat address the base address of the
4545 apertures address can be used. For GFX7-GFX8 these are available in the
4546 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with
4547 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For
4548 GFX9-GFX11 the aperture base addresses are directly available as inline constant
4549 registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit
4550 address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32
4551 which makes it easier to convert from flat to segment or segment to flat.
4553 Image and Samplers
4554 ~~~~~~~~~~~~~~~~~~
4556 Image and sample handles created by an HSA compatible runtime (see
4557 :ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S#
4558 object respectively. In order to support the HSA ``query_sampler`` operations
4559 two extra dwords are used to store the HSA BRIG enumeration values for the
4560 queries that are not trivially deducible from the S# representation.
4562 HSA Signals
4563 ~~~~~~~~~~~
4565 HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`)
4566 are 64-bit addresses of a structure allocated in memory accessible from both the
4567 CPU and GPU. The structure is defined by the runtime and subject to change
4568 between releases. For example, see [AMD-ROCm-github]_.
4570 .. _amdgpu-amdhsa-hsa-aql-queue:
4572 HSA AQL Queue
4573 ~~~~~~~~~~~~~
4575 The HSA AQL queue structure is defined by an HSA compatible runtime (see
4576 :ref:`amdgpu-os`) and subject to change between releases. For example, see
4577 [AMD-ROCm-github]_. For some processors it contains fields needed to implement
4578 certain language features such as the flat address aperture bases. It also
4579 contains fields used by CP such as managing the allocation of scratch memory.
4581 .. _amdgpu-amdhsa-kernel-descriptor:
4583 Kernel Descriptor
4584 ~~~~~~~~~~~~~~~~~
4586 A kernel descriptor consists of the information needed by CP to initiate the
4587 execution of a kernel, including the entry point address of the machine code
4588 that implements the kernel.
4590 Code Object V3 Kernel Descriptor
4591 ++++++++++++++++++++++++++++++++
4593 CP microcode requires the Kernel descriptor to be allocated on 64-byte
4594 alignment.
4596 The fields used by CP for code objects before V3 also match those specified in
4597 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
4599 .. table:: Code Object V3 Kernel Descriptor
4600 :name: amdgpu-amdhsa-kernel-descriptor-v3-table
4602 ======= ======= =============================== ============================
4603 Bits Size Field Name Description
4604 ======= ======= =============================== ============================
4605 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local
4606 address space memory
4607 required for a work-group
4608 in bytes. This does not
4609 include any dynamically
4610 allocated local address
4611 space memory that may be
4612 added when the kernel is
4613 dispatched.
4614 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed
4615 private address space
4616 memory required for a
4617 work-item in bytes. When
4618 this cannot be predicted,
4619 code object v4 and older
4620 sets this value to be
4621 higher than the minimum
4622 requirement.
4623 95:64 4 bytes KERNARG_SIZE The size of the kernarg
4624 memory pointed to by the
4625 AQL dispatch packet. The
4626 kernarg memory is used to
4627 pass arguments to the
4628 kernel.
4630 * If the kernarg pointer in
4631 the dispatch packet is NULL
4632 then there are no kernel
4633 arguments.
4634 * If the kernarg pointer in
4635 the dispatch packet is
4636 not NULL and this value
4637 is 0 then the kernarg
4638 memory size is
4639 unspecified.
4640 * If the kernarg pointer in
4641 the dispatch packet is
4642 not NULL and this value
4643 is not 0 then the value
4644 specifies the kernarg
4645 memory size in bytes. It
4646 is recommended to provide
4647 a value as it may be used
4648 by CP to optimize making
4649 the kernarg memory
4650 visible to the kernel
4651 code.
4653 127:96 4 bytes Reserved, must be 0.
4654 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly
4655 negative) from base
4656 address of kernel
4657 descriptor to kernel's
4658 entry point instruction
4659 which must be 256 byte
4660 aligned.
4661 351:192 20 Reserved, must be 0.
4662 bytes
4663 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9
4664 Reserved, must be 0.
4665 GFX90A, GFX940
4666 Compute Shader (CS)
4667 program settings used by
4668 CP to set up
4669 ``COMPUTE_PGM_RSRC3``
4670 configuration
4671 register. See
4672 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`.
4673 GFX10-GFX11
4674 Compute Shader (CS)
4675 program settings used by
4676 CP to set up
4677 ``COMPUTE_PGM_RSRC3``
4678 configuration
4679 register. See
4680 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-gfx11-table`.
4681 GFX12
4682 Compute Shader (CS)
4683 program settings used by
4684 CP to set up
4685 ``COMPUTE_PGM_RSRC3``
4686 configuration
4687 register. See
4688 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx12-table`.
4689 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS)
4690 program settings used by
4691 CP to set up
4692 ``COMPUTE_PGM_RSRC1``
4693 configuration
4694 register. See
4695 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
4696 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS)
4697 program settings used by
4698 CP to set up
4699 ``COMPUTE_PGM_RSRC2``
4700 configuration
4701 register. See
4702 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
4703 458:448 7 bits *See separate bits below.* Enable the setup of the
4704 SGPR user data registers
4705 (see
4706 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
4708 The total number of SGPR
4709 user data registers
4710 requested must not exceed
4711 16 and match value in
4712 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``.
4713 Any requests beyond 16
4714 will be ignored.
4715 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties*
4716 _BUFFER column of
4717 :ref:`amdgpu-processor-table`
4718 specifies *Architected flat
4719 scratch* then not supported
4720 and must be 0,
4721 >449 1 bit ENABLE_SGPR_DISPATCH_PTR
4722 >450 1 bit ENABLE_SGPR_QUEUE_PTR
4723 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR
4724 >452 1 bit ENABLE_SGPR_DISPATCH_ID
4725 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties*
4726 column of
4727 :ref:`amdgpu-processor-table`
4728 specifies *Architected flat
4729 scratch* then not supported
4730 and must be 0,
4731 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT
4732 _SIZE
4733 457:455 3 bits Reserved, must be 0.
4734 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9
4735 Reserved, must be 0.
4736 GFX10-GFX11
4737 - If 0 execute in
4738 wavefront size 64 mode.
4739 - If 1 execute in
4740 native wavefront size
4741 32 mode.
4742 459 1 bit USES_DYNAMIC_STACK Indicates if the generated
4743 machine code is using a
4744 dynamically sized stack.
4745 This is only set in code
4746 object v5 and later.
4747 463:460 4 bits Reserved, must be 0.
4748 470:464 7 bits KERNARG_PRELOAD_SPEC_LENGTH GFX6-GFX9
4749 - Reserved, must be 0.
4750 GFX90A, GFX940
4751 - The number of dwords from
4752 the kernarg segment to preload
4753 into User SGPRs before kernel
4754 execution. (see
4755 :ref:`amdgpu-amdhsa-kernarg-preload`).
4756 479:471 9 bits KERNARG_PRELOAD_SPEC_OFFSET GFX6-GFX9
4757 - Reserved, must be 0.
4758 GFX90A, GFX940
4759 - An offset in dwords into the
4760 kernarg segment to begin
4761 preloading data into User
4762 SGPRs. (see
4763 :ref:`amdgpu-amdhsa-kernarg-preload`).
4764 511:480 4 bytes Reserved, must be 0.
4765 512 **Total size 64 bytes.**
4766 ======= ====================================================================
4768 ..
4770 .. table:: compute_pgm_rsrc1 for GFX6-GFX12
4771 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table
4773 ======= ======= =============================== ===========================================================================
4774 Bits Size Field Name Description
4775 ======= ======= =============================== ===========================================================================
4776 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register
4777 blocks used by each work-item;
4778 granularity is device
4779 specific:
4781 GFX6-GFX9
4782 - vgprs_used 0..256
4783 - max(0, ceil(vgprs_used / 4) - 1)
4784 GFX90A, GFX940
4785 - vgprs_used 0..512
4786 - vgprs_used = align(arch_vgprs, 4)
4787 + acc_vgprs
4788 - max(0, ceil(vgprs_used / 8) - 1)
4789 GFX10-GFX11 (wavefront size 64)
4790 - max_vgpr 1..256
4791 - max(0, ceil(vgprs_used / 4) - 1)
4792 GFX10-GFX11 (wavefront size 32)
4793 - max_vgpr 1..256
4794 - max(0, ceil(vgprs_used / 8) - 1)
4796 Where vgprs_used is defined
4797 as the highest VGPR number
4798 explicitly referenced plus
4799 one.
4801 Used by CP to set up
4802 ``COMPUTE_PGM_RSRC1.VGPRS``.
4804 The
4805 :ref:`amdgpu-assembler`
4806 calculates this
4807 automatically for the
4808 selected processor from
4809 values provided to the
4810 `.amdhsa_kernel` directive
4811 by the
4812 `.amdhsa_next_free_vgpr`
4813 nested directive (see
4814 :ref:`amdhsa-kernel-directives-table`).
4815 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register
4816 blocks used by a wavefront;
4817 granularity is device
4818 specific:
4820 GFX6-GFX8
4821 - sgprs_used 0..112
4822 - max(0, ceil(sgprs_used / 8) - 1)
4823 GFX9
4824 - sgprs_used 0..112
4825 - 2 * max(0, ceil(sgprs_used / 16) - 1)
4826 GFX10-GFX11
4827 Reserved, must be 0.
4828 (128 SGPRs always
4829 allocated.)
4831 Where sgprs_used is
4832 defined as the highest
4833 SGPR number explicitly
4834 referenced plus one, plus
4835 a target specific number
4836 of additional special
4837 SGPRs for VCC,
4838 FLAT_SCRATCH (GFX7+) and
4839 XNACK_MASK (GFX8+), and
4840 any additional
4841 target specific
4842 limitations. It does not
4843 include the 16 SGPRs added
4844 if a trap handler is
4845 enabled.
4847 The target specific
4848 limitations and special
4849 SGPR layout are defined in
4850 the hardware
4851 documentation, which can
4852 be found in the
4853 :ref:`amdgpu-processors`
4854 table.
4856 Used by CP to set up
4857 ``COMPUTE_PGM_RSRC1.SGPRS``.
4859 The
4860 :ref:`amdgpu-assembler`
4861 calculates this
4862 automatically for the
4863 selected processor from
4864 values provided to the
4865 `.amdhsa_kernel` directive
4866 by the
4867 `.amdhsa_next_free_sgpr`
4868 and `.amdhsa_reserve_*`
4869 nested directives (see
4870 :ref:`amdhsa-kernel-directives-table`).
4871 11:10 2 bits PRIORITY Must be 0.
4873 Start executing wavefront
4874 at the specified priority.
4876 CP is responsible for
4877 filling in
4878 ``COMPUTE_PGM_RSRC1.PRIORITY``.
4879 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution
4880 with specified rounding
4881 mode for single (32
4882 bit) floating point
4883 precision floating point
4884 operations.
4886 Floating point rounding
4887 mode values are defined in
4888 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`.
4890 Used by CP to set up
4891 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``.
4892 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution
4893 with specified rounding
4894 denorm mode for half/double (16
4895 and 64-bit) floating point
4896 precision floating point
4897 operations.
4899 Floating point rounding
4900 mode values are defined in
4901 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`.
4903 Used by CP to set up
4904 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``.
4905 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution
4906 with specified denorm mode
4907 for single (32
4908 bit) floating point
4909 precision floating point
4910 operations.
4912 Floating point denorm mode
4913 values are defined in
4914 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`.
4916 Used by CP to set up
4917 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``.
4918 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution
4919 with specified denorm mode
4920 for half/double (16
4921 and 64-bit) floating point
4922 precision floating point
4923 operations.
4925 Floating point denorm mode
4926 values are defined in
4927 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`.
4929 Used by CP to set up
4930 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``.
4931 20 1 bit PRIV Must be 0.
4933 Start executing wavefront
4934 in privilege trap handler
4935 mode.
4937 CP is responsible for
4938 filling in
4939 ``COMPUTE_PGM_RSRC1.PRIV``.
4940 21 1 bit ENABLE_DX10_CLAMP GFX9-GFX11
4941 Wavefront starts execution
4942 with DX10 clamp mode
4943 enabled. Used by the vector
4944 ALU to force DX10 style
4945 treatment of NaN's (when
4946 set, clamp NaN to zero,
4947 otherwise pass NaN
4948 through).
4950 Used by CP to set up
4951 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``.
4952 WG_RR_EN GFX12
4953 If 1, wavefronts are scheduled
4954 in a round-robin fashion with
4955 respect to the other wavefronts
4956 of the SIMD. Otherwise, wavefronts
4957 are scheduled in oldest age order.
4959 CP is responsible for filling in
4960 ``COMPUTE_PGM_RSRC1.WG_RR_EN``.
4961 22 1 bit DEBUG_MODE Must be 0.
4963 Start executing wavefront
4964 in single step mode.
4966 CP is responsible for
4967 filling in
4968 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``.
4969 23 1 bit ENABLE_IEEE_MODE GFX9-GFX11
4970 Wavefront starts execution
4971 with IEEE mode
4972 enabled. Floating point
4973 opcodes that support
4974 exception flag gathering
4975 will quiet and propagate
4976 signaling-NaN inputs per
4977 IEEE 754-2008. Min_dx10 and
4978 max_dx10 become IEEE
4979 754-2008 compliant due to
4980 signaling-NaN propagation
4981 and quieting.
4983 Used by CP to set up
4984 ``COMPUTE_PGM_RSRC1.IEEE_MODE``.
4985 DISABLE_PERF GFX12
4986 Reserved. Must be 0.
4987 24 1 bit BULKY Must be 0.
4989 Only one work-group allowed
4990 to execute on a compute
4991 unit.
4993 CP is responsible for
4994 filling in
4995 ``COMPUTE_PGM_RSRC1.BULKY``.
4996 25 1 bit CDBG_USER Must be 0.
4998 Flag that can be used to
4999 control debugging code.
5001 CP is responsible for
5002 filling in
5003 ``COMPUTE_PGM_RSRC1.CDBG_USER``.
5004 26 1 bit FP16_OVFL GFX6-GFX8
5005 Reserved, must be 0.
5006 GFX9-GFX11
5007 Wavefront starts execution
5008 with specified fp16 overflow
5009 mode.
5011 - If 0, fp16 overflow generates
5012 +/-INF values.
5013 - If 1, fp16 overflow that is the
5014 result of an +/-INF input value
5015 or divide by 0 produces a +/-INF,
5016 otherwise clamps computed
5017 overflow to +/-MAX_FP16 as
5018 appropriate.
5020 Used by CP to set up
5021 ``COMPUTE_PGM_RSRC1.FP16_OVFL``.
5022 28:27 2 bits Reserved, must be 0.
5023 29 1 bit WGP_MODE GFX6-GFX9
5024 Reserved, must be 0.
5025 GFX10-GFX11
5026 - If 0 execute work-groups in
5027 CU wavefront execution mode.
5028 - If 1 execute work-groups on
5029 in WGP wavefront execution mode.
5031 See :ref:`amdgpu-amdhsa-memory-model`.
5033 Used by CP to set up
5034 ``COMPUTE_PGM_RSRC1.WGP_MODE``.
5035 30 1 bit MEM_ORDERED GFX6-GFX9
5036 Reserved, must be 0.
5037 GFX10-GFX11
5038 Controls the behavior of the
5039 s_waitcnt's vmcnt and vscnt
5040 counters.
5042 - If 0 vmcnt reports completion
5043 of load and atomic with return
5044 out of order with sample
5045 instructions, and the vscnt
5046 reports the completion of
5047 store and atomic without
5048 return in order.
5049 - If 1 vmcnt reports completion
5050 of load, atomic with return
5051 and sample instructions in
5052 order, and the vscnt reports
5053 the completion of store and
5054 atomic without return in order.
5056 Used by CP to set up
5057 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``.
5058 31 1 bit FWD_PROGRESS GFX6-GFX9
5059 Reserved, must be 0.
5060 GFX10-GFX11
5061 - If 0 execute SIMD wavefronts
5062 using oldest first policy.
5063 - If 1 execute SIMD wavefronts to
5064 ensure wavefronts will make some
5065 forward progress.
5067 Used by CP to set up
5068 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``.
5069 32 **Total size 4 bytes**
5070 ======= ===================================================================================================================
5072 ..
5074 .. table:: compute_pgm_rsrc2 for GFX6-GFX12
5075 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table
5077 ======= ======= =============================== ===========================================================================
5078 Bits Size Field Name Description
5079 ======= ======= =============================== ===========================================================================
5080 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the
5081 private segment.
5082 * If the *Target Properties*
5083 column of
5084 :ref:`amdgpu-processor-table`
5085 does not specify
5086 *Architected flat
5087 scratch* then enable the
5088 setup of the SGPR
5089 wavefront scratch offset
5090 system register (see
5091 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
5092 * If the *Target Properties*
5093 column of
5094 :ref:`amdgpu-processor-table`
5095 specifies *Architected
5096 flat scratch* then enable
5097 the setup of the
5098 FLAT_SCRATCH register
5099 pair (see
5100 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
5102 Used by CP to set up
5103 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``.
5104 5:1 5 bits USER_SGPR_COUNT The total number of SGPR
5105 user data
5106 registers requested. This
5107 number must be greater than
5108 or equal to the number of user
5109 data registers enabled.
5111 Used by CP to set up
5112 ``COMPUTE_PGM_RSRC2.USER_SGPR``.
5113 6 1 bit ENABLE_TRAP_HANDLER GFX6-GFX11
5114 Must be 0.
5116 This bit represents
5117 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``,
5118 which is set by the CP if
5119 the runtime has installed a
5120 trap handler.
5121 GFX12
5122 Reserved, must be 0.
5123 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the
5124 system SGPR register for
5125 the work-group id in the X
5126 dimension (see
5127 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
5129 Used by CP to set up
5130 ``COMPUTE_PGM_RSRC2.TGID_X_EN``.
5131 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the
5132 system SGPR register for
5133 the work-group id in the Y
5134 dimension (see
5135 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
5137 Used by CP to set up
5138 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``.
5139 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the
5140 system SGPR register for
5141 the work-group id in the Z
5142 dimension (see
5143 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
5145 Used by CP to set up
5146 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``.
5147 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the
5148 system SGPR register for
5149 work-group information (see
5150 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
5152 Used by CP to set up
5153 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``.
5154 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the
5155 VGPR system registers used
5156 for the work-item ID.
5157 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`
5158 defines the values.
5160 Used by CP to set up
5161 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``.
5162 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0.
5164 Wavefront starts execution
5165 with address watch
5166 exceptions enabled which
5167 are generated when L1 has
5168 witnessed a thread access
5169 an *address of
5170 interest*.
5172 CP is responsible for
5173 filling in the address
5174 watch bit in
5175 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB``
5176 according to what the
5177 runtime requests.
5178 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0.
5180 Wavefront starts execution
5181 with memory violation
5182 exceptions exceptions
5183 enabled which are generated
5184 when a memory violation has
5185 occurred for this wavefront from
5186 L1 or LDS
5187 (write-to-read-only-memory,
5188 mis-aligned atomic, LDS
5189 address out of range,
5190 illegal address, etc.).
5192 CP sets the memory
5193 violation bit in
5194 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB``
5195 according to what the
5196 runtime requests.
5197 23:15 9 bits GRANULATED_LDS_SIZE Must be 0.
5199 CP uses the rounded value
5200 from the dispatch packet,
5201 not this value, as the
5202 dispatch may contain
5203 dynamically allocated group
5204 segment memory. CP writes
5205 directly to
5206 ``COMPUTE_PGM_RSRC2.LDS_SIZE``.
5208 Amount of group segment
5209 (LDS) to allocate for each
5210 work-group. Granularity is
5211 device specific:
5213 GFX6
5214 roundup(lds-size / (64 * 4))
5215 GFX7-GFX11
5216 roundup(lds-size / (128 * 4))
5218 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution
5219 _INVALID_OPERATION with specified exceptions
5220 enabled.
5222 Used by CP to set up
5223 ``COMPUTE_PGM_RSRC2.EXCP_EN``
5224 (set from bits 0..6).
5226 IEEE 754 FP Invalid
5227 Operation
5228 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more
5229 _SOURCE input operands is a
5230 denormal number
5231 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by
5232 _DIVISION_BY_ZERO Zero
5233 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow
5234 _OVERFLOW
5235 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow
5236 _UNDERFLOW
5237 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact
5238 _INEXACT
5239 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero
5240 _ZERO (rcp_iflag_f32 instruction
5241 only)
5242 31 1 bit RESERVED Reserved, must be 0.
5243 32 **Total size 4 bytes.**
5244 ======= ===================================================================================================================
5246 ..
5248 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940
5249 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table
5251 ======= ======= =============================== ===========================================================================
5252 Bits Size Field Name Description
5253 ======= ======= =============================== ===========================================================================
5254 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4.
5255 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ...,
5256 63 - accum-offset = 256.
5257 15:6 10 Reserved, must be 0.
5258 bits
5259 16 1 bit TG_SPLIT - If 0 the waves of a work-group are
5260 launched in the same CU.
5261 - If 1 the waves of a work-group can be
5262 launched in different CUs. The waves
5263 cannot use S_BARRIER or LDS.
5264 31:17 15 Reserved, must be 0.
5265 bits
5266 32 **Total size 4 bytes.**
5267 ======= ===================================================================================================================
5269 ..
5271 .. table:: compute_pgm_rsrc3 for GFX10-GFX11
5272 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-gfx11-table
5274 ======= ======= =============================== ===========================================================================
5275 Bits Size Field Name Description
5276 ======= ======= =============================== ===========================================================================
5277 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For
5278 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity
5279 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does
5280 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0.
5281 9:4 6 bits INST_PREF_SIZE GFX10
5282 Reserved, must be 0.
5283 GFX11
5284 Number of instruction bytes to prefetch, starting at the kernel's entry
5285 point instruction, before wavefront starts execution. The value is 0..63
5286 with a granularity of 128 bytes.
5287 10 1 bit TRAP_ON_START GFX10
5288 Reserved, must be 0.
5289 GFX11
5290 Must be 0.
5292 If 1, wavefront starts execution by trapping into the trap handler.
5294 CP is responsible for filling in the trap on start bit in
5295 ``COMPUTE_PGM_RSRC3.TRAP_ON_START`` according to what the runtime
5296 requests.
5297 11 1 bit TRAP_ON_END GFX10
5298 Reserved, must be 0.
5299 GFX11
5300 Must be 0.
5302 If 1, wavefront execution terminates by trapping into the trap handler.
5304 CP is responsible for filling in the trap on end bit in
5305 ``COMPUTE_PGM_RSRC3.TRAP_ON_END`` according to what the runtime requests.
5306 30:12 19 bits Reserved, must be 0.
5307 31 1 bit IMAGE_OP GFX10
5308 Reserved, must be 0.
5309 GFX11
5310 If 1, the kernel execution contains image instructions. If executed as
5311 part of a graphics pipeline, image read instructions will stall waiting
5312 for any necessary ``WAIT_SYNC`` fence to be performed in order to
5313 indicate that earlier pipeline stages have completed writing to the
5314 image.
5316 Not used for compute kernels that are not part of a graphics pipeline and
5317 must be 0.
5318 32 **Total size 4 bytes.**
5319 ======= ===================================================================================================================
5321 ..
5323 .. table:: compute_pgm_rsrc3 for GFX12
5324 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx12-table
5326 ======= ======= =============================== ===========================================================================
5327 Bits Size Field Name Description
5328 ======= ======= =============================== ===========================================================================
5329 3:0 4 bits RESERVED Reserved, must be 0.
5330 11:4 8 bits INST_PREF_SIZE Number of instruction bytes to prefetch, starting at the kernel's entry
5331 point instruction, before wavefront starts execution. The value is 0..255
5332 with a granularity of 128 bytes.
5333 12 1 bit RESERVED Reserved, must be 0.
5334 13 1 bit GLG_EN If 1, group launch guarantee will be enabled for this dispatch
5335 30:14 17 bits RESERVED Reserved, must be 0.
5336 31 1 bit IMAGE_OP If 1, the kernel execution contains image instructions. If executed as
5337 part of a graphics pipeline, image read instructions will stall waiting
5338 for any necessary ``WAIT_SYNC`` fence to be performed in order to
5339 indicate that earlier pipeline stages have completed writing to the
5340 image.
5342 Not used for compute kernels that are not part of a graphics pipeline and
5343 must be 0.
5344 32 **Total size 4 bytes.**
5345 ======= ===================================================================================================================
5347 ..
5349 .. table:: Floating Point Rounding Mode Enumeration Values
5350 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table
5352 ====================================== ===== ==============================
5353 Enumeration Name Value Description
5354 ====================================== ===== ==============================
5355 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even
5356 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity
5357 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity
5358 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0
5359 ====================================== ===== ==============================
5362 .. table:: Extended FLT_ROUNDS Enumeration Values
5363 :name: amdgpu-rounding-mode-enumeration-values-table
5365 +------------------------+---------------+-------------------+--------------------+----------+
5366 | | F32 NEAR_EVEN | F32 PLUS_INFINITY | F32 MINUS_INFINITY | F32 ZERO |
5367 +------------------------+---------------+-------------------+--------------------+----------+
5368 | F64/F16 NEAR_EVEN | 1 | 11 | 14 | 17 |
5369 +------------------------+---------------+-------------------+--------------------+----------+
5370 | F64/F16 PLUS_INFINITY | 8 | 2 | 15 | 18 |
5371 +------------------------+---------------+-------------------+--------------------+----------+
5372 | F64/F16 MINUS_INFINITY | 9 | 12 | 3 | 19 |
5373 +------------------------+---------------+-------------------+--------------------+----------+
5374 | F64/F16 ZERO | 10 | 13 | 16 | 0 |
5375 +------------------------+---------------+-------------------+--------------------+----------+
5377 ..
5379 .. table:: Floating Point Denorm Mode Enumeration Values
5380 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table
5382 ====================================== ===== ====================================
5383 Enumeration Name Value Description
5384 ====================================== ===== ====================================
5385 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination Denorms
5386 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms
5387 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms
5388 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush
5389 ====================================== ===== ====================================
5391 Denormal flushing is sign respecting. i.e. the behavior expected by
5392 ``"denormal-fp-math"="preserve-sign"``. The behavior is undefined with
5393 ``"denormal-fp-math"="positive-zero"``
5395 ..
5397 .. table:: System VGPR Work-Item ID Enumeration Values
5398 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table
5400 ======================================== ===== ============================
5401 Enumeration Name Value Description
5402 ======================================== ===== ============================
5403 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension
5404 ID.
5405 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y
5406 dimensions ID.
5407 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z
5408 dimensions ID.
5409 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined.
5410 ======================================== ===== ============================
5412 .. _amdgpu-amdhsa-initial-kernel-execution-state:
5414 Initial Kernel Execution State
5415 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5417 This section defines the register state that will be set up by the packet
5418 processor prior to the start of execution of every wavefront. This is limited by
5419 the constraints of the hardware controllers of CP/ADC/SPI.
5421 The order of the SGPR registers is defined, but the compiler can specify which
5422 ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit
5423 fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used
5424 for enabled registers are dense starting at SGPR0: the first enabled register is
5425 SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have
5426 an SGPR number.
5428 The initial SGPRs comprise up to 16 User SGPRs that are set by CP and apply to
5429 all wavefronts of the grid. It is possible to specify more than 16 User SGPRs
5430 using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are
5431 actually initialized. These are then immediately followed by the System SGPRs
5432 that are set up by ADC/SPI and can have different values for each wavefront of
5433 the grid dispatch.
5435 SGPR register initial state is defined in
5436 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.
5438 .. table:: SGPR Register Set Up Order
5439 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table
5441 ========== ========================== ====== ==============================
5442 SGPR Order Name Number Description
5443 (kernel descriptor enable of
5444 field) SGPRs
5445 ========== ========================== ====== ==============================
5446 First Private Segment Buffer 4 See
5447 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`.
5448 _segment_buffer)
5449 then Dispatch Ptr 2 64-bit address of AQL dispatch
5450 (enable_sgpr_dispatch_ptr) packet for kernel dispatch
5451 actually executing.
5452 then Queue Ptr 2 64-bit address of amd_queue_t
5453 (enable_sgpr_queue_ptr) object for AQL queue on which
5454 the dispatch packet was
5455 queued.
5456 then Kernarg Segment Ptr 2 64-bit address of Kernarg
5457 (enable_sgpr_kernarg segment. This is directly
5458 _segment_ptr) copied from the
5459 kernarg_address in the kernel
5460 dispatch packet.
5462 Having CP load it once avoids
5463 loading it at the beginning of
5464 every wavefront.
5465 then Dispatch Id 2 64-bit Dispatch ID of the
5466 (enable_sgpr_dispatch_id) dispatch packet being
5467 executed.
5468 then Flat Scratch Init 2 See
5469 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.
5470 _init)
5471 then Preloaded Kernargs N/A See
5472 (kernarg_preload_spec :ref:`amdgpu-amdhsa-kernarg-preload`.
5473 _length)
5474 then Private Segment Size 1 The 32-bit byte size of a
5475 (enable_sgpr_private single work-item's memory
5476 _segment_size) allocation. This is the
5477 value from the kernel
5478 dispatch packet Private
5479 Segment Byte Size rounded up
5480 by CP to a multiple of
5481 DWORD.
5483 Having CP load it once avoids
5484 loading it at the beginning of
5485 every wavefront.
5487 This is not used for
5488 GFX7-GFX8 since it is the same
5489 value as the second SGPR of
5490 Flat Scratch Init. However, it
5491 may be needed for GFX9-GFX11 which
5492 changes the meaning of the
5493 Flat Scratch Init value.
5494 then Work-Group Id X 1 32-bit work-group id in X
5495 (enable_sgpr_workgroup_id dimension of grid for
5496 _X) wavefront.
5497 then Work-Group Id Y 1 32-bit work-group id in Y
5498 (enable_sgpr_workgroup_id dimension of grid for
5499 _Y) wavefront.
5500 then Work-Group Id Z 1 32-bit work-group id in Z
5501 (enable_sgpr_workgroup_id dimension of grid for
5502 _Z) wavefront.
5503 then Work-Group Info 1 {first_wavefront, 14'b0000,
5504 (enable_sgpr_workgroup ordered_append_term[10:0],
5505 _info) threadgroup_size_in_wavefronts[5:0]}
5506 then Scratch Wavefront Offset 1 See
5507 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.
5508 _segment_wavefront_offset) and
5509 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`.
5510 ========== ========================== ====== ==============================
5512 The order of the VGPR registers is defined, but the compiler can specify which
5513 ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit
5514 fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used
5515 for enabled registers are dense starting at VGPR0: the first enabled register is
5516 VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a
5517 VGPR number.
5519 There are different methods used for the VGPR initial state:
5521 * Unless the *Target Properties* column of :ref:`amdgpu-processor-table`
5522 specifies otherwise, a separate VGPR register is used per work-item ID. The
5523 VGPR register initial state for this method is defined in
5524 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`.
5525 * If *Target Properties* column of :ref:`amdgpu-processor-table`
5526 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used
5527 for all work-item IDs. The register layout for this method is defined in
5528 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`.
5530 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method
5531 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table
5533 ========== ========================== ====== ==============================
5534 VGPR Order Name Number Description
5535 (kernel descriptor enable of
5536 field) VGPRs
5537 ========== ========================== ====== ==============================
5538 First Work-Item Id X 1 32-bit work-item id in X
5539 (Always initialized) dimension of work-group for
5540 wavefront lane.
5541 then Work-Item Id Y 1 32-bit work-item id in Y
5542 (enable_vgpr_workitem_id dimension of work-group for
5543 > 0) wavefront lane.
5544 then Work-Item Id Z 1 32-bit work-item id in Z
5545 (enable_vgpr_workitem_id dimension of work-group for
5546 > 1) wavefront lane.
5547 ========== ========================== ====== ==============================
5549 ..
5551 .. table:: Register Layout for Packed Work-Item ID Method
5552 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table
5554 ======= ======= ================ =========================================
5555 Bits Size Field Name Description
5556 ======= ======= ================ =========================================
5557 0:9 10 bits Work-Item Id X Work-item id in X
5558 dimension of work-group for
5559 wavefront lane.
5561 Always initialized.
5563 10:19 10 bits Work-Item Id Y Work-item id in Y
5564 dimension of work-group for
5565 wavefront lane.
5567 Initialized if enable_vgpr_workitem_id >
5568 0, otherwise set to 0.
5569 20:29 10 bits Work-Item Id Z Work-item id in Z
5570 dimension of work-group for
5571 wavefront lane.
5573 Initialized if enable_vgpr_workitem_id >
5574 1, otherwise set to 0.
5575 30:31 2 bits Reserved, set to 0.
5576 ======= ======= ================ =========================================
5578 The setting of registers is done by GPU CP/ADC/SPI hardware as follows:
5580 1. SGPRs before the Work-Group Ids are set by CP using the 16 User Data
5581 registers.
5582 2. Work-group Id registers X, Y, Z are set by ADC which supports any
5583 combination including none.
5584 3. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why
5585 its value cannot be included with the flat scratch init value which is per
5586 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`).
5587 4. The VGPRs are set by SPI which only supports specifying either (X), (X, Y)
5588 or (X, Y, Z).
5589 5. Flat Scratch register pair initialization is described in
5590 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.
5592 The global segment can be accessed either using buffer instructions (GFX6 which
5593 has V# 64-bit address support), flat instructions (GFX7-GFX11), or global
5594 instructions (GFX9-GFX11).
5596 If buffer operations are used, then the compiler can generate a V# with the
5597 following properties:
5599 * base address of 0
5600 * no swizzle
5601 * ATC: 1 if IOMMU present (such as APU)
5602 * ptr64: 1
5603 * MTYPE set to support memory coherence that matches the runtime (such as CC for
5604 APU and NC for dGPU).
5606 .. _amdgpu-amdhsa-kernarg-preload:
5608 Preloaded Kernel Arguments
5609 ++++++++++++++++++++++++++
5611 On hardware that supports this feature, kernel arguments can be preloaded into
5612 User SGPRs, up to the maximum number of User SGPRs available. The allocation of
5613 Preload SGPRs occurs directly after the last enabled non-kernarg preload User
5614 SGPR. (See :ref:`amdgpu-amdhsa-initial-kernel-execution-state`)
5616 The data preloaded is copied from the kernarg segment, the amount of data is
5617 determined by the value specified in the kernarg_preload_spec_length field of
5618 the kernel descriptor. This data is then loaded into consecutive User SGPRs. The
5619 number of SGPRs receiving preloaded kernarg data corresponds with the value
5620 given by kernarg_preload_spec_length. The preloading starts at the dword offset
5621 within the kernarg segment, which is specified by the
5622 kernarg_preload_spec_offset field.
5624 If the kernarg_preload_spec_length is non-zero, the CP firmware will append an
5625 additional 256 bytes to the kernel_code_entry_byte_offset. This addition
5626 facilitates the incorporation of a prologue to the kernel entry to handle cases
5627 where code designed for kernarg preloading is executed on hardware equipped with
5628 incompatible firmware. If hardware has compatible firmware the 256 bytes at the
5629 start of the kernel entry will be skipped. Additionally, the compiler backend
5630 may insert a trap instruction at the start of the kernel prologue to manage
5631 situations where kernarg preloading is attempted on hardware with incompatible
5632 firmware.
5634 .. _amdgpu-amdhsa-kernel-prolog:
5636 Kernel Prolog
5637 ~~~~~~~~~~~~~
5639 The compiler performs initialization in the kernel prologue depending on the
5640 target and information about things like stack usage in the kernel and called
5641 functions. Some of this initialization requires the compiler to request certain
5642 User and System SGPRs be present in the
5643 :ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the
5644 :ref:`amdgpu-amdhsa-kernel-descriptor`.
5646 .. _amdgpu-amdhsa-kernel-prolog-cfi:
5648 CFI
5649 +++
5651 1. The CFI return address is undefined.
5653 2. The CFI CFA is defined using an expression which evaluates to a location
5654 description that comprises one memory location description for the
5655 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``.
5657 .. _amdgpu-amdhsa-kernel-prolog-m0:
5659 M0
5660 ++
5662 GFX6-GFX8
5663 The M0 register must be initialized with a value at least the total LDS size
5664 if the kernel may access LDS via DS or flat operations. Total LDS size is
5665 available in dispatch packet. For M0, it is also possible to use maximum
5666 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for
5667 GFX7-GFX8).
5668 GFX9-GFX11
5669 The M0 register is not used for range checking LDS accesses and so does not
5670 need to be initialized in the prolog.
5672 .. _amdgpu-amdhsa-kernel-prolog-stack-pointer:
5674 Stack Pointer
5675 +++++++++++++
5677 If the kernel has function calls it must set up the ABI stack pointer described
5678 in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting
5679 SGPR32 to the unswizzled scratch offset of the address past the last local
5680 allocation.
5682 .. _amdgpu-amdhsa-kernel-prolog-frame-pointer:
5684 Frame Pointer
5685 +++++++++++++
5687 If the kernel needs a frame pointer for the reasons defined in
5688 ``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the
5689 kernel prolog. If a frame pointer is not required then all uses of the frame
5690 pointer are replaced with immediate ``0`` offsets.
5692 .. _amdgpu-amdhsa-kernel-prolog-flat-scratch:
5694 Flat Scratch
5695 ++++++++++++
5697 There are different methods used for initializing flat scratch:
5699 * If the *Target Properties* column of :ref:`amdgpu-processor-table`
5700 specifies *Does not support generic address space*:
5702 Flat scratch is not supported and there is no flat scratch register pair.
5704 * If the *Target Properties* column of :ref:`amdgpu-processor-table`
5705 specifies *Offset flat scratch*:
5707 If the kernel or any function it calls may use flat operations to access
5708 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair
5709 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and
5710 Scratch Wavefront Offset SGPR registers (see
5711 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`):
5713 1. The low word of Flat Scratch Init is the 32-bit byte offset from
5714 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory
5715 being managed by SPI for the queue executing the kernel dispatch. This is
5716 the same value used in the Scratch Segment Buffer V# base address.
5718 CP obtains this from the runtime. (The Scratch Segment Buffer base address
5719 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.)
5721 The prolog must add the value of Scratch Wavefront Offset to get the
5722 wavefront's byte scratch backing memory offset from
5723 ``SH_HIDDEN_PRIVATE_BASE_VIMID``.
5725 The Scratch Wavefront Offset must also be used as an offset with Private
5726 segment address when using the Scratch Segment Buffer.
5728 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right
5729 shifted by 8 before moving into FLAT_SCRATCH_HI.
5731 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where
5732 SGPRn is the highest numbered SGPR allocated to the wavefront).
5733 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and
5734 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront
5735 FLAT SCRATCH BASE in flat memory instructions that access the scratch
5736 aperture.
5737 2. The second word of Flat Scratch Init is 32-bit byte size of a single
5738 work-items scratch memory usage.
5740 CP obtains this from the runtime, and it is always a multiple of DWORD. CP
5741 checks that the value in the kernel dispatch packet Private Segment Byte
5742 Size is not larger and requests the runtime to increase the queue's scratch
5743 size if necessary.
5745 CP directly loads from the kernel dispatch packet Private Segment Byte Size
5746 field and rounds up to a multiple of DWORD. Having CP load it once avoids
5747 loading it at the beginning of every wavefront.
5749 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on
5750 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE
5751 in flat memory instructions.
5753 * If the *Target Properties* column of :ref:`amdgpu-processor-table`
5754 specifies *Absolute flat scratch*:
5756 If the kernel or any function it calls may use flat operations to access
5757 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair
5758 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization
5759 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see
5760 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`):
5762 The Flat Scratch Init is the 64-bit address of the base of scratch backing
5763 memory being managed by SPI for the queue executing the kernel dispatch.
5765 CP obtains this from the runtime.
5767 The kernel prolog must add the value of the wave's Scratch Wavefront Offset
5768 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair
5769 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat
5770 memory instructions.
5772 The Scratch Wavefront Offset must also be used as an offset with Private
5773 segment address when using the Scratch Segment Buffer (see
5774 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`).
5776 * If the *Target Properties* column of :ref:`amdgpu-processor-table`
5777 specifies *Architected flat scratch*:
5779 If ENABLE_PRIVATE_SEGMENT is enabled in
5780 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table` then the FLAT_SCRATCH
5781 register pair will be initialized to the 64-bit address of the base of scratch
5782 backing memory being managed by SPI for the queue executing the kernel
5783 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the
5784 flat scratch base in flat memory instructions.
5786 .. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer:
5788 Private Segment Buffer
5789 ++++++++++++++++++++++
5791 If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies
5792 *Architected flat scratch* then a Private Segment Buffer is not supported.
5793 Instead the flat SCRATCH instructions are used.
5795 Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs
5796 that are used as a V# to access scratch. CP uses the value provided by the
5797 runtime. It is used, together with Scratch Wavefront Offset as an offset, to
5798 access the private memory space using a segment address. See
5799 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`.
5801 The scratch V# is a four-aligned SGPR and always selected for the kernel as
5802 follows:
5804 - If it is known during instruction selection that there is stack usage,
5805 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if
5806 optimizations are disabled (``-O0``), if stack objects already exist (for
5807 locals, etc.), or if there are any function calls.
5809 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index
5810 are reserved for the tentative scratch V#. These will be used if it is
5811 determined that spilling is needed.
5813 - If no use is made of the tentative scratch V#, then it is unreserved,
5814 and the register count is determined ignoring it.
5815 - If use is made of the tentative scratch V#, then its register numbers
5816 are shifted to the first four-aligned SGPR index after the highest one
5817 allocated by the register allocator, and all uses are updated. The
5818 register count includes them in the shifted location.
5819 - In either case, if the processor has the SGPR allocation bug, the
5820 tentative allocation is not shifted or unreserved in order to ensure
5821 the register count is higher to workaround the bug.
5823 .. note::
5825 This approach of using a tentative scratch V# and shifting the register
5826 numbers if used avoids having to perform register allocation a second
5827 time if the tentative V# is eliminated. This is more efficient and
5828 avoids the problem that the second register allocation may perform
5829 spilling which will fail as there is no longer a scratch V#.
5831 When the kernel prolog code is being emitted it is known whether the scratch V#
5832 described above is actually used. If it is, the prolog code must set it up by
5833 copying the Private Segment Buffer to the scratch V# registers and then adding
5834 the Private Segment Wavefront Offset to the queue base address in the V#. The
5835 result is a V# with a base address pointing to the beginning of the wavefront
5836 scratch backing memory.
5838 The Private Segment Buffer is always requested, but the Private Segment
5839 Wavefront Offset is only requested if it is used (see
5840 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).
5842 .. _amdgpu-amdhsa-memory-model:
5844 Memory Model
5845 ~~~~~~~~~~~~
5847 This section describes the mapping of the LLVM memory model onto AMDGPU machine
5848 code (see :ref:`memmodel`).
5850 The AMDGPU backend supports the memory synchronization scopes specified in
5851 :ref:`amdgpu-memory-scopes`.
5853 The code sequences used to implement the memory model specify the order of
5854 instructions that a single thread must execute. The ``s_waitcnt`` and cache
5855 management instructions such as ``buffer_wbinvl1_vol`` are defined with respect
5856 to other memory instructions executed by the same thread. This allows them to be
5857 moved earlier or later which can allow them to be combined with other instances
5858 of the same instruction, or hoisted/sunk out of loops to improve performance.
5859 Only the instructions related to the memory model are given; additional
5860 ``s_waitcnt`` instructions are required to ensure registers are defined before
5861 being used. These may be able to be combined with the memory model ``s_waitcnt``
5862 instructions as described above.
5864 The AMDGPU backend supports the following memory models:
5866 HSA Memory Model [HSA]_
5867 The HSA memory model uses a single happens-before relation for all address
5868 spaces (see :ref:`amdgpu-address-spaces`).
5869 OpenCL Memory Model [OpenCL]_
5870 The OpenCL memory model which has separate happens-before relations for the
5871 global and local address spaces. Only a fence specifying both global and
5872 local address space, and seq_cst instructions join the relationships. Since
5873 the LLVM ``memfence`` instruction does not allow an address space to be
5874 specified the OpenCL fence has to conservatively assume both local and
5875 global address space was specified. However, optimizations can often be
5876 done to eliminate the additional ``s_waitcnt`` instructions when there are
5877 no intervening memory instructions which access the corresponding address
5878 space. The code sequences in the table indicate what can be omitted for the
5879 OpenCL memory. The target triple environment is used to determine if the
5880 source language is OpenCL (see :ref:`amdgpu-opencl`).
5882 ``ds/flat_load/store/atomic`` instructions to local memory are termed LDS
5883 operations.
5885 ``buffer/global/flat_load/store/atomic`` instructions to global memory are
5886 termed vector memory operations.
5888 Private address space uses ``buffer_load/store`` using the scratch V#
5889 (GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX11). Since only a single thread
5890 is accessing the memory, atomic memory orderings are not meaningful, and all
5891 accesses are treated as non-atomic.
5893 Constant address space uses ``buffer/global_load`` instructions (or equivalent
5894 scalar memory instructions). Since the constant address space contents do not
5895 change during the execution of a kernel dispatch it is not legal to perform
5896 stores, and atomic memory orderings are not meaningful, and all accesses are
5897 treated as non-atomic.
5899 A memory synchronization scope wider than work-group is not meaningful for the
5900 group (LDS) address space and is treated as work-group.
5902 The memory model does not support the region address space which is treated as
5903 non-atomic.
5905 Acquire memory ordering is not meaningful on store atomic instructions and is
5906 treated as non-atomic.
5908 Release memory ordering is not meaningful on load atomic instructions and is
5909 treated a non-atomic.
5911 Acquire-release memory ordering is not meaningful on load or store atomic
5912 instructions and is treated as acquire and release respectively.
5914 The memory order also adds the single thread optimization constraints defined in
5915 table
5916 :ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`.
5918 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints
5919 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table
5921 ============ ==============================================================
5922 LLVM Memory Optimization Constraints
5923 Ordering
5924 ============ ==============================================================
5925 unordered *none*
5926 monotonic *none*
5927 acquire - If a load atomic/atomicrmw then no following load/load
5928 atomic/store/store atomic/atomicrmw/fence instruction can be
5929 moved before the acquire.
5930 - If a fence then same as load atomic, plus no preceding
5931 associated fence-paired-atomic can be moved after the fence.
5932 release - If a store atomic/atomicrmw then no preceding load/load
5933 atomic/store/store atomic/atomicrmw/fence instruction can be
5934 moved after the release.
5935 - If a fence then same as store atomic, plus no following
5936 associated fence-paired-atomic can be moved before the
5937 fence.
5938 acq_rel Same constraints as both acquire and release.
5939 seq_cst - If a load atomic then same constraints as acquire, plus no
5940 preceding sequentially consistent load atomic/store
5941 atomic/atomicrmw/fence instruction can be moved after the
5942 seq_cst.
5943 - If a store atomic then the same constraints as release, plus
5944 no following sequentially consistent load atomic/store
5945 atomic/atomicrmw/fence instruction can be moved before the
5946 seq_cst.
5947 - If an atomicrmw/fence then same constraints as acq_rel.
5948 ============ ==============================================================
5950 The code sequences used to implement the memory model are defined in the
5951 following sections:
5953 * :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9`
5954 * :ref:`amdgpu-amdhsa-memory-model-gfx90a`
5955 * :ref:`amdgpu-amdhsa-memory-model-gfx942`
5956 * :ref:`amdgpu-amdhsa-memory-model-gfx10-gfx11`
5958 .. _amdgpu-amdhsa-memory-model-gfx6-gfx9:
5960 Memory Model GFX6-GFX9
5961 ++++++++++++++++++++++
5963 For GFX6-GFX9:
5965 * Each agent has multiple shader arrays (SA).
5966 * Each SA has multiple compute units (CU).
5967 * Each CU has multiple SIMDs that execute wavefronts.
5968 * The wavefronts for a single work-group are executed in the same CU but may be
5969 executed by different SIMDs.
5970 * Each CU has a single LDS memory shared by the wavefronts of the work-groups
5971 executing on it.
5972 * All LDS operations of a CU are performed as wavefront wide operations in a
5973 global order and involve no caching. Completion is reported to a wavefront in
5974 execution order.
5975 * The LDS memory has multiple request queues shared by the SIMDs of a
5976 CU. Therefore, the LDS operations performed by different wavefronts of a
5977 work-group can be reordered relative to each other, which can result in
5978 reordering the visibility of vector memory operations with respect to LDS
5979 operations of other wavefronts in the same work-group. A ``s_waitcnt
5980 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and
5981 vector memory operations between wavefronts of a work-group, but not between
5982 operations performed by the same wavefront.
5983 * The vector memory operations are performed as wavefront wide operations and
5984 completion is reported to a wavefront in execution order. The exception is
5985 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of
5986 vector memory order if they access LDS memory, and out of LDS operation order
5987 if they access global memory.
5988 * The vector memory operations access a single vector L1 cache shared by all
5989 SIMDs a CU. Therefore, no special action is required for coherence between the
5990 lanes of a single wavefront, or for coherence between wavefronts in the same
5991 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between
5992 wavefronts executing in different work-groups as they may be executing on
5993 different CUs.
5994 * The scalar memory operations access a scalar L1 cache shared by all wavefronts
5995 on a group of CUs. The scalar and vector L1 caches are not coherent. However,
5996 scalar operations are used in a restricted way so do not impact the memory
5997 model. See :ref:`amdgpu-amdhsa-memory-spaces`.
5998 * The vector and scalar memory operations use an L2 cache shared by all CUs on
5999 the same agent.
6000 * The L2 cache has independent channels to service disjoint ranges of virtual
6001 addresses.
6002 * Each CU has a separate request queue per channel. Therefore, the vector and
6003 scalar memory operations performed by wavefronts executing in different
6004 work-groups (which may be executing on different CUs) of an agent can be
6005 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to
6006 ensure synchronization between vector memory operations of different CUs. It
6007 ensures a previous vector memory operation has completed before executing a
6008 subsequent vector memory or LDS operation and so can be used to meet the
6009 requirements of acquire and release.
6010 * The L2 cache can be kept coherent with other agents on some targets, or ranges
6011 of virtual addresses can be set up to bypass it to ensure system coherence.
6013 Scalar memory operations are only used to access memory that is proven to not
6014 change during the execution of the kernel dispatch. This includes constant
6015 address space and global address space for program scope ``const`` variables.
6016 Therefore, the kernel machine code does not have to maintain the scalar cache to
6017 ensure it is coherent with the vector caches. The scalar and vector caches are
6018 invalidated between kernel dispatches by CP since constant address space data
6019 may change between kernel dispatch executions. See
6020 :ref:`amdgpu-amdhsa-memory-spaces`.
6022 The one exception is if scalar writes are used to spill SGPR registers. In this
6023 case the AMDGPU backend ensures the memory location used to spill is never
6024 accessed by vector memory operations at the same time. If scalar writes are used
6025 then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function
6026 return since the locations may be used for vector memory instructions by a
6027 future wavefront that uses the same scratch area, or a function call that
6028 creates a frame at the same address, respectively. There is no need for a
6029 ``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.
6031 For kernarg backing memory:
6033 * CP invalidates the L1 cache at the start of each kernel dispatch.
6034 * On dGPU the kernarg backing memory is allocated in host memory accessed as
6035 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also
6036 causes it to be treated as non-volatile and so is not invalidated by
6037 ``*_vol``.
6038 * On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent)
6039 and so the L2 cache will be coherent with the CPU and other agents.
6041 Scratch backing memory (which is used for the private address space) is accessed
6042 with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is
6043 only accessed by a single thread, and is always write-before-read, there is
6044 never a need to invalidate these entries from the L1 cache. Hence all cache
6045 invalidates are done as ``*_vol`` to only invalidate the volatile cache lines.
6047 The code sequences used to implement the memory model for GFX6-GFX9 are defined
6048 in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`.
6050 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9
6051 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table
6053 ============ ============ ============== ========== ================================
6054 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code
6055 Ordering Sync Scope Address GFX6-GFX9
6056 Space
6057 ============ ============ ============== ========== ================================
6058 **Non-Atomic**
6059 ------------------------------------------------------------------------------------
6060 load *none* *none* - global - !volatile & !nontemporal
6061 - generic
6062 - private 1. buffer/global/flat_load
6063 - constant
6064 - !volatile & nontemporal
6066 1. buffer/global/flat_load
6067 glc=1 slc=1
6069 - volatile
6071 1. buffer/global/flat_load
6072 glc=1
6073 2. s_waitcnt vmcnt(0)
6075 - Must happen before
6076 any following volatile
6077 global/generic
6078 load/store.
6079 - Ensures that
6080 volatile
6081 operations to
6082 different
6083 addresses will not
6084 be reordered by
6085 hardware.
6087 load *none* *none* - local 1. ds_load
6088 store *none* *none* - global - !volatile & !nontemporal
6089 - generic
6090 - private 1. buffer/global/flat_store
6091 - constant
6092 - !volatile & nontemporal
6094 1. buffer/global/flat_store
6095 glc=1 slc=1
6097 - volatile
6099 1. buffer/global/flat_store
6100 2. s_waitcnt vmcnt(0)
6102 - Must happen before
6103 any following volatile
6104 global/generic
6105 load/store.
6106 - Ensures that
6107 volatile
6108 operations to
6109 different
6110 addresses will not
6111 be reordered by
6112 hardware.
6114 store *none* *none* - local 1. ds_store
6115 **Unordered Atomic**
6116 ------------------------------------------------------------------------------------
6117 load atomic unordered *any* *any* *Same as non-atomic*.
6118 store atomic unordered *any* *any* *Same as non-atomic*.
6119 atomicrmw unordered *any* *any* *Same as monotonic atomic*.
6120 **Monotonic Atomic**
6121 ------------------------------------------------------------------------------------
6122 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load
6123 - wavefront - local
6124 - workgroup - generic
6125 load atomic monotonic - agent - global 1. buffer/global/flat_load
6126 - system - generic glc=1
6127 store atomic monotonic - singlethread - global 1. buffer/global/flat_store
6128 - wavefront - generic
6129 - workgroup
6130 - agent
6131 - system
6132 store atomic monotonic - singlethread - local 1. ds_store
6133 - wavefront
6134 - workgroup
6135 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic
6136 - wavefront - generic
6137 - workgroup
6138 - agent
6139 - system
6140 atomicrmw monotonic - singlethread - local 1. ds_atomic
6141 - wavefront
6142 - workgroup
6143 **Acquire Atomic**
6144 ------------------------------------------------------------------------------------
6145 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load
6146 - wavefront - local
6147 - generic
6148 load atomic acquire - workgroup - global 1. buffer/global_load
6149 load atomic acquire - workgroup - local 1. ds/flat_load
6150 - generic 2. s_waitcnt lgkmcnt(0)
6152 - If OpenCL, omit.
6153 - Must happen before
6154 any following
6155 global/generic
6156 load/load
6157 atomic/store/store
6158 atomic/atomicrmw.
6159 - Ensures any
6160 following global
6161 data read is no
6162 older than a local load
6163 atomic value being
6164 acquired.
6166 load atomic acquire - agent - global 1. buffer/global_load
6167 - system glc=1
6168 2. s_waitcnt vmcnt(0)
6170 - Must happen before
6171 following
6172 buffer_wbinvl1_vol.
6173 - Ensures the load
6174 has completed
6175 before invalidating
6176 the cache.
6178 3. buffer_wbinvl1_vol
6180 - Must happen before
6181 any following
6182 global/generic
6183 load/load
6184 atomic/atomicrmw.
6185 - Ensures that
6186 following
6187 loads will not see
6188 stale global data.
6190 load atomic acquire - agent - generic 1. flat_load glc=1
6191 - system 2. s_waitcnt vmcnt(0) &
6192 lgkmcnt(0)
6194 - If OpenCL omit
6195 lgkmcnt(0).
6196 - Must happen before
6197 following
6198 buffer_wbinvl1_vol.
6199 - Ensures the flat_load
6200 has completed
6201 before invalidating
6202 the cache.
6204 3. buffer_wbinvl1_vol
6206 - Must happen before
6207 any following
6208 global/generic
6209 load/load
6210 atomic/atomicrmw.
6211 - Ensures that
6212 following loads
6213 will not see stale
6214 global data.
6216 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic
6217 - wavefront - local
6218 - generic
6219 atomicrmw acquire - workgroup - global 1. buffer/global_atomic
6220 atomicrmw acquire - workgroup - local 1. ds/flat_atomic
6221 - generic 2. s_waitcnt lgkmcnt(0)
6223 - If OpenCL, omit.
6224 - Must happen before
6225 any following
6226 global/generic
6227 load/load
6228 atomic/store/store
6229 atomic/atomicrmw.
6230 - Ensures any
6231 following global
6232 data read is no
6233 older than a local
6234 atomicrmw value
6235 being acquired.
6237 atomicrmw acquire - agent - global 1. buffer/global_atomic
6238 - system 2. s_waitcnt vmcnt(0)
6240 - Must happen before
6241 following
6242 buffer_wbinvl1_vol.
6243 - Ensures the
6244 atomicrmw has
6245 completed before
6246 invalidating the
6247 cache.
6249 3. buffer_wbinvl1_vol
6251 - Must happen before
6252 any following
6253 global/generic
6254 load/load
6255 atomic/atomicrmw.
6256 - Ensures that
6257 following loads
6258 will not see stale
6259 global data.
6261 atomicrmw acquire - agent - generic 1. flat_atomic
6262 - system 2. s_waitcnt vmcnt(0) &
6263 lgkmcnt(0)
6265 - If OpenCL, omit
6266 lgkmcnt(0).
6267 - Must happen before
6268 following
6269 buffer_wbinvl1_vol.
6270 - Ensures the
6271 atomicrmw has
6272 completed before
6273 invalidating the
6274 cache.
6276 3. buffer_wbinvl1_vol
6278 - Must happen before
6279 any following
6280 global/generic
6281 load/load
6282 atomic/atomicrmw.
6283 - Ensures that
6284 following loads
6285 will not see stale
6286 global data.
6288 fence acquire - singlethread *none* *none*
6289 - wavefront
6290 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0)
6292 - If OpenCL and
6293 address space is
6294 not generic, omit.
6295 - However, since LLVM
6296 currently has no
6297 address space on
6298 the fence need to
6299 conservatively
6300 always generate. If
6301 fence had an
6302 address space then
6303 set to address
6304 space of OpenCL
6305 fence flag, or to
6306 generic if both
6307 local and global
6308 flags are
6309 specified.
6310 - Must happen after
6311 any preceding
6312 local/generic load
6313 atomic/atomicrmw
6314 with an equal or
6315 wider sync scope
6316 and memory ordering
6317 stronger than
6318 unordered (this is
6319 termed the
6320 fence-paired-atomic).
6321 - Must happen before
6322 any following
6323 global/generic
6324 load/load
6325 atomic/store/store
6326 atomic/atomicrmw.
6327 - Ensures any
6328 following global
6329 data read is no
6330 older than the
6331 value read by the
6332 fence-paired-atomic.
6334 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) &
6335 - system vmcnt(0)
6337 - If OpenCL and
6338 address space is
6339 not generic, omit
6340 lgkmcnt(0).
6341 - However, since LLVM
6342 currently has no
6343 address space on
6344 the fence need to
6345 conservatively
6346 always generate
6347 (see comment for
6348 previous fence).
6349 - Could be split into
6350 separate s_waitcnt
6351 vmcnt(0) and
6352 s_waitcnt
6353 lgkmcnt(0) to allow
6354 them to be
6355 independently moved
6356 according to the
6357 following rules.
6358 - s_waitcnt vmcnt(0)
6359 must happen after
6360 any preceding
6361 global/generic load
6362 atomic/atomicrmw
6363 with an equal or
6364 wider sync scope
6365 and memory ordering
6366 stronger than
6367 unordered (this is
6368 termed the
6369 fence-paired-atomic).
6370 - s_waitcnt lgkmcnt(0)
6371 must happen after
6372 any preceding
6373 local/generic load
6374 atomic/atomicrmw
6375 with an equal or
6376 wider sync scope
6377 and memory ordering
6378 stronger than
6379 unordered (this is
6380 termed the
6381 fence-paired-atomic).
6382 - Must happen before
6383 the following
6384 buffer_wbinvl1_vol.
6385 - Ensures that the
6386 fence-paired atomic
6387 has completed
6388 before invalidating
6389 the
6390 cache. Therefore
6391 any following
6392 locations read must
6393 be no older than
6394 the value read by
6395 the
6396 fence-paired-atomic.
6398 2. buffer_wbinvl1_vol
6400 - Must happen before any
6401 following global/generic
6402 load/load
6403 atomic/store/store
6404 atomic/atomicrmw.
6405 - Ensures that
6406 following loads
6407 will not see stale
6408 global data.
6410 **Release Atomic**
6411 ------------------------------------------------------------------------------------
6412 store atomic release - singlethread - global 1. buffer/global/ds/flat_store
6413 - wavefront - local
6414 - generic
6415 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0)
6416 - generic
6417 - If OpenCL, omit.
6418 - Must happen after
6419 any preceding
6420 local/generic
6421 load/store/load
6422 atomic/store
6423 atomic/atomicrmw.
6424 - Must happen before
6425 the following
6426 store.
6427 - Ensures that all
6428 memory operations
6429 to local have
6430 completed before
6431 performing the
6432 store that is being
6433 released.
6435 2. buffer/global/flat_store
6436 store atomic release - workgroup - local 1. ds_store
6437 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &
6438 - system - generic vmcnt(0)
6440 - If OpenCL and
6441 address space is
6442 not generic, omit
6443 lgkmcnt(0).
6444 - Could be split into
6445 separate s_waitcnt
6446 vmcnt(0) and
6447 s_waitcnt
6448 lgkmcnt(0) to allow
6449 them to be
6450 independently moved
6451 according to the
6452 following rules.
6453 - s_waitcnt vmcnt(0)
6454 must happen after
6455 any preceding
6456 global/generic
6457 load/store/load
6458 atomic/store
6459 atomic/atomicrmw.
6460 - s_waitcnt lgkmcnt(0)
6461 must happen after
6462 any preceding
6463 local/generic
6464 load/store/load
6465 atomic/store
6466 atomic/atomicrmw.
6467 - Must happen before
6468 the following
6469 store.
6470 - Ensures that all
6471 memory operations
6472 to memory have
6473 completed before
6474 performing the
6475 store that is being
6476 released.
6478 2. buffer/global/flat_store
6479 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic
6480 - wavefront - local
6481 - generic
6482 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0)
6483 - generic
6484 - If OpenCL, omit.
6485 - Must happen after
6486 any preceding
6487 local/generic
6488 load/store/load
6489 atomic/store
6490 atomic/atomicrmw.
6491 - Must happen before
6492 the following
6493 atomicrmw.
6494 - Ensures that all
6495 memory operations
6496 to local have
6497 completed before
6498 performing the
6499 atomicrmw that is
6500 being released.
6502 2. buffer/global/flat_atomic
6503 atomicrmw release - workgroup - local 1. ds_atomic
6504 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &
6505 - system - generic vmcnt(0)
6507 - If OpenCL, omit
6508 lgkmcnt(0).
6509 - Could be split into
6510 separate s_waitcnt
6511 vmcnt(0) and
6512 s_waitcnt
6513 lgkmcnt(0) to allow
6514 them to be
6515 independently moved
6516 according to the
6517 following rules.
6518 - s_waitcnt vmcnt(0)
6519 must happen after
6520 any preceding
6521 global/generic
6522 load/store/load
6523 atomic/store
6524 atomic/atomicrmw.
6525 - s_waitcnt lgkmcnt(0)
6526 must happen after
6527 any preceding
6528 local/generic
6529 load/store/load
6530 atomic/store
6531 atomic/atomicrmw.
6532 - Must happen before
6533 the following
6534 atomicrmw.
6535 - Ensures that all
6536 memory operations
6537 to global and local
6538 have completed
6539 before performing
6540 the atomicrmw that
6541 is being released.
6543 2. buffer/global/flat_atomic
6544 fence release - singlethread *none* *none*
6545 - wavefront
6546 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0)
6548 - If OpenCL and
6549 address space is
6550 not generic, omit.
6551 - However, since LLVM
6552 currently has no
6553 address space on
6554 the fence need to
6555 conservatively
6556 always generate. If
6557 fence had an
6558 address space then
6559 set to address
6560 space of OpenCL
6561 fence flag, or to
6562 generic if both
6563 local and global
6564 flags are
6565 specified.
6566 - Must happen after
6567 any preceding
6568 local/generic
6569 load/load
6570 atomic/store/store
6571 atomic/atomicrmw.
6572 - Must happen before
6573 any following store
6574 atomic/atomicrmw
6575 with an equal or
6576 wider sync scope
6577 and memory ordering
6578 stronger than
6579 unordered (this is
6580 termed the
6581 fence-paired-atomic).
6582 - Ensures that all
6583 memory operations
6584 to local have
6585 completed before
6586 performing the
6587 following
6588 fence-paired-atomic.
6590 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) &
6591 - system vmcnt(0)
6593 - If OpenCL and
6594 address space is
6595 not generic, omit
6596 lgkmcnt(0).
6597 - If OpenCL and
6598 address space is
6599 local, omit
6600 vmcnt(0).
6601 - However, since LLVM
6602 currently has no
6603 address space on
6604 the fence need to
6605 conservatively
6606 always generate. If
6607 fence had an
6608 address space then
6609 set to address
6610 space of OpenCL
6611 fence flag, or to
6612 generic if both
6613 local and global
6614 flags are
6615 specified.
6616 - Could be split into
6617 separate s_waitcnt
6618 vmcnt(0) and
6619 s_waitcnt
6620 lgkmcnt(0) to allow
6621 them to be
6622 independently moved
6623 according to the
6624 following rules.
6625 - s_waitcnt vmcnt(0)
6626 must happen after
6627 any preceding
6628 global/generic
6629 load/store/load
6630 atomic/store
6631 atomic/atomicrmw.
6632 - s_waitcnt lgkmcnt(0)
6633 must happen after
6634 any preceding
6635 local/generic
6636 load/store/load
6637 atomic/store
6638 atomic/atomicrmw.
6639 - Must happen before
6640 any following store
6641 atomic/atomicrmw
6642 with an equal or
6643 wider sync scope
6644 and memory ordering
6645 stronger than
6646 unordered (this is
6647 termed the
6648 fence-paired-atomic).
6649 - Ensures that all
6650 memory operations
6651 have
6652 completed before
6653 performing the
6654 following
6655 fence-paired-atomic.
6657 **Acquire-Release Atomic**
6658 ------------------------------------------------------------------------------------
6659 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic
6660 - wavefront - local
6661 - generic
6662 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0)
6664 - If OpenCL, omit.
6665 - Must happen after
6666 any preceding
6667 local/generic
6668 load/store/load
6669 atomic/store
6670 atomic/atomicrmw.
6671 - Must happen before
6672 the following
6673 atomicrmw.
6674 - Ensures that all
6675 memory operations
6676 to local have
6677 completed before
6678 performing the
6679 atomicrmw that is
6680 being released.
6682 2. buffer/global_atomic
6684 atomicrmw acq_rel - workgroup - local 1. ds_atomic
6685 2. s_waitcnt lgkmcnt(0)
6687 - If OpenCL, omit.
6688 - Must happen before
6689 any following
6690 global/generic
6691 load/load
6692 atomic/store/store
6693 atomic/atomicrmw.
6694 - Ensures any
6695 following global
6696 data read is no
6697 older than the local load
6698 atomic value being
6699 acquired.
6701 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0)
6703 - If OpenCL, omit.
6704 - Must happen after
6705 any preceding
6706 local/generic
6707 load/store/load
6708 atomic/store
6709 atomic/atomicrmw.
6710 - Must happen before
6711 the following
6712 atomicrmw.
6713 - Ensures that all
6714 memory operations
6715 to local have
6716 completed before
6717 performing the
6718 atomicrmw that is
6719 being released.
6721 2. flat_atomic
6722 3. s_waitcnt lgkmcnt(0)
6724 - If OpenCL, omit.
6725 - Must happen before
6726 any following
6727 global/generic
6728 load/load
6729 atomic/store/store
6730 atomic/atomicrmw.
6731 - Ensures any
6732 following global
6733 data read is no
6734 older than a local load
6735 atomic value being
6736 acquired.
6738 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &
6739 - system vmcnt(0)
6741 - If OpenCL, omit
6742 lgkmcnt(0).
6743 - Could be split into
6744 separate s_waitcnt
6745 vmcnt(0) and
6746 s_waitcnt
6747 lgkmcnt(0) to allow
6748 them to be
6749 independently moved
6750 according to the
6751 following rules.
6752 - s_waitcnt vmcnt(0)
6753 must happen after
6754 any preceding
6755 global/generic
6756 load/store/load
6757 atomic/store
6758 atomic/atomicrmw.
6759 - s_waitcnt lgkmcnt(0)
6760 must happen after
6761 any preceding
6762 local/generic
6763 load/store/load
6764 atomic/store
6765 atomic/atomicrmw.
6766 - Must happen before
6767 the following
6768 atomicrmw.
6769 - Ensures that all
6770 memory operations
6771 to global have
6772 completed before
6773 performing the
6774 atomicrmw that is
6775 being released.
6777 2. buffer/global_atomic
6778 3. s_waitcnt vmcnt(0)
6780 - Must happen before
6781 following
6782 buffer_wbinvl1_vol.
6783 - Ensures the
6784 atomicrmw has
6785 completed before
6786 invalidating the
6787 cache.
6789 4. buffer_wbinvl1_vol
6791 - Must happen before
6792 any following
6793 global/generic
6794 load/load
6795 atomic/atomicrmw.
6796 - Ensures that
6797 following loads
6798 will not see stale
6799 global data.
6801 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &
6802 - system vmcnt(0)
6804 - If OpenCL, omit
6805 lgkmcnt(0).
6806 - Could be split into
6807 separate s_waitcnt
6808 vmcnt(0) and
6809 s_waitcnt
6810 lgkmcnt(0) to allow
6811 them to be
6812 independently moved
6813 according to the
6814 following rules.
6815 - s_waitcnt vmcnt(0)
6816 must happen after
6817 any preceding
6818 global/generic
6819 load/store/load
6820 atomic/store
6821 atomic/atomicrmw.
6822 - s_waitcnt lgkmcnt(0)
6823 must happen after
6824 any preceding
6825 local/generic
6826 load/store/load
6827 atomic/store
6828 atomic/atomicrmw.
6829 - Must happen before
6830 the following
6831 atomicrmw.
6832 - Ensures that all
6833 memory operations
6834 to global have
6835 completed before
6836 performing the
6837 atomicrmw that is
6838 being released.
6840 2. flat_atomic
6841 3. s_waitcnt vmcnt(0) &
6842 lgkmcnt(0)
6844 - If OpenCL, omit
6845 lgkmcnt(0).
6846 - Must happen before
6847 following
6848 buffer_wbinvl1_vol.
6849 - Ensures the
6850 atomicrmw has
6851 completed before
6852 invalidating the
6853 cache.
6855 4. buffer_wbinvl1_vol
6857 - Must happen before
6858 any following
6859 global/generic
6860 load/load
6861 atomic/atomicrmw.
6862 - Ensures that
6863 following loads
6864 will not see stale
6865 global data.
6867 fence acq_rel - singlethread *none* *none*
6868 - wavefront
6869 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0)
6871 - If OpenCL and
6872 address space is
6873 not generic, omit.
6874 - However,
6875 since LLVM
6876 currently has no
6877 address space on
6878 the fence need to
6879 conservatively
6880 always generate
6881 (see comment for
6882 previous fence).
6883 - Must happen after
6884 any preceding
6885 local/generic
6886 load/load
6887 atomic/store/store
6888 atomic/atomicrmw.
6889 - Must happen before
6890 any following
6891 global/generic
6892 load/load
6893 atomic/store/store
6894 atomic/atomicrmw.
6895 - Ensures that all
6896 memory operations
6897 to local have
6898 completed before
6899 performing any
6900 following global
6901 memory operations.
6902 - Ensures that the
6903 preceding
6904 local/generic load
6905 atomic/atomicrmw
6906 with an equal or
6907 wider sync scope
6908 and memory ordering
6909 stronger than
6910 unordered (this is
6911 termed the
6912 acquire-fence-paired-atomic)
6913 has completed
6914 before following
6915 global memory
6916 operations. This
6917 satisfies the
6918 requirements of
6919 acquire.
6920 - Ensures that all
6921 previous memory
6922 operations have
6923 completed before a
6924 following
6925 local/generic store
6926 atomic/atomicrmw
6927 with an equal or
6928 wider sync scope
6929 and memory ordering
6930 stronger than
6931 unordered (this is
6932 termed the
6933 release-fence-paired-atomic).
6934 This satisfies the
6935 requirements of
6936 release.
6938 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) &
6939 - system vmcnt(0)
6941 - If OpenCL and
6942 address space is
6943 not generic, omit
6944 lgkmcnt(0).
6945 - However, since LLVM
6946 currently has no
6947 address space on
6948 the fence need to
6949 conservatively
6950 always generate
6951 (see comment for
6952 previous fence).
6953 - Could be split into
6954 separate s_waitcnt
6955 vmcnt(0) and
6956 s_waitcnt
6957 lgkmcnt(0) to allow
6958 them to be
6959 independently moved
6960 according to the
6961 following rules.
6962 - s_waitcnt vmcnt(0)
6963 must happen after
6964 any preceding
6965 global/generic
6966 load/store/load
6967 atomic/store
6968 atomic/atomicrmw.
6969 - s_waitcnt lgkmcnt(0)
6970 must happen after
6971 any preceding
6972 local/generic
6973 load/store/load
6974 atomic/store
6975 atomic/atomicrmw.
6976 - Must happen before
6977 the following
6978 buffer_wbinvl1_vol.
6979 - Ensures that the
6980 preceding
6981 global/local/generic
6982 load
6983 atomic/atomicrmw
6984 with an equal or
6985 wider sync scope
6986 and memory ordering
6987 stronger than
6988 unordered (this is
6989 termed the
6990 acquire-fence-paired-atomic)
6991 has completed
6992 before invalidating
6993 the cache. This
6994 satisfies the
6995 requirements of
6996 acquire.
6997 - Ensures that all
6998 previous memory
6999 operations have
7000 completed before a
7001 following
7002 global/local/generic
7003 store
7004 atomic/atomicrmw
7005 with an equal or
7006 wider sync scope
7007 and memory ordering
7008 stronger than
7009 unordered (this is
7010 termed the
7011 release-fence-paired-atomic).
7012 This satisfies the
7013 requirements of
7014 release.
7016 2. buffer_wbinvl1_vol
7018 - Must happen before
7019 any following
7020 global/generic
7021 load/load
7022 atomic/store/store
7023 atomic/atomicrmw.
7024 - Ensures that
7025 following loads
7026 will not see stale
7027 global data. This
7028 satisfies the
7029 requirements of
7030 acquire.
7032 **Sequential Consistent Atomic**
7033 ------------------------------------------------------------------------------------
7034 load atomic seq_cst - singlethread - global *Same as corresponding
7035 - wavefront - local load atomic acquire,
7036 - generic except must generate
7037 all instructions even
7038 for OpenCL.*
7039 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0)
7040 - generic
7042 - Must
7043 happen after
7044 preceding
7045 local/generic load
7046 atomic/store
7047 atomic/atomicrmw
7048 with memory
7049 ordering of seq_cst
7050 and with equal or
7051 wider sync scope.
7052 (Note that seq_cst
7053 fences have their
7054 own s_waitcnt
7055 lgkmcnt(0) and so do
7056 not need to be
7057 considered.)
7058 - Ensures any
7059 preceding
7060 sequential
7061 consistent local
7062 memory instructions
7063 have completed
7064 before executing
7065 this sequentially
7066 consistent
7067 instruction. This
7068 prevents reordering
7069 a seq_cst store
7070 followed by a
7071 seq_cst load. (Note
7072 that seq_cst is
7073 stronger than
7074 acquire/release as
7075 the reordering of
7076 load acquire
7077 followed by a store
7078 release is
7079 prevented by the
7080 s_waitcnt of
7081 the release, but
7082 there is nothing
7083 preventing a store
7084 release followed by
7085 load acquire from
7086 completing out of
7087 order. The s_waitcnt
7088 could be placed after
7089 seq_store or before
7090 the seq_load. We
7091 choose the load to
7092 make the s_waitcnt be
7093 as late as possible
7094 so that the store
7095 may have already
7096 completed.)
7098 2. *Following
7099 instructions same as
7100 corresponding load
7101 atomic acquire,
7102 except must generate
7103 all instructions even
7104 for OpenCL.*
7105 load atomic seq_cst - workgroup - local *Same as corresponding
7106 load atomic acquire,
7107 except must generate
7108 all instructions even
7109 for OpenCL.*
7111 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &
7112 - system - generic vmcnt(0)
7114 - Could be split into
7115 separate s_waitcnt
7116 vmcnt(0)
7117 and s_waitcnt
7118 lgkmcnt(0) to allow
7119 them to be
7120 independently moved
7121 according to the
7122 following rules.
7123 - s_waitcnt lgkmcnt(0)
7124 must happen after
7125 preceding
7126 global/generic load
7127 atomic/store
7128 atomic/atomicrmw
7129 with memory
7130 ordering of seq_cst
7131 and with equal or
7132 wider sync scope.
7133 (Note that seq_cst
7134 fences have their
7135 own s_waitcnt
7136 lgkmcnt(0) and so do
7137 not need to be
7138 considered.)
7139 - s_waitcnt vmcnt(0)
7140 must happen after
7141 preceding
7142 global/generic load
7143 atomic/store
7144 atomic/atomicrmw
7145 with memory
7146 ordering of seq_cst
7147 and with equal or
7148 wider sync scope.
7149 (Note that seq_cst
7150 fences have their
7151 own s_waitcnt
7152 vmcnt(0) and so do
7153 not need to be
7154 considered.)
7155 - Ensures any
7156 preceding
7157 sequential
7158 consistent global
7159 memory instructions
7160 have completed
7161 before executing
7162 this sequentially
7163 consistent
7164 instruction. This
7165 prevents reordering
7166 a seq_cst store
7167 followed by a
7168 seq_cst load. (Note
7169 that seq_cst is
7170 stronger than
7171 acquire/release as
7172 the reordering of
7173 load acquire
7174 followed by a store
7175 release is
7176 prevented by the
7177 s_waitcnt of
7178 the release, but
7179 there is nothing
7180 preventing a store
7181 release followed by
7182 load acquire from
7183 completing out of
7184 order. The s_waitcnt
7185 could be placed after
7186 seq_store or before
7187 the seq_load. We
7188 choose the load to
7189 make the s_waitcnt be
7190 as late as possible
7191 so that the store
7192 may have already
7193 completed.)
7195 2. *Following
7196 instructions same as
7197 corresponding load
7198 atomic acquire,
7199 except must generate
7200 all instructions even
7201 for OpenCL.*
7202 store atomic seq_cst - singlethread - global *Same as corresponding
7203 - wavefront - local store atomic release,
7204 - workgroup - generic except must generate
7205 - agent all instructions even
7206 - system for OpenCL.*
7207 atomicrmw seq_cst - singlethread - global *Same as corresponding
7208 - wavefront - local atomicrmw acq_rel,
7209 - workgroup - generic except must generate
7210 - agent all instructions even
7211 - system for OpenCL.*
7212 fence seq_cst - singlethread *none* *Same as corresponding
7213 - wavefront fence acq_rel,
7214 - workgroup except must generate
7215 - agent all instructions even
7216 - system for OpenCL.*
7217 ============ ============ ============== ========== ================================
7219 .. _amdgpu-amdhsa-memory-model-gfx90a:
7221 Memory Model GFX90A
7222 +++++++++++++++++++
7224 For GFX90A:
7226 * Each agent has multiple shader arrays (SA).
7227 * Each SA has multiple compute units (CU).
7228 * Each CU has multiple SIMDs that execute wavefronts.
7229 * The wavefronts for a single work-group are executed in the same CU but may be
7230 executed by different SIMDs. The exception is when in tgsplit execution mode
7231 when the wavefronts may be executed by different SIMDs in different CUs.
7232 * Each CU has a single LDS memory shared by the wavefronts of the work-groups
7233 executing on it. The exception is when in tgsplit execution mode when no LDS
7234 is allocated as wavefronts of the same work-group can be in different CUs.
7235 * All LDS operations of a CU are performed as wavefront wide operations in a
7236 global order and involve no caching. Completion is reported to a wavefront in
7237 execution order.
7238 * The LDS memory has multiple request queues shared by the SIMDs of a
7239 CU. Therefore, the LDS operations performed by different wavefronts of a
7240 work-group can be reordered relative to each other, which can result in
7241 reordering the visibility of vector memory operations with respect to LDS
7242 operations of other wavefronts in the same work-group. A ``s_waitcnt
7243 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and
7244 vector memory operations between wavefronts of a work-group, but not between
7245 operations performed by the same wavefront.
7246 * The vector memory operations are performed as wavefront wide operations and
7247 completion is reported to a wavefront in execution order. The exception is
7248 that ``flat_load/store/atomic`` instructions can report out of vector memory
7249 order if they access LDS memory, and out of LDS operation order if they access
7250 global memory.
7251 * The vector memory operations access a single vector L1 cache shared by all
7252 SIMDs a CU. Therefore:
7254 * No special action is required for coherence between the lanes of a single
7255 wavefront.
7257 * No special action is required for coherence between wavefronts in the same
7258 work-group since they execute on the same CU. The exception is when in
7259 tgsplit execution mode as wavefronts of the same work-group can be in
7260 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in
7261 the following item.
7263 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts
7264 executing in different work-groups as they may be executing on different
7265 CUs.
7267 * The scalar memory operations access a scalar L1 cache shared by all wavefronts
7268 on a group of CUs. The scalar and vector L1 caches are not coherent. However,
7269 scalar operations are used in a restricted way so do not impact the memory
7270 model. See :ref:`amdgpu-amdhsa-memory-spaces`.
7271 * The vector and scalar memory operations use an L2 cache shared by all CUs on
7272 the same agent.
7274 * The L2 cache has independent channels to service disjoint ranges of virtual
7275 addresses.
7276 * Each CU has a separate request queue per channel. Therefore, the vector and
7277 scalar memory operations performed by wavefronts executing in different
7278 work-groups (which may be executing on different CUs), or the same
7279 work-group if executing in tgsplit mode, of an agent can be reordered
7280 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure
7281 synchronization between vector memory operations of different CUs. It
7282 ensures a previous vector memory operation has completed before executing a
7283 subsequent vector memory or LDS operation and so can be used to meet the
7284 requirements of acquire and release.
7285 * The L2 cache of one agent can be kept coherent with other agents by:
7286 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE
7287 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with
7288 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2.
7290 * Any local memory cache lines will be automatically invalidated by writes
7291 from CUs associated with other L2 caches, or writes from the CPU, due to
7292 the cache probe caused by coherent requests. Coherent requests are caused
7293 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over
7294 XGMI, and by PCIe requests that are configured to be coherent requests.
7295 * XGMI accesses from the CPU to local memory may be cached on the CPU.
7296 Subsequent access from the GPU will automatically invalidate or writeback
7297 the CPU cache due to the L2 probe filter and and the PTE C-bit being set.
7298 * Since all work-groups on the same agent share the same L2, no L2
7299 invalidation or writeback is required for coherence.
7300 * To ensure coherence of local and remote memory writes of work-groups in
7301 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2
7302 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC
7303 ()used for remote coarse grain memory). Note that MTYPE CC (used for local
7304 fine grain memory) causes write through to DRAM, and MTYPE UC (used for
7305 remote fine grain memory) bypasses the L2, so both will never result in
7306 dirty L2 cache lines.
7307 * To ensure coherence of local and remote memory reads of work-groups in
7308 different agents a ``buffer_invl2`` is required. It will invalidate L2
7309 cache lines with MTYPE NC (used for remote coarse grain memory). Note that
7310 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local
7311 coarse memory) cause local reads to be invalidated by remote writes with
7312 with the PTE C-bit so these cache lines are not invalidated. Note that
7313 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will
7314 never result in L2 cache lines that need to be invalidated.
7316 * PCIe access from the GPU to the CPU memory is kept coherent by using the
7317 MTYPE UC (uncached) which bypasses the L2.
7319 Scalar memory operations are only used to access memory that is proven to not
7320 change during the execution of the kernel dispatch. This includes constant
7321 address space and global address space for program scope ``const`` variables.
7322 Therefore, the kernel machine code does not have to maintain the scalar cache to
7323 ensure it is coherent with the vector caches. The scalar and vector caches are
7324 invalidated between kernel dispatches by CP since constant address space data
7325 may change between kernel dispatch executions. See
7326 :ref:`amdgpu-amdhsa-memory-spaces`.
7328 The one exception is if scalar writes are used to spill SGPR registers. In this
7329 case the AMDGPU backend ensures the memory location used to spill is never
7330 accessed by vector memory operations at the same time. If scalar writes are used
7331 then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function
7332 return since the locations may be used for vector memory instructions by a
7333 future wavefront that uses the same scratch area, or a function call that
7334 creates a frame at the same address, respectively. There is no need for a
7335 ``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.
7337 For kernarg backing memory:
7339 * CP invalidates the L1 cache at the start of each kernel dispatch.
7340 * On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host
7341 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2
7342 cache. This also causes it to be treated as non-volatile and so is not
7343 invalidated by ``*_vol``.
7344 * On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and
7345 so the L2 cache will be coherent with the CPU and other agents.
7347 Scratch backing memory (which is used for the private address space) is accessed
7348 with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is
7349 only accessed by a single thread, and is always write-before-read, there is
7350 never a need to invalidate these entries from the L1 cache. Hence all cache
7351 invalidates are done as ``*_vol`` to only invalidate the volatile cache lines.
7353 The code sequences used to implement the memory model for GFX90A are defined
7354 in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`.
7356 .. table:: AMDHSA Memory Model Code Sequences GFX90A
7357 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table
7359 ============ ============ ============== ========== ================================
7360 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code
7361 Ordering Sync Scope Address GFX90A
7362 Space
7363 ============ ============ ============== ========== ================================
7364 **Non-Atomic**
7365 ------------------------------------------------------------------------------------
7366 load *none* *none* - global - !volatile & !nontemporal
7367 - generic
7368 - private 1. buffer/global/flat_load
7369 - constant
7370 - !volatile & nontemporal
7372 1. buffer/global/flat_load
7373 glc=1 slc=1
7375 - volatile
7377 1. buffer/global/flat_load
7378 glc=1
7379 2. s_waitcnt vmcnt(0)
7381 - Must happen before
7382 any following volatile
7383 global/generic
7384 load/store.
7385 - Ensures that
7386 volatile
7387 operations to
7388 different
7389 addresses will not
7390 be reordered by
7391 hardware.
7393 load *none* *none* - local 1. ds_load
7394 store *none* *none* - global - !volatile & !nontemporal
7395 - generic
7396 - private 1. buffer/global/flat_store
7397 - constant
7398 - !volatile & nontemporal
7400 1. buffer/global/flat_store
7401 glc=1 slc=1
7403 - volatile
7405 1. buffer/global/flat_store
7406 2. s_waitcnt vmcnt(0)
7408 - Must happen before
7409 any following volatile
7410 global/generic
7411 load/store.
7412 - Ensures that
7413 volatile
7414 operations to
7415 different
7416 addresses will not
7417 be reordered by
7418 hardware.
7420 store *none* *none* - local 1. ds_store
7421 **Unordered Atomic**
7422 ------------------------------------------------------------------------------------
7423 load atomic unordered *any* *any* *Same as non-atomic*.
7424 store atomic unordered *any* *any* *Same as non-atomic*.
7425 atomicrmw unordered *any* *any* *Same as monotonic atomic*.
7426 **Monotonic Atomic**
7427 ------------------------------------------------------------------------------------
7428 load atomic monotonic - singlethread - global 1. buffer/global/flat_load
7429 - wavefront - generic
7430 load atomic monotonic - workgroup - global 1. buffer/global/flat_load
7431 - generic glc=1
7433 - If not TgSplit execution
7434 mode, omit glc=1.
7436 load atomic monotonic - singlethread - local *If TgSplit execution mode,
7437 - wavefront local address space cannot
7438 - workgroup be used.*
7440 1. ds_load
7441 load atomic monotonic - agent - global 1. buffer/global/flat_load
7442 - generic glc=1
7443 load atomic monotonic - system - global 1. buffer/global/flat_load
7444 - generic glc=1
7445 store atomic monotonic - singlethread - global 1. buffer/global/flat_store
7446 - wavefront - generic
7447 - workgroup
7448 - agent
7449 store atomic monotonic - system - global 1. buffer/global/flat_store
7450 - generic
7451 store atomic monotonic - singlethread - local *If TgSplit execution mode,
7452 - wavefront local address space cannot
7453 - workgroup be used.*
7455 1. ds_store
7456 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic
7457 - wavefront - generic
7458 - workgroup
7459 - agent
7460 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic
7461 - generic
7462 atomicrmw monotonic - singlethread - local *If TgSplit execution mode,
7463 - wavefront local address space cannot
7464 - workgroup be used.*
7466 1. ds_atomic
7467 **Acquire Atomic**
7468 ------------------------------------------------------------------------------------
7469 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load
7470 - wavefront - local
7471 - generic
7472 load atomic acquire - workgroup - global 1. buffer/global_load glc=1
7474 - If not TgSplit execution
7475 mode, omit glc=1.
7477 2. s_waitcnt vmcnt(0)
7479 - If not TgSplit execution
7480 mode, omit.
7481 - Must happen before the
7482 following buffer_wbinvl1_vol.
7484 3. buffer_wbinvl1_vol
7486 - If not TgSplit execution
7487 mode, omit.
7488 - Must happen before
7489 any following
7490 global/generic
7491 load/load
7492 atomic/store/store
7493 atomic/atomicrmw.
7494 - Ensures that
7495 following
7496 loads will not see
7497 stale data.
7499 load atomic acquire - workgroup - local *If TgSplit execution mode,
7500 local address space cannot
7501 be used.*
7503 1. ds_load
7504 2. s_waitcnt lgkmcnt(0)
7506 - If OpenCL, omit.
7507 - Must happen before
7508 any following
7509 global/generic
7510 load/load
7511 atomic/store/store
7512 atomic/atomicrmw.
7513 - Ensures any
7514 following global
7515 data read is no
7516 older than the local load
7517 atomic value being
7518 acquired.
7520 load atomic acquire - workgroup - generic 1. flat_load glc=1
7522 - If not TgSplit execution
7523 mode, omit glc=1.
7525 2. s_waitcnt lgkm/vmcnt(0)
7527 - Use lgkmcnt(0) if not
7528 TgSplit execution mode
7529 and vmcnt(0) if TgSplit
7530 execution mode.
7531 - If OpenCL, omit lgkmcnt(0).
7532 - Must happen before
7533 the following
7534 buffer_wbinvl1_vol and any
7535 following global/generic
7536 load/load
7537 atomic/store/store
7538 atomic/atomicrmw.
7539 - Ensures any
7540 following global
7541 data read is no
7542 older than a local load
7543 atomic value being
7544 acquired.
7546 3. buffer_wbinvl1_vol
7548 - If not TgSplit execution
7549 mode, omit.
7550 - Ensures that
7551 following
7552 loads will not see
7553 stale data.
7555 load atomic acquire - agent - global 1. buffer/global_load
7556 glc=1
7557 2. s_waitcnt vmcnt(0)
7559 - Must happen before
7560 following
7561 buffer_wbinvl1_vol.
7562 - Ensures the load
7563 has completed
7564 before invalidating
7565 the cache.
7567 3. buffer_wbinvl1_vol
7569 - Must happen before
7570 any following
7571 global/generic
7572 load/load
7573 atomic/atomicrmw.
7574 - Ensures that
7575 following
7576 loads will not see
7577 stale global data.
7579 load atomic acquire - system - global 1. buffer/global/flat_load
7580 glc=1
7581 2. s_waitcnt vmcnt(0)
7583 - Must happen before
7584 following buffer_invl2 and
7585 buffer_wbinvl1_vol.
7586 - Ensures the load
7587 has completed
7588 before invalidating
7589 the cache.
7591 3. buffer_invl2;
7592 buffer_wbinvl1_vol
7594 - Must happen before
7595 any following
7596 global/generic
7597 load/load
7598 atomic/atomicrmw.
7599 - Ensures that
7600 following
7601 loads will not see
7602 stale L1 global data,
7603 nor see stale L2 MTYPE
7604 NC global data.
7605 MTYPE RW and CC memory will
7606 never be stale in L2 due to
7607 the memory probes.
7609 load atomic acquire - agent - generic 1. flat_load glc=1
7610 2. s_waitcnt vmcnt(0) &
7611 lgkmcnt(0)
7613 - If TgSplit execution mode,
7614 omit lgkmcnt(0).
7615 - If OpenCL omit
7616 lgkmcnt(0).
7617 - Must happen before
7618 following
7619 buffer_wbinvl1_vol.
7620 - Ensures the flat_load
7621 has completed
7622 before invalidating
7623 the cache.
7625 3. buffer_wbinvl1_vol
7627 - Must happen before
7628 any following
7629 global/generic
7630 load/load
7631 atomic/atomicrmw.
7632 - Ensures that
7633 following loads
7634 will not see stale
7635 global data.
7637 load atomic acquire - system - generic 1. flat_load glc=1
7638 2. s_waitcnt vmcnt(0) &
7639 lgkmcnt(0)
7641 - If TgSplit execution mode,
7642 omit lgkmcnt(0).
7643 - If OpenCL omit
7644 lgkmcnt(0).
7645 - Must happen before
7646 following
7647 buffer_invl2 and
7648 buffer_wbinvl1_vol.
7649 - Ensures the flat_load
7650 has completed
7651 before invalidating
7652 the caches.
7654 3. buffer_invl2;
7655 buffer_wbinvl1_vol
7657 - Must happen before
7658 any following
7659 global/generic
7660 load/load
7661 atomic/atomicrmw.
7662 - Ensures that
7663 following
7664 loads will not see
7665 stale L1 global data,
7666 nor see stale L2 MTYPE
7667 NC global data.
7668 MTYPE RW and CC memory will
7669 never be stale in L2 due to
7670 the memory probes.
7672 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic
7673 - wavefront - generic
7674 atomicrmw acquire - singlethread - local *If TgSplit execution mode,
7675 - wavefront local address space cannot
7676 be used.*
7678 1. ds_atomic
7679 atomicrmw acquire - workgroup - global 1. buffer/global_atomic
7680 2. s_waitcnt vmcnt(0)
7682 - If not TgSplit execution
7683 mode, omit.
7684 - Must happen before the
7685 following buffer_wbinvl1_vol.
7686 - Ensures the atomicrmw
7687 has completed
7688 before invalidating
7689 the cache.
7691 3. buffer_wbinvl1_vol
7693 - If not TgSplit execution
7694 mode, omit.
7695 - Must happen before
7696 any following
7697 global/generic
7698 load/load
7699 atomic/atomicrmw.
7700 - Ensures that
7701 following loads
7702 will not see stale
7703 global data.
7705 atomicrmw acquire - workgroup - local *If TgSplit execution mode,
7706 local address space cannot
7707 be used.*
7709 1. ds_atomic
7710 2. s_waitcnt lgkmcnt(0)
7712 - If OpenCL, omit.
7713 - Must happen before
7714 any following
7715 global/generic
7716 load/load
7717 atomic/store/store
7718 atomic/atomicrmw.
7719 - Ensures any
7720 following global
7721 data read is no
7722 older than the local
7723 atomicrmw value
7724 being acquired.
7726 atomicrmw acquire - workgroup - generic 1. flat_atomic
7727 2. s_waitcnt lgkm/vmcnt(0)
7729 - Use lgkmcnt(0) if not
7730 TgSplit execution mode
7731 and vmcnt(0) if TgSplit
7732 execution mode.
7733 - If OpenCL, omit lgkmcnt(0).
7734 - Must happen before
7735 the following
7736 buffer_wbinvl1_vol and
7737 any following
7738 global/generic
7739 load/load
7740 atomic/store/store
7741 atomic/atomicrmw.
7742 - Ensures any
7743 following global
7744 data read is no
7745 older than a local
7746 atomicrmw value
7747 being acquired.
7749 3. buffer_wbinvl1_vol
7751 - If not TgSplit execution
7752 mode, omit.
7753 - Ensures that
7754 following
7755 loads will not see
7756 stale data.
7758 atomicrmw acquire - agent - global 1. buffer/global_atomic
7759 2. s_waitcnt vmcnt(0)
7761 - Must happen before
7762 following
7763 buffer_wbinvl1_vol.
7764 - Ensures the
7765 atomicrmw has
7766 completed before
7767 invalidating the
7768 cache.
7770 3. buffer_wbinvl1_vol
7772 - Must happen before
7773 any following
7774 global/generic
7775 load/load
7776 atomic/atomicrmw.
7777 - Ensures that
7778 following loads
7779 will not see stale
7780 global data.
7782 atomicrmw acquire - system - global 1. buffer/global_atomic
7783 2. s_waitcnt vmcnt(0)
7785 - Must happen before
7786 following buffer_invl2 and
7787 buffer_wbinvl1_vol.
7788 - Ensures the
7789 atomicrmw has
7790 completed before
7791 invalidating the
7792 caches.
7794 3. buffer_invl2;
7795 buffer_wbinvl1_vol
7797 - Must happen before
7798 any following
7799 global/generic
7800 load/load
7801 atomic/atomicrmw.
7802 - Ensures that
7803 following
7804 loads will not see
7805 stale L1 global data,
7806 nor see stale L2 MTYPE
7807 NC global data.
7808 MTYPE RW and CC memory will
7809 never be stale in L2 due to
7810 the memory probes.
7812 atomicrmw acquire - agent - generic 1. flat_atomic
7813 2. s_waitcnt vmcnt(0) &
7814 lgkmcnt(0)
7816 - If TgSplit execution mode,
7817 omit lgkmcnt(0).
7818 - If OpenCL, omit
7819 lgkmcnt(0).
7820 - Must happen before
7821 following
7822 buffer_wbinvl1_vol.
7823 - Ensures the
7824 atomicrmw has
7825 completed before
7826 invalidating the
7827 cache.
7829 3. buffer_wbinvl1_vol
7831 - Must happen before
7832 any following
7833 global/generic
7834 load/load
7835 atomic/atomicrmw.
7836 - Ensures that
7837 following loads
7838 will not see stale
7839 global data.
7841 atomicrmw acquire - system - generic 1. flat_atomic
7842 2. s_waitcnt vmcnt(0) &
7843 lgkmcnt(0)
7845 - If TgSplit execution mode,
7846 omit lgkmcnt(0).
7847 - If OpenCL, omit
7848 lgkmcnt(0).
7849 - Must happen before
7850 following
7851 buffer_invl2 and
7852 buffer_wbinvl1_vol.
7853 - Ensures the
7854 atomicrmw has
7855 completed before
7856 invalidating the
7857 caches.
7859 3. buffer_invl2;
7860 buffer_wbinvl1_vol
7862 - Must happen before
7863 any following
7864 global/generic
7865 load/load
7866 atomic/atomicrmw.
7867 - Ensures that
7868 following
7869 loads will not see
7870 stale L1 global data,
7871 nor see stale L2 MTYPE
7872 NC global data.
7873 MTYPE RW and CC memory will
7874 never be stale in L2 due to
7875 the memory probes.
7877 fence acquire - singlethread *none* *none*
7878 - wavefront
7879 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)
7881 - Use lgkmcnt(0) if not
7882 TgSplit execution mode
7883 and vmcnt(0) if TgSplit
7884 execution mode.
7885 - If OpenCL and
7886 address space is
7887 not generic, omit
7888 lgkmcnt(0).
7889 - If OpenCL and
7890 address space is
7891 local, omit
7892 vmcnt(0).
7893 - However, since LLVM
7894 currently has no
7895 address space on
7896 the fence need to
7897 conservatively
7898 always generate. If
7899 fence had an
7900 address space then
7901 set to address
7902 space of OpenCL
7903 fence flag, or to
7904 generic if both
7905 local and global
7906 flags are
7907 specified.
7908 - s_waitcnt vmcnt(0)
7909 must happen after
7910 any preceding
7911 global/generic load
7912 atomic/
7913 atomicrmw
7914 with an equal or
7915 wider sync scope
7916 and memory ordering
7917 stronger than
7918 unordered (this is
7919 termed the
7920 fence-paired-atomic).
7921 - s_waitcnt lgkmcnt(0)
7922 must happen after
7923 any preceding
7924 local/generic load
7925 atomic/atomicrmw
7926 with an equal or
7927 wider sync scope
7928 and memory ordering
7929 stronger than
7930 unordered (this is
7931 termed the
7932 fence-paired-atomic).
7933 - Must happen before
7934 the following
7935 buffer_wbinvl1_vol and
7936 any following
7937 global/generic
7938 load/load
7939 atomic/store/store
7940 atomic/atomicrmw.
7941 - Ensures any
7942 following global
7943 data read is no
7944 older than the
7945 value read by the
7946 fence-paired-atomic.
7948 2. buffer_wbinvl1_vol
7950 - If not TgSplit execution
7951 mode, omit.
7952 - Ensures that
7953 following
7954 loads will not see
7955 stale data.
7957 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) &
7958 vmcnt(0)
7960 - If TgSplit execution mode,
7961 omit lgkmcnt(0).
7962 - If OpenCL and
7963 address space is
7964 not generic, omit
7965 lgkmcnt(0).
7966 - However, since LLVM
7967 currently has no
7968 address space on
7969 the fence need to
7970 conservatively
7971 always generate
7972 (see comment for
7973 previous fence).
7974 - Could be split into
7975 separate s_waitcnt
7976 vmcnt(0) and
7977 s_waitcnt
7978 lgkmcnt(0) to allow
7979 them to be
7980 independently moved
7981 according to the
7982 following rules.
7983 - s_waitcnt vmcnt(0)
7984 must happen after
7985 any preceding
7986 global/generic load
7987 atomic/atomicrmw
7988 with an equal or
7989 wider sync scope
7990 and memory ordering
7991 stronger than
7992 unordered (this is
7993 termed the
7994 fence-paired-atomic).
7995 - s_waitcnt lgkmcnt(0)
7996 must happen after
7997 any preceding
7998 local/generic load
7999 atomic/atomicrmw
8000 with an equal or
8001 wider sync scope
8002 and memory ordering
8003 stronger than
8004 unordered (this is
8005 termed the
8006 fence-paired-atomic).
8007 - Must happen before
8008 the following
8009 buffer_wbinvl1_vol.
8010 - Ensures that the
8011 fence-paired atomic
8012 has completed
8013 before invalidating
8014 the
8015 cache. Therefore
8016 any following
8017 locations read must
8018 be no older than
8019 the value read by
8020 the
8021 fence-paired-atomic.
8023 2. buffer_wbinvl1_vol
8025 - Must happen before any
8026 following global/generic
8027 load/load
8028 atomic/store/store
8029 atomic/atomicrmw.
8030 - Ensures that
8031 following loads
8032 will not see stale
8033 global data.
8035 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) &
8036 vmcnt(0)
8038 - If TgSplit execution mode,
8039 omit lgkmcnt(0).
8040 - If OpenCL and
8041 address space is
8042 not generic, omit
8043 lgkmcnt(0).
8044 - However, since LLVM
8045 currently has no
8046 address space on
8047 the fence need to
8048 conservatively
8049 always generate
8050 (see comment for
8051 previous fence).
8052 - Could be split into
8053 separate s_waitcnt
8054 vmcnt(0) and
8055 s_waitcnt
8056 lgkmcnt(0) to allow
8057 them to be
8058 independently moved
8059 according to the
8060 following rules.
8061 - s_waitcnt vmcnt(0)
8062 must happen after
8063 any preceding
8064 global/generic load
8065 atomic/atomicrmw
8066 with an equal or
8067 wider sync scope
8068 and memory ordering
8069 stronger than
8070 unordered (this is
8071 termed the
8072 fence-paired-atomic).
8073 - s_waitcnt lgkmcnt(0)
8074 must happen after
8075 any preceding
8076 local/generic load
8077 atomic/atomicrmw
8078 with an equal or
8079 wider sync scope
8080 and memory ordering
8081 stronger than
8082 unordered (this is
8083 termed the
8084 fence-paired-atomic).
8085 - Must happen before
8086 the following buffer_invl2 and
8087 buffer_wbinvl1_vol.
8088 - Ensures that the
8089 fence-paired atomic
8090 has completed
8091 before invalidating
8092 the
8093 cache. Therefore
8094 any following
8095 locations read must
8096 be no older than
8097 the value read by
8098 the
8099 fence-paired-atomic.
8101 2. buffer_invl2;
8102 buffer_wbinvl1_vol
8104 - Must happen before any
8105 following global/generic
8106 load/load
8107 atomic/store/store
8108 atomic/atomicrmw.
8109 - Ensures that
8110 following
8111 loads will not see
8112 stale L1 global data,
8113 nor see stale L2 MTYPE
8114 NC global data.
8115 MTYPE RW and CC memory will
8116 never be stale in L2 due to
8117 the memory probes.
8118 **Release Atomic**
8119 ------------------------------------------------------------------------------------
8120 store atomic release - singlethread - global 1. buffer/global/flat_store
8121 - wavefront - generic
8122 store atomic release - singlethread - local *If TgSplit execution mode,
8123 - wavefront local address space cannot
8124 be used.*
8126 1. ds_store
8127 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)
8128 - generic
8129 - Use lgkmcnt(0) if not
8130 TgSplit execution mode
8131 and vmcnt(0) if TgSplit
8132 execution mode.
8133 - If OpenCL, omit lgkmcnt(0).
8134 - s_waitcnt vmcnt(0)
8135 must happen after
8136 any preceding
8137 global/generic load/store/
8138 load atomic/store atomic/
8139 atomicrmw.
8140 - s_waitcnt lgkmcnt(0)
8141 must happen after
8142 any preceding
8143 local/generic
8144 load/store/load
8145 atomic/store
8146 atomic/atomicrmw.
8147 - Must happen before
8148 the following
8149 store.
8150 - Ensures that all
8151 memory operations
8152 have
8153 completed before
8154 performing the
8155 store that is being
8156 released.
8158 2. buffer/global/flat_store
8159 store atomic release - workgroup - local *If TgSplit execution mode,
8160 local address space cannot
8161 be used.*
8163 1. ds_store
8164 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &
8165 - generic vmcnt(0)
8167 - If TgSplit execution mode,
8168 omit lgkmcnt(0).
8169 - If OpenCL and
8170 address space is
8171 not generic, omit
8172 lgkmcnt(0).
8173 - Could be split into
8174 separate s_waitcnt
8175 vmcnt(0) and
8176 s_waitcnt
8177 lgkmcnt(0) to allow
8178 them to be
8179 independently moved
8180 according to the
8181 following rules.
8182 - s_waitcnt vmcnt(0)
8183 must happen after
8184 any preceding
8185 global/generic
8186 load/store/load
8187 atomic/store
8188 atomic/atomicrmw.
8189 - s_waitcnt lgkmcnt(0)
8190 must happen after
8191 any preceding
8192 local/generic
8193 load/store/load
8194 atomic/store
8195 atomic/atomicrmw.
8196 - Must happen before
8197 the following
8198 store.
8199 - Ensures that all
8200 memory operations
8201 to memory have
8202 completed before
8203 performing the
8204 store that is being
8205 released.
8207 2. buffer/global/flat_store
8208 store atomic release - system - global 1. buffer_wbl2
8209 - generic
8210 - Must happen before
8211 following s_waitcnt.
8212 - Performs L2 writeback to
8213 ensure previous
8214 global/generic
8215 store/atomicrmw are
8216 visible at system scope.
8218 2. s_waitcnt lgkmcnt(0) &
8219 vmcnt(0)
8221 - If TgSplit execution mode,
8222 omit lgkmcnt(0).
8223 - If OpenCL and
8224 address space is
8225 not generic, omit
8226 lgkmcnt(0).
8227 - Could be split into
8228 separate s_waitcnt
8229 vmcnt(0) and
8230 s_waitcnt
8231 lgkmcnt(0) to allow
8232 them to be
8233 independently moved
8234 according to the
8235 following rules.
8236 - s_waitcnt vmcnt(0)
8237 must happen after any
8238 preceding
8239 global/generic
8240 load/store/load
8241 atomic/store
8242 atomic/atomicrmw.
8243 - s_waitcnt lgkmcnt(0)
8244 must happen after any
8245 preceding
8246 local/generic
8247 load/store/load
8248 atomic/store
8249 atomic/atomicrmw.
8250 - Must happen before
8251 the following
8252 store.
8253 - Ensures that all
8254 memory operations
8255 to memory and the L2
8256 writeback have
8257 completed before
8258 performing the
8259 store that is being
8260 released.
8262 3. buffer/global/flat_store
8263 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic
8264 - wavefront - generic
8265 atomicrmw release - singlethread - local *If TgSplit execution mode,
8266 - wavefront local address space cannot
8267 be used.*
8269 1. ds_atomic
8270 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)
8271 - generic
8272 - Use lgkmcnt(0) if not
8273 TgSplit execution mode
8274 and vmcnt(0) if TgSplit
8275 execution mode.
8276 - If OpenCL, omit
8277 lgkmcnt(0).
8278 - s_waitcnt vmcnt(0)
8279 must happen after
8280 any preceding
8281 global/generic load/store/
8282 load atomic/store atomic/
8283 atomicrmw.
8284 - s_waitcnt lgkmcnt(0)
8285 must happen after
8286 any preceding
8287 local/generic
8288 load/store/load
8289 atomic/store
8290 atomic/atomicrmw.
8291 - Must happen before
8292 the following
8293 atomicrmw.
8294 - Ensures that all
8295 memory operations
8296 have
8297 completed before
8298 performing the
8299 atomicrmw that is
8300 being released.
8302 2. buffer/global/flat_atomic
8303 atomicrmw release - workgroup - local *If TgSplit execution mode,
8304 local address space cannot
8305 be used.*
8307 1. ds_atomic
8308 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &
8309 - generic vmcnt(0)
8311 - If TgSplit execution mode,
8312 omit lgkmcnt(0).
8313 - If OpenCL, omit
8314 lgkmcnt(0).
8315 - Could be split into
8316 separate s_waitcnt
8317 vmcnt(0) and
8318 s_waitcnt
8319 lgkmcnt(0) to allow
8320 them to be
8321 independently moved
8322 according to the
8323 following rules.
8324 - s_waitcnt vmcnt(0)
8325 must happen after
8326 any preceding
8327 global/generic
8328 load/store/load
8329 atomic/store
8330 atomic/atomicrmw.
8331 - s_waitcnt lgkmcnt(0)
8332 must happen after
8333 any preceding
8334 local/generic
8335 load/store/load
8336 atomic/store
8337 atomic/atomicrmw.
8338 - Must happen before
8339 the following
8340 atomicrmw.
8341 - Ensures that all
8342 memory operations
8343 to global and local
8344 have completed
8345 before performing
8346 the atomicrmw that
8347 is being released.
8349 2. buffer/global/flat_atomic
8350 atomicrmw release - system - global 1. buffer_wbl2
8351 - generic
8352 - Must happen before
8353 following s_waitcnt.
8354 - Performs L2 writeback to
8355 ensure previous
8356 global/generic
8357 store/atomicrmw are
8358 visible at system scope.
8360 2. s_waitcnt lgkmcnt(0) &
8361 vmcnt(0)
8363 - If TgSplit execution mode,
8364 omit lgkmcnt(0).
8365 - If OpenCL, omit
8366 lgkmcnt(0).
8367 - Could be split into
8368 separate s_waitcnt
8369 vmcnt(0) and
8370 s_waitcnt
8371 lgkmcnt(0) to allow
8372 them to be
8373 independently moved
8374 according to the
8375 following rules.
8376 - s_waitcnt vmcnt(0)
8377 must happen after
8378 any preceding
8379 global/generic
8380 load/store/load
8381 atomic/store
8382 atomic/atomicrmw.
8383 - s_waitcnt lgkmcnt(0)
8384 must happen after
8385 any preceding
8386 local/generic
8387 load/store/load
8388 atomic/store
8389 atomic/atomicrmw.
8390 - Must happen before
8391 the following
8392 atomicrmw.
8393 - Ensures that all
8394 memory operations
8395 to memory and the L2
8396 writeback have
8397 completed before
8398 performing the
8399 store that is being
8400 released.
8402 3. buffer/global/flat_atomic
8403 fence release - singlethread *none* *none*
8404 - wavefront
8405 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)
8407 - Use lgkmcnt(0) if not
8408 TgSplit execution mode
8409 and vmcnt(0) if TgSplit
8410 execution mode.
8411 - If OpenCL and
8412 address space is
8413 not generic, omit
8414 lgkmcnt(0).
8415 - If OpenCL and
8416 address space is
8417 local, omit
8418 vmcnt(0).
8419 - However, since LLVM
8420 currently has no
8421 address space on
8422 the fence need to
8423 conservatively
8424 always generate. If
8425 fence had an
8426 address space then
8427 set to address
8428 space of OpenCL
8429 fence flag, or to
8430 generic if both
8431 local and global
8432 flags are
8433 specified.
8434 - s_waitcnt vmcnt(0)
8435 must happen after
8436 any preceding
8437 global/generic
8438 load/store/
8439 load atomic/store atomic/
8440 atomicrmw.
8441 - s_waitcnt lgkmcnt(0)
8442 must happen after
8443 any preceding
8444 local/generic
8445 load/load
8446 atomic/store/store
8447 atomic/atomicrmw.
8448 - Must happen before
8449 any following store
8450 atomic/atomicrmw
8451 with an equal or
8452 wider sync scope
8453 and memory ordering
8454 stronger than
8455 unordered (this is
8456 termed the
8457 fence-paired-atomic).
8458 - Ensures that all
8459 memory operations
8460 have
8461 completed before
8462 performing the
8463 following
8464 fence-paired-atomic.
8466 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) &
8467 vmcnt(0)
8469 - If TgSplit execution mode,
8470 omit lgkmcnt(0).
8471 - If OpenCL and
8472 address space is
8473 not generic, omit
8474 lgkmcnt(0).
8475 - If OpenCL and
8476 address space is
8477 local, omit
8478 vmcnt(0).
8479 - However, since LLVM
8480 currently has no
8481 address space on
8482 the fence need to
8483 conservatively
8484 always generate. If
8485 fence had an
8486 address space then
8487 set to address
8488 space of OpenCL
8489 fence flag, or to
8490 generic if both
8491 local and global
8492 flags are
8493 specified.
8494 - Could be split into
8495 separate s_waitcnt
8496 vmcnt(0) and
8497 s_waitcnt
8498 lgkmcnt(0) to allow
8499 them to be
8500 independently moved
8501 according to the
8502 following rules.
8503 - s_waitcnt vmcnt(0)
8504 must happen after
8505 any preceding
8506 global/generic
8507 load/store/load
8508 atomic/store
8509 atomic/atomicrmw.
8510 - s_waitcnt lgkmcnt(0)
8511 must happen after
8512 any preceding
8513 local/generic
8514 load/store/load
8515 atomic/store
8516 atomic/atomicrmw.
8517 - Must happen before
8518 any following store
8519 atomic/atomicrmw
8520 with an equal or
8521 wider sync scope
8522 and memory ordering
8523 stronger than
8524 unordered (this is
8525 termed the
8526 fence-paired-atomic).
8527 - Ensures that all
8528 memory operations
8529 have
8530 completed before
8531 performing the
8532 following
8533 fence-paired-atomic.
8535 fence release - system *none* 1. buffer_wbl2
8537 - If OpenCL and
8538 address space is
8539 local, omit.
8540 - Must happen before
8541 following s_waitcnt.
8542 - Performs L2 writeback to
8543 ensure previous
8544 global/generic
8545 store/atomicrmw are
8546 visible at system scope.
8548 2. s_waitcnt lgkmcnt(0) &
8549 vmcnt(0)
8551 - If TgSplit execution mode,
8552 omit lgkmcnt(0).
8553 - If OpenCL and
8554 address space is
8555 not generic, omit
8556 lgkmcnt(0).
8557 - If OpenCL and
8558 address space is
8559 local, omit
8560 vmcnt(0).
8561 - However, since LLVM
8562 currently has no
8563 address space on
8564 the fence need to
8565 conservatively
8566 always generate. If
8567 fence had an
8568 address space then
8569 set to address
8570 space of OpenCL
8571 fence flag, or to
8572 generic if both
8573 local and global
8574 flags are
8575 specified.
8576 - Could be split into
8577 separate s_waitcnt
8578 vmcnt(0) and
8579 s_waitcnt
8580 lgkmcnt(0) to allow
8581 them to be
8582 independently moved
8583 according to the
8584 following rules.
8585 - s_waitcnt vmcnt(0)
8586 must happen after
8587 any preceding
8588 global/generic
8589 load/store/load
8590 atomic/store
8591 atomic/atomicrmw.
8592 - s_waitcnt lgkmcnt(0)
8593 must happen after
8594 any preceding
8595 local/generic
8596 load/store/load
8597 atomic/store
8598 atomic/atomicrmw.
8599 - Must happen before
8600 any following store
8601 atomic/atomicrmw
8602 with an equal or
8603 wider sync scope
8604 and memory ordering
8605 stronger than
8606 unordered (this is
8607 termed the
8608 fence-paired-atomic).
8609 - Ensures that all
8610 memory operations
8611 have
8612 completed before
8613 performing the
8614 following
8615 fence-paired-atomic.
8617 **Acquire-Release Atomic**
8618 ------------------------------------------------------------------------------------
8619 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic
8620 - wavefront - generic
8621 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode,
8622 - wavefront local address space cannot
8623 be used.*
8625 1. ds_atomic
8626 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)
8628 - Use lgkmcnt(0) if not
8629 TgSplit execution mode
8630 and vmcnt(0) if TgSplit
8631 execution mode.
8632 - If OpenCL, omit
8633 lgkmcnt(0).
8634 - Must happen after
8635 any preceding
8636 local/generic
8637 load/store/load
8638 atomic/store
8639 atomic/atomicrmw.
8640 - s_waitcnt vmcnt(0)
8641 must happen after
8642 any preceding
8643 global/generic load/store/
8644 load atomic/store atomic/
8645 atomicrmw.
8646 - s_waitcnt lgkmcnt(0)
8647 must happen after
8648 any preceding
8649 local/generic
8650 load/store/load
8651 atomic/store
8652 atomic/atomicrmw.
8653 - Must happen before
8654 the following
8655 atomicrmw.
8656 - Ensures that all
8657 memory operations
8658 have
8659 completed before
8660 performing the
8661 atomicrmw that is
8662 being released.
8664 2. buffer/global_atomic
8665 3. s_waitcnt vmcnt(0)
8667 - If not TgSplit execution
8668 mode, omit.
8669 - Must happen before
8670 the following
8671 buffer_wbinvl1_vol.
8672 - Ensures any
8673 following global
8674 data read is no
8675 older than the
8676 atomicrmw value
8677 being acquired.
8679 4. buffer_wbinvl1_vol
8681 - If not TgSplit execution
8682 mode, omit.
8683 - Ensures that
8684 following
8685 loads will not see
8686 stale data.
8688 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode,
8689 local address space cannot
8690 be used.*
8692 1. ds_atomic
8693 2. s_waitcnt lgkmcnt(0)
8695 - If OpenCL, omit.
8696 - Must happen before
8697 any following
8698 global/generic
8699 load/load
8700 atomic/store/store
8701 atomic/atomicrmw.
8702 - Ensures any
8703 following global
8704 data read is no
8705 older than the local load
8706 atomic value being
8707 acquired.
8709 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0)
8711 - Use lgkmcnt(0) if not
8712 TgSplit execution mode
8713 and vmcnt(0) if TgSplit
8714 execution mode.
8715 - If OpenCL, omit
8716 lgkmcnt(0).
8717 - s_waitcnt vmcnt(0)
8718 must happen after
8719 any preceding
8720 global/generic load/store/
8721 load atomic/store atomic/
8722 atomicrmw.
8723 - s_waitcnt lgkmcnt(0)
8724 must happen after
8725 any preceding
8726 local/generic
8727 load/store/load
8728 atomic/store
8729 atomic/atomicrmw.
8730 - Must happen before
8731 the following
8732 atomicrmw.
8733 - Ensures that all
8734 memory operations
8735 have
8736 completed before
8737 performing the
8738 atomicrmw that is
8739 being released.
8741 2. flat_atomic
8742 3. s_waitcnt lgkmcnt(0) &
8743 vmcnt(0)
8745 - If not TgSplit execution
8746 mode, omit vmcnt(0).
8747 - If OpenCL, omit
8748 lgkmcnt(0).
8749 - Must happen before
8750 the following
8751 buffer_wbinvl1_vol and
8752 any following
8753 global/generic
8754 load/load
8755 atomic/store/store
8756 atomic/atomicrmw.
8757 - Ensures any
8758 following global
8759 data read is no
8760 older than a local load
8761 atomic value being
8762 acquired.
8764 3. buffer_wbinvl1_vol
8766 - If not TgSplit execution
8767 mode, omit.
8768 - Ensures that
8769 following
8770 loads will not see
8771 stale data.
8773 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &
8774 vmcnt(0)
8776 - If TgSplit execution mode,
8777 omit lgkmcnt(0).
8778 - If OpenCL, omit
8779 lgkmcnt(0).
8780 - Could be split into
8781 separate s_waitcnt
8782 vmcnt(0) and
8783 s_waitcnt
8784 lgkmcnt(0) to allow
8785 them to be
8786 independently moved
8787 according to the
8788 following rules.
8789 - s_waitcnt vmcnt(0)
8790 must happen after
8791 any preceding
8792 global/generic
8793 load/store/load
8794 atomic/store
8795 atomic/atomicrmw.
8796 - s_waitcnt lgkmcnt(0)
8797 must happen after
8798 any preceding
8799 local/generic
8800 load/store/load
8801 atomic/store
8802 atomic/atomicrmw.
8803 - Must happen before
8804 the following
8805 atomicrmw.
8806 - Ensures that all
8807 memory operations
8808 to global have
8809 completed before
8810 performing the
8811 atomicrmw that is
8812 being released.
8814 2. buffer/global_atomic
8815 3. s_waitcnt vmcnt(0)
8817 - Must happen before
8818 following
8819 buffer_wbinvl1_vol.
8820 - Ensures the
8821 atomicrmw has
8822 completed before
8823 invalidating the
8824 cache.
8826 4. buffer_wbinvl1_vol
8828 - Must happen before
8829 any following
8830 global/generic
8831 load/load
8832 atomic/atomicrmw.
8833 - Ensures that
8834 following loads
8835 will not see stale
8836 global data.
8838 atomicrmw acq_rel - system - global 1. buffer_wbl2
8840 - Must happen before
8841 following s_waitcnt.
8842 - Performs L2 writeback to
8843 ensure previous
8844 global/generic
8845 store/atomicrmw are
8846 visible at system scope.
8848 2. s_waitcnt lgkmcnt(0) &
8849 vmcnt(0)
8851 - If TgSplit execution mode,
8852 omit lgkmcnt(0).
8853 - If OpenCL, omit
8854 lgkmcnt(0).
8855 - Could be split into
8856 separate s_waitcnt
8857 vmcnt(0) and
8858 s_waitcnt
8859 lgkmcnt(0) to allow
8860 them to be
8861 independently moved
8862 according to the
8863 following rules.
8864 - s_waitcnt vmcnt(0)
8865 must happen after
8866 any preceding
8867 global/generic
8868 load/store/load
8869 atomic/store
8870 atomic/atomicrmw.
8871 - s_waitcnt lgkmcnt(0)
8872 must happen after
8873 any preceding
8874 local/generic
8875 load/store/load
8876 atomic/store
8877 atomic/atomicrmw.
8878 - Must happen before
8879 the following
8880 atomicrmw.
8881 - Ensures that all
8882 memory operations
8883 to global and L2 writeback
8884 have completed before
8885 performing the
8886 atomicrmw that is
8887 being released.
8889 3. buffer/global_atomic
8890 4. s_waitcnt vmcnt(0)
8892 - Must happen before
8893 following buffer_invl2 and
8894 buffer_wbinvl1_vol.
8895 - Ensures the
8896 atomicrmw has
8897 completed before
8898 invalidating the
8899 caches.
8901 5. buffer_invl2;
8902 buffer_wbinvl1_vol
8904 - Must happen before
8905 any following
8906 global/generic
8907 load/load
8908 atomic/atomicrmw.
8909 - Ensures that
8910 following
8911 loads will not see
8912 stale L1 global data,
8913 nor see stale L2 MTYPE
8914 NC global data.
8915 MTYPE RW and CC memory will
8916 never be stale in L2 due to
8917 the memory probes.
8919 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &
8920 vmcnt(0)
8922 - If TgSplit execution mode,
8923 omit lgkmcnt(0).
8924 - If OpenCL, omit
8925 lgkmcnt(0).
8926 - Could be split into
8927 separate s_waitcnt
8928 vmcnt(0) and
8929 s_waitcnt
8930 lgkmcnt(0) to allow
8931 them to be
8932 independently moved
8933 according to the
8934 following rules.
8935 - s_waitcnt vmcnt(0)
8936 must happen after
8937 any preceding
8938 global/generic
8939 load/store/load
8940 atomic/store
8941 atomic/atomicrmw.
8942 - s_waitcnt lgkmcnt(0)
8943 must happen after
8944 any preceding
8945 local/generic
8946 load/store/load
8947 atomic/store
8948 atomic/atomicrmw.
8949 - Must happen before
8950 the following
8951 atomicrmw.
8952 - Ensures that all
8953 memory operations
8954 to global have
8955 completed before
8956 performing the
8957 atomicrmw that is
8958 being released.
8960 2. flat_atomic
8961 3. s_waitcnt vmcnt(0) &
8962 lgkmcnt(0)
8964 - If TgSplit execution mode,
8965 omit lgkmcnt(0).
8966 - If OpenCL, omit
8967 lgkmcnt(0).
8968 - Must happen before
8969 following
8970 buffer_wbinvl1_vol.
8971 - Ensures the
8972 atomicrmw has
8973 completed before
8974 invalidating the
8975 cache.
8977 4. buffer_wbinvl1_vol
8979 - Must happen before
8980 any following
8981 global/generic
8982 load/load
8983 atomic/atomicrmw.
8984 - Ensures that
8985 following loads
8986 will not see stale
8987 global data.
8989 atomicrmw acq_rel - system - generic 1. buffer_wbl2
8991 - Must happen before
8992 following s_waitcnt.
8993 - Performs L2 writeback to
8994 ensure previous
8995 global/generic
8996 store/atomicrmw are
8997 visible at system scope.
8999 2. s_waitcnt lgkmcnt(0) &
9000 vmcnt(0)
9002 - If TgSplit execution mode,
9003 omit lgkmcnt(0).
9004 - If OpenCL, omit
9005 lgkmcnt(0).
9006 - Could be split into
9007 separate s_waitcnt
9008 vmcnt(0) and
9009 s_waitcnt
9010 lgkmcnt(0) to allow
9011 them to be
9012 independently moved
9013 according to the
9014 following rules.
9015 - s_waitcnt vmcnt(0)
9016 must happen after
9017 any preceding
9018 global/generic
9019 load/store/load
9020 atomic/store
9021 atomic/atomicrmw.
9022 - s_waitcnt lgkmcnt(0)
9023 must happen after
9024 any preceding
9025 local/generic
9026 load/store/load
9027 atomic/store
9028 atomic/atomicrmw.
9029 - Must happen before
9030 the following
9031 atomicrmw.
9032 - Ensures that all
9033 memory operations
9034 to global and L2 writeback
9035 have completed before
9036 performing the
9037 atomicrmw that is
9038 being released.
9040 3. flat_atomic
9041 4. s_waitcnt vmcnt(0) &
9042 lgkmcnt(0)
9044 - If TgSplit execution mode,
9045 omit lgkmcnt(0).
9046 - If OpenCL, omit
9047 lgkmcnt(0).
9048 - Must happen before
9049 following buffer_invl2 and
9050 buffer_wbinvl1_vol.
9051 - Ensures the
9052 atomicrmw has
9053 completed before
9054 invalidating the
9055 caches.
9057 5. buffer_invl2;
9058 buffer_wbinvl1_vol
9060 - Must happen before
9061 any following
9062 global/generic
9063 load/load
9064 atomic/atomicrmw.
9065 - Ensures that
9066 following
9067 loads will not see
9068 stale L1 global data,
9069 nor see stale L2 MTYPE
9070 NC global data.
9071 MTYPE RW and CC memory will
9072 never be stale in L2 due to
9073 the memory probes.
9075 fence acq_rel - singlethread *none* *none*
9076 - wavefront
9077 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)
9079 - Use lgkmcnt(0) if not
9080 TgSplit execution mode
9081 and vmcnt(0) if TgSplit
9082 execution mode.
9083 - If OpenCL and
9084 address space is
9085 not generic, omit
9086 lgkmcnt(0).
9087 - If OpenCL and
9088 address space is
9089 local, omit
9090 vmcnt(0).
9091 - However,
9092 since LLVM
9093 currently has no
9094 address space on
9095 the fence need to
9096 conservatively
9097 always generate
9098 (see comment for
9099 previous fence).
9100 - s_waitcnt vmcnt(0)
9101 must happen after
9102 any preceding
9103 global/generic
9104 load/store/
9105 load atomic/store atomic/
9106 atomicrmw.
9107 - s_waitcnt lgkmcnt(0)
9108 must happen after
9109 any preceding
9110 local/generic
9111 load/load
9112 atomic/store/store
9113 atomic/atomicrmw.
9114 - Must happen before
9115 any following
9116 global/generic
9117 load/load
9118 atomic/store/store
9119 atomic/atomicrmw.
9120 - Ensures that all
9121 memory operations
9122 have
9123 completed before
9124 performing any
9125 following global
9126 memory operations.
9127 - Ensures that the
9128 preceding
9129 local/generic load
9130 atomic/atomicrmw
9131 with an equal or
9132 wider sync scope
9133 and memory ordering
9134 stronger than
9135 unordered (this is
9136 termed the
9137 acquire-fence-paired-atomic)
9138 has completed
9139 before following
9140 global memory
9141 operations. This
9142 satisfies the
9143 requirements of
9144 acquire.
9145 - Ensures that all
9146 previous memory
9147 operations have
9148 completed before a
9149 following
9150 local/generic store
9151 atomic/atomicrmw
9152 with an equal or
9153 wider sync scope
9154 and memory ordering
9155 stronger than
9156 unordered (this is
9157 termed the
9158 release-fence-paired-atomic).
9159 This satisfies the
9160 requirements of
9161 release.
9162 - Must happen before
9163 the following
9164 buffer_wbinvl1_vol.
9165 - Ensures that the
9166 acquire-fence-paired
9167 atomic has completed
9168 before invalidating
9169 the
9170 cache. Therefore
9171 any following
9172 locations read must
9173 be no older than
9174 the value read by
9175 the
9176 acquire-fence-paired-atomic.
9178 2. buffer_wbinvl1_vol
9180 - If not TgSplit execution
9181 mode, omit.
9182 - Ensures that
9183 following
9184 loads will not see
9185 stale data.
9187 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) &
9188 vmcnt(0)
9190 - If TgSplit execution mode,
9191 omit lgkmcnt(0).
9192 - If OpenCL and
9193 address space is
9194 not generic, omit
9195 lgkmcnt(0).
9196 - However, since LLVM
9197 currently has no
9198 address space on
9199 the fence need to
9200 conservatively
9201 always generate
9202 (see comment for
9203 previous fence).
9204 - Could be split into
9205 separate s_waitcnt
9206 vmcnt(0) and
9207 s_waitcnt
9208 lgkmcnt(0) to allow
9209 them to be
9210 independently moved
9211 according to the
9212 following rules.
9213 - s_waitcnt vmcnt(0)
9214 must happen after
9215 any preceding
9216 global/generic
9217 load/store/load
9218 atomic/store
9219 atomic/atomicrmw.
9220 - s_waitcnt lgkmcnt(0)
9221 must happen after
9222 any preceding
9223 local/generic
9224 load/store/load
9225 atomic/store
9226 atomic/atomicrmw.
9227 - Must happen before
9228 the following
9229 buffer_wbinvl1_vol.
9230 - Ensures that the
9231 preceding
9232 global/local/generic
9233 load
9234 atomic/atomicrmw
9235 with an equal or
9236 wider sync scope
9237 and memory ordering
9238 stronger than
9239 unordered (this is
9240 termed the
9241 acquire-fence-paired-atomic)
9242 has completed
9243 before invalidating
9244 the cache. This
9245 satisfies the
9246 requirements of
9247 acquire.
9248 - Ensures that all
9249 previous memory
9250 operations have
9251 completed before a
9252 following
9253 global/local/generic
9254 store
9255 atomic/atomicrmw
9256 with an equal or
9257 wider sync scope
9258 and memory ordering
9259 stronger than
9260 unordered (this is
9261 termed the
9262 release-fence-paired-atomic).
9263 This satisfies the
9264 requirements of
9265 release.
9267 2. buffer_wbinvl1_vol
9269 - Must happen before
9270 any following
9271 global/generic
9272 load/load
9273 atomic/store/store
9274 atomic/atomicrmw.
9275 - Ensures that
9276 following loads
9277 will not see stale
9278 global data. This
9279 satisfies the
9280 requirements of
9281 acquire.
9283 fence acq_rel - system *none* 1. buffer_wbl2
9285 - If OpenCL and
9286 address space is
9287 local, omit.
9288 - Must happen before
9289 following s_waitcnt.
9290 - Performs L2 writeback to
9291 ensure previous
9292 global/generic
9293 store/atomicrmw are
9294 visible at system scope.
9296 2. s_waitcnt lgkmcnt(0) &
9297 vmcnt(0)
9299 - If TgSplit execution mode,
9300 omit lgkmcnt(0).
9301 - If OpenCL and
9302 address space is
9303 not generic, omit
9304 lgkmcnt(0).
9305 - However, since LLVM
9306 currently has no
9307 address space on
9308 the fence need to
9309 conservatively
9310 always generate
9311 (see comment for
9312 previous fence).
9313 - Could be split into
9314 separate s_waitcnt
9315 vmcnt(0) and
9316 s_waitcnt
9317 lgkmcnt(0) to allow
9318 them to be
9319 independently moved
9320 according to the
9321 following rules.
9322 - s_waitcnt vmcnt(0)
9323 must happen after
9324 any preceding
9325 global/generic
9326 load/store/load
9327 atomic/store
9328 atomic/atomicrmw.
9329 - s_waitcnt lgkmcnt(0)
9330 must happen after
9331 any preceding
9332 local/generic
9333 load/store/load
9334 atomic/store
9335 atomic/atomicrmw.
9336 - Must happen before
9337 the following buffer_invl2 and
9338 buffer_wbinvl1_vol.
9339 - Ensures that the
9340 preceding
9341 global/local/generic
9342 load
9343 atomic/atomicrmw
9344 with an equal or
9345 wider sync scope
9346 and memory ordering
9347 stronger than
9348 unordered (this is
9349 termed the
9350 acquire-fence-paired-atomic)
9351 has completed
9352 before invalidating
9353 the cache. This
9354 satisfies the
9355 requirements of
9356 acquire.
9357 - Ensures that all
9358 previous memory
9359 operations have
9360 completed before a
9361 following
9362 global/local/generic
9363 store
9364 atomic/atomicrmw
9365 with an equal or
9366 wider sync scope
9367 and memory ordering
9368 stronger than
9369 unordered (this is
9370 termed the
9371 release-fence-paired-atomic).
9372 This satisfies the
9373 requirements of
9374 release.
9376 3. buffer_invl2;
9377 buffer_wbinvl1_vol
9379 - Must happen before
9380 any following
9381 global/generic
9382 load/load
9383 atomic/store/store
9384 atomic/atomicrmw.
9385 - Ensures that
9386 following
9387 loads will not see
9388 stale L1 global data,
9389 nor see stale L2 MTYPE
9390 NC global data.
9391 MTYPE RW and CC memory will
9392 never be stale in L2 due to
9393 the memory probes.
9395 **Sequential Consistent Atomic**
9396 ------------------------------------------------------------------------------------
9397 load atomic seq_cst - singlethread - global *Same as corresponding
9398 - wavefront - local load atomic acquire,
9399 - generic except must generate
9400 all instructions even
9401 for OpenCL.*
9402 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)
9403 - generic
9404 - Use lgkmcnt(0) if not
9405 TgSplit execution mode
9406 and vmcnt(0) if TgSplit
9407 execution mode.
9408 - s_waitcnt lgkmcnt(0) must
9409 happen after
9410 preceding
9411 local/generic load
9412 atomic/store
9413 atomic/atomicrmw
9414 with memory
9415 ordering of seq_cst
9416 and with equal or
9417 wider sync scope.
9418 (Note that seq_cst
9419 fences have their
9420 own s_waitcnt
9421 lgkmcnt(0) and so do
9422 not need to be
9423 considered.)
9424 - s_waitcnt vmcnt(0)
9425 must happen after
9426 preceding
9427 global/generic load
9428 atomic/store
9429 atomic/atomicrmw
9430 with memory
9431 ordering of seq_cst
9432 and with equal or
9433 wider sync scope.
9434 (Note that seq_cst
9435 fences have their
9436 own s_waitcnt
9437 vmcnt(0) and so do
9438 not need to be
9439 considered.)
9440 - Ensures any
9441 preceding
9442 sequential
9443 consistent global/local
9444 memory instructions
9445 have completed
9446 before executing
9447 this sequentially
9448 consistent
9449 instruction. This
9450 prevents reordering
9451 a seq_cst store
9452 followed by a
9453 seq_cst load. (Note
9454 that seq_cst is
9455 stronger than
9456 acquire/release as
9457 the reordering of
9458 load acquire
9459 followed by a store
9460 release is
9461 prevented by the
9462 s_waitcnt of
9463 the release, but
9464 there is nothing
9465 preventing a store
9466 release followed by
9467 load acquire from
9468 completing out of
9469 order. The s_waitcnt
9470 could be placed after
9471 seq_store or before
9472 the seq_load. We
9473 choose the load to
9474 make the s_waitcnt be
9475 as late as possible
9476 so that the store
9477 may have already
9478 completed.)
9480 2. *Following
9481 instructions same as
9482 corresponding load
9483 atomic acquire,
9484 except must generate
9485 all instructions even
9486 for OpenCL.*
9487 load atomic seq_cst - workgroup - local *If TgSplit execution mode,
9488 local address space cannot
9489 be used.*
9491 *Same as corresponding
9492 load atomic acquire,
9493 except must generate
9494 all instructions even
9495 for OpenCL.*
9497 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &
9498 - system - generic vmcnt(0)
9500 - If TgSplit execution mode,
9501 omit lgkmcnt(0).
9502 - Could be split into
9503 separate s_waitcnt
9504 vmcnt(0)
9505 and s_waitcnt
9506 lgkmcnt(0) to allow
9507 them to be
9508 independently moved
9509 according to the
9510 following rules.
9511 - s_waitcnt lgkmcnt(0)
9512 must happen after
9513 preceding
9514 global/generic load
9515 atomic/store
9516 atomic/atomicrmw
9517 with memory
9518 ordering of seq_cst
9519 and with equal or
9520 wider sync scope.
9521 (Note that seq_cst
9522 fences have their
9523 own s_waitcnt
9524 lgkmcnt(0) and so do
9525 not need to be
9526 considered.)
9527 - s_waitcnt vmcnt(0)
9528 must happen after
9529 preceding
9530 global/generic load
9531 atomic/store
9532 atomic/atomicrmw
9533 with memory
9534 ordering of seq_cst
9535 and with equal or
9536 wider sync scope.
9537 (Note that seq_cst
9538 fences have their
9539 own s_waitcnt
9540 vmcnt(0) and so do
9541 not need to be
9542 considered.)
9543 - Ensures any
9544 preceding
9545 sequential
9546 consistent global
9547 memory instructions
9548 have completed
9549 before executing
9550 this sequentially
9551 consistent
9552 instruction. This
9553 prevents reordering
9554 a seq_cst store
9555 followed by a
9556 seq_cst load. (Note
9557 that seq_cst is
9558 stronger than
9559 acquire/release as
9560 the reordering of
9561 load acquire
9562 followed by a store
9563 release is
9564 prevented by the
9565 s_waitcnt of
9566 the release, but
9567 there is nothing
9568 preventing a store
9569 release followed by
9570 load acquire from
9571 completing out of
9572 order. The s_waitcnt
9573 could be placed after
9574 seq_store or before
9575 the seq_load. We
9576 choose the load to
9577 make the s_waitcnt be
9578 as late as possible
9579 so that the store
9580 may have already
9581 completed.)
9583 2. *Following
9584 instructions same as
9585 corresponding load
9586 atomic acquire,
9587 except must generate
9588 all instructions even
9589 for OpenCL.*
9590 store atomic seq_cst - singlethread - global *Same as corresponding
9591 - wavefront - local store atomic release,
9592 - workgroup - generic except must generate
9593 - agent all instructions even
9594 - system for OpenCL.*
9595 atomicrmw seq_cst - singlethread - global *Same as corresponding
9596 - wavefront - local atomicrmw acq_rel,
9597 - workgroup - generic except must generate
9598 - agent all instructions even
9599 - system for OpenCL.*
9600 fence seq_cst - singlethread *none* *Same as corresponding
9601 - wavefront fence acq_rel,
9602 - workgroup except must generate
9603 - agent all instructions even
9604 - system for OpenCL.*
9605 ============ ============ ============== ========== ================================
9607 .. _amdgpu-amdhsa-memory-model-gfx942:
9609 Memory Model GFX942
9610 +++++++++++++++++++
9612 For GFX942:
9614 * Each agent has multiple shader arrays (SA).
9615 * Each SA has multiple compute units (CU).
9616 * Each CU has multiple SIMDs that execute wavefronts.
9617 * The wavefronts for a single work-group are executed in the same CU but may be
9618 executed by different SIMDs. The exception is when in tgsplit execution mode
9619 when the wavefronts may be executed by different SIMDs in different CUs.
9620 * Each CU has a single LDS memory shared by the wavefronts of the work-groups
9621 executing on it. The exception is when in tgsplit execution mode when no LDS
9622 is allocated as wavefronts of the same work-group can be in different CUs.
9623 * All LDS operations of a CU are performed as wavefront wide operations in a
9624 global order and involve no caching. Completion is reported to a wavefront in
9625 execution order.
9626 * The LDS memory has multiple request queues shared by the SIMDs of a
9627 CU. Therefore, the LDS operations performed by different wavefronts of a
9628 work-group can be reordered relative to each other, which can result in
9629 reordering the visibility of vector memory operations with respect to LDS
9630 operations of other wavefronts in the same work-group. A ``s_waitcnt
9631 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and
9632 vector memory operations between wavefronts of a work-group, but not between
9633 operations performed by the same wavefront.
9634 * The vector memory operations are performed as wavefront wide operations and
9635 completion is reported to a wavefront in execution order. The exception is
9636 that ``flat_load/store/atomic`` instructions can report out of vector memory
9637 order if they access LDS memory, and out of LDS operation order if they access
9638 global memory.
9639 * The vector memory operations access a single vector L1 cache shared by all
9640 SIMDs a CU. Therefore:
9642 * No special action is required for coherence between the lanes of a single
9643 wavefront.
9645 * No special action is required for coherence between wavefronts in the same
9646 work-group since they execute on the same CU. The exception is when in
9647 tgsplit execution mode as wavefronts of the same work-group can be in
9648 different CUs and so a ``buffer_inv sc0`` is required which will invalidate
9649 the L1 cache.
9651 * A ``buffer_inv sc0`` is required to invalidate the L1 cache for coherence
9652 between wavefronts executing in different work-groups as they may be
9653 executing on different CUs.
9655 * Atomic read-modify-write instructions implicitly bypass the L1 cache.
9656 Therefore, they do not use the sc0 bit for coherence and instead use it to
9657 indicate if the instruction returns the original value being updated. They
9658 do use sc1 to indicate system or agent scope coherence.
9660 * The scalar memory operations access a scalar L1 cache shared by all wavefronts
9661 on a group of CUs. The scalar and vector L1 caches are not coherent. However,
9662 scalar operations are used in a restricted way so do not impact the memory
9663 model. See :ref:`amdgpu-amdhsa-memory-spaces`.
9664 * The vector and scalar memory operations use an L2 cache.
9666 * The gfx942 can be configured as a number of smaller agents with each having
9667 a single L2 shared by all CUs on the same agent, or as fewer (possibly one)
9668 larger agents with groups of CUs on each agent each sharing separate L2
9669 caches.
9670 * The L2 cache has independent channels to service disjoint ranges of virtual
9671 addresses.
9672 * Each CU has a separate request queue per channel for its associated L2.
9673 Therefore, the vector and scalar memory operations performed by wavefronts
9674 executing with different L1 caches and the same L2 cache can be reordered
9675 relative to each other.
9676 * A ``s_waitcnt vmcnt(0)`` is required to ensure synchronization between
9677 vector memory operations of different CUs. It ensures a previous vector
9678 memory operation has completed before executing a subsequent vector memory
9679 or LDS operation and so can be used to meet the requirements of acquire and
9680 release.
9681 * An L2 cache can be kept coherent with other L2 caches by using the MTYPE RW
9682 (read-write) for memory local to the L2, and MTYPE NC (non-coherent) with
9683 the PTE C-bit set for memory not local to the L2.
9685 * Any local memory cache lines will be automatically invalidated by writes
9686 from CUs associated with other L2 caches, or writes from the CPU, due to
9687 the cache probe caused by the PTE C-bit.
9688 * XGMI accesses from the CPU to local memory may be cached on the CPU.
9689 Subsequent access from the GPU will automatically invalidate or writeback
9690 the CPU cache due to the L2 probe filter.
9691 * To ensure coherence of local memory writes of CUs with different L1 caches
9692 in the same agent a ``buffer_wbl2`` is required. It does nothing if the
9693 agent is configured to have a single L2, or will writeback dirty L2 cache
9694 lines if configured to have multiple L2 caches.
9695 * To ensure coherence of local memory writes of CUs in different agents a
9696 ``buffer_wbl2 sc1`` is required. It will writeback dirty L2 cache lines.
9697 * To ensure coherence of local memory reads of CUs with different L1 caches
9698 in the same agent a ``buffer_inv sc1`` is required. It does nothing if the
9699 agent is configured to have a single L2, or will invalidate non-local L2
9700 cache lines if configured to have multiple L2 caches.
9701 * To ensure coherence of local memory reads of CUs in different agents a
9702 ``buffer_inv sc0 sc1`` is required. It will invalidate non-local L2 cache
9703 lines if configured to have multiple L2 caches.
9705 * PCIe access from the GPU to the CPU can be kept coherent by using the MTYPE
9706 UC (uncached) which bypasses the L2.
9708 Scalar memory operations are only used to access memory that is proven to not
9709 change during the execution of the kernel dispatch. This includes constant
9710 address space and global address space for program scope ``const`` variables.
9711 Therefore, the kernel machine code does not have to maintain the scalar cache to
9712 ensure it is coherent with the vector caches. The scalar and vector caches are
9713 invalidated between kernel dispatches by CP since constant address space data
9714 may change between kernel dispatch executions. See
9715 :ref:`amdgpu-amdhsa-memory-spaces`.
9717 The one exception is if scalar writes are used to spill SGPR registers. In this
9718 case the AMDGPU backend ensures the memory location used to spill is never
9719 accessed by vector memory operations at the same time. If scalar writes are used
9720 then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function
9721 return since the locations may be used for vector memory instructions by a
9722 future wavefront that uses the same scratch area, or a function call that
9723 creates a frame at the same address, respectively. There is no need for a
9724 ``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.
9726 For kernarg backing memory:
9728 * CP invalidates the L1 cache at the start of each kernel dispatch.
9729 * On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host
9730 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2
9731 cache. This also causes it to be treated as non-volatile and so is not
9732 invalidated by ``*_vol``.
9733 * On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and
9734 so the L2 cache will be coherent with the CPU and other agents.
9736 Scratch backing memory (which is used for the private address space) is accessed
9737 with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is
9738 only accessed by a single thread, and is always write-before-read, there is
9739 never a need to invalidate these entries from the L1 cache. Hence all cache
9740 invalidates are done as ``*_vol`` to only invalidate the volatile cache lines.
9742 The code sequences used to implement the memory model for GFX940, GFX941, GFX942
9743 are defined in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx940-gfx941-gfx942-table`.
9745 .. table:: AMDHSA Memory Model Code Sequences GFX940, GFX941, GFX942
9746 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx940-gfx941-gfx942-table
9748 ============ ============ ============== ========== ================================
9749 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code
9750 Ordering Sync Scope Address GFX940, GFX941, GFX942
9751 Space
9752 ============ ============ ============== ========== ================================
9753 **Non-Atomic**
9754 ------------------------------------------------------------------------------------
9755 load *none* *none* - global - !volatile & !nontemporal
9756 - generic
9757 - private 1. buffer/global/flat_load
9758 - constant
9759 - !volatile & nontemporal
9761 1. buffer/global/flat_load
9762 nt=1
9764 - volatile
9766 1. buffer/global/flat_load
9767 sc0=1 sc1=1
9768 2. s_waitcnt vmcnt(0)
9770 - Must happen before
9771 any following volatile
9772 global/generic
9773 load/store.
9774 - Ensures that
9775 volatile
9776 operations to
9777 different
9778 addresses will not
9779 be reordered by
9780 hardware.
9782 load *none* *none* - local 1. ds_load
9783 store *none* *none* - global - !volatile & !nontemporal
9784 - generic
9785 - private 1. GFX940, GFX941
9786 - constant buffer/global/flat_store
9787 sc0=1 sc1=1
9788 GFX942
9789 buffer/global/flat_store
9791 - !volatile & nontemporal
9793 1. GFX940, GFX941
9794 buffer/global/flat_store
9795 nt=1 sc0=1 sc1=1
9796 GFX942
9797 buffer/global/flat_store
9798 nt=1
9800 - volatile
9802 1. buffer/global/flat_store
9803 sc0=1 sc1=1
9804 2. s_waitcnt vmcnt(0)
9806 - Must happen before
9807 any following volatile
9808 global/generic
9809 load/store.
9810 - Ensures that
9811 volatile
9812 operations to
9813 different
9814 addresses will not
9815 be reordered by
9816 hardware.
9818 store *none* *none* - local 1. ds_store
9819 **Unordered Atomic**
9820 ------------------------------------------------------------------------------------
9821 load atomic unordered *any* *any* *Same as non-atomic*.
9822 store atomic unordered *any* *any* *Same as non-atomic*.
9823 atomicrmw unordered *any* *any* *Same as monotonic atomic*.
9824 **Monotonic Atomic**
9825 ------------------------------------------------------------------------------------
9826 load atomic monotonic - singlethread - global 1. buffer/global/flat_load
9827 - wavefront - generic
9828 load atomic monotonic - workgroup - global 1. buffer/global/flat_load
9829 - generic sc0=1
9830 load atomic monotonic - singlethread - local *If TgSplit execution mode,
9831 - wavefront local address space cannot
9832 - workgroup be used.*
9834 1. ds_load
9835 load atomic monotonic - agent - global 1. buffer/global/flat_load
9836 - generic sc1=1
9837 load atomic monotonic - system - global 1. buffer/global/flat_load
9838 - generic sc0=1 sc1=1
9839 store atomic monotonic - singlethread - global 1. buffer/global/flat_store
9840 - wavefront - generic
9841 store atomic monotonic - workgroup - global 1. buffer/global/flat_store
9842 - generic sc0=1
9843 store atomic monotonic - agent - global 1. buffer/global/flat_store
9844 - generic sc1=1
9845 store atomic monotonic - system - global 1. buffer/global/flat_store
9846 - generic sc0=1 sc1=1
9847 store atomic monotonic - singlethread - local *If TgSplit execution mode,
9848 - wavefront local address space cannot
9849 - workgroup be used.*
9851 1. ds_store
9852 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic
9853 - wavefront - generic
9854 - workgroup
9855 - agent
9856 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic
9857 - generic sc1=1
9858 atomicrmw monotonic - singlethread - local *If TgSplit execution mode,
9859 - wavefront local address space cannot
9860 - workgroup be used.*
9862 1. ds_atomic
9863 **Acquire Atomic**
9864 ------------------------------------------------------------------------------------
9865 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load
9866 - wavefront - local
9867 - generic
9868 load atomic acquire - workgroup - global 1. buffer/global_load sc0=1
9869 2. s_waitcnt vmcnt(0)
9871 - If not TgSplit execution
9872 mode, omit.
9873 - Must happen before the
9874 following buffer_inv.
9876 3. buffer_inv sc0=1
9878 - If not TgSplit execution
9879 mode, omit.
9880 - Must happen before
9881 any following
9882 global/generic
9883 load/load
9884 atomic/store/store
9885 atomic/atomicrmw.
9886 - Ensures that
9887 following
9888 loads will not see
9889 stale data.
9891 load atomic acquire - workgroup - local *If TgSplit execution mode,
9892 local address space cannot
9893 be used.*
9895 1. ds_load
9896 2. s_waitcnt lgkmcnt(0)
9898 - If OpenCL, omit.
9899 - Must happen before
9900 any following
9901 global/generic
9902 load/load
9903 atomic/store/store
9904 atomic/atomicrmw.
9905 - Ensures any
9906 following global
9907 data read is no
9908 older than the local load
9909 atomic value being
9910 acquired.
9912 load atomic acquire - workgroup - generic 1. flat_load sc0=1
9913 2. s_waitcnt lgkm/vmcnt(0)
9915 - Use lgkmcnt(0) if not
9916 TgSplit execution mode
9917 and vmcnt(0) if TgSplit
9918 execution mode.
9919 - If OpenCL, omit lgkmcnt(0).
9920 - Must happen before
9921 the following
9922 buffer_inv and any
9923 following global/generic
9924 load/load
9925 atomic/store/store
9926 atomic/atomicrmw.
9927 - Ensures any
9928 following global
9929 data read is no
9930 older than a local load
9931 atomic value being
9932 acquired.
9934 3. buffer_inv sc0=1
9936 - If not TgSplit execution
9937 mode, omit.
9938 - Ensures that
9939 following
9940 loads will not see
9941 stale data.
9943 load atomic acquire - agent - global 1. buffer/global_load
9944 sc1=1
9945 2. s_waitcnt vmcnt(0)
9947 - Must happen before
9948 following
9949 buffer_inv.
9950 - Ensures the load
9951 has completed
9952 before invalidating
9953 the cache.
9955 3. buffer_inv sc1=1
9957 - Must happen before
9958 any following
9959 global/generic
9960 load/load
9961 atomic/atomicrmw.
9962 - Ensures that
9963 following
9964 loads will not see
9965 stale global data.
9967 load atomic acquire - system - global 1. buffer/global/flat_load
9968 sc0=1 sc1=1
9969 2. s_waitcnt vmcnt(0)
9971 - Must happen before
9972 following
9973 buffer_inv.
9974 - Ensures the load
9975 has completed
9976 before invalidating
9977 the cache.
9979 3. buffer_inv sc0=1 sc1=1
9981 - Must happen before
9982 any following
9983 global/generic
9984 load/load
9985 atomic/atomicrmw.
9986 - Ensures that
9987 following
9988 loads will not see
9989 stale MTYPE NC global data.
9990 MTYPE RW and CC memory will
9991 never be stale due to the
9992 memory probes.
9994 load atomic acquire - agent - generic 1. flat_load sc1=1
9995 2. s_waitcnt vmcnt(0) &
9996 lgkmcnt(0)
9998 - If TgSplit execution mode,
9999 omit lgkmcnt(0).
10000 - If OpenCL omit
10001 lgkmcnt(0).
10002 - Must happen before
10003 following
10004 buffer_inv.
10005 - Ensures the flat_load
10006 has completed
10007 before invalidating
10008 the cache.
10010 3. buffer_inv sc1=1
10012 - Must happen before
10013 any following
10014 global/generic
10015 load/load
10016 atomic/atomicrmw.
10017 - Ensures that
10018 following loads
10019 will not see stale
10020 global data.
10022 load atomic acquire - system - generic 1. flat_load sc0=1 sc1=1
10023 2. s_waitcnt vmcnt(0) &
10024 lgkmcnt(0)
10026 - If TgSplit execution mode,
10027 omit lgkmcnt(0).
10028 - If OpenCL omit
10029 lgkmcnt(0).
10030 - Must happen before
10031 the following
10032 buffer_inv.
10033 - Ensures the flat_load
10034 has completed
10035 before invalidating
10036 the caches.
10038 3. buffer_inv sc0=1 sc1=1
10040 - Must happen before
10041 any following
10042 global/generic
10043 load/load
10044 atomic/atomicrmw.
10045 - Ensures that
10046 following
10047 loads will not see
10048 stale MTYPE NC global data.
10049 MTYPE RW and CC memory will
10050 never be stale due to the
10051 memory probes.
10053 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic
10054 - wavefront - generic
10055 atomicrmw acquire - singlethread - local *If TgSplit execution mode,
10056 - wavefront local address space cannot
10057 be used.*
10059 1. ds_atomic
10060 atomicrmw acquire - workgroup - global 1. buffer/global_atomic
10061 2. s_waitcnt vmcnt(0)
10063 - If not TgSplit execution
10064 mode, omit.
10065 - Must happen before the
10066 following buffer_inv.
10067 - Ensures the atomicrmw
10068 has completed
10069 before invalidating
10070 the cache.
10072 3. buffer_inv sc0=1
10074 - If not TgSplit execution
10075 mode, omit.
10076 - Must happen before
10077 any following
10078 global/generic
10079 load/load
10080 atomic/atomicrmw.
10081 - Ensures that
10082 following loads
10083 will not see stale
10084 global data.
10086 atomicrmw acquire - workgroup - local *If TgSplit execution mode,
10087 local address space cannot
10088 be used.*
10090 1. ds_atomic
10091 2. s_waitcnt lgkmcnt(0)
10093 - If OpenCL, omit.
10094 - Must happen before
10095 any following
10096 global/generic
10097 load/load
10098 atomic/store/store
10099 atomic/atomicrmw.
10100 - Ensures any
10101 following global
10102 data read is no
10103 older than the local
10104 atomicrmw value
10105 being acquired.
10107 atomicrmw acquire - workgroup - generic 1. flat_atomic
10108 2. s_waitcnt lgkm/vmcnt(0)
10110 - Use lgkmcnt(0) if not
10111 TgSplit execution mode
10112 and vmcnt(0) if TgSplit
10113 execution mode.
10114 - If OpenCL, omit lgkmcnt(0).
10115 - Must happen before
10116 the following
10117 buffer_inv and
10118 any following
10119 global/generic
10120 load/load
10121 atomic/store/store
10122 atomic/atomicrmw.
10123 - Ensures any
10124 following global
10125 data read is no
10126 older than a local
10127 atomicrmw value
10128 being acquired.
10130 3. buffer_inv sc0=1
10132 - If not TgSplit execution
10133 mode, omit.
10134 - Ensures that
10135 following
10136 loads will not see
10137 stale data.
10139 atomicrmw acquire - agent - global 1. buffer/global_atomic
10140 2. s_waitcnt vmcnt(0)
10142 - Must happen before
10143 following
10144 buffer_inv.
10145 - Ensures the
10146 atomicrmw has
10147 completed before
10148 invalidating the
10149 cache.
10151 3. buffer_inv sc1=1
10153 - Must happen before
10154 any following
10155 global/generic
10156 load/load
10157 atomic/atomicrmw.
10158 - Ensures that
10159 following loads
10160 will not see stale
10161 global data.
10163 atomicrmw acquire - system - global 1. buffer/global_atomic
10164 sc1=1
10165 2. s_waitcnt vmcnt(0)
10167 - Must happen before
10168 following
10169 buffer_inv.
10170 - Ensures the
10171 atomicrmw has
10172 completed before
10173 invalidating the
10174 caches.
10176 3. buffer_inv sc0=1 sc1=1
10178 - Must happen before
10179 any following
10180 global/generic
10181 load/load
10182 atomic/atomicrmw.
10183 - Ensures that
10184 following
10185 loads will not see
10186 stale MTYPE NC global data.
10187 MTYPE RW and CC memory will
10188 never be stale due to the
10189 memory probes.
10191 atomicrmw acquire - agent - generic 1. flat_atomic
10192 2. s_waitcnt vmcnt(0) &
10193 lgkmcnt(0)
10195 - If TgSplit execution mode,
10196 omit lgkmcnt(0).
10197 - If OpenCL, omit
10198 lgkmcnt(0).
10199 - Must happen before
10200 following
10201 buffer_inv.
10202 - Ensures the
10203 atomicrmw has
10204 completed before
10205 invalidating the
10206 cache.
10208 3. buffer_inv sc1=1
10210 - Must happen before
10211 any following
10212 global/generic
10213 load/load
10214 atomic/atomicrmw.
10215 - Ensures that
10216 following loads
10217 will not see stale
10218 global data.
10220 atomicrmw acquire - system - generic 1. flat_atomic sc1=1
10221 2. s_waitcnt vmcnt(0) &
10222 lgkmcnt(0)
10224 - If TgSplit execution mode,
10225 omit lgkmcnt(0).
10226 - If OpenCL, omit
10227 lgkmcnt(0).
10228 - Must happen before
10229 following
10230 buffer_inv.
10231 - Ensures the
10232 atomicrmw has
10233 completed before
10234 invalidating the
10235 caches.
10237 3. buffer_inv sc0=1 sc1=1
10239 - Must happen before
10240 any following
10241 global/generic
10242 load/load
10243 atomic/atomicrmw.
10244 - Ensures that
10245 following
10246 loads will not see
10247 stale MTYPE NC global data.
10248 MTYPE RW and CC memory will
10249 never be stale due to the
10250 memory probes.
10252 fence acquire - singlethread *none* *none*
10253 - wavefront
10254 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)
10256 - Use lgkmcnt(0) if not
10257 TgSplit execution mode
10258 and vmcnt(0) if TgSplit
10259 execution mode.
10260 - If OpenCL and
10261 address space is
10262 not generic, omit
10263 lgkmcnt(0).
10264 - If OpenCL and
10265 address space is
10266 local, omit
10267 vmcnt(0).
10268 - However, since LLVM
10269 currently has no
10270 address space on
10271 the fence need to
10272 conservatively
10273 always generate. If
10274 fence had an
10275 address space then
10276 set to address
10277 space of OpenCL
10278 fence flag, or to
10279 generic if both
10280 local and global
10281 flags are
10282 specified.
10283 - s_waitcnt vmcnt(0)
10284 must happen after
10285 any preceding
10286 global/generic load
10287 atomic/
10288 atomicrmw
10289 with an equal or
10290 wider sync scope
10291 and memory ordering
10292 stronger than
10293 unordered (this is
10294 termed the
10295 fence-paired-atomic).
10296 - s_waitcnt lgkmcnt(0)
10297 must happen after
10298 any preceding
10299 local/generic load
10300 atomic/atomicrmw
10301 with an equal or
10302 wider sync scope
10303 and memory ordering
10304 stronger than
10305 unordered (this is
10306 termed the
10307 fence-paired-atomic).
10308 - Must happen before
10309 the following
10310 buffer_inv and
10311 any following
10312 global/generic
10313 load/load
10314 atomic/store/store
10315 atomic/atomicrmw.
10316 - Ensures any
10317 following global
10318 data read is no
10319 older than the
10320 value read by the
10321 fence-paired-atomic.
10323 3. buffer_inv sc0=1
10325 - If not TgSplit execution
10326 mode, omit.
10327 - Ensures that
10328 following
10329 loads will not see
10330 stale data.
10332 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) &
10333 vmcnt(0)
10335 - If TgSplit execution mode,
10336 omit lgkmcnt(0).
10337 - If OpenCL and
10338 address space is
10339 not generic, omit
10340 lgkmcnt(0).
10341 - However, since LLVM
10342 currently has no
10343 address space on
10344 the fence need to
10345 conservatively
10346 always generate
10347 (see comment for
10348 previous fence).
10349 - Could be split into
10350 separate s_waitcnt
10351 vmcnt(0) and
10352 s_waitcnt
10353 lgkmcnt(0) to allow
10354 them to be
10355 independently moved
10356 according to the
10357 following rules.
10358 - s_waitcnt vmcnt(0)
10359 must happen after
10360 any preceding
10361 global/generic load
10362 atomic/atomicrmw
10363 with an equal or
10364 wider sync scope
10365 and memory ordering
10366 stronger than
10367 unordered (this is
10368 termed the
10369 fence-paired-atomic).
10370 - s_waitcnt lgkmcnt(0)
10371 must happen after
10372 any preceding
10373 local/generic load
10374 atomic/atomicrmw
10375 with an equal or
10376 wider sync scope
10377 and memory ordering
10378 stronger than
10379 unordered (this is
10380 termed the
10381 fence-paired-atomic).
10382 - Must happen before
10383 the following
10384 buffer_inv.
10385 - Ensures that the
10386 fence-paired atomic
10387 has completed
10388 before invalidating
10389 the
10390 cache. Therefore
10391 any following
10392 locations read must
10393 be no older than
10394 the value read by
10395 the
10396 fence-paired-atomic.
10398 2. buffer_inv sc1=1
10400 - Must happen before any
10401 following global/generic
10402 load/load
10403 atomic/store/store
10404 atomic/atomicrmw.
10405 - Ensures that
10406 following loads
10407 will not see stale
10408 global data.
10410 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) &
10411 vmcnt(0)
10413 - If TgSplit execution mode,
10414 omit lgkmcnt(0).
10415 - If OpenCL and
10416 address space is
10417 not generic, omit
10418 lgkmcnt(0).
10419 - However, since LLVM
10420 currently has no
10421 address space on
10422 the fence need to
10423 conservatively
10424 always generate
10425 (see comment for
10426 previous fence).
10427 - Could be split into
10428 separate s_waitcnt
10429 vmcnt(0) and
10430 s_waitcnt
10431 lgkmcnt(0) to allow
10432 them to be
10433 independently moved
10434 according to the
10435 following rules.
10436 - s_waitcnt vmcnt(0)
10437 must happen after
10438 any preceding
10439 global/generic load
10440 atomic/atomicrmw
10441 with an equal or
10442 wider sync scope
10443 and memory ordering
10444 stronger than
10445 unordered (this is
10446 termed the
10447 fence-paired-atomic).
10448 - s_waitcnt lgkmcnt(0)
10449 must happen after
10450 any preceding
10451 local/generic load
10452 atomic/atomicrmw
10453 with an equal or
10454 wider sync scope
10455 and memory ordering
10456 stronger than
10457 unordered (this is
10458 termed the
10459 fence-paired-atomic).
10460 - Must happen before
10461 the following
10462 buffer_inv.
10463 - Ensures that the
10464 fence-paired atomic
10465 has completed
10466 before invalidating
10467 the
10468 cache. Therefore
10469 any following
10470 locations read must
10471 be no older than
10472 the value read by
10473 the
10474 fence-paired-atomic.
10476 2. buffer_inv sc0=1 sc1=1
10478 - Must happen before any
10479 following global/generic
10480 load/load
10481 atomic/store/store
10482 atomic/atomicrmw.
10483 - Ensures that
10484 following loads
10485 will not see stale
10486 global data.
10488 **Release Atomic**
10489 ------------------------------------------------------------------------------------
10490 store atomic release - singlethread - global 1. GFX940, GFX941
10491 - wavefront - generic buffer/global/flat_store
10492 sc0=1 sc1=1
10493 GFX942
10494 buffer/global/flat_store
10496 store atomic release - singlethread - local *If TgSplit execution mode,
10497 - wavefront local address space cannot
10498 be used.*
10500 1. ds_store
10501 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)
10502 - generic
10503 - Use lgkmcnt(0) if not
10504 TgSplit execution mode
10505 and vmcnt(0) if TgSplit
10506 execution mode.
10507 - If OpenCL, omit lgkmcnt(0).
10508 - s_waitcnt vmcnt(0)
10509 must happen after
10510 any preceding
10511 global/generic load/store/
10512 load atomic/store atomic/
10513 atomicrmw.
10514 - s_waitcnt lgkmcnt(0)
10515 must happen after
10516 any preceding
10517 local/generic
10518 load/store/load
10519 atomic/store
10520 atomic/atomicrmw.
10521 - Must happen before
10522 the following
10523 store.
10524 - Ensures that all
10525 memory operations
10526 have
10527 completed before
10528 performing the
10529 store that is being
10530 released.
10532 2. GFX940, GFX941
10533 buffer/global/flat_store
10534 sc0=1 sc1=1
10535 GFX942
10536 buffer/global/flat_store
10537 sc0=1
10538 store atomic release - workgroup - local *If TgSplit execution mode,
10539 local address space cannot
10540 be used.*
10542 1. ds_store
10543 store atomic release - agent - global 1. buffer_wbl2 sc1=1
10544 - generic
10545 - Must happen before
10546 following s_waitcnt.
10547 - Performs L2 writeback to
10548 ensure previous
10549 global/generic
10550 store/atomicrmw are
10551 visible at agent scope.
10553 2. s_waitcnt lgkmcnt(0) &
10554 vmcnt(0)
10556 - If TgSplit execution mode,
10557 omit lgkmcnt(0).
10558 - If OpenCL and
10559 address space is
10560 not generic, omit
10561 lgkmcnt(0).
10562 - Could be split into
10563 separate s_waitcnt
10564 vmcnt(0) and
10565 s_waitcnt
10566 lgkmcnt(0) to allow
10567 them to be
10568 independently moved
10569 according to the
10570 following rules.
10571 - s_waitcnt vmcnt(0)
10572 must happen after
10573 any preceding
10574 global/generic
10575 load/store/load
10576 atomic/store
10577 atomic/atomicrmw.
10578 - s_waitcnt lgkmcnt(0)
10579 must happen after
10580 any preceding
10581 local/generic
10582 load/store/load
10583 atomic/store
10584 atomic/atomicrmw.
10585 - Must happen before
10586 the following
10587 store.
10588 - Ensures that all
10589 memory operations
10590 to memory have
10591 completed before
10592 performing the
10593 store that is being
10594 released.
10596 3. GFX940, GFX941
10597 buffer/global/flat_store
10598 sc0=1 sc1=1
10599 GFX942
10600 buffer/global/flat_store
10601 sc1=1
10602 store atomic release - system - global 1. buffer_wbl2 sc0=1 sc1=1
10603 - generic
10604 - Must happen before
10605 following s_waitcnt.
10606 - Performs L2 writeback to
10607 ensure previous
10608 global/generic
10609 store/atomicrmw are
10610 visible at system scope.
10612 2. s_waitcnt lgkmcnt(0) &
10613 vmcnt(0)
10615 - If TgSplit execution mode,
10616 omit lgkmcnt(0).
10617 - If OpenCL and
10618 address space is
10619 not generic, omit
10620 lgkmcnt(0).
10621 - Could be split into
10622 separate s_waitcnt
10623 vmcnt(0) and
10624 s_waitcnt
10625 lgkmcnt(0) to allow
10626 them to be
10627 independently moved
10628 according to the
10629 following rules.
10630 - s_waitcnt vmcnt(0)
10631 must happen after any
10632 preceding
10633 global/generic
10634 load/store/load
10635 atomic/store
10636 atomic/atomicrmw.
10637 - s_waitcnt lgkmcnt(0)
10638 must happen after any
10639 preceding
10640 local/generic
10641 load/store/load
10642 atomic/store
10643 atomic/atomicrmw.
10644 - Must happen before
10645 the following
10646 store.
10647 - Ensures that all
10648 memory operations
10649 to memory and the L2
10650 writeback have
10651 completed before
10652 performing the
10653 store that is being
10654 released.
10656 3. buffer/global/flat_store
10657 sc0=1 sc1=1
10658 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic
10659 - wavefront - generic
10660 atomicrmw release - singlethread - local *If TgSplit execution mode,
10661 - wavefront local address space cannot
10662 be used.*
10664 1. ds_atomic
10665 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)
10666 - generic
10667 - Use lgkmcnt(0) if not
10668 TgSplit execution mode
10669 and vmcnt(0) if TgSplit
10670 execution mode.
10671 - If OpenCL, omit
10672 lgkmcnt(0).
10673 - s_waitcnt vmcnt(0)
10674 must happen after
10675 any preceding
10676 global/generic load/store/
10677 load atomic/store atomic/
10678 atomicrmw.
10679 - s_waitcnt lgkmcnt(0)
10680 must happen after
10681 any preceding
10682 local/generic
10683 load/store/load
10684 atomic/store
10685 atomic/atomicrmw.
10686 - Must happen before
10687 the following
10688 atomicrmw.
10689 - Ensures that all
10690 memory operations
10691 have
10692 completed before
10693 performing the
10694 atomicrmw that is
10695 being released.
10697 2. buffer/global/flat_atomic sc0=1
10698 atomicrmw release - workgroup - local *If TgSplit execution mode,
10699 local address space cannot
10700 be used.*
10702 1. ds_atomic
10703 atomicrmw release - agent - global 1. buffer_wbl2 sc1=1
10704 - generic
10705 - Must happen before
10706 following s_waitcnt.
10707 - Performs L2 writeback to
10708 ensure previous
10709 global/generic
10710 store/atomicrmw are
10711 visible at agent scope.
10713 2. s_waitcnt lgkmcnt(0) &
10714 vmcnt(0)
10716 - If TgSplit execution mode,
10717 omit lgkmcnt(0).
10718 - If OpenCL, omit
10719 lgkmcnt(0).
10720 - Could be split into
10721 separate s_waitcnt
10722 vmcnt(0) and
10723 s_waitcnt
10724 lgkmcnt(0) to allow
10725 them to be
10726 independently moved
10727 according to the
10728 following rules.
10729 - s_waitcnt vmcnt(0)
10730 must happen after
10731 any preceding
10732 global/generic
10733 load/store/load
10734 atomic/store
10735 atomic/atomicrmw.
10736 - s_waitcnt lgkmcnt(0)
10737 must happen after
10738 any preceding
10739 local/generic
10740 load/store/load
10741 atomic/store
10742 atomic/atomicrmw.
10743 - Must happen before
10744 the following
10745 atomicrmw.
10746 - Ensures that all
10747 memory operations
10748 to global and local
10749 have completed
10750 before performing
10751 the atomicrmw that
10752 is being released.
10754 3. buffer/global/flat_atomic sc1=1
10755 atomicrmw release - system - global 1. buffer_wbl2 sc0=1 sc1=1
10756 - generic
10757 - Must happen before
10758 following s_waitcnt.
10759 - Performs L2 writeback to
10760 ensure previous
10761 global/generic
10762 store/atomicrmw are
10763 visible at system scope.
10765 2. s_waitcnt lgkmcnt(0) &
10766 vmcnt(0)
10768 - If TgSplit execution mode,
10769 omit lgkmcnt(0).
10770 - If OpenCL, omit
10771 lgkmcnt(0).
10772 - Could be split into
10773 separate s_waitcnt
10774 vmcnt(0) and
10775 s_waitcnt
10776 lgkmcnt(0) to allow
10777 them to be
10778 independently moved
10779 according to the
10780 following rules.
10781 - s_waitcnt vmcnt(0)
10782 must happen after
10783 any preceding
10784 global/generic
10785 load/store/load
10786 atomic/store
10787 atomic/atomicrmw.
10788 - s_waitcnt lgkmcnt(0)
10789 must happen after
10790 any preceding
10791 local/generic
10792 load/store/load
10793 atomic/store
10794 atomic/atomicrmw.
10795 - Must happen before
10796 the following
10797 atomicrmw.
10798 - Ensures that all
10799 memory operations
10800 to memory and the L2
10801 writeback have
10802 completed before
10803 performing the
10804 store that is being
10805 released.
10807 3. buffer/global/flat_atomic
10808 sc0=1 sc1=1
10809 fence release - singlethread *none* *none*
10810 - wavefront
10811 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)
10813 - Use lgkmcnt(0) if not
10814 TgSplit execution mode
10815 and vmcnt(0) if TgSplit
10816 execution mode.
10817 - If OpenCL and
10818 address space is
10819 not generic, omit
10820 lgkmcnt(0).
10821 - If OpenCL and
10822 address space is
10823 local, omit
10824 vmcnt(0).
10825 - However, since LLVM
10826 currently has no
10827 address space on
10828 the fence need to
10829 conservatively
10830 always generate. If
10831 fence had an
10832 address space then
10833 set to address
10834 space of OpenCL
10835 fence flag, or to
10836 generic if both
10837 local and global
10838 flags are
10839 specified.
10840 - s_waitcnt vmcnt(0)
10841 must happen after
10842 any preceding
10843 global/generic
10844 load/store/
10845 load atomic/store atomic/
10846 atomicrmw.
10847 - s_waitcnt lgkmcnt(0)
10848 must happen after
10849 any preceding
10850 local/generic
10851 load/load
10852 atomic/store/store
10853 atomic/atomicrmw.
10854 - Must happen before
10855 any following store
10856 atomic/atomicrmw
10857 with an equal or
10858 wider sync scope
10859 and memory ordering
10860 stronger than
10861 unordered (this is
10862 termed the
10863 fence-paired-atomic).
10864 - Ensures that all
10865 memory operations
10866 have
10867 completed before
10868 performing the
10869 following
10870 fence-paired-atomic.
10872 fence release - agent *none* 1. buffer_wbl2 sc1=1
10874 - If OpenCL and
10875 address space is
10876 local, omit.
10877 - Must happen before
10878 following s_waitcnt.
10879 - Performs L2 writeback to
10880 ensure previous
10881 global/generic
10882 store/atomicrmw are
10883 visible at agent scope.
10885 2. s_waitcnt lgkmcnt(0) &
10886 vmcnt(0)
10888 - If TgSplit execution mode,
10889 omit lgkmcnt(0).
10890 - If OpenCL and
10891 address space is
10892 not generic, omit
10893 lgkmcnt(0).
10894 - If OpenCL and
10895 address space is
10896 local, omit
10897 vmcnt(0).
10898 - However, since LLVM
10899 currently has no
10900 address space on
10901 the fence need to
10902 conservatively
10903 always generate. If
10904 fence had an
10905 address space then
10906 set to address
10907 space of OpenCL
10908 fence flag, or to
10909 generic if both
10910 local and global
10911 flags are
10912 specified.
10913 - Could be split into
10914 separate s_waitcnt
10915 vmcnt(0) and
10916 s_waitcnt
10917 lgkmcnt(0) to allow
10918 them to be
10919 independently moved
10920 according to the
10921 following rules.
10922 - s_waitcnt vmcnt(0)
10923 must happen after
10924 any preceding
10925 global/generic
10926 load/store/load
10927 atomic/store
10928 atomic/atomicrmw.
10929 - s_waitcnt lgkmcnt(0)
10930 must happen after
10931 any preceding
10932 local/generic
10933 load/store/load
10934 atomic/store
10935 atomic/atomicrmw.
10936 - Must happen before
10937 any following store
10938 atomic/atomicrmw
10939 with an equal or
10940 wider sync scope
10941 and memory ordering
10942 stronger than
10943 unordered (this is
10944 termed the
10945 fence-paired-atomic).
10946 - Ensures that all
10947 memory operations
10948 have
10949 completed before
10950 performing the
10951 following
10952 fence-paired-atomic.
10954 fence release - system *none* 1. buffer_wbl2 sc0=1 sc1=1
10956 - Must happen before
10957 following s_waitcnt.
10958 - Performs L2 writeback to
10959 ensure previous
10960 global/generic
10961 store/atomicrmw are
10962 visible at system scope.
10964 2. s_waitcnt lgkmcnt(0) &
10965 vmcnt(0)
10967 - If TgSplit execution mode,
10968 omit lgkmcnt(0).
10969 - If OpenCL and
10970 address space is
10971 not generic, omit
10972 lgkmcnt(0).
10973 - If OpenCL and
10974 address space is
10975 local, omit
10976 vmcnt(0).
10977 - However, since LLVM
10978 currently has no
10979 address space on
10980 the fence need to
10981 conservatively
10982 always generate. If
10983 fence had an
10984 address space then
10985 set to address
10986 space of OpenCL
10987 fence flag, or to
10988 generic if both
10989 local and global
10990 flags are
10991 specified.
10992 - Could be split into
10993 separate s_waitcnt
10994 vmcnt(0) and
10995 s_waitcnt
10996 lgkmcnt(0) to allow
10997 them to be
10998 independently moved
10999 according to the
11000 following rules.
11001 - s_waitcnt vmcnt(0)
11002 must happen after
11003 any preceding
11004 global/generic
11005 load/store/load
11006 atomic/store
11007 atomic/atomicrmw.
11008 - s_waitcnt lgkmcnt(0)
11009 must happen after
11010 any preceding
11011 local/generic
11012 load/store/load
11013 atomic/store
11014 atomic/atomicrmw.
11015 - Must happen before
11016 any following store
11017 atomic/atomicrmw
11018 with an equal or
11019 wider sync scope
11020 and memory ordering
11021 stronger than
11022 unordered (this is
11023 termed the
11024 fence-paired-atomic).
11025 - Ensures that all
11026 memory operations
11027 have
11028 completed before
11029 performing the
11030 following
11031 fence-paired-atomic.
11033 **Acquire-Release Atomic**
11034 ------------------------------------------------------------------------------------
11035 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic
11036 - wavefront - generic
11037 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode,
11038 - wavefront local address space cannot
11039 be used.*
11041 1. ds_atomic
11042 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)
11044 - Use lgkmcnt(0) if not
11045 TgSplit execution mode
11046 and vmcnt(0) if TgSplit
11047 execution mode.
11048 - If OpenCL, omit
11049 lgkmcnt(0).
11050 - Must happen after
11051 any preceding
11052 local/generic
11053 load/store/load
11054 atomic/store
11055 atomic/atomicrmw.
11056 - s_waitcnt vmcnt(0)
11057 must happen after
11058 any preceding
11059 global/generic load/store/
11060 load atomic/store atomic/
11061 atomicrmw.
11062 - s_waitcnt lgkmcnt(0)
11063 must happen after
11064 any preceding
11065 local/generic
11066 load/store/load
11067 atomic/store
11068 atomic/atomicrmw.
11069 - Must happen before
11070 the following
11071 atomicrmw.
11072 - Ensures that all
11073 memory operations
11074 have
11075 completed before
11076 performing the
11077 atomicrmw that is
11078 being released.
11080 2. buffer/global_atomic
11081 3. s_waitcnt vmcnt(0)
11083 - If not TgSplit execution
11084 mode, omit.
11085 - Must happen before
11086 the following
11087 buffer_inv.
11088 - Ensures any
11089 following global
11090 data read is no
11091 older than the
11092 atomicrmw value
11093 being acquired.
11095 4. buffer_inv sc0=1
11097 - If not TgSplit execution
11098 mode, omit.
11099 - Ensures that
11100 following
11101 loads will not see
11102 stale data.
11104 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode,
11105 local address space cannot
11106 be used.*
11108 1. ds_atomic
11109 2. s_waitcnt lgkmcnt(0)
11111 - If OpenCL, omit.
11112 - Must happen before
11113 any following
11114 global/generic
11115 load/load
11116 atomic/store/store
11117 atomic/atomicrmw.
11118 - Ensures any
11119 following global
11120 data read is no
11121 older than the local load
11122 atomic value being
11123 acquired.
11125 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0)
11127 - Use lgkmcnt(0) if not
11128 TgSplit execution mode
11129 and vmcnt(0) if TgSplit
11130 execution mode.
11131 - If OpenCL, omit
11132 lgkmcnt(0).
11133 - s_waitcnt vmcnt(0)
11134 must happen after
11135 any preceding
11136 global/generic load/store/
11137 load atomic/store atomic/
11138 atomicrmw.
11139 - s_waitcnt lgkmcnt(0)
11140 must happen after
11141 any preceding
11142 local/generic
11143 load/store/load
11144 atomic/store
11145 atomic/atomicrmw.
11146 - Must happen before
11147 the following
11148 atomicrmw.
11149 - Ensures that all
11150 memory operations
11151 have
11152 completed before
11153 performing the
11154 atomicrmw that is
11155 being released.
11157 2. flat_atomic
11158 3. s_waitcnt lgkmcnt(0) &
11159 vmcnt(0)
11161 - If not TgSplit execution
11162 mode, omit vmcnt(0).
11163 - If OpenCL, omit
11164 lgkmcnt(0).
11165 - Must happen before
11166 the following
11167 buffer_inv and
11168 any following
11169 global/generic
11170 load/load
11171 atomic/store/store
11172 atomic/atomicrmw.
11173 - Ensures any
11174 following global
11175 data read is no
11176 older than a local load
11177 atomic value being
11178 acquired.
11180 3. buffer_inv sc0=1
11182 - If not TgSplit execution
11183 mode, omit.
11184 - Ensures that
11185 following
11186 loads will not see
11187 stale data.
11189 atomicrmw acq_rel - agent - global 1. buffer_wbl2 sc1=1
11191 - Must happen before
11192 following s_waitcnt.
11193 - Performs L2 writeback to
11194 ensure previous
11195 global/generic
11196 store/atomicrmw are
11197 visible at agent scope.
11199 2. s_waitcnt lgkmcnt(0) &
11200 vmcnt(0)
11202 - If TgSplit execution mode,
11203 omit lgkmcnt(0).
11204 - If OpenCL, omit
11205 lgkmcnt(0).
11206 - Could be split into
11207 separate s_waitcnt
11208 vmcnt(0) and
11209 s_waitcnt
11210 lgkmcnt(0) to allow
11211 them to be
11212 independently moved
11213 according to the
11214 following rules.
11215 - s_waitcnt vmcnt(0)
11216 must happen after
11217 any preceding
11218 global/generic
11219 load/store/load
11220 atomic/store
11221 atomic/atomicrmw.
11222 - s_waitcnt lgkmcnt(0)
11223 must happen after
11224 any preceding
11225 local/generic
11226 load/store/load
11227 atomic/store
11228 atomic/atomicrmw.
11229 - Must happen before
11230 the following
11231 atomicrmw.
11232 - Ensures that all
11233 memory operations
11234 to global have
11235 completed before
11236 performing the
11237 atomicrmw that is
11238 being released.
11240 3. buffer/global_atomic
11241 4. s_waitcnt vmcnt(0)
11243 - Must happen before
11244 following
11245 buffer_inv.
11246 - Ensures the
11247 atomicrmw has
11248 completed before
11249 invalidating the
11250 cache.
11252 5. buffer_inv sc1=1
11254 - Must happen before
11255 any following
11256 global/generic
11257 load/load
11258 atomic/atomicrmw.
11259 - Ensures that
11260 following loads
11261 will not see stale
11262 global data.
11264 atomicrmw acq_rel - system - global 1. buffer_wbl2 sc0=1 sc1=1
11266 - Must happen before
11267 following s_waitcnt.
11268 - Performs L2 writeback to
11269 ensure previous
11270 global/generic
11271 store/atomicrmw are
11272 visible at system scope.
11274 2. s_waitcnt lgkmcnt(0) &
11275 vmcnt(0)
11277 - If TgSplit execution mode,
11278 omit lgkmcnt(0).
11279 - If OpenCL, omit
11280 lgkmcnt(0).
11281 - Could be split into
11282 separate s_waitcnt
11283 vmcnt(0) and
11284 s_waitcnt
11285 lgkmcnt(0) to allow
11286 them to be
11287 independently moved
11288 according to the
11289 following rules.
11290 - s_waitcnt vmcnt(0)
11291 must happen after
11292 any preceding
11293 global/generic
11294 load/store/load
11295 atomic/store
11296 atomic/atomicrmw.
11297 - s_waitcnt lgkmcnt(0)
11298 must happen after
11299 any preceding
11300 local/generic
11301 load/store/load
11302 atomic/store
11303 atomic/atomicrmw.
11304 - Must happen before
11305 the following
11306 atomicrmw.
11307 - Ensures that all
11308 memory operations
11309 to global and L2 writeback
11310 have completed before
11311 performing the
11312 atomicrmw that is
11313 being released.
11315 3. buffer/global_atomic
11316 sc1=1
11317 4. s_waitcnt vmcnt(0)
11319 - Must happen before
11320 following
11321 buffer_inv.
11322 - Ensures the
11323 atomicrmw has
11324 completed before
11325 invalidating the
11326 caches.
11328 5. buffer_inv sc0=1 sc1=1
11330 - Must happen before
11331 any following
11332 global/generic
11333 load/load
11334 atomic/atomicrmw.
11335 - Ensures that
11336 following loads
11337 will not see stale
11338 MTYPE NC global data.
11339 MTYPE RW and CC memory will
11340 never be stale due to the
11341 memory probes.
11343 atomicrmw acq_rel - agent - generic 1. buffer_wbl2 sc1=1
11345 - Must happen before
11346 following s_waitcnt.
11347 - Performs L2 writeback to
11348 ensure previous
11349 global/generic
11350 store/atomicrmw are
11351 visible at agent scope.
11353 2. s_waitcnt lgkmcnt(0) &
11354 vmcnt(0)
11356 - If TgSplit execution mode,
11357 omit lgkmcnt(0).
11358 - If OpenCL, omit
11359 lgkmcnt(0).
11360 - Could be split into
11361 separate s_waitcnt
11362 vmcnt(0) and
11363 s_waitcnt
11364 lgkmcnt(0) to allow
11365 them to be
11366 independently moved
11367 according to the
11368 following rules.
11369 - s_waitcnt vmcnt(0)
11370 must happen after
11371 any preceding
11372 global/generic
11373 load/store/load
11374 atomic/store
11375 atomic/atomicrmw.
11376 - s_waitcnt lgkmcnt(0)
11377 must happen after
11378 any preceding
11379 local/generic
11380 load/store/load
11381 atomic/store
11382 atomic/atomicrmw.
11383 - Must happen before
11384 the following
11385 atomicrmw.
11386 - Ensures that all
11387 memory operations
11388 to global have
11389 completed before
11390 performing the
11391 atomicrmw that is
11392 being released.
11394 3. flat_atomic
11395 4. s_waitcnt vmcnt(0) &
11396 lgkmcnt(0)
11398 - If TgSplit execution mode,
11399 omit lgkmcnt(0).
11400 - If OpenCL, omit
11401 lgkmcnt(0).
11402 - Must happen before
11403 following
11404 buffer_inv.
11405 - Ensures the
11406 atomicrmw has
11407 completed before
11408 invalidating the
11409 cache.
11411 5. buffer_inv sc1=1
11413 - Must happen before
11414 any following
11415 global/generic
11416 load/load
11417 atomic/atomicrmw.
11418 - Ensures that
11419 following loads
11420 will not see stale
11421 global data.
11423 atomicrmw acq_rel - system - generic 1. buffer_wbl2 sc0=1 sc1=1
11425 - Must happen before
11426 following s_waitcnt.
11427 - Performs L2 writeback to
11428 ensure previous
11429 global/generic
11430 store/atomicrmw are
11431 visible at system scope.
11433 2. s_waitcnt lgkmcnt(0) &
11434 vmcnt(0)
11436 - If TgSplit execution mode,
11437 omit lgkmcnt(0).
11438 - If OpenCL, omit
11439 lgkmcnt(0).
11440 - Could be split into
11441 separate s_waitcnt
11442 vmcnt(0) and
11443 s_waitcnt
11444 lgkmcnt(0) to allow
11445 them to be
11446 independently moved
11447 according to the
11448 following rules.
11449 - s_waitcnt vmcnt(0)
11450 must happen after
11451 any preceding
11452 global/generic
11453 load/store/load
11454 atomic/store
11455 atomic/atomicrmw.
11456 - s_waitcnt lgkmcnt(0)
11457 must happen after
11458 any preceding
11459 local/generic
11460 load/store/load
11461 atomic/store
11462 atomic/atomicrmw.
11463 - Must happen before
11464 the following
11465 atomicrmw.
11466 - Ensures that all
11467 memory operations
11468 to global and L2 writeback
11469 have completed before
11470 performing the
11471 atomicrmw that is
11472 being released.
11474 3. flat_atomic sc1=1
11475 4. s_waitcnt vmcnt(0) &
11476 lgkmcnt(0)
11478 - If TgSplit execution mode,
11479 omit lgkmcnt(0).
11480 - If OpenCL, omit
11481 lgkmcnt(0).
11482 - Must happen before
11483 following
11484 buffer_inv.
11485 - Ensures the
11486 atomicrmw has
11487 completed before
11488 invalidating the
11489 caches.
11491 5. buffer_inv sc0=1 sc1=1
11493 - Must happen before
11494 any following
11495 global/generic
11496 load/load
11497 atomic/atomicrmw.
11498 - Ensures that
11499 following loads
11500 will not see stale
11501 MTYPE NC global data.
11502 MTYPE RW and CC memory will
11503 never be stale due to the
11504 memory probes.
11506 fence acq_rel - singlethread *none* *none*
11507 - wavefront
11508 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)
11510 - Use lgkmcnt(0) if not
11511 TgSplit execution mode
11512 and vmcnt(0) if TgSplit
11513 execution mode.
11514 - If OpenCL and
11515 address space is
11516 not generic, omit
11517 lgkmcnt(0).
11518 - If OpenCL and
11519 address space is
11520 local, omit
11521 vmcnt(0).
11522 - However,
11523 since LLVM
11524 currently has no
11525 address space on
11526 the fence need to
11527 conservatively
11528 always generate
11529 (see comment for
11530 previous fence).
11531 - s_waitcnt vmcnt(0)
11532 must happen after
11533 any preceding
11534 global/generic
11535 load/store/
11536 load atomic/store atomic/
11537 atomicrmw.
11538 - s_waitcnt lgkmcnt(0)
11539 must happen after
11540 any preceding
11541 local/generic
11542 load/load
11543 atomic/store/store
11544 atomic/atomicrmw.
11545 - Must happen before
11546 any following
11547 global/generic
11548 load/load
11549 atomic/store/store
11550 atomic/atomicrmw.
11551 - Ensures that all
11552 memory operations
11553 have
11554 completed before
11555 performing any
11556 following global
11557 memory operations.
11558 - Ensures that the
11559 preceding
11560 local/generic load
11561 atomic/atomicrmw
11562 with an equal or
11563 wider sync scope
11564 and memory ordering
11565 stronger than
11566 unordered (this is
11567 termed the
11568 acquire-fence-paired-atomic)
11569 has completed
11570 before following
11571 global memory
11572 operations. This
11573 satisfies the
11574 requirements of
11575 acquire.
11576 - Ensures that all
11577 previous memory
11578 operations have
11579 completed before a
11580 following
11581 local/generic store
11582 atomic/atomicrmw
11583 with an equal or
11584 wider sync scope
11585 and memory ordering
11586 stronger than
11587 unordered (this is
11588 termed the
11589 release-fence-paired-atomic).
11590 This satisfies the
11591 requirements of
11592 release.
11593 - Must happen before
11594 the following
11595 buffer_inv.
11596 - Ensures that the
11597 acquire-fence-paired
11598 atomic has completed
11599 before invalidating
11600 the
11601 cache. Therefore
11602 any following
11603 locations read must
11604 be no older than
11605 the value read by
11606 the
11607 acquire-fence-paired-atomic.
11609 3. buffer_inv sc0=1
11611 - If not TgSplit execution
11612 mode, omit.
11613 - Ensures that
11614 following
11615 loads will not see
11616 stale data.
11618 fence acq_rel - agent *none* 1. buffer_wbl2 sc1=1
11620 - If OpenCL and
11621 address space is
11622 local, omit.
11623 - Must happen before
11624 following s_waitcnt.
11625 - Performs L2 writeback to
11626 ensure previous
11627 global/generic
11628 store/atomicrmw are
11629 visible at agent scope.
11631 2. s_waitcnt lgkmcnt(0) &
11632 vmcnt(0)
11634 - If TgSplit execution mode,
11635 omit lgkmcnt(0).
11636 - If OpenCL and
11637 address space is
11638 not generic, omit
11639 lgkmcnt(0).
11640 - However, since LLVM
11641 currently has no
11642 address space on
11643 the fence need to
11644 conservatively
11645 always generate
11646 (see comment for
11647 previous fence).
11648 - Could be split into
11649 separate s_waitcnt
11650 vmcnt(0) and
11651 s_waitcnt
11652 lgkmcnt(0) to allow
11653 them to be
11654 independently moved
11655 according to the
11656 following rules.
11657 - s_waitcnt vmcnt(0)
11658 must happen after
11659 any preceding
11660 global/generic
11661 load/store/load
11662 atomic/store
11663 atomic/atomicrmw.
11664 - s_waitcnt lgkmcnt(0)
11665 must happen after
11666 any preceding
11667 local/generic
11668 load/store/load
11669 atomic/store
11670 atomic/atomicrmw.
11671 - Must happen before
11672 the following
11673 buffer_inv.
11674 - Ensures that the
11675 preceding
11676 global/local/generic
11677 load
11678 atomic/atomicrmw
11679 with an equal or
11680 wider sync scope
11681 and memory ordering
11682 stronger than
11683 unordered (this is
11684 termed the
11685 acquire-fence-paired-atomic)
11686 has completed
11687 before invalidating
11688 the cache. This
11689 satisfies the
11690 requirements of
11691 acquire.
11692 - Ensures that all
11693 previous memory
11694 operations have
11695 completed before a
11696 following
11697 global/local/generic
11698 store
11699 atomic/atomicrmw
11700 with an equal or
11701 wider sync scope
11702 and memory ordering
11703 stronger than
11704 unordered (this is
11705 termed the
11706 release-fence-paired-atomic).
11707 This satisfies the
11708 requirements of
11709 release.
11711 3. buffer_inv sc1=1
11713 - Must happen before
11714 any following
11715 global/generic
11716 load/load
11717 atomic/store/store
11718 atomic/atomicrmw.
11719 - Ensures that
11720 following loads
11721 will not see stale
11722 global data. This
11723 satisfies the
11724 requirements of
11725 acquire.
11727 fence acq_rel - system *none* 1. buffer_wbl2 sc0=1 sc1=1
11729 - If OpenCL and
11730 address space is
11731 local, omit.
11732 - Must happen before
11733 following s_waitcnt.
11734 - Performs L2 writeback to
11735 ensure previous
11736 global/generic
11737 store/atomicrmw are
11738 visible at system scope.
11740 1. s_waitcnt lgkmcnt(0) &
11741 vmcnt(0)
11743 - If TgSplit execution mode,
11744 omit lgkmcnt(0).
11745 - If OpenCL and
11746 address space is
11747 not generic, omit
11748 lgkmcnt(0).
11749 - However, since LLVM
11750 currently has no
11751 address space on
11752 the fence need to
11753 conservatively
11754 always generate
11755 (see comment for
11756 previous fence).
11757 - Could be split into
11758 separate s_waitcnt
11759 vmcnt(0) and
11760 s_waitcnt
11761 lgkmcnt(0) to allow
11762 them to be
11763 independently moved
11764 according to the
11765 following rules.
11766 - s_waitcnt vmcnt(0)
11767 must happen after
11768 any preceding
11769 global/generic
11770 load/store/load
11771 atomic/store
11772 atomic/atomicrmw.
11773 - s_waitcnt lgkmcnt(0)
11774 must happen after
11775 any preceding
11776 local/generic
11777 load/store/load
11778 atomic/store
11779 atomic/atomicrmw.
11780 - Must happen before
11781 the following
11782 buffer_inv.
11783 - Ensures that the
11784 preceding
11785 global/local/generic
11786 load
11787 atomic/atomicrmw
11788 with an equal or
11789 wider sync scope
11790 and memory ordering
11791 stronger than
11792 unordered (this is
11793 termed the
11794 acquire-fence-paired-atomic)
11795 has completed
11796 before invalidating
11797 the cache. This
11798 satisfies the
11799 requirements of
11800 acquire.
11801 - Ensures that all
11802 previous memory
11803 operations have
11804 completed before a
11805 following
11806 global/local/generic
11807 store
11808 atomic/atomicrmw
11809 with an equal or
11810 wider sync scope
11811 and memory ordering
11812 stronger than
11813 unordered (this is
11814 termed the
11815 release-fence-paired-atomic).
11816 This satisfies the
11817 requirements of
11818 release.
11820 2. buffer_inv sc0=1 sc1=1
11822 - Must happen before
11823 any following
11824 global/generic
11825 load/load
11826 atomic/store/store
11827 atomic/atomicrmw.
11828 - Ensures that
11829 following loads
11830 will not see stale
11831 MTYPE NC global data.
11832 MTYPE RW and CC memory will
11833 never be stale due to the
11834 memory probes.
11836 **Sequential Consistent Atomic**
11837 ------------------------------------------------------------------------------------
11838 load atomic seq_cst - singlethread - global *Same as corresponding
11839 - wavefront - local load atomic acquire,
11840 - generic except must generate
11841 all instructions even
11842 for OpenCL.*
11843 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)
11844 - generic
11845 - Use lgkmcnt(0) if not
11846 TgSplit execution mode
11847 and vmcnt(0) if TgSplit
11848 execution mode.
11849 - s_waitcnt lgkmcnt(0) must
11850 happen after
11851 preceding
11852 local/generic load
11853 atomic/store
11854 atomic/atomicrmw
11855 with memory
11856 ordering of seq_cst
11857 and with equal or
11858 wider sync scope.
11859 (Note that seq_cst
11860 fences have their
11861 own s_waitcnt
11862 lgkmcnt(0) and so do
11863 not need to be
11864 considered.)
11865 - s_waitcnt vmcnt(0)
11866 must happen after
11867 preceding
11868 global/generic load
11869 atomic/store
11870 atomic/atomicrmw
11871 with memory
11872 ordering of seq_cst
11873 and with equal or
11874 wider sync scope.
11875 (Note that seq_cst
11876 fences have their
11877 own s_waitcnt
11878 vmcnt(0) and so do
11879 not need to be
11880 considered.)
11881 - Ensures any
11882 preceding
11883 sequential
11884 consistent global/local
11885 memory instructions
11886 have completed
11887 before executing
11888 this sequentially
11889 consistent
11890 instruction. This
11891 prevents reordering
11892 a seq_cst store
11893 followed by a
11894 seq_cst load. (Note
11895 that seq_cst is
11896 stronger than
11897 acquire/release as
11898 the reordering of
11899 load acquire
11900 followed by a store
11901 release is
11902 prevented by the
11903 s_waitcnt of
11904 the release, but
11905 there is nothing
11906 preventing a store
11907 release followed by
11908 load acquire from
11909 completing out of
11910 order. The s_waitcnt
11911 could be placed after
11912 seq_store or before
11913 the seq_load. We
11914 choose the load to
11915 make the s_waitcnt be
11916 as late as possible
11917 so that the store
11918 may have already
11919 completed.)
11921 2. *Following
11922 instructions same as
11923 corresponding load
11924 atomic acquire,
11925 except must generate
11926 all instructions even
11927 for OpenCL.*
11928 load atomic seq_cst - workgroup - local *If TgSplit execution mode,
11929 local address space cannot
11930 be used.*
11932 *Same as corresponding
11933 load atomic acquire,
11934 except must generate
11935 all instructions even
11936 for OpenCL.*
11938 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &
11939 - system - generic vmcnt(0)
11941 - If TgSplit execution mode,
11942 omit lgkmcnt(0).
11943 - Could be split into
11944 separate s_waitcnt
11945 vmcnt(0)
11946 and s_waitcnt
11947 lgkmcnt(0) to allow
11948 them to be
11949 independently moved
11950 according to the
11951 following rules.
11952 - s_waitcnt lgkmcnt(0)
11953 must happen after
11954 preceding
11955 global/generic load
11956 atomic/store
11957 atomic/atomicrmw
11958 with memory
11959 ordering of seq_cst
11960 and with equal or
11961 wider sync scope.
11962 (Note that seq_cst
11963 fences have their
11964 own s_waitcnt
11965 lgkmcnt(0) and so do
11966 not need to be
11967 considered.)
11968 - s_waitcnt vmcnt(0)
11969 must happen after
11970 preceding
11971 global/generic load
11972 atomic/store
11973 atomic/atomicrmw
11974 with memory
11975 ordering of seq_cst
11976 and with equal or
11977 wider sync scope.
11978 (Note that seq_cst
11979 fences have their
11980 own s_waitcnt
11981 vmcnt(0) and so do
11982 not need to be
11983 considered.)
11984 - Ensures any
11985 preceding
11986 sequential
11987 consistent global
11988 memory instructions
11989 have completed
11990 before executing
11991 this sequentially
11992 consistent
11993 instruction. This
11994 prevents reordering
11995 a seq_cst store
11996 followed by a
11997 seq_cst load. (Note
11998 that seq_cst is
11999 stronger than
12000 acquire/release as
12001 the reordering of
12002 load acquire
12003 followed by a store
12004 release is
12005 prevented by the
12006 s_waitcnt of
12007 the release, but
12008 there is nothing
12009 preventing a store
12010 release followed by
12011 load acquire from
12012 completing out of
12013 order. The s_waitcnt
12014 could be placed after
12015 seq_store or before
12016 the seq_load. We
12017 choose the load to
12018 make the s_waitcnt be
12019 as late as possible
12020 so that the store
12021 may have already
12022 completed.)
12024 2. *Following
12025 instructions same as
12026 corresponding load
12027 atomic acquire,
12028 except must generate
12029 all instructions even
12030 for OpenCL.*
12031 store atomic seq_cst - singlethread - global *Same as corresponding
12032 - wavefront - local store atomic release,
12033 - workgroup - generic except must generate
12034 - agent all instructions even
12035 - system for OpenCL.*
12036 atomicrmw seq_cst - singlethread - global *Same as corresponding
12037 - wavefront - local atomicrmw acq_rel,
12038 - workgroup - generic except must generate
12039 - agent all instructions even
12040 - system for OpenCL.*
12041 fence seq_cst - singlethread *none* *Same as corresponding
12042 - wavefront fence acq_rel,
12043 - workgroup except must generate
12044 - agent all instructions even
12045 - system for OpenCL.*
12046 ============ ============ ============== ========== ================================
12048 .. _amdgpu-amdhsa-memory-model-gfx10-gfx11:
12050 Memory Model GFX10-GFX11
12051 ++++++++++++++++++++++++
12053 For GFX10-GFX11:
12055 * Each agent has multiple shader arrays (SA).
12056 * Each SA has multiple work-group processors (WGP).
12057 * Each WGP has multiple compute units (CU).
12058 * Each CU has multiple SIMDs that execute wavefronts.
12059 * The wavefronts for a single work-group are executed in the same
12060 WGP. In CU wavefront execution mode the wavefronts may be executed by
12061 different SIMDs in the same CU. In WGP wavefront execution mode the
12062 wavefronts may be executed by different SIMDs in different CUs in the same
12063 WGP.
12064 * Each WGP has a single LDS memory shared by the wavefronts of the work-groups
12065 executing on it.
12066 * All LDS operations of a WGP are performed as wavefront wide operations in a
12067 global order and involve no caching. Completion is reported to a wavefront in
12068 execution order.
12069 * The LDS memory has multiple request queues shared by the SIMDs of a
12070 WGP. Therefore, the LDS operations performed by different wavefronts of a
12071 work-group can be reordered relative to each other, which can result in
12072 reordering the visibility of vector memory operations with respect to LDS
12073 operations of other wavefronts in the same work-group. A ``s_waitcnt
12074 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and
12075 vector memory operations between wavefronts of a work-group, but not between
12076 operations performed by the same wavefront.
12077 * The vector memory operations are performed as wavefront wide operations.
12078 Completion of load/store/sample operations are reported to a wavefront in
12079 execution order of other load/store/sample operations performed by that
12080 wavefront.
12081 * The vector memory operations access a vector L0 cache. There is a single L0
12082 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no
12083 special action is required for coherence between the lanes of a single
12084 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between
12085 wavefronts executing in the same work-group as they may be executing on SIMDs
12086 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also
12087 required for coherence between wavefronts executing in different work-groups
12088 as they may be executing on different WGPs.
12089 * The scalar memory operations access a scalar L0 cache shared by all wavefronts
12090 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar
12091 operations are used in a restricted way so do not impact the memory model. See
12092 :ref:`amdgpu-amdhsa-memory-spaces`.
12093 * The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on
12094 the same SA. Therefore, no special action is required for coherence between
12095 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is
12096 required for coherence between wavefronts executing in different work-groups
12097 as they may be executing on different SAs that access different L1s.
12098 * The L1 caches have independent quadrants to service disjoint ranges of virtual
12099 addresses.
12100 * Each L0 cache has a separate request queue per L1 quadrant. Therefore, the
12101 vector and scalar memory operations performed by different wavefronts, whether
12102 executing in the same or different work-groups (which may be executing on
12103 different CUs accessing different L0s), can be reordered relative to each
12104 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure
12105 synchronization between vector memory operations of different wavefronts. It
12106 ensures a previous vector memory operation has completed before executing a
12107 subsequent vector memory or LDS operation and so can be used to meet the
12108 requirements of acquire, release and sequential consistency.
12109 * The L1 caches use an L2 cache shared by all SAs on the same agent.
12110 * The L2 cache has independent channels to service disjoint ranges of virtual
12111 addresses.
12112 * Each L1 quadrant of a single SA accesses a different L2 channel. Each L1
12113 quadrant has a separate request queue per L2 channel. Therefore, the vector
12114 and scalar memory operations performed by wavefronts executing in different
12115 work-groups (which may be executing on different SAs) of an agent can be
12116 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is
12117 required to ensure synchronization between vector memory operations of
12118 different SAs. It ensures a previous vector memory operation has completed
12119 before executing a subsequent vector memory and so can be used to meet the
12120 requirements of acquire, release and sequential consistency.
12121 * The L2 cache can be kept coherent with other agents on some targets, or ranges
12122 of virtual addresses can be set up to bypass it to ensure system coherence.
12123 * On GFX10.3 and GFX11 a memory attached last level (MALL) cache exists for GPU memory.
12124 The MALL cache is fully coherent with GPU memory and has no impact on system
12125 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache.
12127 Scalar memory operations are only used to access memory that is proven to not
12128 change during the execution of the kernel dispatch. This includes constant
12129 address space and global address space for program scope ``const`` variables.
12130 Therefore, the kernel machine code does not have to maintain the scalar cache to
12131 ensure it is coherent with the vector caches. The scalar and vector caches are
12132 invalidated between kernel dispatches by CP since constant address space data
12133 may change between kernel dispatch executions. See
12134 :ref:`amdgpu-amdhsa-memory-spaces`.
12136 The one exception is if scalar writes are used to spill SGPR registers. In this
12137 case the AMDGPU backend ensures the memory location used to spill is never
12138 accessed by vector memory operations at the same time. If scalar writes are used
12139 then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function
12140 return since the locations may be used for vector memory instructions by a
12141 future wavefront that uses the same scratch area, or a function call that
12142 creates a frame at the same address, respectively. There is no need for a
12143 ``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.
12145 For kernarg backing memory:
12147 * CP invalidates the L0 and L1 caches at the start of each kernel dispatch.
12148 * On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid
12149 needing to invalidate the L2 cache.
12150 * On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and
12151 so the L2 cache will be coherent with the CPU and other agents.
12153 Scratch backing memory (which is used for the private address space) is accessed
12154 with MTYPE NC (non-coherent). Since the private address space is only accessed
12155 by a single thread, and is always write-before-read, there is never a need to
12156 invalidate these entries from the L0 or L1 caches.
12158 Wavefronts are executed in native mode with in-order reporting of loads and
12159 sample instructions. In this mode vmcnt reports completion of load, atomic with
12160 return and sample instructions in order, and the vscnt reports the completion of
12161 store and atomic without return in order. See ``MEM_ORDERED`` field in
12162 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
12164 Wavefronts can be executed in WGP or CU wavefront execution mode:
12166 * In WGP wavefront execution mode the wavefronts of a work-group are executed
12167 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per
12168 CU L0 caches is required for work-group synchronization. Also accesses to L1
12169 at work-group scope need to be explicitly ordered as the accesses from
12170 different CUs are not ordered.
12171 * In CU wavefront execution mode the wavefronts of a work-group are executed on
12172 the SIMDs of a single CU of the WGP. Therefore, all global memory access by
12173 the work-group access the same L0 which in turn ensures L1 accesses are
12174 ordered and so do not require explicit management of the caches for
12175 work-group synchronization.
12177 See ``WGP_MODE`` field in
12178 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table` and
12179 :ref:`amdgpu-target-features`.
12181 The code sequences used to implement the memory model for GFX10-GFX11 are defined in
12182 table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
12184 .. table:: AMDHSA Memory Model Code Sequences GFX10-GFX11
12185 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table
12187 ============ ============ ============== ========== ================================
12188 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code
12189 Ordering Sync Scope Address GFX10-GFX11
12190 Space
12191 ============ ============ ============== ========== ================================
12192 **Non-Atomic**
12193 ------------------------------------------------------------------------------------
12194 load *none* *none* - global - !volatile & !nontemporal
12195 - generic
12196 - private 1. buffer/global/flat_load
12197 - constant
12198 - !volatile & nontemporal
12200 1. buffer/global/flat_load
12201 slc=1 dlc=1
12203 - If GFX10, omit dlc=1.
12205 - volatile
12207 1. buffer/global/flat_load
12208 glc=1 dlc=1
12210 2. s_waitcnt vmcnt(0)
12212 - Must happen before
12213 any following volatile
12214 global/generic
12215 load/store.
12216 - Ensures that
12217 volatile
12218 operations to
12219 different
12220 addresses will not
12221 be reordered by
12222 hardware.
12224 load *none* *none* - local 1. ds_load
12225 store *none* *none* - global - !volatile & !nontemporal
12226 - generic
12227 - private 1. buffer/global/flat_store
12228 - constant
12229 - !volatile & nontemporal
12231 1. buffer/global/flat_store
12232 glc=1 slc=1 dlc=1
12234 - If GFX10, omit dlc=1.
12236 - volatile
12238 1. buffer/global/flat_store
12239 dlc=1
12241 - If GFX10, omit dlc=1.
12243 2. s_waitcnt vscnt(0)
12245 - Must happen before
12246 any following volatile
12247 global/generic
12248 load/store.
12249 - Ensures that
12250 volatile
12251 operations to
12252 different
12253 addresses will not
12254 be reordered by
12255 hardware.
12257 store *none* *none* - local 1. ds_store
12258 **Unordered Atomic**
12259 ------------------------------------------------------------------------------------
12260 load atomic unordered *any* *any* *Same as non-atomic*.
12261 store atomic unordered *any* *any* *Same as non-atomic*.
12262 atomicrmw unordered *any* *any* *Same as monotonic atomic*.
12263 **Monotonic Atomic**
12264 ------------------------------------------------------------------------------------
12265 load atomic monotonic - singlethread - global 1. buffer/global/flat_load
12266 - wavefront - generic
12267 load atomic monotonic - workgroup - global 1. buffer/global/flat_load
12268 - generic glc=1
12270 - If CU wavefront execution
12271 mode, omit glc=1.
12273 load atomic monotonic - singlethread - local 1. ds_load
12274 - wavefront
12275 - workgroup
12276 load atomic monotonic - agent - global 1. buffer/global/flat_load
12277 - system - generic glc=1 dlc=1
12279 - If GFX11, omit dlc=1.
12281 store atomic monotonic - singlethread - global 1. buffer/global/flat_store
12282 - wavefront - generic
12283 - workgroup
12284 - agent
12285 - system
12286 store atomic monotonic - singlethread - local 1. ds_store
12287 - wavefront
12288 - workgroup
12289 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic
12290 - wavefront - generic
12291 - workgroup
12292 - agent
12293 - system
12294 atomicrmw monotonic - singlethread - local 1. ds_atomic
12295 - wavefront
12296 - workgroup
12297 **Acquire Atomic**
12298 ------------------------------------------------------------------------------------
12299 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load
12300 - wavefront - local
12301 - generic
12302 load atomic acquire - workgroup - global 1. buffer/global_load glc=1
12304 - If CU wavefront execution
12305 mode, omit glc=1.
12307 2. s_waitcnt vmcnt(0)
12309 - If CU wavefront execution
12310 mode, omit.
12311 - Must happen before
12312 the following buffer_gl0_inv
12313 and before any following
12314 global/generic
12315 load/load
12316 atomic/store/store
12317 atomic/atomicrmw.
12319 3. buffer_gl0_inv
12321 - If CU wavefront execution
12322 mode, omit.
12323 - Ensures that
12324 following
12325 loads will not see
12326 stale data.
12328 load atomic acquire - workgroup - local 1. ds_load
12329 2. s_waitcnt lgkmcnt(0)
12331 - If OpenCL, omit.
12332 - Must happen before
12333 the following buffer_gl0_inv
12334 and before any following
12335 global/generic load/load
12336 atomic/store/store
12337 atomic/atomicrmw.
12338 - Ensures any
12339 following global
12340 data read is no
12341 older than the local load
12342 atomic value being
12343 acquired.
12345 3. buffer_gl0_inv
12347 - If CU wavefront execution
12348 mode, omit.
12349 - If OpenCL, omit.
12350 - Ensures that
12351 following
12352 loads will not see
12353 stale data.
12355 load atomic acquire - workgroup - generic 1. flat_load glc=1
12357 - If CU wavefront execution
12358 mode, omit glc=1.
12360 2. s_waitcnt lgkmcnt(0) &
12361 vmcnt(0)
12363 - If CU wavefront execution
12364 mode, omit vmcnt(0).
12365 - If OpenCL, omit
12366 lgkmcnt(0).
12367 - Must happen before
12368 the following
12369 buffer_gl0_inv and any
12370 following global/generic
12371 load/load
12372 atomic/store/store
12373 atomic/atomicrmw.
12374 - Ensures any
12375 following global
12376 data read is no
12377 older than a local load
12378 atomic value being
12379 acquired.
12381 3. buffer_gl0_inv
12383 - If CU wavefront execution
12384 mode, omit.
12385 - Ensures that
12386 following
12387 loads will not see
12388 stale data.
12390 load atomic acquire - agent - global 1. buffer/global_load
12391 - system glc=1 dlc=1
12393 - If GFX11, omit dlc=1.
12395 2. s_waitcnt vmcnt(0)
12397 - Must happen before
12398 following
12399 buffer_gl*_inv.
12400 - Ensures the load
12401 has completed
12402 before invalidating
12403 the caches.
12405 3. buffer_gl1_inv;
12406 buffer_gl0_inv
12408 - Must happen before
12409 any following
12410 global/generic
12411 load/load
12412 atomic/atomicrmw.
12413 - Ensures that
12414 following
12415 loads will not see
12416 stale global data.
12418 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1
12419 - system
12420 - If GFX11, omit dlc=1.
12422 2. s_waitcnt vmcnt(0) &
12423 lgkmcnt(0)
12425 - If OpenCL omit
12426 lgkmcnt(0).
12427 - Must happen before
12428 following
12429 buffer_gl*_invl.
12430 - Ensures the flat_load
12431 has completed
12432 before invalidating
12433 the caches.
12435 3. buffer_gl1_inv;
12436 buffer_gl0_inv
12438 - Must happen before
12439 any following
12440 global/generic
12441 load/load
12442 atomic/atomicrmw.
12443 - Ensures that
12444 following loads
12445 will not see stale
12446 global data.
12448 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic
12449 - wavefront - local
12450 - generic
12451 atomicrmw acquire - workgroup - global 1. buffer/global_atomic
12452 2. s_waitcnt vm/vscnt(0)
12454 - If CU wavefront execution
12455 mode, omit.
12456 - Use vmcnt(0) if atomic with
12457 return and vscnt(0) if
12458 atomic with no-return.
12459 - Must happen before
12460 the following buffer_gl0_inv
12461 and before any following
12462 global/generic
12463 load/load
12464 atomic/store/store
12465 atomic/atomicrmw.
12467 3. buffer_gl0_inv
12469 - If CU wavefront execution
12470 mode, omit.
12471 - Ensures that
12472 following
12473 loads will not see
12474 stale data.
12476 atomicrmw acquire - workgroup - local 1. ds_atomic
12477 2. s_waitcnt lgkmcnt(0)
12479 - If OpenCL, omit.
12480 - Must happen before
12481 the following
12482 buffer_gl0_inv.
12483 - Ensures any
12484 following global
12485 data read is no
12486 older than the local
12487 atomicrmw value
12488 being acquired.
12490 3. buffer_gl0_inv
12492 - If OpenCL omit.
12493 - Ensures that
12494 following
12495 loads will not see
12496 stale data.
12498 atomicrmw acquire - workgroup - generic 1. flat_atomic
12499 2. s_waitcnt lgkmcnt(0) &
12500 vm/vscnt(0)
12502 - If CU wavefront execution
12503 mode, omit vm/vscnt(0).
12504 - If OpenCL, omit lgkmcnt(0).
12505 - Use vmcnt(0) if atomic with
12506 return and vscnt(0) if
12507 atomic with no-return.
12508 - Must happen before
12509 the following
12510 buffer_gl0_inv.
12511 - Ensures any
12512 following global
12513 data read is no
12514 older than a local
12515 atomicrmw value
12516 being acquired.
12518 3. buffer_gl0_inv
12520 - If CU wavefront execution
12521 mode, omit.
12522 - Ensures that
12523 following
12524 loads will not see
12525 stale data.
12527 atomicrmw acquire - agent - global 1. buffer/global_atomic
12528 - system 2. s_waitcnt vm/vscnt(0)
12530 - Use vmcnt(0) if atomic with
12531 return and vscnt(0) if
12532 atomic with no-return.
12533 - Must happen before
12534 following
12535 buffer_gl*_inv.
12536 - Ensures the
12537 atomicrmw has
12538 completed before
12539 invalidating the
12540 caches.
12542 3. buffer_gl1_inv;
12543 buffer_gl0_inv
12545 - Must happen before
12546 any following
12547 global/generic
12548 load/load
12549 atomic/atomicrmw.
12550 - Ensures that
12551 following loads
12552 will not see stale
12553 global data.
12555 atomicrmw acquire - agent - generic 1. flat_atomic
12556 - system 2. s_waitcnt vm/vscnt(0) &
12557 lgkmcnt(0)
12559 - If OpenCL, omit
12560 lgkmcnt(0).
12561 - Use vmcnt(0) if atomic with
12562 return and vscnt(0) if
12563 atomic with no-return.
12564 - Must happen before
12565 following
12566 buffer_gl*_inv.
12567 - Ensures the
12568 atomicrmw has
12569 completed before
12570 invalidating the
12571 caches.
12573 3. buffer_gl1_inv;
12574 buffer_gl0_inv
12576 - Must happen before
12577 any following
12578 global/generic
12579 load/load
12580 atomic/atomicrmw.
12581 - Ensures that
12582 following loads
12583 will not see stale
12584 global data.
12586 fence acquire - singlethread *none* *none*
12587 - wavefront
12588 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) &
12589 vmcnt(0) & vscnt(0)
12591 - If CU wavefront execution
12592 mode, omit vmcnt(0) and
12593 vscnt(0).
12594 - If OpenCL and
12595 address space is
12596 not generic, omit
12597 lgkmcnt(0).
12598 - If OpenCL and
12599 address space is
12600 local, omit
12601 vmcnt(0) and vscnt(0).
12602 - However, since LLVM
12603 currently has no
12604 address space on
12605 the fence need to
12606 conservatively
12607 always generate. If
12608 fence had an
12609 address space then
12610 set to address
12611 space of OpenCL
12612 fence flag, or to
12613 generic if both
12614 local and global
12615 flags are
12616 specified.
12617 - Could be split into
12618 separate s_waitcnt
12619 vmcnt(0), s_waitcnt
12620 vscnt(0) and s_waitcnt
12621 lgkmcnt(0) to allow
12622 them to be
12623 independently moved
12624 according to the
12625 following rules.
12626 - s_waitcnt vmcnt(0)
12627 must happen after
12628 any preceding
12629 global/generic load
12630 atomic/
12631 atomicrmw-with-return-value
12632 with an equal or
12633 wider sync scope
12634 and memory ordering
12635 stronger than
12636 unordered (this is
12637 termed the
12638 fence-paired-atomic).
12639 - s_waitcnt vscnt(0)
12640 must happen after
12641 any preceding
12642 global/generic
12643 atomicrmw-no-return-value
12644 with an equal or
12645 wider sync scope
12646 and memory ordering
12647 stronger than
12648 unordered (this is
12649 termed the
12650 fence-paired-atomic).
12651 - s_waitcnt lgkmcnt(0)
12652 must happen after
12653 any preceding
12654 local/generic load
12655 atomic/atomicrmw
12656 with an equal or
12657 wider sync scope
12658 and memory ordering
12659 stronger than
12660 unordered (this is
12661 termed the
12662 fence-paired-atomic).
12663 - Must happen before
12664 the following
12665 buffer_gl0_inv.
12666 - Ensures that the
12667 fence-paired atomic
12668 has completed
12669 before invalidating
12670 the
12671 cache. Therefore
12672 any following
12673 locations read must
12674 be no older than
12675 the value read by
12676 the
12677 fence-paired-atomic.
12679 3. buffer_gl0_inv
12681 - If CU wavefront execution
12682 mode, omit.
12683 - Ensures that
12684 following
12685 loads will not see
12686 stale data.
12688 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) &
12689 - system vmcnt(0) & vscnt(0)
12691 - If OpenCL and
12692 address space is
12693 not generic, omit
12694 lgkmcnt(0).
12695 - If OpenCL and
12696 address space is
12697 local, omit
12698 vmcnt(0) and vscnt(0).
12699 - However, since LLVM
12700 currently has no
12701 address space on
12702 the fence need to
12703 conservatively
12704 always generate
12705 (see comment for
12706 previous fence).
12707 - Could be split into
12708 separate s_waitcnt
12709 vmcnt(0), s_waitcnt
12710 vscnt(0) and s_waitcnt
12711 lgkmcnt(0) to allow
12712 them to be
12713 independently moved
12714 according to the
12715 following rules.
12716 - s_waitcnt vmcnt(0)
12717 must happen after
12718 any preceding
12719 global/generic load
12720 atomic/
12721 atomicrmw-with-return-value
12722 with an equal or
12723 wider sync scope
12724 and memory ordering
12725 stronger than
12726 unordered (this is
12727 termed the
12728 fence-paired-atomic).
12729 - s_waitcnt vscnt(0)
12730 must happen after
12731 any preceding
12732 global/generic
12733 atomicrmw-no-return-value
12734 with an equal or
12735 wider sync scope
12736 and memory ordering
12737 stronger than
12738 unordered (this is
12739 termed the
12740 fence-paired-atomic).
12741 - s_waitcnt lgkmcnt(0)
12742 must happen after
12743 any preceding
12744 local/generic load
12745 atomic/atomicrmw
12746 with an equal or
12747 wider sync scope
12748 and memory ordering
12749 stronger than
12750 unordered (this is
12751 termed the
12752 fence-paired-atomic).
12753 - Must happen before
12754 the following
12755 buffer_gl*_inv.
12756 - Ensures that the
12757 fence-paired atomic
12758 has completed
12759 before invalidating
12760 the
12761 caches. Therefore
12762 any following
12763 locations read must
12764 be no older than
12765 the value read by
12766 the
12767 fence-paired-atomic.
12769 2. buffer_gl1_inv;
12770 buffer_gl0_inv
12772 - Must happen before any
12773 following global/generic
12774 load/load
12775 atomic/store/store
12776 atomic/atomicrmw.
12777 - Ensures that
12778 following loads
12779 will not see stale
12780 global data.
12782 **Release Atomic**
12783 ------------------------------------------------------------------------------------
12784 store atomic release - singlethread - global 1. buffer/global/ds/flat_store
12785 - wavefront - local
12786 - generic
12787 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) &
12788 - generic vmcnt(0) & vscnt(0)
12790 - If CU wavefront execution
12791 mode, omit vmcnt(0) and
12792 vscnt(0).
12793 - If OpenCL, omit
12794 lgkmcnt(0).
12795 - Could be split into
12796 separate s_waitcnt
12797 vmcnt(0), s_waitcnt
12798 vscnt(0) and s_waitcnt
12799 lgkmcnt(0) to allow
12800 them to be
12801 independently moved
12802 according to the
12803 following rules.
12804 - s_waitcnt vmcnt(0)
12805 must happen after
12806 any preceding
12807 global/generic load/load
12808 atomic/
12809 atomicrmw-with-return-value.
12810 - s_waitcnt vscnt(0)
12811 must happen after
12812 any preceding
12813 global/generic
12814 store/store
12815 atomic/
12816 atomicrmw-no-return-value.
12817 - s_waitcnt lgkmcnt(0)
12818 must happen after
12819 any preceding
12820 local/generic
12821 load/store/load
12822 atomic/store
12823 atomic/atomicrmw.
12824 - Must happen before
12825 the following
12826 store.
12827 - Ensures that all
12828 memory operations
12829 have
12830 completed before
12831 performing the
12832 store that is being
12833 released.
12835 2. buffer/global/flat_store
12836 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)
12838 - If CU wavefront execution
12839 mode, omit.
12840 - If OpenCL, omit.
12841 - Could be split into
12842 separate s_waitcnt
12843 vmcnt(0) and s_waitcnt
12844 vscnt(0) to allow
12845 them to be
12846 independently moved
12847 according to the
12848 following rules.
12849 - s_waitcnt vmcnt(0)
12850 must happen after
12851 any preceding
12852 global/generic load/load
12853 atomic/
12854 atomicrmw-with-return-value.
12855 - s_waitcnt vscnt(0)
12856 must happen after
12857 any preceding
12858 global/generic
12859 store/store atomic/
12860 atomicrmw-no-return-value.
12861 - Must happen before
12862 the following
12863 store.
12864 - Ensures that all
12865 global memory
12866 operations have
12867 completed before
12868 performing the
12869 store that is being
12870 released.
12872 2. ds_store
12873 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &
12874 - system - generic vmcnt(0) & vscnt(0)
12876 - If OpenCL and
12877 address space is
12878 not generic, omit
12879 lgkmcnt(0).
12880 - Could be split into
12881 separate s_waitcnt
12882 vmcnt(0), s_waitcnt vscnt(0)
12883 and s_waitcnt
12884 lgkmcnt(0) to allow
12885 them to be
12886 independently moved
12887 according to the
12888 following rules.
12889 - s_waitcnt vmcnt(0)
12890 must happen after
12891 any preceding
12892 global/generic
12893 load/load
12894 atomic/
12895 atomicrmw-with-return-value.
12896 - s_waitcnt vscnt(0)
12897 must happen after
12898 any preceding
12899 global/generic
12900 store/store atomic/
12901 atomicrmw-no-return-value.
12902 - s_waitcnt lgkmcnt(0)
12903 must happen after
12904 any preceding
12905 local/generic
12906 load/store/load
12907 atomic/store
12908 atomic/atomicrmw.
12909 - Must happen before
12910 the following
12911 store.
12912 - Ensures that all
12913 memory operations
12914 have
12915 completed before
12916 performing the
12917 store that is being
12918 released.
12920 2. buffer/global/flat_store
12921 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic
12922 - wavefront - local
12923 - generic
12924 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) &
12925 - generic vmcnt(0) & vscnt(0)
12927 - If CU wavefront execution
12928 mode, omit vmcnt(0) and
12929 vscnt(0).
12930 - If OpenCL, omit lgkmcnt(0).
12931 - Could be split into
12932 separate s_waitcnt
12933 vmcnt(0), s_waitcnt
12934 vscnt(0) and s_waitcnt
12935 lgkmcnt(0) to allow
12936 them to be
12937 independently moved
12938 according to the
12939 following rules.
12940 - s_waitcnt vmcnt(0)
12941 must happen after
12942 any preceding
12943 global/generic load/load
12944 atomic/
12945 atomicrmw-with-return-value.
12946 - s_waitcnt vscnt(0)
12947 must happen after
12948 any preceding
12949 global/generic
12950 store/store
12951 atomic/
12952 atomicrmw-no-return-value.
12953 - s_waitcnt lgkmcnt(0)
12954 must happen after
12955 any preceding
12956 local/generic
12957 load/store/load
12958 atomic/store
12959 atomic/atomicrmw.
12960 - Must happen before
12961 the following
12962 atomicrmw.
12963 - Ensures that all
12964 memory operations
12965 have
12966 completed before
12967 performing the
12968 atomicrmw that is
12969 being released.
12971 2. buffer/global/flat_atomic
12972 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)
12974 - If CU wavefront execution
12975 mode, omit.
12976 - If OpenCL, omit.
12977 - Could be split into
12978 separate s_waitcnt
12979 vmcnt(0) and s_waitcnt
12980 vscnt(0) to allow
12981 them to be
12982 independently moved
12983 according to the
12984 following rules.
12985 - s_waitcnt vmcnt(0)
12986 must happen after
12987 any preceding
12988 global/generic load/load
12989 atomic/
12990 atomicrmw-with-return-value.
12991 - s_waitcnt vscnt(0)
12992 must happen after
12993 any preceding
12994 global/generic
12995 store/store atomic/
12996 atomicrmw-no-return-value.
12997 - Must happen before
12998 the following
12999 store.
13000 - Ensures that all
13001 global memory
13002 operations have
13003 completed before
13004 performing the
13005 store that is being
13006 released.
13008 2. ds_atomic
13009 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &
13010 - system - generic vmcnt(0) & vscnt(0)
13012 - If OpenCL, omit
13013 lgkmcnt(0).
13014 - Could be split into
13015 separate s_waitcnt
13016 vmcnt(0), s_waitcnt
13017 vscnt(0) and s_waitcnt
13018 lgkmcnt(0) to allow
13019 them to be
13020 independently moved
13021 according to the
13022 following rules.
13023 - s_waitcnt vmcnt(0)
13024 must happen after
13025 any preceding
13026 global/generic
13027 load/load atomic/
13028 atomicrmw-with-return-value.
13029 - s_waitcnt vscnt(0)
13030 must happen after
13031 any preceding
13032 global/generic
13033 store/store atomic/
13034 atomicrmw-no-return-value.
13035 - s_waitcnt lgkmcnt(0)
13036 must happen after
13037 any preceding
13038 local/generic
13039 load/store/load
13040 atomic/store
13041 atomic/atomicrmw.
13042 - Must happen before
13043 the following
13044 atomicrmw.
13045 - Ensures that all
13046 memory operations
13047 to global and local
13048 have completed
13049 before performing
13050 the atomicrmw that
13051 is being released.
13053 2. buffer/global/flat_atomic
13054 fence release - singlethread *none* *none*
13055 - wavefront
13056 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) &
13057 vmcnt(0) & vscnt(0)
13059 - If CU wavefront execution
13060 mode, omit vmcnt(0) and
13061 vscnt(0).
13062 - If OpenCL and
13063 address space is
13064 not generic, omit
13065 lgkmcnt(0).
13066 - If OpenCL and
13067 address space is
13068 local, omit
13069 vmcnt(0) and vscnt(0).
13070 - However, since LLVM
13071 currently has no
13072 address space on
13073 the fence need to
13074 conservatively
13075 always generate. If
13076 fence had an
13077 address space then
13078 set to address
13079 space of OpenCL
13080 fence flag, or to
13081 generic if both
13082 local and global
13083 flags are
13084 specified.
13085 - Could be split into
13086 separate s_waitcnt
13087 vmcnt(0), s_waitcnt
13088 vscnt(0) and s_waitcnt
13089 lgkmcnt(0) to allow
13090 them to be
13091 independently moved
13092 according to the
13093 following rules.
13094 - s_waitcnt vmcnt(0)
13095 must happen after
13096 any preceding
13097 global/generic
13098 load/load
13099 atomic/
13100 atomicrmw-with-return-value.
13101 - s_waitcnt vscnt(0)
13102 must happen after
13103 any preceding
13104 global/generic
13105 store/store atomic/
13106 atomicrmw-no-return-value.
13107 - s_waitcnt lgkmcnt(0)
13108 must happen after
13109 any preceding
13110 local/generic
13111 load/store/load
13112 atomic/store atomic/
13113 atomicrmw.
13114 - Must happen before
13115 any following store
13116 atomic/atomicrmw
13117 with an equal or
13118 wider sync scope
13119 and memory ordering
13120 stronger than
13121 unordered (this is
13122 termed the
13123 fence-paired-atomic).
13124 - Ensures that all
13125 memory operations
13126 have
13127 completed before
13128 performing the
13129 following
13130 fence-paired-atomic.
13132 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) &
13133 - system vmcnt(0) & vscnt(0)
13135 - If OpenCL and
13136 address space is
13137 not generic, omit
13138 lgkmcnt(0).
13139 - If OpenCL and
13140 address space is
13141 local, omit
13142 vmcnt(0) and vscnt(0).
13143 - However, since LLVM
13144 currently has no
13145 address space on
13146 the fence need to
13147 conservatively
13148 always generate. If
13149 fence had an
13150 address space then
13151 set to address
13152 space of OpenCL
13153 fence flag, or to
13154 generic if both
13155 local and global
13156 flags are
13157 specified.
13158 - Could be split into
13159 separate s_waitcnt
13160 vmcnt(0), s_waitcnt
13161 vscnt(0) and s_waitcnt
13162 lgkmcnt(0) to allow
13163 them to be
13164 independently moved
13165 according to the
13166 following rules.
13167 - s_waitcnt vmcnt(0)
13168 must happen after
13169 any preceding
13170 global/generic
13171 load/load atomic/
13172 atomicrmw-with-return-value.
13173 - s_waitcnt vscnt(0)
13174 must happen after
13175 any preceding
13176 global/generic
13177 store/store atomic/
13178 atomicrmw-no-return-value.
13179 - s_waitcnt lgkmcnt(0)
13180 must happen after
13181 any preceding
13182 local/generic
13183 load/store/load
13184 atomic/store
13185 atomic/atomicrmw.
13186 - Must happen before
13187 any following store
13188 atomic/atomicrmw
13189 with an equal or
13190 wider sync scope
13191 and memory ordering
13192 stronger than
13193 unordered (this is
13194 termed the
13195 fence-paired-atomic).
13196 - Ensures that all
13197 memory operations
13198 have
13199 completed before
13200 performing the
13201 following
13202 fence-paired-atomic.
13204 **Acquire-Release Atomic**
13205 ------------------------------------------------------------------------------------
13206 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic
13207 - wavefront - local
13208 - generic
13209 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) &
13210 vmcnt(0) & vscnt(0)
13212 - If CU wavefront execution
13213 mode, omit vmcnt(0) and
13214 vscnt(0).
13215 - If OpenCL, omit
13216 lgkmcnt(0).
13217 - Must happen after
13218 any preceding
13219 local/generic
13220 load/store/load
13221 atomic/store
13222 atomic/atomicrmw.
13223 - Could be split into
13224 separate s_waitcnt
13225 vmcnt(0), s_waitcnt
13226 vscnt(0), and s_waitcnt
13227 lgkmcnt(0) to allow
13228 them to be
13229 independently moved
13230 according to the
13231 following rules.
13232 - s_waitcnt vmcnt(0)
13233 must happen after
13234 any preceding
13235 global/generic load/load
13236 atomic/
13237 atomicrmw-with-return-value.
13238 - s_waitcnt vscnt(0)
13239 must happen after
13240 any preceding
13241 global/generic
13242 store/store
13243 atomic/
13244 atomicrmw-no-return-value.
13245 - s_waitcnt lgkmcnt(0)
13246 must happen after
13247 any preceding
13248 local/generic
13249 load/store/load
13250 atomic/store
13251 atomic/atomicrmw.
13252 - Must happen before
13253 the following
13254 atomicrmw.
13255 - Ensures that all
13256 memory operations
13257 have
13258 completed before
13259 performing the
13260 atomicrmw that is
13261 being released.
13263 2. buffer/global_atomic
13264 3. s_waitcnt vm/vscnt(0)
13266 - If CU wavefront execution
13267 mode, omit.
13268 - Use vmcnt(0) if atomic with
13269 return and vscnt(0) if
13270 atomic with no-return.
13271 - Must happen before
13272 the following
13273 buffer_gl0_inv.
13274 - Ensures any
13275 following global
13276 data read is no
13277 older than the
13278 atomicrmw value
13279 being acquired.
13281 4. buffer_gl0_inv
13283 - If CU wavefront execution
13284 mode, omit.
13285 - Ensures that
13286 following
13287 loads will not see
13288 stale data.
13290 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)
13292 - If CU wavefront execution
13293 mode, omit.
13294 - If OpenCL, omit.
13295 - Could be split into
13296 separate s_waitcnt
13297 vmcnt(0) and s_waitcnt
13298 vscnt(0) to allow
13299 them to be
13300 independently moved
13301 according to the
13302 following rules.
13303 - s_waitcnt vmcnt(0)
13304 must happen after
13305 any preceding
13306 global/generic load/load
13307 atomic/
13308 atomicrmw-with-return-value.
13309 - s_waitcnt vscnt(0)
13310 must happen after
13311 any preceding
13312 global/generic
13313 store/store atomic/
13314 atomicrmw-no-return-value.
13315 - Must happen before
13316 the following
13317 store.
13318 - Ensures that all
13319 global memory
13320 operations have
13321 completed before
13322 performing the
13323 store that is being
13324 released.
13326 2. ds_atomic
13327 3. s_waitcnt lgkmcnt(0)
13329 - If OpenCL, omit.
13330 - Must happen before
13331 the following
13332 buffer_gl0_inv.
13333 - Ensures any
13334 following global
13335 data read is no
13336 older than the local load
13337 atomic value being
13338 acquired.
13340 4. buffer_gl0_inv
13342 - If CU wavefront execution
13343 mode, omit.
13344 - If OpenCL omit.
13345 - Ensures that
13346 following
13347 loads will not see
13348 stale data.
13350 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) &
13351 vmcnt(0) & vscnt(0)
13353 - If CU wavefront execution
13354 mode, omit vmcnt(0) and
13355 vscnt(0).
13356 - If OpenCL, omit lgkmcnt(0).
13357 - Could be split into
13358 separate s_waitcnt
13359 vmcnt(0), s_waitcnt
13360 vscnt(0) and s_waitcnt
13361 lgkmcnt(0) to allow
13362 them to be
13363 independently moved
13364 according to the
13365 following rules.
13366 - s_waitcnt vmcnt(0)
13367 must happen after
13368 any preceding
13369 global/generic load/load
13370 atomic/
13371 atomicrmw-with-return-value.
13372 - s_waitcnt vscnt(0)
13373 must happen after
13374 any preceding
13375 global/generic
13376 store/store
13377 atomic/
13378 atomicrmw-no-return-value.
13379 - s_waitcnt lgkmcnt(0)
13380 must happen after
13381 any preceding
13382 local/generic
13383 load/store/load
13384 atomic/store
13385 atomic/atomicrmw.
13386 - Must happen before
13387 the following
13388 atomicrmw.
13389 - Ensures that all
13390 memory operations
13391 have
13392 completed before
13393 performing the
13394 atomicrmw that is
13395 being released.
13397 2. flat_atomic
13398 3. s_waitcnt lgkmcnt(0) &
13399 vmcnt(0) & vscnt(0)
13401 - If CU wavefront execution
13402 mode, omit vmcnt(0) and
13403 vscnt(0).
13404 - If OpenCL, omit lgkmcnt(0).
13405 - Must happen before
13406 the following
13407 buffer_gl0_inv.
13408 - Ensures any
13409 following global
13410 data read is no
13411 older than the load
13412 atomic value being
13413 acquired.
13415 3. buffer_gl0_inv
13417 - If CU wavefront execution
13418 mode, omit.
13419 - Ensures that
13420 following
13421 loads will not see
13422 stale data.
13424 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &
13425 - system vmcnt(0) & vscnt(0)
13427 - If OpenCL, omit
13428 lgkmcnt(0).
13429 - Could be split into
13430 separate s_waitcnt
13431 vmcnt(0), s_waitcnt
13432 vscnt(0) and s_waitcnt
13433 lgkmcnt(0) to allow
13434 them to be
13435 independently moved
13436 according to the
13437 following rules.
13438 - s_waitcnt vmcnt(0)
13439 must happen after
13440 any preceding
13441 global/generic
13442 load/load atomic/
13443 atomicrmw-with-return-value.
13444 - s_waitcnt vscnt(0)
13445 must happen after
13446 any preceding
13447 global/generic
13448 store/store atomic/
13449 atomicrmw-no-return-value.
13450 - s_waitcnt lgkmcnt(0)
13451 must happen after
13452 any preceding
13453 local/generic
13454 load/store/load
13455 atomic/store
13456 atomic/atomicrmw.
13457 - Must happen before
13458 the following
13459 atomicrmw.
13460 - Ensures that all
13461 memory operations
13462 to global have
13463 completed before
13464 performing the
13465 atomicrmw that is
13466 being released.
13468 2. buffer/global_atomic
13469 3. s_waitcnt vm/vscnt(0)
13471 - Use vmcnt(0) if atomic with
13472 return and vscnt(0) if
13473 atomic with no-return.
13474 - Must happen before
13475 following
13476 buffer_gl*_inv.
13477 - Ensures the
13478 atomicrmw has
13479 completed before
13480 invalidating the
13481 caches.
13483 4. buffer_gl1_inv;
13484 buffer_gl0_inv
13486 - Must happen before
13487 any following
13488 global/generic
13489 load/load
13490 atomic/atomicrmw.
13491 - Ensures that
13492 following loads
13493 will not see stale
13494 global data.
13496 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &
13497 - system vmcnt(0) & vscnt(0)
13499 - If OpenCL, omit
13500 lgkmcnt(0).
13501 - Could be split into
13502 separate s_waitcnt
13503 vmcnt(0), s_waitcnt
13504 vscnt(0), and s_waitcnt
13505 lgkmcnt(0) to allow
13506 them to be
13507 independently moved
13508 according to the
13509 following rules.
13510 - s_waitcnt vmcnt(0)
13511 must happen after
13512 any preceding
13513 global/generic
13514 load/load atomic
13515 atomicrmw-with-return-value.
13516 - s_waitcnt vscnt(0)
13517 must happen after
13518 any preceding
13519 global/generic
13520 store/store atomic/
13521 atomicrmw-no-return-value.
13522 - s_waitcnt lgkmcnt(0)
13523 must happen after
13524 any preceding
13525 local/generic
13526 load/store/load
13527 atomic/store
13528 atomic/atomicrmw.
13529 - Must happen before
13530 the following
13531 atomicrmw.
13532 - Ensures that all
13533 memory operations
13534 have
13535 completed before
13536 performing the
13537 atomicrmw that is
13538 being released.
13540 2. flat_atomic
13541 3. s_waitcnt vm/vscnt(0) &
13542 lgkmcnt(0)
13544 - If OpenCL, omit
13545 lgkmcnt(0).
13546 - Use vmcnt(0) if atomic with
13547 return and vscnt(0) if
13548 atomic with no-return.
13549 - Must happen before
13550 following
13551 buffer_gl*_inv.
13552 - Ensures the
13553 atomicrmw has
13554 completed before
13555 invalidating the
13556 caches.
13558 4. buffer_gl1_inv;
13559 buffer_gl0_inv
13561 - Must happen before
13562 any following
13563 global/generic
13564 load/load
13565 atomic/atomicrmw.
13566 - Ensures that
13567 following loads
13568 will not see stale
13569 global data.
13571 fence acq_rel - singlethread *none* *none*
13572 - wavefront
13573 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) &
13574 vmcnt(0) & vscnt(0)
13576 - If CU wavefront execution
13577 mode, omit vmcnt(0) and
13578 vscnt(0).
13579 - If OpenCL and
13580 address space is
13581 not generic, omit
13582 lgkmcnt(0).
13583 - If OpenCL and
13584 address space is
13585 local, omit
13586 vmcnt(0) and vscnt(0).
13587 - However,
13588 since LLVM
13589 currently has no
13590 address space on
13591 the fence need to
13592 conservatively
13593 always generate
13594 (see comment for
13595 previous fence).
13596 - Could be split into
13597 separate s_waitcnt
13598 vmcnt(0), s_waitcnt
13599 vscnt(0) and s_waitcnt
13600 lgkmcnt(0) to allow
13601 them to be
13602 independently moved
13603 according to the
13604 following rules.
13605 - s_waitcnt vmcnt(0)
13606 must happen after
13607 any preceding
13608 global/generic
13609 load/load
13610 atomic/
13611 atomicrmw-with-return-value.
13612 - s_waitcnt vscnt(0)
13613 must happen after
13614 any preceding
13615 global/generic
13616 store/store atomic/
13617 atomicrmw-no-return-value.
13618 - s_waitcnt lgkmcnt(0)
13619 must happen after
13620 any preceding
13621 local/generic
13622 load/store/load
13623 atomic/store atomic/
13624 atomicrmw.
13625 - Must happen before
13626 any following
13627 global/generic
13628 load/load
13629 atomic/store/store
13630 atomic/atomicrmw.
13631 - Ensures that all
13632 memory operations
13633 have
13634 completed before
13635 performing any
13636 following global
13637 memory operations.
13638 - Ensures that the
13639 preceding
13640 local/generic load
13641 atomic/atomicrmw
13642 with an equal or
13643 wider sync scope
13644 and memory ordering
13645 stronger than
13646 unordered (this is
13647 termed the
13648 acquire-fence-paired-atomic)
13649 has completed
13650 before following
13651 global memory
13652 operations. This
13653 satisfies the
13654 requirements of
13655 acquire.
13656 - Ensures that all
13657 previous memory
13658 operations have
13659 completed before a
13660 following
13661 local/generic store
13662 atomic/atomicrmw
13663 with an equal or
13664 wider sync scope
13665 and memory ordering
13666 stronger than
13667 unordered (this is
13668 termed the
13669 release-fence-paired-atomic).
13670 This satisfies the
13671 requirements of
13672 release.
13673 - Must happen before
13674 the following
13675 buffer_gl0_inv.
13676 - Ensures that the
13677 acquire-fence-paired
13678 atomic has completed
13679 before invalidating
13680 the
13681 cache. Therefore
13682 any following
13683 locations read must
13684 be no older than
13685 the value read by
13686 the
13687 acquire-fence-paired-atomic.
13689 3. buffer_gl0_inv
13691 - If CU wavefront execution
13692 mode, omit.
13693 - Ensures that
13694 following
13695 loads will not see
13696 stale data.
13698 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) &
13699 - system vmcnt(0) & vscnt(0)
13701 - If OpenCL and
13702 address space is
13703 not generic, omit
13704 lgkmcnt(0).
13705 - If OpenCL and
13706 address space is
13707 local, omit
13708 vmcnt(0) and vscnt(0).
13709 - However, since LLVM
13710 currently has no
13711 address space on
13712 the fence need to
13713 conservatively
13714 always generate
13715 (see comment for
13716 previous fence).
13717 - Could be split into
13718 separate s_waitcnt
13719 vmcnt(0), s_waitcnt
13720 vscnt(0) and s_waitcnt
13721 lgkmcnt(0) to allow
13722 them to be
13723 independently moved
13724 according to the
13725 following rules.
13726 - s_waitcnt vmcnt(0)
13727 must happen after
13728 any preceding
13729 global/generic
13730 load/load
13731 atomic/
13732 atomicrmw-with-return-value.
13733 - s_waitcnt vscnt(0)
13734 must happen after
13735 any preceding
13736 global/generic
13737 store/store atomic/
13738 atomicrmw-no-return-value.
13739 - s_waitcnt lgkmcnt(0)
13740 must happen after
13741 any preceding
13742 local/generic
13743 load/store/load
13744 atomic/store
13745 atomic/atomicrmw.
13746 - Must happen before
13747 the following
13748 buffer_gl*_inv.
13749 - Ensures that the
13750 preceding
13751 global/local/generic
13752 load
13753 atomic/atomicrmw
13754 with an equal or
13755 wider sync scope
13756 and memory ordering
13757 stronger than
13758 unordered (this is
13759 termed the
13760 acquire-fence-paired-atomic)
13761 has completed
13762 before invalidating
13763 the caches. This
13764 satisfies the
13765 requirements of
13766 acquire.
13767 - Ensures that all
13768 previous memory
13769 operations have
13770 completed before a
13771 following
13772 global/local/generic
13773 store
13774 atomic/atomicrmw
13775 with an equal or
13776 wider sync scope
13777 and memory ordering
13778 stronger than
13779 unordered (this is
13780 termed the
13781 release-fence-paired-atomic).
13782 This satisfies the
13783 requirements of
13784 release.
13786 2. buffer_gl1_inv;
13787 buffer_gl0_inv
13789 - Must happen before
13790 any following
13791 global/generic
13792 load/load
13793 atomic/store/store
13794 atomic/atomicrmw.
13795 - Ensures that
13796 following loads
13797 will not see stale
13798 global data. This
13799 satisfies the
13800 requirements of
13801 acquire.
13803 **Sequential Consistent Atomic**
13804 ------------------------------------------------------------------------------------
13805 load atomic seq_cst - singlethread - global *Same as corresponding
13806 - wavefront - local load atomic acquire,
13807 - generic except must generate
13808 all instructions even
13809 for OpenCL.*
13810 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) &
13811 - generic vmcnt(0) & vscnt(0)
13813 - If CU wavefront execution
13814 mode, omit vmcnt(0) and
13815 vscnt(0).
13816 - Could be split into
13817 separate s_waitcnt
13818 vmcnt(0), s_waitcnt
13819 vscnt(0), and s_waitcnt
13820 lgkmcnt(0) to allow
13821 them to be
13822 independently moved
13823 according to the
13824 following rules.
13825 - s_waitcnt lgkmcnt(0) must
13826 happen after
13827 preceding
13828 local/generic load
13829 atomic/store
13830 atomic/atomicrmw
13831 with memory
13832 ordering of seq_cst
13833 and with equal or
13834 wider sync scope.
13835 (Note that seq_cst
13836 fences have their
13837 own s_waitcnt
13838 lgkmcnt(0) and so do
13839 not need to be
13840 considered.)
13841 - s_waitcnt vmcnt(0)
13842 must happen after
13843 preceding
13844 global/generic load
13845 atomic/
13846 atomicrmw-with-return-value
13847 with memory
13848 ordering of seq_cst
13849 and with equal or
13850 wider sync scope.
13851 (Note that seq_cst
13852 fences have their
13853 own s_waitcnt
13854 vmcnt(0) and so do
13855 not need to be
13856 considered.)
13857 - s_waitcnt vscnt(0)
13858 Must happen after
13859 preceding
13860 global/generic store
13861 atomic/
13862 atomicrmw-no-return-value
13863 with memory
13864 ordering of seq_cst
13865 and with equal or
13866 wider sync scope.
13867 (Note that seq_cst
13868 fences have their
13869 own s_waitcnt
13870 vscnt(0) and so do
13871 not need to be
13872 considered.)
13873 - Ensures any
13874 preceding
13875 sequential
13876 consistent global/local
13877 memory instructions
13878 have completed
13879 before executing
13880 this sequentially
13881 consistent
13882 instruction. This
13883 prevents reordering
13884 a seq_cst store
13885 followed by a
13886 seq_cst load. (Note
13887 that seq_cst is
13888 stronger than
13889 acquire/release as
13890 the reordering of
13891 load acquire
13892 followed by a store
13893 release is
13894 prevented by the
13895 s_waitcnt of
13896 the release, but
13897 there is nothing
13898 preventing a store
13899 release followed by
13900 load acquire from
13901 completing out of
13902 order. The s_waitcnt
13903 could be placed after
13904 seq_store or before
13905 the seq_load. We
13906 choose the load to
13907 make the s_waitcnt be
13908 as late as possible
13909 so that the store
13910 may have already
13911 completed.)
13913 2. *Following
13914 instructions same as
13915 corresponding load
13916 atomic acquire,
13917 except must generate
13918 all instructions even
13919 for OpenCL.*
13920 load atomic seq_cst - workgroup - local
13922 1. s_waitcnt vmcnt(0) & vscnt(0)
13924 - If CU wavefront execution
13925 mode, omit.
13926 - Could be split into
13927 separate s_waitcnt
13928 vmcnt(0) and s_waitcnt
13929 vscnt(0) to allow
13930 them to be
13931 independently moved
13932 according to the
13933 following rules.
13934 - s_waitcnt vmcnt(0)
13935 Must happen after
13936 preceding
13937 global/generic load
13938 atomic/
13939 atomicrmw-with-return-value
13940 with memory
13941 ordering of seq_cst
13942 and with equal or
13943 wider sync scope.
13944 (Note that seq_cst
13945 fences have their
13946 own s_waitcnt
13947 vmcnt(0) and so do
13948 not need to be
13949 considered.)
13950 - s_waitcnt vscnt(0)
13951 Must happen after
13952 preceding
13953 global/generic store
13954 atomic/
13955 atomicrmw-no-return-value
13956 with memory
13957 ordering of seq_cst
13958 and with equal or
13959 wider sync scope.
13960 (Note that seq_cst
13961 fences have their
13962 own s_waitcnt
13963 vscnt(0) and so do
13964 not need to be
13965 considered.)
13966 - Ensures any
13967 preceding
13968 sequential
13969 consistent global
13970 memory instructions
13971 have completed
13972 before executing
13973 this sequentially
13974 consistent
13975 instruction. This
13976 prevents reordering
13977 a seq_cst store
13978 followed by a
13979 seq_cst load. (Note
13980 that seq_cst is
13981 stronger than
13982 acquire/release as
13983 the reordering of
13984 load acquire
13985 followed by a store
13986 release is
13987 prevented by the
13988 s_waitcnt of
13989 the release, but
13990 there is nothing
13991 preventing a store
13992 release followed by
13993 load acquire from
13994 completing out of
13995 order. The s_waitcnt
13996 could be placed after
13997 seq_store or before
13998 the seq_load. We
13999 choose the load to
14000 make the s_waitcnt be
14001 as late as possible
14002 so that the store
14003 may have already
14004 completed.)
14006 2. *Following
14007 instructions same as
14008 corresponding load
14009 atomic acquire,
14010 except must generate
14011 all instructions even
14012 for OpenCL.*
14014 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &
14015 - system - generic vmcnt(0) & vscnt(0)
14017 - Could be split into
14018 separate s_waitcnt
14019 vmcnt(0), s_waitcnt
14020 vscnt(0) and s_waitcnt
14021 lgkmcnt(0) to allow
14022 them to be
14023 independently moved
14024 according to the
14025 following rules.
14026 - s_waitcnt lgkmcnt(0)
14027 must happen after
14028 preceding
14029 local load
14030 atomic/store
14031 atomic/atomicrmw
14032 with memory
14033 ordering of seq_cst
14034 and with equal or
14035 wider sync scope.
14036 (Note that seq_cst
14037 fences have their
14038 own s_waitcnt
14039 lgkmcnt(0) and so do
14040 not need to be
14041 considered.)
14042 - s_waitcnt vmcnt(0)
14043 must happen after
14044 preceding
14045 global/generic load
14046 atomic/
14047 atomicrmw-with-return-value
14048 with memory
14049 ordering of seq_cst
14050 and with equal or
14051 wider sync scope.
14052 (Note that seq_cst
14053 fences have their
14054 own s_waitcnt
14055 vmcnt(0) and so do
14056 not need to be
14057 considered.)
14058 - s_waitcnt vscnt(0)
14059 Must happen after
14060 preceding
14061 global/generic store
14062 atomic/
14063 atomicrmw-no-return-value
14064 with memory
14065 ordering of seq_cst
14066 and with equal or
14067 wider sync scope.
14068 (Note that seq_cst
14069 fences have their
14070 own s_waitcnt
14071 vscnt(0) and so do
14072 not need to be
14073 considered.)
14074 - Ensures any
14075 preceding
14076 sequential
14077 consistent global
14078 memory instructions
14079 have completed
14080 before executing
14081 this sequentially
14082 consistent
14083 instruction. This
14084 prevents reordering
14085 a seq_cst store
14086 followed by a
14087 seq_cst load. (Note
14088 that seq_cst is
14089 stronger than
14090 acquire/release as
14091 the reordering of
14092 load acquire
14093 followed by a store
14094 release is
14095 prevented by the
14096 s_waitcnt of
14097 the release, but
14098 there is nothing
14099 preventing a store
14100 release followed by
14101 load acquire from
14102 completing out of
14103 order. The s_waitcnt
14104 could be placed after
14105 seq_store or before
14106 the seq_load. We
14107 choose the load to
14108 make the s_waitcnt be
14109 as late as possible
14110 so that the store
14111 may have already
14112 completed.)
14114 2. *Following
14115 instructions same as
14116 corresponding load
14117 atomic acquire,
14118 except must generate
14119 all instructions even
14120 for OpenCL.*
14121 store atomic seq_cst - singlethread - global *Same as corresponding
14122 - wavefront - local store atomic release,
14123 - workgroup - generic except must generate
14124 - agent all instructions even
14125 - system for OpenCL.*
14126 atomicrmw seq_cst - singlethread - global *Same as corresponding
14127 - wavefront - local atomicrmw acq_rel,
14128 - workgroup - generic except must generate
14129 - agent all instructions even
14130 - system for OpenCL.*
14131 fence seq_cst - singlethread *none* *Same as corresponding
14132 - wavefront fence acq_rel,
14133 - workgroup except must generate
14134 - agent all instructions even
14135 - system for OpenCL.*
14136 ============ ============ ============== ========== ================================
14138 .. _amdgpu-amdhsa-trap-handler-abi:
14140 Trap Handler ABI
14141 ~~~~~~~~~~~~~~~~
14143 For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible
14144 runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that
14145 supports the ``s_trap`` instruction. For usage see:
14147 - :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table`
14148 - :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table`
14149 - :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table`
14151 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2
14152 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table
14154 =================== =============== =============== =======================================
14155 Usage Code Sequence Trap Handler Description
14156 Inputs
14157 =================== =============== =============== =======================================
14158 reserved ``s_trap 0x00`` Reserved by hardware.
14159 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap``
14160 ``queue_ptr`` intrinsic (not implemented).
14161 ``VGPR0``:
14162 ``arg``
14163 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at
14164 ``queue_ptr`` the trap instruction. The associated
14165 queue is signalled to put it into the
14166 error state. When the queue is put in
14167 the error state, the waves executing
14168 dispatches on the queue will be
14169 terminated.
14170 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves
14171 as a no-operation. The trap handler
14172 is entered and immediately returns to
14173 continue execution of the wavefront.
14174 - If the debugger is enabled, causes
14175 the debug trap to be reported by the
14176 debugger and the wavefront is put in
14177 the halt state with the PC at the
14178 instruction. The debugger must
14179 increment the PC and resume the wave.
14180 reserved ``s_trap 0x04`` Reserved.
14181 reserved ``s_trap 0x05`` Reserved.
14182 reserved ``s_trap 0x06`` Reserved.
14183 reserved ``s_trap 0x07`` Reserved.
14184 reserved ``s_trap 0x08`` Reserved.
14185 reserved ``s_trap 0xfe`` Reserved.
14186 reserved ``s_trap 0xff`` Reserved.
14187 =================== =============== =============== =======================================
14189 ..
14191 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3
14192 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table
14194 =================== =============== =============== =======================================
14195 Usage Code Sequence Trap Handler Description
14196 Inputs
14197 =================== =============== =============== =======================================
14198 reserved ``s_trap 0x00`` Reserved by hardware.
14199 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for
14200 breakpoints. Causes wave to be halted
14201 with the PC at the trap instruction.
14202 The debugger is responsible to resume
14203 the wave, including the instruction
14204 that the breakpoint overwrote.
14205 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at
14206 ``queue_ptr`` the trap instruction. The associated
14207 queue is signalled to put it into the
14208 error state. When the queue is put in
14209 the error state, the waves executing
14210 dispatches on the queue will be
14211 terminated.
14212 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves
14213 as a no-operation. The trap handler
14214 is entered and immediately returns to
14215 continue execution of the wavefront.
14216 - If the debugger is enabled, causes
14217 the debug trap to be reported by the
14218 debugger and the wavefront is put in
14219 the halt state with the PC at the
14220 instruction. The debugger must
14221 increment the PC and resume the wave.
14222 reserved ``s_trap 0x04`` Reserved.
14223 reserved ``s_trap 0x05`` Reserved.
14224 reserved ``s_trap 0x06`` Reserved.
14225 reserved ``s_trap 0x07`` Reserved.
14226 reserved ``s_trap 0x08`` Reserved.
14227 reserved ``s_trap 0xfe`` Reserved.
14228 reserved ``s_trap 0xff`` Reserved.
14229 =================== =============== =============== =======================================
14231 ..
14233 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above
14234 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table
14236 =================== =============== ================ ================= =======================================
14237 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX11 Inputs Description
14238 =================== =============== ================ ================= =======================================
14239 reserved ``s_trap 0x00`` Reserved by hardware.
14240 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for
14241 breakpoints. Causes wave to be halted
14242 with the PC at the trap instruction.
14243 The debugger is responsible to resume
14244 the wave, including the instruction
14245 that the breakpoint overwrote.
14246 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at
14247 ``queue_ptr`` the trap instruction. The associated
14248 queue is signalled to put it into the
14249 error state. When the queue is put in
14250 the error state, the waves executing
14251 dispatches on the queue will be
14252 terminated.
14253 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves
14254 as a no-operation. The trap handler
14255 is entered and immediately returns to
14256 continue execution of the wavefront.
14257 - If the debugger is enabled, causes
14258 the debug trap to be reported by the
14259 debugger and the wavefront is put in
14260 the halt state with the PC at the
14261 instruction. The debugger must
14262 increment the PC and resume the wave.
14263 reserved ``s_trap 0x04`` Reserved.
14264 reserved ``s_trap 0x05`` Reserved.
14265 reserved ``s_trap 0x06`` Reserved.
14266 reserved ``s_trap 0x07`` Reserved.
14267 reserved ``s_trap 0x08`` Reserved.
14268 reserved ``s_trap 0xfe`` Reserved.
14269 reserved ``s_trap 0xff`` Reserved.
14270 =================== =============== ================ ================= =======================================
14272 .. _amdgpu-amdhsa-function-call-convention:
14274 Call Convention
14275 ~~~~~~~~~~~~~~~
14277 .. note::
14279 This section is currently incomplete and has inaccuracies. It is WIP that will
14280 be updated as information is determined.
14282 See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled
14283 addresses. Unswizzled addresses are normal linear addresses.
14285 .. _amdgpu-amdhsa-function-call-convention-kernel-functions:
14287 Kernel Functions
14288 ++++++++++++++++
14290 This section describes the call convention ABI for the outer kernel function.
14292 See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call
14293 convention.
14295 The following is not part of the AMDGPU kernel calling convention but describes
14296 how the AMDGPU implements function calls:
14298 1. Clang decides the kernarg layout to match the *HSA Programmer's Language
14299 Reference* [HSA]_.
14301 - All structs are passed directly.
14302 - Lambda values are passed *TBA*.
14304 .. TODO::
14306 - Does this really follow HSA rules? Or are structs >16 bytes passed
14307 by-value struct?
14308 - What is ABI for lambda values?
14310 4. The kernel performs certain setup in its prolog, as described in
14311 :ref:`amdgpu-amdhsa-kernel-prolog`.
14313 .. _amdgpu-amdhsa-function-call-convention-non-kernel-functions:
14315 Non-Kernel Functions
14316 ++++++++++++++++++++
14318 This section describes the call convention ABI for functions other than the
14319 outer kernel function.
14321 If a kernel has function calls then scratch is always allocated and used for
14322 the call stack which grows from low address to high address using the swizzled
14323 scratch address space.
14325 On entry to a function:
14327 1. SGPR0-3 contain a V# with the following properties (see
14328 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`):
14330 * Base address pointing to the beginning of the wavefront scratch backing
14331 memory.
14332 * Swizzled with dword element size and stride of wavefront size elements.
14334 2. The FLAT_SCRATCH register pair is setup. See
14335 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.
14336 3. GFX6-GFX8: M0 register set to the size of LDS in bytes. See
14337 :ref:`amdgpu-amdhsa-kernel-prolog-m0`.
14338 4. The EXEC register is set to the lanes active on entry to the function.
14339 5. MODE register: *TBD*
14340 6. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described
14341 below.
14342 7. SGPR30-31 return address (RA). The code address that the function must
14343 return to when it completes. The value is undefined if the function is *no
14344 return*.
14345 8. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch
14346 offset relative to the beginning of the wavefront scratch backing memory.
14348 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR
14349 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled
14350 manner.
14352 The unswizzled SP value can be converted into the swizzled SP value by:
14354 | swizzled SP = unswizzled SP / wavefront size
14356 This may be used to obtain the private address space address of stack
14357 objects and to convert this address to a flat address by adding the flat
14358 scratch aperture base address.
14360 The swizzled SP value is always 4 bytes aligned for the ``r600``
14361 architecture and 16 byte aligned for the ``amdgcn`` architecture.
14363 .. note::
14365 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the
14366 OpenCL language which has the largest base type defined as 16 bytes.
14368 On entry, the swizzled SP value is the address of the first function
14369 argument passed on the stack. Other stack passed arguments are positive
14370 offsets from the entry swizzled SP value.
14372 The function may use positive offsets beyond the last stack passed argument
14373 for stack allocated local variables and register spill slots. If necessary,
14374 the function may align these to greater alignment than 16 bytes. After these
14375 the function may dynamically allocate space for such things as runtime sized
14376 ``alloca`` local allocations.
14378 If the function calls another function, it will place any stack allocated
14379 arguments after the last local allocation and adjust SGPR32 to the address
14380 after the last local allocation.
14382 9. All other registers are unspecified.
14383 10. Any necessary ``s_waitcnt`` has been performed to ensure memory is available
14384 to the function.
14385 11. Use pass-by-reference (byref) in stead of pass-by-value (byval) for struct
14386 arguments in C ABI. Callee is responsible for allocating stack memory and
14387 copying the value of the struct if modified. Note that the backend still
14388 supports byval for struct arguments.
14390 On exit from a function:
14392 1. VGPR0-31 and SGPR4-29 are used to pass function result arguments as
14393 described below. Any registers used are considered clobbered registers.
14394 2. The following registers are preserved and have the same value as on entry:
14396 * FLAT_SCRATCH
14397 * EXEC
14398 * GFX6-GFX8: M0
14399 * All SGPR registers except the clobbered registers of SGPR4-31.
14400 * VGPR40-47
14401 * VGPR56-63
14402 * VGPR72-79
14403 * VGPR88-95
14404 * VGPR104-111
14405 * VGPR120-127
14406 * VGPR136-143
14407 * VGPR152-159
14408 * VGPR168-175
14409 * VGPR184-191
14410 * VGPR200-207
14411 * VGPR216-223
14412 * VGPR232-239
14413 * VGPR248-255
14415 .. note::
14417 Except the argument registers, the VGPRs clobbered and the preserved
14418 registers are intermixed at regular intervals in order to keep a
14419 similar ratio independent of the number of allocated VGPRs.
14421 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31.
14422 * Lanes of all VGPRs that are inactive at the call site.
14424 For the AMDGPU backend, an inter-procedural register allocation (IPRA)
14425 optimization may mark some of clobbered SGPR and VGPR registers as
14426 preserved if it can be determined that the called function does not change
14427 their value.
14429 2. The PC is set to the RA provided on entry.
14430 3. MODE register: *TBD*.
14431 4. All other registers are clobbered.
14432 5. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by
14433 function is available to the caller.
14435 .. TODO::
14437 - How are function results returned? The address of structured types is passed
14438 by reference, but what about other types?
14440 The function input arguments are made up of the formal arguments explicitly
14441 declared by the source language function plus the implicit input arguments used
14442 by the implementation.
14444 The source language input arguments are:
14446 1. Any source language implicit ``this`` or ``self`` argument comes first as a
14447 pointer type.
14448 2. Followed by the function formal arguments in left to right source order.
14450 The source language result arguments are:
14452 1. The function result argument.
14454 The source language input or result struct type arguments that are less than or
14455 equal to 16 bytes, are decomposed recursively into their base type fields, and
14456 each field is passed as if a separate argument. For input arguments, if the
14457 called function requires the struct to be in memory, for example because its
14458 address is taken, then the function body is responsible for allocating a stack
14459 location and copying the field arguments into it. Clang terms this *direct
14460 struct*.
14462 The source language input struct type arguments that are greater than 16 bytes,
14463 are passed by reference. The caller is responsible for allocating a stack
14464 location to make a copy of the struct value and pass the address as the input
14465 argument. The called function is responsible to perform the dereference when
14466 accessing the input argument. Clang terms this *by-value struct*.
14468 A source language result struct type argument that is greater than 16 bytes, is
14469 returned by reference. The caller is responsible for allocating a stack location
14470 to hold the result value and passes the address as the last input argument
14471 (before the implicit input arguments). In this case there are no result
14472 arguments. The called function is responsible to perform the dereference when
14473 storing the result value. Clang terms this *structured return (sret)*.
14475 *TODO: correct the ``sret`` definition.*
14477 .. TODO::
14479 Is this definition correct? Or is ``sret`` only used if passing in registers, and
14480 pass as non-decomposed struct as stack argument? Or something else? Is the
14481 memory location in the caller stack frame, or a stack memory argument and so
14482 no address is passed as the caller can directly write to the argument stack
14483 location? But then the stack location is still live after return. If an
14484 argument stack location is it the first stack argument or the last one?
14486 Lambda argument types are treated as struct types with an implementation defined
14487 set of fields.
14489 .. TODO::
14491 Need to specify the ABI for lambda types for AMDGPU.
14493 For AMDGPU backend all source language arguments (including the decomposed
14494 struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case
14495 they are passed in SGPRs.
14497 The AMDGPU backend walks the function call graph from the leaves to determine
14498 which implicit input arguments are used, propagating to each caller of the
14499 function. The used implicit arguments are appended to the function arguments
14500 after the source language arguments in the following order:
14502 .. TODO::
14504 Is recursion or external functions supported?
14506 1. Work-Item ID (1 VGPR)
14508 The X, Y and Z work-item ID are packed into a single VGRP with the following
14509 layout. Only fields actually used by the function are set. The other bits
14510 are undefined.
14512 The values come from the initial kernel execution state. See
14513 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`.
14515 .. table:: Work-item implicit argument layout
14516 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table
14518 ======= ======= ==============
14519 Bits Size Field Name
14520 ======= ======= ==============
14521 9:0 10 bits X Work-Item ID
14522 19:10 10 bits Y Work-Item ID
14523 29:20 10 bits Z Work-Item ID
14524 31:30 2 bits Unused
14525 ======= ======= ==============
14527 2. Dispatch Ptr (2 SGPRs)
14529 The value comes from the initial kernel execution state. See
14530 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.
14532 3. Queue Ptr (2 SGPRs)
14534 The value comes from the initial kernel execution state. See
14535 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.
14537 4. Kernarg Segment Ptr (2 SGPRs)
14539 The value comes from the initial kernel execution state. See
14540 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.
14542 5. Dispatch id (2 SGPRs)
14544 The value comes from the initial kernel execution state. See
14545 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.
14547 6. Work-Group ID X (1 SGPR)
14549 The value comes from the initial kernel execution state. See
14550 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.
14552 7. Work-Group ID Y (1 SGPR)
14554 The value comes from the initial kernel execution state. See
14555 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.
14557 8. Work-Group ID Z (1 SGPR)
14559 The value comes from the initial kernel execution state. See
14560 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.
14562 9. Implicit Argument Ptr (2 SGPRs)
14564 The value is computed by adding an offset to Kernarg Segment Ptr to get the
14565 global address space pointer to the first kernarg implicit argument.
14567 The input and result arguments are assigned in order in the following manner:
14569 .. note::
14571 There are likely some errors and omissions in the following description that
14572 need correction.
14574 .. TODO::
14576 Check the Clang source code to decipher how function arguments and return
14577 results are handled. Also see the AMDGPU specific values used.
14579 * VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to
14580 VGPR31.
14582 If there are more arguments than will fit in these registers, the remaining
14583 arguments are allocated on the stack in order on naturally aligned
14584 addresses.
14586 .. TODO::
14588 How are overly aligned structures allocated on the stack?
14590 * SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to
14591 SGPR29.
14593 If there are more arguments than will fit in these registers, the remaining
14594 arguments are allocated on the stack in order on naturally aligned
14595 addresses.
14597 Note that decomposed struct type arguments may have some fields passed in
14598 registers and some in memory.
14600 .. TODO::
14602 So, a struct which can pass some fields as decomposed register arguments, will
14603 pass the rest as decomposed stack elements? But an argument that will not start
14604 in registers will not be decomposed and will be passed as a non-decomposed
14605 stack value?
14607 The following is not part of the AMDGPU function calling convention but
14608 describes how the AMDGPU implements function calls:
14610 1. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an
14611 unswizzled scratch address. It is only needed if runtime sized ``alloca``
14612 are used, or for the reasons defined in ``SIFrameLowering``.
14613 2. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP)
14614 to access the incoming stack arguments in the function. The BP is needed
14615 only when the function requires the runtime stack alignment.
14617 3. Allocating SGPR arguments on the stack are not supported.
14619 4. No CFI is currently generated. See
14620 :ref:`amdgpu-dwarf-call-frame-information`.
14622 .. note::
14624 CFI will be generated that defines the CFA as the unswizzled address
14625 relative to the wave scratch base in the unswizzled private address space
14626 of the lowest address stack allocated local variable.
14628 ``DW_AT_frame_base`` will be defined as the swizzled address in the
14629 swizzled private address space by dividing the CFA by the wavefront size
14630 (since CFA is always at least dword aligned which matches the scratch
14631 swizzle element size).
14633 If no dynamic stack alignment was performed, the stack allocated arguments
14634 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the
14635 local variables and register spill slots are accessed as positive offsets
14636 relative to ``DW_AT_frame_base``.
14638 5. Function argument passing is implemented by copying the input physical
14639 registers to virtual registers on entry. The register allocator can spill if
14640 necessary. These are copied back to physical registers at call sites. The
14641 net effect is that each function call can have these values in entirely
14642 distinct locations. The IPRA can help avoid shuffling argument registers.
14643 6. Call sites are implemented by setting up the arguments at positive offsets
14644 from SP. Then SP is incremented to account for the known frame size before
14645 the call and decremented after the call.
14647 .. note::
14649 The CFI will reflect the changed calculation needed to compute the CFA
14650 from SP.
14652 7. 4 byte spill slots are used in the stack frame. One slot is allocated for an
14653 emergency spill slot. Buffer instructions are used for stack accesses and
14654 not the ``flat_scratch`` instruction.
14656 .. TODO::
14658 Explain when the emergency spill slot is used.
14660 .. TODO::
14662 Possible broken issues:
14664 - Stack arguments must be aligned to required alignment.
14665 - Stack is aligned to max(16, max formal argument alignment)
14666 - Direct argument < 64 bits should check register budget.
14667 - Register budget calculation should respect ``inreg`` for SGPR.
14668 - SGPR overflow is not handled.
14669 - struct with 1 member unpeeling is not checking size of member.
14670 - ``sret`` is after ``this`` pointer.
14671 - Caller is not implementing stack realignment: need an extra pointer.
14672 - Should say AMDGPU passes FP rather than SP.
14673 - Should CFI define CFA as address of locals or arguments. Difference is
14674 apparent when have implemented dynamic alignment.
14675 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be
14676 highest address of stack frame and use negative offset for locals. Would
14677 allow SP to be the same as FP and could support signal-handler-like as now
14678 have a real SP for the top of the stack.
14679 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay
14680 arguments?
14682 AMDPAL
14683 ------
14685 This section provides code conventions used when the target triple OS is
14686 ``amdpal`` (see :ref:`amdgpu-target-triples`).
14688 .. _amdgpu-amdpal-code-object-metadata-section:
14690 Code Object Metadata
14691 ~~~~~~~~~~~~~~~~~~~~
14693 .. note::
14695 The metadata is currently in development and is subject to major
14696 changes. Only the current version is supported. *When this document
14697 was generated the version was 2.6.*
14699 Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note
14700 record (see :ref:`amdgpu-note-records-v3-onwards`).
14702 The metadata is represented as Message Pack formatted binary data (see
14703 [MsgPack]_). The top level is a Message Pack map that includes the keys
14704 defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table`
14705 and referenced tables.
14707 Additional information can be added to the maps. To avoid conflicts, any
14708 key names should be prefixed by "*vendor-name*." where ``vendor-name``
14709 can be the name of the vendor and specific vendor tool that generates the
14710 information. The prefix is abbreviated to simply "." when it appears
14711 within a map that has been added by the same *vendor-name*.
14713 .. table:: AMDPAL Code Object Metadata Map
14714 :name: amdgpu-amdpal-code-object-metadata-map-table
14716 =================== ============== ========= ======================================================================
14717 String Key Value Type Required? Description
14718 =================== ============== ========= ======================================================================
14719 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values
14720 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*.
14721 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See
14722 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the
14723 definition of the keys included in that map.
14724 =================== ============== ========= ======================================================================
14726 ..
14728 .. table:: AMDPAL Code Object Pipeline Metadata Map
14729 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table
14731 ====================================== ============== ========= ===================================================
14732 String Key Value Type Required? Description
14733 ====================================== ============== ========= ===================================================
14734 ".name" string Source name of the pipeline.
14735 ".type" string Pipeline type, e.g. VsPs. Values include:
14737 - "VsPs"
14738 - "Gs"
14739 - "Cs"
14740 - "Ngg"
14741 - "Tess"
14742 - "GsTess"
14743 - "NggTess"
14745 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower
14746 2 integers 64 bits is the "stable" portion of the hash, used
14747 for e.g. shader replacement lookup. Upper 64 bits
14748 is the "unique" portion of the hash, used for
14749 e.g. pipeline cache lookup. The value is
14750 implementation defined, and can not be relied on
14751 between different builds of the compiler.
14752 ".shaders" map Per-API shader metadata. See
14753 :ref:`amdgpu-amdpal-code-object-shader-map-table`
14754 for the definition of the keys included in that
14755 map.
14756 ".hardware_stages" map Per-hardware stage metadata. See
14757 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table`
14758 for the definition of the keys included in that
14759 map.
14760 ".shader_functions" map Per-shader function metadata. See
14761 :ref:`amdgpu-amdpal-code-object-shader-function-map-table`
14762 for the definition of the keys included in that
14763 map.
14764 ".registers" map Required Hardware register configuration. See
14765 :ref:`amdgpu-amdpal-code-object-register-map-table`
14766 for the definition of the keys included in that
14767 map.
14768 ".user_data_limit" integer Number of user data entries accessed by this
14769 pipeline.
14770 ".spill_threshold" integer The user data spill threshold. 0xFFFF for
14771 NoUserDataSpilling.
14772 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the
14773 viewport array index feature. Pipelines which use
14774 this feature can render into all 16 viewports,
14775 whereas pipelines which do not use it are
14776 restricted to viewport #0.
14777 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for
14778 handling data-passing between the ES and GS
14779 shader stages. This can be zero if the data is
14780 passed using off-chip buffers. This value should
14781 be used to program all user-SGPRs which have been
14782 marked with "UserDataMapping::EsGsLdsSize"
14783 (typically only the GS and VS HW stages will ever
14784 have a user-SGPR so marked).
14785 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders
14786 (maximum number of threads in a subgroup).
14787 ".num_interpolants" integer Graphics only. Number of PS interpolants.
14788 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used.
14789 ".api" string Name of the client graphics API.
14790 ".api_create_info" binary Graphics API shader create info binary blob. Can
14791 be defined by the driver using the compiler if
14792 they want to be able to correlate API-specific
14793 information used during creation at a later time.
14794 ====================================== ============== ========= ===================================================
14796 ..
14798 .. table:: AMDPAL Code Object Shader Map
14799 :name: amdgpu-amdpal-code-object-shader-map-table
14802 +-------------+--------------+-------------------------------------------------------------------+
14803 |String Key |Value Type |Description |
14804 +=============+==============+===================================================================+
14805 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` |
14806 |- ".vertex" | |for the definition of the keys included in that map. |
14807 |- ".hull" | | |
14808 |- ".domain" | | |
14809 |- ".geometry"| | |
14810 |- ".pixel" | | |
14811 +-------------+--------------+-------------------------------------------------------------------+
14813 ..
14815 .. table:: AMDPAL Code Object API Shader Metadata Map
14816 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table
14818 ==================== ============== ========= =====================================================================
14819 String Key Value Type Required? Description
14820 ==================== ============== ========= =====================================================================
14821 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value
14822 2 integers is implementation defined, and can not be relied on between
14823 different builds of the compiler.
14824 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values
14825 string include:
14827 - ".ls"
14828 - ".hs"
14829 - ".es"
14830 - ".gs"
14831 - ".vs"
14832 - ".ps"
14833 - ".cs"
14835 ==================== ============== ========= =====================================================================
14837 ..
14839 .. table:: AMDPAL Code Object Hardware Stage Map
14840 :name: amdgpu-amdpal-code-object-hardware-stage-map-table
14842 +-------------+--------------+-----------------------------------------------------------------------+
14843 |String Key |Value Type |Description |
14844 +=============+==============+=======================================================================+
14845 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` |
14846 |- ".hs" | |for the definition of the keys included in that map. |
14847 |- ".es" | | |
14848 |- ".gs" | | |
14849 |- ".vs" | | |
14850 |- ".ps" | | |
14851 |- ".cs" | | |
14852 +-------------+--------------+-----------------------------------------------------------------------+
14854 ..
14856 .. table:: AMDPAL Code Object Hardware Stage Metadata Map
14857 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table
14859 ========================== ============== ========= ===============================================================
14860 String Key Value Type Required? Description
14861 ========================== ============== ========= ===============================================================
14862 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point.
14863 ".scratch_memory_size" integer Scratch memory size in bytes.
14864 ".lds_size" integer Local Data Share size in bytes.
14865 ".perf_data_buffer_size" integer Performance data buffer size in bytes.
14866 ".vgpr_count" integer Number of VGPRs used.
14867 ".agpr_count" integer Number of AGPRs used.
14868 ".sgpr_count" integer Number of SGPRs used.
14869 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a
14870 directive to instruct the compiler to limit the VGPR usage to
14871 be less than or equal to the specified value (only set if
14872 different from HW default).
14873 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW
14874 default).
14875 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only).
14876 3 integers
14877 ".wavefront_size" integer Wavefront size (only set if different from HW default).
14878 ".uses_uavs" boolean The shader reads or writes UAVs.
14879 ".uses_rovs" boolean The shader reads or writes ROVs.
14880 ".writes_uavs" boolean The shader writes to one or more UAVs.
14881 ".writes_depth" boolean The shader writes out a depth value.
14882 ".uses_append_consume" boolean The shader uses append and/or consume operations, either
14883 memory or GDS.
14884 ".uses_prim_id" boolean The shader uses PrimID.
14885 ========================== ============== ========= ===============================================================
14887 ..
14889 .. table:: AMDPAL Code Object Shader Function Map
14890 :name: amdgpu-amdpal-code-object-shader-function-map-table
14892 =============== ============== ====================================================================
14893 String Key Value Type Description
14894 =============== ============== ====================================================================
14895 *symbol name* map *symbol name* is the ELF symbol name of the shader function code
14896 entry address. The value is the function's metadata. See
14897 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`.
14898 =============== ============== ====================================================================
14900 ..
14902 .. table:: AMDPAL Code Object Shader Function Metadata Map
14903 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table
14905 ============================= ============== =================================================================
14906 String Key Value Type Description
14907 ============================= ============== =================================================================
14908 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value
14909 2 integers is implementation defined, and can not be relied on between
14910 different builds of the compiler.
14911 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader.
14912 ".lds_size" integer Size in bytes of LDS memory.
14913 ".vgpr_count" integer Number of VGPRs used by the shader.
14914 ".sgpr_count" integer Number of SGPRs used by the shader.
14915 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader.
14916 ".shader_subtype" string Shader subtype/kind. Values include:
14918 - "Unknown"
14920 ============================= ============== =================================================================
14922 ..
14924 .. table:: AMDPAL Code Object Register Map
14925 :name: amdgpu-amdpal-code-object-register-map-table
14927 ========================== ============== ====================================================================
14928 32-bit Integer Key Value Type Description
14929 ========================== ============== ====================================================================
14930 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of
14931 a GRBM register (i.e., driver accessible GPU register number, not
14932 shader GPR register number). The driver is required to program each
14933 specified register to the corresponding specified value when
14934 executing this pipeline. Typically, the ``reg offsets`` are the
14935 ``uint16_t`` offsets to each register as defined by the hardware
14936 chip headers. The register is set to the provided value. However, a
14937 ``reg offset`` that specifies a user data register (e.g.,
14938 COMPUTE_USER_DATA_0) needs special treatment. See
14939 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more
14940 information.
14941 ========================== ============== ====================================================================
14943 .. _amdgpu-amdpal-code-object-user-data-section:
14945 User Data
14946 +++++++++
14948 Each hardware stage has a set of 32-bit physical SPI *user data registers*
14949 (either 16 or 32 based on graphics IP and the stage) which can be
14950 written from a command buffer and then loaded into SGPRs when waves are
14951 launched via a subsequent dispatch or draw operation. This is the way
14952 most arguments are passed from the application/runtime to a hardware
14953 shader.
14955 PAL abstracts this functionality by exposing a set of 128 *user data
14956 entries* per pipeline a client can use to pass arguments from a command
14957 buffer to one or more shaders in that pipeline. The ELF code object must
14958 specify a mapping from virtualized *user data entries* to physical *user
14959 data registers*, and PAL is responsible for implementing that mapping,
14960 including spilling overflow *user data entries* to memory if needed.
14962 Since the *user data registers* are GRBM-accessible SPI registers, this
14963 mapping is actually embedded in the ``.registers`` metadata entry. For
14964 most registers, the value in that map is a literal 32-bit value that
14965 should be written to the register by the driver. However, when the
14966 register is a *user data register* (any USER_DATA register e.g.,
14967 SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells
14968 the driver to write either a *user data entry* value or one of several
14969 driver-internal values to the register. This encoding is described in
14970 the following table:
14972 .. note::
14974 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0,
14975 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must
14976 always be programmed to the address of the GlobalTable, and *user data
14977 register* 1 must always be programmed to the address of the PerShaderTable.
14979 ..
14981 .. table:: AMDPAL User Data Mapping
14982 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table
14984 ========== ================= ===============================================================================
14985 Value Name Description
14986 ========== ================= ===============================================================================
14987 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()*
14988 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should
14989 always point to *user data register* 0).
14990 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See
14991 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section`
14992 for more detail (should always point to *user data register* 1).
14993 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See
14994 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for
14995 more detail.
14996 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't
14997 reference the draw index in the vertex shader. Only supported by the first
14998 stage in a graphics pipeline.
14999 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in
15000 a graphics pipeline.
15001 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a
15002 graphics pipeline.
15003 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of
15004 a buffer containing the grid dimensions for a Compute dispatch operation. The
15005 high half of the address is stored in the next sequential user-SGPR. Only
15006 supported by compute pipelines.
15007 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS
15008 space used for the ES/GS pseudo-ring-buffer for passing data between shader
15009 stages.
15010 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic
15011 pipeline instancing.
15012 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This
15013 can only appear for one shader stage per pipeline.
15014 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer.
15015 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can
15016 only appear for one shader stage per pipeline.
15017 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can
15018 only appear for one shader stage per pipeline (PS). These replace color targets
15019 and are completely separate from any UAVs used by the shader. This is optional,
15020 and only used by the PS when UAV exports are used to replace color-target
15021 exports to optimize specific shaders.
15022 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by
15023 some NGG pipelines to perform culling. This value contains the address of the
15024 first of two consecutive registers which provide the full GPU address.
15025 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine.
15026 ========== ================= ===============================================================================
15028 .. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section:
15030 Per-Shader Table
15031 ################
15033 Low 32 bits of the GPU address for an optional buffer in the ``.data``
15034 section of the ELF. The high 32 bits of the address match the high 32 bits
15035 of the shader's program counter.
15037 The buffer can be anything the shader compiler needs it for, and
15038 allows each shader to have its own region of the ``.data`` section.
15039 Typically, this could be a table of buffer SRD's and the data pointed to
15040 by the buffer SRD's, but it could be a flat-address region of memory as
15041 well. Its layout and usage are defined by the shader compiler.
15043 Each shader's table in the ``.data`` section is referenced by the symbol
15044 ``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the
15045 hardware shader stage the data is for. E.g.,
15046 ``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage.
15048 .. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section:
15050 Spill Table
15051 ###########
15053 It is possible for a hardware shader to need access to more *user data
15054 entries* than there are slots available in user data registers for one
15055 or more hardware shader stages. In that case, the PAL runtime expects
15056 the necessary *user data entries* to be spilled to GPU memory and use
15057 one user data register to point to the spilled user data memory. The
15058 value of the *user data entry* must then represent the location where
15059 a shader expects to read the low 32-bits of the table's GPU virtual
15060 address. The *spill table* itself represents a set of 32-bit values
15061 managed by the PAL runtime in GPU-accessible memory that can be made
15062 indirectly accessible to a hardware shader.
15064 Unspecified OS
15065 --------------
15067 This section provides code conventions used when the target triple OS is
15068 empty (see :ref:`amdgpu-target-triples`).
15070 Trap Handler ABI
15071 ~~~~~~~~~~~~~~~~
15073 For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does
15074 not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap``
15075 instructions are handled as follows:
15077 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS
15078 :name: amdgpu-trap-handler-for-non-amdhsa-os-table
15080 =============== =============== ===========================================
15081 Usage Code Sequence Description
15082 =============== =============== ===========================================
15083 llvm.trap s_endpgm Causes wavefront to be terminated.
15084 llvm.debugtrap *none* Compiler warning given that there is no
15085 trap handler installed.
15086 =============== =============== ===========================================
15088 Source Languages
15089 ================
15091 .. _amdgpu-opencl:
15093 OpenCL
15094 ------
15096 When the language is OpenCL the following differences occur:
15098 1. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`).
15099 2. The AMDGPU backend appends additional arguments to the kernel's explicit
15100 arguments for the AMDHSA OS (see
15101 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`).
15102 3. Additional metadata is generated
15103 (see :ref:`amdgpu-amdhsa-code-object-metadata`).
15105 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS
15106 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table
15108 ======== ==== ========= ===========================================
15109 Position Byte Byte Description
15110 Size Alignment
15111 ======== ==== ========= ===========================================
15112 1 8 8 OpenCL Global Offset X
15113 2 8 8 OpenCL Global Offset Y
15114 3 8 8 OpenCL Global Offset Z
15115 4 8 8 OpenCL address of printf buffer
15116 5 8 8 OpenCL address of virtual queue used by
15117 enqueue_kernel.
15118 6 8 8 OpenCL address of AqlWrap struct used by
15119 enqueue_kernel.
15120 7 8 8 Pointer argument used for Multi-gird
15121 synchronization.
15122 ======== ==== ========= ===========================================
15124 .. _amdgpu-hcc:
15126 HCC
15127 ---
15129 When the language is HCC the following differences occur:
15131 1. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`).
15133 .. _amdgpu-assembler:
15135 Assembler
15136 ---------
15138 AMDGPU backend has LLVM-MC based assembler which is currently in development.
15139 It supports AMDGCN GFX6-GFX11.
15141 This section describes general syntax for instructions and operands.
15143 Instructions
15144 ~~~~~~~~~~~~
15146 An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`:
15148 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,...
15149 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...``
15151 :doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while
15152 :doc:`modifiers<AMDGPUModifierSyntax>` are space-separated.
15154 The order of operands and modifiers is fixed.
15155 Most modifiers are optional and may be omitted.
15157 Links to detailed instruction syntax description may be found in the following
15158 table. Note that features under development are not included
15159 in this description.
15161 ============= ============================================= =======================================
15162 Architecture Core ISA ISA Variants and Extensions
15163 ============= ============================================= =======================================
15164 GCN 2 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \-
15165 GCN 3, GCN 4 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \-
15166 GCN 5 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>`
15168 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>`
15170 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>`
15172 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>`
15174 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>`
15176 :doc:`gfx90c<AMDGPU/AMDGPUAsmGFX900>`
15178 CDNA 1 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>`
15180 CDNA 2 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>`
15182 CDNA 3 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx940<AMDGPU/AMDGPUAsmGFX940>`
15184 :doc:`gfx941<AMDGPU/AMDGPUAsmGFX940>`
15186 :doc:`gfx942<AMDGPU/AMDGPUAsmGFX940>`
15188 RDNA 1 :doc:`GFX10 RDNA1<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1010<AMDGPU/AMDGPUAsmGFX10>`
15190 :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>`
15192 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>`
15194 :doc:`gfx1013<AMDGPU/AMDGPUAsmGFX1013>`
15196 RDNA 2 :doc:`GFX10 RDNA2<AMDGPU/AMDGPUAsmGFX1030>` :doc:`gfx1030<AMDGPU/AMDGPUAsmGFX1030>`
15198 :doc:`gfx1031<AMDGPU/AMDGPUAsmGFX1030>`
15200 :doc:`gfx1032<AMDGPU/AMDGPUAsmGFX1030>`
15202 :doc:`gfx1033<AMDGPU/AMDGPUAsmGFX1030>`
15204 :doc:`gfx1034<AMDGPU/AMDGPUAsmGFX1030>`
15206 :doc:`gfx1035<AMDGPU/AMDGPUAsmGFX1030>`
15208 :doc:`gfx1036<AMDGPU/AMDGPUAsmGFX1030>`
15210 RDNA 3 :doc:`GFX11<AMDGPU/AMDGPUAsmGFX11>` :doc:`gfx1100<AMDGPU/AMDGPUAsmGFX11>`
15212 :doc:`gfx1101<AMDGPU/AMDGPUAsmGFX11>`
15214 :doc:`gfx1102<AMDGPU/AMDGPUAsmGFX11>`
15216 :doc:`gfx1103<AMDGPU/AMDGPUAsmGFX11>`
15217 ============= ============================================= =======================================
15219 For more information about instructions, their semantics and supported
15220 combinations of operands, refer to one of instruction set architecture manuals
15221 [AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_,
15222 [AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_,
15223 [AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX90A-CDNA2]_,
15224 [AMD-GCN-GFX940-GFX942-CDNA3]_, [AMD-GCN-GFX10-RDNA1]_, [AMD-GCN-GFX10-RDNA2]_
15225 and [AMD-GCN-GFX11-RDNA3]_.
15227 Operands
15228 ~~~~~~~~
15230 Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`.
15232 Modifiers
15233 ~~~~~~~~~
15235 Detailed description of modifiers may be found
15236 :doc:`here<AMDGPUModifierSyntax>`.
15238 Instruction Examples
15239 ~~~~~~~~~~~~~~~~~~~~
15241 DS
15242 ++
15244 .. code-block:: nasm
15246 ds_add_u32 v2, v4 offset:16
15247 ds_write_src2_b64 v2 offset0:4 offset1:8
15248 ds_cmpst_f32 v2, v4, v6
15249 ds_min_rtn_f64 v[8:9], v2, v[4:5]
15251 For full list of supported instructions, refer to "LDS/GDS instructions" in ISA
15252 Manual.
15254 FLAT
15255 ++++
15257 .. code-block:: nasm
15259 flat_load_dword v1, v[3:4]
15260 flat_store_dwordx3 v[3:4], v[5:7]
15261 flat_atomic_swap v1, v[3:4], v5 glc
15262 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc
15263 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc
15265 For full list of supported instructions, refer to "FLAT instructions" in ISA
15266 Manual.
15268 MUBUF
15269 +++++
15271 .. code-block:: nasm
15273 buffer_load_dword v1, off, s[4:7], s1
15274 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe
15275 buffer_store_format_xy v[1:2], off, s[4:7], s1
15276 buffer_wbinvl1
15277 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc
15279 For full list of supported instructions, refer to "MUBUF Instructions" in ISA
15280 Manual.
15282 SMRD/SMEM
15283 +++++++++
15285 .. code-block:: nasm
15287 s_load_dword s1, s[2:3], 0xfc
15288 s_load_dwordx8 s[8:15], s[2:3], s4
15289 s_load_dwordx16 s[88:103], s[2:3], s4
15290 s_dcache_inv_vol
15291 s_memtime s[4:5]
15293 For full list of supported instructions, refer to "Scalar Memory Operations" in
15294 ISA Manual.
15296 SOP1
15297 ++++
15299 .. code-block:: nasm
15301 s_mov_b32 s1, s2
15302 s_mov_b64 s[0:1], 0x80000000
15303 s_cmov_b32 s1, 200
15304 s_wqm_b64 s[2:3], s[4:5]
15305 s_bcnt0_i32_b64 s1, s[2:3]
15306 s_swappc_b64 s[2:3], s[4:5]
15307 s_cbranch_join s[4:5]
15309 For full list of supported instructions, refer to "SOP1 Instructions" in ISA
15310 Manual.
15312 SOP2
15313 ++++
15315 .. code-block:: nasm
15317 s_add_u32 s1, s2, s3
15318 s_and_b64 s[2:3], s[4:5], s[6:7]
15319 s_cselect_b32 s1, s2, s3
15320 s_andn2_b32 s2, s4, s6
15321 s_lshr_b64 s[2:3], s[4:5], s6
15322 s_ashr_i32 s2, s4, s6
15323 s_bfm_b64 s[2:3], s4, s6
15324 s_bfe_i64 s[2:3], s[4:5], s6
15325 s_cbranch_g_fork s[4:5], s[6:7]
15327 For full list of supported instructions, refer to "SOP2 Instructions" in ISA
15328 Manual.
15330 SOPC
15331 ++++
15333 .. code-block:: nasm
15335 s_cmp_eq_i32 s1, s2
15336 s_bitcmp1_b32 s1, s2
15337 s_bitcmp0_b64 s[2:3], s4
15338 s_setvskip s3, s5
15340 For full list of supported instructions, refer to "SOPC Instructions" in ISA
15341 Manual.
15343 SOPP
15344 ++++
15346 .. code-block:: nasm
15348 s_barrier
15349 s_nop 2
15350 s_endpgm
15351 s_waitcnt 0 ; Wait for all counters to be 0
15352 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above
15353 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1.
15354 s_sethalt 9
15355 s_sleep 10
15356 s_sendmsg 0x1
15357 s_sendmsg sendmsg(MSG_INTERRUPT)
15358 s_trap 1
15360 For full list of supported instructions, refer to "SOPP Instructions" in ISA
15361 Manual.
15363 Unless otherwise mentioned, little verification is performed on the operands
15364 of SOPP Instructions, so it is up to the programmer to be familiar with the
15365 range or acceptable values.
15367 VALU
15368 ++++
15370 For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA),
15371 the assembler will automatically use optimal encoding based on its operands. To
15372 force specific encoding, one can add a suffix to the opcode of the instruction:
15374 * _e32 for 32-bit VOP1/VOP2/VOPC
15375 * _e64 for 64-bit VOP3
15376 * _dpp for VOP_DPP
15377 * _e64_dpp for VOP3 with DPP
15378 * _sdwa for VOP_SDWA
15380 VOP1/VOP2/VOP3/VOPC examples:
15382 .. code-block:: nasm
15384 v_mov_b32 v1, v2
15385 v_mov_b32_e32 v1, v2
15386 v_nop
15387 v_cvt_f64_i32_e32 v[1:2], v2
15388 v_floor_f32_e32 v1, v2
15389 v_bfrev_b32_e32 v1, v2
15390 v_add_f32_e32 v1, v2, v3
15391 v_mul_i32_i24_e64 v1, v2, 3
15392 v_mul_i32_i24_e32 v1, -3, v3
15393 v_mul_i32_i24_e32 v1, -100, v3
15394 v_addc_u32 v1, s[0:1], v2, v3, s[2:3]
15395 v_max_f16_e32 v1, v2, v3
15397 VOP_DPP examples:
15399 .. code-block:: nasm
15401 v_mov_b32 v0, v0 quad_perm:[0,2,1,1]
15402 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0
15403 v_mov_b32 v0, v0 wave_shl:1
15404 v_mov_b32 v0, v0 row_mirror
15405 v_mov_b32 v0, v0 row_bcast:31
15406 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0
15407 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0
15408 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0
15411 VOP3_DPP examples (Available on GFX11+):
15413 .. code-block:: nasm
15415 v_add_f32_e64_dpp v0, v1, v2 dpp8:[0,1,2,3,4,5,6,7]
15416 v_sqrt_f32_e64_dpp v0, v1 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0
15417 v_ldexp_f32 v0, v1, v2 dpp8:[0,1,2,3,4,5,6,7]
15419 VOP_SDWA examples:
15421 .. code-block:: nasm
15423 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD
15424 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD
15425 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1
15426 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1
15427 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0
15429 For full list of supported instructions, refer to "Vector ALU instructions".
15431 .. _amdgpu-amdhsa-assembler-predefined-symbols-v2:
15433 Code Object V2 Predefined Symbols
15434 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
15436 .. warning::
15437 Code object V2 generation is no longer supported by this version of LLVM.
15439 The AMDGPU assembler defines and updates some symbols automatically. These
15440 symbols do not affect code generation.
15442 .option.machine_version_major
15443 +++++++++++++++++++++++++++++
15445 Set to the GFX major generation number of the target being assembled for. For
15446 example, when assembling for a "GFX9" target this will be set to the integer
15447 value "9". The possible GFX major generation numbers are presented in
15448 :ref:`amdgpu-processors`.
15450 .option.machine_version_minor
15451 +++++++++++++++++++++++++++++
15453 Set to the GFX minor generation number of the target being assembled for. For
15454 example, when assembling for a "GFX810" target this will be set to the integer
15455 value "1". The possible GFX minor generation numbers are presented in
15456 :ref:`amdgpu-processors`.
15458 .option.machine_version_stepping
15459 ++++++++++++++++++++++++++++++++
15461 Set to the GFX stepping generation number of the target being assembled for.
15462 For example, when assembling for a "GFX704" target this will be set to the
15463 integer value "4". The possible GFX stepping generation numbers are presented
15464 in :ref:`amdgpu-processors`.
15466 .kernel.vgpr_count
15467 ++++++++++++++++++
15469 Set to zero each time a
15470 :ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is
15471 encountered. At each instruction, if the current value of this symbol is less
15472 than or equal to the maximum VGPR number explicitly referenced within that
15473 instruction then the symbol value is updated to equal that VGPR number plus
15474 one.
15476 .kernel.sgpr_count
15477 ++++++++++++++++++
15479 Set to zero each time a
15480 :ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is
15481 encountered. At each instruction, if the current value of this symbol is less
15482 than or equal to the maximum VGPR number explicitly referenced within that
15483 instruction then the symbol value is updated to equal that SGPR number plus
15484 one.
15486 .. _amdgpu-amdhsa-assembler-directives-v2:
15488 Code Object V2 Directives
15489 ~~~~~~~~~~~~~~~~~~~~~~~~~
15491 .. warning::
15492 Code object V2 generation is no longer supported by this version of LLVM.
15494 AMDGPU ABI defines auxiliary data in output code object. In assembly source,
15495 one can specify them with assembler directives.
15497 .hsa_code_object_version major, minor
15498 +++++++++++++++++++++++++++++++++++++
15500 *major* and *minor* are integers that specify the version of the HSA code
15501 object that will be generated by the assembler.
15503 .hsa_code_object_isa [major, minor, stepping, vendor, arch]
15504 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
15507 *major*, *minor*, and *stepping* are all integers that describe the instruction
15508 set architecture (ISA) version of the assembly program.
15510 *vendor* and *arch* are quoted strings. *vendor* should always be equal to
15511 "AMD" and *arch* should always be equal to "AMDGPU".
15513 By default, the assembler will derive the ISA version, *vendor*, and *arch*
15514 from the value of the -mcpu option that is passed to the assembler.
15516 .. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel:
15518 .amdgpu_hsa_kernel (name)
15519 +++++++++++++++++++++++++
15521 This directives specifies that the symbol with given name is a kernel entry
15522 point (label) and the object should contain corresponding symbol of type
15523 STT_AMDGPU_HSA_KERNEL.
15525 .amd_kernel_code_t
15526 ++++++++++++++++++
15528 This directive marks the beginning of a list of key / value pairs that are used
15529 to specify the amd_kernel_code_t object that will be emitted by the assembler.
15530 The list must be terminated by the *.end_amd_kernel_code_t* directive. For any
15531 amd_kernel_code_t values that are unspecified a default value will be used. The
15532 default value for all keys is 0, with the following exceptions:
15534 - *amd_code_version_major* defaults to 1.
15535 - *amd_kernel_code_version_minor* defaults to 2.
15536 - *amd_machine_kind* defaults to 1.
15537 - *amd_machine_version_major*, *machine_version_minor*, and
15538 *amd_machine_version_stepping* are derived from the value of the -mcpu option
15539 that is passed to the assembler.
15540 - *kernel_code_entry_byte_offset* defaults to 256.
15541 - *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards
15542 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5.
15543 Note that wavefront size is specified as a power of two, so a value of **n**
15544 means a size of 2^ **n**.
15545 - *call_convention* defaults to -1.
15546 - *kernarg_segment_alignment*, *group_segment_alignment*, and
15547 *private_segment_alignment* default to 4. Note that alignments are specified
15548 as a power of 2, so a value of **n** means an alignment of 2^ **n**.
15549 - *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for
15550 GFX90A onwards.
15551 - *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for
15552 GFX10 onwards.
15553 - *enable_mem_ordered* defaults to 1 for GFX10 onwards.
15555 The *.amd_kernel_code_t* directive must be placed immediately after the
15556 function label and before any instructions.
15558 For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document,
15559 comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s.
15561 .. _amdgpu-amdhsa-assembler-example-v2:
15563 Code Object V2 Example Source Code
15564 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
15566 .. warning::
15567 Code object V2 generation is no longer supported by this version of LLVM.
15569 Here is an example of a minimal assembly source file, defining one HSA kernel:
15571 .. code::
15572 :number-lines:
15574 .hsa_code_object_version 1,0
15575 .hsa_code_object_isa
15577 .hsatext
15578 .globl hello_world
15579 .p2align 8
15580 .amdgpu_hsa_kernel hello_world
15582 hello_world:
15584 .amd_kernel_code_t
15585 enable_sgpr_kernarg_segment_ptr = 1
15586 is_ptr64 = 1
15587 compute_pgm_rsrc1_vgprs = 0
15588 compute_pgm_rsrc1_sgprs = 0
15589 compute_pgm_rsrc2_user_sgpr = 2
15590 compute_pgm_rsrc1_wgp_mode = 0
15591 compute_pgm_rsrc1_mem_ordered = 0
15592 compute_pgm_rsrc1_fwd_progress = 1
15593 .end_amd_kernel_code_t
15595 s_load_dwordx2 s[0:1], s[0:1] 0x0
15596 v_mov_b32 v0, 3.14159
15597 s_waitcnt lgkmcnt(0)
15598 v_mov_b32 v1, s0
15599 v_mov_b32 v2, s1
15600 flat_store_dword v[1:2], v0
15601 s_endpgm
15602 .Lfunc_end0:
15603 .size hello_world, .Lfunc_end0-hello_world
15605 .. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards:
15607 Code Object V3 and Above Predefined Symbols
15608 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
15610 The AMDGPU assembler defines and updates some symbols automatically. These
15611 symbols do not affect code generation.
15613 .amdgcn.gfx_generation_number
15614 +++++++++++++++++++++++++++++
15616 Set to the GFX major generation number of the target being assembled for. For
15617 example, when assembling for a "GFX9" target this will be set to the integer
15618 value "9". The possible GFX major generation numbers are presented in
15619 :ref:`amdgpu-processors`.
15621 .amdgcn.gfx_generation_minor
15622 ++++++++++++++++++++++++++++
15624 Set to the GFX minor generation number of the target being assembled for. For
15625 example, when assembling for a "GFX810" target this will be set to the integer
15626 value "1". The possible GFX minor generation numbers are presented in
15627 :ref:`amdgpu-processors`.
15629 .amdgcn.gfx_generation_stepping
15630 +++++++++++++++++++++++++++++++
15632 Set to the GFX stepping generation number of the target being assembled for.
15633 For example, when assembling for a "GFX704" target this will be set to the
15634 integer value "4". The possible GFX stepping generation numbers are presented
15635 in :ref:`amdgpu-processors`.
15637 .. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr:
15639 .amdgcn.next_free_vgpr
15640 ++++++++++++++++++++++
15642 Set to zero before assembly begins. At each instruction, if the current value
15643 of this symbol is less than or equal to the maximum VGPR number explicitly
15644 referenced within that instruction then the symbol value is updated to equal
15645 that VGPR number plus one.
15647 May be used to set the `.amdhsa_next_free_vgpr` directive in
15648 :ref:`amdhsa-kernel-directives-table`.
15650 May be set at any time, e.g. manually set to zero at the start of each kernel.
15652 .. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr:
15654 .amdgcn.next_free_sgpr
15655 ++++++++++++++++++++++
15657 Set to zero before assembly begins. At each instruction, if the current value
15658 of this symbol is less than or equal the maximum SGPR number explicitly
15659 referenced within that instruction then the symbol value is updated to equal
15660 that SGPR number plus one.
15662 May be used to set the `.amdhsa_next_free_spgr` directive in
15663 :ref:`amdhsa-kernel-directives-table`.
15665 May be set at any time, e.g. manually set to zero at the start of each kernel.
15667 .. _amdgpu-amdhsa-assembler-directives-v3-onwards:
15669 Code Object V3 and Above Directives
15670 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
15672 Directives which begin with ``.amdgcn`` are valid for all ``amdgcn``
15673 architecture processors, and are not OS-specific. Directives which begin with
15674 ``.amdhsa`` are specific to ``amdgcn`` architecture processors when the
15675 ``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and
15676 :ref:`amdgpu-processors`.
15678 .. _amdgpu-assembler-directive-amdgcn-target:
15680 .amdgcn_target <target-triple> "-" <target-id>
15681 ++++++++++++++++++++++++++++++++++++++++++++++
15683 Optional directive which declares the ``<target-triple>-<target-id>`` supported
15684 by the containing assembler source file. Used by the assembler to validate
15685 command-line options such as ``-triple``, ``-mcpu``, and
15686 ``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See
15687 :ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`.
15689 .. note::
15691 The target ID syntax used for code object V2 to V3 for this directive differs
15692 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`.
15694 .. _amdgpu-assembler-directive-amdhsa-code-object-version:
15696 .amdhsa_code_object_version <version>
15697 +++++++++++++++++++++++++++++++++++++
15699 Optional directive which declares the code object version to be generated by the
15700 assembler. If not present, a default value will be used.
15702 .amdhsa_kernel <name>
15703 +++++++++++++++++++++
15705 Creates a correctly aligned AMDHSA kernel descriptor and a symbol,
15706 ``<name>.kd``, in the current location of the current section. Only valid when
15707 the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first
15708 instruction to execute, and does not need to be previously defined.
15710 Marks the beginning of a list of directives used to generate the bytes of a
15711 kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`.
15712 Directives which may appear in this list are described in
15713 :ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must
15714 be valid for the target being assembled for, and cannot be repeated. Directives
15715 support the range of values specified by the field they reference in
15716 :ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is
15717 assumed to have its default value, unless it is marked as "Required", in which
15718 case it is an error to omit the directive. This list of directives is
15719 terminated by an ``.end_amdhsa_kernel`` directive.
15721 .. table:: AMDHSA Kernel Assembler Directives
15722 :name: amdhsa-kernel-directives-table
15724 ======================================================== =================== ============ ===================
15725 Directive Default Supported On Description
15726 ======================================================== =================== ============ ===================
15727 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX12 Controls GROUP_SEGMENT_FIXED_SIZE in
15728 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15729 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX12 Controls PRIVATE_SEGMENT_FIXED_SIZE in
15730 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15731 ``.amdhsa_kernarg_size`` 0 GFX6-GFX12 Controls KERNARG_SIZE in
15732 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15733 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX12 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2
15734 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`
15735 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in
15736 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15737 GFX940)
15738 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX12 Controls ENABLE_SGPR_DISPATCH_PTR in
15739 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15740 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX12 Controls ENABLE_SGPR_QUEUE_PTR in
15741 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15742 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX12 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in
15743 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15744 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX12 Controls ENABLE_SGPR_DISPATCH_ID in
15745 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15746 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in
15747 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15748 GFX940)
15749 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX12 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in
15750 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15751 ``.amdhsa_wavefront_size32`` Target GFX10-GFX12 Controls ENABLE_WAVEFRONT_SIZE32 in
15752 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15753 Specific
15754 (wavefrontsize64)
15755 ``.amdhsa_uses_dynamic_stack`` 0 GFX6-GFX12 Controls USES_DYNAMIC_STACK in
15756 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15757 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in
15758 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15759 GFX940)
15760 ``.amdhsa_enable_private_segment`` 0 GFX940, Controls ENABLE_PRIVATE_SEGMENT in
15761 GFX11-GFX12 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15762 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX12 Controls ENABLE_SGPR_WORKGROUP_ID_X in
15763 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15764 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX12 Controls ENABLE_SGPR_WORKGROUP_ID_Y in
15765 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15766 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX12 Controls ENABLE_SGPR_WORKGROUP_ID_Z in
15767 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15768 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX12 Controls ENABLE_SGPR_WORKGROUP_INFO in
15769 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15770 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX12 Controls ENABLE_VGPR_WORKITEM_ID in
15771 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15772 Possible values are defined in
15773 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`.
15774 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX12 Maximum VGPR number explicitly referenced, plus one.
15775 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in
15776 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15777 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX12 Maximum SGPR number explicitly referenced, plus one.
15778 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in
15779 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15780 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file.
15781 GFX940 Used to calculate ACCUM_OFFSET in
15782 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`.
15783 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX12 Whether the kernel may use the special VCC SGPR.
15784 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in
15785 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15786 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access
15787 (except scratch memory. Used to calculate
15788 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in
15789 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15790 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay.
15791 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in
15792 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15793 (xnack)
15794 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX12 Controls FLOAT_ROUND_MODE_32 in
15795 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15796 Possible values are defined in
15797 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`.
15798 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX12 Controls FLOAT_ROUND_MODE_16_64 in
15799 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15800 Possible values are defined in
15801 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`.
15802 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX12 Controls FLOAT_DENORM_MODE_32 in
15803 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15804 Possible values are defined in
15805 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`.
15806 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX12 Controls FLOAT_DENORM_MODE_16_64 in
15807 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15808 Possible values are defined in
15809 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`.
15810 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX11 Controls ENABLE_DX10_CLAMP in
15811 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15812 ``.amdhsa_ieee_mode`` 1 GFX6-GFX11 Controls ENABLE_IEEE_MODE in
15813 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15814 ``.amdhsa_round_robin_scheduling`` 0 GFX12 Controls ENABLE_WG_RR_EN in
15815 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15816 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX12 Controls FP16_OVFL in
15817 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15818 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in
15819 Feature GFX940, :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`.
15820 Specific GFX11-GFX12
15821 (tgsplit)
15822 ``.amdhsa_workgroup_processor_mode`` Target GFX10-GFX12 Controls ENABLE_WGP_MODE in
15823 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15824 Specific
15825 (cumode)
15826 ``.amdhsa_memory_ordered`` 1 GFX10-GFX12 Controls MEM_ORDERED in
15827 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15828 ``.amdhsa_forward_progress`` 0 GFX10-GFX12 Controls FWD_PROGRESS in
15829 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.
15830 ``.amdhsa_shared_vgpr_count`` 0 GFX10-GFX11 Controls SHARED_VGPR_COUNT in
15831 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-gfx11-table`.
15832 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX12 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in
15833 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15834 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX12 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in
15835 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15836 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX12 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in
15837 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15838 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX12 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in
15839 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15840 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX12 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in
15841 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15842 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX12 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in
15843 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15844 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX12 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in
15845 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.
15846 ``.amdhsa_user_sgpr_kernarg_preload_length`` 0 GFX90A, Controls KERNARG_PRELOAD_SPEC_LENGTH in
15847 GFX940 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15848 ``.amdhsa_user_sgpr_kernarg_preload_offset`` 0 GFX90A, Controls KERNARG_PRELOAD_SPEC_OFFSET in
15849 GFX940 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.
15850 ======================================================== =================== ============ ===================
15852 .amdgpu_metadata
15853 ++++++++++++++++
15855 Optional directive which declares the contents of the ``NT_AMDGPU_METADATA``
15856 note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`).
15858 The contents must be in the [YAML]_ markup format, with the same structure and
15859 semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`,
15860 :ref:`amdgpu-amdhsa-code-object-metadata-v4` or
15861 :ref:`amdgpu-amdhsa-code-object-metadata-v5`.
15863 This directive is terminated by an ``.end_amdgpu_metadata`` directive.
15865 .. _amdgpu-amdhsa-assembler-example-v3-onwards:
15867 Code Object V3 and Above Example Source Code
15868 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
15870 Here is an example of a minimal assembly source file, defining one HSA kernel:
15872 .. code::
15873 :number-lines:
15875 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional
15877 .text
15878 .globl hello_world
15879 .p2align 8
15880 .type hello_world,@function
15881 hello_world:
15882 s_load_dwordx2 s[0:1], s[0:1] 0x0
15883 v_mov_b32 v0, 3.14159
15884 s_waitcnt lgkmcnt(0)
15885 v_mov_b32 v1, s0
15886 v_mov_b32 v2, s1
15887 flat_store_dword v[1:2], v0
15888 s_endpgm
15889 .Lfunc_end0:
15890 .size hello_world, .Lfunc_end0-hello_world
15892 .rodata
15893 .p2align 6
15894 .amdhsa_kernel hello_world
15895 .amdhsa_user_sgpr_kernarg_segment_ptr 1
15896 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr
15897 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr
15898 .end_amdhsa_kernel
15900 .amdgpu_metadata
15901 ---
15902 amdhsa.version:
15903 - 1
15904 - 0
15905 amdhsa.kernels:
15906 - .name: hello_world
15907 .symbol: hello_world.kd
15908 .kernarg_segment_size: 48
15909 .group_segment_fixed_size: 0
15910 .private_segment_fixed_size: 0
15911 .kernarg_segment_align: 4
15912 .wavefront_size: 64
15913 .sgpr_count: 2
15914 .vgpr_count: 3
15915 .max_flat_workgroup_size: 256
15916 .args:
15917 - .size: 8
15918 .offset: 0
15919 .value_kind: global_buffer
15920 .address_space: global
15921 .actual_access: write_only
15922 //...
15923 .end_amdgpu_metadata
15925 This kernel is equivalent to the following HIP program:
15927 .. code::
15928 :number-lines:
15930 __global__ void hello_world(float *p) {
15931 *p = 3.14159f;
15932 }
15934 If an assembly source file contains multiple kernels and/or functions, the
15935 :ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and
15936 :ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using
15937 the ``.set <symbol>, <expression>`` directive. For example, in the case of two
15938 kernels, where ``function1`` is only called from ``kernel1`` it is sufficient
15939 to group the function with the kernel that calls it and reset the symbols
15940 between the two connected components:
15942 .. code::
15943 :number-lines:
15945 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional
15947 // gpr tracking symbols are implicitly set to zero
15949 .text
15950 .globl kern0
15951 .p2align 8
15952 .type kern0,@function
15953 kern0:
15954 // ...
15955 s_endpgm
15956 .Lkern0_end:
15957 .size kern0, .Lkern0_end-kern0
15959 .rodata
15960 .p2align 6
15961 .amdhsa_kernel kern0
15962 // ...
15963 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr
15964 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr
15965 .end_amdhsa_kernel
15967 // reset symbols to begin tracking usage in func1 and kern1
15968 .set .amdgcn.next_free_vgpr, 0
15969 .set .amdgcn.next_free_sgpr, 0
15971 .text
15972 .hidden func1
15973 .global func1
15974 .p2align 2
15975 .type func1,@function
15976 func1:
15977 // ...
15978 s_setpc_b64 s[30:31]
15979 .Lfunc1_end:
15980 .size func1, .Lfunc1_end-func1
15982 .globl kern1
15983 .p2align 8
15984 .type kern1,@function
15985 kern1:
15986 // ...
15987 s_getpc_b64 s[4:5]
15988 s_add_u32 s4, s4, func1@rel32@lo+4
15989 s_addc_u32 s5, s5, func1@rel32@lo+4
15990 s_swappc_b64 s[30:31], s[4:5]
15991 // ...
15992 s_endpgm
15993 .Lkern1_end:
15994 .size kern1, .Lkern1_end-kern1
15996 .rodata
15997 .p2align 6
15998 .amdhsa_kernel kern1
15999 // ...
16000 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr
16001 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr
16002 .end_amdhsa_kernel
16004 These symbols cannot identify connected components in order to automatically
16005 track the usage for each kernel. However, in some cases careful organization of
16006 the kernels and functions in the source file means there is minimal additional
16007 effort required to accurately calculate GPR usage.
16009 Additional Documentation
16010 ========================
16012 .. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__
16013 .. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_
16014 .. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__
16015 .. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__
16016 .. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__
16017 .. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__
16018 .. [AMD-GCN-GFX90A-CDNA2] `AMD Instinct MI200 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA2_Shader_ISA_4February2022.pdf>`__
16019 .. [AMD-GCN-GFX940-GFX942-CDNA3] `AMD Instinct MI300 Instruction Set Architecture <https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/instruction-set-architectures/amd-instinct-mi300-cdna3-instruction-set-architecture.pdf>`__
16020 .. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__
16021 .. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__
16022 .. [AMD-GCN-GFX11-RDNA3] `AMD RDNA 3 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA3_Shader_ISA_December2022.pdf>`__
16023 .. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__
16024 .. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__
16025 .. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__
16026 .. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__
16027 .. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__
16028 .. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__
16029 .. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__
16030 .. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__
16031 .. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__
16032 .. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__
16033 .. [HRF] `Heterogeneous-race-free Memory Models <https://research.cs.wisc.edu/multifacet/papers/asplos14_hrf.pdf>`__
16034 .. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__
16035 .. [MsgPack] `Message Pack <http://www.msgpack.org/>`__
16036 .. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__
16037 .. [SEMVER] `Semantic Versioning <https://semver.org/>`__
16038 .. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__