1 # New frame format documentation
4 This document is primarily intended for developers only.
6 A major change is the support of variable length frames between host and Proxmark3.
7 This is a step especially important for usage over FPC/USART/BT.
11 - [New frame format documentation](#new-frame-format-documentation)
12 - [Table of Contents](#table-of-contents)
13 - [Old format](#old-format)
14 - [New format](#new-format)
15 - [Transition](#transition)
16 - [New format API](#new-format-api)
17 - [On the client, for sending frames](#on-the-client-for-sending-frames)
18 - [On the Proxmark3, for receiving frames](#on-the-proxmark3-for-receiving-frames)
19 - [On the Proxmark3, for sending frames](#on-the-proxmark3-for-sending-frames)
20 - [On the client, for receiving frames](#on-the-client-for-receiving-frames)
21 - [API transition](#api-transition)
23 - [On the Proxmark3, for receiving frames](#on-the-proxmark3-for-receiving-frames-1)
24 - [On the Proxmark3, for sending frames](#on-the-proxmark3-for-sending-frames-1)
25 - [On the client, for sending frames](#on-the-client-for-sending-frames-1)
26 - [On the client, for receiving frames](#on-the-client-for-receiving-frames-1)
27 - [New usart RX FIFO](#new-usart-rx-fifo)
29 - [Reference frames](#reference-frames)
34 Previously, frames were, in both directions like this:
39 uint8_t asBytes[PM3_CMD_DATA_SIZE];
40 uint32_t asDwords[PM3_CMD_DATA_SIZE / 4];
43 with PM3_CMD_DATA_SIZE = 512 and there was no API abstraction, everybody was forging/parsing these frames.
44 So the frame size was fixed, 544 bytes, even for simple ACKs.
45 When snooping the USB transfers, we can observe the host is sending 544b Bulk USB frames while the Proxmark3 is limited by its internal buffers and is sending 128b, 128b, 128b, 128b, 32b, so in total 5 packets.
50 Even if we make the payload part variable in the old format, we've still a minimum of 32 bytes per frame with fields arbitrarily large.
51 So we designed a new format from scratch:
53 For commands being sent to the Proxmark3:
62 * `magic`: arbitrary magic (`PM3a`) to help re-sync if needed
63 * `length`: length of the variable payload, 0 if none, max 512 (PM3_CMD_DATA_SIZE) for now.
64 * `ng`: flag to tell if the data is following the new format (ng) or the old one, see transition notes below
65 * `cmd`: as previously, on 16b as it's enough
66 * `data`: variable length payload
67 * `crc`: either an actual CRC (crc14a) or a Magic placeholder (`a3`)
69 For responses from the Proxmark3:
80 * `magic`: arbitrary magic (`PM3b`) to help re-sync if needed
81 * `length`: length of the variable payload, 0 if none, max 512 (PM3_CMD_DATA_SIZE) for now.
82 * `ng`: flag to tell if the data is following the new format (ng) or the old one, see transition notes below
83 * `status`: a field to send back the status of the command execution
84 * `reason`: details about what the status indicates for the specified command
85 * `cmd`: as previously, on 16b as it's enough
86 * `data`: variable length payload
87 * `crc`: either an actual CRC (crc14a) or a Magic placeholder (`b3`)
89 We used to send an anonymous ACK, now we're replying with the corresponding command name and a status.
90 CRC is optional and on reception, the magic `a3`/`b3` is accepted as placeholder. If it's different then it's checked as a CRC.
91 By default CRC is used over USART and is disabled over USB, on both directions.
93 Internal structures used to handle these packets are:
94 * PacketCommandNGPreamble
95 * PacketCommandNGPostamble
97 * PacketResponseNGPreamble
98 * PacketResponseNGPostamble
101 But they are abstracted from the developer view with a new API. See below.
106 Because it's a long transition to clean all the code from the old format and because we don't want to break stuffs when flashing the bootloader, the old frames are still supported together with the new frames. The old structure is now called `PacketCommandOLD` and `PacketResponseOLD` and it's also abstracted from the developer view with the new API.
111 So the new API is a merge of the old and the new frame formats, to ensure a smooth transition.
113 The boolean `ng` indicates if the structure is storing data from the old or the new format.
114 Old format can come from either old 544b frames or mixed frames (variable length but still with oldargs).
115 After the full transition, we might remove the fields `oldarg` and `ng`.
116 `PacketCommandNG` and `PacketResponseNG` are the structs used by the API, as seen in a previous section, there are other variable-sized packed structs specifically for the data transmission.
121 uint32_t magic; // NG
123 uint64_t oldarg[3]; // OLD
125 uint8_t asBytes[PM3_CMD_DATA_SIZE];
126 uint32_t asDwords[PM3_CMD_DATA_SIZE / 4];
128 bool ng; // does it store NG data or OLD data?
134 uint32_t magic; // NG
138 uint64_t oldarg[3]; // OLD
140 uint8_t asBytes[PM3_CMD_DATA_SIZE];
141 uint32_t asDwords[PM3_CMD_DATA_SIZE / 4];
143 bool ng; // does it store NG data or OLD data?
147 ### On the client, for sending frames
152 void SendCommandNG(uint16_t cmd, uint8_t *data, size_t len);
153 void SendCommandBL(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, void *data, size_t len);
154 void SendCommandOLD(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, void *data, size_t len);
155 void SendCommandMIX(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, void *data, size_t len);
157 So cmds should make the transition from `SendCommandOLD` to `SendCommandNG` to benefit from smaller frames (and armsrc handlers adjusted accordingly of course).
158 `SendCommandBL` is for Bootloader-related activities, see Bootrom section.
159 `SendCommandMIX` is a transition fct: it uses the same API as `SendCommandOLD` but benefits somehow from variable length frames. It occupies at least 24b of data for the oldargs and real data is therefore limited to PM3_CMD_DATA_SIZE - 24 (defined as PM3_CMD_DATA_SIZE_MIX). Besides the size limitation, the receiver handler doesn't know if this was an OLD frame or a MIX frame, it gets its oldargs and data as usual.
160 Warning : it makes sense to move from `SendCommandOLD` to `SendCommandMIX` only for *commands with small payloads*.
161 * otherwise both have about the same size
162 * `SendCommandMIX` has a smaller payload (PM3_CMD_DATA_SIZE_MIX < PM3_CMD_DATA_SIZE) so it's risky to blindly move from OLD to MIX if there is a large payload.
164 Internally these functions prepare the new or old frames and call `uart_communication` which calls `uart_send`.
166 ### On the Proxmark3, for receiving frames
173 `AppMain` calls `receive_ng`(`common/cmd.c`) which calls `usb_read_ng`/`usart_read_ng` to get a `PacketCommandNG`, then passes it to `PacketReceived`.
174 (no matter if it's an old frame or a new frame, check `PacketCommandNG.ng` field to know if there are `oldargs`)
175 `PacketReceive` is the commands broker.
176 Old handlers will still find their stuff in `PacketCommandNG.oldarg` field.
178 ### On the Proxmark3, for sending frames
183 int reply_old(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, const void *data, size_t len);
184 int reply_ng(uint16_t cmd, int8_t status, const uint8_t *data, size_t len);
185 int reply_mix(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, const void *data, size_t len);
186 int reply_reason(uint16_t cmd, int8_t status, int8_t reason, const uint8_t *data, size_t len);
188 So replies should make the transition from `reply_old` to `reply_ng` to benefit from smaller frames (and client reception adjusted accordingly of course).
189 `reply_mix` is a transition fct: it uses the same API as reply_old but benefits somehow from variable length frames. It occupies at least 24b of data for the oldargs and real data is therefore limited to PM3_CMD_DATA_SIZE - 24. Besides the size limitation, the client command doesn't know if this was an OLD frame or a MIX frame, it gets its oldargs and data as usual.
191 Example of a handler that supports both OLD/MIX and NG command styles and replies with the new frame format when it receives new command format:
194 reply_ng(CMD_FOOBAR, PM3_SUCCESS, packet->data.asBytes, packet->length);
196 // reply_old(CMD_ACK, 0, 0, 0, packet->data.asBytes, packet->length);
197 reply_mix(CMD_ACK, 0, 0, 0, packet->data.asBytes, packet->length);
200 ### On the client, for receiving frames
205 WaitForResponseTimeout ⇒ PacketResponseNG
207 `uart_communication` calls `uart_receive` and create a `PacketResponseNG`, then passes it to `PacketResponseReceived`.
208 `PacketResponseReceived` treats it immediately (prints) or stores it with `storeReply`.
209 Commands do `WaitForResponseTimeoutW` (or `dl_it`) which uses `getReply` to fetch responses.
214 In short, to move from one format to the other, we need for each command:
216 * (client TX) `SendCommandOLD` ⇒ `SendCommandNG` (with all stuff in ad-hoc PACKED structs in `data` field)
217 * (pm3 RX) `PacketCommandNG` parsing, from `oldarg` to only the `data` field
218 * (pm3 TX) `reply_old` ⇒ `reply_ng` (with all stuff in ad-hoc PACKED structs in `data` field)
219 * (client RX) `PacketResponseNG` parsing, from `oldarg` to only the `data` field
221 Meanwhile, a fast transition to MIX frames can be done with:
223 * (client TX) `SendCommandOLD` ⇒ `SendCommandMIX` (but check the limited data size PM3_CMD_DATA_SIZE ⇒ PM3_CMD_DATA_SIZE_MIX)
224 * (pm3 TX) `reply_old` ⇒ `reply_mix` (but check the limited data size PM3_CMD_DATA_SIZE ⇒ PM3_CMD_DATA_SIZE_MIX)
229 Bootrom code will still use the old frame format to remain compatible with other repos supporting the old format and because it would hardly gain anything from the new format:
230 * almost all frames convey 512b of payload, so difference in overhead is negligible
231 * bringing flash over usart sounds risky and would be terribly slow anyway (115200 bauds vs. 7M bauds).
233 `SendCommandBL` is the same as `SendCommandOLD` with a different name to be sure not to migrate it.
235 ### On the Proxmark3, for receiving frames
238 (`bootrom/bootrom.c`)
240 usb_read (common/usb_cdc.c) ⇒ UsbPacketReceived (bootrom.c)
241 ⇒ CMD_DEVICE_INFO / CMD_START_FLASH / CMD_FINISH_WRITE / CMD_HARDWARE_RESET
243 also `usb_enable`, `usb_disable` (`common/usb_cdc.c`)
245 ### On the Proxmark3, for sending frames
248 (`bootrom/bootrom.c`)
250 reply_old (bootrom.c) ⇒ usb_write (common/usb_cdc.c)
252 also `usb_enable`, `usb_disable` (`common/usb_cdc.c`)
254 ### On the client, for sending frames
257 Therefore, the flasher client (`client/flasher.c` + `client/flash.c`) must still use these old frames.
258 It uses a few commands in common with current client code:
263 ⇒ CMD_DEVICE_INFO / CMD_START_FLASH / CMD_FINISH_WRITE / CMD_HARDWARE_RESET
265 ### On the client, for receiving frames
268 As usual, old frames are still supported
270 WaitForResponseTimeout ⇒ PacketResponseNG
275 USART code has been rewritten to cope with unknown size packets.
276 * using USART full duplex with double DMA buffer on RX & TX
277 * using internal FIFO for RX
280 * USART is activated all way long from usart_init(), no need to touch it in RX/TX routines: `pUS1->US_PTCR = AT91C_PDC_RXTEN | AT91C_PDC_TXTEN`
282 `usart_writebuffer_sync`:
283 * still using DMA but accepts arbitrary packet sizes
284 * removed unneeded memcpy
285 * wait for DMA buffer to be treated before returning, therefore "sync"
286 * we could make an async version but caller must be sure the DMA buffer remains available!
287 * as it's sync, no need for next DMA buffer
290 * user tells expected packet length
291 * relies on usart_rxdata_available to know if there is data in our FIFO buffer
292 * fetches data from our FIFO
293 * dynamic number of tries (depending on FPC speed) to wait for asked data
295 `usart_rxdata_available`:
296 * polls usart_fill_rxfifo
297 * returns number of bytes available in our FIFO
300 * if next DMA buffer got moved to current buffer (`US_RNCR == 0`), it means one DMA buffer is full
301 * transfer current DMA buffer data to our FIFO
302 * swap to the other DMA buffer
303 * provide the emptied DMA buffer as next DMA buffer
304 * if current DMA buffer is partially filled
305 * transfer available data to our FIFO
306 * remember how many bytes we already copied to our FIFO
311 Reference (before new format):
313 linux usb: #db# USB Transfer Speed PM3 ⇒ Client = 545109 Bytes/s
316 proxspace usb: #db# USB Transfer Speed PM3 ⇒ Client = 233998 Bytes/s
321 USART_BAUD_RATE defined there
323 9600: #db# USB Transfer Speed PM3 ⇒ Client = 934 Bytes/s
324 115200: #db# USB Transfer Speed PM3 ⇒ Client = 11137 Bytes/s
325 460800: #db# USB Transfer Speed PM3 ⇒ Client = 43119 Bytes/s
326 linux usb: #db# USB Transfer Speed PM3 ⇒ Client = 666624 Bytes/s (equiv. to ~7Mbaud)
331 Receiving from USART need more than 30ms as we used on USB
332 else we get errors about partial packet reception
334 FTDI 9600 hw status ⇒ we need 20ms
335 FTDI 115200 hw status ⇒ we need 50ms
336 FTDI 460800 hw status ⇒ we need 30ms
337 BT 115200 hf mf fchk --1k -f file.dic ⇒ we need 140ms
339 # define UART_FPC_CLIENT_RX_TIMEOUT_MS 200
340 # define UART_USB_CLIENT_RX_TIMEOUT_MS 20
341 # define UART_NET_CLIENT_RX_TIMEOUT_MS 500
342 # define UART_TCP_LOCAL_CLIENT_RX_TIMEOUT_MS 40
343 # define UART_UDP_LOCAL_CLIENT_RX_TIMEOUT_MS 20
345 This goes to `uart_posix.c` `timeval` struct
346 and `uart_win32.c` `serial_port_windows` struct
348 It starts at UART_FPC_CLIENT_RX_TIMEOUT_MS and once we detect we're working over USB
349 it's reduced to UART_USB_CLIENT_RX_TIMEOUT_MS.
351 The timeout is configurable by the `hw timeout` command (since v4.17140).
353 Add automatically some communication delay in the `WaitForResponseTimeout` & `dl_it` timeouts.
354 Only when using FPC, timeout = 2* empirically measured delay (FTDI cable).
355 Empirically measured delay (FTDI cable) with "hw ping -l 512" :
363 static size_t communication_delay(void) {
364 if (conn.send_via_fpc_usart) // needed also for Windows USB USART??
365 return 2 * (12000000 / uart_speed);
369 Because some commands send a lot of frames before finishing (hw status, lf read,...),
370 `WaitForResponseTimeout` & `dl_it` timeouts are reset at each packet reception,
371 so timeout is actually counted after latest received packet,
372 it doesn't depend anymore on the number of received packets.
374 It was needed to tune pm3 RX usart `maxtry` :
378 uint32_t usart_read_ng(uint8_t *data, size_t len) {
379 // Empirical max try observed: 3000000 / USART_BAUD_RATE
381 uint32_t tryconstant = 0;
382 #ifdef USART_SLOW_LINK
383 // Experienced up to 13200 tries on BT link even at 460800
386 uint32_t maxtry = 10 * (3000000 / USART_BAUD_RATE) + tryconstant;
389 `DbpStringEx` using `reply_old`:
391 time client/proxmark3 -p /dev/ttyACM0 -c "hw status"
393 time client/proxmark3 -p /dev/ttyUSB0 -b 460800 -c "hw status"
395 time client/proxmark3 -p /dev/ttyUSB0 -b 115200 -c "hw status"
397 time client/proxmark3 -p /dev/ttyUSB0 -b 9600 -c "hw status"
400 `DbpStringEx` using `reply_mix`:
402 time client/proxmark3 -p /dev/ttyUSB0 -b 9600 -c "hw status"
405 `DbpStringEx` using `reply_ng`:
407 time client/proxmark3 -p /dev/ttyACM0 -c "hw status"
409 time client/proxmark3 -p /dev/ttyUSB0 -b 460800 -c "hw status"
411 time client/proxmark3 -p /dev/ttyUSB0 -b 115200 -c "hw status"
413 time client/proxmark3 -p /dev/ttyUSB0 -b 9600 -c "hw status"
416 time client/proxmark3 -p /dev/ttyUSB0 -b 9600 -c "lf read"
418 time client/proxmark3 -p /dev/ttyUSB0 -b 115200 -c "lf read"
421 time client/proxmark3 -p /dev/ttyACM0 -c "mem dump -f foo_usb"
423 time client/proxmark3 -p /dev/ttyUSB0 -b 115200 -c "mem dump -f foo_fpc"
427 Sending multiple commands can still be slow because it waits regularly for incoming RX frames and the timings are quite conservative because of BT (see struct timeval timeout in uart_posix.c, now at 200ms). When one knows there is no response to wait before the next command, he can use the same trick as in the flasher:
430 conn.block_after_ACK = true;
432 if (sending_last_command)
434 conn.block_after_ACK = false;
435 SendCommandOLD / SendCommandMix
436 if (!WaitForResponseTimeout(CMD_ACK, &resp, some_timeout)) {
438 conn.block_after_ACK = false;
444 Or if it's too complex to determine when we're sending the last command:
447 conn.block_after_ACK = true;
449 SendCommandOLD / SendCommandMIX
450 if (!WaitForResponseTimeout(CMD_ACK, &resp, some_timeout)) {
452 conn.block_after_ACK = false;
456 // Disable fast mode and send a dummy command to make it effective
457 conn.block_after_ACK = false;
458 SendCommandNG(CMD_PING, NULL, 0);
459 WaitForResponseTimeout(CMD_ACK, NULL, 1000);
468 * OLD command and reply packets are 544 bytes.
469 * NG & MIX command packets are between 10 and 522 bytes.
470 * NG & MIX reply packets are between 12 and 524 bytes.
473 * sent packets can be 544
474 * received packets are max 128, so 544 = 128+128+128+128+32
477 * sent packets are max 256, so 544 = 256+256+32
478 * received packets are max 512, so 544 = 512+32
480 `hw ping` (old version, mix reply)
482 TestProxmark: SendCommandOLD(CMD_PING, 0, 0, 0, NULL, 0);
483 ->544=0901000000000000000000000000000000000000000000000000000000000000 -> OLD
484 CMD_PING: reply_mix(CMD_ACK, reply_via_fpc, 0, 0, 0, 0);
485 <-36=504d336218000000ff0000000000000000000000000000000000000000000000 <- MIX
487 `hw ping` (intermediate version using MIX)
489 CmdPing SendCommandMIX(CMD_PING, 0, 0, 0, NULL, 0);
490 ->34=504d336118000901000000000000000000000000000000000000000000000000 -> MIX
491 CMD_PING reply_mix(CMD_ACK, reply_via_fpc, 0, 0, 0, 0);
492 <-36=504d336218000000ff0000000000000000000000000000000000000000000000 <- MIX
494 `hw ping` (current NG version)
496 CmdPing SendCommandNG(CMD_PING, data, len);
497 ->10=504d3361008009016133 -> NG
498 CMD_PING reply_ng(CMD_PING, PM3_SUCCESS, packet->data.asBytes, packet->length);
499 <-12=504d33620080000009016233 <- NG
501 `hw ping -l 512` (NG)
503 CmdPing SendCommandNG(CMD_PING, data, len);
504 ->522=504d336100820901000102030405060708090a0b0c0d0e0f1011121314151617 -> NG
505 CMD_PING reply_ng(CMD_PING, PM3_SUCCESS, packet->data.asBytes, packet->length);
506 <-128=504d3362008200000901000102030405060708090a0b0c0d0e0f101112131415 <- NG
507 <-128=767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495
508 <-128=f6f7f8f9fafbfcfdfeff000102030405060708090a0b0c0d0e0f101112131415
509 <-128=767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495
510 <-12=f6f7f8f9fafbfcfdfeff6233