1 # New frame format documentation
4 This document is primarily intended for developers only.
6 A major change is the support of variable length frames between host and Proxmark3.
7 This is a step especially important for usage over FPC/USART/BT.
11 - [New frame format documentation](#new-frame-format-documentation)
12 - [Table of Contents](#table-of-contents)
13 - [Old format](#old-format)
14 - [New format](#new-format)
15 - [Transition](#transition)
16 - [New format API](#new-format-api)
17 - [On the client, for sending frames](#on-the-client-for-sending-frames)
18 - [On the Proxmark3, for receiving frames](#on-the-proxmark3-for-receiving-frames)
19 - [On the Proxmark3, for sending frames](#on-the-proxmark3-for-sending-frames)
20 - [On the client, for receiving frames](#on-the-client-for-receiving-frames)
21 - [API transition](#api-transition)
23 - [On the Proxmark3, for receiving frames](#on-the-proxmark3-for-receiving-frames-1)
24 - [On the Proxmark3, for sending frames](#on-the-proxmark3-for-sending-frames-1)
25 - [On the client, for sending frames](#on-the-client-for-sending-frames-1)
26 - [On the client, for receiving frames](#on-the-client-for-receiving-frames-1)
27 - [New usart RX FIFO](#new-usart-rx-fifo)
29 - [Reference frames](#reference-frames)
34 Previously, frames were, in both directions like this:
39 uint8_t asBytes[PM3_CMD_DATA_SIZE];
40 uint32_t asDwords[PM3_CMD_DATA_SIZE / 4];
43 with PM3_CMD_DATA_SIZE = 512 and there was no API abstraction, everybody was forging/parsing these frames.
44 So the frame size was fixed, 544 bytes, even for simple ACKs.
45 When snooping the USB transfers, we can observe the host is sending 544b Bulk USB frames while the Proxmark3 is limited by its internal buffers and is sending 128b, 128b, 128b, 128b, 32b, so in total 5 packets.
50 Even if we make the payload part variable in the old format, we've still a minimum of 32 bytes per frame with fields arbitrarily large.
51 So we designed a new format from scratch:
53 For commands being sent to the Proxmark3:
62 * `magic`: arbitrary magic (`PM3a`) to help re-sync if needed
63 * `length`: length of the variable payload, 0 if none, max 512 (PM3_CMD_DATA_SIZE) for now.
64 * `ng`: flag to tell if the data is following the new format (ng) or the old one, see transition notes below
65 * `cmd`: as previously, on 16b as it's enough
66 * `data`: variable length payload
67 * `crc`: either an actual CRC (crc14a) or a Magic placeholder (`a3`)
69 For responses from the Proxmark3:
79 * `magic`: arbitrary magic (`PM3b`) to help re-sync if needed
80 * `length`: length of the variable payload, 0 if none, max 512 (PM3_CMD_DATA_SIZE) for now.
81 * `ng`: flag to tell if the data is following the new format (ng) or the old one, see transition notes below
82 * `status`: a field to send back the status of the command execution
83 * `cmd`: as previously, on 16b as it's enough
84 * `data`: variable length payload
85 * `crc`: either an actual CRC (crc14a) or a Magic placeholder (`a3`)
87 We used to send an anonymous ACK, now we're replying with the corresponding command name and a status.
88 CRC is optional and on reception, the magic `a3` is accepted as placeholder. If it's different then it's checked as a CRC.
89 By default CRC is user over USART and is disabled over USB, on both directions.
91 Internal structures used to handle these packets are:
92 * PacketCommandNGPreamble
93 * PacketCommandNGPostamble
95 * PacketResponseNGPreamble
96 * PacketResponseNGPostamble
99 But they are abstracted from the developer view with a new API. See below.
104 Because it's a long transition to clean all the code from the old format and because we don't want to break stuffs when flashing the bootloader, the old frames are still supported together with the new frames. The old structure is now called `PacketCommandOLD` and `PacketResponseOLD` and it's also abstracted from the developer view with the new API.
109 So the new API is a merge of the old and the new frame formats, to ensure a smooth transition.
111 The boolean `ng` indicates if the structure is storing data from the old or the new format.
112 Old format can come from either old 544b frames or mixed frames (variable length but still with oldargs).
113 After the full transition, we might remove the fields `oldarg` and `ng`.
114 `PacketCommandNG` and `PacketResponseNG` are the structs used by the API, as seen in a previous section, there are other variable-sized packed structs specifically for the data transmission.
119 uint32_t magic; // NG
121 uint64_t oldarg[3]; // OLD
123 uint8_t asBytes[PM3_CMD_DATA_SIZE];
124 uint32_t asDwords[PM3_CMD_DATA_SIZE / 4];
126 bool ng; // does it store NG data or OLD data?
132 uint32_t magic; // NG
133 int16_t status; // NG
135 uint64_t oldarg[3]; // OLD
137 uint8_t asBytes[PM3_CMD_DATA_SIZE];
138 uint32_t asDwords[PM3_CMD_DATA_SIZE / 4];
140 bool ng; // does it store NG data or OLD data?
144 ### On the client, for sending frames
149 void SendCommandNG(uint16_t cmd, uint8_t *data, size_t len);
150 void SendCommandBL(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, void *data, size_t len);
151 void SendCommandOLD(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, void *data, size_t len);
152 void SendCommandMIX(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, void *data, size_t len);
154 So cmds should make the transition from `SendCommandOLD` to `SendCommandNG` to benefit from smaller frames (and armsrc handlers adjusted accordingly of course).
155 `SendCommandBL` is for Bootloader-related activities, see Bootrom section.
156 `SendCommandMIX` is a transition fct: it uses the same API as `SendCommandOLD` but benefits somehow from variable length frames. It occupies at least 24b of data for the oldargs and real data is therefore limited to PM3_CMD_DATA_SIZE - 24 (defined as PM3_CMD_DATA_SIZE_MIX). Besides the size limitation, the receiver handler doesn't know if this was an OLD frame or a MIX frame, it gets its oldargs and data as usual.
157 Warning : it makes sense to move from `SendCommandOLD` to `SendCommandMIX` only for *commands with small payloads*.
158 * otherwise both have about the same size
159 * `SendCommandMIX` has a smaller payload (PM3_CMD_DATA_SIZE_MIX < PM3_CMD_DATA_SIZE) so it's risky to blindly move from OLD to MIX if there is a large payload.
161 Internally these functions prepare the new or old frames and call `uart_communication` which calls `uart_send`.
163 ### On the Proxmark3, for receiving frames
170 `AppMain` calls `receive_ng`(`common/cmd.c`) which calls `usb_read_ng`/`usart_read_ng` to get a `PacketCommandNG`, then passes it to `PacketReceived`.
171 (no matter if it's an old frame or a new frame, check `PacketCommandNG.ng` field to know if there are `oldargs`)
172 `PacketReceive` is the commands broker.
173 Old handlers will still find their stuff in `PacketCommandNG.oldarg` field.
175 ### On the Proxmark3, for sending frames
180 int16_t reply_ng(uint16_t cmd, int16_t status, uint8_t *data, size_t len)
181 int16_t reply_old(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, void *data, size_t len)
182 int16_t reply_mix(uint64_t cmd, uint64_t arg0, uint64_t arg1, uint64_t arg2, void *data, size_t len)
184 So replies should make the transition from `reply_old` to `reply_ng` to benefit from smaller frames (and client reception adjusted accordingly of course).
185 `reply_mix` is a transition fct: it uses the same API as reply_old but benefits somehow from variable length frames. It occupies at least 24b of data for the oldargs and real data is therefore limited to PM3_CMD_DATA_SIZE - 24. Besides the size limitation, the client command doesn't know if this was an OLD frame or a MIX frame, it gets its oldargs and data as usual.
187 Example of a handler that supports both OLD/MIX and NG command styles and replies with the new frame format when it receives new command format:
190 reply_ng(CMD_FOOBAR, PM3_SUCCESS, packet->data.asBytes, packet->length);
192 // reply_old(CMD_ACK, 0, 0, 0, packet->data.asBytes, packet->length);
193 reply_mix(CMD_ACK, 0, 0, 0, packet->data.asBytes, packet->length);
196 ### On the client, for receiving frames
201 WaitForResponseTimeout ⇒ PacketResponseNG
203 `uart_communication` calls `uart_receive` and create a `PacketResponseNG`, then passes it to `PacketResponseReceived`.
204 `PacketResponseReceived` treats it immediately (prints) or stores it with `storeReply`.
205 Commands do `WaitForResponseTimeoutW` (or `dl_it`) which uses `getReply` to fetch responses.
210 In short, to move from one format to the other, we need for each command:
212 * (client TX) `SendCommandOLD` ⇒ `SendCommandNG` (with all stuff in ad-hoc PACKED structs in `data` field)
213 * (pm3 RX) `PacketCommandNG` parsing, from `oldarg` to only the `data` field
214 * (pm3 TX) `reply_old` ⇒ `reply_ng` (with all stuff in ad-hoc PACKED structs in `data` field)
215 * (client RX) `PacketResponseNG` parsing, from `oldarg` to only the `data` field
217 Meanwhile, a fast transition to MIX frames can be done with:
219 * (client TX) `SendCommandOLD` ⇒ `SendCommandMIX` (but check the limited data size PM3_CMD_DATA_SIZE ⇒ PM3_CMD_DATA_SIZE_MIX)
220 * (pm3 TX) `reply_old` ⇒ `reply_mix` (but check the limited data size PM3_CMD_DATA_SIZE ⇒ PM3_CMD_DATA_SIZE_MIX)
225 Bootrom code will still use the old frame format to remain compatible with other repos supporting the old format and because it would hardly gain anything from the new format:
226 * almost all frames convey 512b of payload, so difference in overhead is negligible
227 * bringing flash over usart sounds risky and would be terribly slow anyway (115200 bauds vs. 7M bauds).
229 `SendCommandBL` is the same as `SendCommandOLD` with a different name to be sure not to migrate it.
231 ### On the Proxmark3, for receiving frames
234 (`bootrom/bootrom.c`)
236 usb_read (common/usb_cdc.c) ⇒ UsbPacketReceived (bootrom.c)
237 ⇒ CMD_DEVICE_INFO / CMD_START_FLASH / CMD_FINISH_WRITE / CMD_HARDWARE_RESET
239 also `usb_enable`, `usb_disable` (`common/usb_cdc.c`)
241 ### On the Proxmark3, for sending frames
244 (`bootrom/bootrom.c`)
246 reply_old (bootrom.c) ⇒ usb_write (common/usb_cdc.c)
248 also `usb_enable`, `usb_disable` (`common/usb_cdc.c`)
250 ### On the client, for sending frames
253 Therefore, the flasher client (`client/flasher.c` + `client/flash.c`) must still use these old frames.
254 It uses a few commands in common with current client code:
259 ⇒ CMD_DEVICE_INFO / CMD_START_FLASH / CMD_FINISH_WRITE / CMD_HARDWARE_RESET
261 ### On the client, for receiving frames
264 As usual, old frames are still supported
266 WaitForResponseTimeout ⇒ PacketResponseNG
271 USART code has been rewritten to cope with unknown size packets.
272 * using USART full duplex with double DMA buffer on RX & TX
273 * using internal FIFO for RX
276 * USART is activated all way long from usart_init(), no need to touch it in RX/TX routines: `pUS1->US_PTCR = AT91C_PDC_RXTEN | AT91C_PDC_TXTEN`
278 `usart_writebuffer_sync`:
279 * still using DMA but accepts arbitrary packet sizes
280 * removed unneeded memcpy
281 * wait for DMA buffer to be treated before returning, therefore "sync"
282 * we could make an async version but caller must be sure the DMA buffer remains available!
283 * as it's sync, no need for next DMA buffer
286 * user tells expected packet length
287 * relies on usart_rxdata_available to know if there is data in our FIFO buffer
288 * fetches data from our FIFO
289 * dynamic number of tries (depending on FPC speed) to wait for asked data
291 `usart_rxdata_available`:
292 * polls usart_fill_rxfifo
293 * returns number of bytes available in our FIFO
296 * if next DMA buffer got moved to current buffer (`US_RNCR == 0`), it means one DMA buffer is full
297 * transfer current DMA buffer data to our FIFO
298 * swap to the other DMA buffer
299 * provide the emptied DMA buffer as next DMA buffer
300 * if current DMA buffer is partially filled
301 * transfer available data to our FIFO
302 * remember how many bytes we already copied to our FIFO
307 Reference (before new format):
309 linux usb: #db# USB Transfer Speed PM3 ⇒ Client = 545109 Bytes/s
312 proxspace usb: #db# USB Transfer Speed PM3 ⇒ Client = 233998 Bytes/s
317 USART_BAUD_RATE defined there
319 9600: #db# USB Transfer Speed PM3 ⇒ Client = 934 Bytes/s
320 115200: #db# USB Transfer Speed PM3 ⇒ Client = 11137 Bytes/s
321 460800: #db# USB Transfer Speed PM3 ⇒ Client = 43119 Bytes/s
322 linux usb: #db# USB Transfer Speed PM3 ⇒ Client = 666624 Bytes/s (equiv. to ~7Mbaud)
327 Receiving from USART need more than 30ms as we used on USB
328 else we get errors about partial packet reception
330 FTDI 9600 hw status ⇒ we need 20ms
331 FTDI 115200 hw status ⇒ we need 50ms
332 FTDI 460800 hw status ⇒ we need 30ms
333 BT 115200 hf mf fchk --1k -f file.dic ⇒ we need 140ms
335 # define UART_FPC_CLIENT_RX_TIMEOUT_MS 200
336 # define UART_USB_CLIENT_RX_TIMEOUT_MS 20
337 # define UART_NET_CLIENT_RX_TIMEOUT_MS 500
338 # define UART_TCP_LOCAL_CLIENT_RX_TIMEOUT_MS 40
339 # define UART_UDP_LOCAL_CLIENT_RX_TIMEOUT_MS 20
341 This goes to `uart_posix.c` `timeval` struct
342 and `uart_win32.c` `serial_port_windows` struct
344 It starts at UART_FPC_CLIENT_RX_TIMEOUT_MS and once we detect we're working over USB
345 it's reduced to UART_USB_CLIENT_RX_TIMEOUT_MS.
347 The timeout is configurable by the `hw timeout` command (since v4.17140).
349 Add automatically some communication delay in the `WaitForResponseTimeout` & `dl_it` timeouts.
350 Only when using FPC, timeout = 2* empirically measured delay (FTDI cable).
351 Empirically measured delay (FTDI cable) with "hw ping -l 512" :
359 static size_t communication_delay(void) {
360 if (conn.send_via_fpc_usart) // needed also for Windows USB USART??
361 return 2 * (12000000 / uart_speed);
365 Because some commands send a lot of frames before finishing (hw status, lf read,...),
366 `WaitForResponseTimeout` & `dl_it` timeouts are reset at each packet reception,
367 so timeout is actually counted after latest received packet,
368 it doesn't depend anymore on the number of received packets.
370 It was needed to tune pm3 RX usart `maxtry` :
374 uint32_t usart_read_ng(uint8_t *data, size_t len) {
375 // Empirical max try observed: 3000000 / USART_BAUD_RATE
377 uint32_t tryconstant = 0;
378 #ifdef USART_SLOW_LINK
379 // Experienced up to 13200 tries on BT link even at 460800
382 uint32_t maxtry = 10 * (3000000 / USART_BAUD_RATE) + tryconstant;
385 `DbpStringEx` using `reply_old`:
387 time client/proxmark3 -p /dev/ttyACM0 -c "hw status"
389 time client/proxmark3 -p /dev/ttyUSB0 -b 460800 -c "hw status"
391 time client/proxmark3 -p /dev/ttyUSB0 -b 115200 -c "hw status"
393 time client/proxmark3 -p /dev/ttyUSB0 -b 9600 -c "hw status"
396 `DbpStringEx` using `reply_mix`:
398 time client/proxmark3 -p /dev/ttyUSB0 -b 9600 -c "hw status"
401 `DbpStringEx` using `reply_ng`:
403 time client/proxmark3 -p /dev/ttyACM0 -c "hw status"
405 time client/proxmark3 -p /dev/ttyUSB0 -b 460800 -c "hw status"
407 time client/proxmark3 -p /dev/ttyUSB0 -b 115200 -c "hw status"
409 time client/proxmark3 -p /dev/ttyUSB0 -b 9600 -c "hw status"
412 time client/proxmark3 -p /dev/ttyUSB0 -b 9600 -c "lf read"
414 time client/proxmark3 -p /dev/ttyUSB0 -b 115200 -c "lf read"
417 time client/proxmark3 -p /dev/ttyACM0 -c "mem dump -f foo_usb"
419 time client/proxmark3 -p /dev/ttyUSB0 -b 115200 -c "mem dump -f foo_fpc"
423 Sending multiple commands can still be slow because it waits regularly for incoming RX frames and the timings are quite conservative because of BT (see struct timeval timeout in uart_posix.c, now at 200ms). When one knows there is no response to wait before the next command, he can use the same trick as in the flasher:
426 conn.block_after_ACK = true;
428 if (sending_last_command)
430 conn.block_after_ACK = false;
431 SendCommandOLD / SendCommandMix
432 if (!WaitForResponseTimeout(CMD_ACK, &resp, some_timeout)) {
434 conn.block_after_ACK = false;
440 Or if it's too complex to determine when we're sending the last command:
443 conn.block_after_ACK = true;
445 SendCommandOLD / SendCommandMIX
446 if (!WaitForResponseTimeout(CMD_ACK, &resp, some_timeout)) {
448 conn.block_after_ACK = false;
452 // Disable fast mode and send a dummy command to make it effective
453 conn.block_after_ACK = false;
454 SendCommandNG(CMD_PING, NULL, 0);
455 WaitForResponseTimeout(CMD_ACK, NULL, 1000);
464 * OLD command and reply packets are 544 bytes.
465 * NG & MIX command packets are between 10 and 522 bytes.
466 * NG & MIX reply packets are between 12 and 524 bytes.
469 * sent packets can be 544
470 * received packets are max 128, so 544 = 128+128+128+128+32
473 * sent packets are max 256, so 544 = 256+256+32
474 * received packets are max 512, so 544 = 512+32
476 `hw ping` (old version, mix reply)
478 TestProxmark: SendCommandOLD(CMD_PING, 0, 0, 0, NULL, 0);
479 ->544=0901000000000000000000000000000000000000000000000000000000000000 -> OLD
480 CMD_PING: reply_mix(CMD_ACK, reply_via_fpc, 0, 0, 0, 0);
481 <-36=504d336218000000ff0000000000000000000000000000000000000000000000 <- MIX
483 `hw ping` (intermediate version using MIX)
485 CmdPing SendCommandMIX(CMD_PING, 0, 0, 0, NULL, 0);
486 ->34=504d336118000901000000000000000000000000000000000000000000000000 -> MIX
487 CMD_PING reply_mix(CMD_ACK, reply_via_fpc, 0, 0, 0, 0);
488 <-36=504d336218000000ff0000000000000000000000000000000000000000000000 <- MIX
490 `hw ping` (current NG version)
492 CmdPing SendCommandNG(CMD_PING, data, len);
493 ->10=504d3361008009016133 -> NG
494 CMD_PING reply_ng(CMD_PING, PM3_SUCCESS, packet->data.asBytes, packet->length);
495 <-12=504d33620080000009016233 <- NG
497 `hw ping -l 512` (NG)
499 CmdPing SendCommandNG(CMD_PING, data, len);
500 ->522=504d336100820901000102030405060708090a0b0c0d0e0f1011121314151617 -> NG
501 CMD_PING reply_ng(CMD_PING, PM3_SUCCESS, packet->data.asBytes, packet->length);
502 <-128=504d3362008200000901000102030405060708090a0b0c0d0e0f101112131415 <- NG
503 <-128=767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495
504 <-128=f6f7f8f9fafbfcfdfeff000102030405060708090a0b0c0d0e0f101112131415
505 <-128=767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495
506 <-12=f6f7f8f9fafbfcfdfeff6233