1 .. SPDX-License-Identifier: GPL-2.0
3 ======================================
4 Texas Instruments CPSW ethernet driver
5 ======================================
7 Multiqueue & CBS & MQPRIO
8 =========================
11 The cpsw has 3 CBS shapers for each external ports. This document
12 describes MQPRIO and CBS Qdisc offload configuration for cpsw driver
13 based on examples. It potentially can be used in audio video bridging
14 (AVB) and time sensitive networking (TSN).
16 The following examples were tested on AM572x EVM and BBB boards.
21 Under consideration two examples with AM572x EVM running cpsw driver
24 Several prerequisites:
26 - TX queues must be rated starting from txq0 that has highest priority
27 - Traffic classes are used starting from 0, that has highest priority
28 - CBS shapers should be used with rated queues
29 - The bandwidth for CBS shapers has to be set a little bit more then
30 potential incoming rate, thus, rate of all incoming tx queues has
32 - Real rates can differ, due to discreetness
33 - Map skb-priority to txq is not enough, also skb-priority to l2 prio
34 map has to be created with ip or vconfig tool
35 - Any l2/socket prio (0 - 7) for classes can be used, but for
36 simplicity default values are used: 3 and 2
37 - only 2 classes tested: A and B, but checked and can work with more,
38 maximum allowed 4, but only for 3 rate can be set.
40 Test setup for examples
41 =======================
45 +-------------------------------+
48 |E | MAC 18:03:73:66:87:42 |
49 +-----------------------------+ +--|t | |
50 | | 1 | E | | |h |./tsn_listener -d \ |
51 | Target board: | 0 | t |--+ |0 | 18:03:73:66:87:42 -i eth0 \|
52 | AM572x EVM | 0 | h | | | -s 1500 |
54 | Only 2 classes: |Mb +---| +-------------------------------+
55 | class A, class B | |
56 | | +---| +-------------------------------+
58 | | 0 | t | | | Workstation1 |
59 | | 0 | h |--+ |E | MAC 20:cf:30:85:7d:fd |
61 +-----------------------------+ |h |./tsn_listener -d \ |
62 |0 | 20:cf:30:85:7d:fd -i eth0 \|
65 +-------------------------------+
68 Example 1: One port tx AVB configuration scheme for target board
69 ----------------------------------------------------------------
71 (prints and scheme for AM572x evm, applicable for single port boards)
74 - txq - transmit queue
76 - f - fifo (cpsw fifo)
77 - S - shaper configured
81 +------------------------------------------------------------------+ u
82 | +---------------+ +---------------+ +------+ +------+ | s
84 | | App 1 | | App 2 | | Apps | | Apps | | r
85 | | Class A | | Class B | | Rest | | Rest | |
86 | | Eth0 | | Eth0 | | Eth0 | | Eth1 | | s
87 | | VLAN100 | | VLAN100 | | | | | | | | p
88 | | 40 Mb/s | | 20 Mb/s | | | | | | | | a
89 | | SO_PRIORITY=3 | | SO_PRIORITY=2 | | | | | | | | c
90 | | | | | | | | | | | | | | e
91 | +---|-----------+ +---|-----------+ +---|--+ +---|--+ |
92 +-----|------------------|------------------|--------|-------------+
93 +-+ +------------+ | |
94 | | +-----------------+ +--+
96 +---|-------|-------------|-----------------------|----------------+
97 | +----+ +----+ +----+ +----+ +----+ |
98 | | p3 | | p2 | | p1 | | p0 | | p0 | | k
99 | \ / \ / \ / \ / \ / | e
100 | \ / \ / \ / \ / \ / | r
105 | +----+ +----+ +----+ +----+ | s
106 | |tc0 | |tc1 | |tc2 | |tc0 | | p
107 | \ / \ / \ / \ / | a
108 | \ / \ / \ / \ / | c
114 | +----+ +----+ +----+ +----+ +----+ |
115 | |txq0| |txq1| |txq2| |txq3| |txq4| |
116 | \ / \ / \ / \ / \ / |
117 | \ / \ / \ / \ / \ / |
119 | +-|------|------|------|--+ +--|--------------+ |
120 | | | | | | | Eth0.100 | | Eth1 | |
121 +---|------|------|------|------------------------|----------------+
124 3 2 0-1, 4-7 <- L2 priority |
127 +---|------|------|------|------------------------|----------------+
128 | | | | | |----------+ |
129 | +----+ +----+ +----+ +----+ +----+ |
130 | |dma7| |dma6| |dma5| |dma4| |dma3| |
131 | \ / \ / \ / \ / \ / | c
132 | \S / \S / \ / \ / \ / | p
137 | +----+ +----+ +----+p p+----+ | r
138 | | | | | | |o o| | | i
139 | | f3 | | f2 | | f0 |r r| f0 | | v
140 | |tc0 | |tc1 | |tc2 |t t|tc0 | | e
141 | \CBS / \CBS / \CBS /1 2\CBS / | r
142 | \S / \S / \ / \ / |
144 +------------------------------------------------------------------+
150 // Add 4 tx queues, for interface Eth0, and 1 tx queue for Eth1
151 $ ethtool -L eth0 rx 1 tx 5
152 rx unmodified, ignoring
156 // Check if num of queues is set correctly:
158 Channel parameters for eth0:
164 Current hardware settings:
172 // TX queues must be rated starting from 0, so set bws for tx0 and tx1
173 // Set rates 40 and 20 Mb/s appropriately.
174 // Pay attention, real speed can differ a bit due to discreetness.
175 // Leave last 2 tx queues not rated.
176 $ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate
177 $ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate
181 // Check maximum rate of tx (cpdma) queues:
182 $ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate
191 // Map skb->priority to traffic class:
192 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
193 // Map traffic class to transmit queue:
194 // tc0 -> txq0, tc1 -> txq1, tc2 -> (txq2, txq3)
195 $ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \
196 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@2 hw 1
200 // As two interface sharing same set of tx queues, assign all traffic
201 // coming to interface Eth1 to separate queue in order to not mix it
202 // with traffic from interface Eth0, so use separate txq to send
203 // packets to Eth1, so all prio -> tc0 and tc0 -> txq4
204 // Here hw 0, so here still default configuration for eth1 in hw
205 $ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 1 \
206 map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@4 hw 0
210 // Check classes settings
211 $ tc -g class show dev eth0
212 +---(100:ffe2) mqprio
216 +---(100:ffe1) mqprio
219 +---(100:ffe0) mqprio
222 $ tc -g class show dev eth1
223 +---(100:ffe0) mqprio
228 // Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc
229 // Set it +1 Mb for reserve (important!)
230 // here only idle slope is important, others arg are ignored
231 // Pay attention, real speed can differ a bit due to discreetness
232 $ tc qdisc add dev eth0 parent 100:1 cbs locredit -1438 \
233 hicredit 62 sendslope -959000 idleslope 41000 offload 1
234 net eth0: set FIFO3 bw = 50
238 // Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc:
239 // Set it +1 Mb for reserve (important!)
240 $ tc qdisc add dev eth0 parent 100:2 cbs locredit -1468 \
241 hicredit 65 sendslope -979000 idleslope 21000 offload 1
242 net eth0: set FIFO2 bw = 30
246 // Create vlan 100 to map sk->priority to vlan qos
247 $ ip link add link eth0 name eth0.100 type vlan id 100
248 8021q: 802.1Q VLAN Support v1.8
249 8021q: adding VLAN 0 to HW filter on device eth0
250 8021q: adding VLAN 0 to HW filter on device eth1
251 net eth0: Adding vlanid 100 to vlan filter
255 // Map skb->priority to L2 prio, 1 to 1
256 $ ip link set eth0.100 type vlan \
257 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
261 // Check egress map for vlan 100
262 $ cat /proc/net/vlan/eth0.100
264 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
265 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
269 // Run your appropriate tools with socket option "SO_PRIORITY"
270 // to 3 for class A and/or to 2 for class B
271 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)
272 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500&
273 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500&
277 // run your listener on workstation (should be in same vlan)
278 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)
279 ./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500
280 Receiving data rate: 39012 kbps
281 Receiving data rate: 39012 kbps
282 Receiving data rate: 39012 kbps
283 Receiving data rate: 39012 kbps
284 Receiving data rate: 39012 kbps
285 Receiving data rate: 39012 kbps
286 Receiving data rate: 39012 kbps
287 Receiving data rate: 39012 kbps
288 Receiving data rate: 39012 kbps
289 Receiving data rate: 39012 kbps
290 Receiving data rate: 39012 kbps
291 Receiving data rate: 39012 kbps
292 Receiving data rate: 39000 kbps
296 // Restore default configuration if needed
297 $ ip link del eth0.100
298 $ tc qdisc del dev eth1 root
299 $ tc qdisc del dev eth0 root
300 net eth0: Prev FIFO2 is shaped
301 net eth0: set FIFO3 bw = 0
302 net eth0: set FIFO2 bw = 0
303 $ ethtool -L eth0 rx 1 tx 1
305 Example 2: Two port tx AVB configuration scheme for target board
306 ----------------------------------------------------------------
308 (prints and scheme for AM572x evm, for dual emac boards only)
312 +------------------------------------------------------------------+ u
313 | +----------+ +----------+ +------+ +----------+ +----------+ | s
314 | | | | | | | | | | | | e
315 | | App 1 | | App 2 | | Apps | | App 3 | | App 4 | | r
316 | | Class A | | Class B | | Rest | | Class B | | Class A | |
317 | | Eth0 | | Eth0 | | | | | Eth1 | | Eth1 | | s
318 | | VLAN100 | | VLAN100 | | | | | VLAN100 | | VLAN100 | | p
319 | | 40 Mb/s | | 20 Mb/s | | | | | 10 Mb/s | | 30 Mb/s | | a
320 | | SO_PRI=3 | | SO_PRI=2 | | | | | SO_PRI=3 | | SO_PRI=2 | | c
321 | | | | | | | | | | | | | | | | | e
322 | +---|------+ +---|------+ +---|--+ +---|------+ +---|------+ |
323 +-----|-------------|-------------|---------|-------------|--------+
324 +-+ +-------+ | +----------+ +----+
325 | | +-------+------+ | |
327 +---|-------|-------------|--------------|-------------|-------|---+
328 | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
329 | | p3 | | p2 | | p1 | | p0 | | p0 | | p1 | | p2 | | p3 | | k
330 | \ / \ / \ / \ / \ / \ / \ / \ / | e
331 | \ / \ / \ / \ / \ / \ / \ / \ / | r
332 | \/ \/ \/ \/ \/ \/ \/ \/ | n
334 | | | +----+ +----+ | | | l
336 | +----+ +----+ +----+ +----+ +----+ +----+ | s
337 | |tc0 | |tc1 | |tc2 | |tc2 | |tc1 | |tc0 | | p
338 | \ / \ / \ / \ / \ / \ / | a
339 | \ / \ / \ / \ / \ / \ / | c
340 | \/ \/ \/ \/ \/ \/ | e
341 | | | +-----+ +-----+ | | |
344 | | | | | E E | | | | |
345 | +----+ +----+ +----+ +----+ t t +----+ +----+ +----+ +----+ |
346 | |txq0| |txq1| |txq4| |txq5| h h |txq6| |txq7| |txq3| |txq2| |
347 | \ / \ / \ / \ / 0 1 \ / \ / \ / \ / |
348 | \ / \ / \ / \ / . . \ / \ / \ / \ / |
349 | \/ \/ \/ \/ 1 1 \/ \/ \/ \/ |
350 | +-|------|------|------|--+ 0 0 +-|------|------|------|--+ |
351 | | | | | | | 0 0 | | | | | | |
352 +---|------|------|------|---------------|------|------|------|----+
355 3 2 0-1, 4-7 <-L2 pri-> 0-1, 4-7 2 3
358 +---|------|------|------|---------------|------|------|------|----+
360 | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
361 | |dma7| |dma6| |dma3| |dma2| |dma1| |dma0| |dma4| |dma5| |
362 | \ / \ / \ / \ / \ / \ / \ / \ / | c
363 | \S / \S / \ / \ / \ / \ / \S / \S / | p
364 | \/ \/ \/ \/ \/ \/ \/ \/ | s
365 | | | | +----- | | | | | w
366 | | | | | +----+ | | | |
367 | | | | | | | | | | d
368 | +----+ +----+ +----+p p+----+ +----+ +----+ | r
369 | | | | | | |o o| | | | | | | i
370 | | f3 | | f2 | | f0 |r CPSW r| f3 | | f2 | | f0 | | v
371 | |tc0 | |tc1 | |tc2 |t t|tc0 | |tc1 | |tc2 | | e
372 | \CBS / \CBS / \CBS /1 2\CBS / \CBS / \CBS / | r
373 | \S / \S / \ / \S / \S / \ / |
374 | \/ \/ \/ \/ \/ \/ |
375 +------------------------------------------------------------------+
376 ========================================Eth==========================>
380 // Add 8 tx queues, for interface Eth0, but they are common, so are accessed
381 // by two interfaces Eth0 and Eth1.
382 $ ethtool -L eth1 rx 1 tx 8
383 rx unmodified, ignoring
387 // Check if num of queues is set correctly:
389 Channel parameters for eth0:
395 Current hardware settings:
403 // TX queues must be rated starting from 0, so set bws for tx0 and tx1 for Eth0
404 // and for tx2 and tx3 for Eth1. That is, rates 40 and 20 Mb/s appropriately
405 // for Eth0 and 30 and 10 Mb/s for Eth1.
406 // Real speed can differ a bit due to discreetness
407 // Leave last 4 tx queues as not rated
408 $ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate
409 $ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate
410 $ echo 30 > /sys/class/net/eth1/queues/tx-2/tx_maxrate
411 $ echo 10 > /sys/class/net/eth1/queues/tx-3/tx_maxrate
415 // Check maximum rate of tx (cpdma) queues:
416 $ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate
428 // Map skb->priority to traffic class for Eth0:
429 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
430 // Map traffic class to transmit queue:
431 // tc0 -> txq0, tc1 -> txq1, tc2 -> (txq4, txq5)
432 $ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \
433 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@4 hw 1
437 // Check classes settings
438 $ tc -g class show dev eth0
439 +---(100:ffe2) mqprio
443 +---(100:ffe1) mqprio
446 +---(100:ffe0) mqprio
451 // Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc for Eth0
452 // here only idle slope is important, others ignored
453 // Real speed can differ a bit due to discreetness
454 $ tc qdisc add dev eth0 parent 100:1 cbs locredit -1470 \
455 hicredit 62 sendslope -959000 idleslope 41000 offload 1
456 net eth0: set FIFO3 bw = 50
460 // Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc for Eth0
461 $ tc qdisc add dev eth0 parent 100:2 cbs locredit -1470 \
462 hicredit 65 sendslope -979000 idleslope 21000 offload 1
463 net eth0: set FIFO2 bw = 30
467 // Create vlan 100 to map sk->priority to vlan qos for Eth0
468 $ ip link add link eth0 name eth0.100 type vlan id 100
469 net eth0: Adding vlanid 100 to vlan filter
473 // Map skb->priority to L2 prio for Eth0.100, one to one
474 $ ip link set eth0.100 type vlan \
475 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
479 // Check egress map for vlan 100
480 $ cat /proc/net/vlan/eth0.100
482 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
483 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
487 // Map skb->priority to traffic class for Eth1:
488 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
489 // Map traffic class to transmit queue:
490 // tc0 -> txq2, tc1 -> txq3, tc2 -> (txq6, txq7)
491 $ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 3 \
492 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@2 1@3 2@6 hw 1
496 // Check classes settings
497 $ tc -g class show dev eth1
498 +---(100:ffe2) mqprio
502 +---(100:ffe1) mqprio
505 +---(100:ffe0) mqprio
510 // Set rate for class A - 31 Mbit (tc0, txq2) using CBS Qdisc for Eth1
511 // here only idle slope is important, others ignored, but calculated
512 // for interface speed - 100Mb for eth1 port.
513 // Set it +1 Mb for reserve (important!)
514 $ tc qdisc add dev eth1 parent 100:3 cbs locredit -1035 \
515 hicredit 465 sendslope -69000 idleslope 31000 offload 1
516 net eth1: set FIFO3 bw = 31
520 // Set rate for class B - 11 Mbit (tc1, txq3) using CBS Qdisc for Eth1
521 // Set it +1 Mb for reserve (important!)
522 $ tc qdisc add dev eth1 parent 100:4 cbs locredit -1335 \
523 hicredit 405 sendslope -89000 idleslope 11000 offload 1
524 net eth1: set FIFO2 bw = 11
528 // Create vlan 100 to map sk->priority to vlan qos for Eth1
529 $ ip link add link eth1 name eth1.100 type vlan id 100
530 net eth1: Adding vlanid 100 to vlan filter
534 // Map skb->priority to L2 prio for Eth1.100, one to one
535 $ ip link set eth1.100 type vlan \
536 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
540 // Check egress map for vlan 100
541 $ cat /proc/net/vlan/eth1.100
543 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
544 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
548 // Run appropriate tools with socket option "SO_PRIORITY" to 3
549 // for class A and to 2 for class B. For both interfaces
550 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500&
551 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500&
552 ./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p2 -s 1500&
553 ./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p3 -s 1500&
557 // run your listener on workstation (should be in same vlan)
558 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)
559 ./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500
560 Receiving data rate: 39012 kbps
561 Receiving data rate: 39012 kbps
562 Receiving data rate: 39012 kbps
563 Receiving data rate: 39012 kbps
564 Receiving data rate: 39012 kbps
565 Receiving data rate: 39012 kbps
566 Receiving data rate: 39012 kbps
567 Receiving data rate: 39012 kbps
568 Receiving data rate: 39012 kbps
569 Receiving data rate: 39012 kbps
570 Receiving data rate: 39012 kbps
571 Receiving data rate: 39012 kbps
572 Receiving data rate: 39000 kbps
576 // Restore default configuration if needed
577 $ ip link del eth1.100
578 $ ip link del eth0.100
579 $ tc qdisc del dev eth1 root
580 net eth1: Prev FIFO2 is shaped
581 net eth1: set FIFO3 bw = 0
582 net eth1: set FIFO2 bw = 0
583 $ tc qdisc del dev eth0 root
584 net eth0: Prev FIFO2 is shaped
585 net eth0: set FIFO3 bw = 0
586 net eth0: set FIFO2 bw = 0
587 $ ethtool -L eth0 rx 1 tx 1