[mascara-docs.git] / i386 / junos / MIT / course / lectures / all / 6.828 / 2011 / labs / lab6 / index.html
| blob | 2928d149a77cdeeb922cd6abe19ea0ec1363e41f |
2 <html>
3 <head>
10 </script>
11 </head>
13 <body>
17 <p>
18 <b>
22 </b>
23 </p>
27 <p>
28 Now that you have a file system, no self respecting OS should go
29 without a network stack. In this the lab you are going to write a
30 driver for a network interface card. The card will be based on the
31 Intel 82540EM chip, also known as the E1000.
32 </p>
36 <p>
37 Use Git to commit your Lab 5 source (if you haven't already), fetch
38 the latest version of the course
41 </p>
42 <pre>
46 nothing to commit (working directory clean)
48 Already up-to-date.
50 Branch lab6 set up to track remote branch refs/remotes/origin/lab6.
51 Switched to a new branch "lab6"
53 Merge made by recursive.
54 fs/fs.c | 42 +++++++++++++++++++
56 athena%
57 </pre>
59 <p>
60 The network card driver, however, will not be enough to get your OS hooked up to the
61 Internet. In the new lab6 code, we have provided you with a network stack
62 and a network server. As in previous labs, use git to grab the code for
63 this lab, merge in your own code, and explore the contents of the new
65 </p>
67 <p>
68 In addition to writing the driver, you will need to create a system call
69 interface to give access to your driver. You will implement missing network
70 server code to transfer packets between the network stack and your driver.
71 You will also tie everything together by finishing a web server. With the new
72 web server you will be able to serve files from your file system.
73 </p>
75 <p>
76 Much of the kernel device driver code you will have to write yourself from
77 scratch. This lab provides much less guidance than previous labs:
78 there are no skeleton files, no system call interfaces written in
79 stone, and many design decisions are left up to you. For
80 this reason, we recommend that you read the entire assignment write up
81 before starting any individual exercises.
82 Many students find this lab more difficult than previous labs, so
83 please plan your time accordingly.
84 </p>
88 <p>
89 As before, you will need
90 to do all of the regular exercises described in the lab and <i>at
91 least one</i> challenge problem. Write up brief answers to the
92 questions posed in the lab and a description of your challenge
94 </p>
98 <p>
99 We will be using QEMU's user mode network stack since it requires no administrative
100 privileges to run. QEMU's documentation has more about user-net
101 <a href="http://wiki.qemu.org/download/qemu-doc.html#Using-the-user-mode-network-stack">here</a>.
102 We've updated the makefile to enable QEMU's user-mode network stack
103 and the virtual E1000 network card.
104 </p>
106 <p>
107 By default, QEMU provides a virtual router running on IP 10.0.2.2 and
108 will assign JOS the IP address 10.0.2.15. To keep things simple, we
110 </p>
112 <p>
113 While QEMU's virtual network allows JOS to make arbitrary connections
114 out to the Internet, JOS's 10.0.2.15 address has no meaning outside
115 the virtual network running inside QEMU (that is, QEMU acts as a NAT),
116 so we can't connect directly to servers running inside JOS, even from
117 the host running QEMU. To address this, we configure QEMU to run a
119 through to some port in JOS and shuttles data back and forth between
120 your real host and the virtual network.
121 </p>
123 <p>
125 collisions on shared Athena machines, the makefile generates
126 forwarding ports for these based on your user ID. To find out what
127 ports QEMU is forwarding to on your development host, run <kbd>make
128 which-ports</kbd>. For convenience, the makefile also provides
130 interact directly with servers running on these ports in your
131 terminal. (These targets only connect to a running QEMU instance; you
132 must start QEMU itself separately.)
133 </p>
137 <p>
138 The makefile also configures QEMU's network stack to record all
140 directory.
141 </p>
143 <p>
145 </p>
147 <pre>
149 </pre>
151 <p>
152 Alternatively, you can use <a
154 the pcap file. Wireshark also knows how to decode and inspect
155 hundreds of network protocols. If you're on Athena, you'll have to
156 use Wireshark's predecessor, ethereal, which is in the sipbnet locker.
157 </p>
161 <p>
162 We are very lucky to be using emulated hardware. Since the E1000 is running in
163 software, the emulated E1000 can report to us, in a user readable format, its
164 internal state and any problems it encounters. Normally, such a luxury would
165 not be available to a driver developer writing with bare metal.
166 </p>
168 <p>
169 The E1000 can produce a lot of debug output, so you have to enable
170 specific logging channels. Some channels you might find useful are:
171 </p>
183 registers.</td></tr>
184 </table>
189 <p>
191 </p>
193 <p>
194 You can take debugging using software emulated hardware one step
195 further. If you are ever stuck and do not understand why the E1000 is
196 not responding the way you would expect, you can look at QEMU's E1000
198 </p>
202 <p>
203 Writing a network stack from scratch is hard work. Instead, we will be using
204 lwIP, an open source lightweight TCP/IP protocol suite that among many things
205 includes a network stack. You can find more information on lwIP
207 as we are concerned, lwIP is a black box that implements a BSD socket interface
208 and has a packet input port and packet output port.
209 </p>
211 <p>
212 The network server is actually a combination of four environments:
213 </p>
215 <ul>
220 </ul>
222 <p>
223 The following diagram shows the different environments and their relationships.
224 The diagram shows the entire system including the device driver, which will be
225 covered later. In this lab, you will implement the parts highlighted
226 in green.
227 </p>
233 <p>
234 The core network server environment is composed of the socket call dispatcher
235 and lwIP itself. The socket call dispatcher works exactly like the file server.
237 messages to the core network environment. If you look at
241 looking for this special environment type.
242 For each user environment IPC, the dispatcher in the network server
243 calls the appropriate BSD socket
244 interface function provided by lwIP on behalf of the user.
245 </p>
247 <p>
250 which provides a file descriptor-based sockets API. Thus, user
251 environments refer to sockets via file descriptors, just
252 like how they referred to on-disk files. A number of operations
257 server maintained internal unique ID's for all open files, lwIP also
258 generates unique ID's for all open sockets. In both the file server
259 and the network server, we
261 per-environment file descriptors to these unique ID spaces.
262 </p>
264 <p>
265 Even though it may seem that the IPC dispatchers of the file server and network
266 server act the same, there is a key difference. BSD socket calls like
268 dispatcher were to let lwIP execute one of these blocking calls, the dispatcher
269 would also block and there could only be one outstanding network call
270 at a time for the whole system. Since this is unacceptable, the network
271 server uses user-level threading to avoid blocking the entire server
272 environment. For every incoming IPC message, the dispatcher creates a
273 thread and processes the request in the newly created thread. If the thread
274 blocks, then only that thread is put to sleep while other threads continue
275 to run.
276 </p>
278 <p>
279 In addition to the core network environment there are three helper
280 environments. Besides accepting messages from user applications, the
281 core network environment's dispatcher also accepts messages from the
282 input and timer environments.
283 </p>
287 <p>
288 When servicing user environment socket calls, lwIP will generate packets for
289 the network card to transmit. LwIP will send each packet to be transmitted to
291 with the packet attached in the page argument of the IPC message. The output
292 environment is responsible for accepting
293 these messages and forwarding the packet on to the device driver via the
294 system call interface that you will soon create.
295 </p>
299 <p>
300 Packets received by the network card
301 need to be injected into lwIP. For every packet received by the device driver,
302 the input environment pulls the packet out of kernel space (using kernel
303 system calls that you will implement) and sends the packet to the core
305 </p>
307 <p>
308 The packet input functionality is separated from the core network environment
309 because JOS makes it hard to simultaneous accept IPC messages and poll or wait
311 system call in JOS that would allow environments to monitor multiple input
312 sources to identify which input is ready to be processed.
313 </p>
315 <p>
317 will see that both need to be implemented. This is mainly because the
318 implementation depends on your system call interface. You will write the code
319 for the two helper environments after you implement the driver and system call
320 interface.
321 </p>
325 <p>
326 The timer
328 core network server notifying it that a timer has expired. The timer messages
329 from this thread are used by lwIP to implement various network timeouts.
330 </p>
334 <p>
335 Your kernel does not have a notion of time, so we need to add
336 it. There is
337 currently a clock interrupt that is generated by the hardware every 10ms. On
338 every clock interrupt we can increment a variable to indicate that time has
340 yet fully integrated into your kernel.
341 </p>
348 access to the time.
349 </p></div>
351 <p>
353 code. You
355 The "-DTEST_NO_NS" disables starting the network server environment
356 because it will panic at this point in the lab.
357 </p>
361 <p>
362 Writing a driver requires knowing in depth the hardware and the interface
363 presented to the software. The lab text will provide a high-level
364 overview of how to interface with the E1000, but you'll need to make
365 extensive use of Intel's manual while writing your driver.
366 </p>
371 Manual</a> for the E1000. This manual covers several closely related
375 To write your driver, you'll need to be familiar with chapters 3 and
377 to use chapter 13 as reference. The other chapters mostly cover
378 components of the E1000 that your driver won't have to interact
379 with. Don't worry about the details right now; just get a feel for
380 how the document is structured so you can find things later.</p>
382 <p>While reading the manual, keep in mind that the E1000 is a
383 sophisticated device with many advanced features. A working E1000
384 driver only needs a fraction of the features and interfaces that the
385 NIC provides. Think carefully about the easiest way to interface
386 with the card. We strongly recommend that you get a basic driver
387 working before taking advantage of the advanced features.</p>
388 </div>
392 <p>
393 The E1000 is a PCI device, which means it plugs into the PCI bus on
394 the motherboard. The PCI bus has address, data, and interrupt lines,
395 and allows the CPU to communicate with PCI devices and PCI devices
396 to read and write memory.
397 A PCI device needs to be discovered and
398 initialized before it can be used. Discovery is the process of walking the PCI
399 bus looking for attached devices. Initialization is the process of
400 allocating I/O and memory space as well as negotiating the IRQ line for the
401 device to use.
402 </p>
404 <p>
406 To perform PCI initialization during boot, the PCI code walks the PCI
407 bus looking for devices. When it finds
408 a device, it reads its vendor ID and device ID and uses these two values as a key
411 </p>
413 <pre>
414 struct pci_driver {
415 uint32_t key1, key2;
416 int (*attachfn) (struct pci_func *pcif);
417 };
418 </pre>
420 <p>
421 If the discovered device's vendor ID and device ID match an entry in
423 perform device initialization.
424 (Devices can also be identified by class, which is
426 </p>
428 <p>
430 PCI card can expose multiple functions, though the E1000 exposes only
431 one. Here is how we represent a PCI function in JOS:
432 </p>
434 <pre>
435 struct pci_func {
436 struct pci_bus *bus;
438 uint32_t dev;
439 uint32_t func;
441 uint32_t dev_id;
442 uint32_t dev_class;
444 uint32_t reg_base[6];
445 uint32_t reg_size[6];
446 uint8_t irq_line;
447 };
448 </pre>
450 <p>
451 The above structure reflects some of the entries found in Table
453 developer manual. The last three entries of
455 record the negotiated memory, I/O, and interrupt resources for the
456 device.
458 information for up to six Base Address Registers or BARs.
460 memory-mapped I/O regions (or base I/O ports for I/O port resources),
464 assigned to the device for interrupts. The specific meanings of the
466 </p>
468 <p>
469 When the attach function of a device is called, the
471 not yet determined the resources allocated to the device, such as address space and
472 an IRQ line, and, thus, the last three elements of the <code>struct
473 pci_func</code> structure are not yet filled in. The attach function should call
475 negotiate these resources, and fill in the <code>struct
476 pci_func</code>.
477 </p>
481 Implement an attach function to initialize the E1000. Add an entry to the
483 your function if a matching PCI device is found (be sure to put it
485 table). You can find the
486 vendor ID and device ID of the 82540EM that QEMU emulates in section
487 5.2. You
488 should also see these listed when JOS scans the PCI bus while booting.
489 </p>
490 <p>
491 For now, just enable the E1000 device via
493 throughout the lab.
494 </p>
495 <p>
499 other places in the kernel.
500 </p>
501 <p>
502 When you boot your kernel, you should see it print that the PCI
503 function of the E1000 card was enabled. Your code should now pass the
505 </p>
506 </div>
510 <p>
512 (MMIO). You've seen this twice before in JOS: both the CGA console
513 and the LAPIC are devices that you control and query by writing to
514 and reading from "memory". But these reads and writes don't go to
515 DRAM; they go directly to these devices.
516 </p>
518 <p>
520 E1000 and stores its base and size in BAR 0 (that is,
523 which means you'll have to do something to access it via virtual
524 addresses. Since MMIO regions are assigned very high physical
526 access it because of JOS's 256MB limit. Thus, you'll have to create
527 a new memory mapping. It's up to you where to put this mapping.
529 KSTACKTOP and KERNBASE; or you could map it well above KERNBASE (but
530 don't overwrite the mapping used by the LAPIC). Since PCI device
531 initialization happens before JOS creates user environments, you can
533 available.
534 </p>
538 create a virtual memory mapping for the E1000's BAR 0. Since this
539 is device memory and not regular DRAM, you'll have to tell the CPU
540 that it isn't safe to cache access to this memory. Luckily, the
541 page tables provide bits for this purpose; simply create the mapping
542 with PTE_PCD|PTE_PWT (cache-disable and write-through). (If you're
543 interested in more details on this, see section 10.5 of IA32 volume
546 <p>You'll want to record where you made this mapping in a variable
547 so you can later access the registers you just mapped. Take a look
549 example of one way to do this. If you do use a pointer to the
550 device register mapping, be sure to declare it
552 values and reorder accesses to this memory.</p>
554 <p>To test your mapping, try printing out the device status register
557 indicates a full duplex link is up at 1000 MB/s, among other
558 things.</p>
559 </div>
561 <p>
562 Hint: You'll need a lot of constants, like the locations of
563 registers and values of bit masks. Trying to copy these out of the
564 developer's manual is error-prone and mistakes can lead to painful
565 debugging sessions. We recommend instead using QEMU's <a
567 don't recommend copying it in verbatim, because it defines far more
568 than you actually need and may not define things in the way you
569 need, but it's a good starting point.
570 </p>
574 <p>
575 You could imagine transmitting and receiving packets by writing and
576 reading from the E1000's registers, but this would be slow and would
577 require the E1000 to buffer packet data internally. Instead, the
579 packet data directly from memory without involving the CPU. The
580 driver is responsible for allocating memory for the transmit and
581 receive queues, setting up DMA descriptors, and configuring the
582 E1000 with the location of these queues, but everything after that
583 is asynchronous. To transmit a packet, the driver copies it into
584 the next DMA descriptor in the transmit queue and informs the E1000
585 that another packet is available; the E1000 will copy the data out
586 of the descriptor when there is time to send the packet. Likewise,
587 when the E1000 receives a packet, it copies it into the next DMA
588 descriptor in the receive queue, which the driver can read from at
589 its next opportunity.
590 </p>
592 <p>
593 The receive and transmit queues are very similar at a high level.
595 structure of these descriptors varies, each descriptor contains some
596 flags and the physical address of a buffer containing packet data
597 (either packet data for the card to send, or a buffer allocated by
598 the OS for the card to write a received packet to).
599 </p>
601 <!-- XXX Figure of a descriptor? -->
603 <p>
604 The queues are implemented as circular arrays, meaning that when the
605 card or the driver reach the end of the array, it wraps back around
607 pointer</i> and the contents of the queue are the descriptors
608 between these two pointers. The hardware always consumes
609 descriptors from the head and moves the head pointer, while the
610 driver always add descriptors to the tail and moves the tail
611 pointer. The descriptors in the transmit queue represent packets
612 waiting to be sent (hence, in the steady state, the transmit queue
613 is empty). For the receive queue, the descriptors in the queue are
614 free descriptors that the card can receive packets into (hence, in
615 the steady state, the receive queue consists of all available
616 receive descriptors).
617 </p>
619 <!-- XXX Figure? Cite sections 3.2.6 and section 3.4? -->
621 <p>
622 The pointers to these arrays as well as the addresses of the packet
624 because hardware performs DMA directly to and from physical RAM
625 without going through the MMU.
626 </p>
630 <p>
631 The transmit and receive functions of the E1000 are basically
632 independent of each other, so we can work on one at a time. We'll
633 attack transmitting packets first simply because we can't test
634 receive without transmitting an "I'm here!" packet first.
635 </p>
637 <p>
638 First, you'll have to initialize the card to transmit, following the
639 steps described in section 14.5 (you don't have to worry about the
640 subsections). The first step of transmit initialization is setting
641 up the transmit queue. The precise structure of the queue is
642 described in section 3.4 and the structure of the descriptors is
643 described in section 3.3.3. We won't be using the TCP offload
644 features of the E1000, so you can focus on the "legacy transmit
645 descriptor format." You should read those sections now and
646 familiarize yourself with these structures.
647 </p>
651 <p>
653 the E1000's structures. As you've seen with structures like the
655 precisely layout data in memory. C can insert padding between
656 fields, but the E1000's structures are laid out such that this
657 shouldn't be a problem. If you do encounter field alignment
658 problems, look into GCC's "packed" attribute.
659 </p>
661 <p>
662 As an example, consider the legacy transmit descriptor given in
664 </p>
667 63 48 47 40 39 32 31 24 23 16 15 0
668 +---------------------------------------------------------------+
669 | Buffer address |
670 +---------------+-------+-------+-------+-------+---------------+
671 | Special | CSS | Status| Cmd | CSO | Length |
672 +---------------+-------+-------+-------+-------+---------------+
673 </pre>
675 <p>
676 The first byte of the structure starts at the top right, so to
677 convert this into a C struct, read from right to left, top to
678 bottom. If you squint at it right, you'll see that all of the
679 fields even fit nicely into a standard-size types:
680 </p>
682 <pre>
683 struct tx_desc
684 {
685 uint64_t addr;
686 uint16_t length;
687 uint8_t cso;
688 uint8_t cmd;
689 uint8_t status;
690 uint8_t css;
691 uint16_t special;
692 };
693 </pre>
695 <p>
696 Your driver will have to reserve memory for the transmit descriptor
697 array and the packet buffers pointed to by the transmit descriptors.
698 There are several ways to do this, ranging from dynamically
699 allocating pages to simply declaring them in global variables.
700 Whatever you choose, keep in mind that the E1000 accesses physical
701 memory directly, which means any buffer it accesses must be
702 contiguous in physical memory.
703 </p>
705 <p>
706 There are also multiple ways to handle the packet buffers. The
707 simplest, which we recommend starting with, is to reserve space for
708 a packet buffer for each descriptor during driver initialization and
709 simply copy packet data into and out of these pre-allocated buffers.
710 The maximum size of an Ethernet packet is 1518 bytes, which bounds
711 how big these buffers need to be. More sophisticated drivers could
712 dynamically allocate packet buffers (e.g., to reduce memory overhead
713 when network usage is low) or even pass buffers directly provided by
714 user space (a technique known as "zero copy"), but it's good to
715 start simple.
716 </p>
720 initialization steps described in section 14.5 (but not its
721 subsections). Use section 13 as a reference for the registers the
723 reference to the transmit descriptors and transmit descriptor
724 array.</p>
726 <p>Be mindful of the alignment requirements on the transmit
727 descriptor array and the restrictions on length of this array.
728 Since TDLEN must be 128-byte aligned and each transmit descriptor is
729 16 bytes, your transmit descriptor array will need some multiple of
731 or our tests won't be able to test transmit ring overflow.</p>
733 <p>For the TCTL.COLD, you can assume full-duplex operation.
737 </div>
739 <p>
741 see an "e1000: tx disabled" message when you set the TDT register
742 (since this happens before you set TCTL.EN) and no further "e1000"
743 messages.
744 </p>
746 <p>
747 Now that transmit is initialized, you'll have to write the code to
748 transmit a packet and make it accessible to user space via a system
749 call. To transmit a packet, you have to add it to the tail of the
750 transmit queue, which means copying the packet data into the next
751 packet buffer and then updating the TDT (transmit descriptor tail)
752 register to inform the card that there's another packet in the
754 descriptor array, not a byte offset; the documentation isn't very
755 clear about this.)
756 </p>
758 <p>
759 However, the transmit queue is only so big. What happens if the
760 card has fallen behind transmitting packets and the transmit queue
761 is full? In order to detect this condition, you'll need some
762 feedback from the E1000. Unfortunately, you can't just use the TDH
763 (transmit descriptor head) register; the documentation explicitly
764 states that reading this register from software is unreliable.
765 However, if you set the RS bit in the command field of a transmit
766 descriptor, then, when the card has transmitted the packet in that
767 descriptor, the card will set the DD bit in the status field of the
768 descriptor. If a descriptor's DD bit is set, you know it's safe to
769 recycle that descriptor and use it to transmit another packet.
770 </p>
772 <p>
773 What if the user calls your transmit system call, but the DD bit of
774 the next descriptor isn't set, indicating that the transmit queue is
775 full? You'll have to decide what to do in this situation. You
776 could simply drop the packet. Network protocols are resilient to
777 this, but if you drop a large burst of packets, the protocol may not
778 recover. You could tell the user environment that it has to retry,
780 advantage of pushing back on the environment generating the data.
781 You could spin in the driver until a transmit descriptor frees up,
782 but this may introduce serious performance problems since the JOS
783 kernel isn't designed to block. Finally, you could put the
784 transmitting environment to sleep and request that the card send an
785 interrupt when transmit descriptors free up. We recommend that you
786 start with whatever you think is simplest.
787 </p>
791 transmit a packet by checking that the next descriptor is free,
792 copying the packet data into the next descriptor, and updating TDT.
793 Make sure you handle the transmit queue being full.</p>
794 </div>
796 <p>
797 Now would be a good time to test your packet transmit code. Try
798 transmitting just a few packets by directly calling your transmit
799 function from the kernel. You don't have to create packets that
800 conform to any particular network protocol in order to test this.
802 should see something like
803 </p>
804 <pre>
806 ...
807 </pre>
808 <p>
809 as you transmit packets. Each line gives the index in the transmit
810 array, the buffer address of that transmit descriptor, the
811 cmd/CSO/length fields, and the special/CSS/status fields.
812 If QEMU doesn't print the values you expected from your transmit
813 descriptor, check that you're filling in the right descriptor and
814 that you configured TDBAL and TDBAH correctly.
815 If you get "e1000: TDH wraparound @0, TDT
816 x, TDLEN y" messages, that means the E1000 ran all the way through
817 the transmit queue without stopping (if QEMU didn't check this, it
818 would enter an infinite loop), which probably means you aren't
819 manipulating TDT correctly. If you get lots of "e1000: tx disabled"
820 messages, then you didn't set the transmit control register right.
821 </p>
823 <p>
825 to see the packet data that you transmitted. If you saw the
826 expected "e1000: index" messages from QEMU, but your packet capture
827 is empty, double check that you filled in every necessary field and
828 bit in your transmit descriptors (the E1000 probably went through
829 your transmit descriptors, but didn't think it had to send
830 anything).
831 </p>
835 lets you transmit packets from user space. The exact interface is
836 up to you. Don't forget to check any pointers passed to the kernel
837 from user space.</p>
838 </div>
842 <p>
843 Now that you have a system call interface to the transmit side of your device
844 driver, it's time to send packets. The output helper environment's goal is to
846 send the packets accompanying these IPC message to the network device driver
850 network system. Each IPC will include a page consisting of a
854 </p>
855 <pre>
856 struct jif_pkt {
857 int jp_len;
858 char jp_data[0];
859 };
860 </pre>
862 <p>
864 subsequent bytes on the IPC page are dedicated to the packet contents.
866 struct is a common C trick (some would say abomination) for
867 representing buffers without pre-determined lengths. Since C doesn't
868 do array bounds checking, as long as you ensure there's enough unused
870 were an array of any size.
871 </p>
873 <p>
874 Be aware of the interaction between the device driver, the output environment
875 and the core network server when there is no more space in the device driver's
876 transmit queue. The core network server sends packets to the output environment
877 using IPC. If the output environment is suspended due to a send packet system
878 call because the driver has no more buffer space for new packets, the core
879 network server will block waiting for the output server to accept the IPC call.
880 </p>
885 </p></div>
887 <p>
889 without involving the whole network server. Try running
891 see something like
892 </p>
893 <pre>
894 Transmitting packet 0
896 Transmitting packet 1
898 ...
899 </pre>
900 <p>
902 </p>
903 <pre>
904 reading from file qemu.pcap, link-type EN10MB (Ethernet)
909 ...
910 </pre>
911 <p>
912 To test with a larger packet count, try
913 <kbd>make E1000_DEBUG=TXERR,TX
915 overflows your transmit ring, double check that you're handling the
916 DD status bit correctly and that you've told the hardware to set the
917 DD status bit (using the RS command bit).
918 </p>
920 <p>
922 grade</kbd>.
923 </p>
927 <ol>
928 <li>
929 How did you structure your transmit implementation? In
930 particular, what do you do if the transmit ring is full?
931 </li>
932 </ol>
933 </div>
939 <p>
940 Just like you did for transmitting packets, you'll have to configure
941 the E1000 to receive packets and provide a receive descriptor queue
942 and receive descriptors. Section 3.2 describes how packet reception
943 works, including the receive queue structure and receive
944 descriptors, and the initialization process is detailed in section
945 14.4.
946 </p>
950 can ignore anything about interrupts and checksum offloading (you
951 can return to these sections if you decide to use these features
952 later), and you don't have to be concerned with the details of
953 thresholds and how the card's internal caches work.</p>
954 </div>
956 <p>
957 The receive queue is very similar to the transmit queue, except that
958 it consists of empty packet buffers waiting to be filled with
959 incoming packets. Hence, when the network is idle, the transmit
960 queue is empty (because all packets have been sent), but the receive
961 queue is full (of empty packet buffers).
962 </p>
964 <p>
965 When the E1000 receives a packet, it first checks if it matches the
966 card's configured filters (for example, to see if the packet is
967 addressed to this E1000's MAC address) and ignores the packet if it
968 doesn't match any filters. Otherwise, the E1000 tries to retrieve
969 the next receive descriptor from the head of the receive queue. If
970 the head (RDH) has caught up with the tail (RDT), then the receive
971 queue is out of free descriptors, so the card drops the packet. If
972 there is a free receive descriptor, it copies the packet data into
973 the buffer pointed to by the descriptor, sets the descriptor's DD
974 (Descriptor Done) and EOP (End of Packet) status bits, and
975 increments the RDH.
976 </p>
978 <p>
979 If the E1000 receives a packet that is larger than the packet buffer
980 in one receive descriptor, it will retrieve as many descriptors as
981 necessary from the receive queue to store the entire contents of the
982 packet. To indicate that this has happened, it will set the DD
983 status bit on all of these descriptors, but only set the EOP status
984 bit on the last of these descriptors. You can either deal with this
985 possibility in your driver, or simply configure the card to not
987 sure your receive buffers are large enough to store the largest
988 possible standard Ethernet packet (1518 bytes).
989 </p>
993 and configure the E1000 by following the process in section 14.4.
994 You don't have to support "long packets" or multicast. For now,
995 don't configure the card to use interrupts; you can change that
996 later if you decide to use receive interrupts. Also, configure the
997 E1000 to strip the Ethernet CRC, since the grade script expects it
998 to be stripped.</p>
1001 have to configure the Receive Address Registers (RAL and RAH) with
1002 the card's own MAC address in order to accept packets addressed to
1003 that card. You can simply hard-code QEMU's default MAC address of
1005 here too doesn't make things any worse). Be very careful with the
1006 byte order; MAC addresses are written from lowest-order byte to
1010 <p>The E1000 only supports a specific set of receive buffer sizes
1011 (given in the description of RCTL.BSIZE in 13.4.22). If you make
1012 your receive packet buffers large enough and disable long packets,
1013 you won't have to worry about packets spanning multiple receive
1014 buffers. Also, remember that, just like for transmit, the receive
1015 queue and the packet buffers must be contiguous in physical
1016 memory.</p>
1017 </div>
1019 <p>
1020 You can do a basic test of receive functionality now, even without
1021 writing the code to receive packets. Run
1024 Protocol) announcement packet, which QEMU will automatically reply
1025 to. Even though your driver can't receive this reply yet, you
1026 should see a "e1000: unicast match[0]: 52:54:00:12:34:56" message,
1027 indicating that a packet was received by the E1000 and matched the
1028 configured receive filter. If you see a "e1000: unicast mismatch:
1029 52:54:00:12:34:56" message instead, the E1000 filtered out the
1030 packet, which means you probably didn't configure RAL and RAH
1031 correctly. Make sure you got the byte ordering right and didn't
1032 forget to set the "Address Valid" bit in RAH. If you don't get any
1033 "e1000" messages, you probably didn't enable receive correctly.
1034 </p>
1036 <p>
1037 Now you're ready to implement receiving packets. To receive a
1038 packet, your driver will have to keep track of which descriptor it
1039 expects to hold the next received packet (hint: depending on your
1040 design, there's probably already be a register in the E1000 keeping
1041 track of this). Similar to transmit, the documentation states that
1042 the RDH register cannot be reliably read from software, so in order
1043 to determine if a packet has been delivered to this descriptor's
1044 packet buffer, you'll have to read the DD status bit in the
1045 descriptor. If the DD bit is set, you can copy the packet data out
1046 of that descriptor's packet buffer and then tell the card that the
1047 descriptor is free by updating the queue's tail index, RDT.
1048 </p>
1050 <p>
1051 If the DD bit isn't set, then no packet has been received. This is
1052 the receive-side equivalent of when the transmit queue was full, and
1053 there are several things you can do in this situation. You can
1054 simply return a "try again" error and require the caller to retry.
1055 While this approach works well for full transmit queues because
1056 that's a transient condition, it is less justifiable for empty
1057 receive queues because the receive queue may remain empty for long
1058 stretches of time. A second approach is to suspend the calling
1059 environment until there are packets in the receive queue to process.
1061 in the IPC case, since we have only one kernel stack per CPU, as
1062 soon as we leave the kernel the state on the stack will be lost. We
1063 need to set a flag indicating that an environment has been suspended
1064 by receive queue underflow and record the system call arguments.
1065 The drawback of this approach is complexity: the E1000 must be
1066 instructed to generate receive interrupts and the driver must handle
1067 them in order to resume the environment blocked waiting for a
1068 packet.
1069 </p>
1073 receive a packet from the E1000 and expose it to user space by
1074 adding a system call. Make sure you handle the receive queue being
1075 empty.</p>
1077 <!-- XXX Check this for the E1000 and the LAPIC -->
1078 <p>If you decide to use interrupts to detect when packets are
1079 received, you'll need to write code to handle these interrupts. If
1080 you do use interrupts, note that, once an interrupt is asserted, it
1081 will remain asserted until the driver clears the interrupt. In your
1082 interrupt handler make sure to clear the interrupt handled as soon
1083 as you handle it. If you don't, after returning from your interrupt
1084 handler, the CPU will jump back into it again. In addition to
1085 clearing the interrupts on the E1000 card, interrupts also need to
1087 </div>
1091 <p>
1092 In the network server input environment, you will need to use your new
1093 receive system call to receive packets and pass them to the core
1095 message. These IPC input message should have a page
1097 pkt</code> field filled in with the packet received from the network.
1098 </p>
1103 </p></div>
1105 <p>
1107 run-net_testinput</kbd>. You should see
1108 </p>
1109 <pre>
1110 Sending ARP announcement...
1111 Waiting for packets...
1117 input: 0030 0000 0000 0000 0000 0000 0000 0000 0000
1118 </pre>
1119 <p>
1120 The lines beginning with "input:" are a hexdump of QEMU's ARP reply.
1121 </p>
1123 <p>
1125 grade</kbd>. Note that there's no way to test packet receiving
1126 without sending at least one ARP packet to inform QEMU of JOS' IP
1127 address, so bugs in your transmitting code can cause this test to
1128 fail.
1129 </p>
1131 <p>
1132 To more thoroughly test your networking code, we have provided a daemon called
1134 that will echo back anything sent over a TCP connection. Use
1137 another to connect to it. Every line you type should be echoed back
1138 by the server.
1139 Every time the emulated E1000 receives a packet, QEMU should print
1140 something like the following to the console:
1141 </p>
1142 <pre>
1147 </pre>
1149 <p>
1151 test.
1152 </p>
1157 <li>
1158 How did you structure your receive implementation? In particular,
1159 what do you do if the receive queue is empty and a user environment
1160 requests the next incoming packet?
1161 </li>
1162 </ol>
1163 </div>
1165 <!-- (amdragon: QEMU only has minimal offload support)
1166 <div class="challenge">
1167 <p><span class="header">Challenge!</span> Take advantage of the
1168 E1000's TCP offload features. This lets the E1000 take care of some
1169 of the more computationally-intensive parts of TCP processing like
1170 computing checksums before transmitting and checking checksums of
1171 received packets. You'll have to configure the card to enable TCP
1172 offload and modify lwIP so it doesn't do this work internally.</p>
1173 </div>
1174 -->
1178 Read about the EEPROM in the developer's manual and write the code to
1179 load the E1000's MAC address out of the EEPROM. Currently,
1180 QEMU's default MAC address is hard-coded into both your
1181 receive initialization and lwIP. Fix your initialization to
1182 use the MAC address you read from the EEPROM, add a
1183 system call to pass the MAC address to lwIP, and modify lwIP to
1184 the MAC address read from the card. Test your change by
1185 configuring QEMU to use a different MAC address.
1186 </p></div>
1190 to be "zero copy." Currently, packet data has to be copied from
1191 user-space buffers to transmit packet buffers and from receive
1192 packet buffers back to user-space buffers. A zero copy driver
1193 avoids this by having user space and the E1000 share packet buffer
1194 memory directly. There are many different approaches to this,
1195 including mapping the kernel-allocated structures into user space or
1196 passing user-provided buffers directly to the E1000. Regardless of
1197 your approach, be careful how you reuse buffers so that you don't
1198 introduce races between user-space code and the E1000.</p>
1199 </div>
1203 Take the zero copy concept all the way into lwIP.
1204 </p>
1206 <p>
1207 A typical packet is composed of many headers. The user sends data to be
1208 transmitted to lwIP in one buffer. The TCP layer wants to add a TCP
1209 header, the IP layer an IP header and the MAC layer an Ethernet header.
1210 Even though there are many parts to a packet, right now the parts need
1211 to be joined together so that the device driver can send the final
1212 packet.
1213 </p>
1215 <p>
1216 The E1000's transmit descriptor design is well-suited to
1217 collecting pieces of a packet scattered throughout memory,
1218 like the packet fragments created inside lwIP. If you enqueue
1219 multiple transmit descriptors, but only set the EOP command
1220 bit on the last one, then the E1000 will internally
1221 concatenate the packet buffers from these descriptors and only
1222 transmit the concatenated buffer when it reaches the
1223 EOP-marked descriptor. As a result, the individual packet
1224 pieces never need to be joined together in memory.
1225 </p>
1227 <p>
1228 Change your driver to be able to send packets composed of many
1229 buffers without copying and modify lwIP to avoid merging the
1230 packet pieces as it does right now.
1231 </p>
1232 </div>
1236 Augment your system call interface to service more than one user
1237 environment. This will prove useful if there are multiple network
1238 stacks (and multiple network servers) each with their own IP address
1239 running in user mode. The receive system call will need to decide to
1240 which environment it needs to forward each incoming packet.
1241 </p>
1243 <p>
1244 Note that the current interface cannot tell the difference between two
1245 packets and if multiple environments call the packet receive system
1246 call, each respective environment will get a subset of the incoming
1247 packets and that subset may include packets that are not destined to
1248 the calling environment.
1249 </p>
1251 <p>
1254 Exokernel paper have an in-depth explanation of the problem
1255 and a method of addressing it in a kernel like JOS. Use the paper to
1256 help you get a grip on the problem, chances are you do not need a
1257 solution as complex as presented in the paper.
1258 </p>
1259 </div>
1263 <p>
1264 A web server in its simplest form sends the contents of a file to the
1265 requesting client. We have provided skeleton code for a very simple web server
1267 and parses the headers.
1268 </p>
1272 The web server is missing the code that deals with sending the
1273 contents of a file back to the client. Finish the web server by
1275 </p></div>
1277 <p>
1278 Once you've finished the web server, point your favorite browser at
1284 You should see a web page served by the HTTP server running inside
1285 JOS.
1286 </p>
1288 <p>
1290 </p>
1295 Add a simple chat server to JOS, where multiple people can
1296 connect to the server and anything that any user types is
1297 transmitted to the other users. To do this, you will have to
1298 find a way to communicate with multiple sockets at once
1300 time. There are multiple ways to go about this. lwIP
1301 provides a MSG_DONTWAIT flag for recv (see
1303 you could constantly loop through all open sockets, polling
1305 supported by the network server IPC, they aren't accessible
1307 to pass the flags. A more efficient approach is to start one
1308 or more environments for each connection
1309 and to use IPC to coordinate them. Conveniently, the lwIP
1310 socket ID found in the struct Fd for a socket is global (not
1311 per-environment), so, for example, the child of a
1313 environment can even send on another environment's socket simply by
1314 constructing an Fd containing the right socket ID.
1316 </p>
1317 </div>
1323 <li>
1324 How long approximately did it take you to do this lab?
1325 </li>
1326 </ol>
1327 </div>
1329 <p>
1332 your answers and a description of your challenge exercise solution.
1334 to examine your changes and don't forget to <kbd>git add
1335 answers-lab6.txt</kbd>. When you're ready, commit your changes with
1337 handin</kbd> and follow the directions.
1338 </p>
1340 </body></html>
1342 <!-- LocalWords: IPC exokernel lib inc pingpong forktree Bochs inode inodes
1343 -->
1344 <!-- LocalWords: granularities Superblocks superblocks superblock IDE PIO rm
1345 -->
1346 <!-- LocalWords: IOPL EFLAGS obj gmake AMD's Athlon VM fd alloc FDTABLE MAXFD
1347 -->
1348 <!-- LocalWords: FILEBASE unmap bss libos foo umain int argc argv cputs ptr
1349 -->
1350 <!-- LocalWords: USTACKTOP esp arg init TMPPAGE TMPPAGETOP libmain env's
1351 -->