| blob | 8cac6492aade7c41cc1cffafc4b4d0500ff9d624 |
1 SAS Layer
2 ---------
4 The SAS Layer is a management infrastructure which manages
5 SAS LLDDs. It sits between SCSI Core and SAS LLDDs. The
6 layout is as follows: while SCSI Core is concerned with
7 SAM/SPC issues, and a SAS LLDD+sequencer is concerned with
8 phy/OOB/link management, the SAS layer is concerned with:
10 * SAS Phy/Port/HA event management (LLDD generates,
11 SAS Layer processes),
12 * SAS Port management (creation/destruction),
13 * SAS Domain discovery and revalidation,
14 * SAS Domain device management,
15 * SCSI Host registration/unregistration,
16 * Device registration with SCSI Core (SAS) or libata
17 (SATA), and
18 * Expander management and exporting expander control
19 to user space.
21 A SAS LLDD is a PCI device driver. It is concerned with
22 phy/OOB management, and vendor specific tasks and generates
23 events to the SAS layer.
25 The SAS Layer does most SAS tasks as outlined in the SAS 1.1
26 spec.
28 The sas_ha_struct describes the SAS LLDD to the SAS layer.
29 Most of it is used by the SAS Layer but a few fields need to
30 be initialized by the LLDDs.
32 After initializing your hardware, from the probe() function
33 you call sas_register_ha(). It will register your LLDD with
34 the SCSI subsystem, creating a SCSI host and it will
35 register your SAS driver with the sysfs SAS tree it creates.
36 It will then return. Then you enable your phys to actually
37 start OOB (at which point your driver will start calling the
38 notify_* event callbacks).
40 Structure descriptions:
42 struct sas_phy --------------------
43 Normally this is statically embedded to your driver's
44 phy structure:
45 struct my_phy {
46 blah;
47 struct sas_phy sas_phy;
48 bleh;
49 };
50 And then all the phys are an array of my_phy in your HA
51 struct (shown below).
53 Then as you go along and initialize your phys you also
54 initialize the sas_phy struct, along with your own
55 phy structure.
57 In general, the phys are managed by the LLDD and the ports
58 are managed by the SAS layer. So the phys are initialized
59 and updated by the LLDD and the ports are initialized and
60 updated by the SAS layer.
62 There is a scheme where the LLDD can RW certain fields,
63 and the SAS layer can only read such ones, and vice versa.
64 The idea is to avoid unnecessary locking.
66 enabled -- must be set (0/1)
67 id -- must be set [0,MAX_PHYS)
68 class, proto, type, role, oob_mode, linkrate -- must be set
69 oob_mode -- you set this when OOB has finished and then notify
70 the SAS Layer.
72 sas_addr -- this normally points to an array holding the sas
73 address of the phy, possibly somewhere in your my_phy
74 struct.
76 attached_sas_addr -- set this when you (LLDD) receive an
77 IDENTIFY frame or a FIS frame, _before_ notifying the SAS
78 layer. The idea is that sometimes the LLDD may want to fake
79 or provide a different SAS address on that phy/port and this
80 allows it to do this. At best you should copy the sas
81 address from the IDENTIFY frame or maybe generate a SAS
82 address for SATA directly attached devices. The Discover
83 process may later change this.
85 frame_rcvd -- this is where you copy the IDENTIFY/FIS frame
86 when you get it; you lock, copy, set frame_rcvd_size and
87 unlock the lock, and then call the event. It is a pointer
88 since there's no way to know your hw frame size _exactly_,
89 so you define the actual array in your phy struct and let
90 this pointer point to it. You copy the frame from your
91 DMAable memory to that area holding the lock.
93 sas_prim -- this is where primitives go when they're
94 received. See sas.h. Grab the lock, set the primitive,
95 release the lock, notify.
97 port -- this points to the sas_port if the phy belongs
98 to a port -- the LLDD only reads this. It points to the
99 sas_port this phy is part of. Set by the SAS Layer.
101 ha -- may be set; the SAS layer sets it anyway.
103 lldd_phy -- you should set this to point to your phy so you
104 can find your way around faster when the SAS layer calls one
105 of your callbacks and passes you a phy. If the sas_phy is
106 embedded you can also use container_of -- whatever you
107 prefer.
110 struct sas_port --------------------
111 The LLDD doesn't set any fields of this struct -- it only
112 reads them. They should be self explanatory.
114 phy_mask is 32 bit, this should be enough for now, as I
115 haven't heard of a HA having more than 8 phys.
117 lldd_port -- I haven't found use for that -- maybe other
118 LLDD who wish to have internal port representation can make
119 use of this.
122 struct sas_ha_struct --------------------
123 It normally is statically declared in your own LLDD
124 structure describing your adapter:
125 struct my_sas_ha {
126 blah;
127 struct sas_ha_struct sas_ha;
128 struct my_phy phys[MAX_PHYS];
129 struct sas_port sas_ports[MAX_PHYS]; /* (1) */
130 bleh;
131 };
133 (1) If your LLDD doesn't have its own port representation.
135 What needs to be initialized (sample function given below).
137 pcidev
138 sas_addr -- since the SAS layer doesn't want to mess with
139 memory allocation, etc, this points to statically
140 allocated array somewhere (say in your host adapter
141 structure) and holds the SAS address of the host
142 adapter as given by you or the manufacturer, etc.
143 sas_port
144 sas_phy -- an array of pointers to structures. (see
145 note above on sas_addr).
146 These must be set. See more notes below.
147 num_phys -- the number of phys present in the sas_phy array,
148 and the number of ports present in the sas_port
149 array. There can be a maximum num_phys ports (one per
150 port) so we drop the num_ports, and only use
151 num_phys.
153 The event interface:
155 /* LLDD calls these to notify the class of an event. */
156 void (*notify_ha_event)(struct sas_ha_struct *, enum ha_event);
157 void (*notify_port_event)(struct sas_phy *, enum port_event);
158 void (*notify_phy_event)(struct sas_phy *, enum phy_event);
160 When sas_register_ha() returns, those are set and can be
161 called by the LLDD to notify the SAS layer of such events
162 the SAS layer.
164 The port notification:
166 /* The class calls these to notify the LLDD of an event. */
167 void (*lldd_port_formed)(struct sas_phy *);
168 void (*lldd_port_deformed)(struct sas_phy *);
170 If the LLDD wants notification when a port has been formed
171 or deformed it sets those to a function satisfying the type.
173 A SAS LLDD should also implement at least one of the Task
174 Management Functions (TMFs) described in SAM:
176 /* Task Management Functions. Must be called from process context. */
177 int (*lldd_abort_task)(struct sas_task *);
178 int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);
179 int (*lldd_clear_aca)(struct domain_device *, u8 *lun);
180 int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);
181 int (*lldd_I_T_nexus_reset)(struct domain_device *);
182 int (*lldd_lu_reset)(struct domain_device *, u8 *lun);
183 int (*lldd_query_task)(struct sas_task *);
185 For more information please read SAM from T10.org.
187 Port and Adapter management:
189 /* Port and Adapter management */
190 int (*lldd_clear_nexus_port)(struct sas_port *);
191 int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);
193 A SAS LLDD should implement at least one of those.
195 Phy management:
197 /* Phy management */
198 int (*lldd_control_phy)(struct sas_phy *, enum phy_func);
200 lldd_ha -- set this to point to your HA struct. You can also
201 use container_of if you embedded it as shown above.
203 A sample initialization and registration function
204 can look like this (called last thing from probe())
205 *but* before you enable the phys to do OOB:
207 static int register_sas_ha(struct my_sas_ha *my_ha)
208 {
209 int i;
210 static struct sas_phy *sas_phys[MAX_PHYS];
211 static struct sas_port *sas_ports[MAX_PHYS];
213 my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];
215 for (i = 0; i < MAX_PHYS; i++) {
216 sas_phys[i] = &my_ha->phys[i].sas_phy;
217 sas_ports[i] = &my_ha->sas_ports[i];
218 }
220 my_ha->sas_ha.sas_phy = sas_phys;
221 my_ha->sas_ha.sas_port = sas_ports;
222 my_ha->sas_ha.num_phys = MAX_PHYS;
224 my_ha->sas_ha.lldd_port_formed = my_port_formed;
226 my_ha->sas_ha.lldd_dev_found = my_dev_found;
227 my_ha->sas_ha.lldd_dev_gone = my_dev_gone;
229 my_ha->sas_ha.lldd_execute_task = my_execute_task;
231 my_ha->sas_ha.lldd_abort_task = my_abort_task;
232 my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;
233 my_ha->sas_ha.lldd_clear_aca = my_clear_aca;
234 my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;
235 my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)
236 my_ha->sas_ha.lldd_lu_reset = my_lu_reset;
237 my_ha->sas_ha.lldd_query_task = my_query_task;
239 my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;
240 my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;
242 my_ha->sas_ha.lldd_control_phy = my_control_phy;
244 return sas_register_ha(&my_ha->sas_ha);
245 }
247 (2) SAS 1.1 does not define I_T Nexus Reset TMF.
249 Events
250 ------
252 Events are _the only way_ a SAS LLDD notifies the SAS layer
253 of anything. There is no other method or way a LLDD to tell
254 the SAS layer of anything happening internally or in the SAS
255 domain.
257 Phy events:
258 PHYE_LOSS_OF_SIGNAL, (C)
259 PHYE_OOB_DONE,
260 PHYE_OOB_ERROR, (C)
261 PHYE_SPINUP_HOLD.
263 Port events, passed on a _phy_:
264 PORTE_BYTES_DMAED, (M)
265 PORTE_BROADCAST_RCVD, (E)
266 PORTE_LINK_RESET_ERR, (C)
267 PORTE_TIMER_EVENT, (C)
268 PORTE_HARD_RESET.
270 Host Adapter event:
271 HAE_RESET
273 A SAS LLDD should be able to generate
274 - at least one event from group C (choice),
275 - events marked M (mandatory) are mandatory (only one),
276 - events marked E (expander) if it wants the SAS layer
277 to handle domain revalidation (only one such).
278 - Unmarked events are optional.
280 Meaning:
282 HAE_RESET -- when your HA got internal error and was reset.
284 PORTE_BYTES_DMAED -- on receiving an IDENTIFY/FIS frame
285 PORTE_BROADCAST_RCVD -- on receiving a primitive
286 PORTE_LINK_RESET_ERR -- timer expired, loss of signal, loss
287 of DWS, etc. (*)
288 PORTE_TIMER_EVENT -- DWS reset timeout timer expired (*)
289 PORTE_HARD_RESET -- Hard Reset primitive received.
291 PHYE_LOSS_OF_SIGNAL -- the device is gone (*)
292 PHYE_OOB_DONE -- OOB went fine and oob_mode is valid
293 PHYE_OOB_ERROR -- Error while doing OOB, the device probably
294 got disconnected. (*)
295 PHYE_SPINUP_HOLD -- SATA is present, COMWAKE not sent.
297 (*) should set/clear the appropriate fields in the phy,
298 or alternatively call the inlined sas_phy_disconnected()
299 which is just a helper, from their tasklet.
301 The Execute Command SCSI RPC:
303 int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags);
305 Used to queue a task to the SAS LLDD. @task is the task to be executed.
306 @gfp_mask is the gfp_mask defining the context of the caller.
308 This function should implement the Execute Command SCSI RPC,
310 That is, when lldd_execute_task() is called, the command
311 go out on the transport *immediately*. There is *no*
312 queuing of any sort and at any level in a SAS LLDD.
314 Returns: -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;
315 0, the task(s) were queued.
317 struct sas_task {
318 dev -- the device this task is destined to
319 task_proto -- _one_ of enum sas_proto
320 scatter -- pointer to scatter gather list array
321 num_scatter -- number of elements in scatter
322 total_xfer_len -- total number of bytes expected to be transferred
323 data_dir -- PCI_DMA_...
324 task_done -- callback when the task has finished execution
325 };
327 DISCOVERY
328 ---------
330 The sysfs tree has the following purposes:
331 a) It shows you the physical layout of the SAS domain at
332 the current time, i.e. how the domain looks in the
333 physical world right now.
334 b) Shows some device parameters _at_discovery_time_.
336 This is a link to the tree(1) program, very useful in
337 viewing the SAS domain:
338 ftp://mama.indstate.edu/linux/tree/
339 I expect user space applications to actually create a
340 graphical interface of this.
342 That is, the sysfs domain tree doesn't show or keep state if
343 you e.g., change the meaning of the READY LED MEANING
344 setting, but it does show you the current connection status
345 of the domain device.
347 Keeping internal device state changes is responsibility of
348 upper layers (Command set drivers) and user space.
350 When a device or devices are unplugged from the domain, this
351 is reflected in the sysfs tree immediately, and the device(s)
352 removed from the system.
354 The structure domain_device describes any device in the SAS
355 domain. It is completely managed by the SAS layer. A task
356 points to a domain device, this is how the SAS LLDD knows
357 where to send the task(s) to. A SAS LLDD only reads the
358 contents of the domain_device structure, but it never creates
359 or destroys one.
361 Expander management from User Space
362 -----------------------------------
364 In each expander directory in sysfs, there is a file called
365 "smp_portal". It is a binary sysfs attribute file, which
366 implements an SMP portal (Note: this is *NOT* an SMP port),
367 to which user space applications can send SMP requests and
368 receive SMP responses.
370 Functionality is deceptively simple:
372 1. Build the SMP frame you want to send. The format and layout
373 is described in the SAS spec. Leave the CRC field equal 0.
374 open(2)
375 2. Open the expander's SMP portal sysfs file in RW mode.
376 write(2)
377 3. Write the frame you built in 1.
378 read(2)
379 4. Read the amount of data you expect to receive for the frame you built.
380 If you receive different amount of data you expected to receive,
381 then there was some kind of error.
382 close(2)
383 All this process is shown in detail in the function do_smp_func()
384 and its callers, in the file "expander_conf.c".
386 The kernel functionality is implemented in the file
387 "sas_expander.c".
389 The program "expander_conf.c" implements this. It takes one
390 argument, the sysfs file name of the SMP portal to the
391 expander, and gives expander information, including routing
392 tables.
394 The SMP portal gives you complete control of the expander,
395 so please be careful.