Lib/SocketServer.py

   1 """Generic socket server classes.
   2
   3 This module tries to capture the various aspects of defining a server:
   4
   5 - address family:
   6         - AF_INET: IP (Internet Protocol) sockets (default)
   7         - AF_UNIX: Unix domain sockets
   8         - others, e.g. AF_DECNET are conceivable (see <socket.h>
   9 - socket type:
  10         - SOCK_STREAM (reliable stream, e.g. TCP)
  11         - SOCK_DGRAM (datagrams, e.g. UDP)
  12 - client address verification before further looking at the request
  13         (This is actually a hook for any processing that needs to look
  14          at the request before anything else, e.g. logging)
  15 - how to handle multiple requests:
  16         - synchronous (one request is handled at a time)
  17         - forking (each request is handled by a new process)
  18         - threading (each request is handled by a new thread)
  19
  20 The classes in this module favor the server type that is simplest to
  21 write: a synchronous TCP/IP server.  This is bad class design, but
  22 save some typing.  (There's also the issue that a deep class hierarchy
  23 slows down method lookups.)
  24
  25 There are four classes in an inheritance diagram that represent
  26 synchronous servers of four types:
  27
  28         +-----------+        +------------------+
  29         | TCPServer |------->| UnixStreamServer |
  30         +-----------+        +------------------+
  31               |
  32               v
  33         +-----------+        +--------------------+
  34         | UDPServer |------->| UnixDatagramServer |
  35         +-----------+        +--------------------+
  36
  37 Note that UnixDatagramServer derives from UDPServer, not from
  38 UnixStreamServer -- the only difference between an IP and a Unix
  39 stream server is the address family, which is simply repeated in both
  40 unix server classes.
  41
  42 Forking and threading versions of each type of server can be created
  43 using the ForkingServer and ThreadingServer mix-in classes.  For
  44 instance, a threading UDP server class is created as follows:
  45
  46         class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass
  47
  48 The Mix-in class must come first, since it overrides a method defined
  49 in UDPServer!
  50
  51 To implement a service, you must derive a class from
  52 BaseRequestHandler and redefine its handle() method.  You can then run
  53 various versions of the service by combining one of the server classes
  54 with your request handler class.
  55
  56 The request handler class must be different for datagram or stream
  57 services.  This can be hidden by using the mix-in request handler
  58 classes StreamRequestHandler or DatagramRequestHandler.
  59
  60 Of course, you still have to use your head!
  61
  62 For instance, it makes no sense to use a forking server if the service
  63 contains state in memory that can be modified by requests (since the
  64 modifications in the child process would never reach the initial state
  65 kept in the parent process and passed to each child).  In this case,
  66 you can use a threading server, but you will probably have to use
  67 locks to avoid two requests that come in nearly simultaneous to apply
  68 conflicting changes to the server state.
  69
  70 On the other hand, if you are building e.g. an HTTP server, where all
  71 data is stored externally (e.g. in the file system), a synchronous
  72 class will essentially render the service "deaf" while one request is
  73 being handled -- which may be for a very long time if a client is slow
  74 to reqd all the data it has requested.  Here a threading or forking
  75 server is appropriate.
  76
  77 In some cases, it may be appropriate to process part of a request
  78 synchronously, but to finish processing in a forked child depending on
  79 the request data.  This can be implemented by using a synchronous
  80 server and doing an explicit fork in the request handler class's
  81 handle() method.
  82
  83 Another approach to handling multiple simultaneous requests in an
  84 environment that supports neither threads nor fork (or where these are
  85 too expensive or inappropriate for the service) is to maintain an
  86 explicit table of partially finished requests and to use select() to
  87 decide which request to work on next (or whether to handle a new
  88 incoming request).  This is particularly important for stream services
  89 where each client can potentially be connected for a long time (if
  90 threads or subprocesses can't be used).
  91
  92 Future work:
  93 - Standard classes for Sun RPC (which uses either UDP or TCP)
  94 - Standard mix-in classes to implement various authentication
  95   and encryption schemes
  96 - Standard framework for select-based multiplexing
  97
  98 XXX Open problems:
  99 - What to do with out-of-band data?
 100
 101 """
 102
 103
 104 __version__ = "0.2"
 105
 106
 107 import socket
 108 import sys
 109 import os
 110
 111
 112 class TCPServer:
 113
 114     """Base class for various socket-based server classes.
 115
 116     Defaults to synchronous IP stream (i.e., TCP).
 117
 118     Methods for the caller:
 119
 120     - __init__(server_address, RequestHandlerClass)
 121     - serve_forever()
 122     - handle_request()  # if you don't use serve_forever()
 123     - fileno() -> int   # for select()
 124
 125     Methods that may be overridden:
 126
 127     - server_bind()
 128     - server_activate()
 129     - get_request() -> request, client_address
 130     - verify_request(request, client_address)
 131     - process_request(request, client_address)
 132     - handle_error()
 133
 134     Methods for derived classes:
 135
 136     - finish_request(request, client_address)
 137
 138     Class variables that may be overridden by derived classes or
 139     instances:
 140
 141     - address_family
 142     - socket_type
 143     - request_queue_size (only for stream sockets)
 144     - reuse_address
 145
 146     Instance variables:
 147
 148     - server_address
 149     - RequestHandlerClass
 150     - socket
 151
 152     """
 153
 154     address_family = socket.AF_INET
 155
 156     socket_type = socket.SOCK_STREAM
 157
 158     request_queue_size = 5
 159
 160     allow_reuse_address = 0
 161
 162     def __init__(self, server_address, RequestHandlerClass):
 163         """Constructor.  May be extended, do not override."""
 164         self.server_address = server_address
 165         self.RequestHandlerClass = RequestHandlerClass
 166         self.socket = socket.socket(self.address_family,
 167                                     self.socket_type)
 168         self.server_bind()
 169         self.server_activate()
 170
 171     def server_bind(self):
 172         """Called by constructor to bind the socket.
 173
 174         May be overridden.
 175
 176         """
 177         if self.allow_reuse_address:
 178             self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
 179         self.socket.bind(self.server_address)
 180
 181     def server_activate(self):
 182         """Called by constructor to activate the server.
 183
 184         May be overridden.
 185
 186         """
 187         self.socket.listen(self.request_queue_size)
 188
 189     def fileno(self):
 190         """Return socket file number.
 191
 192         Interface required by select().
 193
 194         """
 195         return self.socket.fileno()
 196
 197     def serve_forever(self):
 198         """Handle one request at a time until doomsday."""
 199         while 1:
 200             self.handle_request()
 201
 202     # The distinction between handling, getting, processing and
 203     # finishing a request is fairly arbitrary.  Remember:
 204     #
 205     # - handle_request() is the top-level call.  It calls
 206     #   get_request(), verify_request() and process_request()
 207     # - get_request() is different for stream or datagram sockets
 208     # - process_request() is the place that may fork a new process
 209     #   or create a new thread to finish the request
 210     # - finish_request() instantiates the request handler class;
 211     #   this constructor will handle the request all by itself
 212
 213     def handle_request(self):
 214         """Handle one request, possibly blocking."""
 215         try:
 216             request, client_address = self.get_request()
 217         except socket.error:
 218             return
 219         if self.verify_request(request, client_address):
 220             try:
 221                 self.process_request(request, client_address)
 222             except:
 223                 self.handle_error(request, client_address)
 224
 225     def get_request(self):
 226         """Get the request and client address from the socket.
 227
 228         May be overridden.
 229
 230         """
 231         return self.socket.accept()
 232
 233     def verify_request(self, request, client_address):
 234         """Verify the request.  May be overridden.
 235
 236         Return true if we should proceed with this request.
 237
 238         """
 239         return 1
 240
 241     def process_request(self, request, client_address):
 242         """Call finish_request.
 243
 244         Overridden by ForkingMixIn and ThreadingMixIn.
 245
 246         """
 247         self.finish_request(request, client_address)
 248
 249     def finish_request(self, request, client_address):
 250         """Finish one request by instantiating RequestHandlerClass."""
 251         self.RequestHandlerClass(request, client_address, self)
 252
 253     def handle_error(self, request, client_address):
 254         """Handle an error gracefully.  May be overridden.
 255
 256         The default is to print a traceback and continue.
 257
 258         """
 259         print '-'*40
 260         print 'Exception happened during processing of request from',
 261         print client_address
 262         import traceback
 263         traceback.print_exc()
 264         print '-'*40
 265
 266
 267 class UDPServer(TCPServer):
 268
 269     """UDP server class."""
 270
 271     socket_type = socket.SOCK_DGRAM
 272
 273     max_packet_size = 8192
 274
 275     def get_request(self):
 276         data, client_addr = self.socket.recvfrom(self.max_packet_size)
 277         return (data, self.socket), client_addr
 278
 279     def server_activate(self):
 280         # No need to call listen() for UDP.
 281         pass
 282
 283
 284 class ForkingMixIn:
 285
 286     """Mix-in class to handle each request in a new process."""
 287
 288     active_children = None
 289     max_children = 40
 290
 291     def collect_children(self):
 292         """Internal routine to wait for died children."""
 293         while self.active_children:
 294             if len(self.active_children) < self.max_children:
 295                 options = os.WNOHANG
 296             else:
 297                 # If the maximum number of children are already
 298                 # running, block while waiting for a child to exit
 299                 options = 0
 300             try:
 301                 pid, status = os.waitpid(0, options)
 302             except os.error:
 303                 pid = None
 304             if not pid: break
 305             self.active_children.remove(pid)
 306
 307     def process_request(self, request, client_address):
 308         """Fork a new subprocess to process the request."""
 309         self.collect_children()
 310         pid = os.fork()
 311         if pid:
 312             # Parent process
 313             if self.active_children is None:
 314                 self.active_children = []
 315             self.active_children.append(pid)
 316             return
 317         else:
 318             # Child process.
 319             # This must never return, hence os._exit()!
 320             try:
 321                 self.socket.close()
 322                 self.finish_request(request, client_address)
 323                 os._exit(0)
 324             except:
 325                 try:
 326                     self.handle_error(request,
 327                                       client_address)
 328                 finally:
 329                     os._exit(1)
 330
 331
 332 class ThreadingMixIn:
 333     """Mix-in class to handle each request in a new thread."""
 334
 335     def process_request(self, request, client_address):
 336         """Start a new thread to process the request."""
 337         import threading
 338         t = threading.Thread(target = self.finish_request,
 339                              args = (request, client_address))
 340         t.start()
 341
 342
 343 class ForkingUDPServer(ForkingMixIn, UDPServer): pass
 344 class ForkingTCPServer(ForkingMixIn, TCPServer): pass
 345
 346 class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass
 347 class ThreadingTCPServer(ThreadingMixIn, TCPServer): pass
 348
 349 if hasattr(socket, 'AF_UNIX'):
 350
 351     class UnixStreamServer(TCPServer):
 352         address_family = socket.AF_UNIX
 353
 354     class UnixDatagramServer(UDPServer):
 355         address_family = socket.AF_UNIX
 356
 357     class ThreadingUnixStreamServer(ThreadingMixIn, UnixStreamServer): pass
 358
 359     class ThreadingUnixDatagramServer(ThreadingMixIn, UnixDatagramServer): pass
 360
 361 class BaseRequestHandler:
 362
 363     """Base class for request handler classes.
 364
 365     This class is instantiated for each request to be handled.  The
 366     constructor sets the instance variables request, client_address
 367     and server, and then calls the handle() method.  To implement a
 368     specific service, all you need to do is to derive a class which
 369     defines a handle() method.
 370
 371     The handle() method can find the request as self.request, the
 372     client address as self.client_address, and the server (in case it
 373     needs access to per-server information) as self.server.  Since a
 374     separate instance is created for each request, the handle() method
 375     can define arbitrary other instance variariables.
 376
 377     """
 378
 379     def __init__(self, request, client_address, server):
 380         self.request = request
 381         self.client_address = client_address
 382         self.server = server
 383         try:
 384             self.setup()
 385             self.handle()
 386             self.finish()
 387         finally:
 388             sys.exc_traceback = None    # Help garbage collection
 389
 390     def setup(self):
 391         pass
 392
 393     def __del__(self):
 394         pass
 395
 396     def handle(self):
 397         pass
 398
 399     def finish(self):
 400         pass
 401
 402
 403 # The following two classes make it possible to use the same service
 404 # class for stream or datagram servers.
 405 # Each class sets up these instance variables:
 406 # - rfile: a file object from which receives the request is read
 407 # - wfile: a file object to which the reply is written
 408 # When the handle() method returns, wfile is flushed properly
 409
 410
 411 class StreamRequestHandler(BaseRequestHandler):
 412
 413     """Define self.rfile and self.wfile for stream sockets."""
 414
 415     def setup(self):
 416         self.connection = self.request
 417         self.rfile = self.connection.makefile('rb', 0)
 418         self.wfile = self.connection.makefile('wb', 0)
 419
 420     def finish(self):
 421         self.wfile.flush()
 422         self.wfile.close()
 423         self.rfile.close()
 424
 425
 426 class DatagramRequestHandler(BaseRequestHandler):
 427
 428     """Define self.rfile and self.wfile for datagram sockets."""
 429
 430     def setup(self):
 431         import StringIO
 432         self.packet, self.socket = self.request
 433         self.rfile = StringIO.StringIO(self.packet)
 434         self.wfile = StringIO.StringIO(self.packet)
 435
 436     def finish(self):
 437         self.socket.sendto(self.wfile.getvalue(), self.client_address)