1"""Generic socket server classes. 2 3This module tries to capture the various aspects of defining a server: 4 5For socket-based servers: 6 7- address family: 8 - AF_INET{,6}: IP (Internet Protocol) sockets (default) 9 - AF_UNIX: Unix domain sockets 10 - others, e.g. AF_DECNET are conceivable (see <socket.h> 11- socket type: 12 - SOCK_STREAM (reliable stream, e.g. TCP) 13 - SOCK_DGRAM (datagrams, e.g. UDP) 14 15For request-based servers (including socket-based): 16 17- client address verification before further looking at the request 18 (This is actually a hook for any processing that needs to look 19 at the request before anything else, e.g. logging) 20- how to handle multiple requests: 21 - synchronous (one request is handled at a time) 22 - forking (each request is handled by a new process) 23 - threading (each request is handled by a new thread) 24 25The classes in this module favor the server type that is simplest to 26write: a synchronous TCP/IP server. This is bad class design, but 27save some typing. (There's also the issue that a deep class hierarchy 28slows down method lookups.) 29 30There are five classes in an inheritance diagram, four of which represent 31synchronous servers of four types: 32 33 +------------+ 34 | BaseServer | 35 +------------+ 36 | 37 v 38 +-----------+ +------------------+ 39 | TCPServer |------->| UnixStreamServer | 40 +-----------+ +------------------+ 41 | 42 v 43 +-----------+ +--------------------+ 44 | UDPServer |------->| UnixDatagramServer | 45 +-----------+ +--------------------+ 46 47Note that UnixDatagramServer derives from UDPServer, not from 48UnixStreamServer -- the only difference between an IP and a Unix 49stream server is the address family, which is simply repeated in both 50unix server classes. 51 52Forking and threading versions of each type of server can be created 53using the ForkingMixIn and ThreadingMixIn mix-in classes. For 54instance, a threading UDP server class is created as follows: 55 56 class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass 57 58The Mix-in class must come first, since it overrides a method defined 59in UDPServer! Setting the various member variables also changes 60the behavior of the underlying server mechanism. 61 62To implement a service, you must derive a class from 63BaseRequestHandler and redefine its handle() method. You can then run 64various versions of the service by combining one of the server classes 65with your request handler class. 66 67The request handler class must be different for datagram or stream 68services. This can be hidden by using the request handler 69subclasses StreamRequestHandler or DatagramRequestHandler. 70 71Of course, you still have to use your head! 72 73For instance, it makes no sense to use a forking server if the service 74contains state in memory that can be modified by requests (since the 75modifications in the child process would never reach the initial state 76kept in the parent process and passed to each child). In this case, 77you can use a threading server, but you will probably have to use 78locks to avoid two requests that come in nearly simultaneous to apply 79conflicting changes to the server state. 80 81On the other hand, if you are building e.g. an HTTP server, where all 82data is stored externally (e.g. in the file system), a synchronous 83class will essentially render the service "deaf" while one request is 84being handled -- which may be for a very long time if a client is slow 85to reqd all the data it has requested. Here a threading or forking 86server is appropriate. 87 88In some cases, it may be appropriate to process part of a request 89synchronously, but to finish processing in a forked child depending on 90the request data. This can be implemented by using a synchronous 91server and doing an explicit fork in the request handler class 92handle() method. 93 94Another approach to handling multiple simultaneous requests in an 95environment that supports neither threads nor fork (or where these are 96too expensive or inappropriate for the service) is to maintain an 97explicit table of partially finished requests and to use select() to 98decide which request to work on next (or whether to handle a new 99incoming request). This is particularly important for stream services 100where each client can potentially be connected for a long time (if 101threads or subprocesses cannot be used). 102 103Future work: 104- Standard classes for Sun RPC (which uses either UDP or TCP) 105- Standard mix-in classes to implement various authentication 106 and encryption schemes 107- Standard framework for select-based multiplexing 108 109XXX Open problems: 110- What to do with out-of-band data? 111 112BaseServer: 113- split generic "request" functionality out into BaseServer class. 114 Copyright (C) 2000 Luke Kenneth Casson Leighton <lkcl@samba.org> 115 116 example: read entries from a SQL database (requires overriding 117 get_request() to return a table entry from the database). 118 entry is processed by a RequestHandlerClass. 119 120""" 121# This file copyright (c) 2001-2015 Python Software Foundation; All Rights Reserved 122 123# Author of the BaseServer patch: Luke Kenneth Casson Leighton 124 125# XXX Warning! 126# There is a test suite for this module, but it cannot be run by the 127# standard regression test. 128# To run it manually, run Lib/test/test_socketserver.py. 129 130__version__ = "0.4" 131 132import socket 133import select 134import sys 135import os 136try: 137 import threading 138except ImportError: 139 import dummy_threading as threading 140 141__all__ = ["TCPServer", "UDPServer", "ForkingUDPServer", "ForkingTCPServer", 142 "ThreadingUDPServer", "ThreadingTCPServer", "BaseRequestHandler", 143 "StreamRequestHandler", "DatagramRequestHandler", 144 "ThreadingMixIn", "ForkingMixIn"] 145if hasattr(socket, "AF_UNIX"): 146 __all__.extend(["UnixStreamServer", "UnixDatagramServer", 147 "ThreadingUnixStreamServer", 148 "ThreadingUnixDatagramServer"]) 149 150 151class BaseServer: 152 """Base class for server classes. 153 154 Methods for the caller: 155 156 - __init__(server_address, RequestHandlerClass) 157 - serve_forever(poll_interval=0.5) 158 - shutdown() 159 - handle_request() # if you do not use serve_forever() 160 - fileno() -> int # for select() 161 162 Methods that may be overridden: 163 164 - server_bind() 165 - server_activate() 166 - get_request() -> request, client_address 167 - handle_timeout() 168 - verify_request(request, client_address) 169 - server_close() 170 - process_request(request, client_address) 171 - close_request(request) 172 - handle_error() 173 174 Methods for derived classes: 175 176 - finish_request(request, client_address) 177 178 Class variables that may be overridden by derived classes or 179 instances: 180 181 - timeout 182 - address_family 183 - socket_type 184 - allow_reuse_address 185 186 Instance variables: 187 188 - RequestHandlerClass 189 - socket 190 191 """ 192 193 timeout = None 194 195 def __init__(self, server_address, RequestHandlerClass): 196 """Constructor. May be extended, do not override.""" 197 self.server_address = server_address 198 self.RequestHandlerClass = RequestHandlerClass 199 self.__is_shut_down = threading.Event() 200 self.__serving = False 201 202 def server_activate(self): 203 """Called by constructor to activate the server. 204 205 May be overridden. 206 207 """ 208 pass 209 210 def serve_forever(self, poll_interval=0.5): 211 """Handle one request at a time until shutdown. 212 213 Polls for shutdown every poll_interval seconds. Ignores 214 self.timeout. If you need to do periodic tasks, do them in 215 another thread. 216 """ 217 self.__serving = True 218 self.__is_shut_down.clear() 219 while self.__serving: 220 # XXX: Consider using another file descriptor or 221 # connecting to the socket to wake this up instead of 222 # polling. Polling reduces our responsiveness to a 223 # shutdown request and wastes cpu at all other times. 224 r, w, e = select.select([self], [], [], poll_interval) 225 if r: 226 self._handle_request_noblock() 227 self.__is_shut_down.set() 228 229 def shutdown(self): 230 """Stops the serve_forever loop. 231 232 Blocks until the loop has finished. This must be called while 233 serve_forever() is running in another thread, or it will 234 deadlock. 235 """ 236 self.__serving = False 237 self.__is_shut_down.wait() 238 239 # The distinction between handling, getting, processing and 240 # finishing a request is fairly arbitrary. Remember: 241 # 242 # - handle_request() is the top-level call. It calls 243 # select, get_request(), verify_request() and process_request() 244 # - get_request() is different for stream or datagram sockets 245 # - process_request() is the place that may fork a new process 246 # or create a new thread to finish the request 247 # - finish_request() instantiates the request handler class; 248 # this constructor will handle the request all by itself 249 250 def handle_request(self): 251 """Handle one request, possibly blocking. 252 253 Respects self.timeout. 254 """ 255 # Support people who used socket.settimeout() to escape 256 # handle_request before self.timeout was available. 257 timeout = self.socket.gettimeout() 258 if timeout is None: 259 timeout = self.timeout 260 elif self.timeout is not None: 261 timeout = min(timeout, self.timeout) 262 fd_sets = select.select([self], [], [], timeout) 263 if not fd_sets[0]: 264 self.handle_timeout() 265 return 266 self._handle_request_noblock() 267 268 def _handle_request_noblock(self): 269 """Handle one request, without blocking. 270 271 I assume that select.select has returned that the socket is 272 readable before this function was called, so there should be 273 no risk of blocking in get_request(). 274 """ 275 try: 276 request, client_address = self.get_request() 277 except socket.error: 278 return 279 if self.verify_request(request, client_address): 280 try: 281 self.process_request(request, client_address) 282 except: 283 self.handle_error(request, client_address) 284 self.close_request(request) 285 286 def handle_timeout(self): 287 """Called if no new request arrives within self.timeout. 288 289 Overridden by ForkingMixIn. 290 """ 291 pass 292 293 def verify_request(self, request, client_address): 294 """Verify the request. May be overridden. 295 296 Return True if we should proceed with this request. 297 298 """ 299 return True 300 301 def process_request(self, request, client_address): 302 """Call finish_request. 303 304 Overridden by ForkingMixIn and ThreadingMixIn. 305 306 """ 307 self.finish_request(request, client_address) 308 self.close_request(request) 309 310 def server_close(self): 311 """Called to clean-up the server. 312 313 May be overridden. 314 315 """ 316 pass 317 318 def finish_request(self, request, client_address): 319 """Finish one request by instantiating RequestHandlerClass.""" 320 self.RequestHandlerClass(request, client_address, self) 321 322 def close_request(self, request): 323 """Called to clean up an individual request.""" 324 pass 325 326 def handle_error(self, request, client_address): 327 """Handle an error gracefully. May be overridden. 328 329 The default is to print a traceback and continue. 330 331 """ 332 print('-' * 40) 333 print('Exception happened during processing of request from %s' % (client_address,)) 334 import traceback 335 traceback.print_exc() # XXX But this goes to stderr! 336 print('-' * 40) 337 338 339class TCPServer(BaseServer): 340 341 """Base class for various socket-based server classes. 342 343 Defaults to synchronous IP stream (i.e., TCP). 344 345 Methods for the caller: 346 347 - __init__(server_address, RequestHandlerClass, bind_and_activate=True) 348 - serve_forever(poll_interval=0.5) 349 - shutdown() 350 - handle_request() # if you don't use serve_forever() 351 - fileno() -> int # for select() 352 353 Methods that may be overridden: 354 355 - server_bind() 356 - server_activate() 357 - get_request() -> request, client_address 358 - handle_timeout() 359 - verify_request(request, client_address) 360 - process_request(request, client_address) 361 - close_request(request) 362 - handle_error() 363 364 Methods for derived classes: 365 366 - finish_request(request, client_address) 367 368 Class variables that may be overridden by derived classes or 369 instances: 370 371 - timeout 372 - address_family 373 - socket_type 374 - request_queue_size (only for stream sockets) 375 - allow_reuse_address 376 377 Instance variables: 378 379 - server_address 380 - RequestHandlerClass 381 - socket 382 383 """ 384 385 address_family = socket.AF_INET 386 387 socket_type = socket.SOCK_STREAM 388 389 request_queue_size = 5 390 391 allow_reuse_address = False 392 393 def __init__(self, server_address, RequestHandlerClass, 394 bind_and_activate=True): 395 """Constructor. May be extended, do not override.""" 396 BaseServer.__init__(self, server_address, RequestHandlerClass) 397 self.socket = socket.socket(self.address_family, 398 self.socket_type) 399 if bind_and_activate: 400 self.server_bind() 401 self.server_activate() 402 403 def server_bind(self): 404 """Called by constructor to bind the socket. 405 406 May be overridden. 407 408 """ 409 if self.allow_reuse_address: 410 self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 411 self.socket.bind(self.server_address) 412 self.server_address = self.socket.getsockname() 413 414 def server_activate(self): 415 """Called by constructor to activate the server. 416 417 May be overridden. 418 419 """ 420 self.socket.listen(self.request_queue_size) 421 422 def server_close(self): 423 """Called to clean-up the server. 424 425 May be overridden. 426 427 """ 428 self.socket.close() 429 430 def fileno(self): 431 """Return socket file number. 432 433 Interface required by select(). 434 435 """ 436 return self.socket.fileno() 437 438 def get_request(self): 439 """Get the request and client address from the socket. 440 441 May be overridden. 442 443 """ 444 return self.socket.accept() 445 446 def close_request(self, request): 447 """Called to clean up an individual request.""" 448 request.close() 449 450 451class UDPServer(TCPServer): 452 453 """UDP server class.""" 454 455 allow_reuse_address = False 456 457 socket_type = socket.SOCK_DGRAM 458 459 max_packet_size = 8192 460 461 def get_request(self): 462 data, client_addr = self.socket.recvfrom(self.max_packet_size) 463 return (data, self.socket), client_addr 464 465 def server_activate(self): 466 # No need to call listen() for UDP. 467 pass 468 469 def close_request(self, request): 470 # No need to close anything. 471 pass 472 473 474class ForkingMixIn: 475 """Mix-in class to handle each request in a new process.""" 476 477 timeout = 300 478 active_children = None 479 max_children = 40 480 481 def collect_children(self): 482 """Internal routine to wait for children that have exited.""" 483 if self.active_children is None: 484 return 485 while len(self.active_children) >= self.max_children: 486 # XXX: This will wait for any child process, not just ones 487 # spawned by this library. This could confuse other 488 # libraries that expect to be able to wait for their own 489 # children. 490 try: 491 pid, status = os.waitpid(0, 0) 492 except os.error: 493 pid = None 494 if pid not in self.active_children: 495 continue 496 self.active_children.remove(pid) 497 498 # XXX: This loop runs more system calls than it ought 499 # to. There should be a way to put the active_children into a 500 # process group and then use os.waitpid(-pgid) to wait for any 501 # of that set, but I couldn't find a way to allocate pgids 502 # that couldn't collide. 503 for child in self.active_children: 504 try: 505 pid, status = os.waitpid(child, os.WNOHANG) 506 except os.error: 507 pid = None 508 if not pid: 509 continue 510 try: 511 self.active_children.remove(pid) 512 except ValueError as e: 513 raise ValueError('%s. x=%d and list=%r' % \ 514 (e.message, pid, self.active_children)) 515 516 def handle_timeout(self): 517 """Wait for zombies after self.timeout seconds of inactivity. 518 519 May be extended, do not override. 520 """ 521 self.collect_children() 522 523 def process_request(self, request, client_address): 524 """Fork a new subprocess to process the request.""" 525 self.collect_children() 526 pid = os.fork() 527 if pid: 528 # Parent process 529 if self.active_children is None: 530 self.active_children = [] 531 self.active_children.append(pid) 532 self.close_request(request) 533 return 534 else: 535 # Child process. 536 # This must never return, hence os._exit()! 537 try: 538 self.finish_request(request, client_address) 539 os._exit(0) 540 except: 541 try: 542 self.handle_error(request, client_address) 543 finally: 544 os._exit(1) 545 546 547class ThreadingMixIn: 548 """Mix-in class to handle each request in a new thread.""" 549 550 # Decides how threads will act upon termination of the 551 # main process 552 daemon_threads = False 553 554 def process_request_thread(self, request, client_address): 555 """Same as in BaseServer but as a thread. 556 557 In addition, exception handling is done here. 558 559 """ 560 try: 561 self.finish_request(request, client_address) 562 self.close_request(request) 563 except: 564 self.handle_error(request, client_address) 565 self.close_request(request) 566 567 def process_request(self, request, client_address): 568 """Start a new thread to process the request.""" 569 t = threading.Thread(target=self.process_request_thread, 570 args=(request, client_address)) 571 if self.daemon_threads: 572 t.setDaemon(1) 573 t.start() 574 575 576class ForkingUDPServer(ForkingMixIn, UDPServer): 577 pass 578 579 580class ForkingTCPServer(ForkingMixIn, TCPServer): 581 pass 582 583 584class ThreadingUDPServer(ThreadingMixIn, UDPServer): 585 pass 586 587 588class ThreadingTCPServer(ThreadingMixIn, TCPServer): 589 pass 590 591 592if hasattr(socket, 'AF_UNIX'): 593 594 class UnixStreamServer(TCPServer): 595 address_family = socket.AF_UNIX 596 597 class UnixDatagramServer(UDPServer): 598 address_family = socket.AF_UNIX 599 600 class ThreadingUnixStreamServer(ThreadingMixIn, UnixStreamServer): 601 pass 602 603 class ThreadingUnixDatagramServer(ThreadingMixIn, UnixDatagramServer): 604 pass 605 606 607class BaseRequestHandler: 608 609 """Base class for request handler classes. 610 611 This class is instantiated for each request to be handled. The 612 constructor sets the instance variables request, client_address 613 and server, and then calls the handle() method. To implement a 614 specific service, all you need to do is to derive a class which 615 defines a handle() method. 616 617 The handle() method can find the request as self.request, the 618 client address as self.client_address, and the server (in case it 619 needs access to per-server information) as self.server. Since a 620 separate instance is created for each request, the handle() method 621 can define arbitrary other instance variariables. 622 623 """ 624 625 def __init__(self, request, client_address, server): 626 self.request = request 627 self.client_address = client_address 628 self.server = server 629 try: 630 self.setup() 631 self.handle() 632 self.finish() 633 finally: 634 sys.exc_traceback = None # Help garbage collection 635 636 def setup(self): 637 pass 638 639 def handle(self): 640 pass 641 642 def finish(self): 643 pass 644 645 646# The following two classes make it possible to use the same service 647# class for stream or datagram servers. 648# Each class sets up these instance variables: 649# - rfile: a file object from which receives the request is read 650# - wfile: a file object to which the reply is written 651# When the handle() method returns, wfile is flushed properly 652 653 654class StreamRequestHandler(BaseRequestHandler): 655 656 """Define self.rfile and self.wfile for stream sockets.""" 657 658 # Default buffer sizes for rfile, wfile. 659 # We default rfile to buffered because otherwise it could be 660 # really slow for large data (a getc() call per byte); we make 661 # wfile unbuffered because (a) often after a write() we want to 662 # read and we need to flush the line; (b) big writes to unbuffered 663 # files are typically optimized by stdio even when big reads 664 # aren't. 665 rbufsize = -1 666 wbufsize = 0 667 668 def setup(self): 669 self.connection = self.request 670 self.rfile = self.connection.makefile('rb', self.rbufsize) 671 self.wfile = self.connection.makefile('wb', self.wbufsize) 672 673 def finish(self): 674 if not self.wfile.closed: 675 self.wfile.flush() 676 self.wfile.close() 677 self.rfile.close() 678 679 680class DatagramRequestHandler(BaseRequestHandler): 681 682 # XXX Regrettably, I cannot get this working on Linux; 683 # s.recvfrom() doesn't return a meaningful client address. 684 685 """Define self.rfile and self.wfile for datagram sockets.""" 686 687 def setup(self): 688 try: 689 from cStringIO import StringIO 690 except ImportError: 691 from StringIO import StringIO 692 self.packet, self.socket = self.request 693 self.rfile = StringIO(self.packet) 694 self.wfile = StringIO() 695 696 def finish(self): 697 self.socket.sendto(self.wfile.getvalue(), self.client_address) 698