1======== 2Tutorial 3======== 4 5.. contents:: Table of Contents 6 7Below is a set of example scripts showing some of the possible customizations 8that can be done with pyftpdlib. Some of them are included in 9`demo <https://github.com/giampaolo/pyftpdlib/blob/master/demo/>`__ 10directory of pyftpdlib source distribution. 11 12A Base FTP server 13================= 14 15The script below uses a basic configuration and it's probably the best 16starting point to understand how things work. It uses the base 17`DummyAuthorizer <api.html#pyftpdlib.authorizers.DummyAuthorizer>`__ 18for adding a bunch of "virtual" users, sets a limit for 19`incoming connections <api.html#pyftpdlib.servers.FTPServer.max_cons>`__ 20and a range of `passive ports <api.html#pyftpdlib.handlers.FTPHandler.passive_ports>`__. 21 22`source code <https://github.com/giampaolo/pyftpdlib/blob/master/demo/basic_ftpd.py>`__ 23 24.. code-block:: python 25 26 import os 27 28 from pyftpdlib.authorizers import DummyAuthorizer 29 from pyftpdlib.handlers import FTPHandler 30 from pyftpdlib.servers import FTPServer 31 32 def main(): 33 # Instantiate a dummy authorizer for managing 'virtual' users 34 authorizer = DummyAuthorizer() 35 36 # Define a new user having full r/w permissions and a read-only 37 # anonymous user 38 authorizer.add_user('user', '12345', '.', perm='elradfmwMT') 39 authorizer.add_anonymous(os.getcwd()) 40 41 # Instantiate FTP handler class 42 handler = FTPHandler 43 handler.authorizer = authorizer 44 45 # Define a customized banner (string returned when client connects) 46 handler.banner = "pyftpdlib based ftpd ready." 47 48 # Specify a masquerade address and the range of ports to use for 49 # passive connections. Decomment in case you're behind a NAT. 50 #handler.masquerade_address = '151.25.42.11' 51 #handler.passive_ports = range(60000, 65535) 52 53 # Instantiate FTP server class and listen on 0.0.0.0:2121 54 address = ('', 2121) 55 server = FTPServer(address, handler) 56 57 # set a limit for connections 58 server.max_cons = 256 59 server.max_cons_per_ip = 5 60 61 # start ftp server 62 server.serve_forever() 63 64 if __name__ == '__main__': 65 main() 66 67 68Logging management 69================== 70 71pyftpdlib uses the 72`logging <http://docs.python.org/library/logging.html logging>`__ 73module to handle logging. If you don't configure logging pyftpdlib will write 74logs to stderr. 75In order to configure logging you should do it *before* calling serve_forever(). 76Example logging to a file: 77 78.. code-block:: python 79 80 import logging 81 82 from pyftpdlib.handlers import FTPHandler 83 from pyftpdlib.servers import FTPServer 84 from pyftpdlib.authorizers import DummyAuthorizer 85 86 authorizer = DummyAuthorizer() 87 authorizer.add_user('user', '12345', '.', perm='elradfmwMT') 88 handler = FTPHandler 89 handler.authorizer = authorizer 90 91 logging.basicConfig(filename='/var/log/pyftpd.log', level=logging.INFO) 92 93 server = FTPServer(('', 2121), handler) 94 server.serve_forever() 95 96DEBUG logging 97^^^^^^^^^^^^^ 98 99You may want to enable DEBUG logging to observe commands and responses 100exchanged by client and server. DEBUG logging will also log internal errors 101which may occur on socket related calls such as ``send()`` and ``recv()``. 102To enable DEBUG logging from code use: 103 104.. code-block:: python 105 106 logging.basicConfig(level=logging.DEBUG) 107 108To enable DEBUG logging from command line use: 109 110.. code-block:: bash 111 112 python -m pyftpdlib -D 113 114DEBUG logs look like this: 115 116:: 117 118 [I 2017-11-07 12:03:44] >>> starting FTP server on 0.0.0.0:2121, pid=22991 <<< 119 [I 2017-11-07 12:03:44] concurrency model: async 120 [I 2017-11-07 12:03:44] masquerade (NAT) address: None 121 [I 2017-11-07 12:03:44] passive ports: None 122 [D 2017-11-07 12:03:44] poller: 'pyftpdlib.ioloop.Epoll' 123 [D 2017-11-07 12:03:44] authorizer: 'pyftpdlib.authorizers.DummyAuthorizer' 124 [D 2017-11-07 12:03:44] use sendfile(2): True 125 [D 2017-11-07 12:03:44] handler: 'pyftpdlib.handlers.FTPHandler' 126 [D 2017-11-07 12:03:44] max connections: 512 127 [D 2017-11-07 12:03:44] max connections per ip: unlimited 128 [D 2017-11-07 12:03:44] timeout: 300 129 [D 2017-11-07 12:03:44] banner: 'pyftpdlib 1.5.4 ready.' 130 [D 2017-11-07 12:03:44] max login attempts: 3 131 [I 2017-11-07 12:03:44] 127.0.0.1:37303-[] FTP session opened (connect) 132 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[] -> 220 pyftpdlib 1.0.0 ready. 133 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[] <- USER user 134 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[] -> 331 Username ok, send password. 135 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] <- PASS ****** 136 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] -> 230 Login successful. 137 [I 2017-11-07 12:03:44] 127.0.0.1:37303-[user] USER 'user' logged in. 138 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] <- TYPE I 139 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] -> 200 Type set to: Binary. 140 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] <- PASV 141 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] -> 227 Entering passive mode (127,0,0,1,233,208). 142 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] <- RETR tmp-pyftpdlib 143 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] -> 125 Data connection already open. Transfer starting. 144 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] -> 226 Transfer complete. 145 [I 2017-11-07 12:03:44] 127.0.0.1:37303-[user] RETR /home/giampaolo/IMG29312.JPG completed=1 bytes=1205012 seconds=0.003 146 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] <- QUIT 147 [D 2017-11-07 12:03:44] 127.0.0.1:37303-[user] -> 221 Goodbye. 148 [I 2017-11-07 12:03:44] 127.0.0.1:37303-[user] FTP session closed (disconnect). 149 150 151Changing log line prefix 152^^^^^^^^^^^^^^^^^^^^^^^^ 153 154.. code-block:: python 155 156 handler = FTPHandler 157 handler.log_prefix = 'XXX [%(username)s]@%(remote_ip)s' 158 server = FTPServer(('localhost', 2121), handler) 159 server.serve_forever() 160 161Logs will now look like this: 162 163:: 164 165 [I 13-02-01 19:12:26] XXX []@127.0.0.1 FTP session opened (connect) 166 [I 13-02-01 19:12:26] XXX [user]@127.0.0.1 USER 'user' logged in. 167 168 169Storing passwords as hash digests 170================================= 171 172Using FTP server library with the default 173`DummyAuthorizer <api.html#pyftpdlib.authorizers.DummyAuthorizer>`__ means that 174passwords will be stored in clear-text. An end-user ftpd using the default 175dummy authorizer would typically require a configuration file for 176authenticating users and their passwords but storing clear-text passwords is of 177course undesirable. The most common way to do things in such case would be 178first creating new users and then storing their usernames + passwords as hash 179digests into a file or wherever you find it convenient. The example below shows 180how to storage passwords as one-way hashes by using md5 algorithm. 181 182`source code <https://github.com/giampaolo/pyftpdlib/blob/master/demo/md5_ftpd.py>`__ 183 184.. code-block:: python 185 186 import os 187 import sys 188 from hashlib import md5 189 190 from pyftpdlib.handlers import FTPHandler 191 from pyftpdlib.servers import FTPServer 192 from pyftpdlib.authorizers import DummyAuthorizer, AuthenticationFailed 193 194 195 class DummyMD5Authorizer(DummyAuthorizer): 196 197 def validate_authentication(self, username, password, handler): 198 if sys.version_info >= (3, 0): 199 password = md5(password.encode('latin1')) 200 hash = md5(password).hexdigest() 201 try: 202 if self.user_table[username]['pwd'] != hash: 203 raise KeyError 204 except KeyError: 205 raise AuthenticationFailed 206 207 208 def main(): 209 # get a hash digest from a clear-text password 210 hash = md5('12345').hexdigest() 211 authorizer = DummyMD5Authorizer() 212 authorizer.add_user('user', hash, os.getcwd(), perm='elradfmwMT') 213 authorizer.add_anonymous(os.getcwd()) 214 handler = FTPHandler 215 handler.authorizer = authorizer 216 server = FTPServer(('', 2121), handler) 217 server.serve_forever() 218 219 if __name__ == "__main__": 220 main() 221 222 223 224Unix FTP Server 225=============== 226 227If you're running a Unix system you may want to configure your ftpd to include 228support for "real" users existing on the system and navigate the real 229filesystem. The example below uses 230`UnixAuthorizer <api.html#pyftpdlib.authorizers.UnixAuthorizer>`__ and 231`UnixFilesystem <api.html#pyftpdlib.filesystems.UnixFilesystem>`__ 232classes to do so. 233 234.. code-block:: python 235 236 from pyftpdlib.handlers import FTPHandler 237 from pyftpdlib.servers import FTPServer 238 from pyftpdlib.authorizers import UnixAuthorizer 239 from pyftpdlib.filesystems import UnixFilesystem 240 241 def main(): 242 authorizer = UnixAuthorizer(rejected_users=["root"], require_valid_shell=True) 243 handler = FTPHandler 244 handler.authorizer = authorizer 245 handler.abstracted_fs = UnixFilesystem 246 server = FTPServer(('', 21), handler) 247 server.serve_forever() 248 249 if __name__ == "__main__": 250 main() 251 252 253Windows FTP Server 254================== 255 256The following code shows how to implement a basic authorizer for a Windows NT 257workstation to authenticate against existing Windows user accounts. This code 258requires Mark Hammond's 259`pywin32 <http://starship.python.net/crew/mhammond/win32/>`__ extension to be 260installed. 261 262`source code <https://github.com/giampaolo/pyftpdlib/blob/master/demo/winnt_ftpd.py>`__ 263 264.. code-block:: python 265 266 from pyftpdlib.handlers import FTPHandler 267 from pyftpdlib.servers import FTPServer 268 from pyftpdlib.authorizers import WindowsAuthorizer 269 270 def main(): 271 authorizer = WindowsAuthorizer() 272 # Use Guest user with empty password to handle anonymous sessions. 273 # Guest user must be enabled first, empty password set and profile 274 # directory specified. 275 #authorizer = WindowsAuthorizer(anonymous_user="Guest", anonymous_password="") 276 handler = FTPHandler 277 handler.authorizer = authorizer 278 server = FTPServer(('', 2121), handler) 279 server.serve_forever() 280 281 if __name__ == "__main__": 282 main() 283 284.. _changing-the-concurrency-model: 285 286Changing the concurrency model 287============================== 288 289By nature pyftpdlib is asynchronous. That means it uses a single process/thread 290to handle multiple client connections and file transfers. This is why it is so 291fast, lightweight and scalable (see `benchmarks <benchmarks.html>`__). The 292async model has one big drawback though: the code cannot contain instructions 293which blocks for a long period of time, otherwise the whole FTP server will 294hang. 295As such the user should avoid calls such as ``time.sleep(3)``, heavy db 296queries, etc. Moreover, there are cases where the async model is not 297appropriate, and that is when you're dealing with a particularly slow 298filesystem (say a network filesystem such as samba). If the filesystem is slow 299(say, a ``open(file, 'r').read(8192)`` takes 2 secs to complete) then you are 300stuck. 301Starting from version 1.0.0 pyftpdlib can change the concurrency model by using 302multiple processes or threads instead. 303In technical (internal) terms that means that every time a client connects a 304separate thread/process is spawned and internally it will run its own IO loop. 305In practical terms this means that you can block as long as you want. 306Changing the concurrency module is easy: you just need to import a substitute 307for `FTPServer <api.html#pyftpdlib.servers.FTPServer>`__. class: 308 309Multiple threads 310^^^^^^^^^^^^^^^^ 311 312.. code-block:: python 313 314 from pyftpdlib.handlers import FTPHandler 315 from pyftpdlib.servers import ThreadedFTPServer # <- 316 from pyftpdlib.authorizers import DummyAuthorizer 317 318 def main(): 319 authorizer = DummyAuthorizer() 320 authorizer.add_user('user', '12345', '.') 321 handler = FTPHandler 322 handler.authorizer = authorizer 323 server = ThreadedFTPServer(('', 2121), handler) 324 server.serve_forever() 325 326 if __name__ == "__main__": 327 main() 328 329 330Multiple processes 331^^^^^^^^^^^^^^^^^^ 332 333.. code-block:: python 334 335 from pyftpdlib.handlers import FTPHandler 336 from pyftpdlib.servers import MultiprocessFTPServer # <- 337 from pyftpdlib.authorizers import DummyAuthorizer 338 339 def main(): 340 authorizer = DummyAuthorizer() 341 authorizer.add_user('user', '12345', '.') 342 handler = FTPHandler 343 handler.authorizer = authorizer 344 server = MultiprocessFTPServer(('', 2121), handler) 345 server.serve_forever() 346 347 if __name__ == "__main__": 348 main() 349 350.. _pre-fork: 351 352Pre-fork 353^^^^^^^^ 354 355There also exists a third option (UNIX only): the pre-fork model. 356Pre-fork means that a certain number of worker processes are ``spawn()``ed 357before starting the server. 358Each worker process will keep using a 1-thread, async concurrency model, 359handling multiple concurrent connections, but the workload is split. 360This way the delay introduced by a blocking function call is amortized and 361divided by the number of workers, and thus also the disk I/O latency is 362minimized. 363Every time a new connection comes in, the parent process will automatically 364delegate the connection to one of the subprocesses, so from the app standpoint 365this is completely transparent. 366As a general rule, it is always a good idea to use this model in production. 367The optimal value depends on many factors including (but not limited to) the 368number of CPU cores, the number of hard disk drives that store data, and load 369pattern. When one is in doubt, setting it to the number of available CPU cores 370would be a good start. 371 372.. code-block:: python 373 374 from pyftpdlib.handlers import FTPHandler 375 from pyftpdlib.servers import FTPServer 376 from pyftpdlib.authorizers import DummyAuthorizer 377 378 def main(): 379 authorizer = DummyAuthorizer() 380 authorizer.add_user('user', '12345', '.') 381 handler = FTPHandler 382 handler.authorizer = authorizer 383 server = FTPServer(('', 2121), handler) 384 server.serve_forever(worker_processes=4) # <- 385 386 if __name__ == "__main__": 387 main() 388 389Throttle bandwidth 390================== 391 392An important feature for an ftpd is limiting the speed for downloads and 393uploads affecting the data channel. 394`ThrottledDTPHandler.banner <api.html#pyftpdlib.handlers.ThrottledDTPHandler>`__ 395can be used to set such limits. 396The basic idea behind ``ThrottledDTPHandler`` is to wrap sending and receiving 397in a data counter and temporary "sleep" the data channel so that you burst to 398no more than x Kb/sec average. When it realizes that more than x Kb in a second 399are being transmitted it temporary blocks the transfer for a certain number of 400seconds. 401 402.. code-block:: python 403 404 import os 405 406 from pyftpdlib.handlers import FTPHandler, ThrottledDTPHandler 407 from pyftpdlib.servers import FTPServer 408 from pyftpdlib.authorizers import DummyAuthorizer 409 410 def main(): 411 authorizer = DummyAuthorizer() 412 authorizer.add_user('user', '12345', os.getcwd(), perm='elradfmwMT') 413 authorizer.add_anonymous(os.getcwd()) 414 415 dtp_handler = ThrottledDTPHandler 416 dtp_handler.read_limit = 30720 # 30 Kb/sec (30 * 1024) 417 dtp_handler.write_limit = 30720 # 30 Kb/sec (30 * 1024) 418 419 ftp_handler = FTPHandler 420 ftp_handler.authorizer = authorizer 421 # have the ftp handler use the alternative dtp handler class 422 ftp_handler.dtp_handler = dtp_handler 423 424 server = FTPServer(('', 2121), ftp_handler) 425 server.serve_forever() 426 427 if __name__ == '__main__': 428 main() 429 430 431FTPS (FTP over TLS/SSL) server 432============================== 433 434Starting from version 0.6.0 pyftpdlib finally includes full FTPS support 435implementing both TLS and SSL protocols and *AUTH*, *PBSZ* and *PROT* commands 436as defined in `RFC-4217 <http://www.ietf.org/rfc/rfc4217.txt>`__. This has been 437implemented by using `PyOpenSSL <http://pypi.python.org/pypi/pyOpenSSL>`__ 438module, which is required in order to run the code below. 439`TLS_FTPHandler <api.html#pyftpdlib.handlers.TLS_FTPHandler>`__ 440class requires at least a ``certfile`` to be specified and optionally a 441``keyfile``. 442`Apache FAQs <https://httpd.apache.org/docs/2.4/ssl/ssl_faq.html#selfcert>`__ provide 443instructions on how to generate them. If you don't care about having your 444personal self-signed certificates you can use the one in the demo directory 445which include both and is available 446`here <https://github.com/giampaolo/pyftpdlib/blob/master/demo/keycert.pem>`__. 447 448`source code <https://github.com/giampaolo/pyftpdlib/blob/master/demo/tls_ftpd.py>`__ 449 450.. code-block:: python 451 452 """ 453 An RFC-4217 asynchronous FTPS server supporting both SSL and TLS. 454 Requires PyOpenSSL module (http://pypi.python.org/pypi/pyOpenSSL). 455 """ 456 457 from pyftpdlib.servers import FTPServer 458 from pyftpdlib.authorizers import DummyAuthorizer 459 from pyftpdlib.handlers import TLS_FTPHandler 460 461 def main(): 462 authorizer = DummyAuthorizer() 463 authorizer.add_user('user', '12345', '.', perm='elradfmwMT') 464 authorizer.add_anonymous('.') 465 handler = TLS_FTPHandler 466 handler.certfile = 'keycert.pem' 467 handler.authorizer = authorizer 468 # requires SSL for both control and data channel 469 #handler.tls_control_required = True 470 #handler.tls_data_required = True 471 server = FTPServer(('', 21), handler) 472 server.serve_forever() 473 474 if __name__ == '__main__': 475 main() 476 477 478Event callbacks 479=============== 480 481A small example which shows how to use callback methods via 482`FTPHandler <api.html#pyftpdlib.handlers.FTPHandler>`__ subclassing: 483 484.. code-block:: python 485 486 from pyftpdlib.handlers import FTPHandler 487 from pyftpdlib.servers import FTPServer 488 from pyftpdlib.authorizers import DummyAuthorizer 489 490 491 class MyHandler(FTPHandler): 492 493 def on_connect(self): 494 print "%s:%s connected" % (self.remote_ip, self.remote_port) 495 496 def on_disconnect(self): 497 # do something when client disconnects 498 pass 499 500 def on_login(self, username): 501 # do something when user login 502 pass 503 504 def on_logout(self, username): 505 # do something when user logs out 506 pass 507 508 def on_file_sent(self, file): 509 # do something when a file has been sent 510 pass 511 512 def on_file_received(self, file): 513 # do something when a file has been received 514 pass 515 516 def on_incomplete_file_sent(self, file): 517 # do something when a file is partially sent 518 pass 519 520 def on_incomplete_file_received(self, file): 521 # remove partially uploaded files 522 import os 523 os.remove(file) 524 525 526 def main(): 527 authorizer = DummyAuthorizer() 528 authorizer.add_user('user', '12345', homedir='.', perm='elradfmwMT') 529 authorizer.add_anonymous(homedir='.') 530 531 handler = MyHandler 532 handler.authorizer = authorizer 533 server = FTPServer(('', 2121), handler) 534 server.serve_forever() 535 536 if __name__ == "__main__": 537 main() 538 539 540Command line usage 541================== 542 543Starting from version 0.6.0 pyftpdlib can be run as a simple stand-alone server 544via Python's -m option, which is particularly useful when you want to quickly 545share a directory. Some examples. 546Anonymous FTPd sharing current directory: 547 548.. code-block:: sh 549 550 $ python -m pyftpdlib 551 [I 13-04-09 17:55:18] >>> starting FTP server on 0.0.0.0:2121, pid=6412 <<< 552 [I 13-04-09 17:55:18] poller: <class 'pyftpdlib.ioloop.Epoll'> 553 [I 13-04-09 17:55:18] masquerade (NAT) address: None 554 [I 13-04-09 17:55:18] passive ports: None 555 [I 13-04-09 17:55:18] use sendfile(2): True 556 557Anonymous FTPd with write permission: 558 559.. code-block:: sh 560 561 $ python -m pyftpdlib -w 562 563Set a different address/port and home directory: 564 565.. code-block:: sh 566 567 $ python -m pyftpdlib -i localhost -p 8021 -d /home/someone 568 569See ``python -m pyftpdlib -h`` for a complete list of options. 570