1==== 2RAET 3==== 4by SaltStack 5============ 6 7.. rubric:: Reliable Asynchronous Event Transport Protocol 8 9Contents: 10 11.. toctree:: 12 :maxdepth: 1 13 14 topics/introduction 15 topics/installation 16 17Motivation 18========== 19 20Modern large scale distributed application architectures where components are 21distributed across the internet on multiple hosts and multiple CPU cores are 22often based on a messaging or event bus that allows the various distributed 23components to communicate asynchronously with each other. Typically the 24messaging bus is some form of messaging queue service such as AMQP or ZeroMQ. 25The message bus supports what is commonly referred to as a publish/subscribe 26methodology for information exchange. 27 28While there are many advantages to a full featured message queuing service, one 29of the disadvantagesis the inability to manage performance at scale. 30 31A message queuing service performs two distinct but complementary functions. 32 33- The first is asynchronous transport of messages over the internet. 34 35- The second is message queue management, that is, the identification, 36 tracking, storage, and distribution of messages between publishers and 37 subscribers via queues. 38 39One of the advantages of a message queuing service for many applications is 40that the service hides the complexities of queue management from the clients 41behind an API. The disadvantage is that at scale where the volume of messages, 42the timing of messages, and the associated demands on memory, network, and cpu 43capacity become critical, the client has little ability to tune the service for 44performance. Often MQ services become bottlenecks for the distributed 45application. The more complicated MQ services like AMQP tend to be unreliable 46under load. 47 48Separating the asynchrounous event over network transport function from the 49message queue management allows independant tuning at scale of each function. 50 51Moreover, most if not all of the MQ services that we are familiar with are 52based on TCP/IP which due to the way it handles connection setup and teardown 53as well as failed connections in order to support streams tends to add high 54latency to the network communications and is not therefore not well suited for 55the asynchronous nature of distibuted event driven application communications. 56 57Because UDP/IP has lower latency and is connectionless, it scales better, but 58it has the serious drawback that it is not reliable. What it needed is a tuned 59transport protocol adds reliability to UDP/IP without sacrificing latency and 60scalability. A transactioned protocol, is much more appropriate for providing 61reliablity to asynchronous event transport than a streaming protocol. 62 63In addition, because most MQ services are based on TCP/IP they tend to also use 64HTTP and therefore TLS/SSL for secure communications. While using HTTP provides 65easy integration with web based systems, it is problematic for high performant 66systems and TLS is also problematic as a security system both from performance 67and vulnerabilty aspects. Elliptic Curve Cryptography provides increases in 68security with lower performance requires over other approaches. LibSodium 69provides an open source Elliptic Curve Cryptographic library with support for 70both authentication and encryption. The CurveCP protocol is based on LibSodium 71and provides a handshake protocol for bootstrapping secure network exchanges of 72information. 73 74Finally, one of the best ways to manage and fine tune processor resources (cpu, 75memory, network) in distrubted concurrent event driven applications is to use 76something called micro-threads. A microthread is typically and in-language 77feature that allows logical concurrency with no more overhead than a function 78call. Micro threading uses cooperative multi-tasking instead of threads and/or 79processes and avoids many of the complexities of resource contention, context 80switching, and interprocess communications while providing much higher total 81performance. 82 83The main limitation of a micro-threaded application is that it is constrained 84to one CPU core because it runs in one process. To enable full utilization of 85all CPU cores, the application needs to be able to run at least one process per 86CPU core. This requires on host inter-process communications. But instead of 87one process per logical concurrent function which is the conventional approach 88to multi-processing, A micro-threaded multi-process application has one 89micro-thread per logical concurrent function and the total number of 90micro-threads is distributed amoungst the minimum number of processes, no more 91than the number of cpu cores. This optimizes the use of the cpu power while 92minimizes the overhead of process context switching. 93 94An example of a framework that uses this type of micro-threaded but 95multi-process architecture is Erlang. Indeed, Erlang provided confirmation that 96the approach to RAET could be viable. Unfortunately, the Erlang ecosystem is 97somewhat limited and the language itself uses what one might describe as a very 98unfortunate syntax. Since we have extensive background in Python we wanted to 99leverage the richness of the Python ecosystem but still be able to develop 100distributed applications on a micro-threaded multi-process capable framework. 101 102 103RAET is designed to provide secure reliable scalable asynchronous message/event 104transport over the internet in a micro-threaded multi-process application 105framework that uses UDP for interhost communication and LibSodium for 106authentication, encryption and the CurveCP handshake for secure bootstrap. 107 108The queue management and micro-threaded application support is provided by 109Ioflo. RAET is a complementary project to Ioflo in that RAET enables multiple 110Ioflo applications to work together over a network as part of a distributed 111application. 112 113The primary use case and motivating problem that resulted in the development of 114RAET was the need to enable SaltStack to scale better. SaltStack is a remote 115execution and configuration management platform written in Python. SaltStack 116uses ZeroMQ (0MQ) as is message bug or message queuing service. ZeroMQ is based 117on 118 119Indices and tables 120================== 121 122* :ref:`genindex` 123* :ref:`modindex` 124* :ref:`search` 125 126