1*2b15cb3dSCy SchubertWhat's New In Libevent 2.0 so far: 2*2b15cb3dSCy Schubert 3*2b15cb3dSCy Schubert1. Meta-issues 4*2b15cb3dSCy Schubert 5*2b15cb3dSCy Schubert1.1. About this document 6*2b15cb3dSCy Schubert 7*2b15cb3dSCy Schubert This document describes the key differences between Libevent 1.4 and 8*2b15cb3dSCy Schubert Libevent 2.0, from a user's point of view. It was most recently 9*2b15cb3dSCy Schubert updated based on features in git master as of August 2010. 10*2b15cb3dSCy Schubert 11*2b15cb3dSCy Schubert NOTE: I am very sure that I missed some thing on this list. Caveat 12*2b15cb3dSCy Schubert haxxor. 13*2b15cb3dSCy Schubert 14*2b15cb3dSCy Schubert1.2. Better documentation 15*2b15cb3dSCy Schubert 16*2b15cb3dSCy Schubert There is now a book-in-progress that explains how to use Libevent and its 17*2b15cb3dSCy Schubert growing pile of APIs. As of this writing, it covers everything except the 18*2b15cb3dSCy Schubert http and rpc code. Check out the latest draft at 19*2b15cb3dSCy Schubert http://www.wangafu.net/~nickm/libevent-book/ . 20*2b15cb3dSCy Schubert 21*2b15cb3dSCy Schubert2. New and Improved Event APIs 22*2b15cb3dSCy Schubert 23*2b15cb3dSCy Schubert Many APIs are improved, refactored, or deprecated in Libevent 2.0. 24*2b15cb3dSCy Schubert 25*2b15cb3dSCy Schubert COMPATIBILITY: 26*2b15cb3dSCy Schubert 27*2b15cb3dSCy Schubert Nearly all existing code that worked with Libevent 1.4 should still 28*2b15cb3dSCy Schubert work correctly with Libevent 2.0. However, if you are writing new code, 29*2b15cb3dSCy Schubert or if you want to port old code, we strongly recommend using the new APIs 30*2b15cb3dSCy Schubert and avoiding deprecated APIs as much as possible. 31*2b15cb3dSCy Schubert 32*2b15cb3dSCy Schubert Binaries linked against Libevent 1.4 will need to be recompiled to link 33*2b15cb3dSCy Schubert against Libevent 2.0. This is nothing new; we have never been good at 34*2b15cb3dSCy Schubert preserving binary compatibility between releases. We'll try harder in the 35*2b15cb3dSCy Schubert future, though: see 2.1 below. 36*2b15cb3dSCy Schubert 37*2b15cb3dSCy Schubert2.1. New header layout for improved forward-compatibility 38*2b15cb3dSCy Schubert 39*2b15cb3dSCy Schubert Libevent 2.0 has a new header layout to make it easier for programmers to 40*2b15cb3dSCy Schubert write good, well-supported libevent code. The new headers are divided 41*2b15cb3dSCy Schubert into three types. 42*2b15cb3dSCy Schubert 43*2b15cb3dSCy Schubert There are *regular headers*, like event2/event.h. These headers contain 44*2b15cb3dSCy Schubert the functions that most programmers will want to use. 45*2b15cb3dSCy Schubert 46*2b15cb3dSCy Schubert There are *backward compatibility headers*, like event2/event_compat.h. 47*2b15cb3dSCy Schubert These headers contain declarations for deprecated functions from older 48*2b15cb3dSCy Schubert versions of Libevent. Documentation in these headers should suggest what's 49*2b15cb3dSCy Schubert wrong with the old functions, and what functions you want to start using 50*2b15cb3dSCy Schubert instead of the old ones. Some of these functions might be removed in a 51*2b15cb3dSCy Schubert future release. New programs should generally not include these headers. 52*2b15cb3dSCy Schubert 53*2b15cb3dSCy Schubert Finally, there are *structure headers*, like event2/event_struct.h. 54*2b15cb3dSCy Schubert These headers contain definitions of some structures that Libevent has 55*2b15cb3dSCy Schubert historically exposed. Exposing them caused problems in the past, 56*2b15cb3dSCy Schubert since programs that were compiled to work with one version of Libevent 57*2b15cb3dSCy Schubert would often stop working with another version that changed the size or 58*2b15cb3dSCy Schubert layout of some object. We've moving them into separate headers so 59*2b15cb3dSCy Schubert that programmers can know that their code is not depending on any 60*2b15cb3dSCy Schubert unstable aspect of the Libvent ABI. New programs should generally not 61*2b15cb3dSCy Schubert include these headers unless they really know what they are doing, are 62*2b15cb3dSCy Schubert willing to rebuild their software whenever they want to link it 63*2b15cb3dSCy Schubert against a new version of Libevent, and are willing to risk their code 64*2b15cb3dSCy Schubert breaking if and when data structures change. 65*2b15cb3dSCy Schubert 66*2b15cb3dSCy Schubert Functionality that once was located in event.h is now more subdivided. 67*2b15cb3dSCy Schubert The core event logic is now in event2/event.h. The "evbuffer" functions 68*2b15cb3dSCy Schubert for low-level buffer manipulation are in event2/buffer.h. The 69*2b15cb3dSCy Schubert "bufferevent" functions for higher-level buffered IO are in 70*2b15cb3dSCy Schubert event2/bufferevent.h. 71*2b15cb3dSCy Schubert 72*2b15cb3dSCy Schubert COMPATIBILITY: 73*2b15cb3dSCy Schubert 74*2b15cb3dSCy Schubert All of the old headers (event.h, evdns.h, evhttp.h, evrpc.h, and 75*2b15cb3dSCy Schubert evutil.h) will continue to work by including the corresponding new 76*2b15cb3dSCy Schubert headers. Old code should not be broken by this change. 77*2b15cb3dSCy Schubert 78*2b15cb3dSCy Schubert2.2. New thread-safe, binary-compatible, harder-to-mess-up APIs 79*2b15cb3dSCy Schubert 80*2b15cb3dSCy Schubert Some aspects of the historical Libevent API have encouraged 81*2b15cb3dSCy Schubert non-threadsafe code, or forced code built against one version of Libevent 82*2b15cb3dSCy Schubert to no longer build with another. The problems with now-deprecated APIs 83*2b15cb3dSCy Schubert fell into two categories: 84*2b15cb3dSCy Schubert 85*2b15cb3dSCy Schubert 1) Dependence on the "current" event_base. In an application with 86*2b15cb3dSCy Schubert multiple event_bases, Libevent previously had a notion of the 87*2b15cb3dSCy Schubert "current" event_base. New events were linked to this base, and 88*2b15cb3dSCy Schubert the caller needed to explicitly reattach them to another base. 89*2b15cb3dSCy Schubert This was horribly error-prone. 90*2b15cb3dSCy Schubert 91*2b15cb3dSCy Schubert Functions like "event_set" that worked with the "current" event_base 92*2b15cb3dSCy Schubert are now deprecated but still available (see 2.1). There are new 93*2b15cb3dSCy Schubert functions like "event_assign" that take an explicit event_base 94*2b15cb3dSCy Schubert argument when setting up a structure. Using these functions will help 95*2b15cb3dSCy Schubert prevent errors in your applications, and to be more threadsafe. 96*2b15cb3dSCy Schubert 97*2b15cb3dSCy Schubert 2) Structure dependence. Applications needed to allocate 'struct 98*2b15cb3dSCy Schubert event' themselves, since there was no function in Libevent to do it 99*2b15cb3dSCy Schubert for them. But since the size and contents of struct event can 100*2b15cb3dSCy Schubert change between libevent versions, this created binary-compatibility 101*2b15cb3dSCy Schubert nightmares. All structures of this kind are now isolated in 102*2b15cb3dSCy Schubert _struct.h header (see 2.1), and there are new allocate-and- 103*2b15cb3dSCy Schubert initialize functions you can use instead of the old initialize-only 104*2b15cb3dSCy Schubert functions. For example, instead of malloc and event_set, you 105*2b15cb3dSCy Schubert can use event_new(). 106*2b15cb3dSCy Schubert 107*2b15cb3dSCy Schubert (For people who do really want to allocate a struct event on the 108*2b15cb3dSCy Schubert stack, or put one inside another structure, you can still use 109*2b15cb3dSCy Schubert event2/event_compat.h.) 110*2b15cb3dSCy Schubert 111*2b15cb3dSCy Schubert So in the case where old code would look like this: 112*2b15cb3dSCy Schubert 113*2b15cb3dSCy Schubert #include <event.h> 114*2b15cb3dSCy Schubert ... 115*2b15cb3dSCy Schubert struct event *ev = malloc(sizeof(struct event)); 116*2b15cb3dSCy Schubert /* This call will cause a buffer overrun if you compile with one version 117*2b15cb3dSCy Schubert of Libevent and link dynamically against another. */ 118*2b15cb3dSCy Schubert event_set(ev, fd, EV_READ, cb, NULL); 119*2b15cb3dSCy Schubert /* If you forget this call, your code will break in hard-to-diagnose 120*2b15cb3dSCy Schubert ways in the presence of multiple event bases. */ 121*2b15cb3dSCy Schubert event_set_base(ev, base); 122*2b15cb3dSCy Schubert 123*2b15cb3dSCy Schubert New code will look more like this: 124*2b15cb3dSCy Schubert 125*2b15cb3dSCy Schubert #include <event2/event.h> 126*2b15cb3dSCy Schubert ... 127*2b15cb3dSCy Schubert struct event *ev; 128*2b15cb3dSCy Schubert ev = event_new(base, fd, EV_READ, cb, NULL); 129*2b15cb3dSCy Schubert 130*2b15cb3dSCy Schubert2.3. Overrideable allocation functions 131*2b15cb3dSCy Schubert 132*2b15cb3dSCy Schubert If you want to override the allocation functions used by libevent 133*2b15cb3dSCy Schubert (for example, to use a specialized allocator, or debug memory 134*2b15cb3dSCy Schubert issues, or so on), you can replace them by calling 135*2b15cb3dSCy Schubert event_set_mem_functions. It takes replacements for malloc(), 136*2b15cb3dSCy Schubert free(), and realloc(). 137*2b15cb3dSCy Schubert 138*2b15cb3dSCy Schubert If you're going to use this facility, you need to call it _before_ 139*2b15cb3dSCy Schubert Libevent does any memory allocation; otherwise, Libevent may allocate some 140*2b15cb3dSCy Schubert memory with malloc(), and free it with the free() function you provide. 141*2b15cb3dSCy Schubert 142*2b15cb3dSCy Schubert You can disable this feature when you are building Libevent by passing 143*2b15cb3dSCy Schubert the --disable-malloc-replacement argument to configure. 144*2b15cb3dSCy Schubert 145*2b15cb3dSCy Schubert2.4. Configurable event_base creation 146*2b15cb3dSCy Schubert 147*2b15cb3dSCy Schubert Older versions of Libevent would always got the fastest backend 148*2b15cb3dSCy Schubert available, unless you reconfigured their behavior with the environment 149*2b15cb3dSCy Schubert variables EVENT_NOSELECT, EVENT_NOPOLL, and so forth. This was annoying 150*2b15cb3dSCy Schubert to programmers who wanted to pick a backend explicitly without messing 151*2b15cb3dSCy Schubert with the environment. 152*2b15cb3dSCy Schubert 153*2b15cb3dSCy Schubert Also, despite our best efforts, not every backend supports every 154*2b15cb3dSCy Schubert operation we might like. Some features (like edge-triggered events, or 155*2b15cb3dSCy Schubert working with non-socket file descriptors) only work with some operating 156*2b15cb3dSCy Schubert systems' fast backends. Previously, programmers who cared about this 157*2b15cb3dSCy Schubert needed to know which backends supported what. This tended to get quite 158*2b15cb3dSCy Schubert ungainly. 159*2b15cb3dSCy Schubert 160*2b15cb3dSCy Schubert There is now an API to choose backends, either by name or by feature. 161*2b15cb3dSCy Schubert Here is an example: 162*2b15cb3dSCy Schubert 163*2b15cb3dSCy Schubert struct event_config_t *config; 164*2b15cb3dSCy Schubert struct event_base *base; 165*2b15cb3dSCy Schubert 166*2b15cb3dSCy Schubert /* Create a new configuration object. */ 167*2b15cb3dSCy Schubert config = event_config_new(); 168*2b15cb3dSCy Schubert /* We don't want to use the "select" method. */ 169*2b15cb3dSCy Schubert event_config_avoid_method(config, "select"); 170*2b15cb3dSCy Schubert /* We want a method that can work with non-socket file descriptors */ 171*2b15cb3dSCy Schubert event_config_require_features(config, EV_FEATURE_FDS); 172*2b15cb3dSCy Schubert 173*2b15cb3dSCy Schubert base = event_base_new_with_config(config); 174*2b15cb3dSCy Schubert if (!base) { 175*2b15cb3dSCy Schubert /* There is no backend method that does what we want. */ 176*2b15cb3dSCy Schubert exit(1); 177*2b15cb3dSCy Schubert } 178*2b15cb3dSCy Schubert event_config_free(config); 179*2b15cb3dSCy Schubert 180*2b15cb3dSCy Schubert Supported features are documented in event2/event.h 181*2b15cb3dSCy Schubert 182*2b15cb3dSCy Schubert2.5. Socket is now an abstract type 183*2b15cb3dSCy Schubert 184*2b15cb3dSCy Schubert All APIs that formerly accepted int as a socket type now accept 185*2b15cb3dSCy Schubert "evutil_socket_t". On Unix, this is just an alias for "int" as 186*2b15cb3dSCy Schubert before. On Windows, however, it's an alias for SOCKET, which can 187*2b15cb3dSCy Schubert be wider than int on 64-bit platforms. 188*2b15cb3dSCy Schubert 189*2b15cb3dSCy Schubert2.6. Timeouts and persistent events work together. 190*2b15cb3dSCy Schubert 191*2b15cb3dSCy Schubert Previously, it wasn't useful to set a timeout on a persistent event: 192*2b15cb3dSCy Schubert the timeout would trigger once, and never again. This is not what 193*2b15cb3dSCy Schubert applications tend to want. Instead, applications tend to want every 194*2b15cb3dSCy Schubert triggering of the event to re-set the timeout. So now, if you set 195*2b15cb3dSCy Schubert up an event like this: 196*2b15cb3dSCy Schubert struct event *ev; 197*2b15cb3dSCy Schubert struct timeval tv; 198*2b15cb3dSCy Schubert ev = event_new(base, fd, EV_READ|EV_PERSIST, cb, NULL); 199*2b15cb3dSCy Schubert tv.tv_sec = 1; 200*2b15cb3dSCy Schubert tv.tv_usec = 0; 201*2b15cb3dSCy Schubert event_add(ev, &tv); 202*2b15cb3dSCy Schubert 203*2b15cb3dSCy Schubert The callback 'cb' will be invoked whenever fd is ready to read, OR whenever 204*2b15cb3dSCy Schubert a second has passed since the last invocation of cb. 205*2b15cb3dSCy Schubert 206*2b15cb3dSCy Schubert2.7. Multiple events allowed per fd 207*2b15cb3dSCy Schubert 208*2b15cb3dSCy Schubert Older versions of Libevent allowed at most one EV_READ event and at most 209*2b15cb3dSCy Schubert one EV_WRITE event per socket, per event base. This restriction is no 210*2b15cb3dSCy Schubert longer present. 211*2b15cb3dSCy Schubert 212*2b15cb3dSCy Schubert2.8. evthread_* functions for thread-safe structures. 213*2b15cb3dSCy Schubert 214*2b15cb3dSCy Schubert Libevent structures can now be built with locking support. This code 215*2b15cb3dSCy Schubert makes it safe to add, remove, and activate events on an event base from a 216*2b15cb3dSCy Schubert different thread. (Previously, if you wanted to write multithreaded code 217*2b15cb3dSCy Schubert with Libevent, you could only an event_base or its events in one thread at 218*2b15cb3dSCy Schubert a time.) 219*2b15cb3dSCy Schubert 220*2b15cb3dSCy Schubert If you want threading support and you're using pthreads, you can just 221*2b15cb3dSCy Schubert call evthread_use_pthreads(). (You'll need to link against the 222*2b15cb3dSCy Schubert libevent_pthreads library in addition to libevent_core. These functions are 223*2b15cb3dSCy Schubert not in libevent_core.) 224*2b15cb3dSCy Schubert 225*2b15cb3dSCy Schubert If you want threading support and you're using Windows, you can just 226*2b15cb3dSCy Schubert call evthread_use_windows_threads(). 227*2b15cb3dSCy Schubert 228*2b15cb3dSCy Schubert If you are using some locking system besides Windows and pthreads, You 229*2b15cb3dSCy Schubert can enable this on a per-event-base level by writing functions to 230*2b15cb3dSCy Schubert implement mutexes, conditions, and thread IDs, and passing them to 231*2b15cb3dSCy Schubert evthread_set_lock_callbacks and related functions in event2/thread.h. 232*2b15cb3dSCy Schubert 233*2b15cb3dSCy Schubert Once locking functions are enabled, every new event_base is created with a 234*2b15cb3dSCy Schubert lock. You can prevent a single event_base from being built with a lock 235*2b15cb3dSCy Schubert disabled by using the EVENT_BASE_FLAG_NOLOCK flag in its 236*2b15cb3dSCy Schubert event_config. If an event_base is created with a lock, it is safe to call 237*2b15cb3dSCy Schubert event_del, event_add, and event_active on its events from any thread. The 238*2b15cb3dSCy Schubert event callbacks themselves are still all executed from the thread running 239*2b15cb3dSCy Schubert the event loop. 240*2b15cb3dSCy Schubert 241*2b15cb3dSCy Schubert To make an evbuffer or a bufferevent object threadsafe, call its 242*2b15cb3dSCy Schubert *_enable_locking() function. 243*2b15cb3dSCy Schubert 244*2b15cb3dSCy Schubert The HTTP api is not currently threadsafe. 245*2b15cb3dSCy Schubert 246*2b15cb3dSCy Schubert To build Libevent with threading support disabled, pass 247*2b15cb3dSCy Schubert --disable-thread-support to the configure script. 248*2b15cb3dSCy Schubert 249*2b15cb3dSCy Schubert2.9. Edge-triggered events on some backends. 250*2b15cb3dSCy Schubert 251*2b15cb3dSCy Schubert With some backends, it's now possible to add the EV_ET flag to an event 252*2b15cb3dSCy Schubert in order to request that the event's semantics be edge-triggered. Right 253*2b15cb3dSCy Schubert now, epoll and kqueue support this. 254*2b15cb3dSCy Schubert 255*2b15cb3dSCy Schubert The corresponding event_config feature is EV_FEATURE_ET; see 2.4 for more 256*2b15cb3dSCy Schubert information. 257*2b15cb3dSCy Schubert 258*2b15cb3dSCy Schubert2.10. Better support for huge numbers of timeouts 259*2b15cb3dSCy Schubert 260*2b15cb3dSCy Schubert The heap-based priority queue timer implementation for Libevent 1.4 is good 261*2b15cb3dSCy Schubert for randomly distributed timeouts, but suboptimal if you have huge numbers 262*2b15cb3dSCy Schubert of timeouts that all expire in the same amount of time after their 263*2b15cb3dSCy Schubert creation. The new event_base_init_common_timeout() logic lets you signal 264*2b15cb3dSCy Schubert that a given timeout interval will be very common, and should use a linked 265*2b15cb3dSCy Schubert list implementation instead of a priority queue. 266*2b15cb3dSCy Schubert 267*2b15cb3dSCy Schubert2.11. Improved debugging support 268*2b15cb3dSCy Schubert 269*2b15cb3dSCy Schubert It's been pretty easy to forget to delete all your events before you 270*2b15cb3dSCy Schubert re-initialize them, or otherwise put Libevent in an internally inconsistent 271*2b15cb3dSCy Schubert state. You can tell libevent to catch these and other common errors with 272*2b15cb3dSCy Schubert the new event_enable_debug_mode() call. Just invoke it before you do 273*2b15cb3dSCy Schubert any calls to other libevent functions, and it'll catch many common 274*2b15cb3dSCy Schubert event-level errors in your code. 275*2b15cb3dSCy Schubert 276*2b15cb3dSCy Schubert2.12. Functions to access all event fields 277*2b15cb3dSCy Schubert 278*2b15cb3dSCy Schubert So that you don't have to access the struct event fields directly, Libevent 279*2b15cb3dSCy Schubert now provides accessor functions to retrieve everything from an event that 280*2b15cb3dSCy Schubert you set during event_new() or event_assign(). 281*2b15cb3dSCy Schubert 282*2b15cb3dSCy Schubert3. Backend-specific and performance improvements. 283*2b15cb3dSCy Schubert 284*2b15cb3dSCy Schubert3.1. Change-minimization on O(1) backends 285*2b15cb3dSCy Schubert 286*2b15cb3dSCy Schubert With previous versions of Libevent, if you called event_del() and 287*2b15cb3dSCy Schubert event_add() repeatedly on a single event between trips to the backend's 288*2b15cb3dSCy Schubert dispatch function, the backend might wind up making unnecessary calls or 289*2b15cb3dSCy Schubert passing unnecessary data to the kernel. The new backend logic batches up 290*2b15cb3dSCy Schubert redundant adds and deletes, and performs no more operations than necessary 291*2b15cb3dSCy Schubert at the kernel level. 292*2b15cb3dSCy Schubert 293*2b15cb3dSCy Schubert This logic is on for the kqueue backend, and available (but off by 294*2b15cb3dSCy Schubert default) for the epoll backend. To turn it on for the epoll backend, 295*2b15cb3dSCy Schubert set the EVENT_BASE_FLAG_EPOLL_USE_CHANGELIST flag in the 296*2b15cb3dSCy Schubert event_base_cofig, or set the EVENT_EPOLL_USE_CHANGELIST environment 297*2b15cb3dSCy Schubert variable. Doing this with epoll may result in weird bugs if you give 298*2b15cb3dSCy Schubert any fds closed by dup() or its variants. 299*2b15cb3dSCy Schubert 300*2b15cb3dSCy Schubert3.2. Improved notification on Linux 301*2b15cb3dSCy Schubert 302*2b15cb3dSCy Schubert When we need to wake the event loop up from another thread, we use 303*2b15cb3dSCy Schubert an epollfd to do so, instead of a socketpair. This is supposed to be 304*2b15cb3dSCy Schubert faster. 305*2b15cb3dSCy Schubert 306*2b15cb3dSCy Schubert3.3. Windows: better support for everything 307*2b15cb3dSCy Schubert 308*2b15cb3dSCy Schubert Bufferevents on Windows can use a new mechanism (off-by-default; see below) 309*2b15cb3dSCy Schubert to send their data via Windows overlapped IO and get their notifications 310*2b15cb3dSCy Schubert via the IOCP API. This should be much faster than using event-based 311*2b15cb3dSCy Schubert notification. 312*2b15cb3dSCy Schubert 313*2b15cb3dSCy Schubert Other functions throughout the code have been fixed to work more 314*2b15cb3dSCy Schubert consistently with Windows. Libevent now builds on Windows using either 315*2b15cb3dSCy Schubert mingw, or using MSVC (with nmake). Libevent works fine with UNICODE 316*2b15cb3dSCy Schubert defined, or not. 317*2b15cb3dSCy Schubert 318*2b15cb3dSCy Schubert Data structures are a little smarter: our lookups from socket to pending 319*2b15cb3dSCy Schubert event are now done with O(1) hash tables rather than O(lg n) red-black 320*2b15cb3dSCy Schubert trees. 321*2b15cb3dSCy Schubert 322*2b15cb3dSCy Schubert Unfortunately, the main Windows backend is still select()-based: from 323*2b15cb3dSCy Schubert testing the IOCP backends on the mailing list, it seems that there isn't 324*2b15cb3dSCy Schubert actually a way to tell for certain whether a socket is writable with IOCP. 325*2b15cb3dSCy Schubert Libevent 2.1 may add a multithreaded WaitForMultipleEvents-based 326*2b15cb3dSCy Schubert backend for better performance with many inactive sockets and better 327*2b15cb3dSCy Schubert integration with Windows events. 328*2b15cb3dSCy Schubert 329*2b15cb3dSCy Schubert4. Improvements to evbuffers 330*2b15cb3dSCy Schubert 331*2b15cb3dSCy Schubert Libevent has long had an "evbuffer" implementation to wrap access to an 332*2b15cb3dSCy Schubert input or output memory buffer. In previous versions, the implementation 333*2b15cb3dSCy Schubert was very inefficient and lacked some desirable features. We've made many 334*2b15cb3dSCy Schubert improvements in Libevent 2.0. 335*2b15cb3dSCy Schubert 336*2b15cb3dSCy Schubert4.1. Chunked-memory internal representation 337*2b15cb3dSCy Schubert 338*2b15cb3dSCy Schubert Previously, each evbuffer was a huge chunk of memory. When we ran out of 339*2b15cb3dSCy Schubert space in an evbuffer, we used realloc() to grow the chunk of memory. When 340*2b15cb3dSCy Schubert data was misaligned, we used memmove to move the data back to the front 341*2b15cb3dSCy Schubert of the buffer. 342*2b15cb3dSCy Schubert 343*2b15cb3dSCy Schubert Needless to say, this is a terrible interface for networked IO. 344*2b15cb3dSCy Schubert 345*2b15cb3dSCy Schubert Now, evbuffers are implemented as a linked list of memory chunks, like 346*2b15cb3dSCy Schubert most Unix kernels use for network IO. (See Linux's skbuf interfaces, 347*2b15cb3dSCy Schubert or *BSD's mbufs). Data is added at the end of the linked list and 348*2b15cb3dSCy Schubert removed from the front, so that we don't ever need realloc huge chunks 349*2b15cb3dSCy Schubert or memmove the whole buffer contents. 350*2b15cb3dSCy Schubert 351*2b15cb3dSCy Schubert To avoid excessive calls to read and write, we use the readv/writev 352*2b15cb3dSCy Schubert interfaces (or WSASend/WSARecv on Windows) to do IO on multiple chunks at 353*2b15cb3dSCy Schubert once with a single system call. 354*2b15cb3dSCy Schubert 355*2b15cb3dSCy Schubert COMPATIBILITY NOTE: 356*2b15cb3dSCy Schubert The evbuffer struct is no longer exposed in a header. The code here is 357*2b15cb3dSCy Schubert too volatile to expose an official evbuffer structure, and there was never 358*2b15cb3dSCy Schubert any means provided to create an evbuffer except via evbuffer_new which 359*2b15cb3dSCy Schubert heap-allocated the buffer. 360*2b15cb3dSCy Schubert 361*2b15cb3dSCy Schubert If you need access to the whole buffer as a linear chunk of memory, the 362*2b15cb3dSCy Schubert EVBUFFER_DATA() function still works. Watch out, though: it needs to copy 363*2b15cb3dSCy Schubert the buffer's contents in a linear chunk before you can use it. 364*2b15cb3dSCy Schubert 365*2b15cb3dSCy Schubert4.2. More flexible readline support 366*2b15cb3dSCy Schubert 367*2b15cb3dSCy Schubert The old evbuffer_readline() function (which accepted any sequence of 368*2b15cb3dSCy Schubert CR and LF characters as a newline, and which couldn't handle lines 369*2b15cb3dSCy Schubert containing NUL characters), is now deprecated. The preferred 370*2b15cb3dSCy Schubert function is evbuffer_readln(), which supports a variety of 371*2b15cb3dSCy Schubert line-ending styles, and which can return the number of characters in 372*2b15cb3dSCy Schubert the line returned. 373*2b15cb3dSCy Schubert 374*2b15cb3dSCy Schubert You can also call evbuffer_search_eol() to find the end of a line 375*2b15cb3dSCy Schubert in an evbuffer without ever extracting the line. 376*2b15cb3dSCy Schubert 377*2b15cb3dSCy Schubert4.3. Support for file-based IO in evbuffers. 378*2b15cb3dSCy Schubert 379*2b15cb3dSCy Schubert You can now add chunks of a file into a evbuffer, and Libevent will have 380*2b15cb3dSCy Schubert your OS use mapped-memory functionality, sendfile, or splice to transfer 381*2b15cb3dSCy Schubert the data without ever copying it to userspace. On OSs where this is not 382*2b15cb3dSCy Schubert supported, Libevent just loads the data. 383*2b15cb3dSCy Schubert 384*2b15cb3dSCy Schubert There are probably some bugs remaining in this code. On some platforms 385*2b15cb3dSCy Schubert (like Windows), it just reads the relevant parts of the file into RAM. 386*2b15cb3dSCy Schubert 387*2b15cb3dSCy Schubert4.4. Support for zero-copy ("scatter/gather") writes in evbuffers. 388*2b15cb3dSCy Schubert 389*2b15cb3dSCy Schubert You can add a piece of memory to an evbuffer without copying it. 390*2b15cb3dSCy Schubert Instead, Libevent adds a new element to the evbuffer's linked list of 391*2b15cb3dSCy Schubert chunks with a pointer to the memory you supplied. You can do this 392*2b15cb3dSCy Schubert either with a reference-counted chunk (via evbuffer_add_reference), or 393*2b15cb3dSCy Schubert by asking Libevent for a pointer to its internal vectors (via 394*2b15cb3dSCy Schubert evbuffer_reserve_space or evbuffer_peek()). 395*2b15cb3dSCy Schubert 396*2b15cb3dSCy Schubert4.5. Multiple callbacks per evbuffer 397*2b15cb3dSCy Schubert 398*2b15cb3dSCy Schubert Previously, you could only have one callback active on an evbuffer at a 399*2b15cb3dSCy Schubert time. In practice, this meant that if one part of Libevent was using an 400*2b15cb3dSCy Schubert evbuffer callback to notice when an internal evbuffer was reading or 401*2b15cb3dSCy Schubert writing data, you couldn't have your own callback on that evbuffer. 402*2b15cb3dSCy Schubert 403*2b15cb3dSCy Schubert Now, you can now use the evbuffer_add_cb() function to add a callback that 404*2b15cb3dSCy Schubert does not interfere with any other callbacks. 405*2b15cb3dSCy Schubert 406*2b15cb3dSCy Schubert The evbuffer_setcb() function is now deprecated. 407*2b15cb3dSCy Schubert 408*2b15cb3dSCy Schubert4.6. New callback interface 409*2b15cb3dSCy Schubert 410*2b15cb3dSCy Schubert Previously, evbuffer callbacks were invoked with the old size of the 411*2b15cb3dSCy Schubert buffer and the new size of the buffer. This interface could not capture 412*2b15cb3dSCy Schubert operations that simultaneously filled _and_ drained a buffer, or handle 413*2b15cb3dSCy Schubert cases where we needed to postpone callbacks until multiple operations were 414*2b15cb3dSCy Schubert complete. 415*2b15cb3dSCy Schubert 416*2b15cb3dSCy Schubert Callbacks that are set with evbuffer_setcb still use the old API. 417*2b15cb3dSCy Schubert Callbacks added with evbuffer_add_cb() use a new interface that takes a 418*2b15cb3dSCy Schubert pointer to a struct holding the total number of bytes drained read and the 419*2b15cb3dSCy Schubert total number of bytes written. See event2/buffer.h for full details. 420*2b15cb3dSCy Schubert 421*2b15cb3dSCy Schubert4.7. Misc new evbuffer features 422*2b15cb3dSCy Schubert 423*2b15cb3dSCy Schubert You can use evbuffer_remove() to move a given number of bytes from one 424*2b15cb3dSCy Schubert buffer to another. 425*2b15cb3dSCy Schubert 426*2b15cb3dSCy Schubert The evbuffer_search() function lets you search for repeated instances of 427*2b15cb3dSCy Schubert a pattern inside an evbuffer. 428*2b15cb3dSCy Schubert 429*2b15cb3dSCy Schubert You can use evbuffer_freeze() to temporarily suspend drains from or adds 430*2b15cb3dSCy Schubert to a given evbuffer. This is useful for code that exposes an evbuffer as 431*2b15cb3dSCy Schubert part of its public API, but wants users to treat it as a pure source or 432*2b15cb3dSCy Schubert sink. 433*2b15cb3dSCy Schubert 434*2b15cb3dSCy Schubert There's an evbuffer_copyout() that looks at the data at the start of an 435*2b15cb3dSCy Schubert evbuffer without doing a drain. 436*2b15cb3dSCy Schubert 437*2b15cb3dSCy Schubert You can have an evbuffer defer all of its callbacks, so that rather than 438*2b15cb3dSCy Schubert being invoked immediately when the evbuffer's length changes, they are 439*2b15cb3dSCy Schubert invoked from within the event_loop. This is useful when you have a 440*2b15cb3dSCy Schubert complex set of callbacks that can change the length of other evbuffers, 441*2b15cb3dSCy Schubert and you want to avoid having them recurse and overflow your stack. 442*2b15cb3dSCy Schubert 443*2b15cb3dSCy Schubert5. Bufferevents improvements 444*2b15cb3dSCy Schubert 445*2b15cb3dSCy Schubert Libevent has long included a "bufferevents" structure and related 446*2b15cb3dSCy Schubert functions that were useful for generic buffered IO on a TCP connection. 447*2b15cb3dSCy Schubert This is what Libevent uses for its HTTP implementation. In addition to 448*2b15cb3dSCy Schubert the improvements that they get for free from the underlying evbuffer 449*2b15cb3dSCy Schubert implementation above, there are many new features in Libevent 2.0's 450*2b15cb3dSCy Schubert evbuffers. 451*2b15cb3dSCy Schubert 452*2b15cb3dSCy Schubert5.1. New OO implementations 453*2b15cb3dSCy Schubert 454*2b15cb3dSCy Schubert The "bufferevent" structure is now an abstract base type with multiple 455*2b15cb3dSCy Schubert implementations. This should not break existing code, which always 456*2b15cb3dSCy Schubert allocated bufferevents with bufferevent_new(). 457*2b15cb3dSCy Schubert 458*2b15cb3dSCy Schubert Current implementations of the bufferevent interface are described below. 459*2b15cb3dSCy Schubert 460*2b15cb3dSCy Schubert5.2. bufferevent_socket_new() replaces bufferevent_new() 461*2b15cb3dSCy Schubert 462*2b15cb3dSCy Schubert Since bufferevents that use a socket are not the only kind, 463*2b15cb3dSCy Schubert bufferevent_new() is now deprecated. Use bufferevent_socket_new() 464*2b15cb3dSCy Schubert instead. 465*2b15cb3dSCy Schubert 466*2b15cb3dSCy Schubert5.3. Filtered bufferevent IO 467*2b15cb3dSCy Schubert 468*2b15cb3dSCy Schubert You can use bufferevent_filter_new() to create a bufferevent that wraps 469*2b15cb3dSCy Schubert around another bufferevent and transforms data it is sending and 470*2b15cb3dSCy Schubert receiving. See test/regress_zlib.c for a toy example that uses zlib to 471*2b15cb3dSCy Schubert compress data before sending it over a bufferevent. 472*2b15cb3dSCy Schubert 473*2b15cb3dSCy Schubert5.3. Linked pairs of bufferevents 474*2b15cb3dSCy Schubert 475*2b15cb3dSCy Schubert You can use bufferevent_pair_new() to produce two linked 476*2b15cb3dSCy Schubert bufferevents. This is like using socketpair, but doesn't require 477*2b15cb3dSCy Schubert system-calls. 478*2b15cb3dSCy Schubert 479*2b15cb3dSCy Schubert5.4. SSL support for bufferevents with OpenSSL 480*2b15cb3dSCy Schubert 481*2b15cb3dSCy Schubert There is now a bufferevent type that supports SSL/TLS using the 482*2b15cb3dSCy Schubert OpenSSL library. The code for this is build in a separate 483*2b15cb3dSCy Schubert library, libevent_openssl, so that your programs don't need to 484*2b15cb3dSCy Schubert link against OpenSSL unless they actually want SSL support. 485*2b15cb3dSCy Schubert 486*2b15cb3dSCy Schubert There are two ways to construct one of these bufferevents, both 487*2b15cb3dSCy Schubert declared in <event2/bufferevent_ssl.h>. If you want to wrap an 488*2b15cb3dSCy Schubert SSL layer around an existing bufferevent, you would call the 489*2b15cb3dSCy Schubert bufferevent_openssl_filter_new() function. If you want to do SSL 490*2b15cb3dSCy Schubert on a socket directly, call bufferevent_openssl_socket_new(). 491*2b15cb3dSCy Schubert 492*2b15cb3dSCy Schubert5.5. IOCP support for bufferevents on Windows 493*2b15cb3dSCy Schubert 494*2b15cb3dSCy Schubert There is now a bufferevents backend that supports IOCP on Windows. 495*2b15cb3dSCy Schubert Supposedly, this will eventually make Windows IO much faster for 496*2b15cb3dSCy Schubert programs using bufferevents. We'll have to see; the code is not 497*2b15cb3dSCy Schubert currently optimized at all. To try it out, call the 498*2b15cb3dSCy Schubert event_base_start_iocp() method on an event_base before contructing 499*2b15cb3dSCy Schubert bufferevents. 500*2b15cb3dSCy Schubert 501*2b15cb3dSCy Schubert This is tricky code; there are probably some bugs hiding here. 502*2b15cb3dSCy Schubert 503*2b15cb3dSCy Schubert5.6. Improved connect support for bufferevents. 504*2b15cb3dSCy Schubert 505*2b15cb3dSCy Schubert You can now create a bufferevent that is not yet connected to any 506*2b15cb3dSCy Schubert host, and tell it to connect, either by address or by hostname. 507*2b15cb3dSCy Schubert 508*2b15cb3dSCy Schubert The functions to do this are bufferevent_socket_connect and 509*2b15cb3dSCy Schubert bufferevent_socket_connect_hostname. 510*2b15cb3dSCy Schubert 511*2b15cb3dSCy Schubert5.7. Rate-limiting for bufferevents 512*2b15cb3dSCy Schubert 513*2b15cb3dSCy Schubert If you need to limit the number of bytes read/written by a single 514*2b15cb3dSCy Schubert bufferevent, or by a group of them, you can do this with a new set of 515*2b15cb3dSCy Schubert bufferevent rate-limiting calls. 516*2b15cb3dSCy Schubert 517*2b15cb3dSCy Schubert6. Other improvements 518*2b15cb3dSCy Schubert 519*2b15cb3dSCy Schubert6.1. DNS improvements 520*2b15cb3dSCy Schubert 521*2b15cb3dSCy Schubert6.1.1. DNS: IPv6 nameservers 522*2b15cb3dSCy Schubert 523*2b15cb3dSCy Schubert The evdns code now lets you have nameservers whose addresses are IPv6. 524*2b15cb3dSCy Schubert 525*2b15cb3dSCy Schubert6.1.2. DNS: Better security 526*2b15cb3dSCy Schubert 527*2b15cb3dSCy Schubert Libevent 2.0 tries harder to resist DNS answer-sniping attacks than 528*2b15cb3dSCy Schubert earlier versions of evdns. See comments in the code for full details. 529*2b15cb3dSCy Schubert 530*2b15cb3dSCy Schubert Notably, evdns now supports the "0x20 hack" to make it harder to 531*2b15cb3dSCy Schubert impersonate a DNS server. Additionally, Libevent now uses a strong 532*2b15cb3dSCy Schubert internal RNG to generate DNS transaction IDs, so you don't need to supply 533*2b15cb3dSCy Schubert your own. 534*2b15cb3dSCy Schubert 535*2b15cb3dSCy Schubert6.1.3. DNS: Getaddrinfo support 536*2b15cb3dSCy Schubert 537*2b15cb3dSCy Schubert There's now an asynchronous getaddrinfo clone, evdns_getaddrinfo(), 538*2b15cb3dSCy Schubert to make the results of the evdns functions more usable. It doesn't 539*2b15cb3dSCy Schubert support every feature of a typical platform getaddrinfo() yet, but it 540*2b15cb3dSCy Schubert is quite close. 541*2b15cb3dSCy Schubert 542*2b15cb3dSCy Schubert There is also a blocking evutil_getaddrinfo() declared in 543*2b15cb3dSCy Schubert event2/util.h, to provide a getaddrinfo() implementation for 544*2b15cb3dSCy Schubert platforms that don't have one, and smooth over the differences in 545*2b15cb3dSCy Schubert various platforms implementations of RFC3493. 546*2b15cb3dSCy Schubert 547*2b15cb3dSCy Schubert Bufferevents provide bufferevent_connect_hostname(), which combines 548*2b15cb3dSCy Schubert the name lookup and connect operations. 549*2b15cb3dSCy Schubert 550*2b15cb3dSCy Schubert6.1.4. DNS: No more evdns globals 551*2b15cb3dSCy Schubert 552*2b15cb3dSCy Schubert Like an event base, evdns operations are now supposed to use an evdns_base 553*2b15cb3dSCy Schubert argument. This makes them easier to wrap for other (more OO) languages, 554*2b15cb3dSCy Schubert and easier to control the lifetime of. The old evdns functions will 555*2b15cb3dSCy Schubert still, of course, continue working. 556*2b15cb3dSCy Schubert 557*2b15cb3dSCy Schubert6.2. Listener support 558*2b15cb3dSCy Schubert 559*2b15cb3dSCy Schubert You can now more easily automate setting up a bound socket to listen for 560*2b15cb3dSCy Schubert TCP connections. Just use the evconnlistener_*() functions in the 561*2b15cb3dSCy Schubert event2/listener.h header. 562*2b15cb3dSCy Schubert 563*2b15cb3dSCy Schubert The listener code supports IOCP on Windows if available. 564*2b15cb3dSCy Schubert 565*2b15cb3dSCy Schubert6.3. Secure RNG support 566*2b15cb3dSCy Schubert 567*2b15cb3dSCy Schubert Network code very frequently needs a secure, hard-to-predict random number 568*2b15cb3dSCy Schubert generator. Some operating systems provide a good C implementation of one; 569*2b15cb3dSCy Schubert others do not. Libevent 2.0 now provides a consistent implementation 570*2b15cb3dSCy Schubert based on the arc4random code originally from OpenBSD. Libevent (and you) 571*2b15cb3dSCy Schubert can use the evutil_secure_rng_*() functions to access a fairly secure 572*2b15cb3dSCy Schubert random stream of bytes. 573*2b15cb3dSCy Schubert 574*2b15cb3dSCy Schubert6.4. HTTP 575*2b15cb3dSCy Schubert 576*2b15cb3dSCy Schubert The evhttp uriencoding and uridecoding APIs have updated versions 577*2b15cb3dSCy Schubert that behave more correctly, and can handle strings with internal NULs. 578*2b15cb3dSCy Schubert 579*2b15cb3dSCy Schubert The evhttp query parsing and URI parsing logic can now detect errors 580*2b15cb3dSCy Schubert more usefully. Moreover, we include an actual URI parsing function 581*2b15cb3dSCy Schubert (evhttp_uri_parse()) to correctly parse URIs, so as to discourage 582*2b15cb3dSCy Schubert people from rolling their own ad-hoc parsing functions. 583*2b15cb3dSCy Schubert 584*2b15cb3dSCy Schubert There are now accessor functions for the useful fields of struct http 585*2b15cb3dSCy Schubert and friends; it shouldn't be necessary to access them directly any 586*2b15cb3dSCy Schubert more. 587*2b15cb3dSCy Schubert 588*2b15cb3dSCy Schubert Libevent now lets you declare support for all specified HTTP methods, 589*2b15cb3dSCy Schubert including OPTIONS, PATCH, and so on. The default list is unchanged. 590*2b15cb3dSCy Schubert 591*2b15cb3dSCy Schubert Numerous evhttp bugs also got fixed. 592*2b15cb3dSCy Schubert 593*2b15cb3dSCy Schubert7. Infrastructure improvements 594*2b15cb3dSCy Schubert 595*2b15cb3dSCy Schubert7.1. Better unit test framework 596*2b15cb3dSCy Schubert 597*2b15cb3dSCy Schubert We now use a unit test framework that Nick wrote called "tinytest". 598*2b15cb3dSCy Schubert The main benefit from Libevent's point of view is that tests which 599*2b15cb3dSCy Schubert might mess with global state can all run each in their own 600*2b15cb3dSCy Schubert subprocess. This way, when there's a bug that makes one unit test 601*2b15cb3dSCy Schubert crash or mess up global state, it doesn't affect any others. 602*2b15cb3dSCy Schubert 603*2b15cb3dSCy Schubert7.2. Better unit tests 604*2b15cb3dSCy Schubert 605*2b15cb3dSCy Schubert Despite all the code we've added, our unit tests are much better than 606*2b15cb3dSCy Schubert before. Right now, iterating over the different backends on various 607*2b15cb3dSCy Schubert platforms, I'm getting between 78% and 81% test coverage, compared 608*2b15cb3dSCy Schubert with less than 45% test coverage in Libevent 1.4. 609*2b15cb3dSCy Schubert 610