README.md
1# rust-rdkafka
2
3[](https://crates.io/crates/rdkafka)
4[](https://docs.rs/rdkafka/)
5[](https://travis-ci.org/fede1024/rust-rdkafka)
6[](https://codecov.io/gh/fede1024/rust-rdkafka/)
7[](https://gitter.im/rust-rdkafka/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
8
9A fully asynchronous, [futures]-enabled [Apache Kafka] client
10library for Rust based on [librdkafka].
11
12## The library
13
14`rust-rdkafka` provides a safe Rust interface to librdkafka. The master
15branch is currently based on librdkafka 1.4.2.
16
17### Documentation
18
19- [Latest release](https://docs.rs/rdkafka/)
20- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md)
21
22### Features
23
24The main features provided at the moment are:
25
26- Support for all Kafka versions since 0.8.x. For more information about
27 broker compatibility options, check the [librdkafka
28 documentation][broker-compat].
29- Consume from single or multiple topics.
30- Automatic consumer rebalancing.
31- Customizable rebalance, with pre and post rebalance callbacks.
32- Synchronous or asynchronous message production.
33- Customizable offset commit.
34- Create and delete topics and add and edit partitions.
35- Alter broker and topic configurations.
36- Access to cluster metadata (list of topic-partitions, replicas, active
37 brokers etc).
38- Access to group metadata (list groups, list members of groups, hostnames,
39 etc.).
40- Access to producer and consumer metrics, errors and callbacks.
41
42### One million messages per second
43
44`rust-rdkafka` is designed to be easy and safe to use thanks to the
45abstraction layer written in Rust, while at the same time being extremely
46fast thanks to the librdkafka C library.
47
48Here are some benchmark results using the [`BaseProducer`],
49sending data to a single Kafka 0.11 process running in localhost (default
50configuration, 3 partitions). Hardware: Dell laptop, with Intel Core
51i7-4712HQ @ 2.30GHz.
52
53- Scenario: produce 5 million messages, 10 bytes each, wait for all of them to be acked
54 - 1045413 messages/s, 9.970 MB/s (average over 5 runs)
55
56- Scenario: produce 100000 messages, 10 KB each, wait for all of them to be acked
57 - 24623 messages/s, 234.826 MB/s (average over 5 runs)
58
59For more numbers, check out the [kafka-benchmark] project.
60
61### Client types
62
63`rust-rdkafka` provides low level and high level consumers and producers.
64
65Low level:
66
67* [`BaseConsumer`]: a simple wrapper around the librdkafka consumer. It
68 must be periodically `poll()`ed in order to execute callbacks, rebalances
69 and to receive messages.
70* [`BaseProducer`]: a simple wrapper around the librdkafka producer. As in
71 the consumer case, the user must call `poll()` periodically to execute
72 delivery callbacks.
73* [`ThreadedProducer`]: a `BaseProducer` with a separate thread dedicated to
74 polling the producer.
75
76High level:
77
78 * [`StreamConsumer`]: a [`Stream`] of messages that takes care of
79 polling the consumer automatically.
80 * [`FutureProducer`]: a [`Future`] that will be completed once
81 the message is delivered to Kafka (or failed).
82
83For more information about consumers and producers, refer to their
84module-level documentation.
85
86*Warning*: the library is under active development and the APIs are likely
87to change.
88
89### Asynchronous data processing with Tokio
90
91[Tokio] is a platform for fast processing of asynchronous events in Rust.
92The interfaces exposed by the [`StreamConsumer`] and the [`FutureProducer`]
93allow rust-rdkafka users to easily integrate Kafka consumers and producers
94within the Tokio platform, and write asynchronous message processing code.
95Note that rust-rdkafka can be used without Tokio.
96
97To see rust-rdkafka in action with Tokio, check out the
98[asynchronous processing example] in the examples folder.
99
100### At-least-once delivery
101
102At-least-once delivery semantics are common in many streaming applications:
103every message is guaranteed to be processed at least once; in case of
104temporary failure, the message can be re-processed and/or re-delivered,
105but no message will be lost.
106
107In order to implement at-least-once delivery the stream processing
108application has to carefully commit the offset only once the message has
109been processed. Committing the offset too early, instead, might cause
110message loss, since upon recovery the consumer will start from the next
111message, skipping the one where the failure occurred.
112
113To see how to implement at-least-once delivery with `rdkafka`, check out the
114[at-least-once delivery example] in the examples folder. To know more about
115delivery semantics, check the [message delivery semantics] chapter in the
116Kafka documentation.
117
118### Users
119
120Here are some of the projects using rust-rdkafka:
121
122- [timely-dataflow]: a distributed data-parallel compute engine. See also
123 the [blog post][timely-blog] announcing its Kafka integration.
124- [kafka-view]: a web interface for Kafka clusters.
125- [kafka-benchmark]: a high performance benchmarking tool for Kafka.
126
127*If you are using rust-rdkafka, please let us know!*
128
129## Installation
130
131Add this to your `Cargo.toml`:
132
133```toml
134[dependencies]
135rdkafka = { version = "0.24", features = ["cmake-build"] }
136```
137
138This crate will compile librdkafka from sources and link it statically to
139your executable. To compile librdkafka you'll need:
140
141* the GNU toolchain
142* GNU `make`
143* `pthreads`
144* `zlib`: optional, but included by default (feature: `libz`)
145* `cmake`: optional, *not* included by default (feature: `cmake-build`)
146* `libssl-dev`: optional, *not* included by default (feature: `ssl`)
147* `libsasl2-dev`: optional, *not* included by default (feature: `gssapi`)
148* `libzstd-dev`: optional, *not* included by default (feature: `zstd-pkg-config`)
149
150Note that using the CMake build system, via the `cmake-build` feature, is
151encouraged if you can take the dependency on CMake.
152
153By default a submodule with the librdkafka sources pinned to a specific
154commit will be used to compile and statically link the library. The
155`dynamic-linking` feature can be used to instead dynamically link rdkafka to
156the system's version of librdkafka. Example:
157
158```toml
159[dependencies]
160rdkafka = { version = "0.24", features = ["dynamic-linking"] }
161```
162
163For a full listing of features, consult the [rdkafka-sys crate's
164documentation][rdkafka-sys-features]. All of rdkafka-sys features are
165re-exported as rdkafka features.
166
167### Asynchronous runtimes
168
169Some features of the [`StreamConsumer`] and [`FutureProducer`] depend on
170Tokio, which can be a heavyweight dependency for users who only intend to
171use the low-level consumers and producers. The Tokio integration is
172enabled by default, but can be disabled by turning off default features:
173
174```toml
175[dependencies]
176rdkafka = { version = "0.24", default-features = false }
177```
178
179If you would like to use an asynchronous runtime besides Tokio, you can
180integrate it with rust-rdkafka by providing a shim that implements the
181[`AsyncRuntime`] trait. See the [smol runtime example] for an example
182integration with [smol].
183
184## Examples
185
186You can find examples in the [`examples`] folder. To run them:
187
188```bash
189cargo run --example <example_name> -- <example_args>
190```
191
192## Debugging
193
194rust-rdkafka uses the [`log`] and [`env_logger`] crates to handle logging.
195Logging can be enabled using the `RUST_LOG` environment variable, for
196example:
197
198```bash
199RUST_LOG="librdkafka=trace,rdkafka::client=debug" cargo test
200```
201
202This will configure the logging level of librdkafka to trace, and the level
203of the client module of the Rust client to debug. To actually receive logs
204from librdkafka, you also have to set the `debug` option in the producer or
205consumer configuration (see librdkafka
206[configuration][librdkafka-config]).
207
208To enable debugging in your project, make sure you initialize the logger
209with `env_logger::init()`, or the equivalent for any `log`-compatible
210logging framework.
211
212[`AsyncRuntime`]: https://docs.rs/rdkafka/*/rdkafka/util/struct.AsyncRuntime.html
213[`BaseConsumer`]: https://docs.rs/rdkafka/*/rdkafka/consumer/base_consumer/struct.BaseConsumer.html
214[`BaseProducer`]: https://docs.rs/rdkafka/*/rdkafka/producer/base_producer/struct.BaseProducer.html
215[`Future`]: https://doc.rust-lang.org/stable/std/future/trait.Future.html
216[`FutureProducer`]: https://docs.rs/rdkafka/*/rdkafka/producer/future_producer/struct.FutureProducer.html
217[`Stream`]: https://docs.rs/futures/*/futures/stream/trait.Stream.html
218[`StreamConsumer`]: https://docs.rs/rdkafka/*/rdkafka/consumer/stream_consumer/struct.StreamConsumer.html
219[`ThreadedProducer`]: https://docs.rs/rdkafka/*/rdkafka/producer/base_producer/struct.ThreadedProducer.html
220[`log`]: https://docs.rs/log
221[`env_logger`]: https://docs.rs/env_logger
222[Apache Kafka]: https://kafka.apache.org
223[asynchronous processing example]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/asynchronous_processing.rs
224[at-least-once delivery example]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/at_least_once.rs
225[smol runtime example]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/smol_runtime.rs
226[broker-compat]: https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#broker-version-compatibility
227[`examples`]: https://github.com/fede1024/rust-rdkafka/blob/master/examples/
228[futures]: https://github.com/rust-lang/futures-rs
229[kafka-benchmark]: https://github.com/fede1024/kafka-benchmark
230[kafka-benchmark]: https://github.com/fede1024/kafka-benchmark
231[kafka-view]: https://github.com/fede1024/kafka-view
232[librdkafka]: https://github.com/edenhill/librdkafka
233[librdkafka-config]: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
234[message delivery semantics]: https://kafka.apache.org/0101/documentation.html#semantics
235[rdkafka-sys-features]: https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys/README.md#features
236[rdkafka-sys-known-issues]: https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys/README.md#known-issues
237[smol]: https://docs.rs/smol
238[timely-blog]: https://github.com/frankmcsherry/blog/blob/master/posts/2017-11-08.md
239[timely-dataflow]: https://github.com/frankmcsherry/timely-dataflow
240[Tokio]: https://tokio.rs/
241
242## rdkafka-sys
243
244See [rdkafka-sys](https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys).
245
246## Contributors
247
248Thanks to:
249* Thijs Cadier - [thijsc](https://github.com/thijsc)
250
251## Alternatives
252
253* [kafka-rust]: a pure Rust implementation of the Kafka client.
254
255[kafka-rust]: https://github.com/spicavigo/kafka-rust
256
){kind=link}
){kind=link}
){kind=link}
){kind=link}
){kind=link}