1 2 3# hackney - HTTP client library in Erlang # 4 5Copyright (c) 2012-2017 Benoît Chesneau. 6 7__Version:__ 1.8.6 8 9# hackney 10 11**hackney** is an HTTP client library for Erlang. 12 13[![Build Status](https://travis-ci.org/benoitc/hackney.png?branch=master)](https://travis-ci.org/benoitc/hackney) 14[![Hex pm](http://img.shields.io/hexpm/v/hackney.svg?style=flat)](https://hex.pm/packages/hackney) 15 16## Main features: 17 18- no message passing (except for asynchronous responses): response is 19 directly streamed to the current process and state is kept in a `#client{}` record. 20- binary streams 21- SSL support 22- Keepalive handling 23- basic authentication 24- stream the response and the requests 25- fetch a response asynchronously 26- multipart support (streamed or not) 27- chunked encoding support 28- Can send files using the sendfile API 29- Optional socket pool 30- REST syntax: `hackney:Method(URL)` (where a method can be get, post, put, delete, ...) 31 32**Supported versions** of Erlang are R16B03-1, 17.3.4 and above. It is 33reported to work with R14B04 and R15B03-1. 34 35**WARNING**: Erlang 17.3 and 17.3.1 have a broken SSL module which 36prevents the usage of SSL connection with some servers. You **must** upgrade 37in that case to Erlang 38[17.3.4](https://github.com/erlang/otp/commit/9417f044ee3c291c2ea343c203aebdcc40597226) 39or superior. 40 41> Note: This is a work in progress, see the 42[TODO](http://github.com/benoitc/hackney/blob/master/TODO.md) for more 43information on what still needs to be done. 44 45#### Useful modules are: 46 47- [`hackney`](hackney.md): main module. It contains all HTTP client functions. 48- [`hackney_http`](hackney_http.md): HTTP parser in pure Erlang. This parser is able 49to parse HTTP responses and requests in a streaming fashion. If not set 50it will be autodetected if it's a request or a response that's needed. 51 52- [`hackney_headers`](hackney_headers.md) Module to manipulate HTTP headers. 53- [`hackney_cookie`](hackney_cookie.md): Module to manipulate cookies. 54- [`hackney_multipart`](hackney_multipart.md): Module to encode/decode multipart. 55- [`hackney_url`](hackney_url.md): Module to parse and create URIs. 56- [`hackney_date`](hackney_date.md): Module to parse HTTP dates. 57 58Read the [NEWS](https://raw.github.com/benoitc/hackney/master/NEWS.md) file 59to get the last changelog. 60 61## Installation 62 63Download the sources from our [Github 64repository](http://github.com/benoitc/hackney) 65 66To build the application simply run 'rebar3 compile'. 67 68To run tests run 'rebar3 eunit'. 69To generate doc, run 'rebar3 edoc'. 70 71Or add it to your rebar config 72 73```erlang 74 75{deps, [ 76 .... 77 {hackney, ".*", {git, "git://github.com/benoitc/hackney.git", {branch, "master"}}} 78]}. 79``` 80 81## Basic usage 82 83The basic usage of hackney is: 84 85### Start hackney 86 87hackney is an 88[OTP](http://www.erlang.org/doc/design_principles/users_guide.html) 89application. You have to start it first before using any of the functions. 90The hackney application will start the default socket pool for you. 91 92To start in the console run: 93 94```erlang-repl 95 96$ ./rebar3 shell 97 98``` 99 100It is suggested that you install rebar3 user-wide as described [here](http://blog.erlware.org/rebar3-features-part-1-local-install-and-upgrade/). 101This fixes zsh (and maybe other shells) escript-related bugs. Also this should speed things up. 102 103```erlang-repl 104 1051>> hackney:start(). 106ok 107``` 108 109It will start hackney and all of the application it depends on: 110 111```erlang 112 113application:start(crypto), 114application:start(public_key), 115application:start(ssl), 116application:start(hackney). 117``` 118 119Or add hackney to the applications property of your .app in a release 120 121### Simple request 122 123Do a simple request that will return a client state: 124 125```erlang 126 127Method = get, 128URL = <<"https://friendpaste.com">>, 129Headers = [], 130Payload = <<>>, 131Options = [], 132{ok, StatusCode, RespHeaders, ClientRef} = hackney:request(Method, URL, 133 Headers, Payload, 134 Options). 135``` 136 137The request method returns the tuple `{ok, StatusCode, Headers, ClientRef}` 138or `{error, Reason}`. A `ClientRef` is simply a reference to the current 139request that you can reuse. 140 141If you prefer the REST syntax, you can also do: 142 143```erlang 144hackney:Method(URL, Headers, Payload, Options) 145``` 146 147where `Method`, can be any HTTP method in lowercase. 148 149### Read the body 150 151```erlang 152{ok, Body} = hackney:body(Client). 153``` 154 155`hackney:body/1` fetch the body. To fetch it by chunk you can use the 156`hackney:stream_body/1` function: 157 158```erlang 159 160read_body(MaxLength, Ref, Acc) when MaxLength > byte_size(Acc) -> 161 case stream_body(Ref) of 162 {ok, Data} -> 163 read_body(MaxLength, Ref, << Acc/binary, Data/binary >>); 164 done -> 165 {ok, Acc}; 166 {error, Reason} -> 167 {error, Reason} 168 end. 169``` 170 171> Note: you can also fetch a multipart response using the functions 172> `hackney:stream_multipart/1` and `hackney:skip_multipart/1`. 173 174> Note 2: using the `with_body` option will return the body directy instead of a reference. 175 176### Reuse a connection 177 178By default all connections are created and closed dynamically by 179hackney but sometimes you may want to reuse the same reference for your 180connections. It's especially useful if you just want to handle serially a 181couple of requests. 182 183> A closed connection will automatically be reconnected. 184 185#### To create a connection: 186 187```erlang 188 189Transport = hackney_tcp, 190Host = << "https://friendpaste.com" >>, 191Port = 443, 192Options = [], 193{ok, ConnRef} = hackney:connect(Transport, Host, Port, Options) 194``` 195 196> To create a connection that will use an HTTP proxy use 197> `hackney_http_proxy:connect_proxy/5` instead. 198 199#### Make a request 200 201Once you created a connection use the `hackney:send_request/2` function 202to make a request: 203 204```erlang 205 206ReqBody = << "{ \"snippet\": \"some snippet\" }" >>, 207ReqHeaders = [{<<"Content-Type">>, <<"application/json">>}], 208NextPath = <<"/">>, 209NextMethod = post, 210NextReq = {NextMethod, NextPath, ReqHeaders, ReqBody} 211{ok, _, _, ConnRef} = hackney:send_request(ConnRef, NextReq). 212{ok, Body1} = hackney:body(ConnRef), 213``` 214 215Here we are posting a JSON payload to '/' on the friendpaste service to 216create a paste. Then we close the client connection. 217 218> If your connection supports keepalive the connection will be kept open until you close it exclusively. 219 220### Send a body 221 222hackney helps you send different payloads by passing different terms as 223the request body: 224 225- `{form, PropList}` : To send a form 226- `{multipart, Parts}` : to send your body using the multipart API. Parts 227 follow this format: 228 - `eof`: end the multipart request 229 - `{file, Path}`: to stream a file 230 - `{file, Path, ExtraHeaders}`: to stream a file 231 - `{Name, Content}`: to send a full part 232 - `{Name, Content, ExtraHeaders}`: to send a full part 233 - `{mp_mixed, Name, MixedBoundary}`: To notify we start a part with a 234 a mixed multipart content 235 - `{mp_mixed_eof, MixedBoundary}`: To notify we end a part with a a 236 mixed multipart content 237- `{file, File}` : To send a file 238- Bin: To send a binary or an iolist 239 240> Note: to send a chunked request, just add the `Transfer-Encoding: chunked` 241> header to your headers. Binary and Iolist bodies will be then sent using 242> the chunked encoding. 243 244#### Send the body by yourself 245 246While the default is to directly send the request and fetch the status 247and headers, if the body is set as the atom `stream` the request and 248send_request function will return {ok, Client}. Then you can use the 249function `hackney:send_body/2` to stream the request body and 250`hackney:start_response/1` to initialize the response. 251 252> Note: The function `hackney:start_response/1` will only accept 253> a Client that is waiting for a response (with a response state 254> equal to the atom `waiting`). 255 256Ex: 257 258```erlang 259 260ReqBody = << "{ 261 \"id\": \"some_paste_id2\", 262 \"rev\": \"some_revision_id\", 263 \"changeset\": \"changeset in unidiff format\" 264}" >>, 265ReqHeaders = [{<<"Content-Type">>, <<"application/json">>}], 266Path = <<"https://friendpaste.com/">>, 267Method = post, 268{ok, ClientRef} = hackney:request(Method, Path, ReqHeaders, stream, []), 269ok = hackney:send_body(ClientRef, ReqBody), 270{ok, _Status, _Headers, ClientRef} = hackney:start_response(ClientRef), 271{ok, Body} = hackney:body(ClientRef), 272``` 273 274> Note: to send a **multipart** body in a streaming fashion use the 275> `hackney:send_multipart_body/2` function. 276 277### Get a response asynchronously 278 279Since the 0.6 version, hackney is able to fetch the response 280asynchronously using the `async` option: 281 282```erlang 283 284Url = <<"https://friendpaste.com/_all_languages">>, 285Opts = [async], 286LoopFun = fun(Loop, Ref) -> 287 receive 288 {hackney_response, Ref, {status, StatusInt, Reason}} -> 289 io:format("got status: ~p with reason ~p~n", [StatusInt, 290 Reason]), 291 Loop(Loop, Ref); 292 {hackney_response, Ref, {headers, Headers}} -> 293 io:format("got headers: ~p~n", [Headers]), 294 Loop(Loop, Ref); 295 {hackney_response, Ref, done} -> 296 ok; 297 {hackney_response, Ref, Bin} -> 298 io:format("got chunk: ~p~n", [Bin]), 299 Loop(Loop, Ref); 300 301 Else -> 302 io:format("else ~p~n", [Else]), 303 ok 304 end 305 end. 306 307{ok, ClientRef} = hackney:get(Url, [], <<>>, Opts), 308LoopFun(LoopFun, ClientRef). 309``` 310 311> **Note 1**: When `{async, once}` is used the socket will receive only once. 312> To receive the other messages use the function `hackney:stream_next/1`. 313 314> **Note 2**: Asynchronous responses automatically checkout the socket at the end. 315 316> **Note 3**: At any time you can go back and receive your response 317> synchronously using the function `hackney:stop_async/1` See the 318> example [test_async_once2](https://github.com/benoitc/hackney/blob/master/examples/test_async_once2.erl) for the usage. 319 320> **Note 4**: When the option `{follow_redirect, true}` is passed to 321> the request, you will receive the folllowing messages on valid 322> redirection: 323> - `{redirect, To, Headers}` 324> - `{see_other, To, Headers}` for status 303 and POST requests. 325 326> **Note 5**: You can send the messages to another process by using the 327> option `{stream_to, Pid}` . 328 329### Use the default pool 330 331To reuse a connection globally in your application you can also use a 332socket pool. On startup, hackney launches a pool named default. To use it 333do the following: 334 335```erlang 336 337Method = get, 338URL = <<"https://friendpaste.com">>, 339Headers = [], 340Payload = <<>>, 341Options = [{pool, default}], 342{ok, StatusCode, RespHeaders, ClientRef} = hackney:request(Method, URL, Headers, 343 Payload, Options). 344``` 345 346By adding the tuple `{pool, default}` to the options, hackney will use 347the connections stored in that pool. 348 349You can also use different pools in your application which allows 350you to maintain a group of connections. 351 352```erlang 353 354PoolName = mypool, 355Options = [{timeout, 150000}, {max_connections, 100}], 356ok = hackney_pool:start_pool(PoolName, Options), 357``` 358 359`timeout` is the time we keep the connection alive in the pool, 360`max_connections` is the number of connections maintained in the pool. Each 361connection in a pool is monitored and closed connections are removed 362automatically. 363 364To close a pool do: 365 366```erlang 367hackney_pool:stop_pool(PoolName). 368``` 369 370> Note: Sometimes you want to disable the default pool in your app 371> without having to set the client option each time. You can now do this 372> by setting the hackney application environment key `use_default_pool` 373> to false. 374 375### Use a custom pool handler. 376 377Since the version 0.8 it is now possible to use your own Pool to 378maintain the connections in hackney. 379 380A pool handler is a module that handles the `hackney_pool_handler` 381behaviour. 382 383See for example the 384[hackney_disp](https://github.com/benoitc/hackney_disp) a load-balanced 385Pool dispatcher based on dispcount].> Note: for now you can`t force the pool handler / client. 386 387### Automatically follow a redirection 388 389If the option `{follow_redirect, true}` is given to the request, the 390client will be able to automatically follow the redirection and 391retrieve the body. The maximum number of connections can be set using the 392`{max_redirect, Max}` option. Default is 5. 393 394The client will follow redirects on 301, 302 & 307 if the method is 395get or head. If another method is used the tuple 396`{ok, maybe_redirect, Status, Headers, Client}` will be returned. It will 397only follow 303 redirects (see other) if the method is a POST. 398 399Last Location is stored in the `location` property of the client state. 400 401ex: 402 403```erlang 404 405Method = get, 406URL = "http://friendpaste.com/", 407ReqHeaders = [{<<"accept-encoding">>, <<"identity">>}], 408ReqBody = <<>>, 409Options = [{follow_redirect, true}, {max_redirect, 5}], 410{ok, S, H, Ref} = hackney:request(Method, URL, ReqHeaders, 411 ReqBody, Options), 412{ok, Body1} = hackney:body(Ref). 413``` 414 415### Proxy a connection 416 417#### HTTP Proxy 418 419To use an HTTP tunnel add the option `{proxy, ProxyUrl}` where 420`ProxyUrl` can be a simple url or an `{Host, Port}` tuple. If you need 421to authenticate set the option `{proxy_auth, {User, Password}}`. 422 423#### SOCKS5 proxy 424 425Hackney supports the connection via a socks5 proxy. To set a socks5 426proxy, use the following settings: 427 428- `{proxy, {socks5, ProxyHost, ProxyPort}}`: to set the host and port of 429 the proxy to connect. 430- `{socks5_user, Username}`: to set the user used to connect to the proxy 431- `{socks5_pass, Password}`: to set the password used to connect to the proxy 432 433SSL and TCP connections can be forwarded via a socks5 proxy. hackney is 434automatically upgrading to an SSL connection if needed. 435 436### Metrics 437 438Hackney offers the following metrics 439 440You can enable metrics collection by adding a `mod_metrics` entry to hackney's 441app config. Metrics are disabled by default. The module specified must have an 442API matching that of the hackney metrics module. 443 444To use [folsom](https://github.com/boundary/folsom), specify `{mod_metrics, 445folsom}`, or if you want to use 446[exometer](https://github.com/feuerlabs/exometer), specify`{mod_metrics, 447exometer}` and ensure that folsom or exometer is in your code path and has 448been started. 449 450#### Generic Hackney metrics 451 452|Name |Type | Description | 453|-------------------------|-------|------------------------------------| 454|hackney.nb_requests |counter| Number of running requests | 455|hackney.total_requests |counter| Total number of requests | 456|hackney.finished_requests|counter| Total number of requests finished | 457 458#### Metrics per Hosts 459 460|Name |Type | Description | 461|----------------------------|---------|----------------------------| 462|hackney.HOST.nb_requests |counter | Number of running requests | 463|hackney.HOST.request_time |histogram| Request time | 464|hackney.HOST.connect_time |histogram| Connect time | 465|hackney.HOST.response_time |histogram| Response time | 466|hackney.HOST.connect_timeout|counter | Number of connect timeout | 467|hackney.HOST.connect_error |counter | Number of timeout errors | 468 469#### Metrics per Pool 470 471|Name |Type | Description | 472|------------------------------|-----------|--------------------------------------------------------------------| 473|hackney.POOLNAME.take_rate |meter | meter recording rate at which a connection is retrieved from the pool| 474|hackney.POOLNAME.no_socket |counter | Count of new connections | 475|hackney.POOLNAME.in_use_count |histogram| How many connections from the pool are used | 476|hackney.POOLNAME.free_count |counter | Number of free sockets in the pool | 477|hackney.POOLNAME.queue_counter|histogram| queued clients | 478 479## Contribute 480 481For issues, comments or feedback please [create an 482issue](http://github.com/benoitc/hackney/issues). 483 484### Notes for developers 485 486If you want to contribute patches or improve the docs, you will need to 487build hackney using the `rebar_dev.config` file. It can also be built 488using the **Makefile**: 489 490```sh 491 492$ rebar3 update 493$ rebar3 compile 494 495``` 496 497For successfully running the hackney test suite locally it is necessary to 498install [httpbin](https://pypi.python.org/pypi/httpbin/0.2.0). 499 500An example installation using virtualenv:: 501 502```sh 503 504$ mkvirtualenv hackney 505$ pip install gunicorn httpbin 506 507``` 508 509Running the tests: 510 511``` 512$ gunicorn --daemon --pid httpbin.pid httpbin:app 513$ make test 514$ kill `cat httpbin.pid` 515``` 516 517 518## Modules ## 519 520 521<table width="100%" border="0" summary="list of modules"> 522<tr><td><a href="hackney.md" class="module">hackney</a></td></tr> 523<tr><td><a href="hackney_app.md" class="module">hackney_app</a></td></tr> 524<tr><td><a href="hackney_bstr.md" class="module">hackney_bstr</a></td></tr> 525<tr><td><a href="hackney_connect.md" class="module">hackney_connect</a></td></tr> 526<tr><td><a href="hackney_cookie.md" class="module">hackney_cookie</a></td></tr> 527<tr><td><a href="hackney_date.md" class="module">hackney_date</a></td></tr> 528<tr><td><a href="hackney_headers.md" class="module">hackney_headers</a></td></tr> 529<tr><td><a href="hackney_headers_new.md" class="module">hackney_headers_new</a></td></tr> 530<tr><td><a href="hackney_http.md" class="module">hackney_http</a></td></tr> 531<tr><td><a href="hackney_http_connect.md" class="module">hackney_http_connect</a></td></tr> 532<tr><td><a href="hackney_local_tcp.md" class="module">hackney_local_tcp</a></td></tr> 533<tr><td><a href="hackney_manager.md" class="module">hackney_manager</a></td></tr> 534<tr><td><a href="hackney_multipart.md" class="module">hackney_multipart</a></td></tr> 535<tr><td><a href="hackney_pool.md" class="module">hackney_pool</a></td></tr> 536<tr><td><a href="hackney_pool_handler.md" class="module">hackney_pool_handler</a></td></tr> 537<tr><td><a href="hackney_request.md" class="module">hackney_request</a></td></tr> 538<tr><td><a href="hackney_response.md" class="module">hackney_response</a></td></tr> 539<tr><td><a href="hackney_socks5.md" class="module">hackney_socks5</a></td></tr> 540<tr><td><a href="hackney_ssl.md" class="module">hackney_ssl</a></td></tr> 541<tr><td><a href="hackney_stream.md" class="module">hackney_stream</a></td></tr> 542<tr><td><a href="hackney_sup.md" class="module">hackney_sup</a></td></tr> 543<tr><td><a href="hackney_tcp.md" class="module">hackney_tcp</a></td></tr> 544<tr><td><a href="hackney_trace.md" class="module">hackney_trace</a></td></tr> 545<tr><td><a href="hackney_url.md" class="module">hackney_url</a></td></tr> 546<tr><td><a href="hackney_util.md" class="module">hackney_util</a></td></tr></table> 547 548