1# Funnel 2 3[![Build Status](https://travis-ci.org/AF83/funnel.png?branch=master)](https://travis-ci.org/AF83/funnel) 4 5Funnel is for building Streaming APIs build upon ElasticSearch’s 6[percolation](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html). 7 8Funnel supports ElasticSearch >= 1.1. 9 10Funnel allow to register users / devices, associates some queries to user. 11 12The common usecase is to store a query from a user and notify this user when a 13new document matching this query is available. 14 15Check out this [example](https://github.com/AF83/funnel_http) for a complete http api using Funnel dsl. 16 17## Installing things 18 19You can use Funnel in your projects with the following steps: 20 211. Adding Funnel to your `mix.exs` dependencies: 22 23 ```elixir 24 def deps do 25 [ 26 {:funnel, "~> 0.1"} 27 ] 28 end 29 ``` 30 312. List the `:funnel` as your application dependencies: 32 33 ```elixir 34 def application do 35 [applications: [:funnel]] 36 end 37 ``` 38 393. Add this sample configuration to `config/config.exs`. 40 41 ```elixir 42 use Mix.Config 43 44 config :funnel, 45 es_host: "http://localhost:9200", 46 percolator_pool_size: 100, 47 percolator_pool_max_overflow: 1000 48 ``` 49 50## Testing things 51 52``` shell 53mix test 54``` 55 56## Doing things 57 58### Transport 59 60Funnel has this notion of `Funnel.Transport`. Anything implementing the 61`Funnel.Transport` protocol can be a transport. 62 63For the [Plug](https://github.com/elixir-lang/plug), the protocol would looks 64like: 65 66```elixir 67defmodule EventStreamMessage do 68 @moduledoc "" 69 This module serialize a given id and body to a ServerSent Events message. 70 "" 71 def to_message(id, data) do 72 "id:#{id}\ndata: #{data}\n\n" 73 end 74end 75 76defimpl Funnel.Transport, for: Elixir.Plug.Conn do 77 import Plug.Conn 78 79 def write(conn, %{:id => id, :item => item}) do 80 chunk conn, EventStreamMessage.to_message(id, item) 81 end 82end 83``` 84 85### Register 86 87A user, or a device, can register to funnel by using the `/register` endpoint. 88This will return a token. This token must be used in all communications with the 89funnel's API. 90 91The first argument is a `transport`, the second one is optionnal, and allow to 92write onto the transport the message seen since the `last_id given`. 93 94```elixir 95{:ok, token} = Funnel.register(transport) 96{:ok, "1a9dc09879374878bd7aab27c7be6bc7"} 97 98{:ok, token} = Funnel.register(self, "422f779c759244d4aad45ac94c83b7da") 99{:ok, "80cfd5a3e1324db8b076defec2ddc1b2"} 100``` 101 102### Creating Indexes 103 104This example will create an empty index: 105 106```elixir 107{:ok, status_code, body} = Funnel.Index.create 108{:ok, 200, %{"body" => %{"acknowledged" => true}, "index_id" => "e4680d88db914ed4854acb8a1a8f317d"}} 109``` 110 111This example will create an index with specific settings: 112 113``` elixir 114settings = '{"settings" : {"number_of_shards" : 1},"mappings" : {"type1" :{"_source" : { "enabled" : false },"properties" : {"field1" : { "type" :"string", "index" : "not_analyzed" }}}}}' |> IO.iodata_to_binary 115{:ok, _status_code, body} = Funnel.Index.create(settings) 116{:ok, 200, %{"body" => %{"acknowledged" => true},"index_id" => "3994bf6c03df412e8b1b05d4aca7a83c"}} 117``` 118 119 120### Deleting an index 121 122``` elixir 123{:ok, status_code, body} = Funnel.Index.destroy("3994bf6c03df412e8b1b05d4aca7a83c") 124{:ok, 200, %{"acknowledged" => true}} 125``` 126 127### Creating a query 128 129``` elixir 130query = '{"query" : {"match" : {"message" : "funnel"}}}' |> IO.iodata_to_binary 131{:ok, status_code, body} = Funnel.Query.create(index_id, token, query) 132{:ok, 201, %{"body" => %{"_id" => "287eae87d5774f2a9d02f5a5dd66856d-0398f4e1a6a34ea4b7ede0c1b7f40f38", "_index" => "3994bf6c03df412e8b1b05d4aca7a83c_dev", "_type" => ".percolator", "_version" => 1, "created" => true}, "index_id" => "3994bf6c03df412e8b1b05d4aca7a83c", "query_id" => "0398f4e1a6a34ea4b7ede0c1b7f40f38"}} 133``` 134 135### Updating a query 136 137``` elixir 138{:ok, status_code, body} = Funnel.Query.update(index_id, token, query_id, query) 139{:ok, 200, %{"body" => %{"_id" => "287eae87d5774f2a9d02f5a5dd66856d-0398f4e1a6a34ea4b7ede0c1b7f40f38", "_index" => "3994bf6c03df412e8b1b05d4aca7a83c_dev", "_type" => ".percolator", "_version" => 2, "created" => false}, "index_id" => "3994bf6c03df412e8b1b05d4aca7a83c", "query_id" => "0398f4e1a6a34ea4b7ede0c1b7f40f38"}} 140``` 141 142### Deleting a query 143 144``` elixir 145{:ok, status_code, body} = Funnel.Query.destroy(index_id, token, query_id) 146{:ok, 200, %{"_id" => "287eae87d5774f2a9d02f5a5dd66856d-0398f4e1a6a34ea4b7ede0c1b7f40f38", "_index" => "3994bf6c03df412e8b1b05d4aca7a83c_dev", "_type" => ".percolator", "_version" => 3, "found" => true}} 147``` 148 149### Finding queries 150 151``` elixir 152{:ok, status_code, body} = Funnel.Query.find(token) 153{:ok, 200, [%{"query_id" => "4f122313862e494b8810f073c27cf43d", "index_id" => "b79d2e9ff8c949e08ba98c4d8c216547", "score" => 1.0}]} 154 155{:ok, status_code, body} = Funnel.Query.find(token, %{index_id: index_id}) 156{:ok, 200, [%{"query_id" => "4f122313862e494b8810f073c27cf43d", "index_id" => "b79d2e9ff8c949e08ba98c4d8c216547", "score" => 1.0}]} 157``` 158 159### Submitting a document to the percolator 160 161``` elixir 162message = '{"message" : "this new elasticsearch percolator feature is nice, borat style"}' |> IO.iodata_to_binary 163Funnel.percolate(index_id, message) 164{:ok} 165``` 166