1--- 2title: Production users 3--- 4 5This document tracks people and use cases for etcd in production. By creating a list of production use cases we hope to build a community of advisors that we can reach out to with experience using various etcd applications, operation environments, and cluster sizes. The etcd development team may reach out periodically to check-in on how etcd is working in the field and update this list. 6 7## All Kubernetes Users 8 9- *Application*: https://kubernetes.io/ 10- *Environments*: AWS, OpenStack, Azure, Google Cloud, Huawei Cloud, Bare Metal, etc 11 12**This is a meta user; please feel free to document specific Kubernetes clusters!** 13 14All Kubernetes clusters use etcd as their primary data store. This means etcd's users include such companies as [Niantic, Inc Pokemon Go](https://cloudplatform.googleblog.com/2016/09/bringing-Pokemon-GO-to-life-on-Google-Cloud.html), [Box](https://blog.box.com/blog/kubernetes-box-microservices-maximum-velocity/), [CoreOS](https://coreos.com/tectonic), [Ticketmaster](https://www.youtube.com/watch?v=wqXVKneP0Hg), [Salesforce](https://www.salesforce.com) and many many more. 15 16## discovery.etcd.io 17 18- *Application*: https://github.com/coreos/discovery.etcd.io 19- *Launched*: Feb. 2014 20- *Cluster Size*: 5 members, 5 discovery proxies 21- *Order of Data Size*: 100s of Megabytes 22- *Operator*: CoreOS, brandon.philips@coreos.com 23- *Environment*: AWS 24- *Backups*: Periodic async to S3 25 26discovery.etcd.io is the longest continuously running etcd backed service that we know about. It is the basis of automatic cluster bootstrap and was launched in Feb. 2014: https://coreos.com/blog/etcd-0.3.0-released/. 27 28## OpenTable 29 30- *Application*: OpenTable internal service discovery and cluster configuration management 31- *Launched*: May 2014 32- *Cluster Size*: 3 members each in 6 independent clusters; approximately 50 nodes reading / writing 33- *Order of Data Size*: 10s of MB 34- *Operator*: OpenTable, Inc; sschlansker@opentable.com 35- *Environment*: AWS, VMWare 36- *Backups*: None, all data can be re-created if necessary. 37 38## cycoresys.com 39 40- *Application*: multiple 41- *Launched*: Jul. 2014 42- *Cluster Size*: 3 members, _n_ proxies 43- *Order of Data Size*: 100s of kilobytes 44- *Operator*: CyCore Systems, Inc, sys@cycoresys.com 45- *Environment*: Baremetal 46- *Backups*: Periodic sync to Ceph RadosGW and DigitalOcean VM 47 48CyCore Systems provides architecture and engineering for computing systems. This cluster provides microservices, virtual machines, databases, storage clusters to a number of clients. It is built on CoreOS machines, with each machine in the cluster running etcd as a peer or proxy. 49 50## Radius Intelligence 51 52- *Application*: multiple internal tools, Kubernetes clusters, bootstrappable system configs 53- *Launched*: June 2015 54- *Cluster Size*: 2 clusters of 5 and 3 members; approximately a dozen nodes read/write 55- *Order of Data Size*: 100s of kilobytes 56- *Operator*: Radius Intelligence; jcderr@radius.com 57- *Environment*: AWS, CoreOS, Kubernetes 58- *Backups*: None, all data can be recreated if necessary. 59 60Radius Intelligence uses Kubernetes running CoreOS to containerize and scale internal toolsets. Examples include running [JetBrains TeamCity][teamcity] and internal AWS security and cost reporting tools. etcd clusters back these clusters as well as provide some basic environment bootstrapping configuration keys. 61 62## Vonage 63 64- *Application*: kubernetes, vault backend, system configuration for microservices, scheduling, locks (future - service discovery) 65- *Launched*: August 2015 66- *Cluster Size*: 2 clusters of 5 members in 2 DCs, n local proxies 1-to-1 with microservice, (ssl and SRV look up) 67- *Order of Data Size*: kilobytes 68- *Operator*: Vonage [devAdmin][raoofm] 69- *Environment*: VMWare, AWS 70- *Backups*: Daily snapshots on VMs. Backups done for upgrades. 71 72## PD 73 74- *Application*: embed etcd 75- *Launched*: Mar 2016 76- *Cluster Size*: 3 or 5 members 77- *Order of Data Size*: megabytes 78- *Operator*: PingCAP, Inc. 79- *Environment*: Bare Metal, AWS, etc. 80- *Backups*: None. 81 82PD(Placement Driver) is the central controller in the TiDB cluster. It saves the cluster meta information, schedule the data, allocate the global unique timestamp for the distributed transaction, etc. It embeds etcd to supply high availability and auto failover. 83 84## Huawei 85 86- *Application*: System configuration for overlay network (Canal) 87- *Launched*: June 2016 88- *Cluster Size*: 3 members for each cluster 89- *Order of Data Size*: kilobytes 90- *Operator*: Huawei Euler Department 91- *Environment*: [Huawei Cloud](http://www.hwclouds.com/product/cce.html) 92- *Backups*: None, all data can be recreated if necessary. 93 94[teamcity]: https://www.jetbrains.com/teamcity/ 95[raoofm]:https://github.com/raoofm 96 97## Qiniu Cloud 98 99- *Application*: system configuration for microservices, distributed locks 100- *Launched*: Jan. 2016 101- *Cluster Size*: 3 members each with several clusters 102- *Order of Data Size*: kilobytes 103- *Operator*: Pandora, chenchao@qiniu.com 104- *Environment*: Baremetal 105- *Backups*: None, all data can be recreated if necessary 106 107## QingCloud 108 109- *Application*: [QingCloud][qingcloud] appcenter cluster for service discovery as [metad][metad] backend. 110- *Launched*: December 2016 111- *Cluster Size*: 1 cluster of 3 members per user. 112- *Order of Data Size*: kilobytes 113- *Operator*: [yunify][yunify] 114- *Environment*: QingCloud IaaS 115- *Backups*: None, all data can be recreated if necessary. 116 117[metad]:https://github.com/yunify/metad 118[yunify]:https://github.com/yunify 119[qingcloud]:https://qingcloud.com/ 120 121 122## Yandex 123 124- *Application*: system configuration for services, service discovery 125- *Launched*: March 2016 126- *Cluster Size*: 3 clusters of 5 members 127- *Order of Data Size*: several gigabytes 128- *Operator*: Yandex; [nekto0n][nekto0n] 129- *Environment*: Bare Metal 130- *Backups*: None 131 132[nekto0n]:https://github.com/nekto0n 133 134## Tencent Games 135 136- *Application*: Meta data and configuration data for service discovery, Kubernetes, etc. 137- *Launched*: Jan. 2015 138- *Cluster Size*: 3 members each with 10s of clusters 139- *Order of Data Size*: 10s of Megabytes 140- *Operator*: Tencent Game Operations Department 141- *Environment*: Baremetal 142- *Backups*: Periodic sync to backup server 143 144In Tencent games, we use Docker and Kubernetes to deploy and run our applications, and use etcd to save meta data for service discovery, Kubernetes, etc. 145 146## Hyper.sh 147 148- *Application*: Kubernetes, distributed locks, etc. 149- *Launched*: April 2016 150- *Cluster Size*: 1 cluster of 3 members 151- *Order of Data Size*: 10s of MB 152- *Operator*: Hyper.sh 153- *Environment*: Baremetal 154- *Backups*: None, all data can be recreated if necessary. 155 156In [hyper.sh][hyper.sh], the container service is backed by [hypernetes][hypernetes], a multi-tenant kubernetes distro. Moreover, we use etcd to coordinate the multiple manage services and store global meta data. 157 158[hypernetes]:https://github.com/hyperhq/hypernetes 159[Hyper.sh]:https://www.hyper.sh 160 161## Meitu 162- *Application*: system configuration for services, service discovery, kubernetes in test environment 163- *Launched*: October 2015 164- *Cluster Size*: 1 cluster of 3 members 165- *Order of Data Size*: megabytes 166- *Operator*: Meitu, hxj@meitu.com, [shafreeck][shafreeck] 167- *Environment*: Bare Metal 168- *Backups*: None, all data can be recreated if necessary. 169 170[shafreeck]:https://github.com/shafreeck 171 172## Grab 173- *Application*: system configuration for services, service discovery 174- *Launched*: June 2016 175- *Cluster Size*: 1 cluster of 7 members 176- *Order of Data Size*: megabytes 177- *Operator*: Grab, [taxitan][taxitan], [reterVision][reterVision] 178- *Environment*: AWS 179- *Backups*: None, all data can be recreated if necessary. 180 181[taxitan]:https://github.com/taxitan 182[reterVision]:https://github.com/reterVision 183 184## DaoCloud.io 185 186- *Application*: container management 187- *Launched*: Sep. 2015 188- *Cluster Size*: 1000+ deployments, each deployment contains a 3 node cluster. 189- *Order of Data Size*: 100s of Megabytes 190- *Operator*: daocloud.io 191- *Environment*: Baremetal and virtual machines 192- *Backups*: None, all data can be recreated if necessary. 193 194In [DaoCloud][DaoCloud], we use Docker and Swarm to deploy and run our applications, and we use etcd to save metadata for service discovery. 195 196[DaoCloud]:https://www.daocloud.io 197 198## Branch.io 199 200- *Application*: Kubernetes 201- *Launched*: April 2016 202- *Cluster Size*: Multiple clusters, multiple sizes 203- *Order of Data Size*: 100s of Megabytes 204- *Operator*: branch.io 205- *Environment*: AWS, Kubernetes 206- *Backups*: EBS volume backups 207 208At [Branch][branch], we use kubernetes heavily as our core microservice platform for staging and production. 209 210[branch]: https://branch.io 211 212## Baidu Waimai 213 214- *Application*: SkyDNS, Kubernetes, UDC, CMDB and other distributed systems 215- *Launched*: April. 2016 216- *Cluster Size*: 3 clusters of 5 members 217- *Order of Data Size*: several gigabytes 218- *Operator*: Baidu Waimai Operations Department 219- *Environment*: CentOS 6.5 220- *Backups*: backup scripts 221 222## Salesforce.com 223 224- *Application*: Kubernetes 225- *Launched*: Jan 2017 226- *Cluster Size*: Multiple clusters of 3 members 227- *Order of Data Size*: 100s of Megabytes 228- *Operator*: Salesforce.com (krmayankk@github) 229- *Environment*: BareMetal 230- *Backups*: None, all data can be recreated 231 232## Hosted Graphite 233 234- *Application*: Service discovery, locking, ephemeral application data 235- *Launched*: January 2017 236- *Cluster Size*: 2 clusters of 7 members 237- *Order of Data Size*: Megabytes 238- *Operator*: Hosted Graphite (sre@hostedgraphite.com) 239- *Environment*: Bare Metal 240- *Backups*: None, all data is considered ephemeral. 241 242## Transwarp 243 244- *Application*: Transwarp Data Cloud, Transwarp Operating System, Transwarp Data Hub, Sophon 245- *Launched*: January 2016 246- *Cluster Size*: Multiple clusters, multiple sizes 247- *Order of Data Size*: Megabytes 248- *Operator*: Trasnwarp Operating System 249- *Environment*: Bare Metal, Container 250- *Backups*: backup scripts 251