1Bolt [![Build Status](https://drone.io/github.com/boltdb/bolt/status.png)](https://drone.io/github.com/boltdb/bolt/latest) [![Coverage Status](https://coveralls.io/repos/boltdb/bolt/badge.png?branch=master)](https://coveralls.io/r/boltdb/bolt?branch=master) [![GoDoc](https://godoc.org/github.com/boltdb/bolt?status.png)](https://godoc.org/github.com/boltdb/bolt) ![Version](http://img.shields.io/badge/version-1.0-green.png) 2==== 3 4Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas] and 5the [LMDB project][lmdb]. The goal of the project is to provide a simple, 6fast, and reliable database for projects that don't require a full database 7server such as Postgres or MySQL. 8 9Since Bolt is meant to be used as such a low-level piece of functionality, 10simplicity is key. The API will be small and only focus on getting values 11and setting values. That's it. 12 13[hyc_symas]: https://twitter.com/hyc_symas 14[lmdb]: http://symas.com/mdb/ 15 16 17## Project Status 18 19Bolt is stable and the API is fixed. Full unit test coverage and randomized 20black box testing are used to ensure database consistency and thread safety. 21Bolt is currently in high-load production environments serving databases as 22large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed 23services every day. 24 25 26## Getting Started 27 28### Installing 29 30To start using Bolt, install Go and run `go get`: 31 32```sh 33$ go get github.com/boltdb/bolt/... 34``` 35 36This will retrieve the library and install the `bolt` command line utility into 37your `$GOBIN` path. 38 39 40### Opening a database 41 42The top-level object in Bolt is a `DB`. It is represented as a single file on 43your disk and represents a consistent snapshot of your data. 44 45To open your database, simply use the `bolt.Open()` function: 46 47```go 48package main 49 50import ( 51 "log" 52 53 "github.com/boltdb/bolt" 54) 55 56func main() { 57 // Open the my.db data file in your current directory. 58 // It will be created if it doesn't exist. 59 db, err := bolt.Open("my.db", 0600, nil) 60 if err != nil { 61 log.Fatal(err) 62 } 63 defer db.Close() 64 65 ... 66} 67``` 68 69Please note that Bolt obtains a file lock on the data file so multiple processes 70cannot open the same database at the same time. Opening an already open Bolt 71database will cause it to hang until the other process closes it. To prevent 72an indefinite wait you can pass a timeout option to the `Open()` function: 73 74```go 75db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second}) 76``` 77 78 79### Transactions 80 81Bolt allows only one read-write transaction at a time but allows as many 82read-only transactions as you want at a time. Each transaction has a consistent 83view of the data as it existed when the transaction started. 84 85Individual transactions and all objects created from them (e.g. buckets, keys) 86are not thread safe. To work with data in multiple goroutines you must start 87a transaction for each one or use locking to ensure only one goroutine accesses 88a transaction at a time. Creating transaction from the `DB` is thread safe. 89 90Read-only transactions and read-write transactions should not depend on one 91another and generally shouldn't be opened simultaneously in the same goroutine. 92This can cause a deadlock as the read-write transaction needs to periodically 93re-map the data file but it cannot do so while a read-only transaction is open. 94 95 96#### Read-write transactions 97 98To start a read-write transaction, you can use the `DB.Update()` function: 99 100```go 101err := db.Update(func(tx *bolt.Tx) error { 102 ... 103 return nil 104}) 105``` 106 107Inside the closure, you have a consistent view of the database. You commit the 108transaction by returning `nil` at the end. You can also rollback the transaction 109at any point by returning an error. All database operations are allowed inside 110a read-write transaction. 111 112Always check the return error as it will report any disk failures that can cause 113your transaction to not complete. If you return an error within your closure 114it will be passed through. 115 116 117#### Read-only transactions 118 119To start a read-only transaction, you can use the `DB.View()` function: 120 121```go 122err := db.View(func(tx *bolt.Tx) error { 123 ... 124 return nil 125}) 126``` 127 128You also get a consistent view of the database within this closure, however, 129no mutating operations are allowed within a read-only transaction. You can only 130retrieve buckets, retrieve values, and copy the database within a read-only 131transaction. 132 133 134#### Batch read-write transactions 135 136Each `DB.Update()` waits for disk to commit the writes. This overhead 137can be minimized by combining multiple updates with the `DB.Batch()` 138function: 139 140```go 141err := db.Batch(func(tx *bolt.Tx) error { 142 ... 143 return nil 144}) 145``` 146 147Concurrent Batch calls are opportunistically combined into larger 148transactions. Batch is only useful when there are multiple goroutines 149calling it. 150 151The trade-off is that `Batch` can call the given 152function multiple times, if parts of the transaction fail. The 153function must be idempotent and side effects must take effect only 154after a successful return from `DB.Batch()`. 155 156For example: don't display messages from inside the function, instead 157set variables in the enclosing scope: 158 159```go 160var id uint64 161err := db.Batch(func(tx *bolt.Tx) error { 162 // Find last key in bucket, decode as bigendian uint64, increment 163 // by one, encode back to []byte, and add new key. 164 ... 165 id = newValue 166 return nil 167}) 168if err != nil { 169 return ... 170} 171fmt.Println("Allocated ID %d", id) 172``` 173 174 175#### Managing transactions manually 176 177The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()` 178function. These helper functions will start the transaction, execute a function, 179and then safely close your transaction if an error is returned. This is the 180recommended way to use Bolt transactions. 181 182However, sometimes you may want to manually start and end your transactions. 183You can use the `Tx.Begin()` function directly but _please_ be sure to close the 184transaction. 185 186```go 187// Start a writable transaction. 188tx, err := db.Begin(true) 189if err != nil { 190 return err 191} 192defer tx.Rollback() 193 194// Use the transaction... 195_, err := tx.CreateBucket([]byte("MyBucket")) 196if err != nil { 197 return err 198} 199 200// Commit the transaction and check for error. 201if err := tx.Commit(); err != nil { 202 return err 203} 204``` 205 206The first argument to `DB.Begin()` is a boolean stating if the transaction 207should be writable. 208 209 210### Using buckets 211 212Buckets are collections of key/value pairs within the database. All keys in a 213bucket must be unique. You can create a bucket using the `DB.CreateBucket()` 214function: 215 216```go 217db.Update(func(tx *bolt.Tx) error { 218 b, err := tx.CreateBucket([]byte("MyBucket")) 219 if err != nil { 220 return fmt.Errorf("create bucket: %s", err) 221 } 222 return nil 223}) 224``` 225 226You can also create a bucket only if it doesn't exist by using the 227`Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this 228function for all your top-level buckets after you open your database so you can 229guarantee that they exist for future transactions. 230 231To delete a bucket, simply call the `Tx.DeleteBucket()` function. 232 233 234### Using key/value pairs 235 236To save a key/value pair to a bucket, use the `Bucket.Put()` function: 237 238```go 239db.Update(func(tx *bolt.Tx) error { 240 b := tx.Bucket([]byte("MyBucket")) 241 err := b.Put([]byte("answer"), []byte("42")) 242 return err 243}) 244``` 245 246This will set the value of the `"answer"` key to `"42"` in the `MyBucket` 247bucket. To retrieve this value, we can use the `Bucket.Get()` function: 248 249```go 250db.View(func(tx *bolt.Tx) error { 251 b := tx.Bucket([]byte("MyBucket")) 252 v := b.Get([]byte("answer")) 253 fmt.Printf("The answer is: %s\n", v) 254 return nil 255}) 256``` 257 258The `Get()` function does not return an error because its operation is 259guarenteed to work (unless there is some kind of system failure). If the key 260exists then it will return its byte slice value. If it doesn't exist then it 261will return `nil`. It's important to note that you can have a zero-length value 262set to a key which is different than the key not existing. 263 264Use the `Bucket.Delete()` function to delete a key from the bucket. 265 266Please note that values returned from `Get()` are only valid while the 267transaction is open. If you need to use a value outside of the transaction 268then you must use `copy()` to copy it to another byte slice. 269 270 271### Iterating over keys 272 273Bolt stores its keys in byte-sorted order within a bucket. This makes sequential 274iteration over these keys extremely fast. To iterate over keys we'll use a 275`Cursor`: 276 277```go 278db.View(func(tx *bolt.Tx) error { 279 b := tx.Bucket([]byte("MyBucket")) 280 c := b.Cursor() 281 282 for k, v := c.First(); k != nil; k, v = c.Next() { 283 fmt.Printf("key=%s, value=%s\n", k, v) 284 } 285 286 return nil 287}) 288``` 289 290The cursor allows you to move to a specific point in the list of keys and move 291forward or backward through the keys one at a time. 292 293The following functions are available on the cursor: 294 295``` 296First() Move to the first key. 297Last() Move to the last key. 298Seek() Move to a specific key. 299Next() Move to the next key. 300Prev() Move to the previous key. 301``` 302 303When you have iterated to the end of the cursor then `Next()` will return `nil`. 304You must seek to a position using `First()`, `Last()`, or `Seek()` before 305calling `Next()` or `Prev()`. If you do not seek to a position then these 306functions will return `nil`. 307 308 309#### Prefix scans 310 311To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`: 312 313```go 314db.View(func(tx *bolt.Tx) error { 315 c := tx.Bucket([]byte("MyBucket")).Cursor() 316 317 prefix := []byte("1234") 318 for k, v := c.Seek(prefix); bytes.HasPrefix(k, prefix); k, v = c.Next() { 319 fmt.Printf("key=%s, value=%s\n", k, v) 320 } 321 322 return nil 323}) 324``` 325 326#### Range scans 327 328Another common use case is scanning over a range such as a time range. If you 329use a sortable time encoding such as RFC3339 then you can query a specific 330date range like this: 331 332```go 333db.View(func(tx *bolt.Tx) error { 334 // Assume our events bucket has RFC3339 encoded time keys. 335 c := tx.Bucket([]byte("Events")).Cursor() 336 337 // Our time range spans the 90's decade. 338 min := []byte("1990-01-01T00:00:00Z") 339 max := []byte("2000-01-01T00:00:00Z") 340 341 // Iterate over the 90's. 342 for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() { 343 fmt.Printf("%s: %s\n", k, v) 344 } 345 346 return nil 347}) 348``` 349 350 351#### ForEach() 352 353You can also use the function `ForEach()` if you know you'll be iterating over 354all the keys in a bucket: 355 356```go 357db.View(func(tx *bolt.Tx) error { 358 b := tx.Bucket([]byte("MyBucket")) 359 b.ForEach(func(k, v []byte) error { 360 fmt.Printf("key=%s, value=%s\n", k, v) 361 return nil 362 }) 363 return nil 364}) 365``` 366 367 368### Nested buckets 369 370You can also store a bucket in a key to create nested buckets. The API is the 371same as the bucket management API on the `DB` object: 372 373```go 374func (*Bucket) CreateBucket(key []byte) (*Bucket, error) 375func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error) 376func (*Bucket) DeleteBucket(key []byte) error 377``` 378 379 380### Database backups 381 382Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()` 383function to write a consistent view of the database to a writer. If you call 384this from a read-only transaction, it will perform a hot backup and not block 385your other database reads and writes. It will also use `O_DIRECT` when available 386to prevent page cache trashing. 387 388One common use case is to backup over HTTP so you can use tools like `cURL` to 389do database backups: 390 391```go 392func BackupHandleFunc(w http.ResponseWriter, req *http.Request) { 393 err := db.View(func(tx *bolt.Tx) error { 394 w.Header().Set("Content-Type", "application/octet-stream") 395 w.Header().Set("Content-Disposition", `attachment; filename="my.db"`) 396 w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size()))) 397 _, err := tx.WriteTo(w) 398 return err 399 }) 400 if err != nil { 401 http.Error(w, err.Error(), http.StatusInternalServerError) 402 } 403} 404``` 405 406Then you can backup using this command: 407 408```sh 409$ curl http://localhost/backup > my.db 410``` 411 412Or you can open your browser to `http://localhost/backup` and it will download 413automatically. 414 415If you want to backup to another file you can use the `Tx.CopyFile()` helper 416function. 417 418 419### Statistics 420 421The database keeps a running count of many of the internal operations it 422performs so you can better understand what's going on. By grabbing a snapshot 423of these stats at two points in time we can see what operations were performed 424in that time range. 425 426For example, we could start a goroutine to log stats every 10 seconds: 427 428```go 429go func() { 430 // Grab the initial stats. 431 prev := db.Stats() 432 433 for { 434 // Wait for 10s. 435 time.Sleep(10 * time.Second) 436 437 // Grab the current stats and diff them. 438 stats := db.Stats() 439 diff := stats.Sub(&prev) 440 441 // Encode stats to JSON and print to STDERR. 442 json.NewEncoder(os.Stderr).Encode(diff) 443 444 // Save stats for the next loop. 445 prev = stats 446 } 447}() 448``` 449 450It's also useful to pipe these stats to a service such as statsd for monitoring 451or to provide an HTTP endpoint that will perform a fixed-length sample. 452 453 454### Read-Only Mode 455 456Sometimes it is useful to create a shared, read-only Bolt database. To this, 457set the `Options.ReadOnly` flag when opening your database. Read-only mode 458uses a shared lock to allow multiple processes to read from the database but 459it will block any processes from opening the database in read-write mode. 460 461```go 462db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true}) 463if err != nil { 464 log.Fatal(err) 465} 466``` 467 468 469## Resources 470 471For more information on getting started with Bolt, check out the following articles: 472 473* [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch). 474* [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville 475 476 477## Comparison with other databases 478 479### Postgres, MySQL, & other relational databases 480 481Relational databases structure data into rows and are only accessible through 482the use of SQL. This approach provides flexibility in how you store and query 483your data but also incurs overhead in parsing and planning SQL statements. Bolt 484accesses all data by a byte slice key. This makes Bolt fast to read and write 485data by key but provides no built-in support for joining values together. 486 487Most relational databases (with the exception of SQLite) are standalone servers 488that run separately from your application. This gives your systems 489flexibility to connect multiple application servers to a single database 490server but also adds overhead in serializing and transporting data over the 491network. Bolt runs as a library included in your application so all data access 492has to go through your application's process. This brings data closer to your 493application but limits multi-process access to the data. 494 495 496### LevelDB, RocksDB 497 498LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that 499they are libraries bundled into the application, however, their underlying 500structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes 501random writes by using a write ahead log and multi-tiered, sorted files called 502SSTables. Bolt uses a B+tree internally and only a single file. Both approaches 503have trade offs. 504 505If you require a high random write throughput (>10,000 w/sec) or you need to use 506spinning disks then LevelDB could be a good choice. If your application is 507read-heavy or does a lot of range scans then Bolt could be a good choice. 508 509One other important consideration is that LevelDB does not have transactions. 510It supports batch writing of key/values pairs and it supports read snapshots 511but it will not give you the ability to do a compare-and-swap operation safely. 512Bolt supports fully serializable ACID transactions. 513 514 515### LMDB 516 517Bolt was originally a port of LMDB so it is architecturally similar. Both use 518a B+tree, have ACID semantics with fully serializable transactions, and support 519lock-free MVCC using a single writer and multiple readers. 520 521The two projects have somewhat diverged. LMDB heavily focuses on raw performance 522while Bolt has focused on simplicity and ease of use. For example, LMDB allows 523several unsafe actions such as direct writes for the sake of performance. Bolt 524opts to disallow actions which can leave the database in a corrupted state. The 525only exception to this in Bolt is `DB.NoSync`. 526 527There are also a few differences in API. LMDB requires a maximum mmap size when 528opening an `mdb_env` whereas Bolt will handle incremental mmap resizing 529automatically. LMDB overloads the getter and setter functions with multiple 530flags whereas Bolt splits these specialized cases into their own functions. 531 532 533## Caveats & Limitations 534 535It's important to pick the right tool for the job and Bolt is no exception. 536Here are a few things to note when evaluating and using Bolt: 537 538* Bolt is good for read intensive workloads. Sequential write performance is 539 also fast but random writes can be slow. You can add a write-ahead log or 540 [transaction coalescer](https://github.com/boltdb/coalescer) in front of Bolt 541 to mitigate this issue. 542 543* Bolt uses a B+tree internally so there can be a lot of random page access. 544 SSDs provide a significant performance boost over spinning disks. 545 546* Try to avoid long running read transactions. Bolt uses copy-on-write so 547 old pages cannot be reclaimed while an old transaction is using them. 548 549* Byte slices returned from Bolt are only valid during a transaction. Once the 550 transaction has been committed or rolled back then the memory they point to 551 can be reused by a new page or can be unmapped from virtual memory and you'll 552 see an `unexpected fault address` panic when accessing it. 553 554* Be careful when using `Bucket.FillPercent`. Setting a high fill percent for 555 buckets that have random inserts will cause your database to have very poor 556 page utilization. 557 558* Use larger buckets in general. Smaller buckets causes poor page utilization 559 once they become larger than the page size (typically 4KB). 560 561* Bulk loading a lot of random writes into a new bucket can be slow as the 562 page will not split until the transaction is committed. Randomly inserting 563 more than 100,000 key/value pairs into a single new bucket in a single 564 transaction is not advised. 565 566* Bolt uses a memory-mapped file so the underlying operating system handles the 567 caching of the data. Typically, the OS will cache as much of the file as it 568 can in memory and will release memory as needed to other processes. This means 569 that Bolt can show very high memory usage when working with large databases. 570 However, this is expected and the OS will release memory as needed. Bolt can 571 handle databases much larger than the available physical RAM. 572 573* The data structures in the Bolt database are memory mapped so the data file 574 will be endian specific. This means that you cannot copy a Bolt file from a 575 little endian machine to a big endian machine and have it work. For most 576 users this is not a concern since most modern CPUs are little endian. 577 578* Because of the way pages are laid out on disk, Bolt cannot truncate data files 579 and return free pages back to the disk. Instead, Bolt maintains a free list 580 of unused pages within its data file. These free pages can be reused by later 581 transactions. This works well for many use cases as databases generally tend 582 to grow. However, it's important to note that deleting large chunks of data 583 will not allow you to reclaim that space on disk. 584 585 For more information on page allocation, [see this comment][page-allocation]. 586 587[page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638 588 589 590## Other Projects Using Bolt 591 592Below is a list of public, open source projects that use Bolt: 593 594* [Operation Go: A Routine Mission](http://gocode.io) - An online programming game for Golang using Bolt for user accounts and a leaderboard. 595* [Bazil](https://github.com/bazillion/bazil) - A file system that lets your data reside where it is most convenient for it to reside. 596* [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb. 597* [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics. 598* [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects. 599* [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday. 600* [ChainStore](https://github.com/nulayer/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations. 601* [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite. 602* [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin". 603* [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka. 604* [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed. 605* [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt. 606* [photosite/session](http://godoc.org/bitbucket.org/kardianos/photosite/session) - Sessions for a photo viewing site. 607* [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage. 608* [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters. 609* [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend. 610* [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend. 611* [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server. 612* [SkyDB](https://github.com/skydb/sky) - Behavioral analytics database. 613* [Seaweed File System](https://github.com/chrislusf/weed-fs) - Highly scalable distributed key~file system with O(1) disk read. 614* [InfluxDB](http://influxdb.com) - Scalable datastore for metrics, events, and real-time analytics. 615* [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data. 616* [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system. 617* [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware. 618* [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistant, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs. 619* [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems. 620 621If you are using Bolt in a project please send a pull request to add it to the list. 622