1bbolt 2===== 3 4[![Go Report Card](https://goreportcard.com/badge/github.com/etcd-io/bbolt?style=flat-square)](https://goreportcard.com/report/github.com/etcd-io/bbolt) 5[![Coverage](https://codecov.io/gh/etcd-io/bbolt/branch/master/graph/badge.svg)](https://codecov.io/gh/etcd-io/bbolt) 6[![Build Status Travis](https://img.shields.io/travis/etcd-io/bboltlabs.svg?style=flat-square&&branch=master)](https://travis-ci.com/etcd-io/bbolt) 7[![Godoc](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)](https://godoc.org/github.com/etcd-io/bbolt) 8[![Releases](https://img.shields.io/github/release/etcd-io/bbolt/all.svg?style=flat-square)](https://github.com/etcd-io/bbolt/releases) 9[![LICENSE](https://img.shields.io/github/license/etcd-io/bbolt.svg?style=flat-square)](https://github.com/etcd-io/bbolt/blob/master/LICENSE) 10 11bbolt is a fork of [Ben Johnson's][gh_ben] [Bolt][bolt] key/value 12store. The purpose of this fork is to provide the Go community with an active 13maintenance and development target for Bolt; the goal is improved reliability 14and stability. bbolt includes bug fixes, performance enhancements, and features 15not found in Bolt while preserving backwards compatibility with the Bolt API. 16 17Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas] 18[LMDB project][lmdb]. The goal of the project is to provide a simple, 19fast, and reliable database for projects that don't require a full database 20server such as Postgres or MySQL. 21 22Since Bolt is meant to be used as such a low-level piece of functionality, 23simplicity is key. The API will be small and only focus on getting values 24and setting values. That's it. 25 26[gh_ben]: https://github.com/benbjohnson 27[bolt]: https://github.com/boltdb/bolt 28[hyc_symas]: https://twitter.com/hyc_symas 29[lmdb]: http://symas.com/mdb/ 30 31## Project Status 32 33Bolt is stable, the API is fixed, and the file format is fixed. Full unit 34test coverage and randomized black box testing are used to ensure database 35consistency and thread safety. Bolt is currently used in high-load production 36environments serving databases as large as 1TB. Many companies such as 37Shopify and Heroku use Bolt-backed services every day. 38 39## Project versioning 40 41bbolt uses [semantic versioning](http://semver.org). 42API should not change between patch and minor releases. 43New minor versions may add additional features to the API. 44 45## Table of Contents 46 47 - [Getting Started](#getting-started) 48 - [Installing](#installing) 49 - [Opening a database](#opening-a-database) 50 - [Transactions](#transactions) 51 - [Read-write transactions](#read-write-transactions) 52 - [Read-only transactions](#read-only-transactions) 53 - [Batch read-write transactions](#batch-read-write-transactions) 54 - [Managing transactions manually](#managing-transactions-manually) 55 - [Using buckets](#using-buckets) 56 - [Using key/value pairs](#using-keyvalue-pairs) 57 - [Autoincrementing integer for the bucket](#autoincrementing-integer-for-the-bucket) 58 - [Iterating over keys](#iterating-over-keys) 59 - [Prefix scans](#prefix-scans) 60 - [Range scans](#range-scans) 61 - [ForEach()](#foreach) 62 - [Nested buckets](#nested-buckets) 63 - [Database backups](#database-backups) 64 - [Statistics](#statistics) 65 - [Read-Only Mode](#read-only-mode) 66 - [Mobile Use (iOS/Android)](#mobile-use-iosandroid) 67 - [Resources](#resources) 68 - [Comparison with other databases](#comparison-with-other-databases) 69 - [Postgres, MySQL, & other relational databases](#postgres-mysql--other-relational-databases) 70 - [LevelDB, RocksDB](#leveldb-rocksdb) 71 - [LMDB](#lmdb) 72 - [Caveats & Limitations](#caveats--limitations) 73 - [Reading the Source](#reading-the-source) 74 - [Other Projects Using Bolt](#other-projects-using-bolt) 75 76## Getting Started 77 78### Installing 79 80To start using Bolt, install Go and run `go get`: 81 82```sh 83$ go get go.etcd.io/bbolt/... 84``` 85 86This will retrieve the library and install the `bolt` command line utility into 87your `$GOBIN` path. 88 89 90### Importing bbolt 91 92To use bbolt as an embedded key-value store, import as: 93 94```go 95import bolt "go.etcd.io/bbolt" 96 97db, err := bolt.Open(path, 0666, nil) 98if err != nil { 99 return err 100} 101defer db.Close() 102``` 103 104 105### Opening a database 106 107The top-level object in Bolt is a `DB`. It is represented as a single file on 108your disk and represents a consistent snapshot of your data. 109 110To open your database, simply use the `bolt.Open()` function: 111 112```go 113package main 114 115import ( 116 "log" 117 118 bolt "go.etcd.io/bbolt" 119) 120 121func main() { 122 // Open the my.db data file in your current directory. 123 // It will be created if it doesn't exist. 124 db, err := bolt.Open("my.db", 0600, nil) 125 if err != nil { 126 log.Fatal(err) 127 } 128 defer db.Close() 129 130 ... 131} 132``` 133 134Please note that Bolt obtains a file lock on the data file so multiple processes 135cannot open the same database at the same time. Opening an already open Bolt 136database will cause it to hang until the other process closes it. To prevent 137an indefinite wait you can pass a timeout option to the `Open()` function: 138 139```go 140db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second}) 141``` 142 143 144### Transactions 145 146Bolt allows only one read-write transaction at a time but allows as many 147read-only transactions as you want at a time. Each transaction has a consistent 148view of the data as it existed when the transaction started. 149 150Individual transactions and all objects created from them (e.g. buckets, keys) 151are not thread safe. To work with data in multiple goroutines you must start 152a transaction for each one or use locking to ensure only one goroutine accesses 153a transaction at a time. Creating transaction from the `DB` is thread safe. 154 155Transactions should not depend on one another and generally shouldn't be opened 156simultaneously in the same goroutine. This can cause a deadlock as the read-write 157transaction needs to periodically re-map the data file but it cannot do so while 158any read-only transaction is open. Even a nested read-only transaction can cause 159a deadlock, as the child transaction can block the parent transaction from releasing 160its resources. 161 162#### Read-write transactions 163 164To start a read-write transaction, you can use the `DB.Update()` function: 165 166```go 167err := db.Update(func(tx *bolt.Tx) error { 168 ... 169 return nil 170}) 171``` 172 173Inside the closure, you have a consistent view of the database. You commit the 174transaction by returning `nil` at the end. You can also rollback the transaction 175at any point by returning an error. All database operations are allowed inside 176a read-write transaction. 177 178Always check the return error as it will report any disk failures that can cause 179your transaction to not complete. If you return an error within your closure 180it will be passed through. 181 182 183#### Read-only transactions 184 185To start a read-only transaction, you can use the `DB.View()` function: 186 187```go 188err := db.View(func(tx *bolt.Tx) error { 189 ... 190 return nil 191}) 192``` 193 194You also get a consistent view of the database within this closure, however, 195no mutating operations are allowed within a read-only transaction. You can only 196retrieve buckets, retrieve values, and copy the database within a read-only 197transaction. 198 199 200#### Batch read-write transactions 201 202Each `DB.Update()` waits for disk to commit the writes. This overhead 203can be minimized by combining multiple updates with the `DB.Batch()` 204function: 205 206```go 207err := db.Batch(func(tx *bolt.Tx) error { 208 ... 209 return nil 210}) 211``` 212 213Concurrent Batch calls are opportunistically combined into larger 214transactions. Batch is only useful when there are multiple goroutines 215calling it. 216 217The trade-off is that `Batch` can call the given 218function multiple times, if parts of the transaction fail. The 219function must be idempotent and side effects must take effect only 220after a successful return from `DB.Batch()`. 221 222For example: don't display messages from inside the function, instead 223set variables in the enclosing scope: 224 225```go 226var id uint64 227err := db.Batch(func(tx *bolt.Tx) error { 228 // Find last key in bucket, decode as bigendian uint64, increment 229 // by one, encode back to []byte, and add new key. 230 ... 231 id = newValue 232 return nil 233}) 234if err != nil { 235 return ... 236} 237fmt.Println("Allocated ID %d", id) 238``` 239 240 241#### Managing transactions manually 242 243The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()` 244function. These helper functions will start the transaction, execute a function, 245and then safely close your transaction if an error is returned. This is the 246recommended way to use Bolt transactions. 247 248However, sometimes you may want to manually start and end your transactions. 249You can use the `DB.Begin()` function directly but **please** be sure to close 250the transaction. 251 252```go 253// Start a writable transaction. 254tx, err := db.Begin(true) 255if err != nil { 256 return err 257} 258defer tx.Rollback() 259 260// Use the transaction... 261_, err := tx.CreateBucket([]byte("MyBucket")) 262if err != nil { 263 return err 264} 265 266// Commit the transaction and check for error. 267if err := tx.Commit(); err != nil { 268 return err 269} 270``` 271 272The first argument to `DB.Begin()` is a boolean stating if the transaction 273should be writable. 274 275 276### Using buckets 277 278Buckets are collections of key/value pairs within the database. All keys in a 279bucket must be unique. You can create a bucket using the `Tx.CreateBucket()` 280function: 281 282```go 283db.Update(func(tx *bolt.Tx) error { 284 b, err := tx.CreateBucket([]byte("MyBucket")) 285 if err != nil { 286 return fmt.Errorf("create bucket: %s", err) 287 } 288 return nil 289}) 290``` 291 292You can also create a bucket only if it doesn't exist by using the 293`Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this 294function for all your top-level buckets after you open your database so you can 295guarantee that they exist for future transactions. 296 297To delete a bucket, simply call the `Tx.DeleteBucket()` function. 298 299 300### Using key/value pairs 301 302To save a key/value pair to a bucket, use the `Bucket.Put()` function: 303 304```go 305db.Update(func(tx *bolt.Tx) error { 306 b := tx.Bucket([]byte("MyBucket")) 307 err := b.Put([]byte("answer"), []byte("42")) 308 return err 309}) 310``` 311 312This will set the value of the `"answer"` key to `"42"` in the `MyBucket` 313bucket. To retrieve this value, we can use the `Bucket.Get()` function: 314 315```go 316db.View(func(tx *bolt.Tx) error { 317 b := tx.Bucket([]byte("MyBucket")) 318 v := b.Get([]byte("answer")) 319 fmt.Printf("The answer is: %s\n", v) 320 return nil 321}) 322``` 323 324The `Get()` function does not return an error because its operation is 325guaranteed to work (unless there is some kind of system failure). If the key 326exists then it will return its byte slice value. If it doesn't exist then it 327will return `nil`. It's important to note that you can have a zero-length value 328set to a key which is different than the key not existing. 329 330Use the `Bucket.Delete()` function to delete a key from the bucket. 331 332Please note that values returned from `Get()` are only valid while the 333transaction is open. If you need to use a value outside of the transaction 334then you must use `copy()` to copy it to another byte slice. 335 336 337### Autoincrementing integer for the bucket 338By using the `NextSequence()` function, you can let Bolt determine a sequence 339which can be used as the unique identifier for your key/value pairs. See the 340example below. 341 342```go 343// CreateUser saves u to the store. The new user ID is set on u once the data is persisted. 344func (s *Store) CreateUser(u *User) error { 345 return s.db.Update(func(tx *bolt.Tx) error { 346 // Retrieve the users bucket. 347 // This should be created when the DB is first opened. 348 b := tx.Bucket([]byte("users")) 349 350 // Generate ID for the user. 351 // This returns an error only if the Tx is closed or not writeable. 352 // That can't happen in an Update() call so I ignore the error check. 353 id, _ := b.NextSequence() 354 u.ID = int(id) 355 356 // Marshal user data into bytes. 357 buf, err := json.Marshal(u) 358 if err != nil { 359 return err 360 } 361 362 // Persist bytes to users bucket. 363 return b.Put(itob(u.ID), buf) 364 }) 365} 366 367// itob returns an 8-byte big endian representation of v. 368func itob(v int) []byte { 369 b := make([]byte, 8) 370 binary.BigEndian.PutUint64(b, uint64(v)) 371 return b 372} 373 374type User struct { 375 ID int 376 ... 377} 378``` 379 380### Iterating over keys 381 382Bolt stores its keys in byte-sorted order within a bucket. This makes sequential 383iteration over these keys extremely fast. To iterate over keys we'll use a 384`Cursor`: 385 386```go 387db.View(func(tx *bolt.Tx) error { 388 // Assume bucket exists and has keys 389 b := tx.Bucket([]byte("MyBucket")) 390 391 c := b.Cursor() 392 393 for k, v := c.First(); k != nil; k, v = c.Next() { 394 fmt.Printf("key=%s, value=%s\n", k, v) 395 } 396 397 return nil 398}) 399``` 400 401The cursor allows you to move to a specific point in the list of keys and move 402forward or backward through the keys one at a time. 403 404The following functions are available on the cursor: 405 406``` 407First() Move to the first key. 408Last() Move to the last key. 409Seek() Move to a specific key. 410Next() Move to the next key. 411Prev() Move to the previous key. 412``` 413 414Each of those functions has a return signature of `(key []byte, value []byte)`. 415When you have iterated to the end of the cursor then `Next()` will return a 416`nil` key. You must seek to a position using `First()`, `Last()`, or `Seek()` 417before calling `Next()` or `Prev()`. If you do not seek to a position then 418these functions will return a `nil` key. 419 420During iteration, if the key is non-`nil` but the value is `nil`, that means 421the key refers to a bucket rather than a value. Use `Bucket.Bucket()` to 422access the sub-bucket. 423 424 425#### Prefix scans 426 427To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`: 428 429```go 430db.View(func(tx *bolt.Tx) error { 431 // Assume bucket exists and has keys 432 c := tx.Bucket([]byte("MyBucket")).Cursor() 433 434 prefix := []byte("1234") 435 for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() { 436 fmt.Printf("key=%s, value=%s\n", k, v) 437 } 438 439 return nil 440}) 441``` 442 443#### Range scans 444 445Another common use case is scanning over a range such as a time range. If you 446use a sortable time encoding such as RFC3339 then you can query a specific 447date range like this: 448 449```go 450db.View(func(tx *bolt.Tx) error { 451 // Assume our events bucket exists and has RFC3339 encoded time keys. 452 c := tx.Bucket([]byte("Events")).Cursor() 453 454 // Our time range spans the 90's decade. 455 min := []byte("1990-01-01T00:00:00Z") 456 max := []byte("2000-01-01T00:00:00Z") 457 458 // Iterate over the 90's. 459 for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() { 460 fmt.Printf("%s: %s\n", k, v) 461 } 462 463 return nil 464}) 465``` 466 467Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable. 468 469 470#### ForEach() 471 472You can also use the function `ForEach()` if you know you'll be iterating over 473all the keys in a bucket: 474 475```go 476db.View(func(tx *bolt.Tx) error { 477 // Assume bucket exists and has keys 478 b := tx.Bucket([]byte("MyBucket")) 479 480 b.ForEach(func(k, v []byte) error { 481 fmt.Printf("key=%s, value=%s\n", k, v) 482 return nil 483 }) 484 return nil 485}) 486``` 487 488Please note that keys and values in `ForEach()` are only valid while 489the transaction is open. If you need to use a key or value outside of 490the transaction, you must use `copy()` to copy it to another byte 491slice. 492 493### Nested buckets 494 495You can also store a bucket in a key to create nested buckets. The API is the 496same as the bucket management API on the `DB` object: 497 498```go 499func (*Bucket) CreateBucket(key []byte) (*Bucket, error) 500func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error) 501func (*Bucket) DeleteBucket(key []byte) error 502``` 503 504Say you had a multi-tenant application where the root level bucket was the account bucket. Inside of this bucket was a sequence of accounts which themselves are buckets. And inside the sequence bucket you could have many buckets pertaining to the Account itself (Users, Notes, etc) isolating the information into logical groupings. 505 506```go 507 508// createUser creates a new user in the given account. 509func createUser(accountID int, u *User) error { 510 // Start the transaction. 511 tx, err := db.Begin(true) 512 if err != nil { 513 return err 514 } 515 defer tx.Rollback() 516 517 // Retrieve the root bucket for the account. 518 // Assume this has already been created when the account was set up. 519 root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10))) 520 521 // Setup the users bucket. 522 bkt, err := root.CreateBucketIfNotExists([]byte("USERS")) 523 if err != nil { 524 return err 525 } 526 527 // Generate an ID for the new user. 528 userID, err := bkt.NextSequence() 529 if err != nil { 530 return err 531 } 532 u.ID = userID 533 534 // Marshal and save the encoded user. 535 if buf, err := json.Marshal(u); err != nil { 536 return err 537 } else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil { 538 return err 539 } 540 541 // Commit the transaction. 542 if err := tx.Commit(); err != nil { 543 return err 544 } 545 546 return nil 547} 548 549``` 550 551 552 553 554### Database backups 555 556Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()` 557function to write a consistent view of the database to a writer. If you call 558this from a read-only transaction, it will perform a hot backup and not block 559your other database reads and writes. 560 561By default, it will use a regular file handle which will utilize the operating 562system's page cache. See the [`Tx`](https://godoc.org/go.etcd.io/bbolt#Tx) 563documentation for information about optimizing for larger-than-RAM datasets. 564 565One common use case is to backup over HTTP so you can use tools like `cURL` to 566do database backups: 567 568```go 569func BackupHandleFunc(w http.ResponseWriter, req *http.Request) { 570 err := db.View(func(tx *bolt.Tx) error { 571 w.Header().Set("Content-Type", "application/octet-stream") 572 w.Header().Set("Content-Disposition", `attachment; filename="my.db"`) 573 w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size()))) 574 _, err := tx.WriteTo(w) 575 return err 576 }) 577 if err != nil { 578 http.Error(w, err.Error(), http.StatusInternalServerError) 579 } 580} 581``` 582 583Then you can backup using this command: 584 585```sh 586$ curl http://localhost/backup > my.db 587``` 588 589Or you can open your browser to `http://localhost/backup` and it will download 590automatically. 591 592If you want to backup to another file you can use the `Tx.CopyFile()` helper 593function. 594 595 596### Statistics 597 598The database keeps a running count of many of the internal operations it 599performs so you can better understand what's going on. By grabbing a snapshot 600of these stats at two points in time we can see what operations were performed 601in that time range. 602 603For example, we could start a goroutine to log stats every 10 seconds: 604 605```go 606go func() { 607 // Grab the initial stats. 608 prev := db.Stats() 609 610 for { 611 // Wait for 10s. 612 time.Sleep(10 * time.Second) 613 614 // Grab the current stats and diff them. 615 stats := db.Stats() 616 diff := stats.Sub(&prev) 617 618 // Encode stats to JSON and print to STDERR. 619 json.NewEncoder(os.Stderr).Encode(diff) 620 621 // Save stats for the next loop. 622 prev = stats 623 } 624}() 625``` 626 627It's also useful to pipe these stats to a service such as statsd for monitoring 628or to provide an HTTP endpoint that will perform a fixed-length sample. 629 630 631### Read-Only Mode 632 633Sometimes it is useful to create a shared, read-only Bolt database. To this, 634set the `Options.ReadOnly` flag when opening your database. Read-only mode 635uses a shared lock to allow multiple processes to read from the database but 636it will block any processes from opening the database in read-write mode. 637 638```go 639db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true}) 640if err != nil { 641 log.Fatal(err) 642} 643``` 644 645### Mobile Use (iOS/Android) 646 647Bolt is able to run on mobile devices by leveraging the binding feature of the 648[gomobile](https://github.com/golang/mobile) tool. Create a struct that will 649contain your database logic and a reference to a `*bolt.DB` with a initializing 650constructor that takes in a filepath where the database file will be stored. 651Neither Android nor iOS require extra permissions or cleanup from using this method. 652 653```go 654func NewBoltDB(filepath string) *BoltDB { 655 db, err := bolt.Open(filepath+"/demo.db", 0600, nil) 656 if err != nil { 657 log.Fatal(err) 658 } 659 660 return &BoltDB{db} 661} 662 663type BoltDB struct { 664 db *bolt.DB 665 ... 666} 667 668func (b *BoltDB) Path() string { 669 return b.db.Path() 670} 671 672func (b *BoltDB) Close() { 673 b.db.Close() 674} 675``` 676 677Database logic should be defined as methods on this wrapper struct. 678 679To initialize this struct from the native language (both platforms now sync 680their local storage to the cloud. These snippets disable that functionality for the 681database file): 682 683#### Android 684 685```java 686String path; 687if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){ 688 path = getNoBackupFilesDir().getAbsolutePath(); 689} else{ 690 path = getFilesDir().getAbsolutePath(); 691} 692Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path) 693``` 694 695#### iOS 696 697```objc 698- (void)demo { 699 NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory, 700 NSUserDomainMask, 701 YES) objectAtIndex:0]; 702 GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path); 703 [self addSkipBackupAttributeToItemAtPath:demo.path]; 704 //Some DB Logic would go here 705 [demo close]; 706} 707 708- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString 709{ 710 NSURL* URL= [NSURL fileURLWithPath: filePathString]; 711 assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]); 712 713 NSError *error = nil; 714 BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES] 715 forKey: NSURLIsExcludedFromBackupKey error: &error]; 716 if(!success){ 717 NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error); 718 } 719 return success; 720} 721 722``` 723 724## Resources 725 726For more information on getting started with Bolt, check out the following articles: 727 728* [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch). 729* [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville 730 731 732## Comparison with other databases 733 734### Postgres, MySQL, & other relational databases 735 736Relational databases structure data into rows and are only accessible through 737the use of SQL. This approach provides flexibility in how you store and query 738your data but also incurs overhead in parsing and planning SQL statements. Bolt 739accesses all data by a byte slice key. This makes Bolt fast to read and write 740data by key but provides no built-in support for joining values together. 741 742Most relational databases (with the exception of SQLite) are standalone servers 743that run separately from your application. This gives your systems 744flexibility to connect multiple application servers to a single database 745server but also adds overhead in serializing and transporting data over the 746network. Bolt runs as a library included in your application so all data access 747has to go through your application's process. This brings data closer to your 748application but limits multi-process access to the data. 749 750 751### LevelDB, RocksDB 752 753LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that 754they are libraries bundled into the application, however, their underlying 755structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes 756random writes by using a write ahead log and multi-tiered, sorted files called 757SSTables. Bolt uses a B+tree internally and only a single file. Both approaches 758have trade-offs. 759 760If you require a high random write throughput (>10,000 w/sec) or you need to use 761spinning disks then LevelDB could be a good choice. If your application is 762read-heavy or does a lot of range scans then Bolt could be a good choice. 763 764One other important consideration is that LevelDB does not have transactions. 765It supports batch writing of key/values pairs and it supports read snapshots 766but it will not give you the ability to do a compare-and-swap operation safely. 767Bolt supports fully serializable ACID transactions. 768 769 770### LMDB 771 772Bolt was originally a port of LMDB so it is architecturally similar. Both use 773a B+tree, have ACID semantics with fully serializable transactions, and support 774lock-free MVCC using a single writer and multiple readers. 775 776The two projects have somewhat diverged. LMDB heavily focuses on raw performance 777while Bolt has focused on simplicity and ease of use. For example, LMDB allows 778several unsafe actions such as direct writes for the sake of performance. Bolt 779opts to disallow actions which can leave the database in a corrupted state. The 780only exception to this in Bolt is `DB.NoSync`. 781 782There are also a few differences in API. LMDB requires a maximum mmap size when 783opening an `mdb_env` whereas Bolt will handle incremental mmap resizing 784automatically. LMDB overloads the getter and setter functions with multiple 785flags whereas Bolt splits these specialized cases into their own functions. 786 787 788## Caveats & Limitations 789 790It's important to pick the right tool for the job and Bolt is no exception. 791Here are a few things to note when evaluating and using Bolt: 792 793* Bolt is good for read intensive workloads. Sequential write performance is 794 also fast but random writes can be slow. You can use `DB.Batch()` or add a 795 write-ahead log to help mitigate this issue. 796 797* Bolt uses a B+tree internally so there can be a lot of random page access. 798 SSDs provide a significant performance boost over spinning disks. 799 800* Try to avoid long running read transactions. Bolt uses copy-on-write so 801 old pages cannot be reclaimed while an old transaction is using them. 802 803* Byte slices returned from Bolt are only valid during a transaction. Once the 804 transaction has been committed or rolled back then the memory they point to 805 can be reused by a new page or can be unmapped from virtual memory and you'll 806 see an `unexpected fault address` panic when accessing it. 807 808* Bolt uses an exclusive write lock on the database file so it cannot be 809 shared by multiple processes. 810 811* Be careful when using `Bucket.FillPercent`. Setting a high fill percent for 812 buckets that have random inserts will cause your database to have very poor 813 page utilization. 814 815* Use larger buckets in general. Smaller buckets causes poor page utilization 816 once they become larger than the page size (typically 4KB). 817 818* Bulk loading a lot of random writes into a new bucket can be slow as the 819 page will not split until the transaction is committed. Randomly inserting 820 more than 100,000 key/value pairs into a single new bucket in a single 821 transaction is not advised. 822 823* Bolt uses a memory-mapped file so the underlying operating system handles the 824 caching of the data. Typically, the OS will cache as much of the file as it 825 can in memory and will release memory as needed to other processes. This means 826 that Bolt can show very high memory usage when working with large databases. 827 However, this is expected and the OS will release memory as needed. Bolt can 828 handle databases much larger than the available physical RAM, provided its 829 memory-map fits in the process virtual address space. It may be problematic 830 on 32-bits systems. 831 832* The data structures in the Bolt database are memory mapped so the data file 833 will be endian specific. This means that you cannot copy a Bolt file from a 834 little endian machine to a big endian machine and have it work. For most 835 users this is not a concern since most modern CPUs are little endian. 836 837* Because of the way pages are laid out on disk, Bolt cannot truncate data files 838 and return free pages back to the disk. Instead, Bolt maintains a free list 839 of unused pages within its data file. These free pages can be reused by later 840 transactions. This works well for many use cases as databases generally tend 841 to grow. However, it's important to note that deleting large chunks of data 842 will not allow you to reclaim that space on disk. 843 844 For more information on page allocation, [see this comment][page-allocation]. 845 846[page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638 847 848 849## Reading the Source 850 851Bolt is a relatively small code base (<5KLOC) for an embedded, serializable, 852transactional key/value database so it can be a good starting point for people 853interested in how databases work. 854 855The best places to start are the main entry points into Bolt: 856 857- `Open()` - Initializes the reference to the database. It's responsible for 858 creating the database if it doesn't exist, obtaining an exclusive lock on the 859 file, reading the meta pages, & memory-mapping the file. 860 861- `DB.Begin()` - Starts a read-only or read-write transaction depending on the 862 value of the `writable` argument. This requires briefly obtaining the "meta" 863 lock to keep track of open transactions. Only one read-write transaction can 864 exist at a time so the "rwlock" is acquired during the life of a read-write 865 transaction. 866 867- `Bucket.Put()` - Writes a key/value pair into a bucket. After validating the 868 arguments, a cursor is used to traverse the B+tree to the page and position 869 where they key & value will be written. Once the position is found, the bucket 870 materializes the underlying page and the page's parent pages into memory as 871 "nodes". These nodes are where mutations occur during read-write transactions. 872 These changes get flushed to disk during commit. 873 874- `Bucket.Get()` - Retrieves a key/value pair from a bucket. This uses a cursor 875 to move to the page & position of a key/value pair. During a read-only 876 transaction, the key and value data is returned as a direct reference to the 877 underlying mmap file so there's no allocation overhead. For read-write 878 transactions, this data may reference the mmap file or one of the in-memory 879 node values. 880 881- `Cursor` - This object is simply for traversing the B+tree of on-disk pages 882 or in-memory nodes. It can seek to a specific key, move to the first or last 883 value, or it can move forward or backward. The cursor handles the movement up 884 and down the B+tree transparently to the end user. 885 886- `Tx.Commit()` - Converts the in-memory dirty nodes and the list of free pages 887 into pages to be written to disk. Writing to disk then occurs in two phases. 888 First, the dirty pages are written to disk and an `fsync()` occurs. Second, a 889 new meta page with an incremented transaction ID is written and another 890 `fsync()` occurs. This two phase write ensures that partially written data 891 pages are ignored in the event of a crash since the meta page pointing to them 892 is never written. Partially written meta pages are invalidated because they 893 are written with a checksum. 894 895If you have additional notes that could be helpful for others, please submit 896them via pull request. 897 898 899## Other Projects Using Bolt 900 901Below is a list of public, open source projects that use Bolt: 902 903* [Algernon](https://github.com/xyproto/algernon) - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend. 904* [Bazil](https://bazil.org/) - A file system that lets your data reside where it is most convenient for it to reside. 905* [bolter](https://github.com/hasit/bolter) - Command-line app for viewing BoltDB file in your terminal. 906* [boltcli](https://github.com/spacewander/boltcli) - the redis-cli for boltdb with Lua script support. 907* [BoltHold](https://github.com/timshannon/bolthold) - An embeddable NoSQL store for Go types built on BoltDB 908* [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt. 909* [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners. 910* [BoltDbWeb](https://github.com/evnix/boltdbweb) - A web based GUI for BoltDB files. 911* [BoltDB Viewer](https://github.com/zc310/rich_boltdb) - A BoltDB Viewer Can run on Windows、Linux、Android system. 912* [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend. 913* [btcwallet](https://github.com/btcsuite/btcwallet) - A bitcoin wallet. 914* [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining 915 simple tx and key scans. 916* [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend. 917* [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations. 918* [ Chestnut](https://github.com/jrapoport/chestnut) - Chestnut is encrypted storage for Go. 919* [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware. 920* [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb. 921* [dcrwallet](https://github.com/decred/dcrwallet) - A wallet for the Decred cryptocurrency. 922* [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems. 923* [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka. 924* [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data. 925* [Go Report Card](https://goreportcard.com/) - Go code quality report cards as a (free and open source) service. 926* [GoWebApp](https://github.com/josephspurrier/gowebapp) - A basic MVC web application in Go using BoltDB. 927* [GoShort](https://github.com/pankajkhairnar/goShort) - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter. 928* [gopherpit](https://github.com/gopherpit/gopherpit) - A web service to manage Go remote import paths with custom domains 929* [gokv](https://github.com/philippgille/gokv) - Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more) 930* [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin". 931* [InfluxDB](https://influxdata.com) - Scalable datastore for metrics, events, and real-time analytics. 932* [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters. 933* [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed. 934* [Ironsmith](https://github.com/timshannon/ironsmith) - A simple, script-driven continuous integration (build - > test -> release) tool, with no external dependencies 935* [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs. 936* [Key Value Access Langusge (KVAL)](https://github.com/kval-access-language) - A proposed grammar for key-value datastores offering a bbolt binding. 937* [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage. 938* [lru](https://github.com/crowdriff/lru) - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores. 939* [mbuckets](https://github.com/abhigupta912/mbuckets) - A Bolt wrapper that allows easy operations on multi level (nested) buckets. 940* [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite. 941* [MuLiFS](https://github.com/dankomiocevic/mulifs) - Music Library Filesystem creates a filesystem to organise your music files. 942* [NATS](https://github.com/nats-io/nats-streaming-server) - NATS Streaming uses bbolt for message and metadata storage. 943* [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system. 944* [Rain](https://github.com/cenkalti/rain) - BitTorrent client and library. 945* [reef-pi](https://github.com/reef-pi/reef-pi) - reef-pi is an award winning, modular, DIY reef tank controller using easy to learn electronics based on a Raspberry Pi. 946* [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service 947* [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read. 948* [stow](https://github.com/djherbis/stow) - a persistence manager for objects 949 backed by boltdb. 950* [Storm](https://github.com/asdine/storm) - Simple and powerful ORM for BoltDB. 951* [SimpleBolt](https://github.com/xyproto/simplebolt) - A simple way to use BoltDB. Deals mainly with strings. 952* [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics. 953* [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects. 954* [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server. 955* [torrent](https://github.com/anacrolix/torrent) - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development. 956* [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday. 957 958If you are using Bolt in a project please send a pull request to add it to the list. 959