1bbolt 2===== 3 4[![Go Report Card](https://goreportcard.com/badge/github.com/etcd-io/bbolt?style=flat-square)](https://goreportcard.com/report/github.com/etcd-io/bbolt) 5[![Coverage](https://codecov.io/gh/etcd-io/bbolt/branch/master/graph/badge.svg)](https://codecov.io/gh/etcd-io/bbolt) 6[![Build Status Travis](https://img.shields.io/travis/etcd-io/bboltlabs.svg?style=flat-square&&branch=master)](https://travis-ci.com/etcd-io/bbolt) 7[![Godoc](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)](https://godoc.org/github.com/etcd-io/bbolt) 8[![Releases](https://img.shields.io/github/release/etcd-io/bbolt/all.svg?style=flat-square)](https://github.com/etcd-io/bbolt/releases) 9[![LICENSE](https://img.shields.io/github/license/etcd-io/bbolt.svg?style=flat-square)](https://github.com/etcd-io/bbolt/blob/master/LICENSE) 10 11bbolt is a fork of [Ben Johnson's][gh_ben] [Bolt][bolt] key/value 12store. The purpose of this fork is to provide the Go community with an active 13maintenance and development target for Bolt; the goal is improved reliability 14and stability. bbolt includes bug fixes, performance enhancements, and features 15not found in Bolt while preserving backwards compatibility with the Bolt API. 16 17Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas] 18[LMDB project][lmdb]. The goal of the project is to provide a simple, 19fast, and reliable database for projects that don't require a full database 20server such as Postgres or MySQL. 21 22Since Bolt is meant to be used as such a low-level piece of functionality, 23simplicity is key. The API will be small and only focus on getting values 24and setting values. That's it. 25 26[gh_ben]: https://github.com/benbjohnson 27[bolt]: https://github.com/boltdb/bolt 28[hyc_symas]: https://twitter.com/hyc_symas 29[lmdb]: http://symas.com/mdb/ 30 31## Project Status 32 33Bolt is stable, the API is fixed, and the file format is fixed. Full unit 34test coverage and randomized black box testing are used to ensure database 35consistency and thread safety. Bolt is currently used in high-load production 36environments serving databases as large as 1TB. Many companies such as 37Shopify and Heroku use Bolt-backed services every day. 38 39## Project versioning 40 41bbolt uses [semantic versioning](http://semver.org). 42API should not change between patch and minor releases. 43New minor versions may add additional features to the API. 44 45## Table of Contents 46 47 - [Getting Started](#getting-started) 48 - [Installing](#installing) 49 - [Opening a database](#opening-a-database) 50 - [Transactions](#transactions) 51 - [Read-write transactions](#read-write-transactions) 52 - [Read-only transactions](#read-only-transactions) 53 - [Batch read-write transactions](#batch-read-write-transactions) 54 - [Managing transactions manually](#managing-transactions-manually) 55 - [Using buckets](#using-buckets) 56 - [Using key/value pairs](#using-keyvalue-pairs) 57 - [Autoincrementing integer for the bucket](#autoincrementing-integer-for-the-bucket) 58 - [Iterating over keys](#iterating-over-keys) 59 - [Prefix scans](#prefix-scans) 60 - [Range scans](#range-scans) 61 - [ForEach()](#foreach) 62 - [Nested buckets](#nested-buckets) 63 - [Database backups](#database-backups) 64 - [Statistics](#statistics) 65 - [Read-Only Mode](#read-only-mode) 66 - [Mobile Use (iOS/Android)](#mobile-use-iosandroid) 67 - [Resources](#resources) 68 - [Comparison with other databases](#comparison-with-other-databases) 69 - [Postgres, MySQL, & other relational databases](#postgres-mysql--other-relational-databases) 70 - [LevelDB, RocksDB](#leveldb-rocksdb) 71 - [LMDB](#lmdb) 72 - [Caveats & Limitations](#caveats--limitations) 73 - [Reading the Source](#reading-the-source) 74 - [Other Projects Using Bolt](#other-projects-using-bolt) 75 76## Getting Started 77 78### Installing 79 80To start using Bolt, install Go and run `go get`: 81 82```sh 83$ go get go.etcd.io/bbolt/... 84``` 85 86This will retrieve the library and install the `bolt` command line utility into 87your `$GOBIN` path. 88 89 90### Importing bbolt 91 92To use bbolt as an embedded key-value store, import as: 93 94```go 95import bolt "go.etcd.io/bbolt" 96 97db, err := bolt.Open(path, 0666, nil) 98if err != nil { 99 return err 100} 101defer db.Close() 102``` 103 104 105### Opening a database 106 107The top-level object in Bolt is a `DB`. It is represented as a single file on 108your disk and represents a consistent snapshot of your data. 109 110To open your database, simply use the `bolt.Open()` function: 111 112```go 113package main 114 115import ( 116 "log" 117 118 bolt "go.etcd.io/bbolt" 119) 120 121func main() { 122 // Open the my.db data file in your current directory. 123 // It will be created if it doesn't exist. 124 db, err := bolt.Open("my.db", 0600, nil) 125 if err != nil { 126 log.Fatal(err) 127 } 128 defer db.Close() 129 130 ... 131} 132``` 133 134Please note that Bolt obtains a file lock on the data file so multiple processes 135cannot open the same database at the same time. Opening an already open Bolt 136database will cause it to hang until the other process closes it. To prevent 137an indefinite wait you can pass a timeout option to the `Open()` function: 138 139```go 140db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second}) 141``` 142 143 144### Transactions 145 146Bolt allows only one read-write transaction at a time but allows as many 147read-only transactions as you want at a time. Each transaction has a consistent 148view of the data as it existed when the transaction started. 149 150Individual transactions and all objects created from them (e.g. buckets, keys) 151are not thread safe. To work with data in multiple goroutines you must start 152a transaction for each one or use locking to ensure only one goroutine accesses 153a transaction at a time. Creating transaction from the `DB` is thread safe. 154 155Read-only transactions and read-write transactions should not depend on one 156another and generally shouldn't be opened simultaneously in the same goroutine. 157This can cause a deadlock as the read-write transaction needs to periodically 158re-map the data file but it cannot do so while a read-only transaction is open. 159 160 161#### Read-write transactions 162 163To start a read-write transaction, you can use the `DB.Update()` function: 164 165```go 166err := db.Update(func(tx *bolt.Tx) error { 167 ... 168 return nil 169}) 170``` 171 172Inside the closure, you have a consistent view of the database. You commit the 173transaction by returning `nil` at the end. You can also rollback the transaction 174at any point by returning an error. All database operations are allowed inside 175a read-write transaction. 176 177Always check the return error as it will report any disk failures that can cause 178your transaction to not complete. If you return an error within your closure 179it will be passed through. 180 181 182#### Read-only transactions 183 184To start a read-only transaction, you can use the `DB.View()` function: 185 186```go 187err := db.View(func(tx *bolt.Tx) error { 188 ... 189 return nil 190}) 191``` 192 193You also get a consistent view of the database within this closure, however, 194no mutating operations are allowed within a read-only transaction. You can only 195retrieve buckets, retrieve values, and copy the database within a read-only 196transaction. 197 198 199#### Batch read-write transactions 200 201Each `DB.Update()` waits for disk to commit the writes. This overhead 202can be minimized by combining multiple updates with the `DB.Batch()` 203function: 204 205```go 206err := db.Batch(func(tx *bolt.Tx) error { 207 ... 208 return nil 209}) 210``` 211 212Concurrent Batch calls are opportunistically combined into larger 213transactions. Batch is only useful when there are multiple goroutines 214calling it. 215 216The trade-off is that `Batch` can call the given 217function multiple times, if parts of the transaction fail. The 218function must be idempotent and side effects must take effect only 219after a successful return from `DB.Batch()`. 220 221For example: don't display messages from inside the function, instead 222set variables in the enclosing scope: 223 224```go 225var id uint64 226err := db.Batch(func(tx *bolt.Tx) error { 227 // Find last key in bucket, decode as bigendian uint64, increment 228 // by one, encode back to []byte, and add new key. 229 ... 230 id = newValue 231 return nil 232}) 233if err != nil { 234 return ... 235} 236fmt.Println("Allocated ID %d", id) 237``` 238 239 240#### Managing transactions manually 241 242The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()` 243function. These helper functions will start the transaction, execute a function, 244and then safely close your transaction if an error is returned. This is the 245recommended way to use Bolt transactions. 246 247However, sometimes you may want to manually start and end your transactions. 248You can use the `DB.Begin()` function directly but **please** be sure to close 249the transaction. 250 251```go 252// Start a writable transaction. 253tx, err := db.Begin(true) 254if err != nil { 255 return err 256} 257defer tx.Rollback() 258 259// Use the transaction... 260_, err := tx.CreateBucket([]byte("MyBucket")) 261if err != nil { 262 return err 263} 264 265// Commit the transaction and check for error. 266if err := tx.Commit(); err != nil { 267 return err 268} 269``` 270 271The first argument to `DB.Begin()` is a boolean stating if the transaction 272should be writable. 273 274 275### Using buckets 276 277Buckets are collections of key/value pairs within the database. All keys in a 278bucket must be unique. You can create a bucket using the `DB.CreateBucket()` 279function: 280 281```go 282db.Update(func(tx *bolt.Tx) error { 283 b, err := tx.CreateBucket([]byte("MyBucket")) 284 if err != nil { 285 return fmt.Errorf("create bucket: %s", err) 286 } 287 return nil 288}) 289``` 290 291You can also create a bucket only if it doesn't exist by using the 292`Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this 293function for all your top-level buckets after you open your database so you can 294guarantee that they exist for future transactions. 295 296To delete a bucket, simply call the `Tx.DeleteBucket()` function. 297 298 299### Using key/value pairs 300 301To save a key/value pair to a bucket, use the `Bucket.Put()` function: 302 303```go 304db.Update(func(tx *bolt.Tx) error { 305 b := tx.Bucket([]byte("MyBucket")) 306 err := b.Put([]byte("answer"), []byte("42")) 307 return err 308}) 309``` 310 311This will set the value of the `"answer"` key to `"42"` in the `MyBucket` 312bucket. To retrieve this value, we can use the `Bucket.Get()` function: 313 314```go 315db.View(func(tx *bolt.Tx) error { 316 b := tx.Bucket([]byte("MyBucket")) 317 v := b.Get([]byte("answer")) 318 fmt.Printf("The answer is: %s\n", v) 319 return nil 320}) 321``` 322 323The `Get()` function does not return an error because its operation is 324guaranteed to work (unless there is some kind of system failure). If the key 325exists then it will return its byte slice value. If it doesn't exist then it 326will return `nil`. It's important to note that you can have a zero-length value 327set to a key which is different than the key not existing. 328 329Use the `Bucket.Delete()` function to delete a key from the bucket. 330 331Please note that values returned from `Get()` are only valid while the 332transaction is open. If you need to use a value outside of the transaction 333then you must use `copy()` to copy it to another byte slice. 334 335 336### Autoincrementing integer for the bucket 337By using the `NextSequence()` function, you can let Bolt determine a sequence 338which can be used as the unique identifier for your key/value pairs. See the 339example below. 340 341```go 342// CreateUser saves u to the store. The new user ID is set on u once the data is persisted. 343func (s *Store) CreateUser(u *User) error { 344 return s.db.Update(func(tx *bolt.Tx) error { 345 // Retrieve the users bucket. 346 // This should be created when the DB is first opened. 347 b := tx.Bucket([]byte("users")) 348 349 // Generate ID for the user. 350 // This returns an error only if the Tx is closed or not writeable. 351 // That can't happen in an Update() call so I ignore the error check. 352 id, _ := b.NextSequence() 353 u.ID = int(id) 354 355 // Marshal user data into bytes. 356 buf, err := json.Marshal(u) 357 if err != nil { 358 return err 359 } 360 361 // Persist bytes to users bucket. 362 return b.Put(itob(u.ID), buf) 363 }) 364} 365 366// itob returns an 8-byte big endian representation of v. 367func itob(v int) []byte { 368 b := make([]byte, 8) 369 binary.BigEndian.PutUint64(b, uint64(v)) 370 return b 371} 372 373type User struct { 374 ID int 375 ... 376} 377``` 378 379### Iterating over keys 380 381Bolt stores its keys in byte-sorted order within a bucket. This makes sequential 382iteration over these keys extremely fast. To iterate over keys we'll use a 383`Cursor`: 384 385```go 386db.View(func(tx *bolt.Tx) error { 387 // Assume bucket exists and has keys 388 b := tx.Bucket([]byte("MyBucket")) 389 390 c := b.Cursor() 391 392 for k, v := c.First(); k != nil; k, v = c.Next() { 393 fmt.Printf("key=%s, value=%s\n", k, v) 394 } 395 396 return nil 397}) 398``` 399 400The cursor allows you to move to a specific point in the list of keys and move 401forward or backward through the keys one at a time. 402 403The following functions are available on the cursor: 404 405``` 406First() Move to the first key. 407Last() Move to the last key. 408Seek() Move to a specific key. 409Next() Move to the next key. 410Prev() Move to the previous key. 411``` 412 413Each of those functions has a return signature of `(key []byte, value []byte)`. 414When you have iterated to the end of the cursor then `Next()` will return a 415`nil` key. You must seek to a position using `First()`, `Last()`, or `Seek()` 416before calling `Next()` or `Prev()`. If you do not seek to a position then 417these functions will return a `nil` key. 418 419During iteration, if the key is non-`nil` but the value is `nil`, that means 420the key refers to a bucket rather than a value. Use `Bucket.Bucket()` to 421access the sub-bucket. 422 423 424#### Prefix scans 425 426To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`: 427 428```go 429db.View(func(tx *bolt.Tx) error { 430 // Assume bucket exists and has keys 431 c := tx.Bucket([]byte("MyBucket")).Cursor() 432 433 prefix := []byte("1234") 434 for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() { 435 fmt.Printf("key=%s, value=%s\n", k, v) 436 } 437 438 return nil 439}) 440``` 441 442#### Range scans 443 444Another common use case is scanning over a range such as a time range. If you 445use a sortable time encoding such as RFC3339 then you can query a specific 446date range like this: 447 448```go 449db.View(func(tx *bolt.Tx) error { 450 // Assume our events bucket exists and has RFC3339 encoded time keys. 451 c := tx.Bucket([]byte("Events")).Cursor() 452 453 // Our time range spans the 90's decade. 454 min := []byte("1990-01-01T00:00:00Z") 455 max := []byte("2000-01-01T00:00:00Z") 456 457 // Iterate over the 90's. 458 for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() { 459 fmt.Printf("%s: %s\n", k, v) 460 } 461 462 return nil 463}) 464``` 465 466Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable. 467 468 469#### ForEach() 470 471You can also use the function `ForEach()` if you know you'll be iterating over 472all the keys in a bucket: 473 474```go 475db.View(func(tx *bolt.Tx) error { 476 // Assume bucket exists and has keys 477 b := tx.Bucket([]byte("MyBucket")) 478 479 b.ForEach(func(k, v []byte) error { 480 fmt.Printf("key=%s, value=%s\n", k, v) 481 return nil 482 }) 483 return nil 484}) 485``` 486 487Please note that keys and values in `ForEach()` are only valid while 488the transaction is open. If you need to use a key or value outside of 489the transaction, you must use `copy()` to copy it to another byte 490slice. 491 492### Nested buckets 493 494You can also store a bucket in a key to create nested buckets. The API is the 495same as the bucket management API on the `DB` object: 496 497```go 498func (*Bucket) CreateBucket(key []byte) (*Bucket, error) 499func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error) 500func (*Bucket) DeleteBucket(key []byte) error 501``` 502 503Say you had a multi-tenant application where the root level bucket was the account bucket. Inside of this bucket was a sequence of accounts which themselves are buckets. And inside the sequence bucket you could have many buckets pertaining to the Account itself (Users, Notes, etc) isolating the information into logical groupings. 504 505```go 506 507// createUser creates a new user in the given account. 508func createUser(accountID int, u *User) error { 509 // Start the transaction. 510 tx, err := db.Begin(true) 511 if err != nil { 512 return err 513 } 514 defer tx.Rollback() 515 516 // Retrieve the root bucket for the account. 517 // Assume this has already been created when the account was set up. 518 root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10))) 519 520 // Setup the users bucket. 521 bkt, err := root.CreateBucketIfNotExists([]byte("USERS")) 522 if err != nil { 523 return err 524 } 525 526 // Generate an ID for the new user. 527 userID, err := bkt.NextSequence() 528 if err != nil { 529 return err 530 } 531 u.ID = userID 532 533 // Marshal and save the encoded user. 534 if buf, err := json.Marshal(u); err != nil { 535 return err 536 } else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil { 537 return err 538 } 539 540 // Commit the transaction. 541 if err := tx.Commit(); err != nil { 542 return err 543 } 544 545 return nil 546} 547 548``` 549 550 551 552 553### Database backups 554 555Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()` 556function to write a consistent view of the database to a writer. If you call 557this from a read-only transaction, it will perform a hot backup and not block 558your other database reads and writes. 559 560By default, it will use a regular file handle which will utilize the operating 561system's page cache. See the [`Tx`](https://godoc.org/go.etcd.io/bbolt#Tx) 562documentation for information about optimizing for larger-than-RAM datasets. 563 564One common use case is to backup over HTTP so you can use tools like `cURL` to 565do database backups: 566 567```go 568func BackupHandleFunc(w http.ResponseWriter, req *http.Request) { 569 err := db.View(func(tx *bolt.Tx) error { 570 w.Header().Set("Content-Type", "application/octet-stream") 571 w.Header().Set("Content-Disposition", `attachment; filename="my.db"`) 572 w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size()))) 573 _, err := tx.WriteTo(w) 574 return err 575 }) 576 if err != nil { 577 http.Error(w, err.Error(), http.StatusInternalServerError) 578 } 579} 580``` 581 582Then you can backup using this command: 583 584```sh 585$ curl http://localhost/backup > my.db 586``` 587 588Or you can open your browser to `http://localhost/backup` and it will download 589automatically. 590 591If you want to backup to another file you can use the `Tx.CopyFile()` helper 592function. 593 594 595### Statistics 596 597The database keeps a running count of many of the internal operations it 598performs so you can better understand what's going on. By grabbing a snapshot 599of these stats at two points in time we can see what operations were performed 600in that time range. 601 602For example, we could start a goroutine to log stats every 10 seconds: 603 604```go 605go func() { 606 // Grab the initial stats. 607 prev := db.Stats() 608 609 for { 610 // Wait for 10s. 611 time.Sleep(10 * time.Second) 612 613 // Grab the current stats and diff them. 614 stats := db.Stats() 615 diff := stats.Sub(&prev) 616 617 // Encode stats to JSON and print to STDERR. 618 json.NewEncoder(os.Stderr).Encode(diff) 619 620 // Save stats for the next loop. 621 prev = stats 622 } 623}() 624``` 625 626It's also useful to pipe these stats to a service such as statsd for monitoring 627or to provide an HTTP endpoint that will perform a fixed-length sample. 628 629 630### Read-Only Mode 631 632Sometimes it is useful to create a shared, read-only Bolt database. To this, 633set the `Options.ReadOnly` flag when opening your database. Read-only mode 634uses a shared lock to allow multiple processes to read from the database but 635it will block any processes from opening the database in read-write mode. 636 637```go 638db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true}) 639if err != nil { 640 log.Fatal(err) 641} 642``` 643 644### Mobile Use (iOS/Android) 645 646Bolt is able to run on mobile devices by leveraging the binding feature of the 647[gomobile](https://github.com/golang/mobile) tool. Create a struct that will 648contain your database logic and a reference to a `*bolt.DB` with a initializing 649constructor that takes in a filepath where the database file will be stored. 650Neither Android nor iOS require extra permissions or cleanup from using this method. 651 652```go 653func NewBoltDB(filepath string) *BoltDB { 654 db, err := bolt.Open(filepath+"/demo.db", 0600, nil) 655 if err != nil { 656 log.Fatal(err) 657 } 658 659 return &BoltDB{db} 660} 661 662type BoltDB struct { 663 db *bolt.DB 664 ... 665} 666 667func (b *BoltDB) Path() string { 668 return b.db.Path() 669} 670 671func (b *BoltDB) Close() { 672 b.db.Close() 673} 674``` 675 676Database logic should be defined as methods on this wrapper struct. 677 678To initialize this struct from the native language (both platforms now sync 679their local storage to the cloud. These snippets disable that functionality for the 680database file): 681 682#### Android 683 684```java 685String path; 686if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){ 687 path = getNoBackupFilesDir().getAbsolutePath(); 688} else{ 689 path = getFilesDir().getAbsolutePath(); 690} 691Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path) 692``` 693 694#### iOS 695 696```objc 697- (void)demo { 698 NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory, 699 NSUserDomainMask, 700 YES) objectAtIndex:0]; 701 GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path); 702 [self addSkipBackupAttributeToItemAtPath:demo.path]; 703 //Some DB Logic would go here 704 [demo close]; 705} 706 707- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString 708{ 709 NSURL* URL= [NSURL fileURLWithPath: filePathString]; 710 assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]); 711 712 NSError *error = nil; 713 BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES] 714 forKey: NSURLIsExcludedFromBackupKey error: &error]; 715 if(!success){ 716 NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error); 717 } 718 return success; 719} 720 721``` 722 723## Resources 724 725For more information on getting started with Bolt, check out the following articles: 726 727* [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch). 728* [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville 729 730 731## Comparison with other databases 732 733### Postgres, MySQL, & other relational databases 734 735Relational databases structure data into rows and are only accessible through 736the use of SQL. This approach provides flexibility in how you store and query 737your data but also incurs overhead in parsing and planning SQL statements. Bolt 738accesses all data by a byte slice key. This makes Bolt fast to read and write 739data by key but provides no built-in support for joining values together. 740 741Most relational databases (with the exception of SQLite) are standalone servers 742that run separately from your application. This gives your systems 743flexibility to connect multiple application servers to a single database 744server but also adds overhead in serializing and transporting data over the 745network. Bolt runs as a library included in your application so all data access 746has to go through your application's process. This brings data closer to your 747application but limits multi-process access to the data. 748 749 750### LevelDB, RocksDB 751 752LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that 753they are libraries bundled into the application, however, their underlying 754structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes 755random writes by using a write ahead log and multi-tiered, sorted files called 756SSTables. Bolt uses a B+tree internally and only a single file. Both approaches 757have trade-offs. 758 759If you require a high random write throughput (>10,000 w/sec) or you need to use 760spinning disks then LevelDB could be a good choice. If your application is 761read-heavy or does a lot of range scans then Bolt could be a good choice. 762 763One other important consideration is that LevelDB does not have transactions. 764It supports batch writing of key/values pairs and it supports read snapshots 765but it will not give you the ability to do a compare-and-swap operation safely. 766Bolt supports fully serializable ACID transactions. 767 768 769### LMDB 770 771Bolt was originally a port of LMDB so it is architecturally similar. Both use 772a B+tree, have ACID semantics with fully serializable transactions, and support 773lock-free MVCC using a single writer and multiple readers. 774 775The two projects have somewhat diverged. LMDB heavily focuses on raw performance 776while Bolt has focused on simplicity and ease of use. For example, LMDB allows 777several unsafe actions such as direct writes for the sake of performance. Bolt 778opts to disallow actions which can leave the database in a corrupted state. The 779only exception to this in Bolt is `DB.NoSync`. 780 781There are also a few differences in API. LMDB requires a maximum mmap size when 782opening an `mdb_env` whereas Bolt will handle incremental mmap resizing 783automatically. LMDB overloads the getter and setter functions with multiple 784flags whereas Bolt splits these specialized cases into their own functions. 785 786 787## Caveats & Limitations 788 789It's important to pick the right tool for the job and Bolt is no exception. 790Here are a few things to note when evaluating and using Bolt: 791 792* Bolt is good for read intensive workloads. Sequential write performance is 793 also fast but random writes can be slow. You can use `DB.Batch()` or add a 794 write-ahead log to help mitigate this issue. 795 796* Bolt uses a B+tree internally so there can be a lot of random page access. 797 SSDs provide a significant performance boost over spinning disks. 798 799* Try to avoid long running read transactions. Bolt uses copy-on-write so 800 old pages cannot be reclaimed while an old transaction is using them. 801 802* Byte slices returned from Bolt are only valid during a transaction. Once the 803 transaction has been committed or rolled back then the memory they point to 804 can be reused by a new page or can be unmapped from virtual memory and you'll 805 see an `unexpected fault address` panic when accessing it. 806 807* Bolt uses an exclusive write lock on the database file so it cannot be 808 shared by multiple processes. 809 810* Be careful when using `Bucket.FillPercent`. Setting a high fill percent for 811 buckets that have random inserts will cause your database to have very poor 812 page utilization. 813 814* Use larger buckets in general. Smaller buckets causes poor page utilization 815 once they become larger than the page size (typically 4KB). 816 817* Bulk loading a lot of random writes into a new bucket can be slow as the 818 page will not split until the transaction is committed. Randomly inserting 819 more than 100,000 key/value pairs into a single new bucket in a single 820 transaction is not advised. 821 822* Bolt uses a memory-mapped file so the underlying operating system handles the 823 caching of the data. Typically, the OS will cache as much of the file as it 824 can in memory and will release memory as needed to other processes. This means 825 that Bolt can show very high memory usage when working with large databases. 826 However, this is expected and the OS will release memory as needed. Bolt can 827 handle databases much larger than the available physical RAM, provided its 828 memory-map fits in the process virtual address space. It may be problematic 829 on 32-bits systems. 830 831* The data structures in the Bolt database are memory mapped so the data file 832 will be endian specific. This means that you cannot copy a Bolt file from a 833 little endian machine to a big endian machine and have it work. For most 834 users this is not a concern since most modern CPUs are little endian. 835 836* Because of the way pages are laid out on disk, Bolt cannot truncate data files 837 and return free pages back to the disk. Instead, Bolt maintains a free list 838 of unused pages within its data file. These free pages can be reused by later 839 transactions. This works well for many use cases as databases generally tend 840 to grow. However, it's important to note that deleting large chunks of data 841 will not allow you to reclaim that space on disk. 842 843 For more information on page allocation, [see this comment][page-allocation]. 844 845[page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638 846 847 848## Reading the Source 849 850Bolt is a relatively small code base (<5KLOC) for an embedded, serializable, 851transactional key/value database so it can be a good starting point for people 852interested in how databases work. 853 854The best places to start are the main entry points into Bolt: 855 856- `Open()` - Initializes the reference to the database. It's responsible for 857 creating the database if it doesn't exist, obtaining an exclusive lock on the 858 file, reading the meta pages, & memory-mapping the file. 859 860- `DB.Begin()` - Starts a read-only or read-write transaction depending on the 861 value of the `writable` argument. This requires briefly obtaining the "meta" 862 lock to keep track of open transactions. Only one read-write transaction can 863 exist at a time so the "rwlock" is acquired during the life of a read-write 864 transaction. 865 866- `Bucket.Put()` - Writes a key/value pair into a bucket. After validating the 867 arguments, a cursor is used to traverse the B+tree to the page and position 868 where they key & value will be written. Once the position is found, the bucket 869 materializes the underlying page and the page's parent pages into memory as 870 "nodes". These nodes are where mutations occur during read-write transactions. 871 These changes get flushed to disk during commit. 872 873- `Bucket.Get()` - Retrieves a key/value pair from a bucket. This uses a cursor 874 to move to the page & position of a key/value pair. During a read-only 875 transaction, the key and value data is returned as a direct reference to the 876 underlying mmap file so there's no allocation overhead. For read-write 877 transactions, this data may reference the mmap file or one of the in-memory 878 node values. 879 880- `Cursor` - This object is simply for traversing the B+tree of on-disk pages 881 or in-memory nodes. It can seek to a specific key, move to the first or last 882 value, or it can move forward or backward. The cursor handles the movement up 883 and down the B+tree transparently to the end user. 884 885- `Tx.Commit()` - Converts the in-memory dirty nodes and the list of free pages 886 into pages to be written to disk. Writing to disk then occurs in two phases. 887 First, the dirty pages are written to disk and an `fsync()` occurs. Second, a 888 new meta page with an incremented transaction ID is written and another 889 `fsync()` occurs. This two phase write ensures that partially written data 890 pages are ignored in the event of a crash since the meta page pointing to them 891 is never written. Partially written meta pages are invalidated because they 892 are written with a checksum. 893 894If you have additional notes that could be helpful for others, please submit 895them via pull request. 896 897 898## Other Projects Using Bolt 899 900Below is a list of public, open source projects that use Bolt: 901 902* [Algernon](https://github.com/xyproto/algernon) - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend. 903* [Bazil](https://bazil.org/) - A file system that lets your data reside where it is most convenient for it to reside. 904* [bolter](https://github.com/hasit/bolter) - Command-line app for viewing BoltDB file in your terminal. 905* [boltcli](https://github.com/spacewander/boltcli) - the redis-cli for boltdb with Lua script support. 906* [BoltHold](https://github.com/timshannon/bolthold) - An embeddable NoSQL store for Go types built on BoltDB 907* [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt. 908* [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners. 909* [BoltDbWeb](https://github.com/evnix/boltdbweb) - A web based GUI for BoltDB files. 910* [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend. 911* [btcwallet](https://github.com/btcsuite/btcwallet) - A bitcoin wallet. 912* [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining 913 simple tx and key scans. 914* [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend. 915* [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations. 916* [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware. 917* [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb. 918* [dcrwallet](https://github.com/decred/dcrwallet) - A wallet for the Decred cryptocurrency. 919* [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems. 920* [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka. 921* [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data. 922* [Go Report Card](https://goreportcard.com/) - Go code quality report cards as a (free and open source) service. 923* [GoWebApp](https://github.com/josephspurrier/gowebapp) - A basic MVC web application in Go using BoltDB. 924* [GoShort](https://github.com/pankajkhairnar/goShort) - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter. 925* [gopherpit](https://github.com/gopherpit/gopherpit) - A web service to manage Go remote import paths with custom domains 926* [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin". 927* [InfluxDB](https://influxdata.com) - Scalable datastore for metrics, events, and real-time analytics. 928* [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters. 929* [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed. 930* [Ironsmith](https://github.com/timshannon/ironsmith) - A simple, script-driven continuous integration (build - > test -> release) tool, with no external dependencies 931* [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs. 932* [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage. 933* [lru](https://github.com/crowdriff/lru) - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores. 934* [mbuckets](https://github.com/abhigupta912/mbuckets) - A Bolt wrapper that allows easy operations on multi level (nested) buckets. 935* [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite. 936* [MuLiFS](https://github.com/dankomiocevic/mulifs) - Music Library Filesystem creates a filesystem to organise your music files. 937* [Operation Go: A Routine Mission](http://gocode.io) - An online programming game for Golang using Bolt for user accounts and a leaderboard. 938* [photosite/session](https://godoc.org/bitbucket.org/kardianos/photosite/session) - Sessions for a photo viewing site. 939* [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system. 940* [reef-pi](https://github.com/reef-pi/reef-pi) - reef-pi is an award winning, modular, DIY reef tank controller using easy to learn electronics based on a Raspberry Pi. 941* [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service 942* [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read. 943* [stow](https://github.com/djherbis/stow) - a persistence manager for objects 944 backed by boltdb. 945* [Storm](https://github.com/asdine/storm) - Simple and powerful ORM for BoltDB. 946* [SimpleBolt](https://github.com/xyproto/simplebolt) - A simple way to use BoltDB. Deals mainly with strings. 947* [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics. 948* [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects. 949* [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server. 950* [torrent](https://github.com/anacrolix/torrent) - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development. 951* [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday. 952 953If you are using Bolt in a project please send a pull request to add it to the list. 954