1Bolt [![Coverage Status](https://coveralls.io/repos/boltdb/bolt/badge.svg?branch=master)](https://coveralls.io/r/boltdb/bolt?branch=master) [![GoDoc](https://godoc.org/github.com/boltdb/bolt?status.svg)](https://godoc.org/github.com/boltdb/bolt) ![Version](https://img.shields.io/badge/version-1.2.1-green.svg) 2==== 3 4Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas] 5[LMDB project][lmdb]. The goal of the project is to provide a simple, 6fast, and reliable database for projects that don't require a full database 7server such as Postgres or MySQL. 8 9Since Bolt is meant to be used as such a low-level piece of functionality, 10simplicity is key. The API will be small and only focus on getting values 11and setting values. That's it. 12 13[hyc_symas]: https://twitter.com/hyc_symas 14[lmdb]: http://symas.com/mdb/ 15 16## Project Status 17 18Bolt is stable, the API is fixed, and the file format is fixed. Full unit 19test coverage and randomized black box testing are used to ensure database 20consistency and thread safety. Bolt is currently in high-load production 21environments serving databases as large as 1TB. Many companies such as 22Shopify and Heroku use Bolt-backed services every day. 23 24## Table of Contents 25 26- [Getting Started](#getting-started) 27 - [Installing](#installing) 28 - [Opening a database](#opening-a-database) 29 - [Transactions](#transactions) 30 - [Read-write transactions](#read-write-transactions) 31 - [Read-only transactions](#read-only-transactions) 32 - [Batch read-write transactions](#batch-read-write-transactions) 33 - [Managing transactions manually](#managing-transactions-manually) 34 - [Using buckets](#using-buckets) 35 - [Using key/value pairs](#using-keyvalue-pairs) 36 - [Autoincrementing integer for the bucket](#autoincrementing-integer-for-the-bucket) 37 - [Iterating over keys](#iterating-over-keys) 38 - [Prefix scans](#prefix-scans) 39 - [Range scans](#range-scans) 40 - [ForEach()](#foreach) 41 - [Nested buckets](#nested-buckets) 42 - [Database backups](#database-backups) 43 - [Statistics](#statistics) 44 - [Read-Only Mode](#read-only-mode) 45 - [Mobile Use (iOS/Android)](#mobile-use-iosandroid) 46- [Resources](#resources) 47- [Comparison with other databases](#comparison-with-other-databases) 48 - [Postgres, MySQL, & other relational databases](#postgres-mysql--other-relational-databases) 49 - [LevelDB, RocksDB](#leveldb-rocksdb) 50 - [LMDB](#lmdb) 51- [Caveats & Limitations](#caveats--limitations) 52- [Reading the Source](#reading-the-source) 53- [Other Projects Using Bolt](#other-projects-using-bolt) 54 55## Getting Started 56 57### Installing 58 59To start using Bolt, install Go and run `go get`: 60 61```sh 62$ go get github.com/boltdb/bolt/... 63``` 64 65This will retrieve the library and install the `bolt` command line utility into 66your `$GOBIN` path. 67 68 69### Opening a database 70 71The top-level object in Bolt is a `DB`. It is represented as a single file on 72your disk and represents a consistent snapshot of your data. 73 74To open your database, simply use the `bolt.Open()` function: 75 76```go 77package main 78 79import ( 80 "log" 81 82 "github.com/boltdb/bolt" 83) 84 85func main() { 86 // Open the my.db data file in your current directory. 87 // It will be created if it doesn't exist. 88 db, err := bolt.Open("my.db", 0600, nil) 89 if err != nil { 90 log.Fatal(err) 91 } 92 defer db.Close() 93 94 ... 95} 96``` 97 98Please note that Bolt obtains a file lock on the data file so multiple processes 99cannot open the same database at the same time. Opening an already open Bolt 100database will cause it to hang until the other process closes it. To prevent 101an indefinite wait you can pass a timeout option to the `Open()` function: 102 103```go 104db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second}) 105``` 106 107 108### Transactions 109 110Bolt allows only one read-write transaction at a time but allows as many 111read-only transactions as you want at a time. Each transaction has a consistent 112view of the data as it existed when the transaction started. 113 114Individual transactions and all objects created from them (e.g. buckets, keys) 115are not thread safe. To work with data in multiple goroutines you must start 116a transaction for each one or use locking to ensure only one goroutine accesses 117a transaction at a time. Creating transaction from the `DB` is thread safe. 118 119Read-only transactions and read-write transactions should not depend on one 120another and generally shouldn't be opened simultaneously in the same goroutine. 121This can cause a deadlock as the read-write transaction needs to periodically 122re-map the data file but it cannot do so while a read-only transaction is open. 123 124 125#### Read-write transactions 126 127To start a read-write transaction, you can use the `DB.Update()` function: 128 129```go 130err := db.Update(func(tx *bolt.Tx) error { 131 ... 132 return nil 133}) 134``` 135 136Inside the closure, you have a consistent view of the database. You commit the 137transaction by returning `nil` at the end. You can also rollback the transaction 138at any point by returning an error. All database operations are allowed inside 139a read-write transaction. 140 141Always check the return error as it will report any disk failures that can cause 142your transaction to not complete. If you return an error within your closure 143it will be passed through. 144 145 146#### Read-only transactions 147 148To start a read-only transaction, you can use the `DB.View()` function: 149 150```go 151err := db.View(func(tx *bolt.Tx) error { 152 ... 153 return nil 154}) 155``` 156 157You also get a consistent view of the database within this closure, however, 158no mutating operations are allowed within a read-only transaction. You can only 159retrieve buckets, retrieve values, and copy the database within a read-only 160transaction. 161 162 163#### Batch read-write transactions 164 165Each `DB.Update()` waits for disk to commit the writes. This overhead 166can be minimized by combining multiple updates with the `DB.Batch()` 167function: 168 169```go 170err := db.Batch(func(tx *bolt.Tx) error { 171 ... 172 return nil 173}) 174``` 175 176Concurrent Batch calls are opportunistically combined into larger 177transactions. Batch is only useful when there are multiple goroutines 178calling it. 179 180The trade-off is that `Batch` can call the given 181function multiple times, if parts of the transaction fail. The 182function must be idempotent and side effects must take effect only 183after a successful return from `DB.Batch()`. 184 185For example: don't display messages from inside the function, instead 186set variables in the enclosing scope: 187 188```go 189var id uint64 190err := db.Batch(func(tx *bolt.Tx) error { 191 // Find last key in bucket, decode as bigendian uint64, increment 192 // by one, encode back to []byte, and add new key. 193 ... 194 id = newValue 195 return nil 196}) 197if err != nil { 198 return ... 199} 200fmt.Println("Allocated ID %d", id) 201``` 202 203 204#### Managing transactions manually 205 206The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()` 207function. These helper functions will start the transaction, execute a function, 208and then safely close your transaction if an error is returned. This is the 209recommended way to use Bolt transactions. 210 211However, sometimes you may want to manually start and end your transactions. 212You can use the `DB.Begin()` function directly but **please** be sure to close 213the transaction. 214 215```go 216// Start a writable transaction. 217tx, err := db.Begin(true) 218if err != nil { 219 return err 220} 221defer tx.Rollback() 222 223// Use the transaction... 224_, err := tx.CreateBucket([]byte("MyBucket")) 225if err != nil { 226 return err 227} 228 229// Commit the transaction and check for error. 230if err := tx.Commit(); err != nil { 231 return err 232} 233``` 234 235The first argument to `DB.Begin()` is a boolean stating if the transaction 236should be writable. 237 238 239### Using buckets 240 241Buckets are collections of key/value pairs within the database. All keys in a 242bucket must be unique. You can create a bucket using the `DB.CreateBucket()` 243function: 244 245```go 246db.Update(func(tx *bolt.Tx) error { 247 b, err := tx.CreateBucket([]byte("MyBucket")) 248 if err != nil { 249 return fmt.Errorf("create bucket: %s", err) 250 } 251 return nil 252}) 253``` 254 255You can also create a bucket only if it doesn't exist by using the 256`Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this 257function for all your top-level buckets after you open your database so you can 258guarantee that they exist for future transactions. 259 260To delete a bucket, simply call the `Tx.DeleteBucket()` function. 261 262 263### Using key/value pairs 264 265To save a key/value pair to a bucket, use the `Bucket.Put()` function: 266 267```go 268db.Update(func(tx *bolt.Tx) error { 269 b := tx.Bucket([]byte("MyBucket")) 270 err := b.Put([]byte("answer"), []byte("42")) 271 return err 272}) 273``` 274 275This will set the value of the `"answer"` key to `"42"` in the `MyBucket` 276bucket. To retrieve this value, we can use the `Bucket.Get()` function: 277 278```go 279db.View(func(tx *bolt.Tx) error { 280 b := tx.Bucket([]byte("MyBucket")) 281 v := b.Get([]byte("answer")) 282 fmt.Printf("The answer is: %s\n", v) 283 return nil 284}) 285``` 286 287The `Get()` function does not return an error because its operation is 288guaranteed to work (unless there is some kind of system failure). If the key 289exists then it will return its byte slice value. If it doesn't exist then it 290will return `nil`. It's important to note that you can have a zero-length value 291set to a key which is different than the key not existing. 292 293Use the `Bucket.Delete()` function to delete a key from the bucket. 294 295Please note that values returned from `Get()` are only valid while the 296transaction is open. If you need to use a value outside of the transaction 297then you must use `copy()` to copy it to another byte slice. 298 299 300### Autoincrementing integer for the bucket 301By using the `NextSequence()` function, you can let Bolt determine a sequence 302which can be used as the unique identifier for your key/value pairs. See the 303example below. 304 305```go 306// CreateUser saves u to the store. The new user ID is set on u once the data is persisted. 307func (s *Store) CreateUser(u *User) error { 308 return s.db.Update(func(tx *bolt.Tx) error { 309 // Retrieve the users bucket. 310 // This should be created when the DB is first opened. 311 b := tx.Bucket([]byte("users")) 312 313 // Generate ID for the user. 314 // This returns an error only if the Tx is closed or not writeable. 315 // That can't happen in an Update() call so I ignore the error check. 316 id, _ := b.NextSequence() 317 u.ID = int(id) 318 319 // Marshal user data into bytes. 320 buf, err := json.Marshal(u) 321 if err != nil { 322 return err 323 } 324 325 // Persist bytes to users bucket. 326 return b.Put(itob(u.ID), buf) 327 }) 328} 329 330// itob returns an 8-byte big endian representation of v. 331func itob(v int) []byte { 332 b := make([]byte, 8) 333 binary.BigEndian.PutUint64(b, uint64(v)) 334 return b 335} 336 337type User struct { 338 ID int 339 ... 340} 341``` 342 343### Iterating over keys 344 345Bolt stores its keys in byte-sorted order within a bucket. This makes sequential 346iteration over these keys extremely fast. To iterate over keys we'll use a 347`Cursor`: 348 349```go 350db.View(func(tx *bolt.Tx) error { 351 // Assume bucket exists and has keys 352 b := tx.Bucket([]byte("MyBucket")) 353 354 c := b.Cursor() 355 356 for k, v := c.First(); k != nil; k, v = c.Next() { 357 fmt.Printf("key=%s, value=%s\n", k, v) 358 } 359 360 return nil 361}) 362``` 363 364The cursor allows you to move to a specific point in the list of keys and move 365forward or backward through the keys one at a time. 366 367The following functions are available on the cursor: 368 369``` 370First() Move to the first key. 371Last() Move to the last key. 372Seek() Move to a specific key. 373Next() Move to the next key. 374Prev() Move to the previous key. 375``` 376 377Each of those functions has a return signature of `(key []byte, value []byte)`. 378When you have iterated to the end of the cursor then `Next()` will return a 379`nil` key. You must seek to a position using `First()`, `Last()`, or `Seek()` 380before calling `Next()` or `Prev()`. If you do not seek to a position then 381these functions will return a `nil` key. 382 383During iteration, if the key is non-`nil` but the value is `nil`, that means 384the key refers to a bucket rather than a value. Use `Bucket.Bucket()` to 385access the sub-bucket. 386 387 388#### Prefix scans 389 390To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`: 391 392```go 393db.View(func(tx *bolt.Tx) error { 394 // Assume bucket exists and has keys 395 c := tx.Bucket([]byte("MyBucket")).Cursor() 396 397 prefix := []byte("1234") 398 for k, v := c.Seek(prefix); bytes.HasPrefix(k, prefix); k, v = c.Next() { 399 fmt.Printf("key=%s, value=%s\n", k, v) 400 } 401 402 return nil 403}) 404``` 405 406#### Range scans 407 408Another common use case is scanning over a range such as a time range. If you 409use a sortable time encoding such as RFC3339 then you can query a specific 410date range like this: 411 412```go 413db.View(func(tx *bolt.Tx) error { 414 // Assume our events bucket exists and has RFC3339 encoded time keys. 415 c := tx.Bucket([]byte("Events")).Cursor() 416 417 // Our time range spans the 90's decade. 418 min := []byte("1990-01-01T00:00:00Z") 419 max := []byte("2000-01-01T00:00:00Z") 420 421 // Iterate over the 90's. 422 for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() { 423 fmt.Printf("%s: %s\n", k, v) 424 } 425 426 return nil 427}) 428``` 429 430Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable. 431 432 433#### ForEach() 434 435You can also use the function `ForEach()` if you know you'll be iterating over 436all the keys in a bucket: 437 438```go 439db.View(func(tx *bolt.Tx) error { 440 // Assume bucket exists and has keys 441 b := tx.Bucket([]byte("MyBucket")) 442 443 b.ForEach(func(k, v []byte) error { 444 fmt.Printf("key=%s, value=%s\n", k, v) 445 return nil 446 }) 447 return nil 448}) 449``` 450 451Please note that keys and values in `ForEach()` are only valid while 452the transaction is open. If you need to use a key or value outside of 453the transaction, you must use `copy()` to copy it to another byte 454slice. 455 456### Nested buckets 457 458You can also store a bucket in a key to create nested buckets. The API is the 459same as the bucket management API on the `DB` object: 460 461```go 462func (*Bucket) CreateBucket(key []byte) (*Bucket, error) 463func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error) 464func (*Bucket) DeleteBucket(key []byte) error 465``` 466 467 468### Database backups 469 470Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()` 471function to write a consistent view of the database to a writer. If you call 472this from a read-only transaction, it will perform a hot backup and not block 473your other database reads and writes. 474 475By default, it will use a regular file handle which will utilize the operating 476system's page cache. See the [`Tx`](https://godoc.org/github.com/boltdb/bolt#Tx) 477documentation for information about optimizing for larger-than-RAM datasets. 478 479One common use case is to backup over HTTP so you can use tools like `cURL` to 480do database backups: 481 482```go 483func BackupHandleFunc(w http.ResponseWriter, req *http.Request) { 484 err := db.View(func(tx *bolt.Tx) error { 485 w.Header().Set("Content-Type", "application/octet-stream") 486 w.Header().Set("Content-Disposition", `attachment; filename="my.db"`) 487 w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size()))) 488 _, err := tx.WriteTo(w) 489 return err 490 }) 491 if err != nil { 492 http.Error(w, err.Error(), http.StatusInternalServerError) 493 } 494} 495``` 496 497Then you can backup using this command: 498 499```sh 500$ curl http://localhost/backup > my.db 501``` 502 503Or you can open your browser to `http://localhost/backup` and it will download 504automatically. 505 506If you want to backup to another file you can use the `Tx.CopyFile()` helper 507function. 508 509 510### Statistics 511 512The database keeps a running count of many of the internal operations it 513performs so you can better understand what's going on. By grabbing a snapshot 514of these stats at two points in time we can see what operations were performed 515in that time range. 516 517For example, we could start a goroutine to log stats every 10 seconds: 518 519```go 520go func() { 521 // Grab the initial stats. 522 prev := db.Stats() 523 524 for { 525 // Wait for 10s. 526 time.Sleep(10 * time.Second) 527 528 // Grab the current stats and diff them. 529 stats := db.Stats() 530 diff := stats.Sub(&prev) 531 532 // Encode stats to JSON and print to STDERR. 533 json.NewEncoder(os.Stderr).Encode(diff) 534 535 // Save stats for the next loop. 536 prev = stats 537 } 538}() 539``` 540 541It's also useful to pipe these stats to a service such as statsd for monitoring 542or to provide an HTTP endpoint that will perform a fixed-length sample. 543 544 545### Read-Only Mode 546 547Sometimes it is useful to create a shared, read-only Bolt database. To this, 548set the `Options.ReadOnly` flag when opening your database. Read-only mode 549uses a shared lock to allow multiple processes to read from the database but 550it will block any processes from opening the database in read-write mode. 551 552```go 553db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true}) 554if err != nil { 555 log.Fatal(err) 556} 557``` 558 559### Mobile Use (iOS/Android) 560 561Bolt is able to run on mobile devices by leveraging the binding feature of the 562[gomobile](https://github.com/golang/mobile) tool. Create a struct that will 563contain your database logic and a reference to a `*bolt.DB` with a initializing 564constructor that takes in a filepath where the database file will be stored. 565Neither Android nor iOS require extra permissions or cleanup from using this method. 566 567```go 568func NewBoltDB(filepath string) *BoltDB { 569 db, err := bolt.Open(filepath+"/demo.db", 0600, nil) 570 if err != nil { 571 log.Fatal(err) 572 } 573 574 return &BoltDB{db} 575} 576 577type BoltDB struct { 578 db *bolt.DB 579 ... 580} 581 582func (b *BoltDB) Path() string { 583 return b.db.Path() 584} 585 586func (b *BoltDB) Close() { 587 b.db.Close() 588} 589``` 590 591Database logic should be defined as methods on this wrapper struct. 592 593To initialize this struct from the native language (both platforms now sync 594their local storage to the cloud. These snippets disable that functionality for the 595database file): 596 597#### Android 598 599```java 600String path; 601if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){ 602 path = getNoBackupFilesDir().getAbsolutePath(); 603} else{ 604 path = getFilesDir().getAbsolutePath(); 605} 606Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path) 607``` 608 609#### iOS 610 611```objc 612- (void)demo { 613 NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory, 614 NSUserDomainMask, 615 YES) objectAtIndex:0]; 616 GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path); 617 [self addSkipBackupAttributeToItemAtPath:demo.path]; 618 //Some DB Logic would go here 619 [demo close]; 620} 621 622- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString 623{ 624 NSURL* URL= [NSURL fileURLWithPath: filePathString]; 625 assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]); 626 627 NSError *error = nil; 628 BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES] 629 forKey: NSURLIsExcludedFromBackupKey error: &error]; 630 if(!success){ 631 NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error); 632 } 633 return success; 634} 635 636``` 637 638## Resources 639 640For more information on getting started with Bolt, check out the following articles: 641 642* [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch). 643* [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville 644 645 646## Comparison with other databases 647 648### Postgres, MySQL, & other relational databases 649 650Relational databases structure data into rows and are only accessible through 651the use of SQL. This approach provides flexibility in how you store and query 652your data but also incurs overhead in parsing and planning SQL statements. Bolt 653accesses all data by a byte slice key. This makes Bolt fast to read and write 654data by key but provides no built-in support for joining values together. 655 656Most relational databases (with the exception of SQLite) are standalone servers 657that run separately from your application. This gives your systems 658flexibility to connect multiple application servers to a single database 659server but also adds overhead in serializing and transporting data over the 660network. Bolt runs as a library included in your application so all data access 661has to go through your application's process. This brings data closer to your 662application but limits multi-process access to the data. 663 664 665### LevelDB, RocksDB 666 667LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that 668they are libraries bundled into the application, however, their underlying 669structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes 670random writes by using a write ahead log and multi-tiered, sorted files called 671SSTables. Bolt uses a B+tree internally and only a single file. Both approaches 672have trade-offs. 673 674If you require a high random write throughput (>10,000 w/sec) or you need to use 675spinning disks then LevelDB could be a good choice. If your application is 676read-heavy or does a lot of range scans then Bolt could be a good choice. 677 678One other important consideration is that LevelDB does not have transactions. 679It supports batch writing of key/values pairs and it supports read snapshots 680but it will not give you the ability to do a compare-and-swap operation safely. 681Bolt supports fully serializable ACID transactions. 682 683 684### LMDB 685 686Bolt was originally a port of LMDB so it is architecturally similar. Both use 687a B+tree, have ACID semantics with fully serializable transactions, and support 688lock-free MVCC using a single writer and multiple readers. 689 690The two projects have somewhat diverged. LMDB heavily focuses on raw performance 691while Bolt has focused on simplicity and ease of use. For example, LMDB allows 692several unsafe actions such as direct writes for the sake of performance. Bolt 693opts to disallow actions which can leave the database in a corrupted state. The 694only exception to this in Bolt is `DB.NoSync`. 695 696There are also a few differences in API. LMDB requires a maximum mmap size when 697opening an `mdb_env` whereas Bolt will handle incremental mmap resizing 698automatically. LMDB overloads the getter and setter functions with multiple 699flags whereas Bolt splits these specialized cases into their own functions. 700 701 702## Caveats & Limitations 703 704It's important to pick the right tool for the job and Bolt is no exception. 705Here are a few things to note when evaluating and using Bolt: 706 707* Bolt is good for read intensive workloads. Sequential write performance is 708 also fast but random writes can be slow. You can use `DB.Batch()` or add a 709 write-ahead log to help mitigate this issue. 710 711* Bolt uses a B+tree internally so there can be a lot of random page access. 712 SSDs provide a significant performance boost over spinning disks. 713 714* Try to avoid long running read transactions. Bolt uses copy-on-write so 715 old pages cannot be reclaimed while an old transaction is using them. 716 717* Byte slices returned from Bolt are only valid during a transaction. Once the 718 transaction has been committed or rolled back then the memory they point to 719 can be reused by a new page or can be unmapped from virtual memory and you'll 720 see an `unexpected fault address` panic when accessing it. 721 722* Be careful when using `Bucket.FillPercent`. Setting a high fill percent for 723 buckets that have random inserts will cause your database to have very poor 724 page utilization. 725 726* Use larger buckets in general. Smaller buckets causes poor page utilization 727 once they become larger than the page size (typically 4KB). 728 729* Bulk loading a lot of random writes into a new bucket can be slow as the 730 page will not split until the transaction is committed. Randomly inserting 731 more than 100,000 key/value pairs into a single new bucket in a single 732 transaction is not advised. 733 734* Bolt uses a memory-mapped file so the underlying operating system handles the 735 caching of the data. Typically, the OS will cache as much of the file as it 736 can in memory and will release memory as needed to other processes. This means 737 that Bolt can show very high memory usage when working with large databases. 738 However, this is expected and the OS will release memory as needed. Bolt can 739 handle databases much larger than the available physical RAM, provided its 740 memory-map fits in the process virtual address space. It may be problematic 741 on 32-bits systems. 742 743* The data structures in the Bolt database are memory mapped so the data file 744 will be endian specific. This means that you cannot copy a Bolt file from a 745 little endian machine to a big endian machine and have it work. For most 746 users this is not a concern since most modern CPUs are little endian. 747 748* Because of the way pages are laid out on disk, Bolt cannot truncate data files 749 and return free pages back to the disk. Instead, Bolt maintains a free list 750 of unused pages within its data file. These free pages can be reused by later 751 transactions. This works well for many use cases as databases generally tend 752 to grow. However, it's important to note that deleting large chunks of data 753 will not allow you to reclaim that space on disk. 754 755 For more information on page allocation, [see this comment][page-allocation]. 756 757[page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638 758 759 760## Reading the Source 761 762Bolt is a relatively small code base (<3KLOC) for an embedded, serializable, 763transactional key/value database so it can be a good starting point for people 764interested in how databases work. 765 766The best places to start are the main entry points into Bolt: 767 768- `Open()` - Initializes the reference to the database. It's responsible for 769 creating the database if it doesn't exist, obtaining an exclusive lock on the 770 file, reading the meta pages, & memory-mapping the file. 771 772- `DB.Begin()` - Starts a read-only or read-write transaction depending on the 773 value of the `writable` argument. This requires briefly obtaining the "meta" 774 lock to keep track of open transactions. Only one read-write transaction can 775 exist at a time so the "rwlock" is acquired during the life of a read-write 776 transaction. 777 778- `Bucket.Put()` - Writes a key/value pair into a bucket. After validating the 779 arguments, a cursor is used to traverse the B+tree to the page and position 780 where they key & value will be written. Once the position is found, the bucket 781 materializes the underlying page and the page's parent pages into memory as 782 "nodes". These nodes are where mutations occur during read-write transactions. 783 These changes get flushed to disk during commit. 784 785- `Bucket.Get()` - Retrieves a key/value pair from a bucket. This uses a cursor 786 to move to the page & position of a key/value pair. During a read-only 787 transaction, the key and value data is returned as a direct reference to the 788 underlying mmap file so there's no allocation overhead. For read-write 789 transactions, this data may reference the mmap file or one of the in-memory 790 node values. 791 792- `Cursor` - This object is simply for traversing the B+tree of on-disk pages 793 or in-memory nodes. It can seek to a specific key, move to the first or last 794 value, or it can move forward or backward. The cursor handles the movement up 795 and down the B+tree transparently to the end user. 796 797- `Tx.Commit()` - Converts the in-memory dirty nodes and the list of free pages 798 into pages to be written to disk. Writing to disk then occurs in two phases. 799 First, the dirty pages are written to disk and an `fsync()` occurs. Second, a 800 new meta page with an incremented transaction ID is written and another 801 `fsync()` occurs. This two phase write ensures that partially written data 802 pages are ignored in the event of a crash since the meta page pointing to them 803 is never written. Partially written meta pages are invalidated because they 804 are written with a checksum. 805 806If you have additional notes that could be helpful for others, please submit 807them via pull request. 808 809 810## Other Projects Using Bolt 811 812Below is a list of public, open source projects that use Bolt: 813 814* [BoltDbWeb](https://github.com/evnix/boltdbweb) - A web based GUI for BoltDB files. 815* [Operation Go: A Routine Mission](http://gocode.io) - An online programming game for Golang using Bolt for user accounts and a leaderboard. 816* [Bazil](https://bazil.org/) - A file system that lets your data reside where it is most convenient for it to reside. 817* [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb. 818* [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics. 819* [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects. 820* [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday. 821* [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations. 822* [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite. 823* [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin". 824* [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka. 825* [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed. 826* [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt. 827* [photosite/session](https://godoc.org/bitbucket.org/kardianos/photosite/session) - Sessions for a photo viewing site. 828* [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage. 829* [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters. 830* [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend. 831* [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend. 832* [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server. 833* [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read. 834* [InfluxDB](https://influxdata.com) - Scalable datastore for metrics, events, and real-time analytics. 835* [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data. 836* [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system. 837* [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware. 838* [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs. 839* [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems. 840* [stow](https://github.com/djherbis/stow) - a persistence manager for objects 841 backed by boltdb. 842* [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining 843 simple tx and key scans. 844* [mbuckets](https://github.com/abhigupta912/mbuckets) - A Bolt wrapper that allows easy operations on multi level (nested) buckets. 845* [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service 846* [Go Report Card](https://goreportcard.com/) - Go code quality report cards as a (free and open source) service. 847* [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners. 848* [lru](https://github.com/crowdriff/lru) - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores. 849* [Storm](https://github.com/asdine/storm) - Simple and powerful ORM for BoltDB. 850* [GoWebApp](https://github.com/josephspurrier/gowebapp) - A basic MVC web application in Go using BoltDB. 851* [SimpleBolt](https://github.com/xyproto/simplebolt) - A simple way to use BoltDB. Deals mainly with strings. 852* [Algernon](https://github.com/xyproto/algernon) - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend. 853* [MuLiFS](https://github.com/dankomiocevic/mulifs) - Music Library Filesystem creates a filesystem to organise your music files. 854* [GoShort](https://github.com/pankajkhairnar/goShort) - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter. 855* [torrent](https://github.com/anacrolix/torrent) - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development. 856 857If you are using Bolt in a project please send a pull request to add it to the list. 858