1Bolt [![Coverage Status](https://coveralls.io/repos/boltdb/bolt/badge.svg?branch=master)](https://coveralls.io/r/boltdb/bolt?branch=master) [![GoDoc](https://godoc.org/github.com/boltdb/bolt?status.svg)](https://godoc.org/github.com/boltdb/bolt) ![Version](https://img.shields.io/badge/version-1.2.1-green.svg)
2====
3
4Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas]
5[LMDB project][lmdb]. The goal of the project is to provide a simple,
6fast, and reliable database for projects that don't require a full database
7server such as Postgres or MySQL.
8
9Since Bolt is meant to be used as such a low-level piece of functionality,
10simplicity is key. The API will be small and only focus on getting values
11and setting values. That's it.
12
13[hyc_symas]: https://twitter.com/hyc_symas
14[lmdb]: http://symas.com/mdb/
15
16## Project Status
17
18Bolt is stable, the API is fixed, and the file format is fixed. Full unit
19test coverage and randomized black box testing are used to ensure database
20consistency and thread safety. Bolt is currently used in high-load production
21environments serving databases as large as 1TB. Many companies such as
22Shopify and Heroku use Bolt-backed services every day.
23
24## Table of Contents
25
26- [Getting Started](#getting-started)
27  - [Installing](#installing)
28  - [Opening a database](#opening-a-database)
29  - [Transactions](#transactions)
30    - [Read-write transactions](#read-write-transactions)
31    - [Read-only transactions](#read-only-transactions)
32    - [Batch read-write transactions](#batch-read-write-transactions)
33    - [Managing transactions manually](#managing-transactions-manually)
34  - [Using buckets](#using-buckets)
35  - [Using key/value pairs](#using-keyvalue-pairs)
36  - [Autoincrementing integer for the bucket](#autoincrementing-integer-for-the-bucket)
37  - [Iterating over keys](#iterating-over-keys)
38    - [Prefix scans](#prefix-scans)
39    - [Range scans](#range-scans)
40    - [ForEach()](#foreach)
41  - [Nested buckets](#nested-buckets)
42  - [Database backups](#database-backups)
43  - [Statistics](#statistics)
44  - [Read-Only Mode](#read-only-mode)
45  - [Mobile Use (iOS/Android)](#mobile-use-iosandroid)
46- [Resources](#resources)
47- [Comparison with other databases](#comparison-with-other-databases)
48  - [Postgres, MySQL, & other relational databases](#postgres-mysql--other-relational-databases)
49  - [LevelDB, RocksDB](#leveldb-rocksdb)
50  - [LMDB](#lmdb)
51- [Caveats & Limitations](#caveats--limitations)
52- [Reading the Source](#reading-the-source)
53- [Other Projects Using Bolt](#other-projects-using-bolt)
54
55## Getting Started
56
57### Installing
58
59To start using Bolt, install Go and run `go get`:
60
61```sh
62$ go get github.com/boltdb/bolt/...
63```
64
65This will retrieve the library and install the `bolt` command line utility into
66your `$GOBIN` path.
67
68
69### Opening a database
70
71The top-level object in Bolt is a `DB`. It is represented as a single file on
72your disk and represents a consistent snapshot of your data.
73
74To open your database, simply use the `bolt.Open()` function:
75
76```go
77package main
78
79import (
80	"log"
81
82	"github.com/boltdb/bolt"
83)
84
85func main() {
86	// Open the my.db data file in your current directory.
87	// It will be created if it doesn't exist.
88	db, err := bolt.Open("my.db", 0600, nil)
89	if err != nil {
90		log.Fatal(err)
91	}
92	defer db.Close()
93
94	...
95}
96```
97
98Please note that Bolt obtains a file lock on the data file so multiple processes
99cannot open the same database at the same time. Opening an already open Bolt
100database will cause it to hang until the other process closes it. To prevent
101an indefinite wait you can pass a timeout option to the `Open()` function:
102
103```go
104db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})
105```
106
107
108### Transactions
109
110Bolt allows only one read-write transaction at a time but allows as many
111read-only transactions as you want at a time. Each transaction has a consistent
112view of the data as it existed when the transaction started.
113
114Individual transactions and all objects created from them (e.g. buckets, keys)
115are not thread safe. To work with data in multiple goroutines you must start
116a transaction for each one or use locking to ensure only one goroutine accesses
117a transaction at a time. Creating transaction from the `DB` is thread safe.
118
119Read-only transactions and read-write transactions should not depend on one
120another and generally shouldn't be opened simultaneously in the same goroutine.
121This can cause a deadlock as the read-write transaction needs to periodically
122re-map the data file but it cannot do so while a read-only transaction is open.
123
124
125#### Read-write transactions
126
127To start a read-write transaction, you can use the `DB.Update()` function:
128
129```go
130err := db.Update(func(tx *bolt.Tx) error {
131	...
132	return nil
133})
134```
135
136Inside the closure, you have a consistent view of the database. You commit the
137transaction by returning `nil` at the end. You can also rollback the transaction
138at any point by returning an error. All database operations are allowed inside
139a read-write transaction.
140
141Always check the return error as it will report any disk failures that can cause
142your transaction to not complete. If you return an error within your closure
143it will be passed through.
144
145
146#### Read-only transactions
147
148To start a read-only transaction, you can use the `DB.View()` function:
149
150```go
151err := db.View(func(tx *bolt.Tx) error {
152	...
153	return nil
154})
155```
156
157You also get a consistent view of the database within this closure, however,
158no mutating operations are allowed within a read-only transaction. You can only
159retrieve buckets, retrieve values, and copy the database within a read-only
160transaction.
161
162
163#### Batch read-write transactions
164
165Each `DB.Update()` waits for disk to commit the writes. This overhead
166can be minimized by combining multiple updates with the `DB.Batch()`
167function:
168
169```go
170err := db.Batch(func(tx *bolt.Tx) error {
171	...
172	return nil
173})
174```
175
176Concurrent Batch calls are opportunistically combined into larger
177transactions. Batch is only useful when there are multiple goroutines
178calling it.
179
180The trade-off is that `Batch` can call the given
181function multiple times, if parts of the transaction fail. The
182function must be idempotent and side effects must take effect only
183after a successful return from `DB.Batch()`.
184
185For example: don't display messages from inside the function, instead
186set variables in the enclosing scope:
187
188```go
189var id uint64
190err := db.Batch(func(tx *bolt.Tx) error {
191	// Find last key in bucket, decode as bigendian uint64, increment
192	// by one, encode back to []byte, and add new key.
193	...
194	id = newValue
195	return nil
196})
197if err != nil {
198	return ...
199}
200fmt.Println("Allocated ID %d", id)
201```
202
203
204#### Managing transactions manually
205
206The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()`
207function. These helper functions will start the transaction, execute a function,
208and then safely close your transaction if an error is returned. This is the
209recommended way to use Bolt transactions.
210
211However, sometimes you may want to manually start and end your transactions.
212You can use the `DB.Begin()` function directly but **please** be sure to close
213the transaction.
214
215```go
216// Start a writable transaction.
217tx, err := db.Begin(true)
218if err != nil {
219    return err
220}
221defer tx.Rollback()
222
223// Use the transaction...
224_, err := tx.CreateBucket([]byte("MyBucket"))
225if err != nil {
226    return err
227}
228
229// Commit the transaction and check for error.
230if err := tx.Commit(); err != nil {
231    return err
232}
233```
234
235The first argument to `DB.Begin()` is a boolean stating if the transaction
236should be writable.
237
238
239### Using buckets
240
241Buckets are collections of key/value pairs within the database. All keys in a
242bucket must be unique. You can create a bucket using the `DB.CreateBucket()`
243function:
244
245```go
246db.Update(func(tx *bolt.Tx) error {
247	b, err := tx.CreateBucket([]byte("MyBucket"))
248	if err != nil {
249		return fmt.Errorf("create bucket: %s", err)
250	}
251	return nil
252})
253```
254
255You can also create a bucket only if it doesn't exist by using the
256`Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this
257function for all your top-level buckets after you open your database so you can
258guarantee that they exist for future transactions.
259
260To delete a bucket, simply call the `Tx.DeleteBucket()` function.
261
262
263### Using key/value pairs
264
265To save a key/value pair to a bucket, use the `Bucket.Put()` function:
266
267```go
268db.Update(func(tx *bolt.Tx) error {
269	b := tx.Bucket([]byte("MyBucket"))
270	err := b.Put([]byte("answer"), []byte("42"))
271	return err
272})
273```
274
275This will set the value of the `"answer"` key to `"42"` in the `MyBucket`
276bucket. To retrieve this value, we can use the `Bucket.Get()` function:
277
278```go
279db.View(func(tx *bolt.Tx) error {
280	b := tx.Bucket([]byte("MyBucket"))
281	v := b.Get([]byte("answer"))
282	fmt.Printf("The answer is: %s\n", v)
283	return nil
284})
285```
286
287The `Get()` function does not return an error because its operation is
288guaranteed to work (unless there is some kind of system failure). If the key
289exists then it will return its byte slice value. If it doesn't exist then it
290will return `nil`. It's important to note that you can have a zero-length value
291set to a key which is different than the key not existing.
292
293Use the `Bucket.Delete()` function to delete a key from the bucket.
294
295Please note that values returned from `Get()` are only valid while the
296transaction is open. If you need to use a value outside of the transaction
297then you must use `copy()` to copy it to another byte slice.
298
299
300### Autoincrementing integer for the bucket
301By using the `NextSequence()` function, you can let Bolt determine a sequence
302which can be used as the unique identifier for your key/value pairs. See the
303example below.
304
305```go
306// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
307func (s *Store) CreateUser(u *User) error {
308    return s.db.Update(func(tx *bolt.Tx) error {
309        // Retrieve the users bucket.
310        // This should be created when the DB is first opened.
311        b := tx.Bucket([]byte("users"))
312
313        // Generate ID for the user.
314        // This returns an error only if the Tx is closed or not writeable.
315        // That can't happen in an Update() call so I ignore the error check.
316        id, _ := b.NextSequence()
317        u.ID = int(id)
318
319        // Marshal user data into bytes.
320        buf, err := json.Marshal(u)
321        if err != nil {
322            return err
323        }
324
325        // Persist bytes to users bucket.
326        return b.Put(itob(u.ID), buf)
327    })
328}
329
330// itob returns an 8-byte big endian representation of v.
331func itob(v int) []byte {
332    b := make([]byte, 8)
333    binary.BigEndian.PutUint64(b, uint64(v))
334    return b
335}
336
337type User struct {
338    ID int
339    ...
340}
341```
342
343### Iterating over keys
344
345Bolt stores its keys in byte-sorted order within a bucket. This makes sequential
346iteration over these keys extremely fast. To iterate over keys we'll use a
347`Cursor`:
348
349```go
350db.View(func(tx *bolt.Tx) error {
351	// Assume bucket exists and has keys
352	b := tx.Bucket([]byte("MyBucket"))
353
354	c := b.Cursor()
355
356	for k, v := c.First(); k != nil; k, v = c.Next() {
357		fmt.Printf("key=%s, value=%s\n", k, v)
358	}
359
360	return nil
361})
362```
363
364The cursor allows you to move to a specific point in the list of keys and move
365forward or backward through the keys one at a time.
366
367The following functions are available on the cursor:
368
369```
370First()  Move to the first key.
371Last()   Move to the last key.
372Seek()   Move to a specific key.
373Next()   Move to the next key.
374Prev()   Move to the previous key.
375```
376
377Each of those functions has a return signature of `(key []byte, value []byte)`.
378When you have iterated to the end of the cursor then `Next()` will return a
379`nil` key.  You must seek to a position using `First()`, `Last()`, or `Seek()`
380before calling `Next()` or `Prev()`. If you do not seek to a position then
381these functions will return a `nil` key.
382
383During iteration, if the key is non-`nil` but the value is `nil`, that means
384the key refers to a bucket rather than a value.  Use `Bucket.Bucket()` to
385access the sub-bucket.
386
387
388#### Prefix scans
389
390To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`:
391
392```go
393db.View(func(tx *bolt.Tx) error {
394	// Assume bucket exists and has keys
395	c := tx.Bucket([]byte("MyBucket")).Cursor()
396
397	prefix := []byte("1234")
398	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
399		fmt.Printf("key=%s, value=%s\n", k, v)
400	}
401
402	return nil
403})
404```
405
406#### Range scans
407
408Another common use case is scanning over a range such as a time range. If you
409use a sortable time encoding such as RFC3339 then you can query a specific
410date range like this:
411
412```go
413db.View(func(tx *bolt.Tx) error {
414	// Assume our events bucket exists and has RFC3339 encoded time keys.
415	c := tx.Bucket([]byte("Events")).Cursor()
416
417	// Our time range spans the 90's decade.
418	min := []byte("1990-01-01T00:00:00Z")
419	max := []byte("2000-01-01T00:00:00Z")
420
421	// Iterate over the 90's.
422	for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
423		fmt.Printf("%s: %s\n", k, v)
424	}
425
426	return nil
427})
428```
429
430Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.
431
432
433#### ForEach()
434
435You can also use the function `ForEach()` if you know you'll be iterating over
436all the keys in a bucket:
437
438```go
439db.View(func(tx *bolt.Tx) error {
440	// Assume bucket exists and has keys
441	b := tx.Bucket([]byte("MyBucket"))
442
443	b.ForEach(func(k, v []byte) error {
444		fmt.Printf("key=%s, value=%s\n", k, v)
445		return nil
446	})
447	return nil
448})
449```
450
451Please note that keys and values in `ForEach()` are only valid while
452the transaction is open. If you need to use a key or value outside of
453the transaction, you must use `copy()` to copy it to another byte
454slice.
455
456### Nested buckets
457
458You can also store a bucket in a key to create nested buckets. The API is the
459same as the bucket management API on the `DB` object:
460
461```go
462func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
463func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
464func (*Bucket) DeleteBucket(key []byte) error
465```
466
467Say you had a multi-tenant application where the root level bucket was the account bucket. Inside of this bucket was a sequence of accounts which themselves are buckets. And inside the sequence bucket you could have many buckets pertaining to the Account itself (Users, Notes, etc) isolating the information into logical groupings.
468
469```go
470
471// createUser creates a new user in the given account.
472func createUser(accountID int, u *User) error {
473    // Start the transaction.
474    tx, err := db.Begin(true)
475    if err != nil {
476        return err
477    }
478    defer tx.Rollback()
479
480    // Retrieve the root bucket for the account.
481    // Assume this has already been created when the account was set up.
482    root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10)))
483
484    // Setup the users bucket.
485    bkt, err := root.CreateBucketIfNotExists([]byte("USERS"))
486    if err != nil {
487        return err
488    }
489
490    // Generate an ID for the new user.
491    userID, err := bkt.NextSequence()
492    if err != nil {
493        return err
494    }
495    u.ID = userID
496
497    // Marshal and save the encoded user.
498    if buf, err := json.Marshal(u); err != nil {
499        return err
500    } else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil {
501        return err
502    }
503
504    // Commit the transaction.
505    if err := tx.Commit(); err != nil {
506        return err
507    }
508
509    return nil
510}
511
512```
513
514
515
516
517### Database backups
518
519Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()`
520function to write a consistent view of the database to a writer. If you call
521this from a read-only transaction, it will perform a hot backup and not block
522your other database reads and writes.
523
524By default, it will use a regular file handle which will utilize the operating
525system's page cache. See the [`Tx`](https://godoc.org/github.com/boltdb/bolt#Tx)
526documentation for information about optimizing for larger-than-RAM datasets.
527
528One common use case is to backup over HTTP so you can use tools like `cURL` to
529do database backups:
530
531```go
532func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
533	err := db.View(func(tx *bolt.Tx) error {
534		w.Header().Set("Content-Type", "application/octet-stream")
535		w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
536		w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
537		_, err := tx.WriteTo(w)
538		return err
539	})
540	if err != nil {
541		http.Error(w, err.Error(), http.StatusInternalServerError)
542	}
543}
544```
545
546Then you can backup using this command:
547
548```sh
549$ curl http://localhost/backup > my.db
550```
551
552Or you can open your browser to `http://localhost/backup` and it will download
553automatically.
554
555If you want to backup to another file you can use the `Tx.CopyFile()` helper
556function.
557
558
559### Statistics
560
561The database keeps a running count of many of the internal operations it
562performs so you can better understand what's going on. By grabbing a snapshot
563of these stats at two points in time we can see what operations were performed
564in that time range.
565
566For example, we could start a goroutine to log stats every 10 seconds:
567
568```go
569go func() {
570	// Grab the initial stats.
571	prev := db.Stats()
572
573	for {
574		// Wait for 10s.
575		time.Sleep(10 * time.Second)
576
577		// Grab the current stats and diff them.
578		stats := db.Stats()
579		diff := stats.Sub(&prev)
580
581		// Encode stats to JSON and print to STDERR.
582		json.NewEncoder(os.Stderr).Encode(diff)
583
584		// Save stats for the next loop.
585		prev = stats
586	}
587}()
588```
589
590It's also useful to pipe these stats to a service such as statsd for monitoring
591or to provide an HTTP endpoint that will perform a fixed-length sample.
592
593
594### Read-Only Mode
595
596Sometimes it is useful to create a shared, read-only Bolt database. To this,
597set the `Options.ReadOnly` flag when opening your database. Read-only mode
598uses a shared lock to allow multiple processes to read from the database but
599it will block any processes from opening the database in read-write mode.
600
601```go
602db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
603if err != nil {
604	log.Fatal(err)
605}
606```
607
608### Mobile Use (iOS/Android)
609
610Bolt is able to run on mobile devices by leveraging the binding feature of the
611[gomobile](https://github.com/golang/mobile) tool. Create a struct that will
612contain your database logic and a reference to a `*bolt.DB` with a initializing
613constructor that takes in a filepath where the database file will be stored.
614Neither Android nor iOS require extra permissions or cleanup from using this method.
615
616```go
617func NewBoltDB(filepath string) *BoltDB {
618	db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
619	if err != nil {
620		log.Fatal(err)
621	}
622
623	return &BoltDB{db}
624}
625
626type BoltDB struct {
627	db *bolt.DB
628	...
629}
630
631func (b *BoltDB) Path() string {
632	return b.db.Path()
633}
634
635func (b *BoltDB) Close() {
636	b.db.Close()
637}
638```
639
640Database logic should be defined as methods on this wrapper struct.
641
642To initialize this struct from the native language (both platforms now sync
643their local storage to the cloud. These snippets disable that functionality for the
644database file):
645
646#### Android
647
648```java
649String path;
650if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
651    path = getNoBackupFilesDir().getAbsolutePath();
652} else{
653    path = getFilesDir().getAbsolutePath();
654}
655Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)
656```
657
658#### iOS
659
660```objc
661- (void)demo {
662    NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
663                                                          NSUserDomainMask,
664                                                          YES) objectAtIndex:0];
665	GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
666	[self addSkipBackupAttributeToItemAtPath:demo.path];
667	//Some DB Logic would go here
668	[demo close];
669}
670
671- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
672{
673    NSURL* URL= [NSURL fileURLWithPath: filePathString];
674    assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);
675
676    NSError *error = nil;
677    BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
678                                  forKey: NSURLIsExcludedFromBackupKey error: &error];
679    if(!success){
680        NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
681    }
682    return success;
683}
684
685```
686
687## Resources
688
689For more information on getting started with Bolt, check out the following articles:
690
691* [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch).
692* [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville
693
694
695## Comparison with other databases
696
697### Postgres, MySQL, & other relational databases
698
699Relational databases structure data into rows and are only accessible through
700the use of SQL. This approach provides flexibility in how you store and query
701your data but also incurs overhead in parsing and planning SQL statements. Bolt
702accesses all data by a byte slice key. This makes Bolt fast to read and write
703data by key but provides no built-in support for joining values together.
704
705Most relational databases (with the exception of SQLite) are standalone servers
706that run separately from your application. This gives your systems
707flexibility to connect multiple application servers to a single database
708server but also adds overhead in serializing and transporting data over the
709network. Bolt runs as a library included in your application so all data access
710has to go through your application's process. This brings data closer to your
711application but limits multi-process access to the data.
712
713
714### LevelDB, RocksDB
715
716LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that
717they are libraries bundled into the application, however, their underlying
718structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes
719random writes by using a write ahead log and multi-tiered, sorted files called
720SSTables. Bolt uses a B+tree internally and only a single file. Both approaches
721have trade-offs.
722
723If you require a high random write throughput (>10,000 w/sec) or you need to use
724spinning disks then LevelDB could be a good choice. If your application is
725read-heavy or does a lot of range scans then Bolt could be a good choice.
726
727One other important consideration is that LevelDB does not have transactions.
728It supports batch writing of key/values pairs and it supports read snapshots
729but it will not give you the ability to do a compare-and-swap operation safely.
730Bolt supports fully serializable ACID transactions.
731
732
733### LMDB
734
735Bolt was originally a port of LMDB so it is architecturally similar. Both use
736a B+tree, have ACID semantics with fully serializable transactions, and support
737lock-free MVCC using a single writer and multiple readers.
738
739The two projects have somewhat diverged. LMDB heavily focuses on raw performance
740while Bolt has focused on simplicity and ease of use. For example, LMDB allows
741several unsafe actions such as direct writes for the sake of performance. Bolt
742opts to disallow actions which can leave the database in a corrupted state. The
743only exception to this in Bolt is `DB.NoSync`.
744
745There are also a few differences in API. LMDB requires a maximum mmap size when
746opening an `mdb_env` whereas Bolt will handle incremental mmap resizing
747automatically. LMDB overloads the getter and setter functions with multiple
748flags whereas Bolt splits these specialized cases into their own functions.
749
750
751## Caveats & Limitations
752
753It's important to pick the right tool for the job and Bolt is no exception.
754Here are a few things to note when evaluating and using Bolt:
755
756* Bolt is good for read intensive workloads. Sequential write performance is
757  also fast but random writes can be slow. You can use `DB.Batch()` or add a
758  write-ahead log to help mitigate this issue.
759
760* Bolt uses a B+tree internally so there can be a lot of random page access.
761  SSDs provide a significant performance boost over spinning disks.
762
763* Try to avoid long running read transactions. Bolt uses copy-on-write so
764  old pages cannot be reclaimed while an old transaction is using them.
765
766* Byte slices returned from Bolt are only valid during a transaction. Once the
767  transaction has been committed or rolled back then the memory they point to
768  can be reused by a new page or can be unmapped from virtual memory and you'll
769  see an `unexpected fault address` panic when accessing it.
770
771* Bolt uses an exclusive write lock on the database file so it cannot be
772  shared by multiple processes.
773
774* Be careful when using `Bucket.FillPercent`. Setting a high fill percent for
775  buckets that have random inserts will cause your database to have very poor
776  page utilization.
777
778* Use larger buckets in general. Smaller buckets causes poor page utilization
779  once they become larger than the page size (typically 4KB).
780
781* Bulk loading a lot of random writes into a new bucket can be slow as the
782  page will not split until the transaction is committed. Randomly inserting
783  more than 100,000 key/value pairs into a single new bucket in a single
784  transaction is not advised.
785
786* Bolt uses a memory-mapped file so the underlying operating system handles the
787  caching of the data. Typically, the OS will cache as much of the file as it
788  can in memory and will release memory as needed to other processes. This means
789  that Bolt can show very high memory usage when working with large databases.
790  However, this is expected and the OS will release memory as needed. Bolt can
791  handle databases much larger than the available physical RAM, provided its
792  memory-map fits in the process virtual address space. It may be problematic
793  on 32-bits systems.
794
795* The data structures in the Bolt database are memory mapped so the data file
796  will be endian specific. This means that you cannot copy a Bolt file from a
797  little endian machine to a big endian machine and have it work. For most
798  users this is not a concern since most modern CPUs are little endian.
799
800* Because of the way pages are laid out on disk, Bolt cannot truncate data files
801  and return free pages back to the disk. Instead, Bolt maintains a free list
802  of unused pages within its data file. These free pages can be reused by later
803  transactions. This works well for many use cases as databases generally tend
804  to grow. However, it's important to note that deleting large chunks of data
805  will not allow you to reclaim that space on disk.
806
807  For more information on page allocation, [see this comment][page-allocation].
808
809[page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638
810
811
812## Reading the Source
813
814Bolt is a relatively small code base (<3KLOC) for an embedded, serializable,
815transactional key/value database so it can be a good starting point for people
816interested in how databases work.
817
818The best places to start are the main entry points into Bolt:
819
820- `Open()` - Initializes the reference to the database. It's responsible for
821  creating the database if it doesn't exist, obtaining an exclusive lock on the
822  file, reading the meta pages, & memory-mapping the file.
823
824- `DB.Begin()` - Starts a read-only or read-write transaction depending on the
825  value of the `writable` argument. This requires briefly obtaining the "meta"
826  lock to keep track of open transactions. Only one read-write transaction can
827  exist at a time so the "rwlock" is acquired during the life of a read-write
828  transaction.
829
830- `Bucket.Put()` - Writes a key/value pair into a bucket. After validating the
831  arguments, a cursor is used to traverse the B+tree to the page and position
832  where they key & value will be written. Once the position is found, the bucket
833  materializes the underlying page and the page's parent pages into memory as
834  "nodes". These nodes are where mutations occur during read-write transactions.
835  These changes get flushed to disk during commit.
836
837- `Bucket.Get()` - Retrieves a key/value pair from a bucket. This uses a cursor
838  to move to the page & position of a key/value pair. During a read-only
839  transaction, the key and value data is returned as a direct reference to the
840  underlying mmap file so there's no allocation overhead. For read-write
841  transactions, this data may reference the mmap file or one of the in-memory
842  node values.
843
844- `Cursor` - This object is simply for traversing the B+tree of on-disk pages
845  or in-memory nodes. It can seek to a specific key, move to the first or last
846  value, or it can move forward or backward. The cursor handles the movement up
847  and down the B+tree transparently to the end user.
848
849- `Tx.Commit()` - Converts the in-memory dirty nodes and the list of free pages
850  into pages to be written to disk. Writing to disk then occurs in two phases.
851  First, the dirty pages are written to disk and an `fsync()` occurs. Second, a
852  new meta page with an incremented transaction ID is written and another
853  `fsync()` occurs. This two phase write ensures that partially written data
854  pages are ignored in the event of a crash since the meta page pointing to them
855  is never written. Partially written meta pages are invalidated because they
856  are written with a checksum.
857
858If you have additional notes that could be helpful for others, please submit
859them via pull request.
860
861
862## Other Projects Using Bolt
863
864Below is a list of public, open source projects that use Bolt:
865
866* [BoltDbWeb](https://github.com/evnix/boltdbweb) - A web based GUI for BoltDB files.
867* [Operation Go: A Routine Mission](http://gocode.io) - An online programming game for Golang using Bolt for user accounts and a leaderboard.
868* [Bazil](https://bazil.org/) - A file system that lets your data reside where it is most convenient for it to reside.
869* [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
870* [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics.
871* [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects.
872* [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday.
873* [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations.
874* [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite.
875* [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
876* [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka.
877* [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed.
878* [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt.
879* [photosite/session](https://godoc.org/bitbucket.org/kardianos/photosite/session) - Sessions for a photo viewing site.
880* [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage.
881* [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters.
882* [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend.
883* [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
884* [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
885* [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read.
886* [InfluxDB](https://influxdata.com) - Scalable datastore for metrics, events, and real-time analytics.
887* [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data.
888* [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system.
889* [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
890* [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
891* [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems.
892* [stow](https://github.com/djherbis/stow) -  a persistence manager for objects
893  backed by boltdb.
894* [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining
895  simple tx and key scans.
896* [mbuckets](https://github.com/abhigupta912/mbuckets) - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
897* [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service
898* [Go Report Card](https://goreportcard.com/) - Go code quality report cards as a (free and open source) service.
899* [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
900* [lru](https://github.com/crowdriff/lru) - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
901* [Storm](https://github.com/asdine/storm) - Simple and powerful ORM for BoltDB.
902* [GoWebApp](https://github.com/josephspurrier/gowebapp) - A basic MVC web application in Go using BoltDB.
903* [SimpleBolt](https://github.com/xyproto/simplebolt) - A simple way to use BoltDB. Deals mainly with strings.
904* [Algernon](https://github.com/xyproto/algernon) - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend.
905* [MuLiFS](https://github.com/dankomiocevic/mulifs) - Music Library Filesystem creates a filesystem to organise your music files.
906* [GoShort](https://github.com/pankajkhairnar/goShort) - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter.
907* [torrent](https://github.com/anacrolix/torrent) - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development.
908* [gopherpit](https://github.com/gopherpit/gopherpit) - A web service to manage Go remote import paths with custom domains
909* [bolter](https://github.com/hasit/bolter) - Command-line app for viewing BoltDB file in your terminal.
910* [btcwallet](https://github.com/btcsuite/btcwallet) - A bitcoin wallet.
911* [dcrwallet](https://github.com/decred/dcrwallet) - A wallet for the Decred cryptocurrency.
912* [Ironsmith](https://github.com/timshannon/ironsmith) - A simple, script-driven continuous integration (build - > test -> release) tool, with no external dependencies
913* [BoltHold](https://github.com/timshannon/bolthold) - An embeddable NoSQL store for Go types built on BoltDB
914* [Ponzu CMS](https://ponzu-cms.org) - Headless CMS + automatic JSON API with auto-HTTPS, HTTP/2 Server Push, and flexible server framework.
915
916If you are using Bolt in a project please send a pull request to add it to the list.
917