1# smat – State Machine Assisted Testing 2 3The concept is simple, describe valid uses of your library as states and actions. States describe which actions are possible, and with what probability they should occur. Actions mutate the context and transition to another state. 4 5By doing this, two things are possible: 6 71. Use [go-fuzz](https://github.com/dvyukov/go-fuzz) to find/test interesting sequences of operations on your library. 8 92. Automate longevity testing of your application by performing long sequences of valid operations. 10 11**NOTE**: both of these can also incorporate validation logic (not just failure detection by building validation into the state machine) 12 13## Status 14 15The API is still not stable. This is brand new and we'll probably change things we don't like... 16 17[![Build Status](https://travis-ci.org/mschoch/smat.svg?branch=master)](https://travis-ci.org/mschoch/smat) 18[![Coverage Status](https://coveralls.io/repos/github/mschoch/smat/badge.svg?branch=master)](https://coveralls.io/github/mschoch/smat?branch=master) 19[![GoDoc](https://godoc.org/github.com/mschoch/smat?status.svg)](https://godoc.org/github.com/mschoch/smat) 20[![codebeat badge](https://codebeat.co/badges/c3ff6180-a241-4128-97f0-fa6bf6f48752)](https://codebeat.co/projects/github-com-mschoch-smat) 21[![Go Report Card](https://goreportcard.com/badge/github.com/mschoch/smat)](https://goreportcard.com/report/github.com/mschoch/smat) 22 23## License 24 25Apache 2.0 26 27## How do I use it? 28 29### smat.Context 30 31Choose a structure to keep track of any state. You pass in an instance of this when you start, and it will be passed to every action when it executes. The actions may mutate this context. 32 33For example, consider a database library, once you open a database handle, you need to use it inside of the other actions. So you might use a structure like: 34 35``` 36type context struct { 37 db *DB 38} 39``` 40 41### smat.State 42 43A state represents a state that your application/library can be in, and the probabilities thats certain actions should be taken. 44 45For example, consider a database library, in a state where the database is open, there many things you can do. Let's consider just two right now, you can set a value, or you can delete a value. 46 47``` 48func dbOpen(next byte) smat.ActionID { 49 return smat.PercentExecute(next, 50 smat.PercentAction{50, setValue}, 51 smat.PercentAction{50, deleteValue}, 52 ) 53} 54``` 55 56This says that in the open state, there are two valid actions, 50% of the time you should set a value and 50% of the time you should delete a value. **NOTE**: these percentages are just for characterizing the test workload. 57 58### smat.Action 59 60Actions are functions that do some work, optionally mutate the context, and indicate the next state to transition to. Below we see an example action to set value in a database. 61 62``` 63func setValueFunc(ctx smat.Context) (next smat.State, err error) { 64 // type assert to our custom context type 65 context := ctx.(*context) 66 // perform the operation 67 err = context.db.Set("k", "v") 68 if err != nil { 69 return nil, err 70 } 71 // return the new state 72 return dbOpen, nil 73} 74``` 75 76### smat.ActionID and smat.ActionMap 77 78Actions are just functions, and since we can't compare functions in Go, we need to introduce an external identifier for them. This allows us to build a bi-directional mapping which we'll take advantage of later. 79 80``` 81const ( 82 setup smat.ActionID = iota 83 teardown 84 setValue 85 deleteValue 86) 87 88var actionMap = smat.ActionMap{ 89 setup: setupFunc, 90 teardown: teardownFunc, 91 setValue: setValueFunc, 92 deleteValue: deleteValueFunc, 93} 94``` 95 96### smat.ActionSeq 97 98A common way that many users think about a library is as a sequence of actions to be performed. Using the ActionID's that we've already seen we can build up sequences of operations. 99 100``` 101 actionSeq := smat.ActionSeq{ 102 open, 103 setValue, 104 setValue, 105 setValue, 106 } 107``` 108 109Notice that we build these actions using the constants we defined above, and because of this we can have a bi-directional mapping between a stream of bytes (driving the state machine) and a sequence of actions to be performed. 110 111## Fuzzing 112 113We've built a lot of pieces, lets wire it up to go-fuzz. 114 115``` 116func Fuzz(data []byte) int { 117 return smat.Fuzz(&context{}, setup, teardown, actionMap, data) 118} 119``` 120 121* The first argument is an instance of context structure. 122* The second argument is the ActionID of our setup function. The setup function does not consume any of the input stream and is used to initialize the context and determine the start state. 123* The third argument is the teardown function. This will be called unconditionally to clean up any resources associated with the test. 124* The fourth argument is the actionMap which maps all ActionIDs to Actions. 125* The fifth argument is the data passed in from the go-fuzz application. 126 127### Generating Initial go-fuzz Corpus 128 129Earlier we mentioned the bi-directional mapping between Actions and the byte stream driving the state machine. We can now leverage this to build the inital go-fuzz corpus. 130 131Using the `ActinSeq`s we learned about earlier we can build up a list of them as: 132 133 var actionSeqs = []smat.ActionSeq{...} 134 135Then, we can write them out to disk using: 136 137``` 138for i, actionSeq := range actionSeqs { 139 byteSequence, err := actionSeq.ByteEncoding(&context{}, setup, teardown, actionMap) 140 if err != nil { 141 // handle error 142 } 143 os.MkdirAll("workdir/corpus", 0700) 144 ioutil.WriteFile(fmt.Sprintf("workdir/corpus/%d", i), byteSequence, 0600) 145} 146``` 147 148You can then either put this into a test case or a main application depending on your needs. 149 150## Longevity Testing 151 152Fuzzing is great, but most of your corpus is likely to be shorter meaningful sequences. And go-fuzz works to find shortest sequences that cause problems, but sometimes you actually want to explore longer sequences that appear to go-fuzz as not triggering additional code coverage. 153 154For these cases we have another helper you can use: 155 156``` 157 Longevity(ctx, setup, teardown, actionMap, 0, closeChan) 158``` 159 160The first four arguments are the same, the last two are: 161* random seed used to ensure repeatable tests 162* closeChan (chan struct{}) - close this channel if you want the function to stop and return ErrClosed, otherwise it will run forever 163 164## Examples 165 166See the examples directory for a working example that tests some BoltDB functionality. 167