README.md
1# smat – State Machine Assisted Testing
2
3The concept is simple, describe valid uses of your library as states and actions. States describe which actions are possible, and with what probability they should occur. Actions mutate the context and transition to another state.
4
5By doing this, two things are possible:
6
71. Use [go-fuzz](https://github.com/dvyukov/go-fuzz) to find/test interesting sequences of operations on your library.
8
92. Automate longevity testing of your application by performing long sequences of valid operations.
10
11**NOTE**: both of these can also incorporate validation logic (not just failure detection by building validation into the state machine)
12
13## Status
14
15The API is still not stable. This is brand new and we'll probably change things we don't like...
16
17[](https://travis-ci.org/mschoch/smat)
18[](https://coveralls.io/github/mschoch/smat?branch=master)
19[](https://godoc.org/github.com/mschoch/smat)
20[](https://codebeat.co/projects/github-com-mschoch-smat)
21[](https://goreportcard.com/report/github.com/mschoch/smat)
22
23## License
24
25Apache 2.0
26
27## How do I use it?
28
29### smat.Context
30
31Choose a structure to keep track of any state. You pass in an instance of this when you start, and it will be passed to every action when it executes. The actions may mutate this context.
32
33For example, consider a database library, once you open a database handle, you need to use it inside of the other actions. So you might use a structure like:
34
35```
36type context struct {
37 db *DB
38}
39```
40
41### smat.State
42
43A state represents a state that your application/library can be in, and the probabilities thats certain actions should be taken.
44
45For example, consider a database library, in a state where the database is open, there many things you can do. Let's consider just two right now, you can set a value, or you can delete a value.
46
47```
48func dbOpen(next byte) smat.ActionID {
49 return smat.PercentExecute(next,
50 smat.PercentAction{50, setValue},
51 smat.PercentAction{50, deleteValue},
52 )
53}
54```
55
56This says that in the open state, there are two valid actions, 50% of the time you should set a value and 50% of the time you should delete a value. **NOTE**: these percentages are just for characterizing the test workload.
57
58### smat.Action
59
60Actions are functions that do some work, optionally mutate the context, and indicate the next state to transition to. Below we see an example action to set value in a database.
61
62```
63func setValueFunc(ctx smat.Context) (next smat.State, err error) {
64 // type assert to our custom context type
65 context := ctx.(*context)
66 // perform the operation
67 err = context.db.Set("k", "v")
68 if err != nil {
69 return nil, err
70 }
71 // return the new state
72 return dbOpen, nil
73}
74```
75
76### smat.ActionID and smat.ActionMap
77
78Actions are just functions, and since we can't compare functions in Go, we need to introduce an external identifier for them. This allows us to build a bi-directional mapping which we'll take advantage of later.
79
80```
81const (
82 setup smat.ActionID = iota
83 teardown
84 setValue
85 deleteValue
86)
87
88var actionMap = smat.ActionMap{
89 setup: setupFunc,
90 teardown: teardownFunc,
91 setValue: setValueFunc,
92 deleteValue: deleteValueFunc,
93}
94```
95
96### smat.ActionSeq
97
98A common way that many users think about a library is as a sequence of actions to be performed. Using the ActionID's that we've already seen we can build up sequences of operations.
99
100```
101 actionSeq := smat.ActionSeq{
102 open,
103 setValue,
104 setValue,
105 setValue,
106 }
107```
108
109Notice that we build these actions using the constants we defined above, and because of this we can have a bi-directional mapping between a stream of bytes (driving the state machine) and a sequence of actions to be performed.
110
111## Fuzzing
112
113We've built a lot of pieces, lets wire it up to go-fuzz.
114
115```
116func Fuzz(data []byte) int {
117 return smat.Fuzz(&context{}, setup, teardown, actionMap, data)
118}
119```
120
121* The first argument is an instance of context structure.
122* The second argument is the ActionID of our setup function. The setup function does not consume any of the input stream and is used to initialize the context and determine the start state.
123* The third argument is the teardown function. This will be called unconditionally to clean up any resources associated with the test.
124* The fourth argument is the actionMap which maps all ActionIDs to Actions.
125* The fifth argument is the data passed in from the go-fuzz application.
126
127### Generating Initial go-fuzz Corpus
128
129Earlier we mentioned the bi-directional mapping between Actions and the byte stream driving the state machine. We can now leverage this to build the inital go-fuzz corpus.
130
131Using the `ActinSeq`s we learned about earlier we can build up a list of them as:
132
133 var actionSeqs = []smat.ActionSeq{...}
134
135Then, we can write them out to disk using:
136
137```
138for i, actionSeq := range actionSeqs {
139 byteSequence, err := actionSeq.ByteEncoding(&context{}, setup, teardown, actionMap)
140 if err != nil {
141 // handle error
142 }
143 os.MkdirAll("workdir/corpus", 0700)
144 ioutil.WriteFile(fmt.Sprintf("workdir/corpus/%d", i), byteSequence, 0600)
145}
146```
147
148You can then either put this into a test case or a main application depending on your needs.
149
150## Longevity Testing
151
152Fuzzing is great, but most of your corpus is likely to be shorter meaningful sequences. And go-fuzz works to find shortest sequences that cause problems, but sometimes you actually want to explore longer sequences that appear to go-fuzz as not triggering additional code coverage.
153
154For these cases we have another helper you can use:
155
156```
157 Longevity(ctx, setup, teardown, actionMap, 0, closeChan)
158```
159
160The first four arguments are the same, the last two are:
161* random seed used to ensure repeatable tests
162* closeChan (chan struct{}) - close this channel if you want the function to stop and return ErrClosed, otherwise it will run forever
163
164## Examples
165
166See the examples directory for a working example that tests some BoltDB functionality.
167
){kind=link}
){kind=link}
){kind=link}