1![afero logo-sm](https://cloud.githubusercontent.com/assets/173412/11490338/d50e16dc-97a5-11e5-8b12-019a300d0fcb.png) 2 3A FileSystem Abstraction System for Go 4 5[![Build Status](https://travis-ci.org/spf13/afero.svg)](https://travis-ci.org/spf13/afero) [![Build status](https://ci.appveyor.com/api/projects/status/github/spf13/afero?branch=master&svg=true)](https://ci.appveyor.com/project/spf13/afero) [![GoDoc](https://godoc.org/github.com/spf13/afero?status.svg)](https://godoc.org/github.com/spf13/afero) [![Join the chat at https://gitter.im/spf13/afero](https://badges.gitter.im/Dev%20Chat.svg)](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) 6 7# Overview 8 9Afero is a filesystem framework providing a simple, uniform and universal API 10interacting with any filesystem, as an abstraction layer providing interfaces, 11types and methods. Afero has an exceptionally clean interface and simple design 12without needless constructors or initialization methods. 13 14Afero is also a library providing a base set of interoperable backend 15filesystems that make it easy to work with afero while retaining all the power 16and benefit of the os and ioutil packages. 17 18Afero provides significant improvements over using the os package alone, most 19notably the ability to create mock and testing filesystems without relying on the disk. 20 21It is suitable for use in any situation where you would consider using the OS 22package as it provides an additional abstraction that makes it easy to use a 23memory backed file system during testing. It also adds support for the http 24filesystem for full interoperability. 25 26 27## Afero Features 28 29* A single consistent API for accessing a variety of filesystems 30* Interoperation between a variety of file system types 31* A set of interfaces to encourage and enforce interoperability between backends 32* An atomic cross platform memory backed file system 33* Support for compositional (union) file systems by combining multiple file systems acting as one 34* Specialized backends which modify existing filesystems (Read Only, Regexp filtered) 35* A set of utility functions ported from io, ioutil & hugo to be afero aware 36 37 38# Using Afero 39 40Afero is easy to use and easier to adopt. 41 42A few different ways you could use Afero: 43 44* Use the interfaces alone to define your own file system. 45* Wrapper for the OS packages. 46* Define different filesystems for different parts of your application. 47* Use Afero for mock filesystems while testing 48 49## Step 1: Install Afero 50 51First use go get to install the latest version of the library. 52 53 $ go get github.com/spf13/afero 54 55Next include Afero in your application. 56```go 57import "github.com/spf13/afero" 58``` 59 60## Step 2: Declare a backend 61 62First define a package variable and set it to a pointer to a filesystem. 63```go 64var AppFs = afero.NewMemMapFs() 65 66or 67 68var AppFs = afero.NewOsFs() 69``` 70It is important to note that if you repeat the composite literal you 71will be using a completely new and isolated filesystem. In the case of 72OsFs it will still use the same underlying filesystem but will reduce 73the ability to drop in other filesystems as desired. 74 75## Step 3: Use it like you would the OS package 76 77Throughout your application use any function and method like you normally 78would. 79 80So if my application before had: 81```go 82os.Open('/tmp/foo') 83``` 84We would replace it with: 85```go 86AppFs.Open('/tmp/foo') 87``` 88 89`AppFs` being the variable we defined above. 90 91 92## List of all available functions 93 94File System Methods Available: 95```go 96Chmod(name string, mode os.FileMode) : error 97Chown(name string, uid, gid int) : error 98Chtimes(name string, atime time.Time, mtime time.Time) : error 99Create(name string) : File, error 100Mkdir(name string, perm os.FileMode) : error 101MkdirAll(path string, perm os.FileMode) : error 102Name() : string 103Open(name string) : File, error 104OpenFile(name string, flag int, perm os.FileMode) : File, error 105Remove(name string) : error 106RemoveAll(path string) : error 107Rename(oldname, newname string) : error 108Stat(name string) : os.FileInfo, error 109``` 110File Interfaces and Methods Available: 111```go 112io.Closer 113io.Reader 114io.ReaderAt 115io.Seeker 116io.Writer 117io.WriterAt 118 119Name() : string 120Readdir(count int) : []os.FileInfo, error 121Readdirnames(n int) : []string, error 122Stat() : os.FileInfo, error 123Sync() : error 124Truncate(size int64) : error 125WriteString(s string) : ret int, err error 126``` 127In some applications it may make sense to define a new package that 128simply exports the file system variable for easy access from anywhere. 129 130## Using Afero's utility functions 131 132Afero provides a set of functions to make it easier to use the underlying file systems. 133These functions have been primarily ported from io & ioutil with some developed for Hugo. 134 135The afero utilities support all afero compatible backends. 136 137The list of utilities includes: 138 139```go 140DirExists(path string) (bool, error) 141Exists(path string) (bool, error) 142FileContainsBytes(filename string, subslice []byte) (bool, error) 143GetTempDir(subPath string) string 144IsDir(path string) (bool, error) 145IsEmpty(path string) (bool, error) 146ReadDir(dirname string) ([]os.FileInfo, error) 147ReadFile(filename string) ([]byte, error) 148SafeWriteReader(path string, r io.Reader) (err error) 149TempDir(dir, prefix string) (name string, err error) 150TempFile(dir, prefix string) (f File, err error) 151Walk(root string, walkFn filepath.WalkFunc) error 152WriteFile(filename string, data []byte, perm os.FileMode) error 153WriteReader(path string, r io.Reader) (err error) 154``` 155For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero) 156 157They are available under two different approaches to use. You can either call 158them directly where the first parameter of each function will be the file 159system, or you can declare a new `Afero`, a custom type used to bind these 160functions as methods to a given filesystem. 161 162### Calling utilities directly 163 164```go 165fs := new(afero.MemMapFs) 166f, err := afero.TempFile(fs,"", "ioutil-test") 167 168``` 169 170### Calling via Afero 171 172```go 173fs := afero.NewMemMapFs() 174afs := &afero.Afero{Fs: fs} 175f, err := afs.TempFile("", "ioutil-test") 176``` 177 178## Using Afero for Testing 179 180There is a large benefit to using a mock filesystem for testing. It has a 181completely blank state every time it is initialized and can be easily 182reproducible regardless of OS. You could create files to your heart’s content 183and the file access would be fast while also saving you from all the annoying 184issues with deleting temporary files, Windows file locking, etc. The MemMapFs 185backend is perfect for testing. 186 187* Much faster than performing I/O operations on disk 188* Avoid security issues and permissions 189* Far more control. 'rm -rf /' with confidence 190* Test setup is far more easier to do 191* No test cleanup needed 192 193One way to accomplish this is to define a variable as mentioned above. 194In your application this will be set to afero.NewOsFs() during testing you 195can set it to afero.NewMemMapFs(). 196 197It wouldn't be uncommon to have each test initialize a blank slate memory 198backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere 199appropriate in my application code. This approach ensures that Tests are order 200independent, with no test relying on the state left by an earlier test. 201 202Then in my tests I would initialize a new MemMapFs for each test: 203```go 204func TestExist(t *testing.T) { 205 appFS := afero.NewMemMapFs() 206 // create test files and directories 207 appFS.MkdirAll("src/a", 0755) 208 afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644) 209 afero.WriteFile(appFS, "src/c", []byte("file c"), 0644) 210 name := "src/c" 211 _, err := appFS.Stat(name) 212 if os.IsNotExist(err) { 213 t.Errorf("file \"%s\" does not exist.\n", name) 214 } 215} 216``` 217 218# Available Backends 219 220## Operating System Native 221 222### OsFs 223 224The first is simply a wrapper around the native OS calls. This makes it 225very easy to use as all of the calls are the same as the existing OS 226calls. It also makes it trivial to have your code use the OS during 227operation and a mock filesystem during testing or as needed. 228 229```go 230appfs := afero.NewOsFs() 231appfs.MkdirAll("src/a", 0755) 232``` 233 234## Memory Backed Storage 235 236### MemMapFs 237 238Afero also provides a fully atomic memory backed filesystem perfect for use in 239mocking and to speed up unnecessary disk io when persistence isn’t 240necessary. It is fully concurrent and will work within go routines 241safely. 242 243```go 244mm := afero.NewMemMapFs() 245mm.MkdirAll("src/a", 0755) 246``` 247 248#### InMemoryFile 249 250As part of MemMapFs, Afero also provides an atomic, fully concurrent memory 251backed file implementation. This can be used in other memory backed file 252systems with ease. Plans are to add a radix tree memory stored file 253system using InMemoryFile. 254 255## Network Interfaces 256 257### SftpFs 258 259Afero has experimental support for secure file transfer protocol (sftp). Which can 260be used to perform file operations over a encrypted channel. 261 262## Filtering Backends 263 264### BasePathFs 265 266The BasePathFs restricts all operations to a given path within an Fs. 267The given file name to the operations on this Fs will be prepended with 268the base path before calling the source Fs. 269 270```go 271bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path") 272``` 273 274### ReadOnlyFs 275 276A thin wrapper around the source Fs providing a read only view. 277 278```go 279fs := afero.NewReadOnlyFs(afero.NewOsFs()) 280_, err := fs.Create("/file.txt") 281// err = syscall.EPERM 282``` 283 284# RegexpFs 285 286A filtered view on file names, any file NOT matching 287the passed regexp will be treated as non-existing. 288Files not matching the regexp provided will not be created. 289Directories are not filtered. 290 291```go 292fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`)) 293_, err := fs.Create("/file.html") 294// err = syscall.ENOENT 295``` 296 297### HttpFs 298 299Afero provides an http compatible backend which can wrap any of the existing 300backends. 301 302The Http package requires a slightly specific version of Open which 303returns an http.File type. 304 305Afero provides an httpFs file system which satisfies this requirement. 306Any Afero FileSystem can be used as an httpFs. 307 308```go 309httpFs := afero.NewHttpFs(<ExistingFS>) 310fileserver := http.FileServer(httpFs.Dir(<PATH>)) 311http.Handle("/", fileserver) 312``` 313 314## Composite Backends 315 316Afero provides the ability have two filesystems (or more) act as a single 317file system. 318 319### CacheOnReadFs 320 321The CacheOnReadFs will lazily make copies of any accessed files from the base 322layer into the overlay. Subsequent reads will be pulled from the overlay 323directly permitting the request is within the cache duration of when it was 324created in the overlay. 325 326If the base filesystem is writeable, any changes to files will be 327done first to the base, then to the overlay layer. Write calls to open file 328handles like `Write()` or `Truncate()` to the overlay first. 329 330To writing files to the overlay only, you can use the overlay Fs directly (not 331via the union Fs). 332 333Cache files in the layer for the given time.Duration, a cache duration of 0 334means "forever" meaning the file will not be re-requested from the base ever. 335 336A read-only base will make the overlay also read-only but still copy files 337from the base to the overlay when they're not present (or outdated) in the 338caching layer. 339 340```go 341base := afero.NewOsFs() 342layer := afero.NewMemMapFs() 343ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second) 344``` 345 346### CopyOnWriteFs() 347 348The CopyOnWriteFs is a read only base file system with a potentially 349writeable layer on top. 350 351Read operations will first look in the overlay and if not found there, will 352serve the file from the base. 353 354Changes to the file system will only be made in the overlay. 355 356Any attempt to modify a file found only in the base will copy the file to the 357overlay layer before modification (including opening a file with a writable 358handle). 359 360Removing and Renaming files present only in the base layer is not currently 361permitted. If a file is present in the base layer and the overlay, only the 362overlay will be removed/renamed. 363 364```go 365 base := afero.NewOsFs() 366 roBase := afero.NewReadOnlyFs(base) 367 ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs()) 368 369 fh, _ = ufs.Create("/home/test/file2.txt") 370 fh.WriteString("This is a test") 371 fh.Close() 372``` 373 374In this example all write operations will only occur in memory (MemMapFs) 375leaving the base filesystem (OsFs) untouched. 376 377 378## Desired/possible backends 379 380The following is a short list of possible backends we hope someone will 381implement: 382 383* SSH 384* S3 385 386# About the project 387 388## What's in the name 389 390Afero comes from the latin roots Ad-Facere. 391 392**"Ad"** is a prefix meaning "to". 393 394**"Facere"** is a form of the root "faciō" making "make or do". 395 396The literal meaning of afero is "to make" or "to do" which seems very fitting 397for a library that allows one to make files and directories and do things with them. 398 399The English word that shares the same roots as Afero is "affair". Affair shares 400the same concept but as a noun it means "something that is made or done" or "an 401object of a particular type". 402 403It's also nice that unlike some of my other libraries (hugo, cobra, viper) it 404Googles very well. 405 406## Release Notes 407 408See the [Releases Page](https://github.com/spf13/afero/releases). 409 410## Contributing 411 4121. Fork it 4132. Create your feature branch (`git checkout -b my-new-feature`) 4143. Commit your changes (`git commit -am 'Add some feature'`) 4154. Push to the branch (`git push origin my-new-feature`) 4165. Create new Pull Request 417 418## Contributors 419 420Names in no particular order: 421 422* [spf13](https://github.com/spf13) 423* [jaqx0r](https://github.com/jaqx0r) 424* [mbertschler](https://github.com/mbertschler) 425* [xor-gate](https://github.com/xor-gate) 426 427## License 428 429Afero is released under the Apache 2.0 license. See 430[LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt) 431