1![afero logo-sm](https://cloud.githubusercontent.com/assets/173412/11490338/d50e16dc-97a5-11e5-8b12-019a300d0fcb.png) 2 3A FileSystem Abstraction System for Go 4 5[![Build Status](https://travis-ci.org/spf13/afero.svg)](https://travis-ci.org/spf13/afero) [![Build status](https://ci.appveyor.com/api/projects/status/github/spf13/afero?branch=master&svg=true)](https://ci.appveyor.com/project/spf13/afero) [![GoDoc](https://godoc.org/github.com/spf13/afero?status.svg)](https://godoc.org/github.com/spf13/afero) [![Join the chat at https://gitter.im/spf13/afero](https://badges.gitter.im/Dev%20Chat.svg)](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) 6 7# Overview 8 9Afero is an filesystem framework providing a simple, uniform and universal API 10interacting with any filesystem, as an abstraction layer providing interfaces, 11types and methods. Afero has an exceptionally clean interface and simple design 12without needless constructors or initialization methods. 13 14Afero is also a library providing a base set of interoperable backend 15filesystems that make it easy to work with afero while retaining all the power 16and benefit of the os and ioutil packages. 17 18Afero provides significant improvements over using the os package alone, most 19notably the ability to create mock and testing filesystems without relying on the disk. 20 21It is suitable for use in a any situation where you would consider using the OS 22package as it provides an additional abstraction that makes it easy to use a 23memory backed file system during testing. It also adds support for the http 24filesystem for full interoperability. 25 26 27## Afero Features 28 29* A single consistent API for accessing a variety of filesystems 30* Interoperation between a variety of file system types 31* A set of interfaces to encourage and enforce interoperability between backends 32* An atomic cross platform memory backed file system 33* Support for compositional (union) file systems by combining multiple file systems acting as one 34* Specialized backends which modify existing filesystems (Read Only, Regexp filtered) 35* A set of utility functions ported from io, ioutil & hugo to be afero aware 36 37 38# Using Afero 39 40Afero is easy to use and easier to adopt. 41 42A few different ways you could use Afero: 43 44* Use the interfaces alone to define you own file system. 45* Wrap for the OS packages. 46* Define different filesystems for different parts of your application. 47* Use Afero for mock filesystems while testing 48 49## Step 1: Install Afero 50 51First use go get to install the latest version of the library. 52 53 $ go get github.com/spf13/afero 54 55Next include Afero in your application. 56```go 57import "github.com/spf13/afero" 58``` 59 60## Step 2: Declare a backend 61 62First define a package variable and set it to a pointer to a filesystem. 63```go 64var AppFs = afero.NewMemMapFs() 65 66or 67 68var AppFs = afero.NewOsFs() 69``` 70It is important to note that if you repeat the composite literal you 71will be using a completely new and isolated filesystem. In the case of 72OsFs it will still use the same underlying filesystem but will reduce 73the ability to drop in other filesystems as desired. 74 75## Step 3: Use it like you would the OS package 76 77Throughout your application use any function and method like you normally 78would. 79 80So if my application before had: 81```go 82os.Open('/tmp/foo') 83``` 84We would replace it with: 85```go 86AppFs.Open('/tmp/foo') 87``` 88 89`AppFs` being the variable we defined above. 90 91 92## List of all available functions 93 94File System Methods Available: 95```go 96Chmod(name string, mode os.FileMode) : error 97Chtimes(name string, atime time.Time, mtime time.Time) : error 98Create(name string) : File, error 99Mkdir(name string, perm os.FileMode) : error 100MkdirAll(path string, perm os.FileMode) : error 101Name() : string 102Open(name string) : File, error 103OpenFile(name string, flag int, perm os.FileMode) : File, error 104Remove(name string) : error 105RemoveAll(path string) : error 106Rename(oldname, newname string) : error 107Stat(name string) : os.FileInfo, error 108``` 109File Interfaces and Methods Available: 110```go 111io.Closer 112io.Reader 113io.ReaderAt 114io.Seeker 115io.Writer 116io.WriterAt 117 118Name() : string 119Readdir(count int) : []os.FileInfo, error 120Readdirnames(n int) : []string, error 121Stat() : os.FileInfo, error 122Sync() : error 123Truncate(size int64) : error 124WriteString(s string) : ret int, err error 125``` 126In some applications it may make sense to define a new package that 127simply exports the file system variable for easy access from anywhere. 128 129## Using Afero's utility functions 130 131Afero provides a set of functions to make it easier to use the underlying file systems. 132These functions have been primarily ported from io & ioutil with some developed for Hugo. 133 134The afero utilities support all afero compatible backends. 135 136The list of utilities includes: 137 138```go 139DirExists(path string) (bool, error) 140Exists(path string) (bool, error) 141FileContainsBytes(filename string, subslice []byte) (bool, error) 142GetTempDir(subPath string) string 143IsDir(path string) (bool, error) 144IsEmpty(path string) (bool, error) 145ReadDir(dirname string) ([]os.FileInfo, error) 146ReadFile(filename string) ([]byte, error) 147SafeWriteReader(path string, r io.Reader) (err error) 148TempDir(dir, prefix string) (name string, err error) 149TempFile(dir, prefix string) (f File, err error) 150Walk(root string, walkFn filepath.WalkFunc) error 151WriteFile(filename string, data []byte, perm os.FileMode) error 152WriteReader(path string, r io.Reader) (err error) 153``` 154For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero) 155 156They are available under two different approaches to use. You can either call 157them directly where the first parameter of each function will be the file 158system, or you can declare a new `Afero`, a custom type used to bind these 159functions as methods to a given filesystem. 160 161### Calling utilities directly 162 163```go 164fs := new(afero.MemMapFs) 165f, err := afero.TempFile(fs,"", "ioutil-test") 166 167``` 168 169### Calling via Afero 170 171```go 172fs := afero.NewMemMapFs() 173afs := &afero.Afero{Fs: fs} 174f, err := afs.TempFile("", "ioutil-test") 175``` 176 177## Using Afero for Testing 178 179There is a large benefit to using a mock filesystem for testing. It has a 180completely blank state every time it is initialized and can be easily 181reproducible regardless of OS. You could create files to your heart’s content 182and the file access would be fast while also saving you from all the annoying 183issues with deleting temporary files, Windows file locking, etc. The MemMapFs 184backend is perfect for testing. 185 186* Much faster than performing I/O operations on disk 187* Avoid security issues and permissions 188* Far more control. 'rm -rf /' with confidence 189* Test setup is far more easier to do 190* No test cleanup needed 191 192One way to accomplish this is to define a variable as mentioned above. 193In your application this will be set to afero.NewOsFs() during testing you 194can set it to afero.NewMemMapFs(). 195 196It wouldn't be uncommon to have each test initialize a blank slate memory 197backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere 198appropriate in my application code. This approach ensures that Tests are order 199independent, with no test relying on the state left by an earlier test. 200 201Then in my tests I would initialize a new MemMapFs for each test: 202```go 203func TestExist(t *testing.T) { 204 appFS := afero.NewMemMapFs() 205 // create test files and directories 206 appFS.MkdirAll("src/a", 0755) 207 afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644) 208 afero.WriteFile(appFS, "src/c", []byte("file c"), 0644) 209 name := "src/c" 210 _, err := appFS.Stat(name) 211 if os.IsNotExist(err) { 212 t.Errorf("file \"%s\" does not exist.\n", name) 213 } 214} 215``` 216 217# Available Backends 218 219## Operating System Native 220 221### OsFs 222 223The first is simply a wrapper around the native OS calls. This makes it 224very easy to use as all of the calls are the same as the existing OS 225calls. It also makes it trivial to have your code use the OS during 226operation and a mock filesystem during testing or as needed. 227 228```go 229appfs := afero.NewOsFs() 230appfs.MkdirAll("src/a", 0755)) 231``` 232 233## Memory Backed Storage 234 235### MemMapFs 236 237Afero also provides a fully atomic memory backed filesystem perfect for use in 238mocking and to speed up unnecessary disk io when persistence isn’t 239necessary. It is fully concurrent and will work within go routines 240safely. 241 242```go 243mm := afero.NewMemMapFs() 244mm.MkdirAll("src/a", 0755)) 245``` 246 247#### InMemoryFile 248 249As part of MemMapFs, Afero also provides an atomic, fully concurrent memory 250backed file implementation. This can be used in other memory backed file 251systems with ease. Plans are to add a radix tree memory stored file 252system using InMemoryFile. 253 254## Network Interfaces 255 256### SftpFs 257 258Afero has experimental support for secure file transfer protocol (sftp). Which can 259be used to perform file operations over a encrypted channel. 260 261## Filtering Backends 262 263### BasePathFs 264 265The BasePathFs restricts all operations to a given path within an Fs. 266The given file name to the operations on this Fs will be prepended with 267the base path before calling the source Fs. 268 269```go 270bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path") 271``` 272 273### ReadOnlyFs 274 275A thin wrapper around the source Fs providing a read only view. 276 277```go 278fs := afero.NewReadOnlyFs(afero.NewOsFs()) 279_, err := fs.Create("/file.txt") 280// err = syscall.EPERM 281``` 282 283# RegexpFs 284 285A filtered view on file names, any file NOT matching 286the passed regexp will be treated as non-existing. 287Files not matching the regexp provided will not be created. 288Directories are not filtered. 289 290```go 291fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`)) 292_, err := fs.Create("/file.html") 293// err = syscall.ENOENT 294``` 295 296### HttpFs 297 298Afero provides an http compatible backend which can wrap any of the existing 299backends. 300 301The Http package requires a slightly specific version of Open which 302returns an http.File type. 303 304Afero provides an httpFs file system which satisfies this requirement. 305Any Afero FileSystem can be used as an httpFs. 306 307```go 308httpFs := afero.NewHttpFs(<ExistingFS>) 309fileserver := http.FileServer(httpFs.Dir(<PATH>))) 310http.Handle("/", fileserver) 311``` 312 313## Composite Backends 314 315Afero provides the ability have two filesystems (or more) act as a single 316file system. 317 318### CacheOnReadFs 319 320The CacheOnReadFs will lazily make copies of any accessed files from the base 321layer into the overlay. Subsequent reads will be pulled from the overlay 322directly permitting the request is within the cache duration of when it was 323created in the overlay. 324 325If the base filesystem is writeable, any changes to files will be 326done first to the base, then to the overlay layer. Write calls to open file 327handles like `Write()` or `Truncate()` to the overlay first. 328 329To writing files to the overlay only, you can use the overlay Fs directly (not 330via the union Fs). 331 332Cache files in the layer for the given time.Duration, a cache duration of 0 333means "forever" meaning the file will not be re-requested from the base ever. 334 335A read-only base will make the overlay also read-only but still copy files 336from the base to the overlay when they're not present (or outdated) in the 337caching layer. 338 339```go 340base := afero.NewOsFs() 341layer := afero.NewMemMapFs() 342ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second) 343``` 344 345### CopyOnWriteFs() 346 347The CopyOnWriteFs is a read only base file system with a potentially 348writeable layer on top. 349 350Read operations will first look in the overlay and if not found there, will 351serve the file from the base. 352 353Changes to the file system will only be made in the overlay. 354 355Any attempt to modify a file found only in the base will copy the file to the 356overlay layer before modification (including opening a file with a writable 357handle). 358 359Removing and Renaming files present only in the base layer is not currently 360permitted. If a file is present in the base layer and the overlay, only the 361overlay will be removed/renamed. 362 363```go 364 base := afero.NewOsFs() 365 roBase := afero.NewReadOnlyFs(base) 366 ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs()) 367 368 fh, _ = ufs.Create("/home/test/file2.txt") 369 fh.WriteString("This is a test") 370 fh.Close() 371``` 372 373In this example all write operations will only occur in memory (MemMapFs) 374leaving the base filesystem (OsFs) untouched. 375 376 377## Desired/possible backends 378 379The following is a short list of possible backends we hope someone will 380implement: 381 382* SSH 383* ZIP 384* TAR 385* S3 386 387# About the project 388 389## What's in the name 390 391Afero comes from the latin roots Ad-Facere. 392 393**"Ad"** is a prefix meaning "to". 394 395**"Facere"** is a form of the root "faciō" making "make or do". 396 397The literal meaning of afero is "to make" or "to do" which seems very fitting 398for a library that allows one to make files and directories and do things with them. 399 400The English word that shares the same roots as Afero is "affair". Affair shares 401the same concept but as a noun it means "something that is made or done" or "an 402object of a particular type". 403 404It's also nice that unlike some of my other libraries (hugo, cobra, viper) it 405Googles very well. 406 407## Release Notes 408 409* **0.10.0** 2015.12.10 410 * Full compatibility with Windows 411 * Introduction of afero utilities 412 * Test suite rewritten to work cross platform 413 * Normalize paths for MemMapFs 414 * Adding Sync to the file interface 415 * **Breaking Change** Walk and ReadDir have changed parameter order 416 * Moving types used by MemMapFs to a subpackage 417 * General bugfixes and improvements 418* **0.9.0** 2015.11.05 419 * New Walk function similar to filepath.Walk 420 * MemMapFs.OpenFile handles O_CREATE, O_APPEND, O_TRUNC 421 * MemMapFs.Remove now really deletes the file 422 * InMemoryFile.Readdir and Readdirnames work correctly 423 * InMemoryFile functions lock it for concurrent access 424 * Test suite improvements 425* **0.8.0** 2014.10.28 426 * First public version 427 * Interfaces feel ready for people to build using 428 * Interfaces satisfy all known uses 429 * MemMapFs passes the majority of the OS test suite 430 * OsFs passes the majority of the OS test suite 431 432## Contributing 433 4341. Fork it 4352. Create your feature branch (`git checkout -b my-new-feature`) 4363. Commit your changes (`git commit -am 'Add some feature'`) 4374. Push to the branch (`git push origin my-new-feature`) 4385. Create new Pull Request 439 440## Contributors 441 442Names in no particular order: 443 444* [spf13](https://github.com/spf13) 445* [jaqx0r](https://github.com/jaqx0r) 446* [mbertschler](https://github.com/mbertschler) 447* [xor-gate](https://github.com/xor-gate) 448 449## License 450 451Afero is released under the Apache 2.0 license. See 452[LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt) 453