1# Garbage Collection 2 3`containerd` has a garbage collector capable of removing resources which are no 4longer being used. The client is responsible for ensuring that all resources 5which are created are either used or held by a lease at all times, else they 6will be considered eligible for removal. The Go client library 7(`github.com/containerd/containerd`) has built-in behavior to ensure resources 8are properly tracked and leased. However, the lifecycles of leases are the 9responsibility of the caller of the library. The `containerd` daemon has strict 10resource management and will garbage collect any unused resource. 11 12## What is a lease? 13 14Leases are a resource in `containerd` which are created by clients and are used 15to reference other resources such as snapshots and content. Leases may be 16configured with an expiration or deleted by clients once it has completed an 17operation. Leases are needed to inform the `containerd` daemon that a resource 18may be used in the future after the client completes an operation, even though 19it currently is not seen as utilized. 20 21## How to use leases 22 23### using Go client 24 25The best way to use leases is to add it to a Go context immediately after the 26context is created. Normally the lifespan of a lease will be the same as the 27lifecycle of a Go context. 28 29```.go 30 ctx, done, err := client.WithLease(ctx) 31 if err != nil { 32 return err 33 } 34 defer done(ctx) 35``` 36 37This will create a lease which will defer its own deletion and have a default 38expiry of 24 hours (in case the process dies before defer). For most use cases, 39this is enough and no more thought needs to be put into leases. 40 41_But, of course, more complicated use cases are supported..._ 42 43If the program or lease are intended to be longer lived, instead of the very 44easy `client.WithLease`, the lease manager can be used directly. This also 45allows for setting custom labels on the lease or manipulating its resources. 46Use `client.LeasesService()` to get a [lease Manager](https://godoc.org/github.com/containerd/containerd/leases#Manager) 47which can be used to create, list, and delete leases as well as manage the 48referenced resources for that lease. 49 50```.go 51 manager := client.LeasesService() 52 53 // this lease will never expire 54 // Use `leases.WithExpiration` to make it expire 55 // Use `leases.WithLabels` to apply any labels 56 l, err := manager.Create(ctx, leases.WithRandomID()) 57 if err != nil { 58 return err 59 } 60 61 // Update current context to add lease 62 ctx = leases.WithLease(ctx, l.ID) 63 64 // Do something, lease will be used... 65 66 // Delete lease at any time, or track it to delete later 67 if err := ls.Delete(ctx, l); err != nil { 68 return err 69 } 70``` 71 72 73### using gRPC 74 75The lease is not an explicit field in the API (except of course the leases 76service), but rather an optional field any API service can use. Leases can 77be set on any gRPC service endpoint using a gRPC header. Set the 78gRPC header `containerd-lease` to the lease identifier and the API 79service will operate within that lease context. 80 81To manage the creation and deletion of leases, use the leases gRPC service. 82 83## Garbage Collection labels 84 85The garbage collection defines relationships between resources in two different 86ways, by type specific resources properties, and by resource labels. The type 87specific properties do not need to be managed by the user as they are part of 88the natural structure of a resource (i.e. a container's snapshot, a snapshot's 89parent, an image's target, etc). However resources may have relationships which 90are not defined by `containerd`, but rather by the client. For example, an OCI 91image has a manifest which references a config file and layer tars. These 92resources are stored in `containerd` as generic blobs, it is the client that 93understands the relationships between these blobs and sets them up using labels 94on the content resources. 95 96Resource labels can also be used to cue the garbage collector on other 97properties, such as expiry, whether an object should be kept without any 98reference or limit what is referenced. 99 100The supported garbage collection labels are: 101 102| Label key | Label value | Supported Resources | Description | 103|---|---|---|---| 104| `containerd.io/gc.root` | _nonempty_ | Content, Snapshots | Keep this object and anything it references. (Clients may set this to a [rfc3339](https://tools.ietf.org/html/rfc3339) timestamp to indicate when this value was set, however, the garbage collector does not parse the value) | 105| `containerd.io/gc.ref.snapshot.<snapshotter>` | `<identifier>` | Content, Snapshots | Resource references the given snapshot `<identifier>` for the snapshotter `<snapshotter>` | 106| `containerd.io/gc.ref.content` | _digest_ | Content, Snapshots, Images, Containers | Resource references the given content blob | 107| `containerd.io/gc.ref.content.<user defined>` | _digest_ | Content, Snapshots, Images, Containers | Resource references the given content blob with a `<user defined>` label key | 108| `containerd.io/gc.expire` | _timestamp_ formatted as [rfc3339](https://tools.ietf.org/html/rfc3339) | Leases | When to expire the lease. The garbage collector will delete the lease after expiration. | 109| `containerd.io/gc.flat` | _nonempty_ | Leases | Ignore label references of leased resources. This only applies when the reference is originating from the lease, if the leased resources are referenced elsewhere, then their label references will be used. | 110 111## Garbage Collection configuration 112 113The garbage collector (gc) is scheduled on a background goroutine and runs based 114on a number of configurable factors. By default the garbage collector will 115attempt to keep the database unlocked 98% of the time based on prior 116calculations of lock time from garbage collection. Also by default, the garbage 117collector will not schedule itself if no deletions occurred or after every 100 118database writes. 119 120The garbage collection scheduler considers the time the database is locked 121as the pause time. The garbage collection will take longer than this when 122resources are being removed, for example cleaning up snapshots may be slow. 123The scheduler will only schedule after the whole garbage collection is 124completed but use the average pause time for determining when the next run 125attempt is. 126 127Garbage collection may be configured using the `containerd` daemon's 128configuration file, usually at `/etc/containerd/config.toml`. The 129configuration is under the `scheduler` plugin. 130 131### Configuration parameters 132 133| Configuration | Default | Description | 134|---|---|---| 135| `pause_threshold` | 0.02 | Represents the maximum amount of time gc should be scheduled based on the average pause time. A maximum value of .5 (50%) is enforced to prevent over scheduling. | 136| `deletion_threshold` | 0 | A threshold of number of deletes to immediately trigger gc. 0 means a gc will not be triggered by deletion count, however a deletion will ensure the next scheduled gc will run. | 137| `mutation_threshold` | 100 | A threshold for running gc after the given number of database mutations. Note any mutation which performed a delete will always cause gc to run, this case handles more rare events such as label reference removal. | 138| `schedule_delay` | "0ms" | The delay between a trigger event and running gc. A non-zero value can be used when mutations may quickly burst. | 139| `startup_delay` | "100ms" | The delay before running the initial garbage collection after daemon startup. This should be run after other startup processes have completed and no gc can be scheduled before this delay. | 140 141The default configuration is represented as... 142```.toml 143[plugins] 144 [plugins.scheduler] 145 pause_threshold = 0.02 146 deletion_threshold = 0 147 mutation_threshold = 100 148 schedule_delay = "0ms" 149 startup_delay = "100ms" 150``` 151 152## Synchronous Garbage Collection 153 154In addition to garbage collections done through the scheduler, the client 155may also request a garbage collection during resource removal. In this case, 156the garbage collection will be scheduled immediately (or after `schedule_delay` 157when configured to non-zero). The service will not return until the garbage 158collection has completed. This is currently supported on removal of images and 159leases. Use [`images.SynchronousDelete()`](https://godoc.org/github.com/containerd/containerd/images#SynchronousDelete) 160for [`images.Store`](https://godoc.org/github.com/containerd/containerd/images#Store)'s 161`Delete` and 162[`leases.SynchronousDelete`](https://godoc.org/github.com/containerd/containerd/leases#SynchronousDelete) 163for [`leases.Manager`](https://godoc.org/github.com/containerd/containerd/leases#Manager)'s 164`Delete`. 165