1--- 2title: "dockerd" 3aliases: ["/engine/reference/commandline/daemon/"] 4description: "The daemon command description and usage" 5keywords: "container, daemon, runtime" 6--- 7 8<!-- This file is maintained within the docker/cli GitHub 9 repository at https://github.com/docker/cli/. Make all 10 pull requests against that repo. If you see this file in 11 another repository, consider it read-only there, as it will 12 periodically be overwritten by the definitive file. Pull 13 requests which include edits to this file in other repositories 14 will be rejected. 15--> 16 17# daemon 18 19```markdown 20Usage: dockerd COMMAND 21 22A self-sufficient runtime for containers. 23 24Options: 25 --add-runtime runtime Register an additional OCI compatible runtime (default []) 26 --allow-nondistributable-artifacts list Push nondistributable artifacts to specified registries (default []) 27 --api-cors-header string Set CORS headers in the Engine API 28 --authorization-plugin list Authorization plugins to load (default []) 29 --bip string Specify network bridge IP 30 -b, --bridge string Attach containers to a network bridge 31 --cgroup-parent string Set parent cgroup for all containers 32 --cluster-advertise string Address or interface name to advertise 33 --cluster-store string URL of the distributed storage backend 34 --cluster-store-opt map Set cluster store options (default map[]) 35 --config-file string Daemon configuration file (default "/etc/docker/daemon.json") 36 --containerd string Path to containerd socket 37 --cpu-rt-period int Limit the CPU real-time period in microseconds 38 --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds 39 --data-root string Root directory of persistent Docker state (default "/var/lib/docker") 40 -D, --debug Enable debug mode 41 --default-gateway ip Container default gateway IPv4 address 42 --default-gateway-v6 ip Container default gateway IPv6 address 43 --default-address-pool Set the default address pool for local node networks 44 --default-runtime string Default OCI runtime for containers (default "runc") 45 --default-ulimit ulimit Default ulimits for containers (default []) 46 --dns list DNS server to use (default []) 47 --dns-opt list DNS options to use (default []) 48 --dns-search list DNS search domains to use (default []) 49 --exec-opt list Runtime execution options (default []) 50 --exec-root string Root directory for execution state files (default "/var/run/docker") 51 --experimental Enable experimental features 52 --fixed-cidr string IPv4 subnet for fixed IPs 53 --fixed-cidr-v6 string IPv6 subnet for fixed IPs 54 -G, --group string Group for the unix socket (default "docker") 55 --help Print usage 56 -H, --host list Daemon socket(s) to connect to (default []) 57 --icc Enable inter-container communication (default true) 58 --init Run an init in the container to forward signals and reap processes 59 --init-path string Path to the docker-init binary 60 --insecure-registry list Enable insecure registry communication (default []) 61 --ip ip Default IP when binding container ports (default 0.0.0.0) 62 --ip-forward Enable net.ipv4.ip_forward (default true) 63 --ip-masq Enable IP masquerading (default true) 64 --iptables Enable addition of iptables rules (default true) 65 --ipv6 Enable IPv6 networking 66 --label list Set key=value labels to the daemon (default []) 67 --live-restore Enable live restore of docker when containers are still running 68 --log-driver string Default driver for container logs (default "json-file") 69 -l, --log-level string Set the logging level ("debug", "info", "warn", "error", "fatal") (default "info") 70 --log-opt map Default log driver options for containers (default map[]) 71 --max-concurrent-downloads int Set the max concurrent downloads for each pull (default 3) 72 --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) 73 --metrics-addr string Set default address and port to serve the metrics api on 74 --mtu int Set the containers network MTU 75 --node-generic-resources list Advertise user-defined resource 76 --no-new-privileges Set no-new-privileges by default for new containers 77 --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) 78 -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") 79 --raw-logs Full timestamps without ANSI coloring 80 --registry-mirror list Preferred Docker registry mirror (default []) 81 --seccomp-profile string Path to seccomp profile 82 --selinux-enabled Enable selinux support 83 --shutdown-timeout int Set the default shutdown timeout (default 15) 84 -s, --storage-driver string Storage driver to use 85 --storage-opt list Storage driver options (default []) 86 --swarm-default-advertise-addr string Set default address or interface for swarm advertised address 87 --tls Use TLS; implied by --tlsverify 88 --tlscacert string Trust certs signed only by this CA (default "~/.docker/ca.pem") 89 --tlscert string Path to TLS certificate file (default "~/.docker/cert.pem") 90 --tlskey string Path to TLS key file (default ~/.docker/key.pem") 91 --tlsverify Use TLS and verify the remote 92 --userland-proxy Use userland proxy for loopback traffic (default true) 93 --userland-proxy-path string Path to the userland proxy binary 94 --userns-remap string User/Group setting for user namespaces 95 -v, --version Print version information and quit 96``` 97 98Options with [] may be specified multiple times. 99 100## Description 101 102`dockerd` is the persistent process that manages containers. Docker 103uses different binaries for the daemon and client. To run the daemon you 104type `dockerd`. 105 106To run the daemon with debug output, use `dockerd -D` or add `"debug": true` to 107the `daemon.json` file. 108 109> **Note**: In Docker 1.13 and higher, enable experimental features by starting 110> `dockerd` with the `--experimental` flag or adding `"experimental": true` to the 111> `daemon.json` file. In earlier Docker versions, a different build was required 112> to enable experimental features. 113 114## Examples 115 116### Daemon socket option 117 118The Docker daemon can listen for [Docker Engine API](../api/) 119requests via three different types of Socket: `unix`, `tcp`, and `fd`. 120 121By default, a `unix` domain socket (or IPC socket) is created at 122`/var/run/docker.sock`, requiring either `root` permission, or `docker` group 123membership. 124 125If you need to access the Docker daemon remotely, you need to enable the `tcp` 126Socket. Beware that the default setup provides un-encrypted and 127un-authenticated direct access to the Docker daemon - and should be secured 128either using the [built in HTTPS encrypted socket](https://docs.docker.com/engine/security/https/), or by 129putting a secure web proxy in front of it. You can listen on port `2375` on all 130network interfaces with `-H tcp://0.0.0.0:2375`, or on a particular network 131interface using its IP address: `-H tcp://192.168.59.103:2375`. It is 132conventional to use port `2375` for un-encrypted, and port `2376` for encrypted 133communication with the daemon. 134 135> **Note**: If you're using an HTTPS encrypted socket, keep in mind that only 136> TLS1.0 and greater are supported. Protocols SSLv3 and under are not 137> supported anymore for security reasons. 138 139On Systemd based systems, you can communicate with the daemon via 140[Systemd socket activation](http://0pointer.de/blog/projects/socket-activation.html), 141use `dockerd -H fd://`. Using `fd://` will work perfectly for most setups but 142you can also specify individual sockets: `dockerd -H fd://3`. If the 143specified socket activated files aren't found, then Docker will exit. You can 144find examples of using Systemd socket activation with Docker and Systemd in the 145[Docker source tree](https://github.com/docker/docker/tree/master/contrib/init/systemd/). 146 147You can configure the Docker daemon to listen to multiple sockets at the same 148time using multiple `-H` options: 149 150```bash 151# listen using the default unix socket, and on 2 specific IP addresses on this host. 152 153$ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2 154``` 155 156The Docker client will honor the `DOCKER_HOST` environment variable to set the 157`-H` flag for the client. Use **one** of the following commands: 158 159```bash 160$ docker -H tcp://0.0.0.0:2375 ps 161``` 162 163```bash 164$ export DOCKER_HOST="tcp://0.0.0.0:2375" 165 166$ docker ps 167``` 168 169Setting the `DOCKER_TLS_VERIFY` environment variable to any value other than 170the empty string is equivalent to setting the `--tlsverify` flag. The following 171are equivalent: 172 173```bash 174$ docker --tlsverify ps 175# or 176$ export DOCKER_TLS_VERIFY=1 177$ docker ps 178``` 179 180The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` 181environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes 182precedence over `HTTP_PROXY`. 183 184Starting with Docker 18.09, the Docker client supports connecting to a remote 185daemon via SSH: 186 187``` 188$ docker -H ssh://me@example.com:22 ps 189$ docker -H ssh://me@example.com ps 190$ docker -H ssh://example.com ps 191``` 192 193To use SSH connection, you need to set up `ssh` so that it can reach the 194remote host with public key authentication. Password authentication is not 195supported. If your key is protected with passphrase, you need to set up 196`ssh-agent`. 197 198Also, you need to have `docker` binary 18.09 or later on the daemon host. 199 200#### Bind Docker to another host/port or a Unix socket 201 202> **Warning**: 203> Changing the default `docker` daemon binding to a 204> TCP port or Unix *docker* user group will increase your security risks 205> by allowing non-root users to gain *root* access on the host. Make sure 206> you control access to `docker`. If you are binding 207> to a TCP port, anyone with access to that port has full Docker access; 208> so it is not advisable on an open network. 209 210With `-H` it is possible to make the Docker daemon to listen on a 211specific IP and port. By default, it will listen on 212`unix:///var/run/docker.sock` to allow only local connections by the 213*root* user. You *could* set it to `0.0.0.0:2375` or a specific host IP 214to give access to everybody, but that is **not recommended** because 215then it is trivial for someone to gain root access to the host where the 216daemon is running. 217 218Similarly, the Docker client can use `-H` to connect to a custom port. 219The Docker client will default to connecting to `unix:///var/run/docker.sock` 220on Linux, and `tcp://127.0.0.1:2376` on Windows. 221 222`-H` accepts host and port assignment in the following format: 223 224 tcp://[host]:[port][path] or unix://path 225 226For example: 227 228- `tcp://` -> TCP connection to `127.0.0.1` on either port `2376` when TLS encryption 229 is on, or port `2375` when communication is in plain text. 230- `tcp://host:2375` -> TCP connection on 231 host:2375 232- `tcp://host:2375/path` -> TCP connection on 233 host:2375 and prepend path to all requests 234- `unix://path/to/socket` -> Unix socket located 235 at `path/to/socket` 236 237`-H`, when empty, will default to the same value as 238when no `-H` was passed in. 239 240`-H` also accepts short form for TCP bindings: `host:` or `host:port` or `:port` 241 242Run Docker in daemon mode: 243 244```bash 245$ sudo <path to>/dockerd -H 0.0.0.0:5555 & 246``` 247 248Download an `ubuntu` image: 249 250```bash 251$ docker -H :5555 pull ubuntu 252``` 253 254You can use multiple `-H`, for example, if you want to listen on both 255TCP and a Unix socket 256 257```bash 258# Run docker in daemon mode 259$ sudo <path to>/dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock & 260# Download an ubuntu image, use default Unix socket 261$ docker pull ubuntu 262# OR use the TCP port 263$ docker -H tcp://127.0.0.1:2375 pull ubuntu 264``` 265 266### Daemon storage-driver 267 268On Linux, the Docker daemon has support for several different image layer storage 269drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay` and `overlay2`. 270 271The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that 272is unlikely to be merged into the main kernel. These are also known to cause 273some serious kernel crashes. However `aufs` allows containers to share 274executable and shared library memory, so is a useful choice when running 275thousands of containers with the same program or libraries. 276 277The `devicemapper` driver uses thin provisioning and Copy on Write (CoW) 278snapshots. For each devicemapper graph location – typically 279`/var/lib/docker/devicemapper` – a thin pool is created based on two block 280devices, one for data and one for metadata. By default, these block devices 281are created automatically by using loopback mounts of automatically created 282sparse files. Refer to [Devicemapper options](#devicemapper-options) below 283for a way how to customize this setup. 284[~jpetazzo/Resizing Docker containers with the Device Mapper plugin](http://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/) 285article explains how to tune your existing setup without the use of options. 286 287The `btrfs` driver is very fast for `docker build` - but like `devicemapper` 288does not share executable memory between devices. Use 289`dockerd -s btrfs -g /mnt/btrfs_partition`. 290 291The `zfs` driver is probably not as fast as `btrfs` but has a longer track record 292on stability. Thanks to `Single Copy ARC` shared blocks between clones will be 293cached only once. Use `dockerd -s zfs`. To select a different zfs filesystem 294set `zfs.fsname` option as described in [ZFS options](#zfs-options). 295 296The `overlay` is a very fast union filesystem. It is now merged in the main 297Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). `overlay` 298also supports page cache sharing, this means multiple containers accessing 299the same file can share a single page cache entry (or entries), it makes 300`overlay` as efficient with memory as `aufs` driver. Call 301`dockerd -s overlay` to use it. 302 303> **Note**: As promising as `overlay` is, the feature is still quite young and 304> should not be used in production. Most notably, using `overlay` can cause 305> excessive inode consumption (especially as the number of images grows), as 306> well as being incompatible with the use of RPMs. 307 308The `overlay2` uses the same fast union filesystem but takes advantage of 309[additional features](https://lkml.org/lkml/2015/2/11/106) added in Linux 310kernel 4.0 to avoid excessive inode consumption. Call `dockerd -s overlay2` 311to use it. 312 313> **Note**: Both `overlay` and `overlay2` are currently unsupported on `btrfs` 314> or any Copy on Write filesystem and should only be used over `ext4` partitions. 315 316On Windows, the Docker daemon supports a single image layer storage driver 317depending on the image platform: `windowsfilter` for Windows images, and 318`lcow` for Linux containers on Windows. 319 320### Options per storage driver 321 322Particular storage-driver can be configured with options specified with 323`--storage-opt` flags. Options for `devicemapper` are prefixed with `dm`, 324options for `zfs` start with `zfs`, options for `btrfs` start with `btrfs` 325and options for `lcow` start with `lcow`. 326 327#### Devicemapper options 328 329This is an example of the configuration file for devicemapper on Linux: 330 331```json 332{ 333 "storage-driver": "devicemapper", 334 "storage-opts": [ 335 "dm.thinpooldev=/dev/mapper/thin-pool", 336 "dm.use_deferred_deletion=true", 337 "dm.use_deferred_removal=true" 338 ] 339} 340``` 341 342##### `dm.thinpooldev` 343 344Specifies a custom block storage device to use for the thin pool. 345 346If using a block device for device mapper storage, it is best to use `lvm` 347to create and manage the thin-pool volume. This volume is then handed to Docker 348to exclusively create snapshot volumes needed for images and containers. 349 350Managing the thin-pool outside of Engine makes for the most feature-rich 351method of having Docker utilize device mapper thin provisioning as the 352backing storage for Docker containers. The highlights of the lvm-based 353thin-pool management feature include: automatic or interactive thin-pool 354resize support, dynamically changing thin-pool features, automatic thinp 355metadata checking when lvm activates the thin-pool, etc. 356 357As a fallback if no thin pool is provided, loopback files are 358created. Loopback is very slow, but can be used without any 359pre-configuration of storage. It is strongly recommended that you do 360not use loopback in production. Ensure your Engine daemon has a 361`--storage-opt dm.thinpooldev` argument provided. 362 363###### Example: 364 365```bash 366$ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool 367``` 368 369##### `dm.directlvm_device` 370 371As an alternative to providing a thin pool as above, Docker can setup a block 372device for you. 373 374###### Example: 375 376```bash 377$ sudo dockerd --storage-opt dm.directlvm_device=/dev/xvdf 378``` 379 380##### `dm.thinp_percent` 381 382Sets the percentage of passed in block device to use for storage. 383 384###### Example: 385 386```bash 387$ sudo dockerd --storage-opt dm.thinp_percent=95 388``` 389 390##### `dm.thinp_metapercent` 391 392Sets the percentage of the passed in block device to use for metadata storage. 393 394###### Example: 395 396```bash 397$ sudo dockerd --storage-opt dm.thinp_metapercent=1 398``` 399 400##### `dm.thinp_autoextend_threshold` 401 402Sets the value of the percentage of space used before `lvm` attempts to 403autoextend the available space [100 = disabled] 404 405###### Example: 406 407```bash 408$ sudo dockerd --storage-opt dm.thinp_autoextend_threshold=80 409``` 410 411##### `dm.thinp_autoextend_percent` 412 413Sets the value percentage value to increase the thin pool by when `lvm` 414attempts to autoextend the available space [100 = disabled] 415 416###### Example: 417 418```bash 419$ sudo dockerd --storage-opt dm.thinp_autoextend_percent=20 420``` 421 422 423##### `dm.basesize` 424 425Specifies the size to use when creating the base device, which limits the 426size of images and containers. The default value is 10G. Note, thin devices 427are inherently "sparse", so a 10G device which is mostly empty doesn't use 42810 GB of space on the pool. However, the filesystem will use more space for 429the empty case the larger the device is. 430 431The base device size can be increased at daemon restart which will allow 432all future images and containers (based on those new images) to be of the 433new base device size. 434 435###### Examples 436 437```bash 438$ sudo dockerd --storage-opt dm.basesize=50G 439``` 440 441This will increase the base device size to 50G. The Docker daemon will throw an 442error if existing base device size is larger than 50G. A user can use 443this option to expand the base device size however shrinking is not permitted. 444 445This value affects the system-wide "base" empty filesystem 446that may already be initialized and inherited by pulled images. Typically, 447a change to this value requires additional steps to take effect: 448 449 ```bash 450$ sudo service docker stop 451 452$ sudo rm -rf /var/lib/docker 453 454$ sudo service docker start 455``` 456 457 458##### `dm.loopdatasize` 459 460> **Note**: This option configures devicemapper loopback, which should not 461> be used in production. 462 463Specifies the size to use when creating the loopback file for the 464"data" device which is used for the thin pool. The default size is 465100G. The file is sparse, so it will not initially take up this 466much space. 467 468###### Example 469 470```bash 471$ sudo dockerd --storage-opt dm.loopdatasize=200G 472``` 473 474##### `dm.loopmetadatasize` 475 476> **Note**: This option configures devicemapper loopback, which should not 477> be used in production. 478 479Specifies the size to use when creating the loopback file for the 480"metadata" device which is used for the thin pool. The default size 481is 2G. The file is sparse, so it will not initially take up 482this much space. 483 484###### Example 485 486```bash 487$ sudo dockerd --storage-opt dm.loopmetadatasize=4G 488``` 489 490##### `dm.fs` 491 492Specifies the filesystem type to use for the base device. The supported 493options are "ext4" and "xfs". The default is "xfs" 494 495###### Example 496 497```bash 498$ sudo dockerd --storage-opt dm.fs=ext4 499``` 500 501##### `dm.mkfsarg` 502 503Specifies extra mkfs arguments to be used when creating the base device. 504 505###### Example 506 507```bash 508$ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" 509``` 510 511##### `dm.mountopt` 512 513Specifies extra mount options used when mounting the thin devices. 514 515###### Example 516 517```bash 518$ sudo dockerd --storage-opt dm.mountopt=nodiscard 519``` 520 521##### `dm.datadev` 522 523(Deprecated, use `dm.thinpooldev`) 524 525Specifies a custom blockdevice to use for data for the thin pool. 526 527If using a block device for device mapper storage, ideally both `datadev` and 528`metadatadev` should be specified to completely avoid using the loopback 529device. 530 531###### Example 532 533```bash 534$ sudo dockerd \ 535 --storage-opt dm.datadev=/dev/sdb1 \ 536 --storage-opt dm.metadatadev=/dev/sdc1 537``` 538 539##### `dm.metadatadev` 540 541(Deprecated, use `dm.thinpooldev`) 542 543Specifies a custom blockdevice to use for metadata for the thin pool. 544 545For best performance the metadata should be on a different spindle than the 546data, or even better on an SSD. 547 548If setting up a new metadata pool it is required to be valid. This can be 549achieved by zeroing the first 4k to indicate empty metadata, like this: 550 551```bash 552$ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 553``` 554 555###### Example 556 557```bash 558$ sudo dockerd \ 559 --storage-opt dm.datadev=/dev/sdb1 \ 560 --storage-opt dm.metadatadev=/dev/sdc1 561``` 562 563##### `dm.blocksize` 564 565Specifies a custom blocksize to use for the thin pool. The default 566blocksize is 64K. 567 568###### Example 569 570```bash 571$ sudo dockerd --storage-opt dm.blocksize=512K 572``` 573 574##### `dm.blkdiscard` 575 576Enables or disables the use of `blkdiscard` when removing devicemapper 577devices. This is enabled by default (only) if using loopback devices and is 578required to resparsify the loopback file on image/container removal. 579 580Disabling this on loopback can lead to *much* faster container removal 581times, but will make the space used in `/var/lib/docker` directory not be 582returned to the system for other use when containers are removed. 583 584###### Examples 585 586```bash 587$ sudo dockerd --storage-opt dm.blkdiscard=false 588``` 589 590##### `dm.override_udev_sync_check` 591 592Overrides the `udev` synchronization checks between `devicemapper` and `udev`. 593`udev` is the device manager for the Linux kernel. 594 595To view the `udev` sync support of a Docker daemon that is using the 596`devicemapper` driver, run: 597 598```bash 599$ docker info 600[...] 601Udev Sync Supported: true 602[...] 603``` 604 605When `udev` sync support is `true`, then `devicemapper` and udev can 606coordinate the activation and deactivation of devices for containers. 607 608When `udev` sync support is `false`, a race condition occurs between 609the`devicemapper` and `udev` during create and cleanup. The race condition 610results in errors and failures. (For information on these failures, see 611[docker#4036](https://github.com/docker/docker/issues/4036)) 612 613To allow the `docker` daemon to start, regardless of `udev` sync not being 614supported, set `dm.override_udev_sync_check` to true: 615 616```bash 617$ sudo dockerd --storage-opt dm.override_udev_sync_check=true 618``` 619 620When this value is `true`, the `devicemapper` continues and simply warns 621you the errors are happening. 622 623> **Note**: The ideal is to pursue a `docker` daemon and environment that does 624> support synchronizing with `udev`. For further discussion on this 625> topic, see [docker#4036](https://github.com/docker/docker/issues/4036). 626> Otherwise, set this flag for migrating existing Docker daemons to 627> a daemon with a supported environment. 628 629##### `dm.use_deferred_removal` 630 631Enables use of deferred device removal if `libdm` and the kernel driver 632support the mechanism. 633 634Deferred device removal means that if device is busy when devices are 635being removed/deactivated, then a deferred removal is scheduled on 636device. And devices automatically go away when last user of the device 637exits. 638 639For example, when a container exits, its associated thin device is removed. 640If that device has leaked into some other mount namespace and can't be 641removed, the container exit still succeeds and this option causes the 642system to schedule the device for deferred removal. It does not wait in a 643loop trying to remove a busy device. 644 645###### Example 646 647```bash 648$ sudo dockerd --storage-opt dm.use_deferred_removal=true 649``` 650 651##### `dm.use_deferred_deletion` 652 653Enables use of deferred device deletion for thin pool devices. By default, 654thin pool device deletion is synchronous. Before a container is deleted, 655the Docker daemon removes any associated devices. If the storage driver 656can not remove a device, the container deletion fails and daemon returns. 657 658```none 659Error deleting container: Error response from daemon: Cannot destroy container 660``` 661 662To avoid this failure, enable both deferred device deletion and deferred 663device removal on the daemon. 664 665```bash 666$ sudo dockerd \ 667 --storage-opt dm.use_deferred_deletion=true \ 668 --storage-opt dm.use_deferred_removal=true 669``` 670 671With these two options enabled, if a device is busy when the driver is 672deleting a container, the driver marks the device as deleted. Later, when 673the device isn't in use, the driver deletes it. 674 675In general it should be safe to enable this option by default. It will help 676when unintentional leaking of mount point happens across multiple mount 677namespaces. 678 679##### `dm.min_free_space` 680 681Specifies the min free space percent in a thin pool require for new device 682creation to succeed. This check applies to both free data space as well 683as free metadata space. Valid values are from 0% - 99%. Value 0% disables 684free space checking logic. If user does not specify a value for this option, 685the Engine uses a default value of 10%. 686 687Whenever a new a thin pool device is created (during `docker pull` or during 688container creation), the Engine checks if the minimum free space is 689available. If sufficient space is unavailable, then device creation fails 690and any relevant `docker` operation fails. 691 692To recover from this error, you must create more free space in the thin pool 693to recover from the error. You can create free space by deleting some images 694and containers from the thin pool. You can also add more storage to the thin 695pool. 696 697To add more space to a LVM (logical volume management) thin pool, just add 698more storage to the volume group container thin pool; this should automatically 699resolve any errors. If your configuration uses loop devices, then stop the 700Engine daemon, grow the size of loop files and restart the daemon to resolve 701the issue. 702 703###### Example 704 705```bash 706$ sudo dockerd --storage-opt dm.min_free_space=10% 707``` 708 709##### `dm.xfs_nospace_max_retries` 710 711Specifies the maximum number of retries XFS should attempt to complete 712IO when ENOSPC (no space) error is returned by underlying storage device. 713 714By default XFS retries infinitely for IO to finish and this can result 715in unkillable process. To change this behavior one can set 716xfs_nospace_max_retries to say 0 and XFS will not retry IO after getting 717ENOSPC and will shutdown filesystem. 718 719###### Example 720 721```bash 722$ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0 723``` 724 725##### `dm.libdm_log_level` 726 727Specifies the maxmimum `libdm` log level that will be forwarded to the 728`dockerd` log (as specified by `--log-level`). This option is primarily 729intended for debugging problems involving `libdm`. Using values other than the 730defaults may cause false-positive warnings to be logged. 731 732Values specified must fall within the range of valid `libdm` log levels. At the 733time of writing, the following is the list of `libdm` log levels as well as 734their corresponding levels when output by `dockerd`. 735 736| `libdm` Level | Value | `--log-level` | 737| ------------- | -----:| ------------- | 738| `_LOG_FATAL` | 2 | error | 739| `_LOG_ERR` | 3 | error | 740| `_LOG_WARN` | 4 | warn | 741| `_LOG_NOTICE` | 5 | info | 742| `_LOG_INFO` | 6 | info | 743| `_LOG_DEBUG` | 7 | debug | 744 745###### Example 746 747```bash 748$ sudo dockerd \ 749 --log-level debug \ 750 --storage-opt dm.libdm_log_level=7 751``` 752 753#### ZFS options 754 755##### `zfs.fsname` 756 757Set zfs filesystem under which docker will create its own datasets. 758By default docker will pick up the zfs filesystem where docker graph 759(`/var/lib/docker`) is located. 760 761###### Example 762 763```bash 764$ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker 765``` 766 767#### Btrfs options 768 769##### `btrfs.min_space` 770 771Specifies the minimum size to use when creating the subvolume which is used 772for containers. If user uses disk quota for btrfs when creating or running 773a container with **--storage-opt size** option, docker should ensure the 774**size** cannot be smaller than **btrfs.min_space**. 775 776###### Example 777 778```bash 779$ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G 780``` 781 782#### Overlay2 options 783 784##### `overlay2.override_kernel_check` 785 786Overrides the Linux kernel version check allowing overlay2. Support for 787specifying multiple lower directories needed by overlay2 was added to the 788Linux kernel in 4.0.0. However, some older kernel versions may be patched 789to add multiple lower directory support for OverlayFS. This option should 790only be used after verifying this support exists in the kernel. Applying 791this option on a kernel without this support will cause failures on mount. 792 793##### `overlay2.size` 794 795Sets the default max size of the container. It is supported only when the 796backing fs is `xfs` and mounted with `pquota` mount option. Under these 797conditions the user can pass any size less then the backing fs size. 798 799###### Example 800 801```bash 802$ sudo dockerd -s overlay2 --storage-opt overlay2.size=1G 803``` 804 805 806#### Windowsfilter options 807 808##### `size` 809 810Specifies the size to use when creating the sandbox which is used for containers. 811Defaults to 20G. 812 813###### Example 814 815```PowerShell 816C:\> dockerd --storage-opt size=40G 817``` 818 819#### LCOW (Linux Containers on Windows) options 820 821##### `lcow.globalmode` 822 823Specifies whether the daemon instantiates utility VM instances as required 824(recommended and default if omitted), or uses single global utility VM (better 825performance, but has security implications and not recommended for production 826deployments). 827 828###### Example 829 830```PowerShell 831C:\> dockerd --storage-opt lcow.globalmode=false 832``` 833 834##### `lcow.kirdpath` 835 836Specifies the folder path to the location of a pair of kernel and initrd files 837used for booting a utility VM. Defaults to `%ProgramFiles%\Linux Containers`. 838 839###### Example 840 841```PowerShell 842C:\> dockerd --storage-opt lcow.kirdpath=c:\path\to\files 843``` 844 845##### `lcow.kernel` 846 847Specifies the filename of a kernel file located in the `lcow.kirdpath` path. 848Defaults to `bootx64.efi`. 849 850###### Example 851 852```PowerShell 853C:\> dockerd --storage-opt lcow.kernel=kernel.efi 854``` 855 856##### `lcow.initrd` 857 858Specifies the filename of an initrd file located in the `lcow.kirdpath` path. 859Defaults to `initrd.img`. 860 861###### Example 862 863```PowerShell 864C:\> dockerd --storage-opt lcow.initrd=myinitrd.img 865``` 866 867##### `lcow.bootparameters` 868 869Specifies additional boot parameters for booting utility VMs when in kernel/ 870initrd mode. Ignored if the utility VM is booting from VHD. These settings 871are kernel specific. 872 873###### Example 874 875```PowerShell 876C:\> dockerd --storage-opt "lcow.bootparameters='option=value'" 877``` 878 879##### `lcow.vhdx` 880 881Specifies a custom VHDX to boot a utility VM, as an alternate to kernel 882and initrd booting. Defaults to `uvm.vhdx` under `lcow.kirdpath`. 883 884###### Example 885 886```PowerShell 887C:\> dockerd --storage-opt lcow.vhdx=custom.vhdx 888``` 889 890##### `lcow.timeout` 891 892Specifies the timeout for utility VM operations in seconds. Defaults 893to 300. 894 895###### Example 896 897```PowerShell 898C:\> dockerd --storage-opt lcow.timeout=240 899``` 900 901##### `lcow.sandboxsize` 902 903Specifies the size in GB to use when creating the sandbox which is used for 904containers. Defaults to 20. Cannot be less than 20. 905 906###### Example 907 908```PowerShell 909C:\> dockerd --storage-opt lcow.sandboxsize=40 910``` 911 912### Docker runtime execution options 913 914The Docker daemon relies on a 915[OCI](https://github.com/opencontainers/runtime-spec) compliant runtime 916(invoked via the `containerd` daemon) as its interface to the Linux 917kernel `namespaces`, `cgroups`, and `SELinux`. 918 919By default, the Docker daemon automatically starts `containerd`. If you want to 920control `containerd` startup, manually start `containerd` and pass the path to 921the `containerd` socket using the `--containerd` flag. For example: 922 923```bash 924$ sudo dockerd --containerd /var/run/dev/docker-containerd.sock 925``` 926 927Runtimes can be registered with the daemon either via the 928configuration file or using the `--add-runtime` command line argument. 929 930The following is an example adding 2 runtimes via the configuration: 931 932```json 933{ 934 "default-runtime": "runc", 935 "runtimes": { 936 "runc": { 937 "path": "runc" 938 }, 939 "custom": { 940 "path": "/usr/local/bin/my-runc-replacement", 941 "runtimeArgs": [ 942 "--debug" 943 ] 944 } 945 } 946} 947``` 948 949This is the same example via the command line: 950 951```bash 952$ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement 953``` 954 955> **Note**: Defining runtime arguments via the command line is not supported. 956 957#### Options for the runtime 958 959You can configure the runtime using options specified 960with the `--exec-opt` flag. All the flag's options have the `native` prefix. A 961single `native.cgroupdriver` option is available. 962 963The `native.cgroupdriver` option specifies the management of the container's 964cgroups. You can only specify `cgroupfs` or `systemd`. If you specify 965`systemd` and it is not available, the system errors out. If you omit the 966`native.cgroupdriver` option,` cgroupfs` is used. 967 968This example sets the `cgroupdriver` to `systemd`: 969 970```bash 971$ sudo dockerd --exec-opt native.cgroupdriver=systemd 972``` 973 974Setting this option applies to all containers the daemon launches. 975 976Also Windows Container makes use of `--exec-opt` for special purpose. Docker user 977can specify default container isolation technology with this, for example: 978 979```console 980> dockerd --exec-opt isolation=hyperv 981``` 982 983Will make `hyperv` the default isolation technology on Windows. If no isolation 984value is specified on daemon start, on Windows client, the default is 985`hyperv`, and on Windows server, the default is `process`. 986 987### Daemon DNS options 988 989To set the DNS server for all Docker containers, use: 990 991```bash 992$ sudo dockerd --dns 8.8.8.8 993``` 994 995To set the DNS search domain for all Docker containers, use: 996 997```bash 998$ sudo dockerd --dns-search example.com 999``` 1000 1001### Allow push of nondistributable artifacts 1002 1003Some images (e.g., Windows base images) contain artifacts whose distribution is 1004restricted by license. When these images are pushed to a registry, restricted 1005artifacts are not included. 1006 1007To override this behavior for specific registries, use the 1008`--allow-nondistributable-artifacts` option in one of the following forms: 1009 1010* `--allow-nondistributable-artifacts myregistry:5000` tells the Docker daemon 1011 to push nondistributable artifacts to myregistry:5000. 1012* `--allow-nondistributable-artifacts 10.1.0.0/16` tells the Docker daemon to 1013 push nondistributable artifacts to all registries whose resolved IP address 1014 is within the subnet described by the CIDR syntax. 1015 1016This option can be used multiple times. 1017 1018This option is useful when pushing images containing nondistributable artifacts 1019to a registry on an air-gapped network so hosts on that network can pull the 1020images without connecting to another server. 1021 1022> **Warning**: Nondistributable artifacts typically have restrictions on how 1023> and where they can be distributed and shared. Only use this feature to push 1024> artifacts to private registries and ensure that you are in compliance with 1025> any terms that cover redistributing nondistributable artifacts. 1026 1027### Insecure registries 1028 1029Docker considers a private registry either secure or insecure. In the rest of 1030this section, *registry* is used for *private registry*, and `myregistry:5000` 1031is a placeholder example for a private registry. 1032 1033A secure registry uses TLS and a copy of its CA certificate is placed on the 1034Docker host at `/etc/docker/certs.d/myregistry:5000/ca.crt`. An insecure 1035registry is either not using TLS (i.e., listening on plain text HTTP), or is 1036using TLS with a CA certificate not known by the Docker daemon. The latter can 1037happen when the certificate was not found under 1038`/etc/docker/certs.d/myregistry:5000/`, or if the certificate verification 1039failed (i.e., wrong CA). 1040 1041By default, Docker assumes all, but local (see local registries below), 1042registries are secure. Communicating with an insecure registry is not possible 1043if Docker assumes that registry is secure. In order to communicate with an 1044insecure registry, the Docker daemon requires `--insecure-registry` in one of 1045the following two forms: 1046 1047* `--insecure-registry myregistry:5000` tells the Docker daemon that 1048 myregistry:5000 should be considered insecure. 1049* `--insecure-registry 10.1.0.0/16` tells the Docker daemon that all registries 1050 whose domain resolve to an IP address is part of the subnet described by the 1051 CIDR syntax, should be considered insecure. 1052 1053The flag can be used multiple times to allow multiple registries to be marked 1054as insecure. 1055 1056If an insecure registry is not marked as insecure, `docker pull`, 1057`docker push`, and `docker search` will result in an error message prompting 1058the user to either secure or pass the `--insecure-registry` flag to the Docker 1059daemon as described above. 1060 1061Local registries, whose IP address falls in the 127.0.0.0/8 range, are 1062automatically marked as insecure as of Docker 1.3.2. It is not recommended to 1063rely on this, as it may change in the future. 1064 1065Enabling `--insecure-registry`, i.e., allowing un-encrypted and/or untrusted 1066communication, can be useful when running a local registry. However, 1067because its use creates security vulnerabilities it should ONLY be enabled for 1068testing purposes. For increased security, users should add their CA to their 1069system's list of trusted CAs instead of enabling `--insecure-registry`. 1070 1071#### Legacy Registries 1072 1073Starting with Docker 17.12, operations against registries supporting only the 1074legacy v1 protocol are no longer supported. Specifically, the daemon will not 1075attempt `push`, `pull` and `login` to v1 registries. The exception to this is 1076`search` which can still be performed on v1 registries. 1077 1078The `disable-legacy-registry` configuration option has been removed and, when 1079used, will produce an error on daemon startup. 1080 1081 1082### Running a Docker daemon behind an HTTPS_PROXY 1083 1084When running inside a LAN that uses an `HTTPS` proxy, the Docker Hub 1085certificates will be replaced by the proxy's certificates. These certificates 1086need to be added to your Docker host's configuration: 1087 10881. Install the `ca-certificates` package for your distribution 10892. Ask your network admin for the proxy's CA certificate and append them to 1090 `/etc/pki/tls/certs/ca-bundle.crt` 10913. Then start your Docker daemon with `HTTPS_PROXY=http://username:password@proxy:port/ dockerd`. 1092 The `username:` and `password@` are optional - and are only needed if your 1093 proxy is set up to require authentication. 1094 1095This will only add the proxy and authentication to the Docker daemon's requests - 1096your `docker build`s and running containers will need extra configuration to 1097use the proxy 1098 1099### Default `ulimit` settings 1100 1101`--default-ulimit` allows you to set the default `ulimit` options to use for 1102all containers. It takes the same options as `--ulimit` for `docker run`. If 1103these defaults are not set, `ulimit` settings will be inherited, if not set on 1104`docker run`, from the Docker daemon. Any `--ulimit` options passed to 1105`docker run` will overwrite these defaults. 1106 1107Be careful setting `nproc` with the `ulimit` flag as `nproc` is designed by Linux to 1108set the maximum number of processes available to a user, not to a container. For details 1109please check the [run](run.md) reference. 1110 1111### Node discovery 1112 1113The `--cluster-advertise` option specifies the `host:port` or `interface:port` 1114combination that this particular daemon instance should use when advertising 1115itself to the cluster. The daemon is reached by remote hosts through this value. 1116If you specify an interface, make sure it includes the IP address of the actual 1117Docker host. For Engine installation created through `docker-machine`, the 1118interface is typically `eth1`. 1119 1120The daemon uses [libkv](https://github.com/docker/libkv/) to advertise 1121the node within the cluster. Some key-value backends support mutual 1122TLS. To configure the client TLS settings used by the daemon can be configured 1123using the `--cluster-store-opt` flag, specifying the paths to PEM encoded 1124files. For example: 1125 1126```bash 1127$ sudo dockerd \ 1128 --cluster-advertise 192.168.1.2:2376 \ 1129 --cluster-store etcd://192.168.1.2:2379 \ 1130 --cluster-store-opt kv.cacertfile=/path/to/ca.pem \ 1131 --cluster-store-opt kv.certfile=/path/to/cert.pem \ 1132 --cluster-store-opt kv.keyfile=/path/to/key.pem 1133``` 1134 1135The currently supported cluster store options are: 1136 1137| Option | Description | 1138|:----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 1139| `discovery.heartbeat` | Specifies the heartbeat timer in seconds which is used by the daemon as a `keepalive` mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds. | 1140| `discovery.ttl` | Specifies the TTL (time-to-live) in seconds which is used by the discovery module to timeout a node if a valid heartbeat is not received within the configured ttl value. If not configured, the default value is 60 seconds. | 1141| `kv.cacertfile` | Specifies the path to a local file with PEM encoded CA certificates to trust. | 1142| `kv.certfile` | Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store. | 1143| `kv.keyfile` | Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store. | 1144| `kv.path` | Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. | 1145 1146### Access authorization 1147 1148Docker's access authorization can be extended by authorization plugins that your 1149organization can purchase or build themselves. You can install one or more 1150authorization plugins when you start the Docker `daemon` using the 1151`--authorization-plugin=PLUGIN_ID` option. 1152 1153```bash 1154$ sudo dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... 1155``` 1156 1157The `PLUGIN_ID` value is either the plugin's name or a path to its specification 1158file. The plugin's implementation determines whether you can specify a name or 1159path. Consult with your Docker administrator to get information about the 1160plugins available to you. 1161 1162Once a plugin is installed, requests made to the `daemon` through the 1163command line or Docker's Engine API are allowed or denied by the plugin. 1164If you have multiple plugins installed, each plugin, in order, must 1165allow the request for it to complete. 1166 1167For information about how to create an authorization plugin, see [authorization 1168plugin](../../extend/plugins_authorization.md) section in the Docker extend section of this documentation. 1169 1170 1171### Daemon user namespace options 1172 1173The Linux kernel 1174[user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) 1175provides additional security by enabling a process, and therefore a container, 1176to have a unique range of user and group IDs which are outside the traditional 1177user and group range utilized by the host system. Potentially the most important 1178security improvement is that, by default, container processes running as the 1179`root` user will have expected administrative privilege (with some restrictions) 1180inside the container but will effectively be mapped to an unprivileged `uid` on 1181the host. 1182 1183For details about how to use this feature, as well as limitations, see 1184[Isolate containers with a user namespace](https://docs.docker.com/engine/security/userns-remap/). 1185 1186### Miscellaneous options 1187 1188IP masquerading uses address translation to allow containers without a public 1189IP to talk to other machines on the Internet. This may interfere with some 1190network topologies and can be disabled with `--ip-masq=false`. 1191 1192Docker supports softlinks for the Docker data directory (`/var/lib/docker`) and 1193for `/var/lib/docker/tmp`. The `DOCKER_TMPDIR` and the data directory can be 1194set like this: 1195 1196 DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 1197 # or 1198 export DOCKER_TMPDIR=/mnt/disk2/tmp 1199 /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 1200 1201#### Default cgroup parent 1202 1203The `--cgroup-parent` option allows you to set the default cgroup parent 1204to use for containers. If this option is not set, it defaults to `/docker` for 1205fs cgroup driver and `system.slice` for systemd cgroup driver. 1206 1207If the cgroup has a leading forward slash (`/`), the cgroup is created 1208under the root cgroup, otherwise the cgroup is created under the daemon 1209cgroup. 1210 1211Assuming the daemon is running in cgroup `daemoncgroup`, 1212`--cgroup-parent=/foobar` creates a cgroup in 1213`/sys/fs/cgroup/memory/foobar`, whereas using `--cgroup-parent=foobar` 1214creates the cgroup in `/sys/fs/cgroup/memory/daemoncgroup/foobar` 1215 1216The systemd cgroup driver has different rules for `--cgroup-parent`. Systemd 1217represents hierarchy by slice and the name of the slice encodes the location in 1218the tree. So `--cgroup-parent` for systemd cgroups should be a slice name. A 1219name can consist of a dash-separated series of names, which describes the path 1220to the slice from the root slice. For example, `--cgroup-parent=user-a-b.slice` 1221means the memory cgroup for the container is created in 1222`/sys/fs/cgroup/memory/user.slice/user-a.slice/user-a-b.slice/docker-<id>.scope`. 1223 1224This setting can also be set per container, using the `--cgroup-parent` 1225option on `docker create` and `docker run`, and takes precedence over 1226the `--cgroup-parent` option on the daemon. 1227 1228#### Daemon metrics 1229 1230The `--metrics-addr` option takes a tcp address to serve the metrics API. 1231This feature is still experimental, therefore, the daemon must be running in experimental 1232mode for this feature to work. 1233 1234To serve the metrics API on `localhost:9323` you would specify `--metrics-addr 127.0.0.1:9323`, 1235allowing you to make requests on the API at `127.0.0.1:9323/metrics` to receive metrics in the 1236[prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/) format. 1237 1238Port `9323` is the [default port associated with Docker 1239metrics](https://github.com/prometheus/prometheus/wiki/Default-port-allocations) 1240to avoid collisions with other prometheus exporters and services. 1241 1242If you are running a prometheus server you can add this address to your scrape configs 1243to have prometheus collect metrics on Docker. For more information 1244on prometheus you can view the website [here](https://prometheus.io/). 1245 1246```none 1247scrape_configs: 1248 - job_name: 'docker' 1249 static_configs: 1250 - targets: ['127.0.0.1:9323'] 1251``` 1252 1253Please note that this feature is still marked as experimental as metrics and metric 1254names could change while this feature is still in experimental. Please provide 1255feedback on what you would like to see collected in the API. 1256 1257#### Node Generic Resources 1258 1259The `--node-generic-resources` option takes a list of key-value 1260pair (`key=value`) that allows you to advertise user defined resources 1261in a swarm cluster. 1262 1263The current expected use case is to advertise NVIDIA GPUs so that services 1264requesting `NVIDIA-GPU=[0-16]` can land on a node that has enough GPUs for 1265the task to run. 1266 1267Example of usage: 1268```json 1269{ 1270 "node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"] 1271} 1272``` 1273 1274### Daemon configuration file 1275 1276The `--config-file` option allows you to set any configuration option 1277for the daemon in a JSON format. This file uses the same flag names as keys, 1278except for flags that allow several entries, where it uses the plural 1279of the flag name, e.g., `labels` for the `label` flag. 1280 1281The options set in the configuration file must not conflict with options set 1282via flags. The docker daemon fails to start if an option is duplicated between 1283the file and the flags, regardless their value. We do this to avoid 1284silently ignore changes introduced in configuration reloads. 1285For example, the daemon fails to start if you set daemon labels 1286in the configuration file and also set daemon labels via the `--label` flag. 1287Options that are not present in the file are ignored when the daemon starts. 1288 1289##### On Linux 1290 1291The default location of the configuration file on Linux is 1292`/etc/docker/daemon.json`. The `--config-file` flag can be used to specify a 1293 non-default location. 1294 1295This is a full example of the allowed configuration options on Linux: 1296 1297```json 1298{ 1299 "authorization-plugins": [], 1300 "data-root": "", 1301 "dns": [], 1302 "dns-opts": [], 1303 "dns-search": [], 1304 "exec-opts": [], 1305 "exec-root": "", 1306 "experimental": false, 1307 "features": {}, 1308 "storage-driver": "", 1309 "storage-opts": [], 1310 "labels": [], 1311 "live-restore": true, 1312 "log-driver": "json-file", 1313 "log-opts": { 1314 "max-size": "10m", 1315 "max-file":"5", 1316 "labels": "somelabel", 1317 "env": "os,customer" 1318 }, 1319 "mtu": 0, 1320 "pidfile": "", 1321 "cluster-store": "", 1322 "cluster-store-opts": {}, 1323 "cluster-advertise": "", 1324 "max-concurrent-downloads": 3, 1325 "max-concurrent-uploads": 5, 1326 "default-shm-size": "64M", 1327 "shutdown-timeout": 15, 1328 "debug": true, 1329 "hosts": [], 1330 "log-level": "", 1331 "tls": true, 1332 "tlsverify": true, 1333 "tlscacert": "", 1334 "tlscert": "", 1335 "tlskey": "", 1336 "swarm-default-advertise-addr": "", 1337 "api-cors-header": "", 1338 "selinux-enabled": false, 1339 "userns-remap": "", 1340 "group": "", 1341 "cgroup-parent": "", 1342 "default-ulimits": { 1343 "nofile": { 1344 "Name": "nofile", 1345 "Hard": 64000, 1346 "Soft": 64000 1347 } 1348 }, 1349 "init": false, 1350 "init-path": "/usr/libexec/docker-init", 1351 "ipv6": false, 1352 "iptables": false, 1353 "ip-forward": false, 1354 "ip-masq": false, 1355 "userland-proxy": false, 1356 "userland-proxy-path": "/usr/libexec/docker-proxy", 1357 "ip": "0.0.0.0", 1358 "bridge": "", 1359 "bip": "", 1360 "fixed-cidr": "", 1361 "fixed-cidr-v6": "", 1362 "default-gateway": "", 1363 "default-gateway-v6": "", 1364 "icc": false, 1365 "raw-logs": false, 1366 "allow-nondistributable-artifacts": [], 1367 "registry-mirrors": [], 1368 "seccomp-profile": "", 1369 "insecure-registries": [], 1370 "no-new-privileges": false, 1371 "default-runtime": "runc", 1372 "oom-score-adjust": -500, 1373 "node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"], 1374 "runtimes": { 1375 "cc-runtime": { 1376 "path": "/usr/bin/cc-runtime" 1377 }, 1378 "custom": { 1379 "path": "/usr/local/bin/my-runc-replacement", 1380 "runtimeArgs": [ 1381 "--debug" 1382 ] 1383 } 1384 }, 1385 "default-address-pools":[{"base":"172.80.0.0/16","size":24}, 1386 {"base":"172.90.0.0/16","size":24}] 1387} 1388``` 1389 1390> **Note:** You cannot set options in `daemon.json` that have already been set on 1391> daemon startup as a flag. 1392> On systems that use `systemd` to start the Docker daemon, `-H` is already set, so 1393> you cannot use the `hosts` key in `daemon.json` to add listening addresses. 1394> See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for how 1395> to accomplish this task with a systemd drop-in file. 1396 1397##### On Windows 1398 1399The default location of the configuration file on Windows is 1400 `%programdata%\docker\config\daemon.json`. The `--config-file` flag can be 1401 used to specify a non-default location. 1402 1403This is a full example of the allowed configuration options on Windows: 1404 1405```json 1406{ 1407 "authorization-plugins": [], 1408 "data-root": "", 1409 "dns": [], 1410 "dns-opts": [], 1411 "dns-search": [], 1412 "exec-opts": [], 1413 "experimental": false, 1414 "features":{}, 1415 "storage-driver": "", 1416 "storage-opts": [], 1417 "labels": [], 1418 "log-driver": "", 1419 "mtu": 0, 1420 "pidfile": "", 1421 "cluster-store": "", 1422 "cluster-advertise": "", 1423 "max-concurrent-downloads": 3, 1424 "max-concurrent-uploads": 5, 1425 "shutdown-timeout": 15, 1426 "debug": true, 1427 "hosts": [], 1428 "log-level": "", 1429 "tlsverify": true, 1430 "tlscacert": "", 1431 "tlscert": "", 1432 "tlskey": "", 1433 "swarm-default-advertise-addr": "", 1434 "group": "", 1435 "default-ulimits": {}, 1436 "bridge": "", 1437 "fixed-cidr": "", 1438 "raw-logs": false, 1439 "allow-nondistributable-artifacts": [], 1440 "registry-mirrors": [], 1441 "insecure-registries": [] 1442} 1443``` 1444 1445#### Feature options 1446The optional field `features` in `daemon.json` allows users to enable or disable specific 1447daemon features. For example, `{"features":{"buildkit": true}}` enables `buildkit` as the 1448default docker image builder. 1449 1450The list of currently supported feature options: 1451- `buildkit`: It enables `buildkit` as default builder when set to `true` or disables it by 1452`false`. Note that if this option is not explicitly set in the daemon config file, then it 1453is up to the cli to determine which builder to invoke. 1454 1455#### Configuration reload behavior 1456 1457Some options can be reconfigured when the daemon is running without requiring 1458to restart the process. We use the `SIGHUP` signal in Linux to reload, and a global event 1459in Windows with the key `Global\docker-daemon-config-$PID`. The options can 1460be modified in the configuration file but still will check for conflicts with 1461the provided flags. The daemon fails to reconfigure itself 1462if there are conflicts, but it won't stop execution. 1463 1464The list of currently supported options that can be reconfigured is this: 1465 1466- `debug`: it changes the daemon to debug mode when set to true. 1467- `cluster-store`: it reloads the discovery store with the new address. 1468- `cluster-store-opts`: it uses the new options to reload the discovery store. 1469- `cluster-advertise`: it modifies the address advertised after reloading. 1470- `labels`: it replaces the daemon labels with a new set of labels. 1471- `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/config/containers/live-restore/). 1472- `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. 1473- `max-concurrent-uploads`: it updates the max concurrent uploads for each push. 1474- `default-runtime`: it updates the runtime to be used if not is 1475 specified at container creation. It defaults to "default" which is 1476 the runtime shipped with the official docker packages. 1477- `runtimes`: it updates the list of available OCI runtimes that can 1478 be used to run containers. 1479- `authorization-plugin`: it specifies the authorization plugins to use. 1480- `allow-nondistributable-artifacts`: Replaces the set of registries to which the daemon will push nondistributable artifacts with a new set of registries. 1481- `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config. 1482- `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config. 1483- `shutdown-timeout`: it replaces the daemon's existing configuration timeout with a new timeout for shutting down all containers. 1484- `features`: it explicitly enables or disables specific features. 1485 1486Updating and reloading the cluster configurations such as `--cluster-store`, 1487`--cluster-advertise` and `--cluster-store-opts` will take effect only if 1488these configurations were not previously configured. If `--cluster-store` 1489has been provided in flags and `cluster-advertise` not, `cluster-advertise` 1490can be added in the configuration file without accompanied by `--cluster-store`. 1491Configuration reload will log a warning message if it detects a change in 1492previously configured cluster configurations. 1493 1494 1495### Run multiple daemons 1496 1497> **Note:** Running multiple daemons on a single host is considered as "experimental". The user should be aware of 1498> unsolved problems. This solution may not work properly in some cases. Solutions are currently under development 1499> and will be delivered in the near future. 1500 1501This section describes how to run multiple Docker daemons on a single host. To 1502run multiple daemons, you must configure each daemon so that it does not 1503conflict with other daemons on the same host. You can set these options either 1504by providing them as flags, or by using a [daemon configuration file](#daemon-configuration-file). 1505 1506The following daemon options must be configured for each daemon: 1507 1508```none 1509-b, --bridge= Attach containers to a network bridge 1510--exec-root=/var/run/docker Root of the Docker execdriver 1511--data-root=/var/lib/docker Root of persisted Docker data 1512-p, --pidfile=/var/run/docker.pid Path to use for daemon PID file 1513-H, --host=[] Daemon socket(s) to connect to 1514--iptables=true Enable addition of iptables rules 1515--config-file=/etc/docker/daemon.json Daemon configuration file 1516--tlscacert="~/.docker/ca.pem" Trust certs signed only by this CA 1517--tlscert="~/.docker/cert.pem" Path to TLS certificate file 1518--tlskey="~/.docker/key.pem" Path to TLS key file 1519``` 1520 1521When your daemons use different values for these flags, you can run them on the same host without any problems. 1522It is very important to properly understand the meaning of those options and to use them correctly. 1523 1524- The `-b, --bridge=` flag is set to `docker0` as default bridge network. It is created automatically when you install Docker. 1525If you are not using the default, you must create and configure the bridge manually or just set it to 'none': `--bridge=none` 1526- `--exec-root` is the path where the container state is stored. The default value is `/var/run/docker`. Specify the path for 1527your running daemon here. 1528- `--data-root` is the path where persisted data such as images, volumes, and 1529cluster state are stored. The default value is `/var/lib/docker`. To avoid any 1530conflict with other daemons, set this parameter separately for each daemon. 1531- `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. Specify the path for your 1532pid file here. 1533- `--host=[]` specifies where the Docker daemon will listen for client connections. If unspecified, it defaults to `/var/run/docker.sock`. 1534- `--iptables=false` prevents the Docker daemon from adding iptables rules. If 1535multiple daemons manage iptables rules, they may overwrite rules set by another 1536daemon. Be aware that disabling this option requires you to manually add 1537iptables rules to expose container ports. If you prevent Docker from adding 1538iptables rules, Docker will also not add IP masquerading rules, even if you set 1539`--ip-masq` to `true`. Without IP masquerading rules, Docker containers will not be 1540able to connect to external hosts or the internet when using network other than 1541default bridge. 1542- `--config-file=/etc/docker/daemon.json` is the path where configuration file is stored. You can use it instead of 1543daemon flags. Specify the path for each daemon. 1544- `--tls*` Docker daemon supports `--tlsverify` mode that enforces encrypted and authenticated remote connections. 1545The `--tls*` options enable use of specific certificates for individual daemons. 1546 1547Example script for a separate “bootstrap” instance of the Docker daemon without network: 1548 1549```bash 1550$ sudo dockerd \ 1551 -H unix:///var/run/docker-bootstrap.sock \ 1552 -p /var/run/docker-bootstrap.pid \ 1553 --iptables=false \ 1554 --ip-masq=false \ 1555 --bridge=none \ 1556 --data-root=/var/lib/docker-bootstrap \ 1557 --exec-root=/var/run/docker-bootstrap 1558``` 1559