1.\" Copyright (c) 2015-2019 The DragonFly Project. All rights reserved. 2.\" 3.\" This code is derived from software contributed to The DragonFly Project 4.\" by Matthew Dillon <dillon@backplane.com> 5.\" 6.\" Redistribution and use in source and binary forms, with or without 7.\" modification, are permitted provided that the following conditions 8.\" are met: 9.\" 10.\" 1. Redistributions of source code must retain the above copyright 11.\" notice, this list of conditions and the following disclaimer. 12.\" 2. Redistributions in binary form must reproduce the above copyright 13.\" notice, this list of conditions and the following disclaimer in 14.\" the documentation and/or other materials provided with the 15.\" distribution. 16.\" 3. Neither the name of The DragonFly Project nor the names of its 17.\" contributors may be used to endorse or promote products derived 18.\" from this software without specific, prior written permission. 19.\" 20.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 21.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 22.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 23.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 24.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 25.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 26.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 27.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 28.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 29.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 30.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 31.\" SUCH DAMAGE. 32.\" 33.Dd November 29, 2020 34.Dt HAMMER2 8 35.Os 36.Sh NAME 37.Nm hammer2 38.Nd hammer2 file system utility 39.Sh SYNOPSIS 40.Nm 41.Fl h 42.Nm 43.Op Fl s Ar path 44.Op Fl t Ar type 45.Op Fl u Ar uuid 46.Op Fl m Ar mem 47.Ar command 48.Op Ar argument ... 49.Sh DESCRIPTION 50The 51.Nm 52utility provides miscellaneous support functions for a 53HAMMER2 file system. 54.Pp 55The options are as follows: 56.Bl -tag -width indent 57.It Fl s Ar path 58Specify the path to a mounted HAMMER2 filesystem. 59At least one PFS on a HAMMER2 filesystem must be mounted for the system 60to act on all PFSs managed by it. 61Every HAMMER2 filesystem typically has a PFS called "LOCAL" for this purpose. 62.It Fl t Ar type 63Specify the type when creating, upgrading, or downgrading a PFS. 64Supported types are MASTER, SLAVE, SOFT_MASTER, SOFT_SLAVE, CACHE, and DUMMY. 65If not specified the pfs-create directive will default to MASTER if no 66UUID is specified, and SLAVE if a UUID is specified. 67.It Fl u Ar uuid 68Specify the cluster UUID when creating a PFS. 69If not specified, a unique, random UUID will be generated. 70Note that every PFS also has a unique pfs_id which is always generated 71and cannot be overridden with an option. 72The { pfs_clid, pfs_fsid } tuple uniquely identifies a component of a cluster. 73.It Fl m Ar mem 74Specify how much tracking memory to use for certain directives. 75At the moment, this option is only applicable to the 76.Cm bulkfree 77directive, allowing it to operate in fewer passes when given more memory. 78A nominal value for a 4TB drive with a ton of stuff on it would be around 79a gigabyte '-m 1g'. 80.El 81.Pp 82.Nm 83directives are as shown below. 84Note that most directives require you to either be CD'd into a hammer2 85filesystem, specify a path to a mounted hammer2 filesystem via the 86.Fl s 87option, or specify a path after the directive. 88It depends on the directive. 89All hammer2 filesystem have a PFS called "LOCAL" which is typically mounted 90locally on the host in order to be able to issue commands for other PFSs 91on the filesystem. 92The mount also enables PFS configuration scanning for that filesystem. 93.Bl -tag -width indent 94.\" ==== cleanup ==== 95.It Cm cleanup Op path 96Perform manual cleanup passes on paths or all mounted partitions. 97.\" ==== connect ==== 98.It Cm connect Ar target 99Add a cluster link entry to the volume header. 100The volume header can support up to 255 link entries. 101This feature is not currently used. 102.\" ==== destroy ==== 103.It Cm destroy Ar path... 104Destroy the specified directory entry in a hammer2 filesystem. 105This bypasses 106all normal checks and will unconditionally destroy the directory entry. 107The underlying inode is not checked and, if it does exist, its nlinks count 108is not decremented. 109This directive should only be used to destroy a corrupted directory entry 110which no longer has a working inode. 111.Pp 112Note that this command may desynchronize the system namecache for the 113specified entry. 114If this happens, you may have to unmount and remount the filesystem. 115.\" ==== destroy-inum ==== 116.It Cm destroy-inum Ar path... 117Destroy the specified inode in a hammer2 filesystem. 118.\" ==== disconnect ==== 119.It Cm disconnect Ar target 120Delete a cluster link entry from the volume header. 121This feature is not currently used. 122.\" ==== emergency-mode-enable === 123.It Cm emergency-mode-enable Ar target 124Flag emergency operations mode in the filesystem. 125This mode may be used 126as a last resort to delete files and directories from a full filesystem. 127Inode creation, file writes, and certain meta-data cleanups are disallowed 128while emergency mode is active. 129File and directory removal and mode/attr setting is still allowed. 130This mode is extremely dangerous and should only be used as a last resort. 131.Pp 132This mode allows the filesystem to modify blocks in-place when it is unable 133to allocate a copy. 134Thus it is possible to chflags and remove files and 135directories even when the filesystem is completely full. 136However, there is a price. 137This mode of operation WILL LIKELY CORRUPT ANY SNAPSHOTS related 138to this filesystem. 139The filesystem will report this condition if it encounters 140it but if you are forced to use this mode to fix a filesystem full condition 141your snapshots can get a bit dicey. 142It is usually safest to delete any related snapshots when using this mode. 143.Pp 144You can detect whether related snapshots have been corrupted by running 145a bulkfree pass and checking the console output for reported CRC errors. 146If no errors are reported, your snapshots are fine. 147If errors are reported 148you should delete related snapshots until bulkfree reports no further errors. 149.Pp 150The emergency mode will also make meta-data updates unsafe due to the lack of 151copy-on-write, causing potential harm if the system unexpectedly panics or 152loses power. 153GREAT CARE MUST BE TAKEN WHILE THIS MODE IS ACTIVE. 154.Bl -enum 155.It 156Determine that you are unable to recover space with normal file and directory 157removal commands due to 158.Er ENOSPC 159errors being returned by 'rm', or through the 160removal of snapshots (if any). The 'bulkfree' directive must be issued to 161scan the filesystem and free up the actual space, then check with 'df'. 162Continue if you still have insufficient space and are unable to remove items 163normally. 164.It 165If you need any related snapshots, this is a good time to copy them elsewhere. 166.It 167Idle or kill any processes trying to use the filesystem. 168.It 169Issue the emergency-mode-enable directive on the filesystem. 170Once enabled, run 'sync' to update any dirty inodes which may still 171be dirty due to not being able to flush. 172Please remember that this 173directive is a LAST RESORT, is dangerous, and will likely corrupt any 174other snapshots you have based on the filesystem you are removing files 175from. 176.It 177Remove file trees as necessary with 'rm -rf' to free space, being cognizant 178of any warnings issued by the kernel on the console (via 'dmesg') while 179doing so. 180.It 181Issue the 'bulkfree' directive to actually free the space and check that 182sufficient space has been freed with 'df'. 183.It 184If bulkfree reports CHECK errors, or if you have snapshots and insufficient 185space has been freed, you will need to delete snapshots. 186Re-run bulkfree and delete snapshots until no errors are reported. 187.It 188Issue the emergency-mode-disable directive when done. 189It might also be a 190good idea to reboot after using this mode, but theoretically you should not 191have to. 192.It 193Restore services using the filesystem. 194.El 195.\" ==== emergency-mode-disable === 196.It Cm emergency-mode-disable Ar target 197Turn off the emergency operations mode on a filesystem, restoring normal 198operation. 199.\" ==== growfs ==== 200.It Cm growfs Op fspath... 201After resizing the disk partition you can issue this command on a 202mounted hammer2 filesystem to grow into the new space in the partition. 203This command is run on a live hammer2 filesystem. 204.\" ==== info ==== 205.It Cm info Op devpath... 206Access and print the status and super-root entries for all HAMMER2 207partitions found in /dev/serno or the specified device path(s). 208The partitions do not have to be mounted. 209Note that only mounted partitions will be under active management. 210This is accomplished by mounting at least one PFS within the partition. 211Typically at least the @LOCAL PFS is mounted. 212.\" ==== mountall ==== 213.It Cm mountall Op devpath... 214This directive mounts the @LOCAL PFS on all HAMMER2 partitions found 215in /dev/serno, or the specified device path(s). 216The partitions are mounted as /var/hammer2/LOCAL.<id>. 217Mounts are executed in the background and this command will wait a 218limited amount of time for the mounts to complete before returning. 219.\" ==== status ==== 220.It Cm status Op path... 221Dump a list of all cluster link entries configured in the volume header. 222.\" ==== hash ==== 223.It Cm hash Op filename... 224Compute and print the directory hash for any number of filenames. 225.\" ==== dhash ==== 226.It Cm dhash Op filename... 227Compute and print the data hash for long directory entry for any number of filenames. 228.\" ==== pfs-list ==== 229.It Cm pfs-list Op path... 230List all PFSs associated with all mounted hammer2 storage devices. 231The list may be restricted to a particular filesystem using 232.Fl s Ar mount . 233.Pp 234Note that hammer2 PFSs associated with storage devices which have not been 235mounted in any fashion will not be listed. At least one hammer2 label must 236be mounted for the PFSs on that device to be visible. 237.\" ==== pfs-clid ==== 238.It Cm pfs-clid Ar label 239Print the cluster id for a PFS specified by name. 240.\" ==== pfs-fsid ==== 241.It Cm pfs-fsid Ar label 242Print the unique filesystem id for a PFS specified by name. 243.\" ==== pfs-create ==== 244.It Cm pfs-create Ar label 245Create a local PFS on the mounted HAMMER2 filesystem represented 246by the current directory, or specified via 247.Fl s Ar mount . 248If no UUID is specified the pfs-type defaults to MASTER. 249If a UUID is specified via the 250.Fl u 251option the pfs-type defaults to SLAVE. 252Other types can be specified with the 253.Fl t 254option. 255.Pp 256If you wish to add a MASTER to an existing cluster, you must first add it as 257a SLAVE and then upgrade it to MASTER to properly synchronize it. 258.Pp 259The DUMMY pfs-type is used to tie network-accessible clusters into the local 260machine when no local storage is desired. 261This type should be used on minimal H2 partitions or entirely in ram for 262netboot-centric systems to provide a tie-in point for the mount command, 263or on more complex systems where you need to also access network-centric 264clusters. 265.Pp 266The CACHE or SLAVE pfs-type is typically used when the main store is on 267the network but local storage is desired to improve performance. 268SLAVE is also used when a backup is desired. 269.Pp 270Generally speaking, you can mount any PFS element of a cluster in order to 271access the cluster via the full cluster protocol. 272There are two exceptions. 273If you mount a SOFT_SLAVE or a SOFT_MASTER then soft quorum semantics are 274employed... the soft slave or soft master's current state will always be used 275and the quorum protocol will not be used. 276The soft PFS will still be 277synchronized to masters in the background when available. 278Also, you can use 279.Sq mount -o local 280to mount ONLY a local HAMMER2 PFS and 281not run any network or quorum protocols for the mount. 282All such mounts except for a SOFT_MASTER mount will be read-only. 283Other than that, you will be mounting the whole cluster when you mount any 284PFS within the cluster. 285.Pp 286DUMMY - Create a PFS skeleton intended to be the mount point for a 287more complex cluster, probably one that is entirely network based. 288No data will be synchronized to this PFS so it is suitable for use 289in a network boot image or memory filesystem. 290This allows you to create placeholders for mount points on your local 291disk, SSD, or memory disk. 292.Pp 293CACHE - Create a PFS for caching portions of the cluster piecemeal. 294This is similar to a SLAVE but does not synchronize the entire contents of 295the cluster to the PFS. 296Elements found in the CACHE PFS which are validated against the cluster 297will be read, presumably a faster access than having to go to the cluster. 298Only local CACHEs will be updated. 299Network-accessible CACHE PFSs might be read but will not be written to. 300If you have a large hard-drive-based cluster you can set up localized 301SSD CACHE PFSs to improve performance. 302.Pp 303SLAVE - Create a PFS which maintains synchronization with and provides a 304read-only copy of the cluster. 305HAMMER2 will prioritize local SLAVEs for data retrieval after validating 306their transaction id against the cluster. 307The difference between a CACHE and a SLAVE is that the SLAVE is synchronized 308to a full copy of the cluster and thus can serve as a backup or be staged 309for use as a MASTER later on. 310.Pp 311SOFT_SLAVE - Create a PFS which maintains synchronization with and provides 312a read-only copy of the cluster. 313This is one of the special mount cases. 314A SOFT_SLAVE will synchronize with 315the cluster when the cluster is available, but can still be accessed when 316the cluster is not available. 317.Pp 318MASTER - Create a PFS which will hold a master copy of the cluster. 319If you create several MASTER PFSs with the same cluster id you are 320effectively creating a multi-master cluster and causing a quorum and 321cache coherency protocol to be used to validate operations. 322The total number of masters is stored in each PFSs making up the cluster. 323Filesystem operations will stall for normal mounts if a quorum cannot be 324obtained to validate the operation. 325MASTER nodes which go offline and return later will synchronize in the 326background. 327Note that when adding a MASTER to an existing cluster you must add the 328new PFS as a SLAVE and then upgrade it to a MASTER. 329.Pp 330SOFT_MASTER - Create a PFS which maintains synchronization with and provides 331a read-write copy of the cluster. 332This is one of the special mount cases. 333A SOFT_MASTER will synchronize with 334the cluster when the cluster is available, but can still be read AND written 335to even when the cluster is not available. 336Modifications made to a SOFT_MASTER will be automatically flushed to the 337cluster when it becomes accessible again, and vise-versa. 338Manual intervention may be required if a conflict occurs during 339synchronization. 340.\" ==== pfs-delete ==== 341.It Cm pfs-delete Op label... 342Destroy a PFS by name. All hammer2 mount points will be checked, however 343this directive will refuse to delete a PFS whos name is duplicated on 344multiple mount points. A specific mount point may be specified to restrict 345the deletion via the 346.Fl s Ar mount 347option. 348.\" ==== snapshot ==== 349.It Cm snapshot Ar path Op label 350Create a snapshot of a directory. 351The snapshot will be created on the same hammer2 storage device as the 352directory. 353This can only be used on a local PFS, and is only really useful if the PFS 354contains a complete copy of what you desire to snapshot so that typically 355means a local MASTER, SOFT_MASTER, SLAVE, or SOFT_SLAVE must be present. 356Snapshots are created simply by flushing a PFS mount to disk and then copying 357the directory inode to the PFS. 358The topology is snapshotted without having to be copied or scanned and 359take no additional space. 360However, bulkfree scans may take longer. 361Snapshots are effectively separate from the cluster they came from 362and can be used as a starting point for a new cluster. 363So unless you build a new cluster from the snapshot, it will stay local 364to the machine it was made on. 365.Pp 366Snapshots can be maintained automatically with 367.Xr periodic 8 . 368See 369.Xr periodic.conf 5 370for details of enabling and configuring the functionality. 371.\" ==== snapshot-debug ==== 372.It Cm snapshot-debug Ar path Op label 373Snapshot without filesystem sync. 374.\" ==== service ==== 375.It Cm service 376Start the 377.Nm 378service daemon. 379This daemon is also automatically started when you run 380.Xr mount_hammer2 8 . 381The hammer2 service daemon handles incoming TCP connections and maintains 382outgoing TCP connections. 383It will interconnect available services on the 384machine (e.g. hammer2 mounts and xdisks) to the network. 385.\" ==== stat ==== 386.It Cm stat Op path... 387Print the inode statistics, compression, and other meta-data associated 388with a list of paths. 389.\" ==== leaf ==== 390.It Cm leaf 391XXX 392.\" ==== shell ==== 393.It Cm shell Op host 394Start a debug shell to the local hammer2 service daemon via the DMSG protocol. 395.\" ==== debugspan ==== 396.It Cm debugspan Ar target 397(do not use) 398.\" ==== rsainit ==== 399.It Cm rsainit Op path 400Create the 401.Pa /etc/hammer2 402directory and initialize a public/private keypair in that directory for 403use by the network cluster protocols. 404.\" ==== show ==== 405.It Cm show Ar devpath 406Dump the radix tree for the HAMMER2 filesystem by scanning a 407block device directly. 408No mount is required. 409.\" ==== freemap ==== 410.It Cm freemap Ar devpath 411Dump the freemap tree for the HAMMER2 filesystem by scanning a 412block device directly. 413No mount is required. 414.\" ==== volhdr ==== 415.It Cm volhdr Ar devpath 416Dump the volume header for the HAMMER2 filesystem by scanning a 417block device directly. 418No mount is required. 419.\" ==== volume-list ==== 420.It Cm volume-list Op path... 421List all volumes associated with all mounted hammer2 storage devices. 422The list may be restricted to a particular filesystem using 423.Fl s Ar mount . 424.Pp 425Note that hammer2 volumes associated with storage devices which have not been 426mounted in any fashion will not be listed. At least one hammer2 label must 427be mounted for the volumes on that device to be visible. 428.\" ==== setcomp ==== 429.It Cm setcomp Ar mode[:level] Ar path... 430Set the compression mode as specified for any newly created elements at or 431under the path if not overridden by deeper elements. 432Available modes are none, autozero, lz4, or zlib. 433When zlib is used the compression level can be set. 434The default will be 6 which is the best trade-off between performance and 435time. 436.Pp 437newfs_hammer2 will set the default compression to lz4 which prioritizes 438speed over performance. 439Also note that HAMMER2 contains a heuristic and will not attempt to 440compress every block if it detects a sufficient amount of uncompressable 441data. 442.Pp 443Hammer2 compression is only effective when it can reduce the size of dataset 444(typically a 64KB block) by one or more powers of 2. A 64K block which 445only compresses to 40K will not yield any storage improvement. 446.Pp 447Generally speaking you do not want to set the compression mode to 448.Sq none , 449as this will cause blocks of all-zeros to be written as all-zero blocks, 450instead of holes. 451The 452.Sq autozero 453compression mode detects blocks of all-zeros 454and writes them as holes. 455.\" ==== setcheck ==== 456.It Cm setcheck Ar check Ar path... 457Set the check code as specified for any newly created elements at or under 458the path if not overridden by deeper elements. 459Available codes are default, disabled, crc32, xxhash64, or sha192. 460.Pp 461Normally HAMMER2 does not overwrite data blocks on the media in order 462to ensure snapshot integrity. Replacement data blocks will be reallocated. 463However, if the compression mode is set to 464.Sq none 465and the check code is set to 466.Sq disabled 467HAMMER2 will overwrite data on the media in-place. 468In this mode of operation, 469snapshots will not be able to snapshot the data against later changes 470made to the file, and de-duplication will no longer function on any 471data related to the file. 472However, you can still recover the most recent data from previously 473taken snapshots if you accidentally remove the file. 474.\" ==== clrcheck ==== 475.It Cm clrcheck Op path... 476Clear the check code override for the specified paths. 477Overrides may still be present in deeper elements. 478.\" ==== setcrc32 ==== 479.It Cm setcrc32 Op path... 480Set the check code to the ISCSI 32-bit CRC for any newly created elements 481at or under the path if not overridden by deeper elements. 482.\" ==== setxxhash64 ==== 483.It Cm setxxhash64 Op path... 484Set the check code to XXHASH64, a fast 64-bit hash 485.\" ==== setsha192 ==== 486.It Cm setsha192 Op path... 487Set the check code to SHA192 for any newly created elements at or under 488the path if not overridden by deeper elements. 489.\" ==== bulkfree ==== 490.It Cm bulkfree Ar path 491Run a bulkfree pass on a HAMMER2 mount. 492You can specify any PFS for the mount, the bulkfree pass is run on the 493entire partition. 494Note that it takes two passes to actually free space. 495By default this directive will use up to 1/16 physical memory to track 496the freemap. 497The amount of memory used may be overridden with the 498.Op Fl m Ar mem 499option. 500.\" ==== printinode ==== 501.It Cm printinode Ar path 502Dump inode. 503.\" ==== dumpchain ==== 504.It Cm dumpchain Op path Op chnflags 505Dump in-memory chain topology. 506.El 507.Sh SYSCTLS 508.Bl -tag -width indent 509.It Va vfs.hammer2.dedup_enable (default on) 510Enables live de-duplication. 511Any recently read data that is on-media 512(already synchronized to media) is tested against pending writes for 513compatibility. 514If a match is found, the write will reference the 515existing on-media data instead of writing new data. 516.It Va vfs.hammer2.always_compress (default off) 517This disables the H2 compression heuristic and forces H2 to always 518try to compress data blocks, even if they look uncompressable. 519Enabling this option reduces performance but has higher de-duplication 520repeatability. 521.It Va vfs.hammer2.cluster_data_read (default 4) 522.It Va vfs.hammer2.cluster_meta_read (default 1) 523Set the amount of read-ahead clustering to perform on data and meta-data 524blocks. 525.It Va vfs.hammer2.cluster_write (default 0) 526Set the amount of write-behind clustering to perform in buffers. 527Each buffer represents 64KB. 528The default is 4 and higher values typically do not improve performance. 529A value of 0 disables clustered writes. 530This variable applies to the underlying media device, not to logical 531file writes, so it should not interfere with temporary file optimization. 532Generally speaking you want this enabled to generate smoothly pipelined 533writes to the media. 534.It Va vfs.hammer2.bulkfree_tps (default 5000) 535Set bulkfree's maximum scan rate. 536This is primarily intended to limit 537I/O utilization on SSDs and CPU utilization when the meta-data is mostly 538cached in memory. 539.El 540.Sh SETTING UP /etc/hammer2 541The 542.Sq rsainit 543directive will create the 544.Pa /etc/hammer2 545directory with appropriate permissions and also generate a public key 546pair in this directory for the machine. 547These files will be 548.Pa rsa.pub 549and 550.Pa rsa.prv 551and needless to say, the private key shouldn't leave the host. 552.Pp 553The service daemon will also scan the 554.Pa /etc/hammer2/autoconn 555file which contains a list of hosts which it will automatically maintain 556connections to to form your cluster. 557The service daemon will automatically reconnect on any failure and will 558also monitor the file for changes. 559.Pp 560When the service daemon receives a connection it expects to find a 561public key for that connection in a file in 562.Pa /etc/hammer2/remote/ 563called 564.Pa <IPADDR>.pub . 565You normally copy the 566.Pa rsa.pub 567key from the host in question to this file. 568The IP address must match exactly or the connection will not be allowed. 569.Pp 570If you want to use an unencrypted connection you can create empty, 571dummy files in the remote directory in the form 572.Pa <IPADDR>.none . 573We do not recommend using unencrypted connections. 574.Sh CLUSTER SERVICES 575Currently there are two services which use the cluster network infrastructure, 576HAMMER2 mounts and XDISK. 577Any HAMMER2 mount will make all PFSs for that filesystem available to the 578cluster. 579And if the XDISK kernel module is loaded, the hammer2 service daemon will make 580your machine's block devices available to the cluster (you must load the 581xdisk.ko kernel module before starting the hammer2 service). 582They will show up as 583.Pa /dev/xa* 584and 585.Pa /dev/serno/* 586devices on the remote machines making up the cluster. 587Remote block devices are just what they appear to be... direct access to a 588block device on a remote machine. 589If the link goes down remote accesses 590will stall until it comes back up again, then automatically requeue any 591pending I/O and resume as if nothing happened. 592However, if the server hosting the physical disks crashes or is rebooted, 593any remote opens to its devices will see a permanent I/O failure requiring a 594close and open sequence to re-establish. 595The latter is necessary because the server's drives might not have committed 596the data before the crash, but had already acknowledged the transfer. 597.Pp 598Data commits work exactly the same as they do for real block devices. 599The originater must issue a BUF_CMD_FLUSH. 600.Sh ADDING A NEW MASTER TO A CLUSTER 601When you 602.Xr newfs_hammer2 8 603a HAMMER2 filesystem or use the 604.Sq pfs-create 605directive on one already mounted 606to create a new PFS, with no special options, you wind up with a PFS 607typed as a MASTER and a unique cluster UUID, but because there is only one 608PFS for that cluster (for each PFS you create via pfs-create), it will 609act just like a normal filesystem would act and does not require any special 610protocols to operate. 611.Pp 612If you use the 613.Sq pfs-create 614directive along with the 615.Fl u 616option to specify a cluster UUID that already exists in the cluster, 617you are adding a PFS to an existing cluster and this can trigger a whole 618series of events in the background. 619When you specify the 620.Fl u 621option in a 622.Sq pfs-create , 623.Nm 624will by default create a SLAVE PFS. 625In fact, this is what must be created first even if you want to add a new 626MASTER to your cluster. 627.Pp 628The most common action a system admin will want to take is to upgrade or 629downgrade a PFS. 630A new MASTER can be added to the cluster by upgrading an existing SLAVE 631to MASTER. 632A MASTER can be removed from the cluster by downgrading it to a SLAVE. 633Upgrades and downgrades will put nodes in the cluster in a transition state 634until the operation is complete. 635For downgrades the transition state is fleeting unless one or more other 636masters has not acknowledged the change. 637For upgrades a background synchronization process must complete before the 638transition can be said to be complete, and the node remains (really) a SLAVE 639until that transition is complete. 640.Sh USE CASES FOR A SOFT_MASTER 641The SOFT_MASTER PFS type is a special type which must be specifically 642mounted by a machine. 643It is a R/W mount which does not use the quorum protocol and is not 644cache coherent with the cluster, but which synchronizes from the cluster 645and allows modifying operations which will synchronize to the cluster. 646The most common case is to use a SOFT_MASTER PFS in a laptop allowing you 647to work on your laptop when you are on the road and not connected to 648your main servers, and for the laptop to synchronize when a connection is 649available. 650.Sh USE CASES FOR A SOFT_SLAVE 651A SOFT_SLAVE PFS type is a special type which must be specifically mounted 652by a machine. 653It is a RO mount which does not use the quorum protocol and is not 654cache coherent with the cluster. 655It will receive synchronization from 656the cluster when network connectivity is available but will not stall if 657network connectivity is lost. 658.Sh FSYNC FLUSH MODES 659TODO. 660.Sh RESTORING FROM A SNAPSHOT BACKUP 661TODO. 662.Sh PERFORMANCE TUNING 663Because HAMMER2 implements compression, decompression, and dedup natively, 664it always double-buffers file data. 665This means that the file data is 666cached via the device vnode (in compressed / dedupped-form) and the same 667data is also cached by the file vnode (in decompressed / non-dedupped form). 668.Pp 669While HAMMER2 will try to age the logical file buffers on its, some 670additional performance tuning may be necessary for optimal operation 671whether swapcache is used or not. 672Our recommendation is to reduce the 673number of vnodes (and thus also the logical buffer cache behind the 674vnodes) that the system caches via the 675.Va kern.maxvnodes 676sysctl. 677.Pp 678Too-large a value will result in excessive double-caching and can cause 679unnecessary read disk I/O. 680We recommend a number between 25000 and 250000 vnodes, depending on your 681use case. 682Keep in mind that even though the vnode cache is smaller, this will make 683room for a great deal more device-level buffer caching which can encompasses 684far more data and meta-data than the vnode-level caching. 685.Sh ENVIRONMENT 686TODO. 687.Sh FILES 688.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 689.It Pa /etc/hammer2/ 690.It Pa /etc/hammer2/rsa.pub 691.It Pa /etc/hammer2/rsa.prv 692.It Pa /etc/hammer2/autoconn 693.It Pa /etc/hammer2/remote/<IP>.pub 694.It Pa /etc/hammer2/remote/<IP>.none 695.El 696.Sh EXIT STATUS 697.Ex -std 698.Sh SEE ALSO 699.Xr periodic.conf 5 , 700.Xr mount_hammer2 8 , 701.Xr mount_null 8 , 702.Xr newfs_hammer2 8 , 703.Xr swapcache 8 , 704.Xr sysctl 8 705.Sh HISTORY 706The 707.Nm 708utility first appeared in 709.Dx 4.1 . 710.Sh AUTHORS 711.An Matthew Dillon Aq Mt dillon@backplane.com 712