1.\" Copyright (c) 2015-2023 The DragonFly Project. All rights reserved. 2.\" 3.\" This code is derived from software contributed to The DragonFly Project 4.\" by Matthew Dillon <dillon@backplane.com> 5.\" 6.\" Redistribution and use in source and binary forms, with or without 7.\" modification, are permitted provided that the following conditions 8.\" are met: 9.\" 10.\" 1. Redistributions of source code must retain the above copyright 11.\" notice, this list of conditions and the following disclaimer. 12.\" 2. Redistributions in binary form must reproduce the above copyright 13.\" notice, this list of conditions and the following disclaimer in 14.\" the documentation and/or other materials provided with the 15.\" distribution. 16.\" 3. Neither the name of The DragonFly Project nor the names of its 17.\" contributors may be used to endorse or promote products derived 18.\" from this software without specific, prior written permission. 19.\" 20.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 21.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 22.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 23.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 24.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 25.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 26.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 27.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 28.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 29.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 30.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 31.\" SUCH DAMAGE. 32.\" 33.Dd November 2, 2021 34.Dt HAMMER2 8 35.Os 36.Sh NAME 37.Nm hammer2 38.Nd hammer2 file system utility 39.Sh SYNOPSIS 40.Nm 41.Fl h 42.Nm 43.Op Fl d 44.Op Fl s Ar path 45.Op Fl t Ar type 46.Op Fl u Ar uuid 47.Op Fl m Ar mem 48.Ar command 49.Op Ar argument ... 50.Sh DESCRIPTION 51The 52.Nm 53utility provides miscellaneous support functions for a 54HAMMER2 file system. 55.Pp 56The options are as follows: 57.Bl -tag -width indent 58.It Fl d 59Enables debug mode. 60It can be specified more than once to enable additional debugging messages. 61In some directives it may cause 62.Nm 63to run in foreground instead of being daemonized. 64.It Fl s Ar path 65Specify the path to a mounted HAMMER2 filesystem. 66At least one PFS on a HAMMER2 filesystem must be mounted for the system 67to act on all PFSs managed by it. 68Every HAMMER2 filesystem typically has a PFS called "LOCAL" for this purpose. 69.It Fl t Ar type 70Specify the type when creating, upgrading, or downgrading a PFS. 71Supported types are MASTER, SLAVE, SOFT_MASTER, SOFT_SLAVE, CACHE, and DUMMY. 72If not specified the pfs-create directive will default to MASTER if no 73UUID is specified, and SLAVE if a UUID is specified. 74.It Fl u Ar uuid 75Specify the cluster UUID when creating a PFS. 76If not specified, a unique, random UUID will be generated. 77Note that every PFS also has a unique pfs_id which is always generated 78and cannot be overridden with an option. 79The { pfs_clid, pfs_fsid } tuple uniquely identifies a component of a cluster. 80.It Fl m Ar mem 81Specify how much tracking memory to use for certain directives. 82At the moment, this option is only applicable to the 83.Cm bulkfree 84directive, allowing it to operate in fewer passes when given more memory. 85A nominal value for a 4TB drive with a ton of stuff on it would be around 86a gigabyte '-m 1g'. 87.El 88.Pp 89.Nm 90directives are as shown below. 91Note that most directives require you to either be CD'd into a hammer2 92filesystem, specify a path to a mounted hammer2 filesystem via the 93.Fl s 94option, or specify a path after the directive. 95It depends on the directive. 96All hammer2 filesystems have a PFS called "LOCAL" which is typically mounted 97locally on the host in order to be able to issue commands for other PFSs 98on the filesystem. 99The mount also enables PFS configuration scanning for that filesystem. 100.Bl -tag -width indent 101.\" ==== cleanup ==== 102.It Cm cleanup Op path 103Perform manual cleanup passes on paths or all mounted partitions. 104.\" ==== connect ==== 105.It Cm connect Ar target 106Add a cluster link entry to the volume header. 107The volume header can support up to 255 link entries. 108This feature is not currently used. 109.\" ==== destroy ==== 110.It Cm destroy Ar path... 111Destroy the specified directory entry in a hammer2 filesystem. 112This bypasses 113all normal checks and will unconditionally destroy the directory entry. 114The underlying inode is not checked and, if it does exist, its nlinks count 115is not decremented. 116This directive should only be used to destroy a corrupted directory entry 117which no longer has a working inode. 118.Pp 119Note that this command may desynchronize the system namecache for the 120specified entry. 121If this happens, you may have to unmount and remount the filesystem. 122.\" ==== destroy-inum ==== 123.It Cm destroy-inum Ar path... 124Destroy the specified inode in a hammer2 filesystem. 125.\" ==== disconnect ==== 126.It Cm disconnect Ar target 127Delete a cluster link entry from the volume header. 128This feature is not currently used. 129.\" ==== emergency-mode-enable === 130.It Cm emergency-mode-enable Ar target 131Flag emergency operations mode in the filesystem. 132This mode may be used 133as a last resort to delete files and directories from a full filesystem. 134Inode creation, file writes, and certain meta-data cleanups are disallowed 135while emergency mode is active. 136File and directory removal and mode/attr setting is still allowed. 137This mode is extremely dangerous and should only be used as a last resort. 138.Pp 139This mode allows the filesystem to modify blocks in-place when it is unable 140to allocate a copy. 141Thus it is possible to chflags and remove files and 142directories even when the filesystem is completely full. 143However, there is a price. 144This mode of operation WILL LIKELY CORRUPT ANY SNAPSHOTS related 145to this filesystem. 146The filesystem will report this condition if it encounters 147it but if you are forced to use this mode to fix a filesystem full condition 148your snapshots can get a bit dicey. 149It is usually safest to delete any related snapshots when using this mode. 150.Pp 151You can detect whether related snapshots have been corrupted by running 152a bulkfree pass and checking the console output for reported CRC errors. 153If no errors are reported, your snapshots are fine. 154If errors are reported 155you should delete related snapshots until bulkfree reports no further errors. 156.Pp 157The emergency mode will also make meta-data updates unsafe due to the lack of 158copy-on-write, causing potential harm if the system unexpectedly panics or 159loses power. 160GREAT CARE MUST BE TAKEN WHILE THIS MODE IS ACTIVE. 161.Bl -enum 162.It 163Determine that you are unable to recover space with normal file and directory 164removal commands due to 165.Er ENOSPC 166errors being returned by 'rm', or through the 167removal of snapshots (if any). The 'bulkfree' directive must be issued to 168scan the filesystem and free up the actual space, then check with 'df'. 169Continue if you still have insufficient space and are unable to remove items 170normally. 171.It 172If you need any related snapshots, this is a good time to copy them elsewhere. 173.It 174Idle or kill any processes trying to use the filesystem. 175.It 176Issue the emergency-mode-enable directive on the filesystem. 177Once enabled, run 'sync' to update any dirty inodes which may still 178be dirty due to not being able to flush. 179Please remember that this 180directive is a LAST RESORT, is dangerous, and will likely corrupt any 181other snapshots you have based on the filesystem you are removing files 182from. 183.It 184Remove file trees as necessary with 'rm -rf' to free space, being cognizant 185of any warnings issued by the kernel on the console (via 'dmesg') while 186doing so. 187.It 188Issue the 'bulkfree' directive to actually free the space and check that 189sufficient space has been freed with 'df'. 190.It 191If bulkfree reports CHECK errors, or if you have snapshots and insufficient 192space has been freed, you will need to delete snapshots. 193Re-run bulkfree and delete snapshots until no errors are reported. 194.It 195Issue the emergency-mode-disable directive when done. 196It might also be a 197good idea to reboot after using this mode, but theoretically you should not 198have to. 199.It 200Restore services using the filesystem. 201.El 202.\" ==== emergency-mode-disable === 203.It Cm emergency-mode-disable Ar target 204Turn off the emergency operations mode on a filesystem, restoring normal 205operation. 206.\" ==== growfs ==== 207.It Cm growfs Op fspath... 208After resizing the disk partition you can issue this command on a 209mounted hammer2 filesystem to grow into the new space in the partition. 210This command is run on a live hammer2 filesystem. 211.\" ==== info ==== 212.It Cm info Op devpath... 213Access and print the status and super-root entries for all HAMMER2 214partitions found in /dev/serno or the specified device path(s). 215The partitions do not have to be mounted. 216Note that only mounted partitions will be under active management. 217This is accomplished by mounting at least one PFS within the partition. 218Typically at least the @LOCAL PFS is mounted. 219.\" ==== mountall ==== 220.It Cm mountall Op devpath... 221This directive mounts the @LOCAL PFS on all HAMMER2 partitions found 222in /dev/serno, or the specified device path(s). 223The partitions are mounted as /var/hammer2/LOCAL.<id>. 224Mounts are executed in the background and this command will wait a 225limited amount of time for the mounts to complete before returning. 226.\" ==== status ==== 227.It Cm status Op path... 228Dump a list of all cluster link entries configured in the volume header. 229.\" ==== hash ==== 230.It Cm hash Op filename... 231Compute and print the directory hash for any number of filenames. 232.\" ==== dhash ==== 233.It Cm dhash Op filename... 234Compute and print the data hash for long directory entry for any number of 235filenames. 236.\" ==== pfs-list ==== 237.It Cm pfs-list Op path... 238List all PFSs associated with all mounted hammer2 storage devices. 239The list may be restricted to a particular filesystem using 240.Fl s Ar mount . 241.Pp 242Note that hammer2 PFSs associated with storage devices which have not been 243mounted in any fashion will not be listed. 244At least one hammer2 label must be mounted for the PFSs on that device to be 245visible. 246.\" ==== pfs-clid ==== 247.It Cm pfs-clid Ar label 248Print the cluster id for a PFS specified by name. 249.\" ==== pfs-fsid ==== 250.It Cm pfs-fsid Ar label 251Print the unique filesystem id for a PFS specified by name. 252.\" ==== pfs-create ==== 253.It Cm pfs-create Ar label 254Create a local PFS on the mounted HAMMER2 filesystem represented 255by the current directory, or specified via 256.Fl s Ar mount . 257If no UUID is specified the pfs-type defaults to MASTER. 258If a UUID is specified via the 259.Fl u 260option the pfs-type defaults to SLAVE. 261Other types can be specified with the 262.Fl t 263option. 264.Pp 265If you wish to add a MASTER to an existing cluster, you must first add it as 266a SLAVE and then upgrade it to MASTER to properly synchronize it. 267.Pp 268The DUMMY pfs-type is used to tie network-accessible clusters into the local 269machine when no local storage is desired. 270This type should be used on minimal H2 partitions or entirely in ram for 271netboot-centric systems to provide a tie-in point for the mount command, 272or on more complex systems where you need to also access network-centric 273clusters. 274.Pp 275The CACHE or SLAVE pfs-type is typically used when the main store is on 276the network but local storage is desired to improve performance. 277SLAVE is also used when a backup is desired. 278.Pp 279Generally speaking, you can mount any PFS element of a cluster in order to 280access the cluster via the full cluster protocol. 281There are two exceptions. 282If you mount a SOFT_SLAVE or a SOFT_MASTER then soft quorum semantics are 283employed... the soft slave or soft master's current state will always be used 284and the quorum protocol will not be used. 285The soft PFS will still be 286synchronized to masters in the background when available. 287Also, you can use 288.Sq mount -o local 289to mount ONLY a local HAMMER2 PFS and 290not run any network or quorum protocols for the mount. 291All such mounts except for a SOFT_MASTER mount will be read-only. 292Other than that, you will be mounting the whole cluster when you mount any 293PFS within the cluster. 294.Pp 295DUMMY - Create a PFS skeleton intended to be the mount point for a 296more complex cluster, probably one that is entirely network based. 297No data will be synchronized to this PFS so it is suitable for use 298in a network boot image or memory filesystem. 299This allows you to create placeholders for mount points on your local 300disk, SSD, or memory disk. 301.Pp 302CACHE - Create a PFS for caching portions of the cluster piecemeal. 303This is similar to a SLAVE but does not synchronize the entire contents of 304the cluster to the PFS. 305Elements found in the CACHE PFS which are validated against the cluster 306will be read, presumably a faster access than having to go to the cluster. 307Only local CACHEs will be updated. 308Network-accessible CACHE PFSs might be read but will not be written to. 309If you have a large hard-drive-based cluster you can set up localized 310SSD CACHE PFSs to improve performance. 311.Pp 312SLAVE - Create a PFS which maintains synchronization with and provides a 313read-only copy of the cluster. 314HAMMER2 will prioritize local SLAVEs for data retrieval after validating 315their transaction id against the cluster. 316The difference between a CACHE and a SLAVE is that the SLAVE is synchronized 317to a full copy of the cluster and thus can serve as a backup or be staged 318for use as a MASTER later on. 319.Pp 320SOFT_SLAVE - Create a PFS which maintains synchronization with and provides 321a read-only copy of the cluster. 322This is one of the special mount cases. 323A SOFT_SLAVE will synchronize with 324the cluster when the cluster is available, but can still be accessed when 325the cluster is not available. 326.Pp 327MASTER - Create a PFS which will hold a master copy of the cluster. 328If you create several MASTER PFSs with the same cluster id you are 329effectively creating a multi-master cluster and causing a quorum and 330cache coherency protocol to be used to validate operations. 331The total number of masters is stored in each PFSs making up the cluster. 332Filesystem operations will stall for normal mounts if a quorum cannot be 333obtained to validate the operation. 334MASTER nodes which go offline and return later will synchronize in the 335background. 336Note that when adding a MASTER to an existing cluster you must add the 337new PFS as a SLAVE and then upgrade it to a MASTER. 338.Pp 339SOFT_MASTER - Create a PFS which maintains synchronization with and provides 340a read-write copy of the cluster. 341This is one of the special mount cases. 342A SOFT_MASTER will synchronize with 343the cluster when the cluster is available, but can still be read AND written 344to even when the cluster is not available. 345Modifications made to a SOFT_MASTER will be automatically flushed to the 346cluster when it becomes accessible again, and vise-versa. 347Manual intervention may be required if a conflict occurs during 348synchronization. 349.\" ==== pfs-delete ==== 350.It Cm pfs-delete Op label... 351Destroy a PFS by name. 352All hammer2 mount points will be checked, however this directive will refuse to 353delete a PFS whos name is duplicated on multiple mount points. 354A specific mount point may be specified to restrict the deletion via the 355.Fl s Ar mount 356option. 357.\" ==== recover ==== 358.It Cm recover Ar media Ar path Ar destdir 359Recover a file or directory tree by scanning the raw media partition, 360with minimal requirements for an intact topology. The results are written 361to the destination directory. 362.Pp 363The recovery path can be anchored at (any) root by prefixing it with a "/". 364If not anchored, any matching sub-tree will be recovered and combined into 365one place on the destination (not as separate sub-trees since that requires 366topological knowledge that might not be available). Roots include all 367PFS roots and snapshot roots, as well as disconnected roots from COW updates 368(aka to be able to undelete a file or directory sub-tree). 369.Pp 370The path may specify a directory or file to restore. Note that if you 371specify something like ".cshrc", then all ".cshrc" files found in the 372entire filesystem will be recovered. Multiple references to the same 373physical inode with the same filename in the same directory are filtered out. 374.Pp 375This function is meant for catastrophic recovery or corrupted media or 376recovery for deleted files that are not otherwise available in snapshots. 377All possible versions of files will be recovered and suffixed as "*.%05d" 378with an iterator. 379.Pp 380The hammer2 recovery function is not meant to generate a fully operational 381filesystem in the target directory. All files will be versioned and contain 382iteration suffixes. Many files and sub-directory trees may wind up glommed 383together in one place. However, this function will recover mtime, ownership, 384group, modes, and flags. Recovered files are fully validated and any missing 385data will cause the file to be renamed with a ".corrupted" suffix. 386.Pp 387Memory requirements - This function needs to track inodes and paths and 388can eat a considerable amount of ram, and it tends to be random-access 389so while swap can make ends meet, it might slow the scan down drastically 390if it has to eat into it much. 391Ram consumption scales with the number of inodes found. Call it a few 392gigabytes up to 500GB of media and approaching 64GB on busy 4TB media. 393Busy meaning, say, 50 million topological inodes turning into 1 billion 394inodes located by the media scan. 395.\" ==== recover-relaxed ==== 396.It Cm recover-relaxed Ar media Ar path Ar destdir 397This version of the recover directive relaxes bref checks. Under normal 398operation, only brefs with check codes are allowed because we get too many 399false hits otherwise. In relaxed mode, bref's with the check code disabled 400are also allowed. 401.Pp 402You should only need to use the relaxed version if you have turned off 403CRCs on the files you want to recover. 404.\" ==== recover-file ==== 405.It Cm recover-file Ar media Ar path Ar destdir 406This version of recover is the normal strict-mode recover but explicitly 407indicates that the path being recovered is a regular file and not a 408directory. Prevents junk recursions when entries corresponding to the 409last path element appear to be directories. 410.\" ==== snapshot ==== 411.It Cm snapshot Ar path Op label 412Create a snapshot of a directory. 413The snapshot will be created on the same hammer2 storage device as the 414directory. 415This can only be used on a local PFS, and is only really useful if the PFS 416contains a complete copy of what you desire to snapshot so that typically 417means a local MASTER, SOFT_MASTER, SLAVE, or SOFT_SLAVE must be present. 418Snapshots are created simply by flushing a PFS mount to disk and then copying 419the directory inode to the PFS. 420The topology is snapshotted without having to be copied or scanned and 421take no additional space. 422However, bulkfree scans may take longer. 423Snapshots are effectively separate from the cluster they came from 424and can be used as a starting point for a new cluster. 425So unless you build a new cluster from the snapshot, it will stay local 426to the machine it was made on. 427.Pp 428Snapshots can be maintained automatically with 429.Xr periodic 8 . 430See 431.Xr periodic.conf 5 432for details of enabling and configuring the functionality. 433.\" ==== snapshot-debug ==== 434.It Cm snapshot-debug Ar path Op label 435Snapshot without filesystem sync. 436.\" ==== service ==== 437.It Cm service 438Start the 439.Nm 440service daemon. 441This daemon is also automatically started when you run 442.Xr mount_hammer2 8 . 443The hammer2 service daemon handles incoming TCP connections and maintains 444outgoing TCP connections. 445It will interconnect available services on the 446machine (e.g. hammer2 mounts and xdisks) to the network. 447.\" ==== stat ==== 448.It Cm stat Op path... 449Print the inode statistics, compression, and other meta-data associated 450with a list of paths. 451.\" ==== leaf ==== 452.It Cm leaf 453XXX 454.\" ==== shell ==== 455.It Cm shell Op host 456Start a debug shell to the local hammer2 service daemon via the DMSG protocol. 457.\" ==== debugspan ==== 458.It Cm debugspan Ar target 459(do not use) 460.\" ==== rsainit ==== 461.It Cm rsainit Op path 462Create the 463.Pa /etc/hammer2 464directory and initialize a public/private keypair in that directory for 465use by the network cluster protocols. 466.\" ==== show ==== 467.It Cm show Ar devpath 468Dump the radix tree for the HAMMER2 filesystem by scanning a 469block device directly. 470No mount is required. 471.\" ==== freemap ==== 472.It Cm freemap Ar devpath 473Dump the freemap tree for the HAMMER2 filesystem by scanning a 474block device directly. 475No mount is required. 476.\" ==== volhdr ==== 477.It Cm volhdr Ar devpath 478Dump the volume header for the HAMMER2 filesystem by scanning a 479block device directly. 480No mount is required. 481.\" ==== volume-list ==== 482.It Cm volume-list Op path... 483List all volumes associated with all mounted hammer2 storage devices. 484The list may be restricted to a particular filesystem using 485.Fl s Ar mount . 486.Pp 487Note that hammer2 volumes associated with storage devices which have not been 488mounted in any fashion will not be listed. 489At least one hammer2 label must be mounted for the volumes on that device to be 490visible. 491.\" ==== setcomp ==== 492.It Cm setcomp Ar mode[:level] Ar path... 493Set the compression mode as specified for any newly created elements at or 494under the path if not overridden by deeper elements. 495Available modes are none, autozero, lz4, or zlib. 496When zlib is used the compression level can be set. 497The default will be 6 which is the best trade-off between performance and 498time. 499.Pp 500newfs_hammer2 will set the default compression to lz4 which prioritizes 501speed over performance. 502Also note that HAMMER2 contains a heuristic and will not attempt to 503compress every block if it detects a sufficient amount of uncompressable 504data. 505.Pp 506Hammer2 compression is only effective when it can reduce the size of dataset 507(typically a 64KB block) by one or more powers of 2. A 64K block which 508only compresses to 40K will not yield any storage improvement. 509.Pp 510Generally speaking you do not want to set the compression mode to 511.Sq none , 512as this will cause blocks of all-zeros to be written as all-zero blocks, 513instead of holes. 514The 515.Sq autozero 516compression mode detects blocks of all-zeros 517and writes them as holes. 518.\" ==== setcheck ==== 519.It Cm setcheck Ar check Ar path... 520Set the check code as specified for any newly created elements at or under 521the path if not overridden by deeper elements. 522Available codes are default, disabled, crc32, xxhash64, or sha192. 523.Pp 524Normally HAMMER2 does not overwrite data blocks on the media in order to ensure 525snapshot integrity. 526Replacement data blocks will be reallocated. 527However, if the compression mode is set to 528.Sq none 529and the check code is set to 530.Sq disabled 531HAMMER2 will overwrite data on the media in-place. 532In this mode of operation, 533snapshots will not be able to snapshot the data against later changes 534made to the file, and de-duplication will no longer function on any 535data related to the file. 536However, you can still recover the most recent data from previously 537taken snapshots if you accidentally remove the file. 538.\" ==== clrcheck ==== 539.It Cm clrcheck Op path... 540Clear the check code override for the specified paths. 541Overrides may still be present in deeper elements. 542.\" ==== setcrc32 ==== 543.It Cm setcrc32 Op path... 544Set the check code to the ISCSI 32-bit CRC for any newly created elements 545at or under the path if not overridden by deeper elements. 546.\" ==== setxxhash64 ==== 547.It Cm setxxhash64 Op path... 548Set the check code to XXHASH64, a fast 64-bit hash 549.\" ==== setsha192 ==== 550.It Cm setsha192 Op path... 551Set the check code to SHA192 for any newly created elements at or under 552the path if not overridden by deeper elements. 553.\" ==== bulkfree ==== 554.It Cm bulkfree Ar path 555Run a bulkfree pass on a HAMMER2 mount. 556You can specify any PFS for the mount, the bulkfree pass is run on the 557entire partition. 558Note that it takes two passes to actually free space. 559By default this directive will use up to 1/16 physical memory to track 560the freemap. 561The amount of memory used may be overridden with the 562.Op Fl m Ar mem 563option. 564.\" ==== printinode ==== 565.It Cm printinode Ar path 566Dump inode. 567.\" ==== dumpchain ==== 568.It Cm dumpchain Op path Op chnflags 569Dump in-memory chain topology. 570.El 571.Sh SYSCTLS 572.Bl -tag -width indent 573.It Va vfs.hammer2.dedup_enable "(default on)" 574Enables live de-duplication. 575Any recently read data that is on-media 576(already synchronized to media) is tested against pending writes for 577compatibility. 578If a match is found, the write will reference the 579existing on-media data instead of writing new data. 580.It Va vfs.hammer2.always_compress "(default off)" 581This disables the H2 compression heuristic and forces H2 to always 582try to compress data blocks, even if they look uncompressable. 583Enabling this option reduces performance but has higher de-duplication 584repeatability. 585.It Va vfs.hammer2.cluster_data_read "(default 4)" 586.It Va vfs.hammer2.cluster_meta_read "(default 1)" 587Set the amount of read-ahead clustering to perform on data and meta-data 588blocks. 589.It Va vfs.hammer2.cluster_write "(default 0)" 590Set the amount of write-behind clustering to perform in buffers. 591Each buffer represents 64KB. 592The default is 0 and higher values typically do not improve performance. 593A value of 0 disables clustered writes. 594This variable applies to the underlying media device, not to logical 595file writes, so it should not interfere with temporary file optimization. 596Generally speaking you want this enabled to generate smoothly pipelined 597writes to the media. 598.It Va vfs.hammer2.bulkfree_tps "(default 5000)" 599Set bulkfree's maximum scan rate. 600This is primarily intended to limit 601I/O utilization on SSDs and CPU utilization when the meta-data is mostly 602cached in memory. 603.El 604.Sh SETTING UP /etc/hammer2 605The 606.Sq rsainit 607directive will create the 608.Pa /etc/hammer2 609directory with appropriate permissions and also generate a public key 610pair in this directory for the machine. 611These files will be 612.Pa rsa.pub 613and 614.Pa rsa.prv 615and needless to say, the private key shouldn't leave the host. 616.Pp 617The service daemon will also scan the 618.Pa /etc/hammer2/autoconn 619file which contains a list of hosts which it will automatically maintain 620connections to to form your cluster. 621The service daemon will automatically reconnect on any failure and will 622also monitor the file for changes. 623.Pp 624When the service daemon receives a connection it expects to find a 625public key for that connection in a file in 626.Pa /etc/hammer2/remote/ 627called 628.Pa <IPADDR>.pub . 629You normally copy the 630.Pa rsa.pub 631key from the host in question to this file. 632The IP address must match exactly or the connection will not be allowed. 633.Pp 634If you want to use an unencrypted connection you can create empty, 635dummy files in the remote directory in the form 636.Pa <IPADDR>.none . 637We do not recommend using unencrypted connections. 638.Sh CLUSTER SERVICES 639Currently there are two services which use the cluster network infrastructure, 640HAMMER2 mounts and XDISK. 641Any HAMMER2 mount will make all PFSs for that filesystem available to the 642cluster. 643And if the XDISK kernel module is loaded, the hammer2 service daemon will make 644your machine's block devices available to the cluster (you must load the 645xdisk.ko kernel module before starting the hammer2 service). 646They will show up as 647.Pa /dev/xa* 648and 649.Pa /dev/serno/* 650devices on the remote machines making up the cluster. 651Remote block devices are just what they appear to be... direct access to a 652block device on a remote machine. 653If the link goes down remote accesses 654will stall until it comes back up again, then automatically requeue any 655pending I/O and resume as if nothing happened. 656However, if the server hosting the physical disks crashes or is rebooted, 657any remote opens to its devices will see a permanent I/O failure requiring a 658close and open sequence to re-establish. 659The latter is necessary because the server's drives might not have committed 660the data before the crash, but had already acknowledged the transfer. 661.Pp 662Data commits work exactly the same as they do for real block devices. 663The originater must issue a BUF_CMD_FLUSH. 664.Sh ADDING A NEW MASTER TO A CLUSTER 665When you 666.Xr newfs_hammer2 8 667a HAMMER2 filesystem or use the 668.Sq pfs-create 669directive on one already mounted 670to create a new PFS, with no special options, you wind up with a PFS 671typed as a MASTER and a unique cluster UUID, but because there is only one 672PFS for that cluster (for each PFS you create via pfs-create), it will 673act just like a normal filesystem would act and does not require any special 674protocols to operate. 675.Pp 676If you use the 677.Sq pfs-create 678directive along with the 679.Fl u 680option to specify a cluster UUID that already exists in the cluster, 681you are adding a PFS to an existing cluster and this can trigger a whole 682series of events in the background. 683When you specify the 684.Fl u 685option in a 686.Sq pfs-create , 687.Nm 688will by default create a SLAVE PFS. 689In fact, this is what must be created first even if you want to add a new 690MASTER to your cluster. 691.Pp 692The most common action a system admin will want to take is to upgrade or 693downgrade a PFS. 694A new MASTER can be added to the cluster by upgrading an existing SLAVE 695to MASTER. 696A MASTER can be removed from the cluster by downgrading it to a SLAVE. 697Upgrades and downgrades will put nodes in the cluster in a transition state 698until the operation is complete. 699For downgrades the transition state is fleeting unless one or more other 700masters has not acknowledged the change. 701For upgrades a background synchronization process must complete before the 702transition can be said to be complete, and the node remains (really) a SLAVE 703until that transition is complete. 704.Sh USE CASES FOR A SOFT_MASTER 705The SOFT_MASTER PFS type is a special type which must be specifically 706mounted by a machine. 707It is a R/W mount which does not use the quorum protocol and is not 708cache coherent with the cluster, but which synchronizes from the cluster 709and allows modifying operations which will synchronize to the cluster. 710The most common case is to use a SOFT_MASTER PFS in a laptop allowing you 711to work on your laptop when you are on the road and not connected to 712your main servers, and for the laptop to synchronize when a connection is 713available. 714.Sh USE CASES FOR A SOFT_SLAVE 715A SOFT_SLAVE PFS type is a special type which must be specifically mounted 716by a machine. 717It is a RO mount which does not use the quorum protocol and is not 718cache coherent with the cluster. 719It will receive synchronization from 720the cluster when network connectivity is available but will not stall if 721network connectivity is lost. 722.Sh FSYNC FLUSH MODES 723TODO. 724.Sh RESTORING FROM A SNAPSHOT BACKUP 725TODO. 726.Sh PERFORMANCE TUNING 727Because HAMMER2 implements compression, decompression, and dedup natively, 728it always double-buffers file data. 729This means that the file data is 730cached via the device vnode (in compressed / dedupped-form) and the same 731data is also cached by the file vnode (in decompressed / non-dedupped form). 732.Pp 733While HAMMER2 will try to age the logical file buffers on its, some 734additional performance tuning may be necessary for optimal operation 735whether swapcache is used or not. 736Our recommendation is to reduce the 737number of vnodes (and thus also the logical buffer cache behind the 738vnodes) that the system caches via the 739.Va kern.maxvnodes 740sysctl. 741.Pp 742Too-large a value will result in excessive double-caching and can cause 743unnecessary read disk I/O. 744We recommend a number between 25000 and 250000 vnodes, depending on your 745use case. 746Keep in mind that even though the vnode cache is smaller, this will make 747room for a great deal more device-level buffer caching which can encompasses 748far more data and meta-data than the vnode-level caching. 749.Sh ENVIRONMENT 750TODO. 751.Sh FILES 752.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 753.It Pa /etc/hammer2/ 754.It Pa /etc/hammer2/rsa.pub 755.It Pa /etc/hammer2/rsa.prv 756.It Pa /etc/hammer2/autoconn 757.It Pa /etc/hammer2/remote/<IP>.pub 758.It Pa /etc/hammer2/remote/<IP>.none 759.El 760.Sh EXIT STATUS 761.Ex -std 762.Sh SEE ALSO 763.Xr xdisk 4 , 764.Xr periodic.conf 5 , 765.Xr mount_hammer2 8 , 766.Xr mount_null 8 , 767.Xr newfs_hammer2 8 , 768.Xr swapcache 8 , 769.Xr sysctl 8 770.Sh HISTORY 771The 772.Nm 773utility first appeared in 774.Dx 4.1 . 775.Sh AUTHORS 776.An Matthew Dillon Aq Mt dillon@backplane.com 777