hammer2: Use HAMMER2_CHECK_DEFAULT when creating inode

There are a few cases where inode check type is explicitly
specified (usually derived from parent directory inode).

These cases should use HAM

hammer2: Use HAMMER2_CHECK_DEFAULT when creating inode

There are a few cases where inode check type is explicitly
specified (usually derived from parent directory inode).

These cases should use HAMMER2_CHECK_DEFAULT rather than
HAMMER2_CHECK_XXHASH64, in the same way HAMMER2_COMP_DEFAULT
is being used.

show more ...

hammer2: Rename HAMMER2_COMP_NEWFS_DEFAULT -> HAMMER2_COMP_DEFAULT

This was originally only used within newfs_hammer2(8) when first
appeared in f481450ffccdbf50f8c9604067c25eb642a0c562 in 2013,
but

hammer2: Rename HAMMER2_COMP_NEWFS_DEFAULT -> HAMMER2_COMP_DEFAULT

This was originally only used within newfs_hammer2(8) when first
appeared in f481450ffccdbf50f8c9604067c25eb642a0c562 in 2013,
but it's now always used when creating a new inode (e.g. ioctl).

The check version of this is HAMMER2_CHECK_DEFAULT without "NEWFS_".

show more ...

usr.sbin/makefs: Add -o c|C option to specify comp|check type

mainly for debugging purpose to use "none" by default.
These options aren't available in newfs_hammer2(8) atm.

Note that level specific

usr.sbin/makefs: Add -o c|C option to specify comp|check type

mainly for debugging purpose to use "none" by default.
These options aren't available in newfs_hammer2(8) atm.

Note that level specification for "-o c" is currently unsupported,
and "sha192" for "-o C" is currently unsupported at runtime.

show more ...

sbin/{mount,newfs}_hammer2: style(9)

> normally, include <sys/types.h> OR <sys/param.h>, but not both.

sbin/newfs_hammer2: Don't allow >64 values for -s option

HAMMER2_MAX_VOLUMES == 64 is maximum number of volumes.

sbin/newfs_hammer2: Expand -s option to allow >1 values for multi-volumes

If -s option takes a single value, the behavior doesn't change.
The value is considered a limit for the total space.

If -s

sbin/newfs_hammer2: Expand -s option to allow >1 values for multi-volumes

If -s option takes a single value, the behavior doesn't change.
The value is considered a limit for the total space.

If -s option takes multiple values using ':' separator, each value
is considered a limit for each multi-volumes volume, rather than each
volume size deducted from remaining size.

If multiple values is specified, number of values must match
number of volumes.

show more ...

sbin/newfs_hammer2: Add newfs(8) -s option equivalent option

Very useful for testing, file backed volume, etc.
The difference vs newfs(8) is newfs_hammer2(8) takes filesystem
size in bytes, using th

sbin/newfs_hammer2: Add newfs(8) -s option equivalent option

Very useful for testing, file backed volume, etc.
The difference vs newfs(8) is newfs_hammer2(8) takes filesystem
size in bytes, using the same format as -b or -r options.

The volume size is deducted from the value (in argv[i] order,
where argc is number of volumes), and if the value reaches zero
while there are remaining volumes, the program fails.

Since -s option requires 1GiB (level1 size) or larger,
the failure mentioned above happens only for multi-volumes.

show more ...

usr.sbin/makefs: Add HAMMER2 support

This commit adds HAMMER2 image creation support for makefs(8).
It runs newfs_hammer2(8) and then sys/vfs/hammer2 logic in userspace
to create HAMMER2 image from

usr.sbin/makefs: Add HAMMER2 support

This commit adds HAMMER2 image creation support for makefs(8).
It runs newfs_hammer2(8) and then sys/vfs/hammer2 logic in userspace
to create HAMMER2 image from a given directory.

This commit splits newfs_hammer2(8) into newfs and mkfs part simlarly
to newfs_msdos(8), so that makefs(8) can use newfs functionality.
The entire sys/vfs/hammer2 (with exception of unneeded
hammer2_{bulkfree,ccms,iocom,ioctl,msgops,synchro}.[hc] and reusable
hammer2_disk.h) is copied to usr.sbin/makefs with below modification.
It intends to have minimum amount of diff against sys/vfs/hammer2.

* Header includes are modified so that it compiles in userspace.
* VFS and other kernel functions are usually implemented as simple
stub functions in hammer2_compat.h and hammer2_buf.c, but some are
commented out.
* Kernel functions such as kprintf, kmalloc, kprintf, kstrdup, etc
are implemented using corresponding libc functions.
* Lock primitives are basically NOP, and they (should) never block
as makefs(8) is a single thread program.
* struct vnode and struct buf (the ones defined locally in makefs(8),
not sys/sys/*) have new struct members only used by HAMMER2 to
emulate VFS behavior required by HAMMER2.
* Since makefs(8) is write-only, VOP_{NRESOLVE,NCREATE,NMKDIR,NLINK,
NSYMLINK,WRITE,STRATEGY} are implemented, but other VOPs just
return EOPNOTSUPP.
* VOP_{INACTIVE,RECLAIM} may be implemented and used in future to
better emulate VFS behavior to address current limitation.
* VOP_WRITE is modified to directly call VOP_STRATEGY function.
* The XOP kernel thread is modified to act as a regular function
called from VOPs, along with simplified admin code.

It currently has following limitations.

* multi-volumes is unsupported, simply due to makefs(8) only taking 1
image file path.
* Not necessarily a limitation, but it only supports populating 1 PFS,
which is "DATA" by default. Other PFSes if any won't have anything
under the root PFS inode.
* makefs(8) process gets killed by OOM for a directory with *extremely*
large number of files, depending on available memory. This is due to
the way it currently tries to flush all chains in a single VFS_SYNC.
Supporting multiple VFS_SYNC calls by checking available memory along
the way gives chance to free unused vnodes/inodes and chains. This
may be implemented in future. This limitation is specific to HAMMER2,
as all other makefs(8) filesystems are not CoW, meaning they allow
in-place write based objects creation from a top directory to bottom
whereas HAMMER2 flushes chains in bottom-up direction.

show more ...

sys/vfs/hammer2: Add initial multi-volumes support for HAMMER2

This commit adds initial multi-volumes support for HAMMER2. Maximum
supported volumes is 64. The feature and implementation is similar

sys/vfs/hammer2: Add initial multi-volumes support for HAMMER2

This commit adds initial multi-volumes support for HAMMER2. Maximum
supported volumes is 64. The feature and implementation is similar to
multi-volumes support in HAMMER1.

1. ondisk changes
=================
This commit bumps volume header version from 1 to 2, and adds four new
volume header fields using reserved fields in version 1. Other ondisk
structures are unchanged.
* "volu_id" - volume id from 0 to 63, where 0 represents root volume.
* "nvolumes" - number of volumes. All volumes have same the same value.
* "total_size" - sum of "volu_size" in volumes. All volumes have the
same value.
* "volu_loff[HAMMER2_MAX_VOLUMES]" - A 512 bytes table which contains
start offset of max 64 volumes within "total_size". All volumes have
the same value.

Version 1 volume header has 0 for above fields, so HAMMER2 internally
treats "nvolumes" as 1, and "total_size" as "volu_size" to be able to
handle version 1 and 2 transparently.

All volumes have 4 headers, but only root volume ones are relevant.
Non-root volume headers have their own unique "volu_id" and "volu_size",
but other fields are unimportant and never used. Non-root volume headers
have sroot blockset[i] whose type is HAMMER2_BREF_TYPE_INVALID. Non-root
volume headers don't have boot/aux area, so freemap area start from
offset 0. Non-root volume headers are readonly and never updated after
creation. This means non-root volumes are just extra storage to extend
fs size and internally make up a single virtual volume whose size is
"total_size".

It currently doesn't automatically upgrade an existing version 1 fs to
version 2. Only newly created fs becomes version 2 for now.

2. volumes layout
=================
Basically similar to HAMMER1. A first block device argument provided for
newfs_hammer2(8) becomes the root volume, and if specified remaining
devices extend "total_size" as non-root volumes. All volumes except for
the last one have 1GiB (freemap level1) aligned "volu_size".

This means each volume's start offset within "total_size" is also 1GiB
(freemap level1) aligned. The start offsets of volumes are stored in
volu_loff[HAMMER2_MAX_VOLUMES]. Each volu_loff[n] (0 <= n < nvolumes)
represents start offset of volume n within "total_size". Unused volumes
have -1 for volu_loff[n].
e.g. If a fs consists of 1 volume, volu_loff[0] has 0 and rests have -1.
e.g. If a fs consists of 3 volumes, x GiB root volume, y GiB volume,
and z GiB volume, volu_loff[0] has 0, volu_loff[1] has x, volu_loff[2]
has x+y, and rests have -1.

Low level I/O function in HAMMER2 uses this linear offsets table to
determine a device vnode to use and relative offset within the device
vnode, for a given blockref's "data_off". This is different from HAMMER1
where logical offset had embedded volume id bits (i.e. there were holes
in logical address space). HAMMER2 needs this table to support multi-
volumes without changing current logical offset mechanism.

Unless all volumes are specified and mountable, mount_hammer2(8) fails
like it failed in HAMMER1. This also applies to other userspace commands
which require volumes specification, except for fstyp(8).

3. userspace commands
=====================
Basically same as or similar to HAMMER1.
* newfs_hammer2(8) takes a list of block device paths as argv[].
* mount_hammer2(8) takes block device paths or names in "a:b:c:..."
format.
* hammer2(8) takes block device paths or names in "a:b:c:..." format for
directives which require volumes specification. This commit also adds
"volume-list" directive and an ioctl command HAMMER2IOC_VOLUME_LIST,
which are similar to the one in HAMMER1.
* fsck_hammer2(8) takes device paths or names in "a:b:c:..." format.
* fstyp(8) takes device paths in "path1:path2:path3:..." format.

4. limitations
==============
* hammer2(8) "info" directive ignores multi-volumes block devices.
* hammer2(8) "growfs" directive doesn't support multi-volumes fs.
* fstyp(8) is unable to find PFS label via -l option if the PFS inode or
its parent indirect blocks are located beyond root volume.
* hammer2(8) doesn't support "volume-add" and "volume-del" directives
which existed in HAMMER1, and there is currently no plan to support.

show more ...

sbin/newfs_hammer2: Minor cleanup

to make multi-volumes diff (which uses *vol for other purpose) clearer.
In HAMMER2 the volume header variable is basically always *voldata.

sbin/newfs_hammer2: Remove redundant \n in err/errx

newfs_hammer2 - Cleanup slight code mess

* Cleanup the use of NLabels in the strlen() tests.

* Require -L labels to not be 0-length, and change the string length
limit from 64 to 63. 64 should w

newfs_hammer2 - Cleanup slight code mess

* Cleanup the use of NLabels in the strlen() tests.

* Require -L labels to not be 0-length, and change the string length
limit from 64 to 63. 64 should work just fine since we store the
length field, but its easier to debug with termination.

Reported-by: kHacks

show more ...

sbin/newfs_hammer2: Fix wrong comments on blockrefs in sroot inode

newfs_hammer2 initially only used 1 blockref within sroot inode when
these were written in b33a7e92438b9c9fc915e7c432666385051369db

sbin/newfs_hammer2: Fix wrong comments on blockrefs in sroot inode

newfs_hammer2 initially only used 1 blockref within sroot inode when
these were written in b33a7e92438b9c9fc915e7c432666385051369db in 2012.

show more ...

sbin/newfs_hammer2: Remove/fix obsolete stuff from HAMMER1

Remove unused -f option. HAMMER1 had this to force creation when
devices were smaller than expected, but HAMMER2 has no such limitation.
Th

sbin/newfs_hammer2: Remove/fix obsolete stuff from HAMMER1

Remove unused -f option. HAMMER1 had this to force creation when
devices were smaller than expected, but HAMMER2 has no such limitation.
The man page also had incorrect description.

Fix incorrect code/description for -r option originally from HAMMER1.

show more ...

sbin/newfs_hammer2: Fix comments/assert on 4MB segments

The 4MB segments exist at every 1GB (every level1 freemap),
not every 2GB "zone".

sbin/newfs_hammer2: Assert inodes/volhdrs don't cross 64KB boundary

Assert they fit within a pbuf in addition to existing asserts.

sbin/newfs_hammer2: Fail if no free space left

stat[v]fs shows negative free space if newfs/mount'd with <0 free_space.

sbin/newfs_hammer2: Use ~HAMMER2_PBUFMASK64 to obtain pbuf offset

HAMMER2 in practice uses ~HAMMER2_PBUFMASK64 for this purpose
rather than HAMMER2_OFF_MASK_HI which is only used here.
Also more rea

sbin/newfs_hammer2: Use ~HAMMER2_PBUFMASK64 to obtain pbuf offset

HAMMER2 in practice uses ~HAMMER2_PBUFMASK64 for this purpose
rather than HAMMER2_OFF_MASK_HI which is only used here.
Also more readable considering use of PBUF macros in HAMMER2.

show more ...

sbin/newfs_hammer2: Remove extra line break after printf outputs

This probably came from newfs_hammer.c, but HAMMER2 doesn't have
a note after this.

sbin/newfs_hammer2: Remove "m:" from getopt(3)

No such option.

sbin/hammer2: Add sbin/hammer2/hammer2_subs.h

Separate a header for subs.c from <hammer2.h>.

This lets other HAMMER2 binaries drop unneeded dependencies
required to use sbin/hammer2/subs.c (various

sbin/hammer2: Add sbin/hammer2/hammer2_subs.h

Separate a header for subs.c from <hammer2.h>.

This lets other HAMMER2 binaries drop unneeded dependencies
required to use sbin/hammer2/subs.c (various unneeded OpenSSL
header includes via <dmsg.h> via <hammer2.h>, global variables
via <hammer2.h>).

This doesn't affect existing files which include <hammer2.h>.

show more ...

sbin/newfs_hammer2: Use sbin/hammer2/subs.c

This is same as how sbin/newfs_hammer makes use of common code
in sbin/hammer.

sbin/newfs_hammer2: Eliminate (an only)tab in result output

Other lines are aligned with space.

sbin/newfs_hammer2: Zero clear boot/aux area

Although currently unused, don't leave unrelated garbage.
This increases runtime by 1 sec or so with default boot/aux size.

Also drop redundant bzero()

sbin/newfs_hammer2: Zero clear boot/aux area

Although currently unused, don't leave unrelated garbage.
This increases runtime by 1 sec or so with default boot/aux size.

Also drop redundant bzero() call against zero'd buffer.

show more ...

sbin/newfs_hammer2: Make ascii-art a bit more descriptive