sys/vfs/hammer: Use hammer_crc_t

Use hammer_crc_t rather than uint32_t which is the same thing.
This commit doesn't (and shouldn't) replace CRC variables that are
not HAMMER's ondisk CRC field or io

sys/vfs/hammer: Use hammer_crc_t

Use hammer_crc_t rather than uint32_t which is the same thing.
This commit doesn't (and shouldn't) replace CRC variables that are
not HAMMER's ondisk CRC field or ioctl, but for general purpose.

show more ...

sys/vfs/hammer: Add inline CRC functions for mrec head

sys/vfs/hammer: Use typedef'd for struct hammer_base_elm*

The whole hammer code is mix of using struct and typedef'd.
Use typedef'd because majority of the code use typedef'd.

sys/vfs/hammer: Add hammer_btree_extract_data() [2/2]

This commit replaces
hammer_btree_extract(cursor, HAMMER_CURSOR_GET_DATA)
with
hammer_btree_extract_data(&cursor)
which are the remaining ones f

sys/vfs/hammer: Add hammer_btree_extract_data() [2/2]

This commit replaces
hammer_btree_extract(cursor, HAMMER_CURSOR_GET_DATA)
with
hammer_btree_extract_data(&cursor)
which are the remaining ones from the previous commit.

From the way hammer_btree_extract_data() is implemented,
hammer_btree_extract(cursor, HAMMER_CURSOR_GET_DATA)
is the same as
hammer_btree_extract(cursor, HAMMER_CURSOR_GET_LEAF | HAMMER_CURSOR_GET_DATA)
and it probably should have been (LEAF | DATA) instead of just DATA,
according to the way hammer_get_inode(), hammer_update_inode() and
hammer_update_itimes() set cursor flag. It's either LEAF or
(LEAF | DATA), but not just DATA that makes sense on calling
hammer_btree_extract().

show more ...

sys/vfs/hammer: Add lo_to_pfs()/pfs_to_lo() macros

These macros hide 16 bits shift and cast to int/uint32_t
that are necessary on localization to/from PFS id conversion.

Some code are better withou

sys/vfs/hammer: Add lo_to_pfs()/pfs_to_lo() macros

These macros hide 16 bits shift and cast to int/uint32_t
that are necessary on localization to/from PFS id conversion.

Some code are better without details being hidden by macros,
but this seems to be more readable with x_to_y macros that
hide bunch of >>16, <<16, and casts.

(No diff in binary on x86_64/gcc5)

show more ...

sys/vfs/hammer: Use bitwise OR to generate ondisk localization

Use |= to generate localization field for B-Tree elements and
cursor keys instead of +=, since lower 16 bits are bitfields
(or safer to

sys/vfs/hammer: Use bitwise OR to generate ondisk localization

Use |= to generate localization field for B-Tree elements and
cursor keys instead of +=, since lower 16 bits are bitfields
(or safer to treat INODE=0x1 and MISC=0x2 as bitfields).

The typical code to generate ondisk localization value is to
do either of the followings.

ondisk_lo = local_variable + {INODE or MISC};
ondisk_lo = ip->obj_localization + {INODE or MISC};
ondisk_lo = HAMMER_XXX_LOCALIZATION + {INODE or MISC};
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
(A)32 bits localization (B)lower 16 bits
with usually 0 for
lower 16 bits for type

Adding (A) and (B) to synthesize PFS id and localization type
could lead to a potential bug if (A) already has type bits set
to either INODE or MISC. For example if (A) had INODE for type
bits and the code is to += INODE for (B), then type bits turn
into MISC (1+1=2) which is not the intention of the code.

This could potentially occur with the first example of above
three where (A) is a local variable or a function argument.
It is not too obvious from the code whether that local variable
has 0 for the lower 16 bits (which basically should be).
If the code just always uses |= no such thing will happen.

show more ...

sys/vfs/hammer: Remove #if0'd hammer_mirror_write()

and its sub function hammer_mirror_localize_data().

These two functions are replaced by hammer_create_at_cursor()
and hammer_cursor_localize_data

sys/vfs/hammer: Remove #if0'd hammer_mirror_write()

and its sub function hammer_mirror_localize_data().

These two functions are replaced by hammer_create_at_cursor()
and hammer_cursor_localize_data() which basically do the same
thing in 83f2a3aa, as the comment says.

Fix some comments that haven't been updated after replacing.

show more ...

hammer: Change u_int{8,16,32,64}_t to uint{8,16,32,64}_t

in the following directories and/or files.

* sys/vfs/hammer
* sbin/hammer
* sbin/newfs_hammer
* sbin/mount_hammer
* usr.bin/fstat/hammer.c
*

hammer: Change u_int{8,16,32,64}_t to uint{8,16,32,64}_t

in the following directories and/or files.

* sys/vfs/hammer
* sbin/hammer
* sbin/newfs_hammer
* sbin/mount_hammer
* usr.bin/fstat/hammer.c
* usr.bin/undo

show more ...

sys/vfs/hammer: Add error messages for crc test failure

Add crc error messages based on the existing error message
in hammer_btree_extract(). The one in hammer_mirror.c is
a part of #if0'd function.

sys/vfs/hammer: Add error messages for crc test failure

Add crc error messages based on the existing error message
in hammer_btree_extract(). The one in hammer_mirror.c is
a part of #if0'd function.

Reported-by: profmakx
Suggested-by: dillon

23:27 (tkusumi) too many error = EIO; but possibly bad crc ?
23:28 (tkusumi) if bad crc is the case turning on hammer_debug_critical leads to Debugger("CRC FAILED: B-TREE NODE");
23:40 (dillon) it's likely a CRC failure, though I think it should print those out on the console too
23:54 (tkusumi) some bad crc are missing "CRC DATA @ %016llx/%d FAILED\n"
23:59 (dillon) those could be added for clarity. Almost universally EIO from hammer means a bad CRC and not an actual disk read error.
23:59 (dillon) USB-connected hard drives seem to get these. I've never gotten a bad CRC from a SATA-connected HDD via the AHCI driver.

show more ...

sys/vfs/hammer: Adjust raw kprintfs using hkprintf variants

This is part3 of hkprintf related after the following two.
sys/vfs/hammer: Change hkprintf() to macro and add variants [2/2]
sys/vfs/hamme

sys/vfs/hammer: Adjust raw kprintfs using hkprintf variants

This is part3 of hkprintf related after the following two.
sys/vfs/hammer: Change hkprintf() to macro and add variants [2/2]
sys/vfs/hammer: Change hkprintf() to macro and add variants [1/2]

Above two commits have replaced the existing kprintf calls
using "HAMMER:" or "HAMMER(label)" or function name prefix
with hkprintf and newly added variants, which basically didn't
change actual output other than fixing wrong function names
to the right ones, etc.

This commit continues replacing remaining kprintfs to make
output more understandable than raw kprintf calls with no clue
that they're hammer related.

For example, an error message like
"BIGBLOCK UNDERFLOW\n"
or a debug message like
"rt %3u, xt %3u, tt %3u\n"
become more understanbale with "HAMMER:" prefix or the name
of the function.

This commit is based on the followings.
1. Use hdkprintf which is hkprintf variant with __func__ prefix
if that kprintf call is used when vfs.hammer.debug_xxx is
enabled. This implies the messages are only for debugging
and those are usually better and more understandable with
a function name prefix as mentioned above. Also this is
what's been done mostly in the existing hammer code.
2. Use hkprintf which has "HAMMER:" prefix if that kprintf
call is a regular hammer message that appears in regular
filesystem operations such as
"Formatting of valid HAMMER volume %s denied. Erase with dd!\n".
3. Use h[vm]kprintf which are hkprintf variants with hammer
label prefix "HAMMER(label)" if that kprintf can safely
access the label via vol or hmp pointer. Some kprintfs in
hammer does this rather than just "HAMMER:" and this seems
to be better, however this commit doesn't go far as to
aggressively replace the existing ones with this because
a caller safely dereferencing hmp or vol is a different
topic from merely replacing.

show more ...

hammer: Remove trailing whitespaces

- (Non-functional commits could make it difficult to git-blame
the history if there are too many of those)

hammer: Remove trailing tabs

- (Non-functional commits could make it difficult to git-blame
the history if there are too many of those)

sys/vfs/hammer: Change "bigblock" to "big-block"

- There are(or were) several terms for 8MB chunk, for example
"big-block", "bigblock", "big block", "large-block", etc but
"big-block" seems to b

sys/vfs/hammer: Change "bigblock" to "big-block"

- There are(or were) several terms for 8MB chunk, for example
"big-block", "bigblock", "big block", "large-block", etc but
"big-block" seems to be the canonical term.

- Changes are mostly comments and some in printf and hammer(8).
Variable names (e.g. xxx_bigblock_xxx) remain unchanged.

- The official design document as well as much of the existing
code (excluding variable and macro names) use "big-block".
https://www.dragonflybsd.org/hammer/hammer.pdf

- Also see e04ee2de and the previous commit.

show more ...

sys/vfs/hammer: Add inline functions hammer_modify_buffer|volume_noundo()

- Add noundo wrappers hammer_modify_buffer|volume_noundo() similar to
the existing inline function hammer_modify_node_noun

sys/vfs/hammer: Add inline functions hammer_modify_buffer|volume_noundo()

- Add noundo wrappers hammer_modify_buffer|volume_noundo() similar to
the existing inline function hammer_modify_node_noundo() for better
readability.

- A pair of args (NULL, 0) indicating that it's not generating undo is
a bit unclear (and there are even comments for them).

- (The compiler doesn't actually inline hammer_modify_node_noundo()
in my environment, but these one-line wrappers are inlined)

show more ...

kernel/hammer: Remove unused variables and add __debugvar.

Initial import of binutils 2.22 on the new vendor branch

Future versions of binutils will also reside on this branch rather
than continuing to create new binutils branches for each new version.

HAMMER - Fix cursor tracking bugs and a few other issues

* When recursively removing empty internal nodes from the B-Tree only
call hammer_cursor_deleted_element() if the related internal
elemen

HAMMER - Fix cursor tracking bugs and a few other issues

* When recursively removing empty internal nodes from the B-Tree only
call hammer_cursor_deleted_element() if the related internal
element is actually removed. The element might not be removed due
to the deadlock fail path.

* If hammer_cursor_up_locked() fails fully restore the cursor before
returning. The index field was not being restored.

* Acquire the sync lock when recovering a cursor lost due to a deadlock
in the mirroring code.

* Document and fix an issue in the rebalancing code which could cause a
cursor to fall off the end of the B-Tree.

Reported-by: YONETANI Tomokazu <qhwt+dfly@les.ath.cx>

show more ...

HAMMER - Add version 3 meta-data features

* These features are available for filesystem version 3. Version 2 may be upgraded
to version 3 in-place. These features are not usable until you upgrad

HAMMER - Add version 3 meta-data features

* These features are available for filesystem version 3. Version 2 may be upgraded
to version 3 in-place. These features are not usable until you upgrade.

* Definitively store snapshots in filesystem meta-data. Softlinks still
work. The new snapshot directives (snap, snaplo, snapq, etc) also allow
you to specify up to a 64-character note for each snapshot you create.
The snapls directive may be used to list all snapshots stored in meta-data.

'hammer cleanup' will move all softlink-based snapshots residing in the
<fs>/snapshots directory to meta-data when it next snapshots the filesystem
(within a day of upgrading, usually). The snapshot softlinks are left intact.

Storing snapshot information in meta-data means that accidental wipes of
your <fs>/snapshots directory will NOT cause later hammer cleanup runs to
destroy your snapshots! The meta-data snapshots are also removed if you
do a prune-everything, or through normal pruning expirations, and thus
'hammer snapls' will definitively list your valid snapshots.

This feature also means that you can obtain a definitive list of snapshots
available on mirroring slaves.

* Definitively store the hammer cleanup configuration file in filesystem meta-data.
This meta-data is not mirrored. 'hammer cleanup' will move <fs>/snapshots/config
to the new meta-data config and deletes <fs>/snapshots/config after you've upgraded
the filesystem. You can edit the configuration with the 'viconfig' directive.

* The HAMMER utility has new directives: snap, snaplo, snapq, snaprm, snapls,
config, and viconfig.

* WARNING! Filesystems mounted 'nohistory' and files chflagged similarly do not
have snapshots, but the hammer utility still allows the directives to be run.
This is a bug that needs to be fixed.

show more ...

HAMMER - Fix root inode creation for slave.

* The root inode for a PFS is only supposed to be created for PFS masters.
PFS slaves get their root inode from the mirroring operation. No longer
cr

HAMMER - Fix root inode creation for slave.

* The root inode for a PFS is only supposed to be created for PFS masters.
PFS slaves get their root inode from the mirroring operation. No longer
create the root inode for a PFS slave.

The root inode was being incorrectly created for slaves as well as
masters, causing the root inode from the mirroring operation to be
ignored.

This caused the root directory for a slave to show up as empty when
doing a mirror-copy from a HAMMER version 1 filesystem to a HAMMER
version 2 filesystem.

Reported-by: "Simon 'corecode' Schubert" <corecode@fs.ei.tum.de>

show more ...

HAMMER - Fix lost inode issue (primarily with nohistory mounts)

* When a HAMMER cursor is unlocked it becomes tracked and unrelated
B-Tree operations will cause the tracked cursor's nodes and indi

HAMMER - Fix lost inode issue (primarily with nohistory mounts)

* When a HAMMER cursor is unlocked it becomes tracked and unrelated
B-Tree operations will cause the tracked cursor's nodes and indices
to be updated. The cursor structure also has a leaf element pointer
which was not being properly updated. This could lead to panics and
lost inodes.

Properly adjust the leaf element pointer in tracked cursors.

* The bug primarily occurs with nohistory mounts or nohistory sub-trees
due to the larger number of physical deletions made to the B-Tree, but
could also occur (rarely) with normal mounts.

* Add additional assertions to catch any further occurrences (though I
think all the cases have been covered now).

* Add a new sysctl vfs.hammer.error_panic which can be set to e.g. 9 to
cause critical errors to panic immediately instead of returning
through the call stack, making debugging possible.

Reported-by: Numerous people

show more ...

HAMMER mirroring - Multiple adjustments to support better incremental ops

* Read side - Ignore records created after the end point, reducing
network bandwidth for incremental operations.

* Read s

HAMMER mirroring - Multiple adjustments to support better incremental ops

* Read side - Ignore records created after the end point, reducing
network bandwidth for incremental operations.

* Read side - Generate PASS for records created prior to tid_beg, instead
of pushing the whole record, reducting network bandwidth for incremental
operations.

* Write side - Ensure that deletions are end-exclusive.

* Write side - Do not delete records created >= end point, supporting
incremental operations.

show more ...

HAMMER VFS - Add hinting capability to block allocator, hint B-Tree

* A hammer_off_t can now be supplied to the blockmap allocator as a hint.

* Use the hinting mechanism to better-localize B-Tree n

HAMMER VFS - Add hinting capability to block allocator, hint B-Tree

* A hammer_off_t can now be supplied to the blockmap allocator as a hint.

* Use the hinting mechanism to better-localize B-Tree node allocations
and meta-data updates.

show more ...

HAMMER VFS - Better CRC handling, bad-file handling.

Data CRC errors should now generate EIO instead of panic()ing the system.
B-Tree CRC errors might still panic() and freemap CRC errors WILL still

HAMMER VFS - Better CRC handling, bad-file handling.

Data CRC errors should now generate EIO instead of panic()ing the system.
B-Tree CRC errors might still panic() and freemap CRC errors WILL still
panic().

Continuing from DDB on a B-Tree node CRC error when debugging is enabled
now no longer marks the B-Tree node as good.

The mirror-read command will now transfer data records with bad CRCs
instead of aborting the transfer, identifying them with a new type field.
The mirror-write ioctl currently ignores such records.

If a directory entry is encountered and the related inode cannot be
looked up, generate a dummy in-memory inode of type FIFO to placemark
the bad directory entry, allowing it to be removed. Currently it is
possible for a directory entry to be synced to the media in a different
transaction then the related inode (a bug which needs to be fixed).
If a crash occurs at the wrong time the recovery code can leave the media
in a state where the directory entry exists but the inode does not. This
change allows the bad directory entry to be removed.

Reported-by: Antonio Huete Jimenez

show more ...

HAMMER VFS - Fix edge case in mirroring code.

There is an edge case where a record is physically destroyed on the master
and a mirroring or mirror-streaming operation attempts to delete the record
o

HAMMER VFS - Fix edge case in mirroring code.

There is an edge case where a record is physically destroyed on the master
and a mirroring or mirror-streaming operation attempts to delete the record
on the slave using a mirror-end TID that in fact matches the create_tid of
the record. This results in a protective panic.

Fix the problem simply by not deleting such edge-case records. They will
be deleted in the next pass, since in order to have been destroyed on the
master the master must have done the operation with a larger TID then
the create_tid of the record that was destroyed.

Reported-by: Brad Frank <brad.frank@gmail.com>

show more ...

HAMMER VFS - Remove the unused also_ip argument from the cursor API