hammer - Disallow modifying ioctls when filesystem is read-only

* Disallow modifying ioctls if the filesystem has been mounted read-only
or gone into read-only mode due to an I/O error.

* This is

hammer - Disallow modifying ioctls when filesystem is read-only

* Disallow modifying ioctls if the filesystem has been mounted read-only
or gone into read-only mode due to an I/O error.

* This is only a partial fix. There are still error-pathing problems
in numerous procedures, particularly the node locking code, that might
result in a token life-lock.

Reported-by: Peter Avalos

show more ...

sys/vfs/hammer: Use "%016jx",intmax_t

Some kprintf use long long, others use intmax_t
for HAMMER offsets and B-Tree keys in hexadecimal.
Make them all use intmax_t.

sys/vfs/hammer: Add hammer_btree_extract_data() [2/2]

This commit replaces
hammer_btree_extract(cursor, HAMMER_CURSOR_GET_DATA)
with
hammer_btree_extract_data(&cursor)
which are the remaining ones f

sys/vfs/hammer: Add hammer_btree_extract_data() [2/2]

This commit replaces
hammer_btree_extract(cursor, HAMMER_CURSOR_GET_DATA)
with
hammer_btree_extract_data(&cursor)
which are the remaining ones from the previous commit.

From the way hammer_btree_extract_data() is implemented,
hammer_btree_extract(cursor, HAMMER_CURSOR_GET_DATA)
is the same as
hammer_btree_extract(cursor, HAMMER_CURSOR_GET_LEAF | HAMMER_CURSOR_GET_DATA)
and it probably should have been (LEAF | DATA) instead of just DATA,
according to the way hammer_get_inode(), hammer_update_inode() and
hammer_update_itimes() set cursor flag. It's either LEAF or
(LEAF | DATA), but not just DATA that makes sense on calling
hammer_btree_extract().

show more ...

sys/vfs/hammer: Add hammer_btree_extract_leaf()

hammer_btree_extract() doesn't read data from block devices
unless flag has HAMMER_CURSOR_GET_DATA. It doesn't really matter
if the flag is HAMMER_CUR

sys/vfs/hammer: Add hammer_btree_extract_leaf()

hammer_btree_extract() doesn't read data from block devices
unless flag has HAMMER_CURSOR_GET_DATA. It doesn't really matter
if the flag is HAMMER_CURSOR_GET_LEAF or not in order to just
extract the node element (without reading data) as long as the
flag doesn't have HAMMER_CURSOR_GET_DATA. Calling this function
will cause cursor->leaf to point to the node element in question
regardless of the flag value.

This commit adds hammer_btree_extract_leaf() which is just a
wrapper over hammer_btree_extract(cursor, HAMMER_CURSOR_GET_LEAF).
This hides complexity of using HAMMER_CURSOR_GET_LEAF or 0 or
anything other than HAMMER_CURSOR_GET_DATA which are all the same.

show more ...

sys/vfs/hammer: Use bitwise OR to generate ondisk localization

Use |= to generate localization field for B-Tree elements and
cursor keys instead of +=, since lower 16 bits are bitfields
(or safer to

sys/vfs/hammer: Use bitwise OR to generate ondisk localization

Use |= to generate localization field for B-Tree elements and
cursor keys instead of +=, since lower 16 bits are bitfields
(or safer to treat INODE=0x1 and MISC=0x2 as bitfields).

The typical code to generate ondisk localization value is to
do either of the followings.

ondisk_lo = local_variable + {INODE or MISC};
ondisk_lo = ip->obj_localization + {INODE or MISC};
ondisk_lo = HAMMER_XXX_LOCALIZATION + {INODE or MISC};
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
(A)32 bits localization (B)lower 16 bits
with usually 0 for
lower 16 bits for type

Adding (A) and (B) to synthesize PFS id and localization type
could lead to a potential bug if (A) already has type bits set
to either INODE or MISC. For example if (A) had INODE for type
bits and the code is to += INODE for (B), then type bits turn
into MISC (1+1=2) which is not the intention of the code.

This could potentially occur with the first example of above
three where (A) is a local variable or a function argument.
It is not too obvious from the code whether that local variable
has 0 for the lower 16 bits (which basically should be).
If the code just always uses |= no such thing will happen.

show more ...

hammer: Change u_int{8,16,32,64}_t to uint{8,16,32,64}_t

in the following directories and/or files.

* sys/vfs/hammer
* sbin/hammer
* sbin/newfs_hammer
* sbin/mount_hammer
* usr.bin/fstat/hammer.c
*

hammer: Change u_int{8,16,32,64}_t to uint{8,16,32,64}_t

in the following directories and/or files.

* sys/vfs/hammer
* sbin/hammer
* sbin/newfs_hammer
* sbin/mount_hammer
* usr.bin/fstat/hammer.c
* usr.bin/undo

show more ...

sys/vfs/hammer: Adjust raw kprintfs using hkprintf variants

This is part3 of hkprintf related after the following two.
sys/vfs/hammer: Change hkprintf() to macro and add variants [2/2]
sys/vfs/hamme

sys/vfs/hammer: Adjust raw kprintfs using hkprintf variants

This is part3 of hkprintf related after the following two.
sys/vfs/hammer: Change hkprintf() to macro and add variants [2/2]
sys/vfs/hammer: Change hkprintf() to macro and add variants [1/2]

Above two commits have replaced the existing kprintf calls
using "HAMMER:" or "HAMMER(label)" or function name prefix
with hkprintf and newly added variants, which basically didn't
change actual output other than fixing wrong function names
to the right ones, etc.

This commit continues replacing remaining kprintfs to make
output more understandable than raw kprintf calls with no clue
that they're hammer related.

For example, an error message like
"BIGBLOCK UNDERFLOW\n"
or a debug message like
"rt %3u, xt %3u, tt %3u\n"
become more understanbale with "HAMMER:" prefix or the name
of the function.

This commit is based on the followings.
1. Use hdkprintf which is hkprintf variant with __func__ prefix
if that kprintf call is used when vfs.hammer.debug_xxx is
enabled. This implies the messages are only for debugging
and those are usually better and more understandable with
a function name prefix as mentioned above. Also this is
what's been done mostly in the existing hammer code.
2. Use hkprintf which has "HAMMER:" prefix if that kprintf
call is a regular hammer message that appears in regular
filesystem operations such as
"Formatting of valid HAMMER volume %s denied. Erase with dd!\n".
3. Use h[vm]kprintf which are hkprintf variants with hammer
label prefix "HAMMER(label)" if that kprintf can safely
access the label via vol or hmp pointer. Some kprintfs in
hammer does this rather than just "HAMMER:" and this seems
to be better, however this commit doesn't go far as to
aggressively replace the existing ones with this because
a caller safely dereferencing hmp or vol is a different
topic from merely replacing.

show more ...

hammer: Remove trailing whitespaces

- (Non-functional commits could make it difficult to git-blame
the history if there are too many of those)

hammer: Remove trailing tabs

- (Non-functional commits could make it difficult to git-blame
the history if there are too many of those)

sys/vfs/hammer: Cleanup cursor initialization code on prune

- Just make things a bit more clear (for the upcoming changes).

- The rule is the ioctl caller sets localization type to prune,
and the

sys/vfs/hammer: Cleanup cursor initialization code on prune

- Just make things a bit more clear (for the upcoming changes).

- The rule is the ioctl caller sets localization type to prune,
and the ioctl code adds up ip localization to initialize cursor.

show more ...

sys/vfs/hammer: Make hammer_delete_at_cursor() properly set *stat_bytes

- This commit makes hammer_delete_at_cursor() set *stat_bytes arg.
*stat_bytes has been ignored although it was supposed to

sys/vfs/hammer: Make hammer_delete_at_cursor() properly set *stat_bytes

- This commit makes hammer_delete_at_cursor() set *stat_bytes arg.
*stat_bytes has been ignored although it was supposed to set bytes
physically deleted from the btree.

- hammer prune command expects this function to set above bytes and
print it after pruning, however it's always 0 because the function
ignores it.

- Note that the actual size freed from zone address space may differ
from *stat_bytes (or what the hammer prune command shows), because
hammer_blockmap_free() may round it up with its alignment size.

==========
example1 - with existing hammer
# hammer prune ./test
TID 000000010404e750 - 0000000104071090
TID 0000000104034060 - 000000010404e750
TID 0000000104019cb0 - 0000000104034060
TID 0000000000000001 - 0000000104019cb0
Prune /HAMMER/: 4 snapshots
Prune /HAMMER/: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 0
Prune /HAMMER/: prune_min is 0d/00:00:00
Prune /HAMMER/ succeeded
Pruned 51/769595 records (10 directory entries) and 0 bytes

example2 - with this commit
# hammer prune ./test
TID 000000010404ea10 - 0000000104071390
TID 0000000104034360 - 000000010404ea10
TID 0000000104019fb0 - 0000000104034360
TID 0000000000000001 - 0000000104019fb0
Prune /HAMMER/: 4 snapshots
Prune /HAMMER/: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 0
Prune /HAMMER/: prune_min is 0d/00:00:00
Prune /HAMMER/ succeeded
Pruned 51/769596 records (11 directory entries) and 6710 bytes

show more ...

Initial import of binutils 2.22 on the new vendor branch

Future versions of binutils will also reside on this branch rather
than continuing to create new binutils branches for each new version.

HAMMER - Fix lost inode issue (primarily with nohistory mounts)

* When a HAMMER cursor is unlocked it becomes tracked and unrelated
B-Tree operations will cause the tracked cursor's nodes and indi

HAMMER - Fix lost inode issue (primarily with nohistory mounts)

* When a HAMMER cursor is unlocked it becomes tracked and unrelated
B-Tree operations will cause the tracked cursor's nodes and indices
to be updated. The cursor structure also has a leaf element pointer
which was not being properly updated. This could lead to panics and
lost inodes.

Properly adjust the leaf element pointer in tracked cursors.

* The bug primarily occurs with nohistory mounts or nohistory sub-trees
due to the larger number of physical deletions made to the B-Tree, but
could also occur (rarely) with normal mounts.

* Add additional assertions to catch any further occurrences (though I
think all the cases have been covered now).

* Add a new sysctl vfs.hammer.error_panic which can be set to e.g. 9 to
cause critical errors to panic immediately instead of returning
through the call stack, making debugging possible.

Reported-by: Numerous people

show more ...

AMD64 - Fix many compile-time warnings. int/ptr type mismatches, %llx, etc.

HAMMER VFS - Remove the unused also_ip argument from the cursor API

Change the autoflush code to autoflush when a flush group reaches a
certain point rather then trying to do it from the reclaim code. This
smooths out the flush sequences. The default is 2000, adjus

Change the autoflush code to autoflush when a flush group reaches a
certain point rather then trying to do it from the reclaim code. This
smooths out the flush sequences. The default is 2000, adjustable via
the vfs.hammer.autoflush sysctl.

Move hammer_inode_waitreclaims() calls. In particular, remove the calls
in the VOP_CLOSE() path. The problem with waiting for excessive reclaims to
drop in these paths is that the inode/vnode operation in question is probably
cached. Thus unrelated programs, even those just opening /dev/null (if /dev
is on a HAMMER filesystem), can wind up blocking in hmrrcm for no good reason.

Instead defer the hammer_inode_waitreclaims() call to the end of the
transaction code if a new inode had to be created during the transaction.
Thus we tend to block on operations that did not have previously cached
vnodes to work with instead of operations on cached vnodes.

Reported-by: Hasso Tepper <hasso@estpak.ee>

show more ...

HAMMER 61F/Many: Stabilization w/ simultanious pruning and reblocking

* BUG FIX: When doing direct-read check to see if any device buffers
are aliasing the disk block and flush any we find which a

HAMMER 61F/Many: Stabilization w/ simultanious pruning and reblocking

* BUG FIX: When doing direct-read check to see if any device buffers
are aliasing the disk block and flush any we find which are dirty.
This ensures that reblocked data gets to disk before a direct-read
tries to read them FROM the disk.

* BUG FIX: Fix a bug introduced in a recent commit where the flusher
wlll not always completely flush the UNDO FIFO or completely flush
all meta-data, resulting in a rollback after a normal umount/mount.

* BUG FIX: Direct-writes queue I/O independant of the in-memory record.
When the backend flusher flushes the record, making it available in the
B-Tree, make sure that the indepent I/O has completed. Otherwise
a later reblocking operation might read the media before the direct-write
has actually completed.

* BUG FIX: In-memory records are not subject direct-IO, since their data
is not yet on the media.

* BUG FIX: Do not allow mount to succeed unless all volumes have been found.
(Reported-by: Sascha Wildner <saw@online.de>)

* BUG FIX: The bd_heatup() call in the reblocker was in the wrong place,
potentially causing the cursor to shift unexpectedly.

* Reorient some of the buffer invalidation code by enhancing
the reservation code.

* Add read CRC verification logic for some direct-reads, but comment it out
because the VM system's bogus-page replacement breaks it.

show more ...

HAMMER 61G/Many: Stabilization of new flush_group code

* The mirroring, reblocking, and pruning code are now more robust in
checking that too many meta-data buffers have not built up.

* Fix a bug

HAMMER 61G/Many: Stabilization of new flush_group code

* The mirroring, reblocking, and pruning code are now more robust in
checking that too many meta-data buffers have not built up.

* Fix a bug in the flusher (today's flusher work) where pure meta-data
changes to the media were not being flushed unless there was also a
modified inode or two to flush.

show more ...

HAMMER 61E/Many: Stabilization, Performance

* The UNDO FIFO could get over-full, causing an assertion. This was because
certain load tests could cuase directories to end up with tens of
thousan

HAMMER 61E/Many: Stabilization, Performance

* The UNDO FIFO could get over-full, causing an assertion. This was because
certain load tests could cuase directories to end up with tens of
thousands of directory entries and all of them were being flushed in
a single flush group.

Rewrite the flush group infrastructure to fix the issue. Instead of
having a two-stage flush we now have flush compartmentalization with
the new hammer_flush_group structure. Flushes can thus be broken down
into transactions that don't blow out the UNDO FIFO.

* Improve flush performance by quite a bit. The new infrastructure
generates fewer B-Tree conflicts and appears to flush the work more
smoothly, probably owing to the fact that the upper size of each
flush group is now bounded.

show more ...

HAMMER 61D/Many: Mirroring features

* Split PFS ioctls into their own source file.

* Add additional PFS/mirroring directives: pfs-upgrade, pfs-downgrade, and
finish implementing pfs-destroy. (Y

HAMMER 61D/Many: Mirroring features

* Split PFS ioctls into their own source file.

* Add additional PFS/mirroring directives: pfs-upgrade, pfs-downgrade, and
finish implementing pfs-destroy. (Yes, that means you can change the
master/slave mode for a PFS now).

* Consolidate some of the B-Tree deletion code.

* Fix another sync_lock deadlock.

show more ...

HAMMER 61C/Many: Stabilization

* Clean up flusher sequencing for the mirroring, pruning, and reblocking
code. Check free space and don't let the buffer cache implode.

HAMMER 61B/Many: Stabilization

* Giving the sync_lock's exclusive locks priority over shared locks caused
a 3-way deadlock, revert to using normal shared locks.

* Move the syncer_lock deeper, clo

HAMMER 61B/Many: Stabilization

* Giving the sync_lock's exclusive locks priority over shared locks caused
a 3-way deadlock, revert to using normal shared locks.

* Move the syncer_lock deeper, closer to the code that actually needs to
hold but still paying attention to atomicy requirements. This reduces
lock contention and improves performance with parallel loads.

Reported-by: Michael Neumann (using his file-creator program)

show more ...

HAMMER 60C/many: Mirroring

* Clean up the B-Tree mirror_tid update code, rewiring it a bit
to reduce code pollution.

* Properly detect the mirroring mode (master, slave, or no-mirror mode)
via

HAMMER 60C/many: Mirroring

* Clean up the B-Tree mirror_tid update code, rewiring it a bit
to reduce code pollution.

* Properly detect the mirroring mode (master, slave, or no-mirror mode)
via the pfs configuration.

show more ...

HAMMER 59J/Many: Features

* The pruning code now disposes of any disconnected inodes it encounters.
A disconnected inode can be created when a process holds an open
descriptor on an unlink()'d f

HAMMER 59J/Many: Features

* The pruning code now disposes of any disconnected inodes it encounters.
A disconnected inode can be created when a process holds an open
descriptor on an unlink()'d file, and the machine then crashes.

show more ...

HAMMER 59D/Many: Stabilization pass

* Fix a panic that occurs when the crash recovery undo space is larger
then conveniently fits in the buffer cache.

* Fix a bug in the truncation code (detected

HAMMER 59D/Many: Stabilization pass

* Fix a panic that occurs when the crash recovery undo space is larger
then conveniently fits in the buffer cache.

* Fix a bug in the truncation code (detected by fsx). The bug was related
to moving an inode from one flush group to another (which itself was done
to fix a bug related to blowing out the buffer cache).

show more ...