world: Staticize various variables and also remove a couple of unused ones.

sbin/hammer: Fix compile error on Linux

Failed with gcc 4.4.7 in CentOS 6.x.
--
cmd_recover.c:78: error: redefinition of typedef 'bigblock_t'
cmd_recover.c:55: note: previous declaration of 'bigbloc

sbin/hammer: Fix compile error on Linux

Failed with gcc 4.4.7 in CentOS 6.x.
--
cmd_recover.c:78: error: redefinition of typedef 'bigblock_t'
cmd_recover.c:55: note: previous declaration of 'bigblock_t' was here

show more ...

sbin/hammer: Cleanup header includes

sbin/*hammer: Add missing braces to conform to code style

These were the missing ones due to
1. the code were written by several devs
2. they were missing ever since originally written by dillon@

sbin/*hammer: Use consistent static/inline/returntype format for functions

Change them to HAMMER2 style which has much more stable code style
than HAMMER1. HAMMER1 is basically a mix of every possib

sbin/*hammer: Use consistent static/inline/returntype format for functions

Change them to HAMMER2 style which has much more stable code style
than HAMMER1. HAMMER1 is basically a mix of every possible pattern
(even in the code originally written only by dillon@).

show more ...

sbin/*hammer: Use typedef'd for struct buffer_info*

This is a part of series of commits started from 21e9e7d5ff in 2016.
This is for userspace.

sbin/*hammer: Use typedef'd for struct volume_info*

This is a part of series of commits started from 21e9e7d5ff in 2016.
This is for userspace.

sbin/hammer: Add /* not reached */

that were originally missing.
Recommended by dillon@.
No diff in sbin/hammer binary when assert(3) is disabled.

sbin/hammer: Partly bring back removed braces from 52e2f1b5

based on comments from dillon@.

(dillon) three types, yah. (1) multi-line condition in if(), (2) if TRUE or FALSE body spans more than o

sbin/hammer: Partly bring back removed braces from 52e2f1b5

based on comments from dillon@.

(dillon) three types, yah. (1) multi-line condition in if(), (2) if TRUE or FALSE body spans more than one line, put braces around both
(dillon) (including if a comment is part of the body)

show more ...

sbin/hammer: Cleanup blocks with a single statement

This commit basically does the same as what Linux kernel's coding
style mentions for braces, which is basically the same with BSDs.
https://github

sbin/hammer: Cleanup blocks with a single statement

This commit basically does the same as what Linux kernel's coding
style mentions for braces, which is basically the same with BSDs.
https://github.com/torvalds/linux/blob/master/Documentation/process/coding-style.rst#3-placing-braces-and-spaces

No diff in sbin/hammer/hammer binary when assert(3) is disabled.

show more ...

hammer - HAMMER Version 7

* Add support for version 7 which changes the CRC mechanic from the
old slow CRC code to the faster ISCSI CRC code. We don't use the CRC
instruction yet but ths base I

hammer - HAMMER Version 7

* Add support for version 7 which changes the CRC mechanic from the
old slow CRC code to the faster ISCSI CRC code. We don't use the CRC
instruction yet but ths base ISCSI CRC from FreeBSD is 6x faster than
the CRC code we were using before.

* Change newfs_hammer default to version 7 (for master).

show more ...

sbin/hammer: Have consistent naming for volume variables

Use *volume.
Userspace is mix of *vol and *volume for no reason.
sys/vfs/hammer is always *volume.

sbin/hammer: Properly use calloc(3)

It's supposed to be number and then size.

sbin/hammer: Change fprintf/exit to err variants [2/2]

Change
fprintf(stderr, ...); exit(1);
and
perror(...); exit(1);
to
err(1, ...) or errx(1, ...);
where possible for consistency.

This commit is

sbin/hammer: Change fprintf/exit to err variants [2/2]

Change
fprintf(stderr, ...); exit(1);
and
perror(...); exit(1);
to
err(1, ...) or errx(1, ...);
where possible for consistency.

This commit is just conversion to err(3) variants.
Messages themselves are not changed, except for removing
strerror(errno) for err(3), and removing trailing \n,
though err variants add program name in the messages.

err(3) and variants are non standard BSD functions, but HAMMER
userspace has been using BSD stuff aside from the existing
err/errx. Also note that err(3) is available in Linux as well.

show more ...

sbin/hammer: Fix known bug in full scan recovery mentioned in f2dd4b0c

As mentioned in 3d900665, introducing scan range limit by default,
and preserving the original default behavior as full scan mo

sbin/hammer: Fix known bug in full scan recovery mentioned in f2dd4b0c

As mentioned in 3d900665, introducing scan range limit by default,
and preserving the original default behavior as full scan mode
worked around a bug mentioned in f2dd4b0c, but possible assertion
error (by having access to not existing volumes) has still been
there as far as full scan mode is concerned.

This commit is to fix that.

show more ...

sbin/hammer: Fix rename printfs to differentiate recover paths

This helps understand recovery path from stdout.

It doesn't really matter if the change makes any sense to real users,
because most pr

sbin/hammer: Fix rename printfs to differentiate recover paths

This helps understand recovery path from stdout.

It doesn't really matter if the change makes any sense to real users,
because most printfs by this command aren't understandable anyway
unless one looks at the code.

show more ...

sbin/hammer: Use big-block append offset to limit recovery scan range

This commit is to fix a remaining issue mentioned in e3cefcca,
which recovers irrelevant files from old filesystem even with the

sbin/hammer: Use big-block append offset to limit recovery scan range

This commit is to fix a remaining issue mentioned in e3cefcca,
which recovers irrelevant files from old filesystem even with the
scan range limit introduced by e3cefcca and quick scan mode
introduced by e819b271.

As shown in an example below, whenever a filesystem is recreated
and the current one uses less space than the old filesystem, the
command is likely to recover files from old filesystem (even with
e3cefcca and e819b271), because B-Tree big-blocks could have nodes
from old filesystem after their append offset, especially if the
block is the last one in B-Tree zone.

In order to avoid recovery of irrelevant files, the command needs
to check if scanning offset is beyond append offset of the B-Tree
big-block that contains this offset, and ignore all nodes beyond
the append offset. [*] shows this situation. Note that the append
offset is checked only if layer1/2 entries that point to this
B-Tree big-block have good CRC result.

This applies to both default and quick scan mode, but not to full
scan mode. Full scan scans everything no matter what.

--------------------------------------------------------> offset
|--------------------------------------------------| volume size
|<----------------------------------------->| previously used
|<---->| previously unused
|<----------------------------------->| currently used
|<---------->| currently unused

... -------------------------->| full scan
... ---------------->| default scan
... --->||<------->||<------->||<--->| default scan [*]
... |<-->| ... |<-->| ... |<-->| quick scan
... |<->| ... |<->| ... |<->| quick scan [*]

===== comparison of recovered files
1. Zero clear the first 1GB of /dev/da1.
# dd if=/dev/zero of=/dev/da1 bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 2.714761 secs (395519867 bytes/sec)

2. Create a filesystem and clone 968MB dragonfly source.
# newfs_hammer -L TEST /dev/da1 > /dev/null
# mount_hammer /dev/da1 /HAMMER
# cd /HAMMER
# git clone /usr/local/src/dragonfly > /dev/null 2>&1
# du -sh .
968M .
# cd
# umount /HAMMER

3. Create a filesystem again with 1 regular file.
# newfs_hammer -L TEST /dev/da1 > /dev/null
# mount_hammer /dev/da1 /HAMMER
# cd /HAMMER
# ls -l
total 0
# echo test > test
# cat ./test
test
# cd
# umount /HAMMER

4-1. Recover a filesystem assuming it only has 1 regular file.
# rm -rf /tmp/a
# hammer -f /dev/da1 recover /tmp/a recover > /dev/null
# cat /tmp/a/PFS00000/test
test
# tree /tmp/a | wc -l
19659
# du -a /tmp/a | grep obj_0x | wc -l
19661

4-2. Do the same as 4-1 using this commit.
# rm -rf /tmp/b
# hammer -f /dev/da1 recover /tmp/b recover > /dev/null
# cat /tmp/b/PFS00000/test
test
# tree /tmp/b
/tmp/b
`-- PFS00000
`-- test

1 directory, 1 file
#

show more ...

sbin/hammer: Cleanup hammer recover

Separate debug code into a different inlined function.

sbin/hammer: Add full mode for hammer recover to revive full scan

This commit revives the original full scan recovery by adding full
option, after the previous commit introduced offset limit.
Appare

sbin/hammer: Add full mode for hammer recover to revive full scan

This commit revives the original full scan recovery by adding full
option, after the previous commit introduced offset limit.
Apparently, both full option and quick option can't be specified.

To summarize 3 modes,

1. default - Full scan, but only upto the last big-block being used.
2. full - Full scan, which scans the entire fs image with no limit.
3. quick - B-Tree only scan, plus associated records in other zones.

1. was introduced (by the previous commit) to fix a bug, as well as
to avoid irrelevant files.
2. was introduced (by this commit) to revive the original full scan
recovery behavior, which is by far the slowest, but most reliable
in terms of recovery except for the above bug.
3. was introduced (by e819b271) to speed up the recovery process,
provided B-Tree zone is not corrupted. This is the fastest.

show more ...

sbin/hammer: Use last active big-block to limit recovery scan range

This commit is to fix a bug mentioned in f2dd4b0c. This commit
uses offset of the last active big-block (big-block with maximum
zo

sbin/hammer: Use last active big-block to limit recovery scan range

This commit is to fix a bug mentioned in f2dd4b0c. This commit
uses offset of the last active big-block (big-block with maximum
zone-2 offset whose layer2->zone is neither 0,4,15), as an upper
limit of scan range, so it doesn't scan beyond actual consumption.

Note that this upper limit is used only if all layer1/2 entries
have correct CRC values. Otherwise the entire image is scanned
as usual (unless quick option is used).

Note that this upper limit doesn't necessarily equal a big-block
before the first unused big-block offset (i.e. layer2->zone == 0),
because reblock could locate unused big-block between used ones.

Note that using the upper limit also tries to avoid recovery of
irrelevant files from old filesystem that could exist beyond the
upper limit (if not perfect). It also speeds up recovery process.

show more ...

sbin/hammer: Minor fix for hammer recover quick mode

* Remove assert(b); since it's totally possible that the whole
B-Tree zone was corrupted and nothing was found.

* Print B-Tree zone info only

sbin/hammer: Minor fix for hammer recover quick mode

* Remove assert(b); since it's totally possible that the whole
B-Tree zone was corrupted and nothing was found.

* Print B-Tree zone info only when using quick option.

* Rename a local variable limit to zone_limit for the next commit.

show more ...

sbin/hammer: Add quick mode for hammer recover

Since hammer recover command tries to recover filesystem data
based on assumption on ondisk data bytes that look like B-Tree
nodes/elms, the command ca

sbin/hammer: Add quick mode for hammer recover

Since hammer recover command tries to recover filesystem data
based on assumption on ondisk data bytes that look like B-Tree
nodes/elms, the command can tell the recovery process is done
once scanning offset gets to the point where there is no more
big-blocks for B-Tree zone, without scanning through the whole
address space of all volumes (provided B-Tree zone is alive).

By specifying quick option after the target directory option,
this command makes use of B-Tree big-block info prefetched before
recovery process, and stops recovery once all B-Tree big-blocks
are scanned. As shown in below example, this makes recovery
much faster by cutting unnecessary I/Os.

The drawback is that quick mode is based on assumption that
B-Tree zone isn't corrupted. If B-Tree zone is somehow corrupted,
prefetched info is incomplete or totally wrong, so one needs
to linearly scan the whole address space of all volumes to
check every possible B-Tree nodes/elms without using quick mode
which is what's been done by default.

-- example of default and quick mode
# newfs_hammer -L TEST /dev/da1 > /dev/null
# mount_hammer /dev/da1 /HAMMER
# cd /HAMMER
# git clone /usr/local/src/dragonfly > /dev/null 2>&1
# cd
# umount /HAMMER

# time hammer -f /dev/da1 recover /tmp/a > /dev/null
hammer -f /dev/da1 recover /tmp/a > /dev/null 309.51s user 122.96s system 21% cpu 33:50.17 total
# cd /tmp/a/PFS00000/dragonfly/sys/vfs/hammer
# make > /dev/null 2>&1; echo $?
0
# file hammer.ko
hammer.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

# time hammer -f /dev/da1 recover /tmp/b quick > /dev/null
hammer -f /dev/da1 recover /tmp/b quick > /dev/null 0.41s user 3.41s system 14% cpu 26.652 total
# cd /tmp/b/PFS00000/dragonfly/sys/vfs/hammer
# make > /dev/null 2>&1; echo $?
0
# file hammer.ko
hammer.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

show more ...

sbin/hammer: Use HAMMER_OBJID_ROOT

sbin/hammer: Fix direntry message in hammer recover

name could have already been free'd, so move it to the beginning.
Also enable it only on -v, just like inode/data rectype cases.

sbin/hammer: Fix inode/data messages in hammer recover

Based on other printf messages where "file" indicates regfile,
the first one should be "inode" rather than "file" because it
could be both dire

sbin/hammer: Fix inode/data messages in hammer recover

Based on other printf messages where "file" indicates regfile,
the first one should be "inode" rather than "file" because it
could be both directory and regfile.

The second one could be "file" because it's for file data, but
just sync with the first format.

show more ...