kernel - __read_mostly pass on vm_fault

* Qualify a number of variables with __read_mostly

kernel: Stop including userland-only includes in kernel code (1/x).

Specifically <stdbool.h>, <arpa/inet.h>, <dlfcn.h>, <limits.h>, and
<unistd.h>.

Only in hammer2's zlib it needed to be exchanged

kernel: Stop including userland-only includes in kernel code (1/x).

Specifically <stdbool.h>, <arpa/inet.h>, <dlfcn.h>, <limits.h>, and
<unistd.h>.

Only in hammer2's zlib it needed to be exchanged with a more suitable
header. The rest could be removed.

show more ...

kernel - Fix serious bug in MAP_STACK, deprecate auto-grow semantics

* When MAP_STACK is used without MAP_TRYFIXED, the address the kernel
determines for the stack was *NOT* being returned to user

kernel - Fix serious bug in MAP_STACK, deprecate auto-grow semantics

* When MAP_STACK is used without MAP_TRYFIXED, the address the kernel
determines for the stack was *NOT* being returned to userland. Instead,
userland always got only the hint address.

* This fixes ruby MAP_STACK use cases and possibly more.

* Deprecate MAP_STACK auto-grow semantics. All user mmap() calls with
MAP_STACK are now converted to normal MAP_ANON mmaps. The kernel will
continue to create an auto-grow stack segment for the primary user stack
in exec(), allowing older pthread libraries to continue working, but this
feature is deprecated and will be removed in a future release.

show more ...

pthreads and kernel - change MAP_STACK operation

* Only allow new mmap()'s to intrude on ungrown MAP_STACK areas when
MAP_TRYFIXED is specified. This was not working as intended before.
Adjust

pthreads and kernel - change MAP_STACK operation

* Only allow new mmap()'s to intrude on ungrown MAP_STACK areas when
MAP_TRYFIXED is specified. This was not working as intended before.
Adjust the manual page to be more clear.

* Make kern.maxssiz (the maximum user stack size) visible via sysctl.

* Add kern.maxthrssiz, indicating the approximate space for placement
of pthread stacks. This defaults to 128GB.

* The old libthread_xu stack code did use TRYFIXED and will work
with the kernel changes, but change how it works to not assume
that the user stack should suddenly be limited to the pthread stack
size (~2MB).

Just use a normal mmap() now without TRYFIXED and a hint based on
kern.maxthrssiz (defaults to 512GB), calculating a starting address
hint that is (_usrstack - maxssiz - maxthrssiz).

* Adjust procfs to report MAP_STACK segments as 'STK'.

show more ...

kernel - Limit console message rate for swap-full warnings

* Limit the message rate on the console for swap-full warnings
to 1hz. This prevents console spam due to a swap-full condition
from ma

kernel - Limit console message rate for swap-full warnings

* Limit the message rate on the console for swap-full warnings
to 1hz. This prevents console spam due to a swap-full condition
from making the system unusable.

show more ...

kernel - Fix more wiring / fictitious bugs w/recent VM work

* Fictitious pages cannot be placed on any VM paging queue,
assert this and fix a few places where it could.

* This will remove most of

kernel - Fix more wiring / fictitious bugs w/recent VM work

* Fictitious pages cannot be placed on any VM paging queue,
assert this and fix a few places where it could.

* This will remove most of the 'Encountered wired page %p on queue'
warnings reported. This kprintf() is still in place for the moment
because generally speaking pages should be unwired before VM objects
are destroyed.

But other than VM object destruction, its actually ok for wired pages
to be on the paging queues now.

show more ...

kernel - Refactor swapcache heuristic

* Refactor the swapcache inactive queue heuristic to remove a
write to a global variable that is in the critical path, and
to improve operation. This shoul

kernel - Refactor swapcache heuristic

* Refactor the swapcache inactive queue heuristic to remove a
write to a global variable that is in the critical path, and
to improve operation. This should reduce cpu cache ping-ponging.

* Change vpgqueues.lcnt from an int to a long, change misc use
cases in the pageout code to a long.

* Use __aligned(64) to 64-byte-align vm_page_queues[]. It was
previously only 32-byte aligned.

show more ...

kernel - Preliminary vm_page hash lookup (2), cleanups, page wiring

* Correct a bug in vm.fault_quick operation. Soft-busied pages cannot
be safely wired or unwired. This fixes a wire/unwire rac

kernel - Preliminary vm_page hash lookup (2), cleanups, page wiring

* Correct a bug in vm.fault_quick operation. Soft-busied pages cannot
be safely wired or unwired. This fixes a wire/unwire race related panic.

* Optimize vm_page_unhold() to normally not have to obtain any spin-locks
at all, since related pages are almost never in the PQ_HOLD VM page
queue. This leaves open a minor race condition where pages with
a hold_count of 0 can accumulate in PQ_HOLD.

* Add vm_page_scan_hold() to the pageout daemon. Unconditionally scan
PQ_HOLD very slowly to remove any pages whos hold_count is 0.

* REFACTOR PAGE WIRING. Wiring vm_page's no longer removes them from
whatever paging queue they are on. Instead, proactively remove
such pages from the queue only when we need to (typically in the
pageout code).

* Remove unused PV_FLAG_VMOBJECT.

* Fix missing atomic-op in pc64/x86_64/efirt.c

* Do not use m->md.pv_list for pagetable pages. It is now only used for
terminal pages.

* Properly initialize pv_flags to 0 when a pv_entry is allocated.

* Add debugging to detect managed pmap_enter()s without an object.

* Conditionalize the setting of PG_MAPPED and PG_WRITEABLE in the
pmap code to avoid unnecessary cpu cache mastership changes.

* Move assertions in vm_pageout.c that could trigger improperly due
to a race.

show more ...

kernel - Optimize vm_page_wakeup(), vm_page_hold(), vm_page_unhold()

* vm_page_wakeup() does not need to acquire the vm_page spin-lock.
The caller holding the page busied is sufficient.

* vm_page

kernel - Optimize vm_page_wakeup(), vm_page_hold(), vm_page_unhold()

* vm_page_wakeup() does not need to acquire the vm_page spin-lock.
The caller holding the page busied is sufficient.

* vm_page_hold() does not need to acquire the vm_page spin-lock as
the caller is expected to hold the page busied, soft-busied,
or stabilized via an interlock (such as the vm_object interlock).

* vm_page_unhold() only needs to acquire the vm_page spin-lock on
the 1->0 transition of m->hold_count.

show more ...

kernel - Preliminary vm_page hash lookup

* Add preliminary vm_page hash lookup code which avoids most
locks, plus support in vm_fault. Default disabled, with debugging
for now.

* This code sti

kernel - Preliminary vm_page hash lookup

* Add preliminary vm_page hash lookup code which avoids most
locks, plus support in vm_fault. Default disabled, with debugging
for now.

* This code still soft-busies the vm_page, which is an improvement over
hard-busying it in that it won't contend, but we will eventually want
to entirely avoid all atomic ops on the vm_page to *really* get the
concurrent fault performance.

show more ...

kernel - Refactor vm_map structure 1/2

* Remove the embedded vm_map_entry 'header' from vm_map.

* Remove the prev and next fields from vm_map_entry.

* Refactor the code to iterate only via the RB

kernel - Refactor vm_map structure 1/2

* Remove the embedded vm_map_entry 'header' from vm_map.

* Remove the prev and next fields from vm_map_entry.

* Refactor the code to iterate only via the RB tree. This is not as
optimal as the prev/next fields were, but we can improve the RB tree
code later to recover the performance.

show more ...

sys/vm: Fix vnode_pager_haspage() description

Not uptodate with the code since
1b9d3514e66df9d4bc5a25c898c7c8cbe414e3e4 in 2010.

kernel - Correct null pointer panic in debug code

* vm_fault pager read errors in kernel threads could panic the
system due to a null pointer indirection in a kprintf(). Fix
the kprintf().

kernel - Remove vm_map min_offset and max_offset macros

* The symbols 'min_offset' and 'max_offset' are common, do not use
them as a #define'd macro for vm_map's header.start and header.end.

* Fi

kernel - Remove vm_map min_offset and max_offset macros

* The symbols 'min_offset' and 'max_offset' are common, do not use
them as a #define'd macro for vm_map's header.start and header.end.

* Fixes symbol collision related to the drm work.

show more ...

kernel: Remove numerous #include <sys/thread2.h>.

Most of them were added when we converted spl*() calls to
crit_enter()/crit_exit(), almost 14 years ago. We can now
remove a good chunk of them agai

kernel: Remove numerous #include <sys/thread2.h>.

Most of them were added when we converted spl*() calls to
crit_enter()/crit_exit(), almost 14 years ago. We can now
remove a good chunk of them again for where crit_*() are
no longer used.

I had to adjust some files that were relying on thread2.h
or headers that it includes coming in via other headers
that it was removed from.

show more ...

kernel - Reduce vm_page_list_find2() stalls

* Reduce stalls in vm_page_list_find2() which can occur in low-memory
situations, as well as in other situations. The problem is two fold.

First, th

kernel - Reduce vm_page_list_find2() stalls

* Reduce stalls in vm_page_list_find2() which can occur in low-memory
situations, as well as in other situations. The problem is two fold.

First, that potentially all cpu cores can wind up waiting for a
single vm_page's spin-lock to be released.

Second, that a long-held vm_page spin-lock can cause the VM system
to stall unnecessarily long.

* Change vm_page_list_find() and vm_page_list_find2() to no longer
unconditionally spinlock a vm_page candidate and then retry if it
is found to be on the wrong queue.

Instead the code now spinlocks the queue, then iterates vm_page
candidates using spin_trylock(), skipping any pages whos spinlocks
cannot be immediately acquired. This is lock-order-reversed but
is ok because we use trylock. Also, by locking the queue first we
guarantee that a successfully spinlocked vm_page will be on the
correct queue.

* Should also reduce IPIQ drain stalls reported to the console as
shown below. The %rip sample is often found in vm_page_list_find2().

send_ipiq X->Y tgt not draining (STALL_SECONDS)

show more ...

sys/vm: Fix debug message in generic VOP_GETPAGES() handler

kernel - Implement sbrk(), change low-address mmap hinting

* Change mmap()'s internal lower address bound from dmax (32GB)
to RLIMIT_DATA's current value. This allows the rlimit to be
e.g. redu

kernel - Implement sbrk(), change low-address mmap hinting

* Change mmap()'s internal lower address bound from dmax (32GB)
to RLIMIT_DATA's current value. This allows the rlimit to be
e.g. reduced and for hinted mmap()s to then map space below
the 4GB mark. The default data rlimit is 32GB.

This change is needed to support several languages, at least
lua and probably another one or two, who use mmap hinting
under the assumption that it can map space below the 4GB
address mark. The data limit must be lowered with a limit command
too, which can be scripted or patched for such programs.

* Implement the sbrk() system call. This system call was already
present but just returned EOPNOTSUPP and libc previously had its
own shim for sbrk() which used the ancient break() system call.
(Note that the prior implementation did not ENOSYS or signal).

sbrk() in the kernel is thread-safe for positive increments and
is also byte-granular (the old libc sbrk() was only page-granular).

sbrk() in the kernel does not implement negative increments and
will return EOPNOTSUPP if asked to. Negative increments were
historically designed to be able to 'free' memory allocated with
sbrk(), but it is not possible to implement the case in a modern
VM system due to the mmap changes above.

(1) Because the new mmap hinting changes make it possible for
normal mmap()s to have mapped space prior to the RLIMIT_DATA resource
limit being increased, causing intermingling of sbrk() and user mmap()d
regions. (2) because negative increments are not even remotely
thread-safe.

* Note the previous commit refactored libc to use the kernel sbrk()
and fall-back to its previous emulation code on failure, so libc
supports both new and old kernels.

* Remove the brk() shim from libc. brk() is not implemented by the
kernel. Symbol removed. Requires testing against ports so we may
have to add it back in but basically there is no way to implement
brk() properly with the mmap() hinting fix

* Adjust manual pages.

show more ...

kernel - Fix panic in mlock() and munlock()

* Fix a degenerate case when mlock() or munlock() is called
with a size of 0. Silently allow the case and return success
(0).

* Fixes vm_map panic.