kernel/libc: Remove the old vmm code.

Removes the kernel code and two system calls.

Bump __DragonFly_version too.

Reviewed-by: aly, dillon

vm: Change 'kernel_object' global to pointer type

Following the previous commits, this commit changes the 'kernel_object'
to pointer type of 'struct vm_object *'. This makes it align better
with 'k

vm: Change 'kernel_object' global to pointer type

Following the previous commits, this commit changes the 'kernel_object'
to pointer type of 'struct vm_object *'. This makes it align better
with 'kernel_map' and simplifies the code a bit.

No functional changes.

show more ...

vm: Change 'kernel_map' global to type of 'struct vm_map *'

Change the global variable 'kernel_map' from type 'struct vm_map' to a
pointer to this struct. This simplify the code a bit since all
inv

vm: Change 'kernel_map' global to type of 'struct vm_map *'

Change the global variable 'kernel_map' from type 'struct vm_map' to a
pointer to this struct. This simplify the code a bit since all
invocations take its address. This change also aligns with NetBSD's
'kernal_map' that it's also a pointer, which also helps the porting of
NVMM.

No functional changes.

show more ...

vkernel: Include <sys/malloc.h> for malloc_reinit_ncpus() prototype.

Fixes the VKERNEL64 build.

kernel - Refactor malloc_type to reduce static data in image

* malloc_type was embedding a SMP_MAXCPU array of kmalloc_use
structures, which winds up being 16KB a pop x 400+ MALLOC_DEFINE()
decl

kernel - Refactor malloc_type to reduce static data in image

* malloc_type was embedding a SMP_MAXCPU array of kmalloc_use
structures, which winds up being 16KB a pop x 400+ MALLOC_DEFINE()
declarations.

This was over 6MB of static data in the kernel binary, and it
wasn't BSS because the declaration is initialized with some defaults.
So this reduction is significant and directly impacts both memory
use and kernel boot times.

* Change malloc_type->ks_use from an array to a pointer. Embed
a single kmalloc_use structure (ks_use0) as the default.

When ncpus is probed, the kernel now goes through all malloc_type
structures and dynamically allocates a properly-sized ks_use
array. Any new malloc hoppers after that point will also dynamically
allocate ks_use.

show more ...

Add <sys/cpumask.h>.

Collect and gather all scatter cpumask bits to correct headers. This
cleans up the namespace and simplifies platform handling in asm macros.
The cpumask_t together with its m

Add <sys/cpumask.h>.

Collect and gather all scatter cpumask bits to correct headers. This
cleans up the namespace and simplifies platform handling in asm macros.
The cpumask_t together with its macros is already non MI feature that is
used in userland utilities, libraries, kernel scheduler and syscalls.
It deserves sys/ header. Adjust syscalls.master and rerun sysent.

While there, fix an issue in ports that set POSIX env, but has
implementation of setting thread names through pthread_set_name_np().

show more ...

kernel: Remove numerous #include <sys/thread2.h>.

Most of them were added when we converted spl*() calls to
crit_enter()/crit_exit(), almost 14 years ago. We can now
remove a good chunk of them agai

kernel: Remove numerous #include <sys/thread2.h>.

Most of them were added when we converted spl*() calls to
crit_enter()/crit_exit(), almost 14 years ago. We can now
remove a good chunk of them again for where crit_*() are
no longer used.

I had to adjust some files that were relying on thread2.h
or headers that it includes coming in via other headers
that it was removed from.

show more ...

kern: Remove ncpus2 and friends.

They were no longer used, after netisr_ncpus was delployed.

Reminded-by: dillon@

arc4random: Make arc4random context per-cpu.

Critical section is commented out, no consumers from ISRs/ithreads.

vkernel - Restabilize pmap code, redo kqueue, systimer, and console code

* Remove vm_token and add necessary vm_page spin locks to the vkernel's
pmap code, improving its stability.

* Separate the

vkernel - Restabilize pmap code, redo kqueue, systimer, and console code

* Remove vm_token and add necessary vm_page spin locks to the vkernel's
pmap code, improving its stability.

* Separate the systimer interrupt and console tty support from the
kqueue subsystem.

Uses SIGURG for systimer
Uses SIGIO for kqueue
Uses SIGALRM for cothread signalling

* The vkernel systimer code now uses a dedicated cothread for timing. The
cothread is a bit of a hack at the moment but is a more direct way of
handling systimers.

* Attempt to fix user%/sys%/intr%/idle% in the systat -vm and
systat -pv output. Still isn't perfect, but it is now more
accurate.

show more ...

kernel - Tag vm_map_entry structure, slight optimization to zalloc, misc.

* Tag the vm_map_entry structure, allowing debugging programs to
break-down how KMEM is being used more easily.

This re

kernel - Tag vm_map_entry structure, slight optimization to zalloc, misc.

* Tag the vm_map_entry structure, allowing debugging programs to
break-down how KMEM is being used more easily.

This requires an additional argument to vm_map_find() and most
kmem_alloc*() functions.

* Remove the page chunking parameter to zinit() and zinitna(). It was
only being used degeneratively. Increase the chunking from one page
to four pages, which will reduce the amount of vm_map_entry spam in
the kernel_map.

* Use atomic ops when adjusting zone_kern_pages.

show more ...

vkernel - Fix FP corruption in VMX/EPT mode

* Properly invalidating the EPT TLB cache when it potentially becomes
stale.

* When destroying a VMX context, make sure the EPT TLB cache has been
in

vkernel - Fix FP corruption in VMX/EPT mode

* Properly invalidating the EPT TLB cache when it potentially becomes
stale.

* When destroying a VMX context, make sure the EPT TLB cache has been
invalidated for that context as a safety.

* Fix a bug in vmx_vminit() where it was losing track of the currently
loaded VMX.

* Setup the VMX to clear the host CR0_TS, and the host makes sure the FP
state is correct prior to vmlaunch.

* Make sure the GUEST_CR0's CR0_TS reflects the vkernel says it should
reflect.

* The vkernel has a choice of asking the guest user context to #NM fault
on FP use or not. Both mechanics work, but its probably better for it
to ensure that the FP state is valid and tell the user context to not
fault. However, this commit makes sure that both mechanics work.

* Document why we shouldn't vmclear the old current context when loading
a different context.

* Cleanup some of the vkernel's pmap handling. This isn't perfect and
probably needs to be rewritten (we need a more powerful guest pmap
adjustment system call to properly synchronize changes). For now
we try to avoid races against detecting the (M)odified flag by clearing
the RW flag first.

show more ...

vkernel - Fix a vkernel lockup on startup

- During ap_init() any pending IPIs is processed manually so
clear gd_npoll as the real kernel does.
- Do not disable interrupts for vkernels during lwkt_

vkernel - Fix a vkernel lockup on startup

- During ap_init() any pending IPIs is processed manually so
clear gd_npoll as the real kernel does.
- Do not disable interrupts for vkernels during lwkt_send_ipiq3()
because they don't seem to be re-enabled afterwards as they should.
I'm not entirely sure this is the right fix, more investigation
is required.

show more ...

kernel: Move semicolon from the definition of SYSINIT() to its invocations.

This affected around 70 of our (more or less) 270 SYSINIT() calls.

style(9) advocates the terminating semicolon to be sup

kernel: Move semicolon from the definition of SYSINIT() to its invocations.

This affected around 70 of our (more or less) 270 SYSINIT() calls.

style(9) advocates the terminating semicolon to be supplied by the
invocation too, because it can make life easier for editors and other
source code parsing programs.

show more ...

vkernel64 - Make operational again after bootstrap changes

* vkernel64's need the same cpu_idle_restore() fix that x86-64 got to
properly handle pending migrations on the BSP at bootstrap time.

vkernel64 - Make operational again after bootstrap changes

* vkernel64's need the same cpu_idle_restore() fix that x86-64 got to
properly handle pending migrations on the BSP at bootstrap time.
(this occurs due to the way initclocks_other() works now).

* Add a warning/reminder in param.h regarding increasing MAXCPU beyond 256.

show more ...

build - Fix vkernel64 build

* Fix build due to missing declaration from recent commit.

kernel - extend cpus past 64 - fixes and adjustments

* Reorder the SMP cpu boot code to remove a great deal of lock contention.
The APs must still loop waiting for the BSP to adjust the stage, but

kernel - extend cpus past 64 - fixes and adjustments

* Reorder the SMP cpu boot code to remove a great deal of lock contention.
The APs must still loop waiting for the BSP to adjust the stage, but
they no longer need to hold a token or spinlock so startup under emulation
is considerably faster.

* Do not initialize our systimer periodics on each target cpu from the
idle thread bootstrap. Previously with the MP lock held the locks acquired
during this initialization were serialized and could not block. Now
that cpu startup runs mostly concurrently, that is no longer the
case.

Instead, systimer periodics are handled by process 0 as a post-smp-startup
call.

* statclock() now uses sys_cputimer() directly to calculate the delta time.

* The TSC is now implemented as sys_cputimer before any systimer periodics
(particularly statclock()) are set-up, allowing the system to take control
away from the i8254 earlier.

* Clean up struct lwkt_ipiq. Remove the 'lwkt_ipiq' typedef. Calculate
allocation sizes separately.

* Add a new loader.conf tunable, hw.tsc_cputimer_force. If set to 1 and
a TSC is present, the system will force invariant and mpsync operation
and always use the TSC as the cputimer (primarily useful for qemu).

* Remove unnecessary kmem_alloc() of the globaldata structure. We are
using a static array now. This access was wasting memory for a long
time.

* Make the boot stack bigger for the APs.

show more ...

kernel: Unbreak vkernel64 build.

kernel - Refactor cpumask_t to extend cpus past 64, part 1/2

* 64-bit systems only. 32-bit builds use the macros but cannot be expanded
past 32 cpus.

* Change cpumask_t from __uint64_t to a stru

kernel - Refactor cpumask_t to extend cpus past 64, part 1/2

* 64-bit systems only. 32-bit builds use the macros but cannot be expanded
past 32 cpus.

* Change cpumask_t from __uint64_t to a structure. This commit implements
one 64-bit sub-element (the next one will implement four for 256 cpus).

* Create a CPUMASK_*() macro API for non-atomic and atomic cpumask
manipulation. These macros generally take lvalues as arguments, allowing
for a fairly optimal implementation.

* Change all C code operating on cpumask's to use the newly created CPUMASK_*()
macro API.

* Compile-test 32 and 64-bit. Run-test 64-bit.

* Adjust sbin/usched, usr.sbin/powerd. usched currently needs more work.

show more ...

hammer2 - Merge Mihai Carabas's VKERNEL/VMM GSOC project into the main tree

* This merge contains work primarily by Mihai Carabas, with some misc
fixes also by Matthew Dillon.

* Special note on G

hammer2 - Merge Mihai Carabas's VKERNEL/VMM GSOC project into the main tree

* This merge contains work primarily by Mihai Carabas, with some misc
fixes also by Matthew Dillon.

* Special note on GSOC core

This is, needless to say, a huge amount of work compressed down into a
few paragraphs of comments. Adds the pc64/vmm subdirectory and tons
of stuff to support hardware virtualization in guest-user mode, plus
the ability for programs (vkernels) running in this mode to make normal
system calls to the host.

* Add system call infrastructure for VMM mode operations in kern/sys_vmm.c
which vectors through a structure to machine-specific implementations.

vmm_guest_ctl_args()
vmm_guest_sync_addr_args()

vmm_guest_ctl_args() - bootstrap VMM and EPT modes. Copydown the original
user stack for EPT (since EPT 'physical' addresses cannot reach that far
into the backing store represented by the process's original VM space).
Also installs the GUEST_CR3 for the guest using parameters supplied by
the guest.

vmm_guest_sync_addr_args() - A host helper function that the vkernel can
use to invalidate page tables on multiple real cpus. This is a lot more
efficient than having the vkernel try to do it itself with IPI signals
via cpusync*().

* Add Intel VMX support to the host infrastructure. Again, tons of work
compressed down into a one paragraph commit message. Intel VMX support
added. AMD SVM support is not part of this GSOC and not yet supported
by DragonFly.

* Remove PG_* defines for PTE's and related mmu operations. Replace with
a table lookup so the same pmap code can be used for normal page tables
and also EPT tables.

* Also include X86_PG_V defines specific to normal page tables for a few
situations outside the pmap code.

* Adjust DDB to disassemble SVM related (intel) instructions.

* Add infrastructure to exit1() to deal related structures.

* Optimize pfind() and pfindn() to remove the global token when looking
up the current process's PID (Matt)

* Add support for EPT (double layer page tables). This primarily required
adjusting the pmap code to use a table lookup to get the PG_* bits.

Add an indirect vector for copyin, copyout, and other user address space
copy operations to support manual walks when EPT is in use.

A multitude of system calls which manually looked up user addresses via
the vm_map now need a VMM layer call to translate EPT.

* Remove the MP lock from trapsignal() use cases in trap().

* (Matt) Add pthread_yield()s in most spin loops to help situations where
the vkernel is running on more cpu's than the host has, and to help with
scheduler edge cases on the host.

* (Matt) Add a pmap_fault_page_quick() infrastructure that vm_fault_page()
uses to try to shortcut operations and avoid locks. Implement it for
pc64. This function checks whether the page is already faulted in as
requested by looking up the PTE. If not it returns NULL and the full
blown vm_fault_page() code continues running.

* (Matt) Remove the MP lock from most the vkernel's trap() code

* (Matt) Use a shared spinlock when possible for certain critical paths
related to the copyin/copyout path.

show more ...

kernel: Make SMP support default (and non-optional).

The 'SMP' kernel option gets removed with this commit, so it has to
be removed from everybody's configs.

Reviewed-by: sjg
Approved-by: many

vkernel{,64} - CPU topology support

* Part of "Add SMT/HT awareness to DragonFly BSD scheduler" GSoC
project.

* Details at: http://leaf.dragonflybsd.org/mailarchive/kernel/2012-08/msg00009.htm

vkernel{,64} - CPU topology support

* Part of "Add SMT/HT awareness to DragonFly BSD scheduler" GSoC
project.

* Details at: http://leaf.dragonflybsd.org/mailarchive/kernel/2012-08/msg00009.html

Mentored-by: Alex Hornung (alexh@)
Sponsored-by: Google Summer of Code 2012

show more ...

Initial import of binutils 2.22 on the new vendor branch

Future versions of binutils will also reside on this branch rather
than continuing to create new binutils branches for each new version.

kernel - Major SMP performance patch / VM system, bus-fault/seg-fault fixes

This is a very large patch which reworks locking in the entire VM subsystem,
concentrated on VM objects and the x86-64 pma

kernel - Major SMP performance patch / VM system, bus-fault/seg-fault fixes

This is a very large patch which reworks locking in the entire VM subsystem,
concentrated on VM objects and the x86-64 pmap code. These fixes remove
nearly all the spin lock contention for non-threaded VM faults and narrows
contention for threaded VM faults to just the threads sharing the pmap.

Multi-socket many-core machines will see a 30-50% improvement in parallel
build performance (tested on a 48-core opteron), depending on how well
the build parallelizes.

As part of this work a long-standing problem on 64-bit systems where programs
would occasionally seg-fault or bus-fault for no reason has been fixed. The
problem was related to races between vm_fault, the vm_object collapse code,
and the vm_map splitting code.

* Most uses of vm_token have been removed. All uses of vm_spin have been
removed. These have been replaced with per-object tokens and per-queue
(vm_page_queues[]) spin locks.

Note in particular that since we still have the page coloring code the
PQ_FREE and PQ_CACHE queues are actually many queues, individually
spin-locked, resulting in very excellent MP page allocation and freeing
performance.

* Reworked vm_page_lookup() and vm_object->rb_memq. All (object,pindex)
lookup operations are now covered by the vm_object hold/drop system,
which utilize pool tokens on vm_objects. Calls now require that the
VM object be held in order to ensure a stable outcome.

Also added vm_page_lookup_busy_wait(), vm_page_lookup_busy_try(),
vm_page_busy_wait(), vm_page_busy_try(), and other API functions
which integrate the PG_BUSY handling.

* Added OBJ_CHAINLOCK. Most vm_object operations are protected by
the vm_object_hold/drop() facility which is token-based. Certain
critical functions which must traverse backing_object chains use
a hard-locking flag and lock almost the entire chain as it is traversed
to prevent races against object deallocation, collapses, and splits.

The last object in the chain (typically a vnode) is NOT locked in
this manner, so concurrent faults which terminate at the same vnode will
still have good performance. This is important e.g. for parallel compiles
which might be running dozens of the same compiler binary concurrently.

* Created a per vm_map token and removed most uses of vmspace_token.

* Removed the mp_lock in sys_execve(). It has not been needed in a while.

* Add kmem_lim_size() which returns approximate available memory (reduced
by available KVM), in megabytes. This is now used to scale up the
slab allocator cache and the pipe buffer caches to reduce unnecessary
global kmem operations.

* Rewrote vm_page_alloc(), various bits in vm/vm_contig.c, the swapcache
scan code, and the pageout scan code. These routines were rewritten
to use the per-queue spin locks.

* Replaced the exponential backoff in the spinlock code with something
a bit less complex and cleaned it up.

* Restructured the IPIQ func/arg1/arg2 array for better cache locality.
Removed the per-queue ip_npoll and replaced it with a per-cpu gd_npoll,
which is used by other cores to determine if they need to issue an
actual hardware IPI or not. This reduces hardware IPI issuance
considerably (and the removal of the decontention code reduced it even
more).

* Temporarily removed the lwkt thread fairq code and disabled a number of
features. These will be worked back in once we track down some of the
remaining performance issues.

Temproarily removed the lwkt thread resequencer for tokens for the same
reason. This might wind up being permanent.

Added splz_check()s in a few critical places.

* Increased the number of pool tokens from 1024 to 4001 and went to a
prime-number mod algorithm to reduce overlaps.

* Removed the token decontention code. This was a bit of an eyesore and
while it did its job when we had global locks it just gets in the way now
that most of the global locks are gone.

Replaced the decontention code with a fall back which acquires the
tokens in sorted order, to guarantee that deadlocks will always be
resolved eventually in the scheduler.

* Introduced a simplified spin-for-a-little-while function
_lwkt_trytoken_spin() that the token code now uses rather than giving
up immediately.

* The vfs_bio subsystem no longer uses vm_token and now uses the
vm_object_hold/drop API for buffer cache operations, resulting
in very good concurrency.

* Gave the vnode its own spinlock instead of sharing vp->v_lock.lk_spinlock,
which fixes a deadlock.

* Adjusted all platform pamp.c's to handle the new main kernel APIs. The
i386 pmap.c is still a bit out of date but should be compatible.

* Completely rewrote very large chunks of the x86-64 pmap.c code. The
critical path no longer needs pmap_spin but pmap_spin itself is still
used heavily, particularin the pv_entry handling code.

A per-pmap token and per-pmap object are now used to serialize pmamp
access and vm_page lookup operations when needed.

The x86-64 pmap.c code now uses only vm_page->crit_count instead of
both crit_count and hold_count, which fixes races against other parts of
the kernel uses vm_page_hold().

_pmap_allocpte() mechanics have been completely rewritten to remove
potential races. Much of pmap_enter() and pmap_enter_quick() has also
been rewritten.

Many other changes.

* The following subsystems (and probably more) no longer use the vm_token
or vmobj_token in critical paths:

x The swap_pager now uses the vm_object_hold/drop API instead of vm_token.

x mmap() and vm_map/vm_mmap in general now use the vm_object_hold/drop API
instead of vm_token.

x vnode_pager

x zalloc

x vm_page handling

x vfs_bio

x umtx system calls

x vm_fault and friends

* Minor fixes to fill_kinfo_proc() to deal with process scan panics (ps)
revealed by recent global lock removals.

* lockmgr() locks no longer support LK_NOSPINWAIT. Spin locks are
unconditionally acquired.

* Replaced netif/e1000's spinlocks with lockmgr locks. The spinlocks
were not appropriate owing to the large context they were covering.

* Misc atomic ops added

show more ...

kernel - Fix rare IPIQ freezes

* Ensure that an IPI interrupt is sent went waiting for an IPIQ
to drain. The IPIQ can be pushed up by passive IPIs and not
necessarily have a signal pending on t

kernel - Fix rare IPIQ freezes

* Ensure that an IPI interrupt is sent went waiting for an IPIQ
to drain. The IPIQ can be pushed up by passive IPIs and not
necessarily have a signal pending on the target cpu, so we
have to put the check in our drain loop rather than outside
the loop.

* Add a cpu_pause() to reduce power use for the IPIQ drain case.

* Normalize the use of gd_intr_nesting_level and clean up the
code syntax for the ipiq_optimized case.

* Remove the previous temporary IPI interrupt signaling fix, it
was incomplete.

* Fix a missing crit_exit() in the ENOENT case for
lwkt_send_ipiq3_nowait().

* Track cpu's which are in the middle of ipiq processing and
assert that a cpu is not in an IPIQ processing loop when
switching between threads.

* Normalize the use of ip->ip_npoll in the IPIQ code. This
field is used to avoid unnecessary IPI interrupts.

show more ...