Use per-domain locks for the bucket cache.

This gives much better concurrency when there are a large number of
cores per-domain and multiple domains. Avoid taking the lock entirely
if it will not b

Use per-domain locks for the bucket cache.

This gives much better concurrency when there are a large number of
cores per-domain and multiple domains. Avoid taking the lock entirely
if it will not be productive. ROUNDROBIN domains will have mixed
memory in each domain and will load balance to all domains.

While here refactor the zone/domain separation and bucket limits to
simplify callers.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D23673

show more ...

Add a simple accessor that returns the bytes of memory consumed by a zone.

UMA has become more particular about zone types. Use the right allocator
calls in uma_zwait().

Slightly restructure uma_zalloc* to generate better code from clang and
reduce duplication among zalloc functions.

Reviewed by: markj
Discussed with: mjg
Differential Revision: https://reviews.freeb

Slightly restructure uma_zalloc* to generate better code from clang and
reduce duplication among zalloc functions.

Reviewed by: markj
Discussed with: mjg
Differential Revision: https://reviews.freebsd.org/D23672

show more ...

Update the zone-global count of cached items in bucket_cache_reclaim().

This was missed in r351673. The count is used to enfore cache limits,
which are rarely used.

Discussed with: jeff
Sponsored

Update the zone-global count of cached items in bucket_cache_reclaim().

This was missed in r351673. The count is used to enfore cache limits,
which are rarely used.

Discussed with: jeff
Sponsored by: The FreeBSD Foundation

show more ...

Fix a case where ub_seq would fail to be set if the cross bucket was
flushed due to memory pressure.

Reviewed by: markj
Differential Revision: http://reviews.freebsd.org/D23614

Store offset into zpcpu allocations in the per-cpu area.

This shorten zpcpu_get and allows more optimizations.

Reviewed by: jeff
Differential Revision: https://reviews.freebsd.org/D23570

Reduce lock hold time in keg_drain().

Maintain a count of free slabs in the per-domain keg structure and use
that to clear the free slab list in constant time for most cases. This
helps minimize lo

Reduce lock hold time in keg_drain().

Maintain a count of free slabs in the per-domain keg structure and use
that to clear the free slab list in constant time for most cases. This
helps minimize lock contention induced by reclamation, in preparation
for proactive trimming of excesses of free memory.

Reviewed by: jeff, rlibby
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D23532

show more ...

uma: remove UMA_ZFLAG_CACHEONLY flag

UMA_ZFLAG_CACHEONLY was essentially the same thing as UMA_ZONE_VM, but
with a more confusing name. Remove the flag, make UMA_ZONE_VM an
inherit flag, and replac

uma: remove UMA_ZFLAG_CACHEONLY flag

UMA_ZFLAG_CACHEONLY was essentially the same thing as UMA_ZONE_VM, but
with a more confusing name. Remove the flag, make UMA_ZONE_VM an
inherit flag, and replace all references.

Reviewed by: markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D23516

show more ...

uma: multipage chicken switch

Add a switch to allow disabling multipage slabs, in order to facilitate
measuring memory usage and performance effects. The tunable
vm.debug.uma_multipage_slabs defaul

uma: multipage chicken switch

Add a switch to allow disabling multipage slabs, in order to facilitate
measuring memory usage and performance effects. The tunable
vm.debug.uma_multipage_slabs defaults to 1 and can be set to 0 to
disable. The name may change soon.

Reviewed by: markj (previous version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D23487

show more ...

uma: grow slabs to enforce minimum memory efficiency

Memory efficiency can be poor with awkward item sizes (e.g. 1/2 or 1
page size + epsilon). In order to achieve a minimum memory efficiency,
sele

uma: grow slabs to enforce minimum memory efficiency

Memory efficiency can be poor with awkward item sizes (e.g. 1/2 or 1
page size + epsilon). In order to achieve a minimum memory efficiency,
select a slab size with a potentially larger number of pages if it
yields a lower portion of waste.

This may mean using page_alloc instead of uma_small_alloc, which could
be more costly.

Discussed with: jeff, mckusick
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D23239

show more ...

uma: add UMA_ZONE_CONTIG, and a default contig_alloc

For now, copy the mbuf allocator.

Reviewed by: jeff, markj (previous version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://revie

uma: add UMA_ZONE_CONTIG, and a default contig_alloc

For now, copy the mbuf allocator.

Reviewed by: jeff, markj (previous version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D23237

show more ...

uma: pcpu_page_free needs to startup_free pages from startup_alloc

After r357392, it is apparent that we do have some early-boot PCPU
zones. Make it so we can safely free pages from them if they ar

uma: pcpu_page_free needs to startup_free pages from startup_alloc

After r357392, it is apparent that we do have some early-boot PCPU
zones. Make it so we can safely free pages from them if they are
actually used during early boot.

Reviewed by: jeff, markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D23496

show more ...

Use literal bucket sizes for smaller buckets rather than the rounding
system. Small bucket sizes already pack well even if they are an odd
number of words. This prevents any potential new instances

Use literal bucket sizes for smaller buckets rather than the rounding
system. Small bucket sizes already pack well even if they are an odd
number of words. This prevents any potential new instances of the
problem fixed in r357463 as well as making the system easier to
understand.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D23494

show more ...

Use STAILQ instead of TAILQ for bucket lists. We only need FIFO behavior
and this is more space efficient.

Stop queueing recently used buckets to the head of the list. If the bucket
goes to a diff

Use STAILQ instead of TAILQ for bucket lists. We only need FIFO behavior
and this is more space efficient.

Stop queueing recently used buckets to the head of the list. If the bucket
goes to a different processor the cache coherency will be more expensive.
We already try to encourage cache-hot behavior in the per-cpu layer.

Reviewed by: rlibby
Differential Revision: https://reviews.freebsd.org/D23493

show more ...

Disable the smallest UMA bucket size on 32-bit platforms.

With r357314, sizeof(struct uma_bucket) grew to 16 bytes on 32-bit
platforms, so BUCKET_SIZE(4) is 0. This resulted in the creation of a
bu

Disable the smallest UMA bucket size on 32-bit platforms.

With r357314, sizeof(struct uma_bucket) grew to 16 bytes on 32-bit
platforms, so BUCKET_SIZE(4) is 0. This resulted in the creation of a
bucket zone for buckets with zero capacity. A more general fix is
planned, but for now this bandaid allows 32-bit platforms to boot again.

PR: 243837
Discussed with: jeff
Reported by: pho, Jenkins via lwhsu
Tested by: pho
Sponsored by: The FreeBSD Foundation

show more ...

Fix a bug in r356776 where the page allocator was not properly restored to
the percpu page allocator after it had been temporarily overridden by
startup_alloc.

Reported by: pho, bdragon

Fix LINT build with MEMGUARD.

Implement a safe memory reclamation feature that is tightly coupled with UMA.

This is in the same family of algorithms as Epoch/QSBR/RCU/PARSEC but is
a unique algorithm. This has 3x the performanc

Implement a safe memory reclamation feature that is tightly coupled with UMA.

This is in the same family of algorithms as Epoch/QSBR/RCU/PARSEC but is
a unique algorithm. This has 3x the performance of epoch in a write heavy
workload with less than half of the read side cost. The memory overhead
is significantly lessened by limiting the free-to-use latency. A synthetic
test uses 1/20th of the memory vs Epoch. There is significant further
discussion in the comments and code review.

This code should be considered experimental. I will write a man page after
it has settled. After further validation the VM will begin using this
feature to permit lockless page lookups.

Both markj and cperciva tested on arm64 at large core counts to verify
fences on weaker ordering architectures. I will commit a stress testing
tool in a follow-up.

Reviewed by: mmacy, markj, rlibby, hselasky
Discussed with: sbahara
Differential Revision: https://reviews.freebsd.org/D22586

show more ...

uma: fix zone domain overlaying pcpu cache with disabled cpus

UMA zone structures have two arrays at the end which are sized according
to the machine: an array of CPU count length, and an array of N

uma: fix zone domain overlaying pcpu cache with disabled cpus

UMA zone structures have two arrays at the end which are sized according
to the machine: an array of CPU count length, and an array of NUMA
domain count length. The CPU counting was wrong in the case where some
CPUs are disabled (when mp_ncpus != mp_maxid + 1), and this caused the
second array to be overlaid with the first.

Reported by: olivier
Reviewed by: jeff, markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D23318

show more ...

uma: report leaks more accurately

Previously UMA had some false negatives in the leak report at keg
destruction time, where it only reported leaks if there were free items
in the slab layer (rather

uma: report leaks more accurately

Previously UMA had some false negatives in the leak report at keg
destruction time, where it only reported leaks if there were free items
in the slab layer (rather than allocated items), which notably would not
be true for single-item slabs (large items). Now, report a leak if
there are any allocated pages, and calculate and report the number of
allocated items rather than free items.

Reviewed by: jeff, markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D23275

show more ...

Some architectures with DMAP still consume boot kva. Simplify the test for
claiming kva in uma_startup2() to handle this.

Reported by: bdragon

pcpu_page_alloc: guard against empty NUMA domains

Some systems, such as higher end Threadripper, may have
NUMA domains with no physical memory, Don't allocate
from these domains.

This fixes a "pani

pcpu_page_alloc: guard against empty NUMA domains

Some systems, such as higher end Threadripper, may have
NUMA domains with no physical memory, Don't allocate
from these domains.

This fixes a "panic: vm_wait in early boot" on my 2990WX desktop

Reviewed by: jeff
Sponsored by: Netflix

show more ...

Simplify VM and UMA startup by eliminating boot pages. Instead use careful
ordering to allocate early pages in the same way boot pages were but only
as needed. After the KVA allocator has started u

Simplify VM and UMA startup by eliminating boot pages. Instead use careful
ordering to allocate early pages in the same way boot pages were but only
as needed. After the KVA allocator has started up we allocate the KVA that
we consumed during boot. This also makes the boot pages freeable since they
have vm_page structures allocated with the rest of memory.

Parts of this patch were written and tested by markj.

Reviewed by: glebius, markj
Differential Revision: https://reviews.freebsd.org/D23102

show more ...

uma: split slabzone into two sizes

By allowing more items per slab, we can improve memory efficiency for
small allocs. If we were just to increase the bitmap size of the
slabzone, we would then was

uma: split slabzone into two sizes

By allowing more items per slab, we can improve memory efficiency for
small allocs. If we were just to increase the bitmap size of the
slabzone, we would then waste slabzone memory. So, split slabzone into
two zones, one especially for 8-byte allocs (512 per slab). The
practical effect should be reduced memory usage for counter(9).

Reviewed by: jeff, markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D23149

show more ...