sys/malloc.h: move sys/proc.h MALLOC_DECLAREs here

This avoids the need to explicitly include sys/malloc.h before
sys/proc.h.

Suggested by: kib
Reviewed by: kib
Differential Revision: https://revie

sys/malloc.h: move sys/proc.h MALLOC_DECLAREs here

This avoids the need to explicitly include sys/malloc.h before
sys/proc.h.

Suggested by: kib
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D44766

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .h pattern

Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/

Remove bzero declaration

The kernel was migrated to memset in ba96f37758412151 ("Use __builtin
for various mem* and b* (e.g. bzero) routines.") and there are no
remaining architectures using the fil

Remove bzero declaration

The kernel was migrated to memset in ba96f37758412151 ("Use __builtin
for various mem* and b* (e.g. bzero) routines.") and there are no
remaining architectures using the file.

malloc is augmented to prevent KMSAN from breaking.

Sponsored by: Rubicon Communications, LLC ("Netgate")

show more ...

malloc(9): Document/complete aligned variants

Comments on a pending kvmclock driver suggested adding a
malloc_aligned() to complement malloc_domainset_aligned(); add it now,
and document both.

Revi

malloc(9): Document/complete aligned variants

Comments on a pending kvmclock driver suggested adding a
malloc_aligned() to complement malloc_domainset_aligned(); add it now,
and document both.

Reviewed by: imp, kib, allanjude (manpages)
Differential Revision: https://reviews.freebsd.org/D31004

show more ...

stand/kmem_zalloc: panic when a M_WAITOK allocation fails

Malloc() might return NULL, in which case we will panic with a NULL
pointer deref. Make it panic when the allocation fails to preserve the
p

stand/kmem_zalloc: panic when a M_WAITOK allocation fails

Malloc() might return NULL, in which case we will panic with a NULL
pointer deref. Make it panic when the allocation fails to preserve the
postcondtion that we never return a non-NULL value.

Reviewed by: tsoome
PR: 249859
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D31106

show more ...

Implement mallocarray_domainset(9) variant of mallocarray(9).

Reviewed by: kib @
MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

Add a VM flag to prevent reclaim on a failed contig allocation

If a M_WAITOK contig alloc fails, the VM subsystem will try to
reclaim contiguous memory twice before actually failing the
request. On

Add a VM flag to prevent reclaim on a failed contig allocation

If a M_WAITOK contig alloc fails, the VM subsystem will try to
reclaim contiguous memory twice before actually failing the
request. On a system with 64GB of RAM I've observed this take
400-500ms before it finally gives up, and I believe that this
will only be worse on systems with even more memory.

In certain contexts this delay is extremely harmful, so add a flag
that will skip reclaim for allocation requests to allow those
paths to opt-out of doing an expensive reclaim.

Sponsored by: Dell Inc
Differential Revision: https://reviews.freebsd.org/D28422
Reviewed by: markj, kib

show more ...

Implement malloc_domainset_aligned(9).

Change the power-of-two malloc zones to require alignment equal to the
size [*]. Current uma allocator already provides such alignment, so in
fact this change

Implement malloc_domainset_aligned(9).

Change the power-of-two malloc zones to require alignment equal to the
size [*]. Current uma allocator already provides such alignment, so in
fact this change does not change anything except providing future-proof
setup.

Suggested by: markj [*]
Reviewed by: andrew, jah, markj
Tested by: pho
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28147

show more ...

[malloc] quieten -Werror=missing-braces with malloc.h wth gcc-6.4

This sets off gcc-6.4 to spit out a 'error: missing braces around initializer'
error when compiling this.

Remove it as it isn't nee

[malloc] quieten -Werror=missing-braces with malloc.h wth gcc-6.4

This sets off gcc-6.4 to spit out a 'error: missing braces around initializer'
error when compiling this.

Remove it as it isn't needed.

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D27183

show more ...

malloc: retire malloc_last_fail

The routine does not serve any practical purpose.

Memory can be allocated in many other ways and most consumers pass the
M_WAITOK flag, making malloc not fail in the

malloc: retire malloc_last_fail

The routine does not serve any practical purpose.

Memory can be allocated in many other ways and most consumers pass the
M_WAITOK flag, making malloc not fail in the first place.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27143

show more ...

malloc: retire mt_stats_zone in favor of pcpu_zone_64

Reviewed by: markj, imp
Differential Revision: https://reviews.freebsd.org/D27142

malloc: move malloc_type_internal into malloc_type

According to code comments the original motivation was to allow for
malloc_type_internal changes without ABI breakage. This can be trivially
accomp

malloc: move malloc_type_internal into malloc_type

According to code comments the original motivation was to allow for
malloc_type_internal changes without ABI breakage. This can be trivially
accomplished by providing spare fields and versioning the struct, as
implemented in the patch below.

The upshots are one less memory indirection on each alloc and disappearance
of mt_zone.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27104

show more ...

malloc: add a helper returning size allocated for given request

Sample usage: kernel modules can decide whether to stick to malloc or
create their own zone.

Reviewed by: markj
Differential Revision

malloc: add a helper returning size allocated for given request

Sample usage: kernel modules can decide whether to stick to malloc or
create their own zone.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27097

show more ...

malloc: delegate M_EXEC handling to dedicacted routines

It is almost never needed and adds an avoidable branch.

While here do minior clean ups in preparation for larger changes.

Reviewed by: markj

malloc: delegate M_EXEC handling to dedicacted routines

It is almost never needed and adds an avoidable branch.

While here do minior clean ups in preparation for larger changes.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27019

show more ...

Create a standalone version of sys/malloc.h

The ZSTD support for the boot loader will need to include files that
use the kernel's malloc interface. Create a standalone stub version
that's functional

Create a standalone version of sys/malloc.h

The ZSTD support for the boot loader will need to include files that
use the kernel's malloc interface. Create a standalone stub version
that's functional enough to allow this to work. There's some
limitations in this interface, and it's not quite a perfect
match. Specifically, M_WAITOK allocations can fail because there's
nothing that can be done we no memory is available.

show more ...

LinuxKPI: Implement ksize() function.

In Linux, ksize() gets the actual amount of memory allocated for a given
object. This commit adds malloc_usable_size() to FreeBSD KPI which does
the same. It al

LinuxKPI: Implement ksize() function.

In Linux, ksize() gets the actual amount of memory allocated for a given
object. This commit adds malloc_usable_size() to FreeBSD KPI which does
the same. It also maps LinuxKPI ksize() to newly created function.

ksize() function is used by drm-kmod.

Reviewed by: hselasky, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26215

show more ...

Remove free_domain() and uma_zfree_domain().

These functions were introduced before UMA started ensuring that freed
memory gets placed in domain-local caches. They no longer serve any
purpose since

Remove free_domain() and uma_zfree_domain().

These functions were introduced before UMA started ensuring that freed
memory gets placed in domain-local caches. They no longer serve any
purpose since UMA now provides their functionality by default. Remove
them to simplyify the kernel memory allocator interfaces a bit.

Reviewed by: cem, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D25937

show more ...

Add zfree to zero allocation before free

Key and cookie management typically wants to
avoid information leaks by explicitly zeroing
before free. This routine simplifies that by
permitting consumers

Add zfree to zero allocation before free

Key and cookie management typically wants to
avoid information leaks by explicitly zeroing
before free. This routine simplifies that by
permitting consumers to do so without carrying
the size around.

Reviewed by: jeff@, jhb@
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC (Netgate)
Differential Revision: https://reviews.freebsd.org/D22790

show more ...

The alignment is passed into contigmalloc_domainset in the 7th argument.

KUBSAN was complaining the pointer contigmalloc_domainset returned was
misaligned. Fix this by using the correct argument to

The alignment is passed into contigmalloc_domainset in the 7th argument.

KUBSAN was complaining the pointer contigmalloc_domainset returned was
misaligned. Fix this by using the correct argument to find the alignment
in the function signature.

Reported by: KUBSAN
MFC after: 2 weeks
Sponsored by: DARPA, AFRL

show more ...

Implement the M_NEXTFIT allocation strategy for vmem(9).

This is described in the vmem paper: "directs vmem to use the next free
segment after the one previously allocated." The implementation adds

Implement the M_NEXTFIT allocation strategy for vmem(9).

This is described in the vmem paper: "directs vmem to use the next free
segment after the one previously allocated." The implementation adds a
new boundary tag type, M_CURSOR, which is linked into the segment list
and precedes the segment following the previous M_NEXTFIT allocation.
The cursor is used to locate the next free segment satisfying the
allocation constraints.

This implementation isn't O(1) since busy tags aren't coalesced, and we
may potentially scan the entire segment list during an M_NEXTFIT
allocation.

Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D17226

show more ...

Add malloc_domainset(9) and _domainset variants to other allocator KPIs.

Remove malloc_domain(9) and most other _domain KPIs added in r327900.
The new functions allow the caller to specify a general

Add malloc_domainset(9) and _domainset variants to other allocator KPIs.

Remove malloc_domain(9) and most other _domain KPIs added in r327900.
The new functions allow the caller to specify a general NUMA domain
selection policy, rather than specifically requesting an allocation from
a specific domain. The latter policy tends to interact poorly with
M_WAITOK, resulting in situations where a caller is blocked indefinitely
because the specified domain is depleted. Most existing consumers of
the _domain KPIs are converted to instead use a DOMAINSET_PREF() policy,
in which we fall back to other domains to satisfy the allocation
request.

This change also defines a set of DOMAINSET_FIXED() policies, which
only permit allocations from the specified domain.

Discussed with: gallatin, jeff
Reported and tested by: pho (previous version)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17418

show more ...

Eliminate false sharing in malloc due to statistic collection

Currently stats are collected in a MAXCPU-sized array which is not
aligned and suffers enormous false-sharing. Fix the problem by
utiliz

Eliminate false sharing in malloc due to statistic collection

Currently stats are collected in a MAXCPU-sized array which is not
aligned and suffers enormous false-sharing. Fix the problem by
utilizing per-cpu allocation.

The counter(9) API is not used here as it is too incomplete and does
not provide a win over per-cpu zone sized for malloc stats struct. In
particular stats are being reported for each cpu separately by just
copying what is supposed to be an array element for given cpu.

This eliminates significant false-sharing during malloc-heavy tests
e.g. on Skylake. See the review for details.

Reviewed by: markj
Approved by: re (kib)
Differential Revision: https://reviews.freebsd.org/D17289

show more ...

Remove a redundant #ifdef _KERNEL.

Submitted by: Sebastian Huber <sebastian.huber@embedded-brains.de>
MFC after: 3 days

inpcb: don't gratuitously defer frees

Don't defer frees in sysctl handlers. It isn't necessary
and it just confuses things.
revert: r333911, r334104, and r334125

Requested by: jtl