Fix annotation of smr_tqh_last

smr_tqh_last is not intended for lockless use with SMR_PTR_GET().

Fix mistypes within sys/smr.h

LIST_END -> SMR_LIST_END
TAILQ_END -> SMR_TAILQ_END

ok visa@

Expose SMR list and pointer macros to userspace. This enables the use
of SMR lists in userspace-visible parts of system headers. In addition,
the macros allow libkvm to examine SMR data structures.

Expose SMR list and pointer macros to userspace. This enables the use
of SMR lists in userspace-visible parts of system headers. In addition,
the macros allow libkvm to examine SMR data structures.

Initial diff by and OK claudio@

show more ...

Mention tail queue in comments.

Implement a SMR TAILQ implementation. The only operations which can be used
in SMR read-side critical sections are SMR_TAILQ_FOREACH(), SMR_TAILQ_FIRST()
and SMR_TAILQ_NEXT(). Most notably the last e

Implement a SMR TAILQ implementation. The only operations which can be used
in SMR read-side critical sections are SMR_TAILQ_FOREACH(), SMR_TAILQ_FIRST()
and SMR_TAILQ_NEXT(). Most notably the last element can not be accessed
in a read-side critical section.
OK visa@

show more ...

Adjust SMR_ASSERT_CRITICAL() and SMR_ASSERT_NONCRITICAL() so that the
panic message shows the actual code location of the assert. Do this by
moving the assert logic inside the macros.

Prompted by an

Adjust SMR_ASSERT_CRITICAL() and SMR_ASSERT_NONCRITICAL() so that the
panic message shows the actual code location of the assert. Do this by
moving the assert logic inside the macros.

Prompted by and OK claudio@
OK mpi@

show more ...

Start the SMR thread when all CPUs are ready for scheduling. This
prevents the appearance of a "smr: dispatch took N seconds" message
during boot when there is an early smr_call(). Such a call can ha

Start the SMR thread when all CPUs are ready for scheduling. This
prevents the appearance of a "smr: dispatch took N seconds" message
during boot when there is an early smr_call(). Such a call can happen
with mfii(4). The initial dispatch cannot make progress until
smr_grace_wait() can visit all CPUs.

This fix is essentially a hack. It makes use of the fact that there
is no hard guarantee on how quickly the callback of smr_call() gets
invoked. It is assumed that the SMR call backlog does not grow large
during boot.

An alternative fix is to make smr_grace_wait() skip secondary CPUs
until they have been started. However, this could break if the spinup
logic of secondary CPUs was changed.

Delayed SMR dispatch reported and fix tested by Hrvoje Popovski
Discussed with and OK kettenis@, claudio@

show more ...

Use READ_ONCE() and WRITE_ONCE() in <sys/smr.h>.

OK guenther@

Introduce safe memory reclamation, a mechanism for reclaiming shared
objects that readers can access without locking. This provides a basis
for read-copy-update operations.

Readers access SMR-protec

Introduce safe memory reclamation, a mechanism for reclaiming shared
objects that readers can access without locking. This provides a basis
for read-copy-update operations.

Readers access SMR-protected shared objects inside SMR read-side
critical section where sleeping is not allowed. To reclaim
an SMR-protected object, the writer has to ensure mutual exclusion of
other writers, remove the object's shared reference and wait until
read-side references cannot exist any longer. As an alternative to
waiting, the writer can schedule a callback that gets invoked when
reclamation is safe.

The mechanism relies on CPU quiescent states to determine when an
SMR-protected object is ready for reclamation.

The <sys/smr.h> header additionally provides an implementation of
singly- and doubly-linked lists that can be used together with SMR.
These lists allow lockless read access with a concurrent writer.

Discussed with many
OK mpi@ sashan@

show more ...