Rework per proc and per process time usage accounting

For procs (threads) the accounting happens now lockless by curproc using
a generation counter. Callers need to use tu_enter() and tu_leave() for

Rework per proc and per process time usage accounting

For procs (threads) the accounting happens now lockless by curproc using
a generation counter. Callers need to use tu_enter() and tu_leave() for this.
To read the proc p_tu struct tuagg_get_proc() should be used. It ensures
that the values read is consistent.

For processes only the time of exited threads is accumulated in ps_tu and
to get the proper process time usage tuagg_get_process() needs to be called.
tuagg_get_process() will sum up all procs p_tu plus the ps_tu.

This removes another SCHED_LOCK() dependency. Adjust the code in
exit1() and exit2() to correctly account for the full run time.
For this adjust sched_exit() to do the runtime accounting like it is done
in mi_switch().

OK jca@ dlg@

show more ...

kernel: disable hardclock() on secondary CPUs

There is no useful work left for secondary CPUs to do in hardclock().
Disable cq_hardclock on secondary CPUs and remove the now-unnecessary
early-return

kernel: disable hardclock() on secondary CPUs

There is no useful work left for secondary CPUs to do in hardclock().
Disable cq_hardclock on secondary CPUs and remove the now-unnecessary
early-return from hardclock().

This change reduces every system's normal clock interrupt rate by
(HZ - HZ/10) per secondary CPU. For example, an 8-core machine
with a HZ=100 kernel should see its clock interrupt rate drop from
~1600 to ~960.

Thread: https://marc.info/?l=openbsd-tech&m=170750140915898&w=2

ok kettenis@

show more ...

dt(4): move interval/profile entry points to dedicated clockintr callback

To improve the utility of dt(4)'s interval and profile probes we need
to move the probe entry points from the fixed-frequenc

dt(4): move interval/profile entry points to dedicated clockintr callback

To improve the utility of dt(4)'s interval and profile probes we need
to move the probe entry points from the fixed-frequency hardclock() to
a dedicated clock interrupt callback so that the probes can fire at
arbitrary frequencies.

- Remove entry points for interval/profile probes from hardclock().

- Merge dt_prov_profile_enter(), dt_prov_interval_enter(), and
dt_prov_profile_fire() into one function, dt_clock(). This is
the now-unified callback for interval/profile probes. dt_clock()
will consume multiple events during a single execution if it is
delayed, but on platforms with high quality interrupt clocks this
should be rare.

- Each struct dt_pcb gets its own clockintr handle, dp_clockintr.

- In struct dt_pcb, replace dp_maxtick/dp_nticks with dp_nsecs,
the PCB's sampling period. Aynchronous probes must initialize
dp_nsecs to a non-zero value during dtpv_alloc().

- In struct dt_pcb, replace dp_cpuid with dp_cpu so that
dt_ioctl_record_start() knows where to bind the PCB's
dp_clockintr.

- dt_ioctl_record_start() binds, staggers, and starts all
interval/profile PCBs on the given dt_softc. Each dp_clockintr
is given a reference to its enclosing PCB so that dt_clock()
doesn't need to search for it. The staggering sort-of simulates
the current behavior under hardclock().

- dt_ioctl_record_stop() unbinds all interval/profile PCBs. The
CL_BARRIER ensures that dp_clockintr's PCB reference is not in
use by dt_clock() so that the PCB may be safely freed upon
return from dt_ioctl_record_stop(). Blocking while holding
dt_lock is not ideal, but in practice blocking in this spot is
rare and dt_clock() completes quickly on all but the oldest
hardware. An extremely unlucky thread could block for every
interval/profile PCB on the softc, but this is implausible.

DT_FA_PROFILE values are up-to-date for amd64, i386, and macppc.
Somebody with the right hardware needs to check-and-maybe-fix the
values on octeon, powerpc64, and sparc64.

Joint effort with mpi@.

Thread: https://marc.info/?l=openbsd-tech&m=170629371821879&w=2

ok mpi@

show more ...

clockintr: move callback-specific API behaviors to "clockrequest" namespace

The API's behavior when invoked from a callback function is impossible
to document. Move the special behavior into a dist

clockintr: move callback-specific API behaviors to "clockrequest" namespace

The API's behavior when invoked from a callback function is impossible
to document. Move the special behavior into a distinct namespace,
"clockrequest".

- Add a 'struct clockrequest'. Basically a stripped-down 'struct clockintr'
for exclusive use during clockintr_dispatch().
- In clockintr_queue, replace the "cq_shadow" clockintr with a "cq_request"
clockrequest. They serve the same purpose.
- CLST_SHADOW_PENDING -> CR_RESCHEDULE; different namespace, same meaning.
- CLST_IGNORE_SHADOW -> CLST_IGNORE_REQUEST; same meaning.
- Move shadow branch in clockintr_advance() to clockrequest_advance().
- clockintr_request_random() becomes clockrequest_advance_random().
- Delete dead shadow branches in clockintr_cancel(), clockintr_schedule().
- Callback functions now get a clockrequest pointer instead of a special
clockintr pointer: update all prototypes, callers.

No functional change intended.

show more ...

kernel: expand fixed clock interrupt periods to 64-bit values

Technically, all the current fixed clock interrupt periods fit within
an unsigned 32-bit value. But 32-bit multiplication is an acciden

kernel: expand fixed clock interrupt periods to 64-bit values

Technically, all the current fixed clock interrupt periods fit within
an unsigned 32-bit value. But 32-bit multiplication is an accident
waiting to happen. So, expand the fixed periods for hardclock,
statclock, profclock, and roundrobin to 64-bit values.

One exception: statclock_mask remains 32-bit because random(9) yields
32-bit values. Update the initclocks() comment to make it clear that
this is not an accident.

show more ...

clockintr: move hz(9)-based initialization out to initclocks()

To separate the hardclock from the clock interrupt subsystem we'll
need to move all related state out first.

hz(9) is set when we retu

clockintr: move hz(9)-based initialization out to initclocks()

To separate the hardclock from the clock interrupt subsystem we'll
need to move all related state out first.

hz(9) is set when we return from cpu_initclocks(), so it's safe to
move hardclock_period and roundrobin_period initialization out into
initclocks(). Move hardclock_period itself out into kern_clock.c
alongside the statclock variables.

show more ...

clockintr, statclock: eliminate clockintr_statclock() wrapper

- Move remaining statclock variables from kern_clockintr.c to
kern_clock.c. Move statclock variable initialization from
clockintr_i

clockintr, statclock: eliminate clockintr_statclock() wrapper

- Move remaining statclock variables from kern_clockintr.c to
kern_clock.c. Move statclock variable initialization from
clockintr_init() into initclocks().

- Change statclock() prototype to make it a legal clockintr
callback function and establish the handle with statclock()
instead clockintr_statclock().

- Merge the contents of clockintr_statclock() into statclock().
statclock() can now reschedule itself and handles multiple
expirations transparently.

- Make statclock_avg visible from sys/systm.h so that clockintr_cpu_init()
can use it to advance the statclock across suspend/hibernate.

Thread: https://marc.info/?l=openbsd-tech&m=169428749720476&w=2

show more ...

clockintr: replace CL_RNDSTAT with global variable statclock_is_randomized

In order to separate the statclock from the clock interrupt subsystem
we need to move all statclock state out into the broa

clockintr: replace CL_RNDSTAT with global variable statclock_is_randomized

In order to separate the statclock from the clock interrupt subsystem
we need to move all statclock state out into the broader kernel.

Start by replacing the CL_RNDSTAT flag with a new global variable,
"statclock_is_randomized", in kern_clock.c. Update all clockintr_init()
callers to set the boolean instead of passing the flag.

Thread: https://marc.info/?l=openbsd-tech&m=169428749720476&w=2

show more ...

kernel: remove schedhz

Now that alpha no longer sets schedhz, schedhz is a dead variable.
Remove it.

For now, leave the schedclock() call in place in statclock(). It
still runs at its default rate

kernel: remove schedhz

Now that alpha no longer sets schedhz, schedhz is a dead variable.
Remove it.

For now, leave the schedclock() call in place in statclock(). It
still runs at its default rate of (stathz / 4).

Part of mpi@'s WIP scheduler patch. Suggested by mpi@.

Thread: https://marc.info/?l=openbsd-tech&m=169419781317781&w=2

ok mpi@

show more ...

all platforms: separate cpu_initclocks() from cpu_startclock()

To give the primary CPU an opportunity to perform clock interrupt
preparation in a machine-independent manner we need to separate the
"

all platforms: separate cpu_initclocks() from cpu_startclock()

To give the primary CPU an opportunity to perform clock interrupt
preparation in a machine-independent manner we need to separate the
"initialization" parts of cpu_initclocks() from the "start the clock
interrupt" parts. Currently, cpu_initclocks() does everything all at
once, so there is no space for this MI setup.

Many platforms have more-or-less already done this separation by
implementing a separate routine named "cpu_startclock()". This patch
promotes cpu_startclock() from de facto standard to mandatory API.

- Prototype cpu_startclock() in sys/systm.h alongside cpu_initclocks().
The separation of responsibility between the two routines is a bit
fuzzy but the basic guidelines are as follows:

+ cpu_initclocks() must initialize hz, stathz, and profhz, and call
clockintr_init().

+ cpu_startclock() must call clockintr_cpu_init() and start the clock
interrupt cycle on the calling CPU.

These guidelines will shift in the future, but that's the way things
stand as of *this* commit.

- In initclocks(): first call cpu_initclocks(), then do MI setup, and
last call cpu_startclock().

- On platforms where cpu_startclock() already exists: don't call
cpu_startclock() from cpu_initclocks() anymore.

- On platforms where cpu_startclock() doesn't yet exist: implement it.
Usually this is as simple as dividing cpu_initclocks() in two.

Tested on amd64 (i8254, lapic), arm64, i386 (i8254, lapic), macppc,
mips64/octeon, and sparc64. Tested on arm/armv7 (agtimer(4)) by
phessler@ and jmatthew@. Tested on m88k/luna88k by aoyama@. Tested
on powerpc64 by gkoehler@ and mlarkin@. Tested on riscv64 by
jmatthew@.

Thread: https://marc.info/?l=openbsd-tech&m=169195251322149&w=2

show more ...

avoid an ifdef in hardclock()
ok miod@ cheloha@

Repair compilability for non-MULTIPROCESSOR kernels.

hardclock(9), roundrobin: make roundrobin() an independent clock interrupt

- Remove the roundrobin() call from hardclock(9).

- Revise roundrobin() to make it a valid clock interrupt callback.
It

hardclock(9), roundrobin: make roundrobin() an independent clock interrupt

- Remove the roundrobin() call from hardclock(9).

- Revise roundrobin() to make it a valid clock interrupt callback.
It is still periodic and it still runs at one tenth of the hardclock
frequency.

- Account for multiple expirations in roundrobin(): if two or more
roundrobin periods have elapsed, set SPCF_SHOULDYIELD on the running
thread immediately to simulate normal behavior.

- Each schedstate_percpu has its own roundrobin() handle, spc_roundrobin.
spc_roundrobin is started/advanced during clockintr_cpu_init().
Intervals elapsed across suspend/resume are discarded.

- rrticks_init and schedstate_percpu.spc_rrticks are now useless:
delete them.

Tweaked by mpi@. With input from mpi@ and claudio@.

Thread: https://marc.info/?l=openbsd-tech&m=169127381314651&w=2

ok mpi@ claudio@

show more ...

hardclock(9): move setitimer(2) code into itimer_update()

- Move the setitimer(2) code responsible for updating the ITIMER_VIRTUAL
and ITIMER_PROF timers from hardclock(9) into a new clock interru

hardclock(9): move setitimer(2) code into itimer_update()

- Move the setitimer(2) code responsible for updating the ITIMER_VIRTUAL
and ITIMER_PROF timers from hardclock(9) into a new clock interrupt
routine, itimer_update(). itimer_update() is periodic and runs at the
same frequency as the hardclock.

+ Revise itimerdecr() to run within itimer_mtx instead of entering
and leaving it.

- Each schedstate_percpu has its own itimer_update() handle, spc_itimer.
A new scheduler flag, SPCF_ITIMER, indicates whether spc_itimer was
started during the last mi_switch() and needs to be stopped during the
next mi_switch() or sched_exit().

- A new per-process flag, PS_ITIMER, indicates whether ITIMER_VIRTUAL
and/or ITIMER_PROF are running. Checking the flag is easier than
entering itimer_mtx to check process.ps_timer[]. The flag is set
and cleared in a new helper function, process_reset_itimer_flag().

- In setitimer(), call need_resched() when the state of ITIMER_VIRTUAL
or ITIMER_PROF is changed to force an mi_switch() and update
spc_itimer.

claudio@ notes that ITIMER_PROF could be implemented as a high-res
timer using the thread's execution time as a guide for when to
interrupt the process and assert SIGPROF. This would probably work
really well in single-threaded processes. ITIMER_VIRTUAL would be
more difficult to make high-res, though, as you need to exclude time
spent in the kernel.

Tested on powerpc64 by gkoehler@. With input from claudio@.

Thread: https://marc.info/?l=openbsd-tech&m=169038818517101&w=2

ok claudio@

show more ...

Don't force early wrap around for jiffies.

In inteldrm the function intel_dp_wait_source_oui() can be called before
last_oui_write is set and so the code requires a positive initial jiffies
value. O

Don't force early wrap around for jiffies.

In inteldrm the function intel_dp_wait_source_oui() can be called before
last_oui_write is set and so the code requires a positive initial jiffies
value. On linux this is the case for 64bit systems (but not for 32bit) and
because of this idiosyncracy this bug was introduced in upstream intel code.
Instead of adding another OpenBSD specific quirk to drm code alter our
jiffies initalisation.

Systems affected are at least "ALDERLAKE_P, gen 12" and "TIGERLAKE, gen 12"
Suggested fix by kettenis@
OK jsg@

show more ...

statclock: move profil(2), GPROF code to profclock(), gmonclock()

This patch isolates profil(2) and GPROF from statclock(). Currently,
statclock() implements both profil(2) and GPROF through a comp

statclock: move profil(2), GPROF code to profclock(), gmonclock()

This patch isolates profil(2) and GPROF from statclock(). Currently,
statclock() implements both profil(2) and GPROF through a complex
mechanism involving both platform code (setstatclockrate) and the
scheduler (pscnt, psdiv, and psratio). We have a machine-independent
interface to the clock interrupt hardware now, so we no longer need to
do it this way.

- Move profil(2)-specific code from statclock() to a new clock
interrupt callback, profclock(), in subr_prof.c. Each
schedstate_percpu has its own profclock handle. The profclock is
enabled/disabled for a given CPU when it is needed by the running
thread during mi_switch() and sched_exit().

- Move GPROF-specific code from statclock() to a new clock interrupt
callback, gmonclock(), in subr_prof.c. Where available, each cpu_info
has its own gmonclock handle . The gmonclock is enabled/disabled for
a given CPU via sysctl(2) in prof_state_toggle().

- Both profclock() and gmonclock() have a fixed period, profclock_period,
that is initialized during initclocks().

- Export clockintr_advance(), clockintr_cancel(), clockintr_establish(),
and clockintr_stagger() via <sys/clockintr.h>. They have external
callers now.

- Delete pscnt, psdiv, psratio. From schedstate_percpu, also delete
spc_pscnt and spc_psdiv. The statclock frequency is not dynamic
anymore so these variables are now useless.

- Delete code/state related to the dynamic statclock frequency from
kern_clockintr.c. The statclock frequency can still be pseudo-random,
so move the contents of clockintr_statvar_init() into clockintr_init().

With input from miod@, deraadt@, and claudio@. Early revisions
cleaned up by claudio. Early revisions tested by claudio@. Tested by
cheloha@ on amd64, arm64, macppc, octeon, and sparc64 (sun4v).
Compile- and boot- tested on i386 by mlarkin@. riscv64 compilation
bugs found by mlarkin@. Tested on riscv64 by jca@. Tested on
powerpc64 by gkoehler@.

show more ...

addupc_intr: support adding multiple profiling ticks at once

Add a third parameter to addupc_intr(), "u_long nticks". This will
allow us to credit more than one profiling tick to the thread at once

addupc_intr: support adding multiple profiling ticks at once

Add a third parameter to addupc_intr(), "u_long nticks". This will
allow us to credit more than one profiling tick to the thread at once.
Should be useful in the unusual case where the clock interrupt is
masked for an extended period.

show more ...

initclocks: don't reinitialize ticks, jiffies at runtime

Various drivers use ticks/jiffies before initclocks(). It isn't
generally safe to reinitialize them at runtime. Hoist the conditional
defin

initclocks: don't reinitialize ticks, jiffies at runtime

Various drivers use ticks/jiffies before initclocks(). It isn't
generally safe to reinitialize them at runtime. Hoist the conditional
definition of HZ from param.c into sys/kernel.h so we can see it from
kern_clock.c and statically initialize ticks/jiffies to the desired
offset.

With this change, timeouts scheduled before initclocks() do not all
fire immediately during the first softclock().

With input from kettenis@.

Link: https://marc.info/?l=openbsd-tech&m=167753870803378&w=2

show more ...

kernel: stathz is always non-zero after cpu_initclocks()

Now that the clockintr switch is complete, cpu_initclocks() always
initializes stathz to a non-zero value. We don't call statclock()
from ha

kernel: stathz is always non-zero after cpu_initclocks()

Now that the clockintr switch is complete, cpu_initclocks() always
initializes stathz to a non-zero value. We don't call statclock()
from hardclock(9) anymore and, more broadly, we don't need to test
whether stathz is non-zero before using it.

With input from kettenis@.

Link: https://marc.info/?l=openbsd-tech&m=167434223309668&w=2

ok kettenis@ miod@

show more ...

remove unneeded includes in sys/kern
ok mpi@ miod@

Remove trailing whitespace. No code change.

unifdef PROC_PC
ok guenther@ rob@

kernel, sysctl(8): remove dead variable: tickadj

The global "tickadj" variable is a remnant of the old NTP adjustment
code we used in the kernel before the current timecounter subsystem
was imported

kernel, sysctl(8): remove dead variable: tickadj

The global "tickadj" variable is a remnant of the old NTP adjustment
code we used in the kernel before the current timecounter subsystem
was imported from FreeBSD circa 2004 or 2005.

Fifteen years hence it is completely vestigial and we can remove it.
We probably should have removed it long ago but I guess it slipped
through the cracks. FreeBSD removed it in 2002:

https://cgit.freebsd.org/src/commit/?id=e1d970f1811e5e1e9c912c032acdcec6521b2a6d

NetBSD and DragonflyBSD can probably remove it, too.

We export tickadj via the kern.clockrate sysctl(2), so update sysctl.2
and sysctl(8) accordingly. Hypothetically this change could break
someone's sysctl(8) parsing script. I don't think that's very likely.

ok mvs@

show more ...

Import dt(4) a driver and framework for Dynamic Profiling.

The design is fairly simple: events, in the form of descriptors on a
ring, are being produced in any kernel context and being consumed by
a

Import dt(4) a driver and framework for Dynamic Profiling.

The design is fairly simple: events, in the form of descriptors on a
ring, are being produced in any kernel context and being consumed by
a userland process reading /dev/dt.

Code and hooks are all guarded under '#if NDT > 0' so this commit
shouldn't introduce any change as long as dt(4) is disable in GENERIC.

ok kettenis@, visa@, jasper@, deraadt@

show more ...

softclock: move softintr registration/scheduling into timeout module

softclock() is scheduled from hardclock(9) because long ago callouts were
processed from hardclock(9) directly. The introduction

softclock: move softintr registration/scheduling into timeout module

softclock() is scheduled from hardclock(9) because long ago callouts were
processed from hardclock(9) directly. The introduction of timeout(9) circa
2000 moved all callout processing into a dedicated module, but the softclock
scheduling stayed behind in hardclock(9).

We can move all the softclock() "stuff" into the timeout module to make
kern_clock.c a bit cleaner. Neither initclocks() nor hardclock(9) need
to "know" about softclock(). The initial softclock() softintr registration
can be done from timeout_proc_init() and softclock() can be scheduled
from timeout_hardclock_update().

ok visa@

show more ...