Use __LP64__ to detect presence of suword64() to fix linking and
loading of the LinuxKPI on 32-bit platforms.

Reported by: lwhsu @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement more userspace memory access functions in the LinuxKPI.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Add basic support for VIMAGE to the LinuxKPI and ibcore.

Support is implemented by mapping Linux's "struct net" into FreeBSD's
"struct vnet". Currently only vnet0 is supported by ibcore.

MFC after:

Add basic support for VIMAGE to the LinuxKPI and ibcore.

Support is implemented by mapping Linux's "struct net" into FreeBSD's
"struct vnet". Currently only vnet0 is supported by ibcore.

MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Set "current" pointer for LinuxKPI interrupts and timer callbacks.

MFC after: 1 week
Sponsored by: Mellanox Technologies

LinuxKPI workqueue cleanup.

This change makes the workqueue implementation behave more like in
Linux, both functionality wise and structure wise.

All workqueue code has been moved to linux_work.c

LinuxKPI workqueue cleanup.

This change makes the workqueue implementation behave more like in
Linux, both functionality wise and structure wise.

All workqueue code has been moved to linux_work.c

Add an atomic based statemachine to the work_struct to ensure proper
operation. Prior to this change struct_work was directly mapped to a
FreeBSD task. When a taskqueue has multiple threads the same task may
end up being executed on more than one worker thread simultaneously.
This might cause problems with code coming from Linux, which expects
serial behaviour, similar to Linux tasklets.

Move all global workqueue function names into the linux_xxx domain to
avoid symbol name clashes in the future.

Implement a few more workqueue related functions and macros.

Create two multithreaded taskqueues for the LinuxKPI during module
load, one for time-consuming callbacks and one for non-time consuming
callbacks.

MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Implement add_timer_on() function in the LinuxKPI.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK.

- Optimise the RCU implementation to not allocate and free
ck_epoch_records during runtime. Instead allocate two sets of
ck_epo

Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK.

- Optimise the RCU implementation to not allocate and free
ck_epoch_records during runtime. Instead allocate two sets of
ck_epoch_records per CPU for general purpose use. The first set is
only used for reader locks and the second set is only used for
synchronization and barriers and is protected with a regular mutex to
prevent simultaneous issues.

- Move the task structure away from the rcu_head structure and into
the per-CPU structures. This allows the size of the rcu_head structure
to be reduced down to the size of two pointers.

- Fix a bug where the linux_rcu_barrier() function only waited for one
per-CPU epoch record to be completed instead of all.

- Use a critical section or a mutex to protect ck_epoch_begin() and
ck_epoch_end() depending on RCU or SRCU type. All the ck_epoch_xxx()
functions, except ck_epoch_register(), ck_epoch_unregister() and
ck_epoch_recycle() are not re-entrant and needs a critical section or
a mutex to operate in the LinuxKPI, after inspecting the CK
implementation of the above mentioned functions. The simultaneous
issues arise from per-CPU epoch records being shared between multiple
threads depending on the amount of taskswitching and how many threads
are involved with the RCU and SRCU operations.

- Properly free all epoch records by using safe list traversal at
LinuxKPI module unload. It turns out the ck_epoch_recycle() always
have the records on an internal list and use a flag in the epoch
record to track allocated and free entries. This would lead to use
after free during module unload.

- Remove redundant synchronize_rcu() call from the
linux_compat_uninit() function. Let the linux_rcu_runtime_uninit()
function do the final rcu_barrier() instead.

MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Replace dummy implementation of RCU in the LinuxKPI with one based on
the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx()
functions into own rculist.h header file.

Obtained from: k

Replace dummy implementation of RCU in the LinuxKPI with one based on
the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx()
functions into own rculist.h header file.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Make the LinuxKPI task struct persistent accross system calls.

A set of helper functions have been added to manage the life of the
LinuxKPI task struct. When an external system call or task is invok

Make the LinuxKPI task struct persistent accross system calls.

A set of helper functions have been added to manage the life of the
LinuxKPI task struct. When an external system call or task is invoked,
a check is made to create the task struct by demand. A thread
destructor callback is registered to free the task struct when a
thread exits to avoid memory leaks.

This change lays the ground for emulating the Linux kernel more
closely which is a dependency by the code using the LinuxKPI APIs.

Add new dedicated td_lkpi_task field has been added to struct thread
instead of abusing td_retval[1].

Fix some header file inclusions to make LINT kernel build properly
after this change.

Bump the __FreeBSD_version to force a rebuild of all kernel modules.

MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Implement register and unregister chrdev in the LinuxKPI.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

The SCHEDULER_STOPPED() macro already contains a predict false statement.
Remove superfluous unlikely() wrapper.

Suggested by: glebius
MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement Linux module parameters as read-only tunable SYSCTLs.

Bool module parameters are no longer supported, because there is no
equivalent in FreeBSD.

There are two macros available which contr

Implement Linux module parameters as read-only tunable SYSCTLs.

Bool module parameters are no longer supported, because there is no
equivalent in FreeBSD.

There are two macros available which control the behaviour of the
LinuxKPI module parameters:

- LINUXKPI_PARAM_PARENT allows the consumer to set the SYSCTL parent
where the modules parameters will be created.

- LINUXKPI_PARAM_PREFIX defines a parameter name prefix, which is
added to all created module parameters.

Sponsored by: Mellanox Technologies
MFC after: 1 week

show more ...

Add checks for SCHEDULER_STOPPED() so that code using the LinuxKPI can
run after a panic(). This for example allows a LinuxKPI based graphics
stack to receive prints during a panic.

Obtained from: k

Add checks for SCHEDULER_STOPPED() so that code using the LinuxKPI can
run after a panic(). This for example allows a LinuxKPI based graphics
stack to receive prints during a panic.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Set "current" for all PCI enumeration callbacks.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Add more list_xxx() functions to the LinuxKPI.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Properly implement "cpu_has_clflush" macro.

Suggested by: kib, jhb
MFC after: 1 week
Sponsored by: Mellanox Technologies

Fix handling of IOCTLs in the LinuxKPI.

Linux requires that all IOCTL data resides in userspace. FreeBSD
always moves the main IOCTL structure into a kernel buffer before
invoking the IOCTL handler

Fix handling of IOCTLs in the LinuxKPI.

Linux requires that all IOCTL data resides in userspace. FreeBSD
always moves the main IOCTL structure into a kernel buffer before
invoking the IOCTL handler and then copies it back into userspace,
before returning. Hide this difference in the "linux_copyin()" and
"linux_copyout()" functions by remapping userspace addresses in the
range from 0x10000 to 0x20000, to the kernel IOCTL data buffer.

It is assumed that the userspace code, data and stack segments starts
no lower than memory address 0x400000, which is also stated by "man 1
ld", which means any valid userspace pointer can be passed to regular
LinuxKPI handled IOCTLs.

Bump the FreeBSD version to force recompilation of all kernel modules.

Discussed with: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Create a dummy "task_struct" on the stack which is returned by
"current" inside all LinuxKPI file operation callbacks. The "current"
is frequently used for various debug prints, printing the thread n

Create a dummy "task_struct" on the stack which is returned by
"current" inside all LinuxKPI file operation callbacks. The "current"
is frequently used for various debug prints, printing the thread name
and thread ID for example.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Return a proper error code instead of panicing when an I/O vector
having the wrong number of entries is detected.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Fix file polling bug.

Ensure the actual poll result is returned by the "linux_file_poll()"
function instead of zero which means no data is available.

MFC after: 3 days
Sponsored by: Mellanox Techno

Fix file polling bug.

Ensure the actual poll result is returned by the "linux_file_poll()"
function instead of zero which means no data is available.

MFC after: 3 days
Sponsored by: Mellanox Technologies

show more ...

Add wait_event_interruptible_timeout to linuxkpi.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Reviewed by: hselasky@
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.fr

Add wait_event_interruptible_timeout to linuxkpi.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Reviewed by: hselasky@
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D5776

show more ...

As <machine/pmap.h> is included from <vm/pmap.h>, there is no need to
include it explicitly when <vm/pmap.h> is already included.

Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd

As <machine/pmap.h> is included from <vm/pmap.h>, there is no need to
include it explicitly when <vm/pmap.h> is already included.

Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D5373

show more ...

Initialize vm_page_prot to VM_MEMATTR_DEFAULT instead of 0.

If a driver's Linux mmap callback passed vm_page_prot through unchanged,
then linux_dev_mmap_single() would try to apply whatever VM_MEMAT

Initialize vm_page_prot to VM_MEMATTR_DEFAULT instead of 0.

If a driver's Linux mmap callback passed vm_page_prot through unchanged,
then linux_dev_mmap_single() would try to apply whatever VM_MEMATTR_xxx
value 0 is to the mapping. On x86, VM_MEMATTR_DEFAULT is the PAT value
for write-back (WB) which is 6, while 0 maps to the PAT value for
uncacheable (UC). Thus, any mmap request that did not explicitly set
page_prot was tried to map memory as UC triggering the warning in
sg_pager_getpages().

Tested by: np
Reported by: Krishnamraju Eraparaju @ Chelsio
MFC after: 3 days
Sponsored by: Chelsio Communications

show more ...

LinuxKPI style changes:
- Properly prefix internal functions with "linux_" instead of only a
single underscore to avoid future namespace collisions.
- Make some functions global instead of inline t

LinuxKPI style changes:
- Properly prefix internal functions with "linux_" instead of only a
single underscore to avoid future namespace collisions.
- Make some functions global instead of inline to ease debugging and
to avoid unnecessary code duplication.
- Remove no longer existing kthread_create() function's prototype.

MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Implement RCU mechanism using shared exclusive locks.

MFC after: 1 week
Sponsored by: Mellanox Technologies