block: move AioContext, QEMUTimer, main-loop to libqemuutil

AioContext is fairly self contained, the only dependency is QEMUTimer but
that in turn doesn't need anything else. So move them out of bl

block: move AioContext, QEMUTimer, main-loop to libqemuutil

AioContext is fairly self contained, the only dependency is QEMUTimer but
that in turn doesn't need anything else. So move them out of block-obj-y
to avoid introducing a dependency from io/ to block-obj-y.

main-loop and its dependency iohandler also need to be moved, because
later in this series io/ will call iohandler_get_aio_context.

[Changed copyright "the QEMU team" to "other QEMU contributors" as
suggested by Daniel Berrange and agreed by Paolo.
--Stefan]

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Message-id: 20170213135235.12274-2-pbonzini@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: clean up trace-events files

There are a number of unused trace events that
scripts/cleanup-trace-events.pl finds. The "hw/vfio/pci-quirks.c"
filename was typoed and "qapi/qapi-visit-core.c"

trace: clean up trace-events files

There are a number of unused trace events that
scripts/cleanup-trace-events.pl finds. The "hw/vfio/pci-quirks.c"
filename was typoed and "qapi/qapi-visit-core.c" was missing the qapi/
directory prefix.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20170126171613.1399-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: move hw/xen events to correct subdir

The trace-events for a given source file should generally
always live in the same directory as the source file.

Reviewed-by: Stefan Hajnoczi <stefanha@re

trace: move hw/xen events to correct subdir

The trace-events for a given source file should generally
always live in the same directory as the source file.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-id: 20170125161417.31949-4-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: Add event "guest_cpu_exit"

Signals the hot-unplugging of a virtual (guest) CPU.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 148278748597.1404.10546320797997984932.stgit@f

trace: Add event "guest_cpu_exit"

Signals the hot-unplugging of a virtual (guest) CPU.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 148278748597.1404.10546320797997984932.stgit@fimbulvetr.bsc.es
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace-events: spelling fix

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-id: 20161212221759.28949-1-marcandre.lureau@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redh

trace-events: spelling fix

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-id: 20161212221759.28949-1-marcandre.lureau@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

aio: self-tune polling time

This patch is based on the algorithm for the kvm.ko halt_poll_ns
parameter in Linux. The initial polling time is zero.

If the event loop is woken up within the maximum

aio: self-tune polling time

This patch is based on the algorithm for the kvm.ko halt_poll_ns
parameter in Linux. The initial polling time is zero.

If the event loop is woken up within the maximum polling time it means
polling could be effective, so grow polling time.

If the event loop is woken up beyond the maximum polling time it means
polling is not effective, so shrink polling time.

If the event loop makes progress within the current polling time then
the sweet spot has been reached.

This algorithm adjusts the polling time so it can adapt to variations in
workloads. The goal is to reach the sweet spot while also recognizing
when polling would hurt more than help.

Two new trace events, poll_grow and poll_shrink, are added for observing
polling time adjustment.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20161201192652.9509-13-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

aio: add polling mode to AioContext

The AioContext event loop uses ppoll(2) or epoll_wait(2) to monitor file
descriptors or until a timer expires. In cases like virtqueues, Linux
AIO, and ThreadPoo

aio: add polling mode to AioContext

The AioContext event loop uses ppoll(2) or epoll_wait(2) to monitor file
descriptors or until a timer expires. In cases like virtqueues, Linux
AIO, and ThreadPool it is technically possible to wait for events via
polling (i.e. continuously checking for events without blocking).

Polling can be faster than blocking syscalls because file descriptors,
the process scheduler, and system calls are bypassed.

The main disadvantage to polling is that it increases CPU utilization.
In classic polling configuration a full host CPU thread might run at
100% to respond to events as quickly as possible. This patch implements
a timeout so we fall back to blocking syscalls if polling detects no
activity. After the timeout no CPU cycles are wasted on polling until
the next event loop iteration.

The run_poll_handlers_begin() and run_poll_handlers_end() trace events
are added to aid performance analysis and troubleshooting. If you need
to know whether polling mode is being used, trace these events to find
out.

Note that the AioContext is now re-acquired before disabling notify_me
in the non-polling case. This makes the code cleaner since notify_me
was enabled outside the non-polling AioContext release region. This
change is correct since it's safe to keep notify_me enabled longer
(disabling is an optimization) but potentially causes unnecessary
event_notifer_set() calls. I think the chance of performance regression
is small here.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20161201192652.9509-4-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

memory: Don't use memcpy for ram_device regions

With a vfio assigned device we lay down a base MemoryRegion registered
as an IO region, giving us read & write accessors. If the region
supports mmap

memory: Don't use memcpy for ram_device regions

With a vfio assigned device we lay down a base MemoryRegion registered
as an IO region, giving us read & write accessors. If the region
supports mmap, we lay down a higher priority sub-region MemoryRegion
on top of the base layer initialized as a RAM device pointer to the
mmap. Finally, if we have any quirks for the device (ie. address
ranges that need additional virtualization support), we put another IO
sub-region on top of the mmap MemoryRegion. When this is flattened,
we now potentially have sub-page mmap MemoryRegions exposed which
cannot be directly mapped through KVM.

This is as expected, but a subtle detail of this is that we end up
with two different access mechanisms through QEMU. If we disable the
mmap MemoryRegion, we make use of the IO MemoryRegion and service
accesses using pread and pwrite to the vfio device file descriptor.
If the mmap MemoryRegion is enabled and results in one of these
sub-page gaps, QEMU handles the access as RAM, using memcpy to the
mmap. Using either pread/pwrite or the mmap directly should be
correct, but using memcpy causes us problems. I expect that not only
does memcpy not necessarily honor the original width and alignment in
performing a copy, but it potentially also uses processor instructions
not intended for MMIO spaces. It turns out that this has been a
problem for Realtek NIC assignment, which has such a quirk that
creates a sub-page mmap MemoryRegion access.

To resolve this, we disable memory_access_is_direct() for ram_device
regions since QEMU assumes that it can use memcpy for those regions.
Instead we access through MemoryRegionOps, which replaces the memcpy
with simple de-references of standard sizes to the host memory.

With this patch we attempt to provide unrestricted access to the RAM
device, allowing byte through qword access as well as unaligned
access. The assumption here is that accesses initiated by the VM are
driven by a device specific driver, which knows the device
capabilities. If unaligned accesses are not supported by the device,
we don't want them to work in a VM by performing multiple aligned
accesses to compose the unaligned access. A down-side of this
philosophy is that the xp command from the monitor attempts to use
the largest available access weidth, unaware of the underlying
device. Using memcpy had this same restriction, but at least now an
operator can dump individual registers, even if blocks of device
memory may result in access widths beyond the capabilities of a
given device (RTL NICs only support up to dword).

Reported-by: Thorsten Kohfeldt <thorsten.kohfeldt@gmx.de>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

trace: Add missing execution mode of guest events

Add missing execution mode documentation for the 'guest_cpu_enter' and
'guest_cpu_reset' events.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu

trace: Add missing execution mode of guest events

Add missing execution mode documentation for the 'guest_cpu_enter' and
'guest_cpu_reset' events.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 147566900921.7708.656450813307396468.stgit@fimbulvetr.bsc.es
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: move colo trace events to net/ sub-directory

The colo patch series added various trace events to the top
level trace-events file, despite the files using them being
in a sub-dir.

commit 30

trace: move colo trace events to net/ sub-directory

The colo patch series added various trace events to the top
level trace-events file, despite the files using them being
in a sub-dir.

commit 30656b097e9dd7978d3fe9416cb9f5a421a9e63e
Author: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Date: Tue Sep 27 10:22:34 2016 +0800

filter-rewriter: rewrite tcp packet to keep secondary connection

commit f4b618360e5a81b097e2e35d52011bec3c63af68
Author: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Date: Tue Sep 27 10:22:31 2016 +0800

colo-compare: add TCP, UDP, ICMP packet comparison

We add TCP,UDP,ICMP packet comparison to replace
IP packet comparison. This can increase the
accuracy of the package comparison.
Less checkpoint more efficiency.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>

commit 0682e15b19b2f41c0568142b42518b9471168597
Author: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Date: Tue Sep 27 10:22:30 2016 +0800

colo-compare: introduce packet comparison thread

commit 59509ec16b7ee92b3f8261c554023aa1d3169317
Author: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Date: Tue Sep 27 10:22:27 2016 +0800

net/colo.c: add colo.c to define and handle packet

This moves all events into net/trace-events where they
were supposed to live.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-id: 1475588159-30598-2-git-send-email-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: Document the execution mode of guest events

Explicitly state in which execution mode (user, softmmu, all) are guest
events available for tracing.

Signed-off-by: Lluís Vilanova <vilanova@ac.u

trace: Document the execution mode of guest events

Explicitly state in which execution mode (user, softmmu, all) are guest
events available for tracing.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 147456962135.11114.6146034359114598596.stgit@fimbulvetr.bsc.es
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: Add event "guest_cpu_reset"

Signals the reset of the state a virtual (guest) CPU.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 147428971851.15111.8799439252178273840.stgit

trace: Add event "guest_cpu_reset"

Signals the reset of the state a virtual (guest) CPU.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 147428971851.15111.8799439252178273840.stgit@fimbulvetr.bsc.es
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: Add event "guest_cpu_enter"

Signals the hot-plugging of a new virtual (guest) CPU.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 147428971313.15111.18023030883528426840.stg

trace: Add event "guest_cpu_enter"

Signals the hot-plugging of a new virtual (guest) CPU.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 147428971313.15111.18023030883528426840.stgit@fimbulvetr.bsc.es
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: move hw/virtio/virtio-balloon.c trace points into correct file

The trace points for hw/virtio/virtio-balloon.c were mistakenly put
in the top level trace-events file, instead of util/trace-ev

trace: move hw/virtio/virtio-balloon.c trace points into correct file

The trace points for hw/virtio/virtio-balloon.c were mistakenly put
in the top level trace-events file, instead of util/trace-events in

commit 270ab88f7c1112389a02cee0e3e03b20fcc7547e
Author: Daniel P. Berrange <berrange@redhat.com>
Date: Thu Jun 16 09:39:57 2016 +0100

trace: split out trace events for hw/virtio/ directory

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 1473872624-23285-5-git-send-email-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: move util/qemu-coroutine*.c trace points into correct file

The trace points for util/qemu-coroutine*.c were mistakenly left
in the top level trace-events file, instead of util/trace-events
in

trace: move util/qemu-coroutine*.c trace points into correct file

The trace points for util/qemu-coroutine*.c were mistakenly left
in the top level trace-events file, instead of util/trace-events
in

commit 492bb2dd651e780c0723580880acbedb5661e5ad
Author: Daniel P. Berrange <berrange@redhat.com>
Date: Thu Jun 16 09:39:48 2016 +0100

trace: split out trace events for util/ directory

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 1473872624-23285-3-git-send-email-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

filter-rewriter: rewrite tcp packet to keep secondary connection

We will rewrite tcp packet secondary received and sent.
When colo guest is a tcp server.

Firstly, client start a tcp handshake. the

filter-rewriter: rewrite tcp packet to keep secondary connection

We will rewrite tcp packet secondary received and sent.
When colo guest is a tcp server.

Firstly, client start a tcp handshake. the packet's seq=client_seq,
ack=0,flag=SYN. COLO primary guest get this pkt and mirror(filter-mirror)
to secondary guest, secondary get it use filter-redirector.
Then,primary guest response pkt
(seq=primary_seq,ack=client_seq+1,flag=ACK|SYN).
secondary guest response pkt
(seq=secondary_seq,ack=client_seq+1,flag=ACK|SYN).
In here,we use filter-rewriter save the secondary_seq to it's tcp connection.
Finally handshake,client send pkt
(seq=client_seq+1,ack=primary_seq+1,flag=ACK).
Here,filter-rewriter can get primary_seq, and rewrite ack from primary_seq+1
to secondary_seq+1, recalculate checksum. So the secondary tcp connection
kept good.

When we send/recv packet.
client send pkt(seq=client_seq+1+data_len,ack=primary_seq+1,flag=ACK|PSH).
filter-rewriter rewrite ack and send to secondary guest.

primary guest response pkt
(seq=primary_seq+1,ack=client_seq+1+data_len,flag=ACK)
secondary guest response pkt
(seq=secondary_seq+1,ack=client_seq+1+data_len,flag=ACK)
we rewrite secondary guest seq from secondary_seq+1 to primary_seq+1.
So tcp connection kept good.

In code We use offset( = secondary_seq - primary_seq )
to rewrite seq or ack.
handle_primary_tcp_pkt: tcp_pkt->th_ack += offset;
handle_secondary_tcp_pkt: tcp_pkt->th_seq -= offset;

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>

show more ...

colo-compare: add TCP, UDP, ICMP packet comparison

We add TCP,UDP,ICMP packet comparison to replace
IP packet comparison. This can increase the
accuracy of the package comparison.
Less checkpoint mo

colo-compare: add TCP, UDP, ICMP packet comparison

We add TCP,UDP,ICMP packet comparison to replace
IP packet comparison. This can increase the
accuracy of the package comparison.
Less checkpoint more efficiency.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>

show more ...

colo-compare: introduce packet comparison thread

If primary packet is same with secondary packet,
we will send primary packet and drop secondary
packet, otherwise notify COLO frame to do checkpoint.

colo-compare: introduce packet comparison thread

If primary packet is same with secondary packet,
we will send primary packet and drop secondary
packet, otherwise notify COLO frame to do checkpoint.
If primary packet comes but secondary packet does not,
after REGULAR_PACKET_CHECK_MS milliseconds we set
the primary packet as old_packet,then do a checkpoint.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>

show more ...

net/colo.c: add colo.c to define and handle packet

The net/colo.c is used by colo-compare and filter-rewriter.
this can share common data structure like net packet,
and other functions.

Signed-off-

net/colo.c: add colo.c to define and handle packet

The net/colo.c is used by colo-compare and filter-rewriter.
this can share common data structure like net packet,
and other functions.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>

show more ...

monitor: use qmp_dispatch()

Replace the old manual dispatch and validation code by the generic one
provided by qapi common code.

Note that it is now possible to call the following commands that use

monitor: use qmp_dispatch()

Replace the old manual dispatch and validation code by the generic one
provided by qapi common code.

Note that it is now possible to call the following commands that used to
be disabled by compile-time conditionals:
- dump-skeys
- query-spice
- rtc-reset-reinjection
- query-gic-capabilities

Their fallback functions return an appropriate "feature disabled" error.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20160912091913.15831-16-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

xen: handle inbound migration of VMs without ioreq server pages

VMs created on older versions on Xen will not have been provisioned with
pages to support creation of non-default ioreq servers. In th

xen: handle inbound migration of VMs without ioreq server pages

VMs created on older versions on Xen will not have been provisioned with
pages to support creation of non-default ioreq servers. In this case
the ioreq server API is not supported and QEMU's only option is to fall
back to using the default ioreq server pages as it did prior to
commit 3996e85c ("Xen: Use the ioreq-server API when available").

This patch therefore changes the code in xen_common.h to stop considering
a failure of xc_hvm_create_ioreq_server() as a hard failure but simply
as an indication that the guest is too old to support the ioreq server
API. Instead a boolean is set to cause reversion to old behaviour such
that the default ioreq server is then used.

Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
Signed-off-by: Stefano Stabellini <sstabellini@kernel.org>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>

show more ...

kvm-all: add trace events for kvm irqchip ops

These will help us monitoring irqchip route activities more easily.

Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@re

kvm-all: add trace events for kvm irqchip ops

These will help us monitoring irqchip route activities more easily.

Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

show more ...

trace: [*-user] Add events to trace guest syscalls in syscall emulation mode

Adds two events to trace syscalls in syscall emulation mode (*-user):

* guest_user_syscall: Emitted before the syscall i

trace: [*-user] Add events to trace guest syscalls in syscall emulation mode

Adds two events to trace syscalls in syscall emulation mode (*-user):

* guest_user_syscall: Emitted before the syscall is emulated; contains
the syscall number and arguments.

* guest_user_syscall_ret: Emitted after the syscall is emulated;
contains the syscall number and return value.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
Message-id: 146651712411.12388.10024905980452504938.stgit@fimbulvetr.bsc.es
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: split out trace events for linux-user/ directory

Move all trace-events for files in the linux-user/ directory to
their own file.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Revie

trace: split out trace events for linux-user/ directory

Move all trace-events for files in the linux-user/ directory to
their own file.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Laurent Vivier <laurent@vivier.eu>
Message-id: 1466066426-16657-41-git-send-email-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

trace: split out trace events for qom/ directory

Move all trace-events for files in the qom/ directory to
their own file.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-id: 1466066

trace: split out trace events for qom/ directory

Move all trace-events for files in the qom/ directory to
their own file.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-id: 1466066426-16657-40-git-send-email-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...