json: Tighten and simplify qstring_from_escaped_str()'s loop

Simplify loop control, and assert that the string ends with the
appropriate quote (the lexer ensures it does).

Signed-off-by: Markus Arm

json: Tighten and simplify qstring_from_escaped_str()'s loop

Simplify loop control, and assert that the string ends with the
appropriate quote (the lexer ensures it does).

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-21-armbru@redhat.com>

show more ...

json: Revamp lexer documentation

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-20-armbru@redhat.com>

json: Reject unescaped control characters

Fix the lexer to reject unescaped control characters in JSON strings,
in accordance with RFC 8259 "The JavaScript Object Notation (JSON)
Data Interchange Fo

json: Reject unescaped control characters

Fix the lexer to reject unescaped control characters in JSON strings,
in accordance with RFC 8259 "The JavaScript Object Notation (JSON)
Data Interchange Format".

Bonus: we now recover more nicely from unclosed strings. E.g.

{"one: 1}\n{"two": 2}

now recovers cleanly after the newline, where before the lexer
remained confused until the next unpaired double quote or lexical
error.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-19-armbru@redhat.com>

show more ...

json: Fix lexer to include the bad character in JSON_ERROR token

json_lexer[] maps (lexer state, input character) to the new lexer
state. The input character is consumed unless the new state is
ter

json: Fix lexer to include the bad character in JSON_ERROR token

json_lexer[] maps (lexer state, input character) to the new lexer
state. The input character is consumed unless the new state is
terminal and the input character doesn't belong to this token,
i.e. the state transition uses look-ahead. When this is the case,
input character '\0' would result in the same state transition.
TERMINAL_NEEDED_LOOKAHEAD() exploits this.

Except this is wrong for transitions to IN_ERROR. There, the
offending input character is in fact consumed: case IN_ERROR returns.
It isn't added to the JSON_ERROR token, though.

Fix that by making TERMINAL_NEEDED_LOOKAHEAD() return false for
transitions to IN_ERROR.

There's a slight complication. json_lexer_flush() passes input
character '\0' to flush an incomplete token. If this results in
JSON_ERROR, we'd now add the '\0' to the token. Suppress that.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-18-armbru@redhat.com>

show more ...

qobject: qobject_from_jsonv() is dangerous, hide it away

qobject_from_jsonv() takes ownership of %p arguments. On failure, we
can't generally know whether we failed before or after %p, so
ownership

qobject: qobject_from_jsonv() is dangerous, hide it away

qobject_from_jsonv() takes ownership of %p arguments. On failure, we
can't generally know whether we failed before or after %p, so
ownership becomes indeterminate. To avoid leaks, callers passing %p
must terminate on error, e.g. by passing &error_abort. Trap for the
unwary; document and give the function internal linkage.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180806065344.7103-11-armbru@redhat.com>

show more ...

qobject: New qobject_from_vjsonf_nofail(), qdict_from_vjsonf_nofail()

Every printf()-like function sooner or later needs its vprintf()-like
buddy. The next commit will need qobject_from_jsonf_nofai

qobject: New qobject_from_vjsonf_nofail(), qdict_from_vjsonf_nofail()

Every printf()-like function sooner or later needs its vprintf()-like
buddy. The next commit will need qobject_from_jsonf_nofail()'s buddy,
and qdict_from_jsonf_nofail()'s buddy will be used later in this
series. Add both.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180806065344.7103-8-armbru@redhat.com>

show more ...

qobject: Replace qobject_from_jsonf() by qobject_from_jsonf_nofail()

Commit ab45015a968 "qobject: Let qobject_from_jsonf() fail instead of
abort" fails to accomplish its stated aim: the function can

qobject: Replace qobject_from_jsonf() by qobject_from_jsonf_nofail()

Commit ab45015a968 "qobject: Let qobject_from_jsonf() fail instead of
abort" fails to accomplish its stated aim: the function can still
abort due to its use of &error_abort.

Its rationale for letting it fail is that all remaining users cope
fine with failure. Well, they're just fine with aborting, too; it's
what they do on failure.

Simply reverting the broken commit would bring back the unfortunate
asymmetry between qobject_from_jsonf() and qobject_from_jsonv(): one
aborts, the other returns null. So also rename it to
qobject_from_jsonf_nofail().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180806065344.7103-7-armbru@redhat.com>

show more ...

qdict: Make qdict_extract_subqdict() accept dst = NULL

This function extracts all options from a QDict starting with a
certain prefix and puts them in a new QDict.

We'll have a couple of cases wher

qdict: Make qdict_extract_subqdict() accept dst = NULL

This function extracts all options from a QDict starting with a
certain prefix and puts them in a new QDict.

We'll have a couple of cases where we simply want to discard those
options instead of copying them, and that's what this patch does.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

qstring: Move qstring_from_substr()'s @end one to the right

qstring_from_substr() takes the index of the substring's first and
last character. qstring_from_substr(s, 0, SIZE_MAX) denotes an empty
s

qstring: Move qstring_from_substr()'s @end one to the right

qstring_from_substr() takes the index of the substring's first and
last character. qstring_from_substr(s, 0, SIZE_MAX) denotes an empty
substring. Awkward.

Shift the end index one to the right. This simplifies both
qstring_from_substr() and its callers.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180727062204.10401-3-armbru@redhat.com>

show more ...

qstring: Assert size calculations don't overflow

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180727062204.10401-2-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.co

qstring: Assert size calculations don't overflow

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180727062204.10401-2-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

qstring: Fix qstring_from_substr() not to provoke int overflow

qstring_from_substr() parameters @start and @end are of type int.
blkdebug_parse_filename(), blkverify_parse_filename(), nbd_parse_uri(

qstring: Fix qstring_from_substr() not to provoke int overflow

qstring_from_substr() parameters @start and @end are of type int.
blkdebug_parse_filename(), blkverify_parse_filename(), nbd_parse_uri(),
and qstring_from_str() pass @end values of type size_t or ptrdiff_t.
Values exceeding INT_MAX get truncated, with possibly disastrous
results.

Such huge substrings seem unlikely, but we found one in a core dump,
where "info tlb" executed via QMP's human-monitor-command apparently
produced 35 GiB of output.

Fix by changing the parameters size_t.

Signed-off-by: liujunjie <liujunjie23@huawei.com>
Message-Id: <20180724134339.17832-1-liujunjie23@huawei.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

qobject: Let qobject_from_jsonf() fail instead of abort

qobject_from_jsonf() aborts on error, unlike qobject_from_jsonv(),
which returns null. Since all remaining users of qobject_from_jsonf()
cope

qobject: Let qobject_from_jsonf() fail instead of abort

qobject_from_jsonf() aborts on error, unlike qobject_from_jsonv(),
which returns null. Since all remaining users of qobject_from_jsonf()
cope fine with null, change it to return null.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180703085358.13941-30-armbru@redhat.com>

show more ...

qobject: New qdict_from_jsonf_nofail()

Many uses of qobject_from_jsonf() convert JSON objects. Create new
convenience function qdict_from_jsonf_nofail() that includes the
conversion to QDict. The

qobject: New qdict_from_jsonf_nofail()

Many uses of qobject_from_jsonf() convert JSON objects. Create new
convenience function qdict_from_jsonf_nofail() that includes the
conversion to QDict. The next few commits will put it to use.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180703085358.13941-22-armbru@redhat.com>

show more ...

block-qdict: Pacify Coverity after commit f1b34a248e9

Commit f1b34a248e9 replaced less-than-obvious test in
qdict_flatten_qdict() by the obvious one. Sadly, it made something
else non-obvious: the

block-qdict: Pacify Coverity after commit f1b34a248e9

Commit f1b34a248e9 replaced less-than-obvious test in
qdict_flatten_qdict() by the obvious one. Sadly, it made something
else non-obvious: the fact that @new_key passed to qdict_put_obj()
can't be null, because that depends on the function's precondition
(target == qdict) == !prefix.

Tweak the function some more to help Coverity and human readers alike.

Fixes: CID 1393620
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

qdict: Make qdict_flatten() shallow-clone-friendly

In its current form, qdict_flatten() removes all entries from nested
QDicts that are moved to the root QDict. It is completely sufficient to
remov

qdict: Make qdict_flatten() shallow-clone-friendly

In its current form, qdict_flatten() removes all entries from nested
QDicts that are moved to the root QDict. It is completely sufficient to
remove all old entries from the root QDict, however. If the nested
dicts have a refcount of 1, this will automatically delete them, too.
And if they have a greater refcount, we probably do not want to modify
them in the first place.

The latter observation means that it was currently (in general)
impossible to qdict_flatten() a shallowly cloned dict because that would
empty nested QDicts in the original dict as well. This patch changes
this, so you can now use qdict_flatten(qdict_shallow_clone(dict)) to get
a flattened copy without disturbing the original.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20180611205203.2624-7-mreitz@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

block: Fix -blockdev / blockdev-add for empty objects and arrays

-blockdev and blockdev-add silently ignore empty objects and arrays in
their argument. That's because qmp_blockdev_add() converts th

block: Fix -blockdev / blockdev-add for empty objects and arrays

-blockdev and blockdev-add silently ignore empty objects and arrays in
their argument. That's because qmp_blockdev_add() converts the
argument to a flat QDict, and qdict_flatten() eats empty QDict and
QList members. For instance, we ignore an empty BlockdevOptions
member @cache. No real harm, as absent means the same as empty there.

Thus, the flaw puts an artificial restriction on the QAPI schema: we
can't have potentially empty objects and arrays within
BlockdevOptions, except when they're optional and "empty" has the same
meaning as "absent".

Our QAPI schema satisfies this restriction (I checked), but it's a
trap for the unwary, and a temptation to employ awkward workarounds
for the wary. Let's get rid of it.

Change qdict_flatten() and qdict_crumple() to treat empty dictionaries
and lists exactly like scalars.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block-qdict: Simplify qdict_is_list() some

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

block-qdict: Clean up qdict_crumple() a bit

When you mix scalar and non-scalar keys, whether you get an "already
set as scalar" or an "already set as dict" error depends on qdict
iteration order. N

block-qdict: Clean up qdict_crumple() a bit

When you mix scalar and non-scalar keys, whether you get an "already
set as scalar" or an "already set as dict" error depends on qdict
iteration order. Neither message makes much sense. Replace by
""Cannot mix scalar and non-scalar keys". This is similar to the
message we get for mixing list and non-list keys.

I find qdict_crumple()'s first loop hard to understand. Rearrange it
and add a comment.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block-qdict: Tweak qdict_flatten_qdict(), qdict_flatten_qlist()

qdict_flatten_qdict() skips copying scalars from @qdict to @target
when the two are the same. Fair enough, but it uses a non-obvious

block-qdict: Tweak qdict_flatten_qdict(), qdict_flatten_qlist()

qdict_flatten_qdict() skips copying scalars from @qdict to @target
when the two are the same. Fair enough, but it uses a non-obvious
test for "same". Replace it by the obvious one. While there, improve
comments.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block-qdict: Simplify qdict_flatten_qdict()

There's no need to restart the loop. We don't elsewhere, e.g. in
qdict_extract_subqdict(), qdict_join() and qemu_opts_absorb_qdict().
Simplify accordingl

block-qdict: Simplify qdict_flatten_qdict()

There's no need to restart the loop. We don't elsewhere, e.g. in
qdict_extract_subqdict(), qdict_join() and qemu_opts_absorb_qdict().
Simplify accordingly.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block: Factor out qobject_input_visitor_new_flat_confused()

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.co

block: Factor out qobject_input_visitor_new_flat_confused()

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block: Fix -blockdev for certain non-string scalars

Configuration flows through the block subsystem in a rather peculiar
way. Configuration made with -drive enters it as QemuOpts.
Configuration mad

block: Fix -blockdev for certain non-string scalars

Configuration flows through the block subsystem in a rather peculiar
way. Configuration made with -drive enters it as QemuOpts.
Configuration made with -blockdev / blockdev-add enters it as QAPI
type BlockdevOptions. The block subsystem uses QDict, QemuOpts and
QAPI types internally. The precise flow is next to impossible to
explain (I tried for this commit message, but gave up after wasting
several hours). What I can explain is a flaw in the BlockDriver
interface that leads to this bug:

$ qemu-system-x86_64 -blockdev node-name=n1,driver=nfs,server.type=inet,server.host=localhost,path=/foo/bar,user=1234
qemu-system-x86_64: -blockdev node-name=n1,driver=nfs,server.type=inet,server.host=localhost,path=/foo/bar,user=1234: Internal error: parameter user invalid

QMP blockdev-add is broken the same way.

Here's what happens. The block layer passes configuration represented
as flat QDict (with dotted keys) to BlockDriver methods
.bdrv_file_open(). The QDict's members are typed according to the
QAPI schema.

nfs_file_open() converts it to QAPI type BlockdevOptionsNfs, with
qdict_crumple() and a qobject input visitor.

This visitor comes in two flavors. The plain flavor requires scalars
to be typed according to the QAPI schema. That's the case here. The
keyval flavor requires string scalars. That's not the case here.
nfs_file_open() uses the latter, and promptly falls apart for members
@user, @group, @tcp-syn-count, @readahead-size, @page-cache-size,
@debug.

Switching to the plain flavor would fix -blockdev, but break -drive,
because there the scalars arrive in nfs_file_open() as strings.

The proper fix would be to replace the QDict by QAPI type
BlockdevOptions in the BlockDriver interface. Sadly, that's beyond my
reach right now.

Next best would be to fix the block layer to always pass correctly
typed QDicts to the BlockDriver methods. Also beyond my reach.

What I can do is throw another hack onto the pile: have
nfs_file_open() convert all members to string, so use of the keyval
flavor actually works, by replacing qdict_crumple() by new function
qdict_crumple_for_keyval_qiv().

The pattern "pass result of qdict_crumple() to
qobject_input_visitor_new_keyval()" occurs several times more:

* qemu_rbd_open()

Same issue as nfs_file_open(), but since BlockdevOptionsRbd has only
string members, its only a latent bug. Fix it anyway.

* parallels_co_create_opts(), qcow_co_create_opts(),
qcow2_co_create_opts(), bdrv_qed_co_create_opts(),
sd_co_create_opts(), vhdx_co_create_opts(), vpc_co_create_opts()

These work, because they create the QDict with
qemu_opts_to_qdict_filtered(), which creates only string scalars.
The function sports a TODO comment asking for better typing; that's
going to be fun. Use qdict_crumple_for_keyval_qiv() to be safe.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

qobject: Move block-specific qdict code to block-qdict.c

Pure code motion, except for two brace placements and a comment
tweaked to appease checkpatch.

Signed-off-by: Markus Armbruster <armbru@redh

qobject: Move block-specific qdict code to block-qdict.c

Pure code motion, except for two brace placements and a comment
tweaked to appease checkpatch.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block: Add block-specific QDict header

There are numerous QDict functions that have been introduced for and are
used only by the block layer. Move their declarations into an own
header file to refl

block: Add block-specific QDict header

There are numerous QDict functions that have been introduced for and are
used only by the block layer. Move their declarations into an own
header file to reflect that.

While qdict_extract_subqdict() is in fact used outside of the block
layer (in util/qemu-config.c), it is still a function related very
closely to how the block layer works with nested QDicts, namely by
sometimes flattening them. Therefore, its declaration is put into this
header as well and util/qemu-config.c includes it with a comment stating
exactly which function it needs.

Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20180509165530.29561-7-mreitz@redhat.com>
[Copyright note tweaked, superfluous includes dropped]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

qobject: Modify qobject_ref() to return obj

For convenience and clarity, make it possible to call qobject_ref() at
the time when the reference is associated with a variable, or
argument, by making q

qobject: Modify qobject_ref() to return obj

For convenience and clarity, make it possible to call qobject_ref() at
the time when the reference is associated with a variable, or
argument, by making qobject_ref() return the same pointer as given.
Use that to simplify the callers.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180419150145.24795-5-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Useless change to qobject_ref_impl() dropped, commit message improved
slightly]
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...