In man(1) mode, i.e. when asking for a single manual page by name,
prefer file name matches over .Dt/.TH matches over first NAME matches
over later NAME matches, but do not change the ordering for ap

In man(1) mode, i.e. when asking for a single manual page by name,
prefer file name matches over .Dt/.TH matches over first NAME matches
over later NAME matches, but do not change the ordering for apropos(1)
nor for man -a.

This reverts main.c rev. 1.213 and mansearch.h rev. 1.23
and includes a partial revert of mansearch.c rev. 1.62.

Regression reported by Lorenzo Beretta <loreb at github>
as part of https://github.com/void-linux/void-packages/issues/9868 .

show more ...

In apropos(1) output, stop sorting .Nm search results by name
priorities (bits). The obscure feature wasn't documented and merely
confused people - for example Edward Tomasz Napierala <trasz at
Free

In apropos(1) output, stop sorting .Nm search results by name
priorities (bits). The obscure feature wasn't documented and merely
confused people - for example Edward Tomasz Napierala <trasz at
FreeBSD>, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227408.

Smaller patch provided by Yuri Pankov <yuripv at FreeBSD>, but i'm
also retiring the now unused "bits" member from struct manpage.
Simplification is good.

show more ...

Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order.
They got broken in the SQLite removal.
As opposed to the rest of -kO, they are no longer very useful,
but they are certainly not suppos

Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order.
They got broken in the SQLite removal.
As opposed to the rest of -kO, they are no longer very useful,
but they are certainly not supposed to fail assertions.
Issue reported by Gonzalo Tornaria <tornaria at cmat dot edu dot uy>.

show more ...

Remove the dependency on SQLite without loss of functionality.
Drop the obsolete names_check() now that we deleted MLINKS.
Run "doas makewhatis" after compiling and installing this.

Earlier version

Remove the dependency on SQLite without loss of functionality.
Drop the obsolete names_check() now that we deleted MLINKS.
Run "doas makewhatis" after compiling and installing this.

Earlier version tested by jmc@ and jturner@;
"commit it all" deraadt@ "commit and dodge" krw@

show more ...

In private header files, __BEGIN_DECLS and __END_DECLS are pointless.
Because these work slightly differently on different systems,
they are becoming a maintenance burden in the portable version,
so

In private header files, __BEGIN_DECLS and __END_DECLS are pointless.
Because these work slightly differently on different systems,
they are becoming a maintenance burden in the portable version,
so delete them.

Besides, one of the chief design goals of the mandoc toolbox is to
make sure that nothing related to documentation requires C++.
Consequently, linking mandoc against any kind of C++ program would
defeat the purpose and is not supported.
I don't understand why kristaps@ added them in the first place.

show more ...

header cleanup:
* add missing forward declarations
* remove needless header inclusions
* some style unification

remove unneccessary inclusion protection; ok schwarze

Make makewhatis(8) understand .so links to .gz pages.
Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong
because it ought to be in the mlinks table: An uncompressed .so link fi

Make makewhatis(8) understand .so links to .gz pages.
Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong
because it ought to be in the mlinks table: An uncompressed .so link file
can point to a compressed manual page file and vice versa.
Besides, it is no longer needed because mparse_open() handles it all.
Sprinkle some KNF while here.

show more ...

In man(1) mode, prefer file name matches over .Dt name matches over
first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries.
For example, this makes sure "man ypbind" does not return

In man(1) mode, prefer file name matches over .Dt name matches over
first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries.
For example, this makes sure "man ypbind" does not return yp(8).
Re-run "makewhatis" to profit from this change.

show more ...

In man(1) mode without -a, stop searching after the first manual tree
that contained at least one match in order to not prefer mdoc(1) from
ports over mdoc(7). As a bonus, this results in a speedup.

If a manual page is installed gzip(1)ed, let makewhatis(8) take
note in mandoc.db(5), such that man(1) -w and apropos(1) -w can
report the correct filename.
This is a prerequisite for letting apropos

If a manual page is installed gzip(1)ed, let makewhatis(8) take
note in mandoc.db(5), such that man(1) -w and apropos(1) -w can
report the correct filename.
This is a prerequisite for letting apropos -a and man support
gzip'ed manuals in the future, which doesn't work yet.

show more ...

In man(1) mode, change to the right directory before starting the parser,
just like traditional man(1) does, such that .so links have a chance to
work. After this point, we don't need the current di

In man(1) mode, change to the right directory before starting the parser,
just like traditional man(1) does, such that .so links have a chance to
work. After this point, we don't need the current directory for anything
else before exit, so we don't need to worry about getting back and we can
safely ignore failure.
This lets man(1) find more Xenocara manuals, but not all of them yet.
Other issues remain that need to be fixed, too.

show more ...

Add an implementation of man(1) into the /usr/bin/mandoc binary and
provide a unified set of command line options for mandoc(1), man(1),
apropos(1), and whatis(1), each option doing the same for all

Add an implementation of man(1) into the /usr/bin/mandoc binary and
provide a unified set of command line options for mandoc(1), man(1),
apropos(1), and whatis(1), each option doing the same for all four.
Not adding any completely new options, only extending exiting ones
from one tool to the others. New options are:
* apropos & whatis -acfkw (in the past, these were man(1) only)
* apropos & whatis -a -IOTW (in the past, mandoc(1) only)
* mandoc -ac (in the past, man(1) only)
* man -IOTW (in the past, mandoc(1) only)

Before we can decide whether or not we want to replace src/usr.bin/man
with this implementation, considerable bugfixing, testing, and
performance measurements are needed, which i'd rather do in the tree
than outside. Note that these bugs only affect the new man(1) mode,
existing mandoc(1), apropos(1), and whatis(1) is fine.

The new functionality in mandoc(1), apropos(1), and whatis(1)
is fully enabled. To play with the new man(1), you can try:
# mv /usr/bin/man /usr/bin/oman
# ln -s /usr/bin/mandoc /usr/bin/man

Positive feedback about the general direction from sthen@ and jmc@,
and deraadt@ is not against it.

show more ...

Sort result pages first by section number, then by name.
By moving the sort from cgi.c to mansearch.c, we get two advantages:
Easier access to the data needed for sorting, in particular the section
n

Sort result pages first by section number, then by name.
By moving the sort from cgi.c to mansearch.c, we get two advantages:
Easier access to the data needed for sorting, in particular the section
number, and the apropos(1) command line utility profits as well.

Feature requested by deraadt@.

show more ...

Fix whatis(1) to correctly match words instead of any substrings.
While here, also provide an internal mode (MANSEARCH_MAN) to match
complete names, to be used by man.cgi(8).

Further apropos(1) speed optimization was trickier than anticipated.
Contrary to what i initially thought, almost all time is now spent
inside sqlite3(3) routines, and i found no easy way calling les

Further apropos(1) speed optimization was trickier than anticipated.
Contrary to what i initially thought, almost all time is now spent
inside sqlite3(3) routines, and i found no easy way calling less of them.
However, sqlite(3) spends substantial time in malloc(3), and even more
(twice that) in its immediate malloc wrapper, sqlite3MemMalloc(),
keeping track of all individual malloc chunk sizes. Typically about
90% of the malloced memory is used for purposes of the pagecache.

By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution
time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for
non-NAME queries, and even apropos(1) resident memory size as reported
by top(1) decreases by 20% for simple and by 60% for non-NAME queries.
The new function, mansearch_setup(), spends no measurable time.

The pagesize chosen is optimal:
* Substantially smaller pages yield no gain at all.
* Larger pages provide no additional benefit and just waste memory.

The chosen number of pages in the cache is a compromise:
* For simple queries, a handful of pages would suffice to get the full
speed effect, at an apropos(1) resident memory size of about 2.0 MB.
* For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might
gain a few more percent in speed, but at the expense of doubling the
apropos(1) resident memory size for *all* queries.
* The chosen number of 256 pages (330 kB) allows nearly full speed gain
for all queries at the price of a 15% resident memory size increase.

show more ...

Next speed optimization step for the new apropos(1).
Split manual names out of the common "keys" table into their
own "names" table. This reduces standard apropos(1) search
times (i.e. searching for

Next speed optimization step for the new apropos(1).
Split manual names out of the common "keys" table into their
own "names" table. This reduces standard apropos(1) search
times (i.e. searching for names and descriptions only) by
typically about 70% for the full /usr/share/man database.
(Yes, that multiplies with the previous optimization step,
so both together have reduced search times by a factor of
more than six. I'm not done yet, expect more to come.)

Even with the minimal databases built with makewhatis(8) -Q,
this step still reduces search times by 15-20%. For both cases,
database sizes and build times hardly change (+/-2%).

show more ...

After careful gprof(1)ing of the new apropos(1), move the descriptions
back from the keys table to the mpages table: I found a good way
to still use them in searches, without complication of the cod

After careful gprof(1)ing of the new apropos(1), move the descriptions
back from the keys table to the mpages table: I found a good way
to still use them in searches, without complication of the code.

On my notebook, this reduces typical apropos(1) search times by about 40%,
it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode
and by 2% in standard mode (less overhead storing pointers to mpages),
and it doesn't measurably change database build times (may even be
going down by a percent or so because less data is being copied
around in ohashes).

show more ...

Remember which names are in the NAME section.
This helps to find missing MLINKS.
Database build times do not change and database growth is minimal
(1.2% with -Q, 0.7% without -Q in /usr/share/man),
s

Remember which names are in the NAME section.
This helps to find missing MLINKS.
Database build times do not change and database growth is minimal
(1.2% with -Q, 0.7% without -Q in /usr/share/man),
so making this optional would be pointless.

show more ...

Sort the macro keys by their real-world frequency to reduce the average
mask size. No functional change.

This shrinks the standard /usr/share/man database by 7%, now at 10.3x
the size of whatis.db,

Sort the macro keys by their real-world frequency to reduce the average
mask size. No functional change.

This shrinks the standard /usr/share/man database by 7%, now at 10.3x
the size of whatis.db, and with -Q even by 11%, now at 3.0x of whatis.db.
Now i'm out of ideas to easily shrink the size of the database.

Optimization found somewhere above the West Australian desert.
Committing from Melbourne, Victoria.

show more ...

Drop Nd from the mpages table, it is still in the keys table.
This shrinks the database in standard mode by 3%, in -Q mode by 9%,
without loss of functionality.

Put section and architecture info into the keys table,
in preparation for removing them from the mpages table,
aiming for cleaner and more uniform interfaces.
Database growth is below 4%, part of whi

Put section and architecture info into the keys table,
in preparation for removing them from the mpages table,
aiming for cleaner and more uniform interfaces.
Database growth is below 4%, part of which will be reclaimed.

As a bonus, this allows searches like:
./obj/apropos An=kettenis -a arch=ppc
./obj/apropos An=kettenis -a sec~[^4]

show more ...

Experimental feature to let apropos(1) show different keys than .Nd.
This really takes us beyond what grep -R /usr/*/man/ can do
because now you can search for pages by *one* criterion and then
displ

Experimental feature to let apropos(1) show different keys than .Nd.
This really takes us beyond what grep -R /usr/*/man/ can do
because now you can search for pages by *one* criterion and then
display the contents of *another* macro from those pages, like in
$ apropos -O Ox Fa~wchar
to get an impression how long wide character handling is available.

show more ...

Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version,
aiming for more flexible development and optimization options.
Kristaps started this during the summer 2012, i did some very hea

Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version,
aiming for more flexible development and optimization options.
Kristaps started this during the summer 2012, i did some very heavy
bugfixing during t2k13 and finally, during the last few days,
got it to a state where it is ripe for in-tree development.
Beware, neither the user interfaces nor the database formats
are expected to be stable just yet.

Will not be installed or activated until further discussion.
No functional change to mandoc(1).

"As long as it remains off until we decide the cost, fine." deraadt@

show more ...