1# The QDirStat Unpackaged Files View 2 3QDirStat can now visualize the files in system directories that are not 4packaged, i.e. that are not part of any file list of any installed software 5package. 6 7This reads the complete file lists first (i.e. all file lists from all 8installed packages), then reads the requested directory as usual and puts the 9files that are packaged in a special branch `<Ignored>` in the tree view. 10 11Those files are _not_ displayed in the treemap, i.e. the treemap now only 12contains unpackaged files. 13 14 15## What is this Good For? 16 17- Recovering a wrecked system after botched upgrades 18 19- Recovering from too much `sudo make install` 20 21- Fixing packaging problems, e.g. find leftover shared libs that tend to get in 22 the way 23 24- QA for distro release managers, QA engineers, PMs 25 26- Satisfying user curiosity 27 28- _Because we can!_ ;-) 29 30 31## Starting the Unpackaged Files View 32 33Menu "File" -> "Show Unpackaged Files" opens this dialog: 34 35 36 37Select the starting directory from the combo box or start typing (it has 38directory name completion) to enter the path of an existing directory. 39 40A number of directories are excluded because they are expected to contain a lot 41of unpackaged files. You can add more directories here (or remove ones from the 42default list). 43 44_`/usr/lib/sysimage/rpm` is the RPM database; you can safely leave that 45directory there even on non-RPM systems because it doesn't exist on such 46systems._ 47 48"Ignore Patterns" are wildcard patterns that are ignored in addition to the 49packaged files. `*.pyc` (byte-compiled Python files) is there because Python 50packages tend to leave behind a zillion unpackaged files of that kind that 51would otherwise very much be in the way when exploring the result. 52 53You can add more wildcard patterns here. If a pattern does not contain a slash, 54only the filename is used for matching. If it does contain a slash, the 55complete path is used. 56 57Notice that a wildcard matches any levels of directories, so you could for 58example use `*/__pycache__/*` to ignore all files in all Python cache 59directories everywhere. 60 61All changes in this dialog are saved for the next use, including starting that 62view from the command line (e.g. `qdirstat unpkg:/usr/share`). 63 64 65## Analyzing the Result 66 67 68 69_Notice that the treemap now only contains unpackaged files. Each of the 70colored rectangles corresponds to one unpackaged file._ 71 72This screenshot shows some files that obviously belong to Qt 5 (judging from 73the `/usr/share/qt5` path) on openSUSE Tumbleweed, but that are not in the file 74list of any package. Is that a bug or not? It is still possible that some 75post-uninstall script removes them, but it is also possible that they will be 76left behind when the packages are uninstalled. 77 78Notice the greyed-out ignored directories which only contain files that are 79properly packaged. Those `<Ignored>` branches are there to give you some 80perspective what else is in each directory; otherwise users might be tempted to 81believe that the unpackaged files are all that there is, and maybe remove that 82entire directory and thus wreck the system. 83 84 85 86 87This screenshot shows a custom Grub2 font that was manually created to have a 88larger, much better readable boot font on a high-resolution monitor. That one 89file clearly stands out from all the packaged files there. 90 91`/boot/grub2` and `/etc` are some of those directories that contain a wild 92mixture of packaged and unpackaged files; some are generated, some are just 93backups of the previous version, some were created as templates for users to 94edit. 95 96 97### Ignored Files 98 99In this view, all files that are in the file list of any installed package are 100ignored. As a result, there are many directories that don't contain any 101unpackaged file; that should actually be the norm. 102 103But since the purpose of this view is to visualize unpackaged files, 104directories that don't contain any packaged files (including completely empty 105directories) are also moved to the highest possible `<Ignored>` branch so they 106get out of the way. 107 108 109## Use Cases 110 111### Accumulated Cruft after many Package Updates 112 113Package maintainers are also just humans, and humans make mistakes. Every once 114in a while a file that belongs to a package is not listed or listed the wrong 115way. 116 117If you have been using a Linux distribution for a long time, you will get many 118package updates; using a rolling release like openSUSE Tumbleweed will get you 119even more. 120 121After a while, those little packaging bugs may accumulate, and your system may 122behave strangely. Some things may stop working or give you very weird error 123messages. 124 125Before you nuke your installation completely and reinstall, consider having a 126look with this unpackaged files view. You may discover shared libs or symlinks 127left behind in crucial system directories like `/usr/lib` or `/usr/lib64`. As a 128general rule of thumb, there should be no unpackaged files in those 129directories. If you find one, ask in a user forum what they are and why they 130might plausibly not be part of a software package. 131 132Or, if you feel brave, move them away (don't delete them right away!) to a 133subdirectory, rebuild the linker cache (!) with `ldconfig` and reboot. Make 134sure you have some live media (a bootable USB stick or a CD/DVD with a recent 135version of the distro you are running) so you can do disaster recovery if 136something goes terribly wrong. 137 138**If you don't know what an unpackaged shared lib is that you found, google it 139first!** 140 141You may find that it is part of some piece of software that you installed 142without using your package manager (some hardware support software comes as a 143ZIP file or as a tarball). It might just be part of the crucial driver for that 144fancy wireless keyboard or mouse that you use, and using your system without 145that hardware may be no fun. 146 147 148### Recovering from too much make install 149 150This is very similar to the accumulated cruft after many package updates 151problem, but it is typically your own fault: When you build software yourself, 152it is much easier to install it directly to the system rather than building a 153.deb or .rpm package: 154 155``` 156make && sudo make install 157``` 158 159If you don't set the install prefix to `/usr/local`, you will end up with 160additional or modified files in system directories like `/usr/bin` or 161`/usr/lib`. Of course, the package manager doesn't know anything about those 162files, so it cannot handle them. 163 164So if you just added a newer version of an important shared lib to your system, 165that version may have priority over the system's own version of that lib. 166 167QDirStat's unpackaged files view may help you to identify such files. 168 169Of course it's much better not to get into that kind of trouble in the first 170place and install your own custom-built software to a dedicated place like 171`/usr/local`, but many users just don't think about that. 172 173 174### A Useful Tool for Package Maintainers 175 176Maintainers of Linux software packages can use this view to analyze their 177packaging work: 178 179- Does the file list contain everything? The packaging tools should complain 180 if you forget to list anything, but they may also not be perfect. 181 182- Does a package upgrade / uninstall behave properly? Are all files that the 183 old version installed or created during its life time properly removed? Or is 184 anything left behind? Do the pre-install / post-install / pre-uninstall / 185 post-uninstall scripts do the right thing? 186 187 188### A Useful Tool for Distro QA Engineers and Release Managers 189 190Building and maintaining a Linux distribution is a complex task. There are many 191packages involved; a typical Linux installation today has some 2500-4500 192packages installed. 193 194Maintaining a software package involves getting the recent sources from 195upstream, getting them to build, patching them to integrate well into the 196distribution, setting up package dependencies, maintaining the file list and 197often enough also writing scripts: pre-install, post-install, pre-uninstall, 198post-uninstall. 199 200Upgrading from the old distro version to the new one comes with its own 201challenges, in particular if upgrading not only from the last version is 202required, but also two or three versions before that. 203 204There is a lot of potential for bugs to sneak in. Multiply that with the number 205of packages, and you end up with insane numbers of pitfalls. 206 207So distro QA engineers test at least the most common cases: Upgrade from 208release X-1 to X with the standard package set, with the minimal package set, 209with the maximum package set; upgrade from release X-2 to X, from X-3 to X. 210 211All those cases should work well with no major problems. Finding leftover files 212may contribute to find problems quicker. 213 214 215### Be Curious! Stay Curious! 216 217What are all those files on my Linux system? Where does all that stuff belong 218to? 219 220Remember that QDirStat will show you what package a file belongs to (if any) 221when you click on it; the details panel on the right shows the package name. 222 223The package view will show you the complete contents (at least the part that 224was installed to your system) in the package view (Menu "File" -> "Show 225Installed Packages"). 226 227And this view solves the inverse problem: It shows you what files do _not_ 228belong to any package. Of course that makes most sense for system directories. 229 230If there are unpackaged files in system directories, that may be a bug, or it 231may be normal. But it is clearly a starting point for further investigation and 232for asking experts. 233 234Linux (and BSD and other Unix-like systems) was meant to be open and easy to 235explore. So, become involved. Become knowledgeable what all that stuff on your 236system is. You may be able to contribute to finding problems; you don't need to 237be a developer for that. Just exploring things and asking the right questions 238may help the community as a whole to identify problems. 239 240 241 242## Target Platforms 243 244This works on all Linux distributions using one of those low-level package 245managers: 246 247- Dpkg 248- RPM 249- ~~PacMan~~ 250 251...and of course all higher-level package managers that are based on any of 252them. I.e. 253 254- Debian 255 256- Ubuntu / Kubuntu / Xubuntu / Lubuntu 257 258- SUSE (openSUSE Tumbleweed or Leap, SLES) 259 260- Red Hat (Fedora, RHEL) 261 262- ~~Arch Linux~~ 263 264- ~~Manjaro~~ 265 266Please notice that _apt_, _synaptic_, _zypper_, _pkgkit_ and whatnot are all 267higher level package managers that ultimately use one of the low level ones, so 268even if you only use a higher level package manager, it still works without 269restriction. 270 271_Sorry, no support for PacMan as of now: There does not appear to be a single 272command to generate the complete file list for PacMan. Advanced users can try to use the [script-based version](../scripts/pkg-tools/README.md)._ 273 274_If anybody knows a PacMan command to do this, please let me know!_ 275 276 277## Command Line: Unpkg URLs 278 279As an alternative to using the menu, you can also start QDirStat from the 280command line with an _unpkg_ URL: 281 282``` 283qdirstat unpkg:/usr/share 284``` 285 286This starts the unpackaged files view with `/usr/share` as the starting 287directory. It will use the same exclude directories and ignore patterns as the 288last time you used that view from the menu. 289 290``` 291qdirstat unpkg:/ 292``` 293 294This starts the unpackaged files view with the root directory as the starting 295directory. 296 297 298``` 299qdirstat -d 300``` 301 302This starts QDirStat without asking for a directory to scan, so you can go 303straight to the menu.
){kind=link}
){kind=link}
){kind=link}