1\chapter{New Features in Enterprise 4.0.x}
2There are new features in the Bacula Enterprise version.
3This is an older version and this documentation remains
4for historical reasons.
5
6\section{New Features in Version 4.0.8}
7
8\subsection{Always Backup a File}
9
10When the Accurate mode is turned on, you can decide to always backup a file
11by using the following option:
12
13\begin{bVerbatim}
14Job {
15   Name = ...
16   FileSet = FS_Example
17   Accurate = yes
18   ...
19}
20
21FileSet {
22 Name = FS_Example
23 Include {
24   Options {
25     Accurate = A
26   }
27   File = /file
28   File = /file2
29 }
30 ...
31}
32\end{bVerbatim}
33
34\section{New Features in 4.0.5}
35
36There are new features in version 4.0.5 and this version fixes a number of bugs
37found in version 4.0.4.
38
39\subsection{Support for the VSS plugin}
40
41The System State component of the VSS plugin (see below) is now supported.
42All tests indicate that it is functioning correctly.
43
44The Exchange component of the VSS plugin appears to work in Full backup
45mode only.  Incremental restores fail, so please do not attempt Incremental
46backups.  We are therefore releasing this plugin for testing in Full backup
47mode only.  However, please carefully test it before using it.  We are
48working on fixing the problem with Incremental restores.
49
50The MSSQL component of the VSS plugin works in Full backup mode only.
51Incremental backups and restores do not work because they need the delta
52backup capability that is only in the next major version (not yet
53released), so please do not attempt Incremental backups.  We are therefore
54releasing this plugin for testing in Full backup mode only.  However,
55please carefully test it before using it.
56
57The Sharepoint component of the VSS plugin has not been tested. Any
58feedback on testing it would be appreciated.
59
60
61\subsection{Support for NDMP Protocol}
62
63The new \texttt{ndmp} Plugin is able to backup a NAS through NDMP protocol
64using \textbf{Filer to server} approach, where the Filer is backing up across
65the LAN to your Bacula server.
66
67Accurate option should be turned on in the Job resource.
68\begin{bVerbatim}
69Job {
70 Accurate = yes
71 FileSet = NDMPFS
72 ...
73}
74
75FileSet {
76 Name = NDMPFS
77 ...
78 Include {
79   Plugin = "ndmp:host=nasbox user=root pass=root file=/vol/vol1"
80 }
81}
82\end{bVerbatim}
83
84This plugin is available as an option. Please
85contact Bacula Systems to get access to the NDMP Plugin packages and the
86documentation.
87
88\smallskip{}
89
90This project was funded by Bacula Systems and is available with Bacula
91Enterprise Edition.
92
93\subsection{Include All Windows Drives in FileSet}
94
95The \texttt{alldrives} Windows Plugin allows you to include all local drives
96with a simple directive. This plugin is available in the Windows 64 and 32 bit
97installer.
98
99\begin{bVerbatim}
100FileSet {
101 Name = EverythingFS
102 ...
103 Include {
104   Plugin = "alldrives"
105 }
106}
107\end{bVerbatim}
108
109You exclude some specific drives with the \texttt{exclude} option.
110
111\begin{bVerbatim}
112FileSet {
113 Name = EverythingFS
114 ...
115 Include {
116   Plugin = "alldrives: exclude=D,E"
117 }
118}
119\end{bVerbatim}
120
121
122This project was funded by Bacula Systems and is available with Bacula
123Enterprise Edition.
124
125\subsection{Additions to RunScript variables}
126You can have access to JobBytes and JobFiles using \%b and \%f in your runscript
127command.
128
129\begin{bVerbatim}
130RunAfterJob = "/bin/echo Job=%j JobBytes=%b JobFiles=%f"
131\end{bVerbatim}
132
133\section{Release Version 4.0.1 to 4.0.4}
134
135There are no new features between version 4.0.1 and 4.0.4.  These versions
136simply fixe a number of bugs found in previous version during the onging
137development process.
138
139\section{New Features in 4.0.0}
140This chapter presents the new features that have been added to the
141current version of the Bacula Enterprise Edition since the previous
142versions.
143
144\subsection{Microsoft VSS Writer Plugin}
145\index[general]{Microsoft VSS writer plugin}
146We provide a single plugin named \btool{vss-fd.dll} that
147permits you to backup a number of different components
148on Windows machines.   This plugin is available from Bacula Systems
149as an option.
150
151Only the System State component is currently supported.  The Sharepoint,
152MSSQL, and Exchange components are available only for testing.
153
154\begin{bitemize}
155\item System State writers
156   \begin{bitemize}
157   \item Registry
158   \item Event Logs
159   \item COM+ REGDB (COM Registration Database)
160   \item System (Systems files -- most of what is under \bdirectoryname{c:/windows} and more)
161   \item WMI (Windows Management and Instrumentation)
162   \item NTDS (Active Directory)
163   \item NTFRS (SYSVOL etc replication -- Windows 2003 domains)
164   \item DFS Replication (SYSVOLS etc replication -- Windows 2008 domains)
165   \item ASR Writer
166   \end{bitemize}
167   This component is known to work.
168\item Sharepoint writers \\
169  This component has not yet been tested. It is included so that you
170  may test it, but please do not use it in production without careful
171  testing.
172\item MSSQL databases (except those owned by Sharepoint if that plugin is
173specified). \\
174  This component has been tested, but only works for Full backups. Please
175  do not attempt to use it for incremental backups. The Windows writer
176  performs block level delta for Incremental backups, which are only
177  supported by Bacula version 4.2.0, not yet released.  If you use
178  this component, please do not use it in production without careful
179  testing.
180\item Exchange (all exchange databases) \\
181  We have tested this component and found it to work, but only for Full
182  backups.  Please do not attempt to use it for incremental or differential
183  backups.  We are including this component for you to test.  Please do not
184  use it in production without careful testing.  \\ Bacula Systems has a
185  White Paper that describes backup and restore of MS Exchange 2010 in
186  detail.
187\end{bitemize}
188
189Each of the above specified Microsoft components can be backed up
190by specifying a different plugin option within the Bacula FileSet.
191All specifications must start with \textbf{vss:} and be followed
192with a keyword which indicates the writer, such as \textbf{/@SYSTEMSTATE/}
193(see below).
194To activate each component you use the following:
195
196\begin{bitemize}
197\item System State writers
198  \begin{bVerbatim}
199  Plugin = "vss:/@SYSTEMSTATE/"
200  \end{bVerbatim}
201  Note, exactly which subcomponents will be backed up depends on
202  which ones you have enabled within Windows.  For example, on a standard
203  default Vista system only ASR Writer, COM+ REGDB, System State, and WMI
204  are enabled.
205\item Sharepoint writers
206  \begin{bVerbatim}
207  Plugin = "vss:/@SHAREPOINT/"
208  \end{bVerbatim}
209\item MSSQL databases (except those owned by Sharepoint if that plugin is
210specified)
211  \begin{bVerbatim}
212  Plugin = "vss:/@MSSQL/"
213  \end{bVerbatim}
214   To use the sharepoint writer you'll need to enable the mssql writer
215   which is not enabled by default (a Microsoft restriction). The Microsoft
216   literature says that the mssql writer is only good for snapshots
217   and it needs to be
218   enabled via a registry tweak or else the older MSDE writer will be
219   invoked instead.
220\item Exchange (all exchange databases)
221  \begin{bVerbatim}
222  Plugin = "vss:/@EXCHANGE/"
223  \end{bVerbatim}
224\end{bitemize}
225
226The plugin directives must be specified exactly as shown above.
227A Job may have one or more of the \textbf{vss} plugins components specified.
228
229
230Also ensure that the vss-fd.dll plugin is in the plugins directory
231on the FD doing the backup, and that the plugin directory config line is
232present in the FD's configuration file (bacula-fd.conf).
233
234\subsubsection{Backup}
235If everything is set up correctly as above then the backup should
236include the system state. The system state files backed up will appear
237in a \btool{bconsole} or \bat{} restore like:
238
239\begin{bVerbatim}
240/@SYSTEMSTATE/
241/@SYSTEMSTATE/ASR Writer/
242/@SYSTEMSTATE/COM+ REGDB Writer/
243etc
244\end{bVerbatim}
245
246Only a complete backup of the system state is supported at this time.  That
247is it is not currently possible to just back up the Registry or Active
248Directory by itself.  In almost all cases a complete backup is a good idea
249anyway as most of the components are interconnected in some way.  Also, if
250an incremental or differential backup is specified on the backup Job then a
251full backup of the system state will still be done.  The size varies
252according to your installation.  We have seen up to 6GB
253under Windows 2008, mostly because of the "System" writer, and
254up to 20GB on Vista.  The actual size depends on how many Windows
255components are enabled.
256
257The system state component automatically respects all the excludes present
258in the FilesNotToBackup registry key, which includes things like \%TEMP\%,
259pagefile.sys, hiberfil.sys, etc.  Each plugin may additionally specify
260files to exclude, eg the VSS Registry Writer will tell Bacula to not back
261up the registry hives under \bdirectoryname{C:\textbackslash{}WINDOWS\textbackslash{}system32\textbackslash{}config} because they
262are backed up as part of the system state.
263
264\subsubsection{Restore}
265In most cases a restore of the entire backed up system state is
266recommended. Individual writers can be selected for restore, but currently
267not individual components of those writers. To restore just the Registry,
268you would need to mark @SYSTEMSTATE (only the directory, not the
269subdirectories), and then do \textbf{mark Registry*} to mark the Registry writer
270and everything under it.
271
272Restoring anything less than a single component may not produce the
273intended results and should only be done if a specific need arises and you
274know what you are doing, and not without testing on a non-critical system
275first.
276
277To restore Active Directory, the system will need to be booted into
278Directory Services Restore Mode, an option at Windows boot time.
279
280Only a non-authoritative restore of NTFRS/DFSR is supported at this
281time. There exists Windows literature to turn a Domain Controller
282restored in non-authoritative mode back into an authoritative Domain
283Controller. If only one DC exists it appears that Windows does an
284authoritative restore anyway.
285
286Most VSS components will want to restore to files that are currently in
287use. A reboot will be required to complete the restore (eg to bring the
288restored registry online).
289
290Starting another restore of VSS data after the restore of the registry
291without first rebooting will not produce the intended results as the 'to be
292replaced next reboot' file list will only be updated in the 'to be
293replaced' copy of the registry and so will not be actioned.
294
295\subsubsection{Example}
296Suppose you have the following backup FileSet:
297
298\begin{bVerbatim}
299@SYSTEMSTATE/
300  System Writer/
301    instance_{GUID}
302    System Files/
303  Registry Writer/
304    instance_{GUID}
305    Registry/
306  COM+ REGDB Writer/
307    instance_{GUID}
308    COM+ REGDB/
309  NTDS/
310    instance_{GUID}
311    ntds/
312\end{bVerbatim}
313
314If only the Registry needs to be restored, then you could use the
315following commands in \btool{bconsole}:
316
317\begin{bVerbatim}
318markdir @SYSTEMSTATE
319cd @SYSTEMSTATE
320markdir "Registry Writer"
321cd "Registry Writer"
322mark instance*
323mark "Registry"
324\end{bVerbatim}
325
326\subsubsection{Windows Plugins Items to Note}
327\begin{bitemize}
328\item Reboot Required after a Plugin Restore\\
329In general after any VSS plugin is used to restore a component, you will
330need to reboot the system.  This is required because in-use files cannot be
331replaced during restore time, so they are noted in the registry and
332replaced when the system reboots.
333\item After a System State restore, a reboot will generally take
334longer than normal because the pre-boot process must move the newly restored
335files into their final place prior to actually booting the OS.
336\item One File from Each Drive needed by the Plugins must be backed up\\
337At least one file from each drive that will be needed by the plugin must
338have a regular file that is marked for backup.  This is to ensure that the
339main Bacula code does a snapshot of all the required drives. At a later
340time, we will find a way to accomplish this automatically.
341\item Bacula does not Automatically Backup Mounted Drives\\
342Any drive that is mounted in the normal file structure using a mount point
343or junction point will not be backed up by Bacula.  If you want it backed
344up, you must explicitly mention it in a Bacula "File" directive in your
345FileSet.
346\item When doing a backup that is to be used as a Bare Metal Recovery, do
347not use the VSS plugin. The reason is that during a Bare Metal Recovery,
348VSS is not available nor are the writers from the various components that
349are needed to do the restore.  You might do full backup to be used with
350a Bare Metal Recovery once a month or once a week, and all other days,
351do a backup using the VSS plugin, but under a different Job name.  Then
352to restore your system, use the last Full non-VSS backup to restore your
353system, and after rebooting do a restore with the VSS plugin to get
354everything fully up to date.
355\end{bitemize}
356
357\subsubsection{Bare Metal Restore}
358Depending on the bare metal restore environment, the VSS writers may not
359be running correctly so this may not work. If this is the case,
360the System State must be restored after the Bare Metal Recovery procedure
361is complete and the system and Bacula are running normally.
362
363\subsection{Additions to the Plugin API}
364The bfuncs structure has been extended to include a number of
365new entrypoints.
366
367
368\subsection{Truncate Volume after Purge}
369\label{blb:sec:actiononpurge}
370
371The Pool directive \textbf{ActionOnPurge=Truncate} instructs Bacula to truncate
372the volume when it is purged with the new command \texttt{purge volume
373  action}. It is useful to prevent disk based volumes from consuming too much
374space.
375
376\begin{bVerbatim}
377Pool {
378  Name = Default
379  Action On Purge = Truncate
380  ...
381}
382\end{bVerbatim}
383
384As usual you can also set this property with the \texttt{update volume} command
385\begin{bVerbatim}
386*update volume=xxx ActionOnPurge=Truncate
387*update volume=xxx actiononpurge=None
388\end{bVerbatim}
389
390To ask Bacula to truncate your \texttt{Purged} volumes, you need to use the
391following command in interactive mode or in a RunScript as shown after:
392\begin{bVerbatim}
393*purge volume action=truncate storage=File allpools
394# or by default, action=all
395*purge volume action storage=File pool=Default
396\end{bVerbatim}
397
398This is possible to specify the volume name, the media type, the pool, the
399storage, etc\dots (see \texttt{help purge}) Be sure that your storage device is
400idle when you decide to run this command.
401
402\begin{bVerbatim}
403Job {
404 Name = CatalogBackup
405 ...
406 RunScript {
407   RunsWhen=After
408   RunsOnClient=No
409   Console = "purge volume action=all allpools storage=File"
410 }
411}
412\end{bVerbatim}
413
414\paragraph{Important note}: This feature doesn't work as
415expected in version 5.0.0. Please do not use it before version 5.0.1.
416
417\subsection{Allow Higher Duplicates}
418This directive did not work correctly and has been depreciated
419(disabled) in version 5.0.1. Please remove it from your bacula-dir.conf
420file as it will be removed in a future rlease.
421
422\subsection{Cancel Lower Level Duplicates}
423This directive was added in Bacula version 5.0.1.  It compares the
424level of a new backup job to old jobs of the same name, if any,
425and will kill the job which has a lower level than the other one.
426If the levels are the same (i.e. both are Full backups), then
427nothing is done and the other Cancel XXX Duplicate directives
428will be examined.
429
430
431\textbf{Maximum Concurrent Jobs} is a new Device directive in the Storage
432Daemon configuration permits setting the maximum number of Jobs that can
433run concurrently on a specified Device.  Using this directive, it is
434possible to have different Jobs using multiple drives, because when the
435Maximum Concurrent Jobs limit is reached, the Storage Daemon will start new
436Jobs on any other available compatible drive.  This facilitates writing to
437multiple drives with multiple Jobs that all use the same Pool.
438
439This project was funded by Bacula Systems.
440
441\subsection{Restore from Multiple Storage Daemons}
442\index[general]{Restore}
443
444Previously, you were able to restore from multiple devices in a single Storage
445Daemon. Now, Bacula is able to restore from multiple Storage Daemons. For
446example, if your full backup runs on a Storage Daemon with an autochanger, and
447your incremental jobs use another Storage Daemon with lots of disks, Bacula
448will switch automatically from one Storage Daemon to an other within the same
449Restore job.
450
451You must upgrade your File Daemon to version 3.1.3 or greater to use this
452feature.
453
454This project was funded by Bacula Systems with the help of Equiinet.
455
456\subsection{File Deduplication using Base Jobs}
457A base job is sort of like a Full save except that you will want the FileSet to
458contain only files that are unlikely to change in the future (i.e.  a snapshot
459of most of your system after installing it).  After the base job has been run,
460when you are doing a Full save, you specify one or more Base jobs to be used.
461All files that have been backed up in the Base job/jobs but not modified will
462then be excluded from the backup.  During a restore, the Base jobs will be
463automatically pulled in where necessary.
464
465This is something none of the competition does, as far as we know (except
466perhaps BackupPC, which is a Perl program that saves to disk only).  It is big
467win for the user, it makes Bacula stand out as offering a unique optimization
468that immediately saves time and money.  Basically, imagine that you have 100
469nearly identical Windows or Linux machine containing the OS and user files.
470Now for the OS part, a Base job will be backed up once, and rather than making
471100 copies of the OS, there will be only one.  If one or more of the systems
472have some files updated, no problem, they will be automatically restored.
473
474See the \bilink{Base Job Chapter}{blb:basejobs} for more information.
475
476This project was funded by Bacula Systems.
477
478\subsection{AllowCompression = \byesorno{}}
479\index[dir]{AllowCompression}
480
481This new directive may be added to Storage resource within the Director's
482configuration to allow users to selectively disable the client compression for
483any job which writes to this storage resource.
484
485For example:
486\begin{bVerbatim}
487Storage {
488  Name = UltriumTape
489  Address = ultrium-tape
490  Password = storage_password # Password for Storage Daemon
491  Device = Ultrium
492  Media Type = LTO 3
493  AllowCompression = No # Tape drive has hardware compression
494}
495\end{bVerbatim}
496The above example would cause any jobs running with the UltriumTape storage
497resource to run without compression from the client file daemons.  This
498effectively overrides any compression settings defined at the FileSet level.
499
500This feature is probably most useful if you have a tape drive which supports
501hardware compression.  By setting the \texttt{AllowCompression = No} directive
502for your tape drive storage resource, you can avoid additional load on the file
503daemon and possibly speed up tape backups.
504
505This project was funded by Collaborative Fusion, Inc.
506
507\subsection{Accurate Fileset Options}
508\label{blb:sec:accuratefileset}
509
510In previous versions, the accurate code used the file creation and modification
511times to determine if a file was modified or not. Now you can specify which
512attributes to use (time, size, checksum, permission, owner, group, \dots),
513similar to the Verify options.
514
515\begin{bVerbatim}
516FileSet {
517  Name = Full
518  Include = {
519    Options {
520       Accurate = mcs
521       Verify   = pin5
522    }
523    File = /
524  }
525}
526\end{bVerbatim}
527
528\begin{bdescription}
529\item [i]  compare the inodes
530\item [p]  compare the permission bits
531\item [n]  compare the number of links
532\item [u]  compare the user id
533\item [g]  compare the group id
534\item [s]  compare the size
535\item [a]  compare the access time
536\item [m]  compare the modification time (st\_mtime)
537\item [c]  compare the change time (st\_ctime)
538\item [d]  report file size decreases
539\item [5]  compare the MD5 signature
540\item [1]  compare the SHA1 signature
541\end{bdescription}
542
543\paragraph{Important note}: If you decide to use checksum in Accurate jobs,
544the File Daemon will have to read all files even if they normally would not
545be saved.  This increases the I/O load, but also the accuracy of the
546deduplication.  By default, Bacula will check modification/creation time
547and size.
548
549This project was funded by Bacula Systems.
550
551\subsection{Tab-completion for Bconsole}
552\label{blb:sec:tabcompletion}
553
554If you build \texttt{bconsole} with readline support, you will be able to use
555the new auto-completion mode. This mode supports all commands, gives help
556inside command, and lists resources when required. It works also in the restore
557mode.
558
559To use this feature, you should have readline development package loaded on
560your system, and use the following option in configure.
561\begin{bVerbatim}
562./configure --with-readline=/usr/include/readline --disable-conio ...
563\end{bVerbatim}
564
565The new bconsole won't be able to tab-complete with older directors.
566
567This project was funded by Bacula Systems.
568
569\subsection{Pool File and Job retention}
570\label{blb:sec:poolfilejobretention}
571
572% TODO check
573We added two new Pool directives, \bdirectivename{FileRetention} and
574\bdirectivename{JobRetention}, that take precedence over Client directives of the same
575name. It allows you to control the Catalog pruning algorithm Pool by Pool. For
576example, you can decide to increase Retention times for Archive or OffSite Pool.
577
578\subsection{Read-only File Daemon using capabilities}
579\label{blb:sec:fdreadonly}
580This feature implements support of keeping \textbf{ReadAll} capabilities after
581UID/GID switch, this allows FD to keep root read but drop write permission.
582
583It introduces new \texttt{bacula-fd} option (\texttt{-k}) specifying that
584\textbf{ReadAll} capabilities should be kept after UID/GID switch.
585
586\begin{bVerbatim}
587root@localhost:~# bacula-fd -k -u nobody -g nobody
588\end{bVerbatim}
589
590The code for this feature was contributed by our friends at AltLinux.
591
592\subsection{Bvfs API}
593\label{blb:sec:bvfs}
594
595To help developers of restore GUI interfaces, we have added new \textsl{dot
596  commands} that permit browsing the catalog in a very simple way.
597
598\begin{bitemize}
599\item \texttt{.bvfs\_update [jobid=x,y,z]} This command is required to update
600  the Bvfs cache in the catalog. You need to run it before any access to the
601  Bvfs layer.
602
603\item \texttt{.bvfs\_lsdirs jobid=x,y,z path=/path | pathid=101} This command
604  will list all directories in the specified \texttt{path} or
605  \texttt{pathid}. Using \texttt{pathid} avoids problems with character
606  encoding of path/filenames.
607
608\item \texttt{.bvfs\_lsfiles jobid=x,y,z path=/path | pathid=101} This command
609  will list all files in the specified \texttt{path} or \texttt{pathid}. Using
610  \texttt{pathid} avoids problems with character encoding.
611\end{bitemize}
612
613You can use \texttt{limit=xxx} and \texttt{offset=yyy} to limit the amount of
614data that will be displayed.
615
616\begin{bVerbatim}
617* .bvfs_update jobid=1,2
618* .bvfs_update
619* .bvfs_lsdir path=/ jobid=1,2
620\end{bVerbatim}
621
622This project was funded by Bacula Systems.
623
624\subsection{Testing your Tape Drive}
625\label{blb:sec:btapespeed}
626
627To determine the best configuration of your tape drive, you can run the new
628\texttt{speed} command available in the \bcommandname{btape} program.
629
630This command can have the following arguments:
631\begin{bitemize}
632\item[\texttt{file\_size=n}] Specify the Maximum File Size for this test
633  (between 1 and 5GB). This counter is in GB.
634\item[\texttt{nb\_file=n}] Specify the number of file to be written. The amount
635  of data should be greater than your memory ($file\_size*nb\_file$).
636\item[\texttt{skip\_zero}] This flag permits to skip tests with constant
637  data.
638\item[\texttt{skip\_random}] This flag permits to skip tests with random
639  data.
640\item[\texttt{skip\_raw}] This flag permits to skip tests with raw access.
641\item[\texttt{skip\_block}] This flag permits to skip tests with Bacula block
642  access.
643\end{bitemize}
644
645\begin{bVerbatim}
646*speed file_size=3 skip_raw
647btape.c:1078 Test with zero data and bacula block structure.
648btape.c:956 Begin writing 3 files of 3.221 GB with blocks of 129024 bytes.
649++++++++++++++++++++++++++++++++++++++++++
650btape.c:604 Wrote 1 EOF to "Drive-0" (/dev/nst0)
651btape.c:406 Volume bytes=3.221 GB. Write rate = 44.128 MB/s
652...
653btape.c:383 Total Volume bytes=9.664 GB. Total Write rate = 43.531 MB/s
654
655btape.c:1090 Test with random data, should give the minimum throughput.
656btape.c:956 Begin writing 3 files of 3.221 GB with blocks of 129024 bytes.
657+++++++++++++++++++++++++++++++++++++++++++
658btape.c:604 Wrote 1 EOF to "Drive-0" (/dev/nst0)
659btape.c:406 Volume bytes=3.221 GB. Write rate = 7.271 MB/s
660+++++++++++++++++++++++++++++++++++++++++++
661...
662btape.c:383 Total Volume bytes=9.664 GB. Total Write rate = 7.365 MB/s
663
664\end{bVerbatim}
665
666When using compression, the random test will give your the minimum throughput
667of your drive . The test using constant string will give you the maximum speed
668of your hardware chain. (cpu, memory, scsi card, cable, drive, tape).
669
670You can change the block size in the Storage Daemon configuration file.
671
672\subsection{New Block Checksum Device Directive}
673You may now turn off the Block Checksum (CRC32) code
674that Bacula uses when writing blocks to a Volume.  This is
675done by adding:
676
677\begin{bVerbatim}
678Block Checksum = no
679\end{bVerbatim}
680
681doing so can reduce the Storage daemon CPU usage slightly.  It
682will also permit Bacula to read a Volume that has corrupted data.
683
684The default is \bdefaultvalue{yes} -- i.e. the checksum is computed on write
685and checked on read.
686
687We do not recommend to turn this off particularly on older tape
688drives or for disk Volumes where doing so may allow corrupted data
689to go undetected.
690
691\subsection{New Bat Features}
692
693Those new features were funded by Bacula Systems.
694
695\subsubsection{Media List View}
696
697By clicking on \bog{}Media\cog{}, you can see the list of all your volumes. You will be
698able to filter by Pool, Media Type, Location,\dots And sort the result directly
699in the table. The old \bog{}Media\cog{} view is now known as \bog{}Pool\cog{}.
700\bimageH{bat-mediaview}{List volumes with BAT}{figbs4:mediaview}
701
702
703\subsubsection{Media Information View}
704
705By double-clicking on a volume (on the Media list, in the Autochanger content
706or in the Job information panel), you can access a detailed overview of your
707Volume. (cf. figure \vref{figbs4:mediainfo}.)
708
709\bimageH{bat11}{Media information}{figbs4:mediainfo}
710
711
712\subsubsection{Job Information View}
713
714By double-clicking on a Job record (on the Job run list or in the Media
715information panel), you can access a detailed overview of your Job. (cf. figure
716\vref{figbs4:jobinfo}.)
717
718\bimageH{bat12}{Job information}{figbs4:jobinfo}
719
720\subsubsection{Autochanger Content View}
721
722By double-clicking on a Storage record (on the Storage list panel), you can
723access a detailed overview of your Autochanger. (cf. figure \vref{figbs4:jobinfo}.)
724
725\bimageH{bat13}{Autochanger content}{figbs4:achcontent}
726
727To use this feature, you need to use the latest mtx-changer script
728version. (With new \texttt{listall} and \texttt{transfer} commands)
729
730\subsection{Bat on Windows}
731We have ported \bat{} to Windows and it is now installed
732by default when the installer is run.  It works quite well
733on Win32, but has not had a lot of testing there, so your
734feedback would be welcome.  Unfortunately, eventhough it is
735installed by default, it does not yet work on 64 bit Windows
736operating systems.
737
738\subsection{New Win32 Installer}
739The Win32 installer has been modified in several very important
740ways.
741\begin{bitemize}
742\item You must deinstall any current version of the
743Win32 File daemon before upgrading to the new one.
744If you forget to do so, the new installation will fail.
745To correct this failure, you must manually shutdown
746and deinstall the old File daemon.
747\item All files (other than menu links) are installed
748in \bdirectoryname{c:/Program Files/Bacula}.
749\item The installer no longer sets this
750file to require administrator privileges by default. If you want
751to do so, please do it manually using the \btool{cacls} program.
752For example:
753\begin{bVerbatim}
754cacls "C:\Program Files\Bacula" /T /G SYSTEM:F Administrators:F
755\end{bVerbatim}
756\item The server daemons (Director and Storage daemon) are
757no longer included in the Windows installer.  If you want the
758Windows servers, you will either need to build them yourself (note
759they have not been ported to 64 bits), or you can contact
760Bacula Systems about this.
761\end{bitemize}
762
763\subsection{Win64 Installer}
764We have corrected a number of problems that required manual
765editing of the conf files.  In most cases, it should now
766install and work.  \bat{} is by default installed in
767\bdirectoryname{c:/Program Files/Bacula/bin32} rather than
768\bdirectoryname{c:/Program Files/Bacula} as is the case with the 32
769bit Windows installer.
770
771\subsection{Linux Bare Metal Recovery USB Key}
772We have made a number of significant improvements in the
773Bare Metal Recovery USB key.  Please see the README file
774in the \textbf{rescue} release for more details.
775
776We are working on an equivalent USB key for Windows bare
777metal recovery, but it will take some time to develop it (best
778estimate 3Q2010 or 4Q2010)
779
780
781\subsection{bconsole Timeout Option}
782You can now use the -u option of \btool{bconsole} to set a timeout in seconds
783for commands. This is useful with GUI programs that use \btool{bconsole}
784to interface to the Director.
785
786\subsection{Important Changes}
787\label{blb:sec:importantchanges}
788
789\begin{bitemize}
790\item You are now allowed to Migrate, Copy, and Virtual Full to read and write
791  to the same Pool. The Storage daemon ensures that you do not read and
792  write to the same Volume.
793\item The \texttt{Device Poll Interval} is now 5 minutes. (previously did not
794  poll by default).
795\item Virtually all the features of \mtxchanger{} have
796  now been parameterized, which allows you to configure
797  \mtxchanger{} without changing it. There is a new configuration file
798  \bfilename{mtx-changer.conf}
799  that contains variables that you can set to configure \mtxchanger{}.
800  This configuration file will not be overwritten during upgrades.
801  We encourage you to submit any changes
802  that are made to \mtxchanger{} and to parameterize it all in
803  \bfilename{mtx-changer.conf} so that all configuration will be done by
804  changing only \bfilename{mtx-changer.conf}.
805\item The new \mtxchanger{} script has two new options, \texttt{listall}
806  and \texttt{transfer}. Please configure them as appropriate
807  in \bfilename{mtx-changer.conf}.
808\item To enhance security of the \texttt{BackupCatalog} job, we provide a new
809  script (\btool{make\_catalog\_backup.pl}) that does not expose your catalog
810  password. If you want to use the new script, you will need to
811  manually change the \texttt{BackupCatalog} Job definition.
812\item The \btool{bconsole} \bcommandname{help} command now accepts
813  an argument, which if provided produces information on that
814  command (ex: \btool{help run}).
815\end{bitemize}
816
817
818\subsubsection*{Truncate volume after purge}
819
820Note that the Truncate Volume after purge feature doesn't work as expected
821in 5.0.0 version. Please, don't use it before version 5.0.1.
822
823\subsubsection{Custom Catalog queries}
824
825If you wish to add specialized commands that list the contents of the catalog,
826you can do so by adding them to the \texttt{query.sql} file. This
827\bfilename{query.sql} file is now empty by default.  The file
828\bfilename{examples/sample-query.sql} has an a number of sample commands
829you might find useful.
830
831\subsubsection{Deprecated parts}
832
833The following items have been \textbf{deprecated} for a long time, and are now
834removed from the code.
835\begin{bitemize}
836\item Gnome console
837\item Support for SQLite 2
838\end{bitemize}
839
840\subsection{Misc Changes}
841\label{blb:sec:miscchanges}
842
843\begin{bitemize}
844\item Updated Nagios check\_bacula
845\item Updated man files
846\item Added OSX package generation script in platforms/darwin
847\item Added Spanish and Ukrainian Bacula translations
848\item Enable/disable command shows only Jobs that can change
849\item Added \bcommandname{show disabled} command to show disabled Jobs
850\item Many \acs{ACL} improvements
851\item Added Level to FD status Job output
852\item Begin Ingres DB driver (not yet working)
853\item Split RedHat spec files into bacula, bat, mtx, and docs
854\item Reorganized the manuals (fewer separate manuals)
855\item Added lock/unlock order protection in lock manager
856\item Allow 64 bit sizes for a number of variables
857\item Fixed several deadlocks or potential race conditions in the SD
858\end{bitemize}
859
860\subsection{Full Restore from a Given JobId}
861\index[general]{Restore menu}
862
863This feature allows selecting a single JobId and having Bacula
864automatically select all the other jobs that comprise a full backup up to
865and including the selected date (through JobId).
866
867Assume we start with the following jobs:
868\begin{bVerbatim}
869+-------+--------------+---------------------+-------+----------+------------+
870| jobid | client       | starttime           | level | jobfiles | jobbytes   |
871+-------+--------------+---------------------+-------+----------+------------+
872| 6     | localhost-fd | 2009-07-15 11:45:49 | I     | 2        | 0          |
873| 5     | localhost-fd | 2009-07-15 11:45:45 | I     | 15       | 44143      |
874| 3     | localhost-fd | 2009-07-15 11:45:38 | I     | 1        | 10         |
875| 1     | localhost-fd | 2009-07-15 11:45:30 | F     | 1527     | 44143073   |
876+-------+--------------+---------------------+-------+----------+------------+
877\end{bVerbatim}
878
879Below is an example of this new feature (which is number 12 in the menu).
880
881\begin{bVerbatim}
882* restore
883To select the JobIds, you have the following choices:
884     1: List last 20 Jobs run
885     2: List Jobs where a given File is saved
886...
887    12: Select full restore to a specified Job date
888    13: Cancel
889
890Select item:  (1-13): 12
891Enter JobId to get the state to restore: 5
892Selecting jobs to build the Full state at 2009-07-15 11:45:45
893You have selected the following JobIds: 1,3,5
894
895Building directory tree for JobId(s) 1,3,5 ...  +++++++++++++++++++
8961,444 files inserted into the tree.
897\end{bVerbatim}
898
899This project was funded by Bacula Systems.
900
901\subsection{Source Address}
902\index[general]{Source address}
903
904A feature has been added which allows the administrator to specify the address
905from which the Director and File daemons will establish connections.  This
906may be used to simplify system configuration overhead when working in complex
907networks utilizing multi-homing and policy-routing.
908
909To accomplish this, two new configuration directives have been implemented:
910\begin{bVerbatim}
911FileDaemon {
912  FDSourceAddress=10.0.1.20    # Always initiate connections from this address
913}
914
915Director {
916  DirSourceAddress=10.0.1.10   # Always initiate connections from this address
917}
918\end{bVerbatim}
919
920Simply adding specific host routes on the OS
921would have an undesirable side-effect: any
922application trying to contact the destination host would be forced to use the
923more specific route possibly diverting management traffic onto a backup VLAN.
924Instead of adding host routes for each client connected to a multi-homed backup
925server (for example where there are management and backup \acsp{VLAN}), one can
926use the new directives to specify a specific source address at the application
927level.
928
929Additionally, this allows the simplification and abstraction of firewall rules
930when dealing with a Hot-Standby director or storage daemon configuration.  The
931Hot-standby pair may share a CARP address, which connections must be sourced
932from, while system services listen and act from the unique interface addresses.
933
934This project was funded by Collaborative Fusion, Inc.
935
936\subsection{Show volume availability when doing restore}
937
938When doing a restore the selection dialog ends by displaying this
939screen:
940
941\begin{bVerbatim}
942The job will require the following
943   Volume(s)                 Storage(s)                SD Device(s)
944   ===========================================================================
945   *000741L3                  LTO-4                     LTO3
946   *000866L3                  LTO-4                     LTO3
947   *000765L3                  LTO-4                     LTO3
948   *000764L3                  LTO-4                     LTO3
949   *000756L3                  LTO-4                     LTO3
950   *001759L3                  LTO-4                     LTO3
951   *001763L3                  LTO-4                     LTO3
952    001762L3                  LTO-4                     LTO3
953    001767L3                  LTO-4                     LTO3
954
955Volumes marked with \bog{}*\cog{} are online (in the autochanger).
956\end{bVerbatim}
957
958This should help speed up large restores by minimizing the time spent
959waiting for the operator to discover that he must change tapes in the library.
960
961This project was funded by Bacula Systems.
962
963\subsection{Accurate estimate command}
964
965The \bcommandname{estimate} command can now use the accurate code to detect changes
966and give a better estimation.
967
968You can set the accurate behavior on the command line by using
969\texttt{accurate=\byesno{}} or use the Job setting as default value.
970
971\begin{bVerbatim}
972* estimate listing accurate=yes level=incremental job=BackupJob
973\end{bVerbatim}
974
975This project was funded by Bacula Systems.
976
977\subsection{Accurate Backup}
978\index[general]{Accurate backup}
979
980As with most other backup programs, by default Bacula decides what files to
981backup for Incremental and Differental backup by comparing the change
982(st\_ctime) and modification (\texttt{st\_mtime}) times of the file to the time the last
983backup completed.  If one of those two times is later than the last backup
984time, then the file will be backed up.  This does not, however, permit tracking
985what files have been deleted and will miss any file with an old time that may
986have been restored to or moved onto the client filesystem.
987
988\subsubsection{Accurate = \byesorno{}}
989If the \bdirectivename{Accurate} directive is enabled (default no) in
990the Job resource, the job will be run as an Accurate Job. For a \textbf{Full}
991backup, there is no difference, but for \textbf{Differential} and
992\textbf{Incremental} backups, the Director will send a list of all previous files
993backed up, and the File daemon will use that list to determine if any new files
994have been added or or moved and if any files have been deleted. This allows
995Bacula to make an accurate backup of your system to that point in time so that
996if you do a restore, it will restore your system exactly.
997
998One note of caution
999about using Accurate backup is that it requires more resources (CPU and memory)
1000on both the Director and the Client machines to create the list of previous
1001files backed up, to send that list to the File daemon, for the File daemon to
1002keep the list (possibly very big) in memory, and for the File daemon to do
1003comparisons between every file in the FileSet and the list.  In particular,
1004if your client has lots of files (more than a few million), you will need
1005lots of memory on the client machine.
1006
1007Accurate must not be enabled when backing up with a plugin that is not
1008specially designed to work with Accurate. If you enable it, your restores
1009will probably not work correctly.
1010
1011This project was funded by Bacula Systems.
1012
1013
1014
1015\subsection{Copy Jobs}
1016\index[general]{Copy jobs}
1017
1018A new \textbf{Copy} job type 'C' has been implemented. It is similar to the
1019existing Migration feature with the exception that the Job that is copied is
1020left unchanged.  This essentially creates two identical copies of the same
1021backup. However, the copy is treated as a copy rather than a backup job, and
1022hence is not directly available for restore.  The \bcommandname{restore} command lists
1023copy jobs and allows selection of copies by using \texttt{jobid=}
1024option. If the keyword \textbf{copies} is present on the command line, Bacula will
1025display the list of all copies for selected jobs.
1026
1027\begin{bVerbatim}
1028* restore copies
1029[...]
1030These JobIds have copies as follows:
1031+-------+------------------------------------+-----------+------------------+
1032| JobId | Job                                | CopyJobId | MediaType        |
1033+-------+------------------------------------+-----------+------------------+
1034| 2     | CopyJobSave.2009-02-17_16.31.00.11 | 7         | DiskChangerMedia |
1035+-------+------------------------------------+-----------+------------------+
1036+-------+-------+----------+----------+---------------------+------------------+
1037| JobId | Level | JobFiles | JobBytes | StartTime           | VolumeName       |
1038+-------+-------+----------+----------+---------------------+------------------+
1039| 19    | F     | 6274     | 76565018 | 2009-02-17 16:30:45 | ChangerVolume002 |
1040| 2     | I     | 1        | 5        | 2009-02-17 16:30:51 | FileVolume001    |
1041+-------+-------+----------+----------+---------------------+------------------+
1042You have selected the following JobIds: 19,2
1043
1044Building directory tree for JobId(s) 19,2 ...  ++++++++++++++++++++++++++++++++++++++++++++
10455,611 files inserted into the tree.
1046...
1047\end{bVerbatim}
1048
1049
1050The Copy Job runs without using the File daemon by copying the data from the
1051old backup Volume to a different Volume in a different Pool. See the Migration
1052documentation for additional details. For copy Jobs there is a new selection
1053directive named \textbf{PoolUncopiedJobs} which selects all Jobs that were
1054not already copied to another Pool.
1055
1056As with Migration, the Client, Volume, Job, or SQL query, are
1057other possible ways of selecting the Jobs to be copied. Selection
1058types like SmallestVolume, OldestVolume, PoolOccupancy and PoolTime also
1059work, but are probably more suited for Migration Jobs.
1060
1061If Bacula finds a Copy of a job record that is purged (deleted) from the catalog,
1062it will promote the Copy to a \textsl{real} backup job and will make it available for
1063automatic restore. If more than one Copy is available, it will promote the copy
1064with the smallest JobId.
1065
1066A nice solution which can be built with the new Copy feature is often
1067called disk-to-disk-to-tape backup (DTDTT). A sample config could
1068look something like the one below:
1069
1070\begin{bVerbatim}
1071Pool {
1072  Name = FullBackupsVirtualPool
1073  Pool Type = Backup
1074  Purge Oldest Volume = Yes
1075  Storage = vtl
1076  NextPool = FullBackupsTapePool
1077}
1078
1079Pool {
1080  Name = FullBackupsTapePool
1081  Pool Type = Backup
1082  Recycle = Yes
1083  AutoPrune = Yes
1084  Volume Retention = 365 days
1085  Storage = superloader
1086}
1087
1088#
1089# Fake fileset for copy jobs
1090#
1091Fileset {
1092  Name = None
1093  Include {
1094    Options {
1095      signature = MD5
1096    }
1097  }
1098}
1099
1100#
1101# Fake client for copy jobs
1102#
1103Client {
1104  Name = None
1105  Address = localhost
1106  Password = "NoNe"
1107  Catalog = MyCatalog
1108}
1109
1110#
1111# Default template for a CopyDiskToTape Job
1112#
1113JobDefs {
1114  Name = CopyDiskToTape
1115  Type = Copy
1116  Messages = StandardCopy
1117  Client = None
1118  FileSet = None
1119  Selection Type = PoolUncopiedJobs
1120  Maximum Concurrent Jobs = 10
1121  SpoolData = No
1122  Allow Duplicate Jobs = Yes
1123  Cancel Queued Duplicates = No
1124  Cancel Running Duplicates = No
1125  Priority = 13
1126}
1127
1128Schedule {
1129   Name = DaySchedule7:00
1130   Run = Level=Full daily at 7:00
1131}
1132
1133Job {
1134  Name = CopyDiskToTapeFullBackups
1135  Enabled = Yes
1136  Schedule = DaySchedule7:00
1137  Pool = FullBackupsVirtualPool
1138  JobDefs = CopyDiskToTape
1139}
1140\end{bVerbatim}
1141
1142The example above had 2 pool which are copied using the PoolUncopiedJobs
1143selection criteria. Normal Full backups go to the Virtual pool and are copied
1144to the Tape pool the next morning.
1145
1146The command \texttt{list copies [jobid=x,y,z]} lists copies for a given
1147\textbf{jobid}.
1148
1149\begin{bVerbatim}
1150*list copies
1151+-------+------------------------------------+-----------+------------------+
1152| JobId | Job                                | CopyJobId | MediaType        |
1153+-------+------------------------------------+-----------+------------------+
1154|     9 | CopyJobSave.2008-12-20_22.26.49.05 |        11 | DiskChangerMedia |
1155+-------+------------------------------------+-----------+------------------+
1156\end{bVerbatim}
1157
1158\subsection{ACL Updates}
1159\index[general]{ACL updates}
1160The whole ACL code had been overhauled and in this version each platforms has
1161different streams for each type of acl available on such an platform. As ACLs
1162between platforms tend to be not that portable (most implement POSIX acls but
1163some use an other draft or a completely different format) we currently only
1164allow certain platform specific ACL streams to be decoded and restored on the
1165same platform that they were created on.  The old code allowed to restore ACL
1166cross platform but the comments already mention that not being to wise. For
1167backward compatability the new code will accept the two old ACL streams and
1168handle those with the platform specific handler. But for all new backups it
1169will save the ACLs using the new streams.
1170
1171Currently the following platforms support ACLs:
1172
1173\begin{bitemize}
1174 \item \textbf{AIX}
1175 \item \textbf{Darwin/OSX}
1176 \item \textbf{FreeBSD}
1177 \item \textbf{HPUX}
1178 \item \textbf{IRIX}
1179 \item \textbf{Linux}
1180 \item \textbf{Tru64}
1181 \item \textbf{Solaris}
1182\end{bitemize}
1183
1184Currently we support the following ACL types (these ACL streams use a reserved
1185part of the stream numbers):
1186
1187\begin{bitemize}
1188\item \textbf{STREAM\_ACL\_AIX\_TEXT} 1000 AIX specific string representation from
1189  acl\_get
1190 \item \textbf{STREAM\_ACL\_DARWIN\_ACCESS\_ACL} 1001 Darwin (OSX) specific acl\_t
1191   string representation from acl\_to\_text (POSIX acl)
1192  \item \textbf{STREAM\_ACL\_FREEBSD\_DEFAULT\_ACL} 1002 FreeBSD specific acl\_t
1193    string representation from acl\_to\_text (POSIX acl) for default acls.
1194  \item \textbf{STREAM\_ACL\_FREEBSD\_ACCESS\_ACL} 1003 FreeBSD specific acl\_t
1195    string representation from acl\_to\_text (POSIX acl) for access acls.
1196  \item \textbf{STREAM\_ACL\_HPUX\_ACL\_ENTRY} 1004 HPUX specific acl\_entry
1197    string representation from acltostr (POSIX acl)
1198  \item \textbf{STREAM\_ACL\_IRIX\_DEFAULT\_ACL} 1005 IRIX specific acl\_t string
1199    representation from acl\_to\_text (POSIX acl) for default acls.
1200  \item \textbf{STREAM\_ACL\_IRIX\_ACCESS\_ACL} 1006 IRIX specific acl\_t string
1201    representation from acl\_to\_text (POSIX acl) for access acls.
1202  \item \textbf{STREAM\_ACL\_LINUX\_DEFAULT\_ACL} 1007 Linux specific acl\_t
1203    string representation from acl\_to\_text (POSIX acl) for default acls.
1204  \item \textbf{STREAM\_ACL\_LINUX\_ACCESS\_ACL} 1008 Linux specific acl\_t string
1205    representation from acl\_to\_text (POSIX acl) for access acls.
1206  \item \textbf{STREAM\_ACL\_TRU64\_DEFAULT\_ACL} 1009 Tru64 specific acl\_t
1207    string representation from acl\_to\_text (POSIX acl) for default acls.
1208  \item \textbf{STREAM\_ACL\_TRU64\_DEFAULT\_DIR\_ACL} 1010 Tru64 specific acl\_t
1209    string representation from acl\_to\_text (POSIX acl) for default acls.
1210  \item \textbf{STREAM\_ACL\_TRU64\_ACCESS\_ACL} 1011 Tru64 specific acl\_t string
1211    representation from acl\_to\_text (POSIX acl) for access acls.
1212  \item \textbf{STREAM\_ACL\_SOLARIS\_ACLENT} 1012 Solaris specific aclent\_t
1213    string representation from acltotext or acl\_totext (POSIX acl)
1214  \item \textbf{STREAM\_ACL\_SOLARIS\_ACE} 1013 Solaris specific ace\_t string
1215    representation from from acl\_totext (NFSv4 or ZFS acl)
1216\end{bitemize}
1217
1218In future versions we might support conversion functions from one type of acl
1219into an other for types that are either the same or easily convertable. For now
1220the streams are seperate and restoring them on a platform that doesn't
1221recognize them will give you a warning.
1222
1223\subsection{Extended Attributes}
1224\index[general]{Extended attributes}
1225Something that was on the project list for some time is now implemented for
1226platforms that support a similar kind of interface. Its the support for backup
1227and restore of so called extended attributes. As extended attributes are so
1228platform specific these attributes are saved in seperate streams for each
1229platform.  Restores of the extended attributes can only be performed on the
1230same platform the backup was done.  There is support for all types of extended
1231attributes, but restoring from one type of filesystem onto an other type of
1232filesystem on the same platform may lead to supprises.  As extended attributes
1233can contain any type of data they are stored as a series of so called
1234value-pairs.  This data must be seen as mostly binary and is stored as such.
1235As security labels from selinux are also extended attributes this option also
1236stores those labels and no specific code is enabled for handling selinux
1237security labels.
1238
1239Currently the following platforms support extended attributes:
1240\begin{bitemize}
1241 \item \textbf{Darwin/OSX}
1242 \item \textbf{FreeBSD}
1243 \item \textbf{Linux}
1244 \item \textbf{NetBSD}
1245\end{bitemize}
1246
1247On linux acls are also extended attributes, as such when you enable ACLs on a
1248Linux platform it will NOT save the same data twice e.g. it will save the ACLs
1249and not the same exteneded attribute.
1250
1251To enable the backup of extended attributes please add the following to your
1252fileset definition.
1253\begin{bVerbatim}
1254  FileSet {
1255    Name = "MyFileSet"
1256    Include {
1257      Options {
1258        signature = MD5
1259        xattrsupport = yes
1260      }
1261      File = ...
1262    }
1263  }
1264\end{bVerbatim}
1265
1266\subsection{Shared objects}
1267\index[general]{Shared objects}
1268A default build of Bacula will now create the libraries as shared objects
1269(.so) rather than static libraries as was previously the case.
1270The shared libraries are built using \textbf{libtool} so it should be quite
1271portable.
1272
1273An important advantage of using shared objects is that on a machine with the
1274Directory, File daemon, the Storage daemon, and a console, you will have only
1275one copy of the code in memory rather than four copies.  Also the total size of
1276the binary release is smaller since the library code appears only once rather
1277than once for every program that uses it; this results in significant reduction
1278in the size of the binaries particularly for the utility tools.
1279
1280In order for the system loader to find the shared objects when loading the
1281Bacula binaries, the Bacula shared objects must either be in a shared object
1282directory known to the loader (typically /usr/lib) or they must be in the
1283directory that may be specified on the \btool{./configure} line using the
1284\textbf{{-}{-}libdir} option as:
1285
1286\begin{bVerbatim}
1287./configure --libdir=/full-path/dir
1288\end{bVerbatim}
1289
1290the default is \bdefaultvalue{/usr/lib}. If \texttt{{-}{-}libdir}
1291is specified, there should be
1292no need to modify your loader configuration provided that
1293the shared objects are installed in that directory (Bacula
1294does this with the make install command). The shared objects
1295that Bacula references are:
1296
1297\begin{bVerbatim}
1298libbaccfg.so
1299libbacfind.so
1300libbacpy.so
1301libbac.so
1302\end{bVerbatim}
1303
1304These files are symbolically linked to the real shared object file,
1305which has a version number to permit running multiple versions of
1306the libraries if desired (not normally the case).
1307
1308If you have problems with libtool or you wish to use the old
1309way of building static libraries, or you want to build a static
1310version of Bacula you may disable
1311libtool on the configure command line with:
1312
1313\begin{bVerbatim}
1314  ./configure --disable-libtool
1315\end{bVerbatim}
1316
1317
1318\subsection{Building Static versions of Bacula}
1319\index[general]{Static linking}
1320In order to build static versions of Bacula, in addition
1321to configuration options that were needed you now must
1322also add \texttt{\-\-disable-libtool}.  Example
1323
1324\begin{bVerbatim}
1325./configure --enable-static-client-only --disable-libtool
1326\end{bVerbatim}
1327
1328
1329\subsection{Virtual Backup (Vbackup)}
1330\index[general]{Virtual backup}
1331\index[general]{Vbackup}
1332
1333Bacula's virtual backup feature is often called Synthetic Backup or
1334Consolidation in other backup products.  It permits you to consolidate the
1335previous Full backup plus the most recent Differential backup and any
1336subsequent Incremental backups into a new Full backup.  This new Full
1337backup will then be considered as the most recent Full for any future
1338Incremental or Differential backups.  The VirtualFull backup is
1339accomplished without contacting the client by reading the previous backup
1340data and writing it to a volume in a different pool.
1341
1342In some respects the Vbackup feature works similar to a Migration job, in
1343that Bacula normally reads the data from the pool specified in the
1344Job resource, and writes it to the \textbf{Next Pool} specified in the
1345Job resource. Note, this means that usually the output from the Virtual
1346Backup is written into a different pool from where your prior backups
1347are saved. Doing it this way guarantees that you will not get a deadlock
1348situation attempting to read and write to the same volume in the Storage
1349daemon. If you then want to do subsequent backups, you may need to
1350move the Virtual Full Volume back to your normal backup pool.
1351Alternatively, you can set your \textbf{Next Pool} to point to the current
1352pool.  This will cause Bacula to read and write to Volumes in the
1353current pool. In general, this will work, because Bacula will
1354not allow reading and writing on the same Volume. In any case, once
1355a VirtualFull has been created, and a restore is done involving the
1356most current Full, it will read the Volume or Volumes by the VirtualFull
1357regardless of in which Pool the Volume is found.
1358
1359The Vbackup is enabled on a Job by Job in the Job resource by specifying
1360a level of \textbf{VirtualFull}.
1361
1362A typical Job resource definition might look like the following:
1363
1364\begin{bVerbatim}
1365Job {
1366  Name = "MyBackup"
1367  Type = Backup
1368  Client=localhost-fd
1369  FileSet = "Full Set"
1370  Storage = File
1371  Messages = Standard
1372  Pool = Default
1373  SpoolData = yes
1374}
1375
1376# Default pool definition
1377Pool {
1378  Name = Default
1379  Pool Type = Backup
1380  Recycle = yes            # Automatically recycle Volumes
1381  AutoPrune = yes          # Prune expired volumes
1382  Volume Retention = 365d  # one year
1383  NextPool = Full
1384  Storage = File
1385}
1386
1387Pool {
1388  Name = Full
1389  Pool Type = Backup
1390  Recycle = yes            # Automatically recycle Volumes
1391  AutoPrune = yes          # Prune expired volumes
1392  Volume Retention = 365d  # one year
1393  Storage = DiskChanger
1394}
1395
1396# Definition of file storage device
1397Storage {
1398  Name = File
1399  Address = localhost
1400  Password = "xxx"
1401  Device = FileStorage
1402  Media Type = File
1403  Maximum Concurrent Jobs = 5
1404}
1405
1406# Definition of DDS Virtual tape disk storage device
1407Storage {
1408  Name = DiskChanger
1409  Address = localhost  # N.B. Use a fully qualified name here
1410  Password = "yyy"
1411  Device = DiskChanger
1412  Media Type = DiskChangerMedia
1413  Maximum Concurrent Jobs = 4
1414  Autochanger = yes
1415}
1416\end{bVerbatim}
1417
1418Then in bconsole or via a Run schedule, you would run the job as:
1419
1420\begin{bVerbatim}
1421run job=MyBackup level=Full
1422run job=MyBackup level=Incremental
1423run job=MyBackup level=Differential
1424run job=MyBackup level=Incremental
1425run job=MyBackup level=Incremental
1426\end{bVerbatim}
1427
1428So providing there were changes between each of those jobs, you would end up
1429with a Full backup, a Differential, which includes the first Incremental
1430backup, then two Incremental backups.  All the above jobs would be written to
1431the \textbf{Default} pool.
1432
1433To consolidate those backups into a new Full backup, you would run the
1434following:
1435
1436\begin{bVerbatim}
1437run job=MyBackup level=VirtualFull
1438\end{bVerbatim}
1439
1440And it would produce a new Full backup without using the client, and the output
1441would be written to the \textbf{Full} Pool which uses the Diskchanger Storage.
1442
1443If the Virtual Full is run, and there are no prior Jobs, the Virtual Full will
1444fail with an error.
1445
1446Note, the Start and End time of the Virtual Full backup is set to the
1447values for the last job included in the Virtual Full (in the above example,
1448it is an Increment). This is so that if another incremental is done, which
1449will be based on the Virtual Full, it will backup all files from the
1450last Job included in the Virtual Full rather than from the time the Virtual
1451Full was actually run.
1452
1453
1454
1455\subsection{Catalog Format}
1456\index[general]{Catalog format}
1457Bacula 3.0 comes with some changes to the catalog format.  The upgrade
1458operation will convert the FileId field of the File table from 32 bits (max 4
1459billion table entries) to 64 bits (very large number of items).  The
1460conversion process can take a bit of time and will likely DOUBLE THE SIZE of
1461your catalog during the conversion.  Also you won't be able to run jobs during
1462this conversion period.  For example, a 3 million file catalog will take 2
1463minutes to upgrade on a normal machine.  Please don't forget to make a valid
1464backup of your database before executing the upgrade script. See the
1465ReleaseNotes for additional details.
1466
1467\subsection{64 bit Windows Client}
1468\index[general]{Win64 client}
1469Unfortunately, Microsoft's implementation of Volume Shadown Copy (VSS) on
1470their 64 bit OS versions is not compatible with a 32 bit Bacula Client.
1471As a consequence, we are also releasing a 64 bit version of the Bacula
1472Windows Client (win64bacula-3.0.0.exe) that does work with VSS.
1473These binaries should only be installed on 64 bit Windows operating systems.
1474What is important is not your hardware but whether or not you have
1475a 64 bit version of the Windows OS.
1476
1477Compared to the Win32 Bacula Client, the 64 bit release contains a few differences:
1478\begin{benumerate}
1479\item Before installing the Win64 Bacula Client, you must totally
1480      deinstall any prior 2.4.x Client installation using the
1481      Bacula deinstallation (see the menu item). You may want
1482      to save your .conf files first.
1483\item Only the Client (File daemon) is ported to Win64, the Director
1484      and the Storage daemon are not in the 64 bit Windows installer.
1485\item bwx-console is not yet ported.
1486\item bconsole is ported but it has not been tested.
1487\item The documentation is not included in the installer.
1488\item Due to Vista security restrictions imposed on a default installation
1489      of Vista, before upgrading the Client, you must manually stop
1490      any prior version of Bacula from running, otherwise the install
1491      will fail.
1492\item Due to Vista security restrictions imposed on a default installation
1493      of Vista, attempting to edit the conf files via the menu items
1494      will fail. You must directly edit the files with appropriate
1495      permissions.  Generally double clicking on the appropriate .conf
1496      file will work providing you have sufficient permissions.
1497\item All Bacula files are now installed in
1498      \bdirectoryname{C:/Program Files/Bacula} except the main menu items,
1499      which are installed as before. This vastly simplifies the installation.
1500\item If you are running on a foreign language version of Windows, most
1501      likely \bdirectoryname{C:/Program Files} does not exist, so you should use the
1502      Custom installation and enter an appropriate location to install
1503      the files.
1504\item The 3.0.0 Win32 Client continues to install files in the locations used
1505      by prior versions. For the next version we will convert it to use
1506      the same installation conventions as the Win64 version.
1507\end{benumerate}
1508
1509This project was funded by Bacula Systems.
1510
1511
1512\subsection{Duplicate Job Control}
1513\index[general]{Duplicate jobs}
1514The new version of Bacula provides four new directives that
1515give additional control over what Bacula does if duplicate jobs
1516are started.  A duplicate job in the sense we use it here means
1517a second or subsequent job with the same name starts.  This
1518happens most frequently when the first job runs longer than expected because no
1519tapes are available.
1520
1521The four directives each take as an argument a \textbf{yes} or \textbf{no} value and
1522are specified in the Job resource.
1523
1524They are:
1525
1526\subsubsection{Allow Duplicate Jobs = \byesorno{}}
1527\index[general]{Allow Duplicate Jobs}
1528  If this directive is set to \textbf{yes}, duplicate jobs will be run.  If
1529  the directive is set to \bdefaultvalue{no} (default) then only one job of a given name
1530  may run at one time, and the action that Bacula takes to ensure only
1531  one job runs is determined by the other directives (see below).
1532
1533  If \textbf{Allow Duplicate Jobs} is set to \textbf{no} and two jobs
1534  are present and none of the three directives given below permit
1535  cancelling a job, then the current job (the second one started)
1536  will be cancelled.
1537
1538\subsubsection{Allow Higher Duplicates = \byesorno{}}
1539\index[general]{Allow Higher Duplicates}
1540  This directive was in version 5.0.0, but does not work as
1541  expected. If used, it should always be set to no.  In later versions
1542  of Bacula the directive is disabled (disregarded).
1543
1544\subsubsection{Cancel Running Duplicates = \byesorno{}}
1545\index[general]{Cancel Running Duplicates}
1546  If \textbf{Allow Duplicate Jobs} is set to \textbf{no} and
1547  if this directive is set to \textbf{yes} any job that is already running
1548  will be canceled.  The default is \bdefaultvalue{no}.
1549
1550\subsubsection{Cancel Queued Duplicates = \byesorno{}}
1551\index[general]{Cancel Queued Duplicates}
1552  If \textbf{Allow Duplicate Jobs} is set to \textbf{no} and
1553  if this directive is set to \textbf{yes} any job that is
1554  already queued to run but not yet running will be canceled.
1555  The default is \bdefaultvalue{no}.
1556
1557
1558\subsection{TLS Authentication}
1559\index[general]{TLS authentication}
1560In Bacula version 2.5.x and later, in addition to the normal Bacula
1561CRAM-MD5 authentication that is used to authenticate each Bacula
1562connection, you can specify that you want TLS Authentication as well,
1563which will provide more secure authentication.
1564
1565This new feature uses Bacula's existing TLS code (normally used for
1566communications encryption) to do authentication.  To use it, you must
1567specify all the TLS directives normally used to enable communications
1568encryption (TLS Enable, TLS Verify Peer, TLS Certificate, \ldots{}) and
1569a new directive:
1570
1571\subsubsection{TLS Authenticate = \byesorno{}}
1572\begin{bVerbatim}
1573TLS Authenticate = yes
1574\end{bVerbatim}
1575
1576in the main daemon configuration resource (Director for the Director,
1577Client for the File daemon, and Storage for the Storage daemon).
1578
1579When \textbf{TLS Authenticate} is enabled, after doing the CRAM-MD5
1580authentication, Bacula will also do TLS authentication, then TLS
1581encryption will be turned off, and the rest of the communication between
1582the two Bacula daemons will be done without encryption.
1583
1584If you want to encrypt communications data, use the normal TLS directives
1585but do not turn on \textbf{TLS Authenticate}.
1586
1587\subsection{bextract non-portable Win32 data}
1588\index[general]{bextract handles Win32 non-portable data}
1589\index[general]{Win32!bextract handles non-portable data}
1590\btool{bextract} has been enhanced to be able to restore
1591non-portable Win32 data to any OS.  Previous versions were
1592unable to restore non-portable Win32 data to machines that
1593did not have the Win32 BackupRead and BackupWrite API calls.
1594
1595\subsection{State File updated at Job Termination}
1596\index[general]{State file}
1597In previous versions of Bacula, the state file, which provides a
1598summary of previous jobs run in the \bcommandname{status} command output was
1599updated only when Bacula terminated, thus if the daemon crashed, the
1600state file might not contain all the run data.  This version of
1601the Bacula daemons updates the state file on each job termination.
1602
1603\subsection{MaxFullInterval = \bbracket{time-interval}}
1604\index[general]{MaxFullInterval}
1605The new Job resource directive \textbf{Max Full Interval = \bbracket{time-interval}}
1606can be used to specify the maximum time interval between \textbf{Full} backup
1607jobs. When a job starts, if the time since the last Full backup is
1608greater than the specified interval, and the job would normally be an
1609\textbf{Incremental} or \textbf{Differential}, it will be automatically
1610upgraded to a \textbf{Full} backup.
1611
1612\subsection{MaxDiffInterval = \bbracket{time-interval}}
1613\index[general]{MaxDiffInterval}
1614The new Job resource directive \textbf{Max Diff Interval = \bbracket{time-interval}}
1615can be used to specify the maximum time interval between \textbf{Differential} backup
1616jobs. When a job starts, if the time since the last Differential backup is
1617greater than the specified interval, and the job would normally be an
1618\textbf{Incremental}, it will be automatically
1619upgraded to a \textbf{Differential} backup.
1620
1621\subsection{Honor No Dump Flag = \byesorno{}}
1622\index[general]{MaxDiffInterval}
1623On FreeBSD systems, each file has a \textbf{no dump flag} that can be set
1624by the user, and when it is set it is an indication to backup programs
1625to not backup that particular file.  This version of Bacula contains a
1626new Options directive within a FileSet resource, which instructs Bacula to
1627obey this flag.  The new directive is:
1628
1629\begin{bVerbatim}
1630  Honor No Dump Flag = <yes|no>
1631\end{bVerbatim}
1632
1633The default value is \bdefaultvalue{no}.
1634
1635
1636\subsection{Exclude Dir Containing = \bbracket{filename-string}}
1637\index[general]{IgnoreDir}
1638The \textbf{ExcludeDirContaining = \bbracket{filename}} is a new directive that
1639can be added to the Include section of the FileSet resource.  If the specified
1640filename (\textbf{filename-string}) is found on the Client in any directory to be
1641backed up, the whole directory will be ignored (not backed up).  For example:
1642
1643\begin{bVerbatim}
1644  # List of files to be backed up
1645  FileSet {
1646    Name = "MyFileSet"
1647    Include {
1648      Options {
1649        signature = MD5
1650      }
1651      File = /home
1652      Exclude Dir Containing = .excludeme
1653    }
1654  }
1655\end{bVerbatim}
1656
1657But in /home, there may be hundreds of directories of users and some
1658people want to indicate that they don't want to have certain
1659directories backed up. For example, with the above FileSet, if
1660the user or sysadmin creates a file named \bfilename{.excludeme} in
1661specific directories, such as
1662
1663\begin{bVerbatim}
1664   /home/user/www/cache/.excludeme
1665   /home/user/temp/.excludeme
1666\end{bVerbatim}
1667
1668then Bacula will not backup the two directories named:
1669
1670\begin{bVerbatim}
1671   /home/user/www/cache
1672   /home/user/temp
1673\end{bVerbatim}
1674
1675NOTE: subdirectories will not be backed up.  That is, the directive
1676applies to the two directories in question and any children (be they
1677files, directories, etc).
1678
1679\subsubsection{bfuncs}
1680The bFuncs structure defines the callback entry points within Bacula
1681that the plugin can use register events, get Bacula values, set
1682Bacula values, and send messages to the Job output or debug output.
1683
1684The exact definition as of this writing is:
1685\begin{bVerbatim}
1686typedef struct s_baculaFuncs {
1687   uint32_t size;
1688   uint32_t version;
1689   bRC (*registerBaculaEvents)(bpContext *ctx, ...);
1690   bRC (*getBaculaValue)(bpContext *ctx, bVariable var, void *value);
1691   bRC (*setBaculaValue)(bpContext *ctx, bVariable var, void *value);
1692   bRC (*JobMessage)(bpContext *ctx, const char *file, int line,
1693       int type, utime_t mtime, const char *fmt, ...);
1694   bRC (*DebugMessage)(bpContext *ctx, const char *file, int line,
1695       int level, const char *fmt, ...);
1696   void *(*baculaMalloc)(bpContext *ctx, const char *file, int line,
1697       size_t size);
1698   void (*baculaFree)(bpContext *ctx, const char *file, int line, void *mem);
1699
1700   /* New functions follow */
1701   bRC (*AddExclude)(bpContext *ctx, const char *file);
1702   bRC (*AddInclude)(bpContext *ctx, const char *file);
1703   bRC (*AddIncludeOptions)(bpContext *ctx, const char *opts);
1704   bRC (*AddRegexToInclude)(bpContext *ctx, const char *item, int type);
1705   bRC (*AddWildToInclude)(bpContext *ctx, const char *item, int type);
1706
1707} bFuncs;
1708\end{bVerbatim}
1709
1710\begin{bdescription}
1711\item [AddExclude] can be called to exclude a file. The file
1712  string passed may include wildcards that will be interpreted by
1713  the \textbf{fnmatch} subroutine. This function can be called
1714  multiple times, and each time the file specified will be added
1715  to the list of files to be excluded. Note, this function only
1716  permits adding excludes of specific file or directory names,
1717  or files matched by the rather simple fnmatch mechanism.
1718  See below for information on doing wild-card and regex excludes.
1719
1720\item [NewInclude] can be called to create a new Include block. This
1721  block will be added before any user defined Include blocks. This
1722  function can be called multiple times, but each time, it will create
1723  a new Include section (not normally needed). This function should
1724  be called only if you want to add an entirely new Include block.
1725
1726\item [AddInclude] can be called to add new files/directories to
1727  be included.  They are added to the current Include block. If
1728  NewInclude has not been included, the current Include block is
1729  the last one that the user created. This function
1730  should be used only if you want to add totally new files/directories
1731  to be included in the backup.
1732
1733\item [NewOptions] adds a new Options block to the current Include
1734  in front of any other Options blocks. This permits the plugin to
1735  add exclude directives (wild-cards and regexes) in front of the
1736  user Options, and thus prevent certain files from being backed up.
1737  This can be useful if the plugin backs up files, and they should
1738  not be also backed up by the main Bacula code.  This function
1739  may be called multiple times, and each time, it creates a new
1740  prepended Options block. Note: normally you want to call this
1741  entry point prior to calling AddOptions, AddRegex, or AddWild.
1742
1743\item [AddOptions] allows the plugin it set options in
1744  the current Options block, which is normally created with the
1745  NewOptions call just prior to adding Include Options.
1746  The permitted options are passed as a character string, where
1747  each character has a specific meaning as defined below:
1748
1749  \begin{bdescription}
1750  \item [a] always replace files (default).
1751  \item [e] exclude rather than include.
1752  \item [h] no recursion into subdirectories.
1753  \item [H] do not handle hard links.
1754  \item [i] ignore case in wildcard and regex matches.
1755  \item [M] compute an MD5 sum.
1756  \item [p] use a portable data format on Windows (not recommended).
1757  \item [R] backup resource forks and Findr Info.
1758  \item [r] read from a fifo
1759  \item [S1] compute an SHA1 sum.
1760  \item [S2] compute an SHA256 sum.
1761  \item [S3] comput an SHA512 sum.
1762  \item [s] handle sparse files.
1763  \item [m] use st\_mtime only for file differences.
1764  \item [k] restore the st\_atime after accessing a file.
1765  \item [A] enable ACL backup.
1766  \item [Vxxx:] specify verify options. Must terminate with :
1767  \item [Cxxx:] specify accurate options. Must terminate with :
1768  \item [Jxxx:] specify base job Options. Must terminate with :
1769  \item [Pnnn:] specify integer nnn paths to strip. Must terminate with :
1770  \item [w] if newer
1771  \item [Zn] specify gzip compression level n.
1772  \item [K] do not use st\_atime in backup decision.
1773  \item [c] check if file changed during backup.
1774  \item [N] honor no dump flag.
1775  \item [X] enable backup of extended attributes.
1776  \end{bdescription}
1777
1778\item [AddRegex] adds a regex expression to the current Options block.
1779  The fillowing options are permitted:
1780  \begin{bdescription}
1781  \item [ ] (a blank) regex applies to whole path and filename.
1782  \item [F] regex applies only to the filename (directory or path stripped).
1783  \item [D] regex applies only to the directory (path) part of the name.
1784  \end{bdescription}
1785
1786\item [AddWild] adds a wildcard expression to the current Options block.
1787  The fillowing options are permitted:
1788  \begin{bdescription}
1789  \item [ ] (a blank) regex applies to whole path and filename.
1790  \item [F] regex applies only to the filename (directory or path stripped).
1791  \item [D] regex applies only to the directory (path) part of the name.
1792  \end{bdescription}
1793
1794\end{bdescription}
1795
1796
1797\subsubsection{Bacula events}
1798The list of events has been extended to include:
1799
1800\begin{bVerbatim}
1801typedef enum {
1802  bEventJobStart        = 1,
1803  bEventJobEnd          = 2,
1804  bEventStartBackupJob  = 3,
1805  bEventEndBackupJob    = 4,
1806  bEventStartRestoreJob = 5,
1807  bEventEndRestoreJob   = 6,
1808  bEventStartVerifyJob  = 7,
1809  bEventEndVerifyJob    = 8,
1810  bEventBackupCommand   = 9,
1811  bEventRestoreCommand  = 10,
1812  bEventLevel           = 11,
1813  bEventSince           = 12,
1814
1815  /* New events */
1816  bEventCancelCommand                   = 13,
1817  bEventVssBackupAddComponents          = 14,
1818  bEventVssRestoreLoadComponentMetadata = 15,
1819  bEventVssRestoreSetComponentsSelected = 16,
1820  bEventRestoreObject                   = 17,
1821  bEventEndFileSet                      = 18,
1822  bEventPluginCommand                   = 19
1823
1824} bEventType;
1825\end{bVerbatim}
1826
1827\begin{bdescription}
1828\item [bEventCancelCommand] is called whenever the currently
1829  running Job is cancelled
1830
1831\item [bEventVssBackupAddComponents]
1832\item [bEventPluginCommand] is called for each PluginCommand present in the
1833  current FileSet. The event will be sent only on plugin specifed in the
1834  command. The argument is the PluginCommand (read-only).
1835\end{bdescription}
1836
1837
1838\subsection{Bacula Plugins}
1839\index[general]{Plugin}
1840Support for shared object plugins has been implemented in the Linux, Unix
1841and Win32 File daemons. The API will be documented separately in
1842the Developer's Guide or in a new document.  For the moment, there is
1843a single plugin named \btool{bpipe} that allows an external program to
1844get control to backup and restore a file.
1845
1846Plugins are also planned (partially implemented) in the Director and the
1847Storage daemon.
1848
1849\subsubsection{Plugin Directory}
1850\index[general]{Plugin Directory}
1851Each daemon (DIR, FD, SD) has a new \textbf{Plugin Directory} directive that may
1852be added to the daemon definition resource. The directory takes a quoted
1853string argument, which is the name of the directory in which the daemon can
1854find the Bacula plugins. If this directive is not specified, Bacula will not
1855load any plugins. Since each plugin has a distinctive name, all the daemons
1856can share the same plugin directory.
1857
1858\subsubsection{Plugin Options}
1859\index[general]{Plugin Options}
1860The \textbf{Plugin Options} directive takes a quoted string
1861arguement (after the equal sign) and may be specified in the
1862Job resource.  The options specified will be passed to all plugins
1863when they are run.  This each plugin must know what it is looking
1864for. The value defined in the Job resource can be modified
1865by the user when he runs a Job via the \btool{bconsole} command line
1866prompts.
1867
1868Note: this directive may be specified, and there is code to modify
1869the string in the run command, but the plugin options are not yet passed to
1870the plugin (i.e. not fully implemented).
1871
1872\subsubsection{Plugin Options ACL}
1873\index[general]{Plugin Options ACL}
1874The \textbf{Plugin Options ACL} directive may be specified in the
1875Director's Console resource. It functions as all the other ACL commands
1876do by permitting users running restricted consoles to specify a
1877\textbf{Plugin Options} that overrides the one specified in the Job
1878definition. Without this directive restricted consoles may not modify
1879the Plugin Options.
1880
1881\subsubsection{Plugin = \bbracket{plugin-command-string}}
1882\index[general]{Plugin}
1883The \textbf{Plugin} directive is specified in the Include section of
1884a FileSet resource where you put your \textbf{File = xxx} directives.
1885For example:
1886
1887\begin{bVerbatim}
1888  FileSet {
1889    Name = "MyFileSet"
1890    Include {
1891      Options {
1892        signature = MD5
1893      }
1894      File = /home
1895      Plugin = "bpipe:..."
1896    }
1897  }
1898\end{bVerbatim}
1899
1900In the above example, when the File daemon is processing the directives
1901in the Include section, it will first backup all the files in \bdirectoryname{/home}
1902then it will load the plugin named \btool{bpipe} (actually bpipe-dir.so) from
1903the Plugin Directory.  The syntax and semantics of the Plugin directive
1904require the first part of the string up to the colon (:) to be the name
1905of the plugin. Everything after the first colon is ignored by the File daemon but
1906is passed to the plugin. Thus the plugin writer may define the meaning of the
1907rest of the string as he wishes.
1908
1909Please see the next section for information about the \btool{bpipe} Bacula
1910plugin.
1911
1912\subsection{The bpipe Plugin}
1913\index[general]{The bpipe plugin}
1914The \btool{bpipe} plugin is provided in the directory src/plugins/fd/bpipe-fd.c of
1915the Bacula source distribution. When the plugin is compiled and linking into
1916the resulting dynamic shared object (DSO), it will have the name \bfilename{bpipe-fd.so}.
1917Please note that this is a very simple plugin that was written for
1918demonstration and test purposes. It is and can be used in production, but
1919that was never really intended.
1920
1921The purpose of the plugin is to provide an interface to any system program for
1922backup and restore. As specified above the \btool{bpipe} plugin is specified in
1923the Include section of your Job's FileSet resource.  The full syntax of the
1924plugin directive as interpreted by the \btool{bpipe} plugin (each plugin is free
1925to specify the sytax as it wishes) is:
1926
1927\begin{bVerbatim}
1928  Plugin = "<field1>:<field2>:<field3>:<field4>"
1929\end{bVerbatim}
1930
1931where
1932\begin{bdescription}
1933\item \textbf{field1} is the name of the plugin with the trailing \textbf{-fd.so}
1934stripped off, so in this case, we would put \btool{bpipe} in this field.
1935
1936\item \textbf{field2} specifies the namespace, which for \btool{bpipe} is the
1937pseudo path and filename under which the backup will be saved. This pseudo
1938path and filename will be seen by the user in the restore file tree.
1939For example, if the value is \bfilename{/MYSQL/regress.sql}, the data
1940backed up by the plugin will be put under that "pseudo" path and filename.
1941You must be careful to choose a naming convention that is unique to avoid
1942a conflict with a path and filename that actually exists on your system.
1943
1944\item \textbf{field3} for the \btool{bpipe} plugin
1945specifies the "reader" program that is called by the plugin during
1946backup to read the data. \btool{bpipe} will call this program by doing a
1947\btool{popen} on it.
1948
1949\item \textbf{field4} for the \btool{bpipe} plugin
1950specifies the "writer" program that is called by the plugin during
1951restore to write the data back to the filesystem.
1952\end{bdescription}
1953
1954Please note that for two items above describing the "reader" and "writer"
1955fields, these programs are "executed" by Bacula, which
1956means there is no shell interpretation of any command line arguments
1957you might use.  If you want to use shell characters (redirection of input
1958or output, \ldots{}), then we recommend that you put your command or commands
1959in a shell script and execute the script. In addition if you backup a
1960file with the reader program, when running the writer program during
1961the restore, Bacula will not automatically create the path to the file.
1962Either the path must exist, or you must explicitly do so with your command
1963or in a shell script.
1964
1965Putting it all together, the full plugin directive line might look
1966like the following:
1967
1968\begin{bVerbatim}
1969Plugin = "bpipe:/MYSQL/regress.sql:mysqldump -f
1970          --opt --databases bacula:mysql"
1971\end{bVerbatim}
1972
1973The directive has been split into two lines, but within the \bfilename{bacula-dir.conf} file
1974would be written on a single line.
1975
1976This causes the File daemon to call the \btool{bpipe} plugin, which will write
1977its data into the "pseudo" file \bfilename{/MYSQL/regress.sql} by calling the
1978program \btool{mysqldump -f \-\-opt --database bacula} to read the data during
1979backup. The mysqldump command outputs all the data for the database named
1980\textbf{bacula}, which will be read by the plugin and stored in the backup.
1981During restore, the data that was backed up will be sent to the program
1982specified in the last field, which in this case is \textbf{mysql}.  When
1983\textbf{mysql} is called, it will read the data sent to it by the plugn
1984then write it back to the same database from which it came (\textbf{bacula}
1985in this case).
1986
1987The \btool{bpipe} plugin is a generic pipe program, that simply transmits
1988the data from a specified program to Bacula for backup, and then from Bacula to
1989a specified program for restore.
1990
1991By using different command lines to \btool{bpipe},
1992you can backup any kind of data (ASCII or binary) depending
1993on the program called.
1994
1995\subsection{Microsoft Exchange Server 2003/2007 Plugin}
1996\index[general]{Microsoft Exchange Server 2003/2007 plugin}
1997\subsubsection{Background}
1998The Exchange plugin was made possible by a funded development project
1999between Equiinet Ltd -- www.equiinet.com (many thanks) and Bacula Systems.
2000The code for the plugin was written by James Harper, and the Bacula core
2001code by Kern Sibbald.  All the code for this funded development has become
2002part of the Bacula project.  Thanks to everyone who made it happen.
2003
2004\subsubsection{Concepts}
2005Although it is possible to backup Exchange using Bacula VSS the Exchange
2006plugin adds a good deal of functionality, because while Bacula VSS
2007completes a full backup (snapshot) of Exchange, it does
2008not support Incremental or Differential backups, restoring is more
2009complicated, and a single database restore is not possible.
2010
2011Microsoft Exchange organises its storage into Storage Groups with
2012Databases inside them. A default installation of Exchange will have a
2013single Storage Group called 'First Storage Group', with two Databases
2014inside it, "Mailbox Store (SERVER NAME)" and
2015"Public Folder Store (SERVER NAME)",
2016which hold user email and public folders respectively.
2017
2018In the default configuration, Exchange logs everything that happens to
2019log files, such that if you have a backup, and all the log files since,
2020you can restore to the present time. Each Storage Group has its own set
2021of log files and operates independently of any other Storage Groups. At
2022the Storage Group level, the logging can be turned off by enabling a
2023function called "Enable circular logging". At this time the Exchange
2024plugin will not function if this option is enabled.
2025
2026The plugin allows backing up of entire storage groups, and the restoring
2027of entire storage groups or individual databases. Backing up and
2028restoring at the individual mailbox or email item is not supported but
2029can be simulated by use of the "Recovery" Storage Group (see below).
2030
2031\subsubsection{Installing}
2032The Exchange plugin requires a DLL that is shipped with Microsoft
2033Exchanger Server called \btool{esebcli2.dll}. Assuming Exchange is installed
2034correctly the Exchange plugin should find this automatically and run
2035without any additional installation.
2036
2037If the DLL can not be found automatically it will need to be copied into
2038the Bacula installation
2039directory (eg \bdirectoryname{C:\textbackslash{}Program Files\textbackslash{}Bacula\textbackslash{}bin}). The Exchange API DLL is
2040named \bfilename{esebcli2.dll} and is found in \bdirectoryname{C:\textbackslash{}Program Files\textbackslash{}Exchsrvr\textbackslash{}bin} on a
2041default Exchange installation.
2042
2043\subsubsection{Backing Up}
2044To back up an Exchange server the Fileset definition must contain at
2045least \textbf{Plugin = "exchange\string:/@EXCHANGE/Microsoft Information Store"} for
2046the backup to work correctly. The 'exchange:' bit tells Bacula to look
2047for the exchange plugin, the '@EXCHANGE' bit makes sure all the backed
2048up files are prefixed with something that isn't going to share a name
2049with something outside the plugin, and the 'Microsoft Information Store'
2050bit is required also. It is also possible to add the name of a storage
2051group to the "Plugin =" line, eg \\
2052\textbf{Plugin = "exchange\string:/@EXCHANGE/Microsoft Information Store/First Storage Group"} \\
2053if you want only a single storage group backed up.
2054
2055Additionally, you can suffix the 'Plugin =' directive with
2056":notrunconfull" which will tell the plugin not to truncate the Exchange
2057database at the end of a full backup.
2058
2059An Incremental or Differential backup will backup only the database logs
2060for each Storage Group by inspecting the "modified date" on each
2061physical log file. Because of the way the Exchange API works, the last
2062logfile backed up on each backup will always be backed up by the next
2063Incremental or Differential backup too. This adds 5MB to each
2064Incremental or Differential backup size but otherwise does not cause any
2065problems.
2066
2067By default, a normal VSS fileset containing all the drive letters will
2068also back up the Exchange databases using VSS. This will interfere with
2069the plugin and Exchange's shared ideas of when the last full backup was
2070done, and may also truncate log files incorrectly. It is important,
2071therefore, that the Exchange database files be excluded from the backup,
2072although the folders the files are in should be included, or they will
2073have to be recreated manually if a baremetal restore is done.
2074
2075\begin{bVerbatim}
2076FileSet {
2077   Include {
2078      File = C:/Program Files/Exchsrvr/mdbdata
2079      Plugin = "exchange:..."
2080   }
2081   Exclude {
2082      File = C:/Program Files/Exchsrvr/mdbdata/E00.chk
2083      File = C:/Program Files/Exchsrvr/mdbdata/E00.log
2084      File = C:/Program Files/Exchsrvr/mdbdata/E000000F.log
2085      File = C:/Program Files/Exchsrvr/mdbdata/E0000010.log
2086      File = C:/Program Files/Exchsrvr/mdbdata/E0000011.log
2087      File = C:/Program Files/Exchsrvr/mdbdata/E00tmp.log
2088      File = C:/Program Files/Exchsrvr/mdbdata/priv1.edb
2089   }
2090}
2091\end{bVerbatim}
2092
2093The advantage of excluding the above files is that you can significantly
2094reduce the size of your backup since all the important Exchange files
2095will be properly saved by the Plugin.
2096
2097
2098\subsubsection{Restoring}
2099The restore operation is much the same as a normal Bacula restore, with
2100the following provisos:
2101
2102\begin{bitemize}
2103\item  The \textbf{Where} restore option must not be specified
2104\item Each Database directory must be marked as a whole. You cannot just
2105     select (say) the .edb file and not the others.
2106\item If a Storage Group is restored, the directory of the Storage Group
2107     must be marked too.
2108\item  It is possible to restore only a subset of the available log files,
2109     but they \textbf{must} be contiguous. Exchange will fail to restore correctly
2110     if a log file is missing from the sequence of log files
2111\item Each database to be restored must be dismounted and marked as \bog{}Can be
2112    overwritten by restore\cog{}
2113\item If an entire Storage Group is to be restored (eg all databases and
2114   logs in the Storage Group), then it is best to manually delete the
2115   database files from the server (eg \bdirectoryname{C:\textbackslash{}Program Files\textbackslash{}Exchsrvr\textbackslash{}mdbdata\textbackslash{}*})
2116   as Exchange can get confused by stray log files lying around.
2117\end{bitemize}
2118
2119\subsubsection{Restoring to the Recovery Storage Group}
2120The concept of the Recovery Storage Group is well documented by
2121Microsoft
2122\bref{http://support.microsoft.com/kb/824126}{support.microsoft.com/kb/824126},
2123but to briefly summarize\ldots{}
2124
2125Microsoft Exchange allows the creation of an additional Storage Group
2126called the Recovery Storage Group, which is used to restore an older
2127copy of a database (e.g. before a mailbox was deleted) into without
2128messing with the current live data. This is required as the Standard and
2129Small Business Server versions of Exchange can not ordinarily have more
2130than one Storage Group.
2131
2132To create the Recovery Storage Group, drill down to the Server in Exchange
2133System Manager, right click, and select
2134\textbf{"New \textrightarrow{} Recovery Storage Group!\ldots{}"}.  Accept or change the file
2135locations and click OK. On the Recovery Storage Group, right click and
2136select \textbf{"Add Database to Recover\ldots{}"} and select the database you will
2137be restoring.
2138
2139Restore only the single database nominated as the database in the
2140Recovery Storage Group. Exchange will redirect the restore to the
2141Recovery Storage Group automatically.
2142Then run the restore.
2143
2144\subsubsection{Restoring on Microsoft Server 2007}
2145Apparently the \btool{Exmerge} program no longer exists in Microsoft Server
21462007, and hence you use a new procedure for recovering a single mail box.
2147This procedure is documented by Microsoft at:
2148\bref{http://technet.microsoft.com/en-us/library/aa997694.aspx}{technet.microsoft.com/en-us/library/aa997694.aspx},
2149and involves using the \btool{Restore-Mailbox} and
2150\btool{Get-MailboxStatistics} shell commands.
2151
2152\subsubsection{Caveats}
2153This plugin is still being developed, so you should consider it
2154currently in BETA test, and thus use in a production environment
2155should be done only after very careful testing.
2156
2157When doing a full backup, the Exchange database logs are truncated by
2158Exchange as soon as the plugin has completed the backup. If the data
2159never makes it to the backup medium (eg because of spooling) then the
2160logs will still be truncated, but they will also not have been backed
2161up. A solution to this is being worked on. You will have to schedule a
2162new Full backup to ensure that your next backups will be usable.
2163
2164The \bog{}Enable Circular Logging\cog{} option cannot be enabled or the plugin
2165will fail.
2166
2167Exchange insists that a successful Full backup must have taken place if
2168an Incremental or Differential backup is desired, and the plugin will
2169fail if this is not the case. If a restore is done, Exchange will
2170require that a Full backup be done before an Incremental or Differential
2171backup is done.
2172
2173The plugin will most likely not work well if another backup application
2174(eg \btool{NTBACKUP}) is backing up the Exchange database, especially if the
2175other backup application is truncating the log files.
2176
2177The Exchange plugin has not been tested with the \textbf{Accurate} option, so
2178we recommend either carefully testing or that you avoid this option for
2179the current time.
2180
2181The Exchange plugin is not called during processing the bconsole
2182\bcommandname{estimate} command, and so anything that would be backed up by the plugin
2183will not be added to the estimate total that is displayed.
2184
2185
2186\subsection{libdbi Framework}
2187\index[general]{libdbi framework}
2188As a general guideline, \bacula{} has support for a few catalog database drivers
2189(\mysql{}, \postgresql{}, \sqlite{})
2190coded natively by the Bacula team.  With the libdbi implementation, which is a
2191\bacula{} driver that uses libdbi to access the catalog, we have an open field to
2192use many different kinds database engines following the needs of users.
2193
2194The according to libdbi (\bref{http://libdbi.sourceforge.net/}{libdbi.sourceforge.net/}) project: libdbi
2195implements a database-independent abstraction layer in C, similar to the
2196DBI/DBD layer in Perl. Writing one generic set of code, programmers can
2197leverage the power of multiple databases and multiple simultaneous database
2198connections by using this framework.
2199
2200Currently the libdbi driver in Bacula project only supports the same drivers
2201natively coded in \bacula{}.  However the libdbi project has support for many
2202others database engines. You can view the list at
2203\bref{http://libdbi-drivers.sourceforge.net/}{libdbi-drivers.sourceforge.net/}. In the future all those drivers can be
2204supported by Bacula, however, they must be tested properly by the Bacula team.
2205
2206Some of benefits of using libdbi are:
2207\begin{bitemize}
2208\item The possibility to use proprietary databases engines in which your
2209  proprietary licenses prevent the Bacula team from developing the driver.
2210 \item The possibility to use the drivers written for the libdbi project.
2211 \item The possibility to use other database engines without recompiling \bacula{}
2212   to use them.  Just change one line in \bfilename{bacula-dir.conf}
2213 \item Abstract Database access, this is, unique point to code and profiling
2214   catalog database access.
2215 \end{bitemize}
2216
2217 The following drivers have been tested:
2218 \begin{bitemize}
2219 \item \postgresql{}, with and without batch insert
2220 \item \mysql{}, with and without batch insert
2221 \item \sqlite{}
2222 \item \sqlite{}3
2223 \end{bitemize}
2224
2225 In the future, we will test and approve to use others databases engines
2226 (proprietary or not) like DB2, Oracle, Microsoft \acs{SQL}.
2227
2228 To compile Bacula to support libdbi we need to configure the code with the
2229 \texttt{\-\-with-dbi} and \texttt{\-\-with-dbi-driver=[database]} \btool{./configure} options, where
2230 \texttt{[database]} is the database engine to be used with Bacula (of course we can
2231 change the driver in file \bfilename{bacula-dir.conf}, see below).  We must configure the
2232 access port of the database engine with the option \texttt{\-\-with-db-port}, because the
2233 libdbi framework doesn't know the default access port of each database.
2234
2235The next phase is checking (or configuring) the \bfilename{bacula-dir.conf}, example:
2236\begin{bVerbatim}
2237Catalog {
2238  Name = MyCatalog
2239  dbdriver = dbi:mysql; dbaddress = 127.0.0.1; dbport = 3306
2240  dbname = regress; user = regress; password = ""
2241}
2242\end{bVerbatim}
2243
2244The parameter \textbf{dbdriver} indicates that we will use the driver dbi with a
2245\mysql{} database.  Currently the drivers supported by \bacula{} are: \texttt{postgresql},
2246\texttt{mysql}, \texttt{sqlite}, \texttt{sqlite3}; these are the names that may be added to string \bog{}\texttt{dbi:}".
2247
2248The following limitations apply when \bacula{} is set to use the libdbi framework:
2249\begin{bitemize}
2250\item Not tested on the Win32 platform
2251\item A little performance is lost if comparing with native database driver.
2252   The reason is bound with the database driver provided by libdbi and the
2253   simple fact that one more layer of code was added.
2254\end{bitemize}
2255It is important to remember, when compiling \bacula{} with libdbi, the
2256following packages are needed:
2257 \begin{bitemize}
2258  \item libdbi version 1.0.0, \url{http://libdbi.sourceforge.net/}
2259  \item libdbi-drivers 1.0.0, \url{http://libdbi-drivers.sourceforge.net/}
2260 \end{bitemize}
2261
2262 You can download them and compile them on your system or install the packages
2263 from your OS distribution.
2264
2265\subsection{Console Command Additions and Enhancements}
2266\index[general]{Console additions}
2267
2268\subsubsection{Display Autochanger Content}
2269\index[general]{StatusSlots}
2270
2271The \bcommandname{status slots storage=\bbracket{storage-name}} command displays
2272autochanger content.
2273
2274\begin{bVerbatim}
2275 Slot |  Volume Name  |  Status  |  Media Type       |   Pool     |
2276------+---------------+----------+-------------------+------------|
2277    1 |         00001 |   Append |  DiskChangerMedia |    Default |
2278    2 |         00002 |   Append |  DiskChangerMedia |    Default |
2279    3*|         00003 |   Append |  DiskChangerMedia |    Scratch |
2280    4 |               |          |                   |            |
2281\end{bVerbatim}
2282
2283If an asterisk (\textbf{*}) appears after the slot number, you must run an
2284\bcommandname{update slots} command to synchronize autochanger content with your
2285catalog.
2286
2287\subsubsection{list joblog job=xxx or jobid=nnn}
2288\index[general]{list joblog}
2289A new list command has been added that allows you to list the contents
2290of the Job Log stored in the catalog for either a Job Name (fully qualified)
2291or for a particular JobId.  The \bcommandname{llist} command will include a line with
2292the time and date of the entry.
2293
2294Note for the catalog to have Job Log entries, you must have a directive
2295such as:
2296
2297\begin{bVerbatim}
2298  catalog = all
2299\end{bVerbatim}
2300
2301In your Director's \textbf{Messages} resource.
2302
2303\subsubsection{Use separator for multiple commands}
2304\index[general]{Command separator}
2305  When using bconsole with readline, you can set the command separator with
2306  \textbf{@separator} command to one
2307  of those characters to write commands who require multiple input in one line.
2308\begin{bVerbatim}
2309  !$%&'()*+,-/:;<>?[]^`{|}~
2310\end{bVerbatim}
2311
2312\subsubsection{Deleting Volumes}
2313The delete volume bconsole command has been modified to
2314require an asterisk (*) in front of a MediaId otherwise the
2315value you enter is a taken to be a Volume name. This is so that
2316users may delete numeric Volume names. The previous Bacula versions
2317assumed that all input that started with a number was a MediaId.
2318
2319This new behavior is indicated in the prompt if you read it
2320carefully.
2321
2322\subsection{Bare Metal Recovery}
2323The old bare metal recovery project is essentially dead. One
2324of the main features of it was that it would build a recovery
2325CD based on the kernel on your system. The problem was that
2326every distribution has a different boot procedure and different
2327scripts, and worse yet, the boot procedures and scripts change
2328from one distribution to another.  This meant that maintaining
2329(keeping up with the changes) the rescue CD was too much work.
2330
2331To replace it, a new bare metal recovery USB boot stick has been developed
2332by Bacula Systems.  This technology involves remastering a Ubuntu LiveCD to
2333boot from a USB key.
2334
2335Advantages:
2336\begin{benumerate}
2337\item Recovery can be done from within graphical environment.
2338\item Recovery can be done in a shell.
2339\item Ubuntu boots on a large number of Linux systems.
2340\item The process of updating the system and adding new
2341   packages is not too difficult.
2342\item The USB key can easily be upgraded to newer Ubuntu versions.
2343\item The USB key has writable partitions for modifications to
2344   the OS and for modification to your home directory.
2345\item You can add new files/directories to the USB key very easily.
2346\item You can save the environment from multiple machines on
2347   one USB key.
2348\item Bacula Systems is funding its ongoing development.
2349\end{benumerate}
2350
2351The disadvantages are:
2352\begin{benumerate}
2353\item The USB key is usable but currently under development.
2354\item Not everyone may be familiar with Ubuntu (no worse
2355  than using Knoppix)
2356\item Some older OSes cannot be booted from USB. This can
2357   be resolved by first booting a Ubuntu LiveCD then plugging
2358   in the USB key.
2359\item Currently the documentation is sketchy and not yet added
2360   to the main manual. See below \ldots{}
2361\end{benumerate}
2362
2363The documentation and the code can be found in the \textbf{rescue} package
2364in the directory \textbf{linux/usb}.
2365
2366\subsection{Miscellaneous}
2367\index[general]{Misc new features}
2368
2369\subsubsection{Allow Mixed Priority = \byesorno{}}
2370\index[general]{Allow Mixed Priority}
2371   This directive is only implemented in version 2.5 and later.  When
2372   set to \textbf{yes} (default \bdefaultvalue{no}), this job may run even if lower
2373   priority jobs are already running.  This means a high priority job
2374   will not have to wait for other jobs to finish before starting.
2375   The scheduler will only mix priorities when all running jobs have
2376   this set to true.
2377
2378   Note that only higher priority jobs will start early.  Suppose the
2379   director will allow two concurrent jobs, and that two jobs with
2380   priority 10 are running, with two more in the queue.  If a job with
2381   priority 5 is added to the queue, it will be run as soon as one of
2382   the running jobs finishes.  However, new priority 10 jobs will not
2383   be run until the priority 5 job has finished.
2384
2385\subsubsection{Bootstrap File Directive -- FileRegex}
2386\index[general]{Bootstrap File directive}
2387  \textbf{FileRegex} is a new command that can be added to the bootstrap
2388  (.bsr) file.  The value is a regular expression.  When specified, only
2389  matching filenames will be restored.
2390
2391  During a restore, if all File records are pruned from the catalog
2392  for a Job, normally \bacula{} can restore only all files saved. That
2393  is there is no way using the catalog to select individual files.
2394  With this new feature, \bacula{} will ask if you want to specify a Regex
2395  expression for extracting only a part of the full backup.
2396
2397\begin{bVerbatim}
2398  Building directory tree for JobId(s) 1,3 ...
2399  There were no files inserted into the tree, so file selection
2400  is not possible.Most likely your retention policy pruned the files
2401
2402  Do you want to restore all the files? (yes|no): no
2403
2404  Regexp matching files to restore? (empty to abort): /tmp/regress/(bin|tests)/
2405  Bootstrap records written to /tmp/regress/working/zog4-dir.restore.1.bsr
2406\end{bVerbatim}
2407
2408\subsubsection{Bootstrap File Optimization Changes}
2409In order to permit proper seeking on disk files, we have extended the bootstrap
2410file format to include a \textbf{VolStartAddr} and \textbf{VolEndAddr} records. Each
2411takes a 64 bit unsigned integer range (i.e. nnn-mmm) which defines the start
2412address range and end address range respectively.  These two directives replace
2413the \textbf{VolStartFile}, \textbf{VolEndFile}, \textbf{VolStartBlock} and \textbf{
2414  VolEndBlock} directives.  Bootstrap files containing the old directives will
2415still work, but will not properly take advantage of proper disk seeking, and
2416may read completely to the end of a disk volume during a restore.  With the new
2417format (automatically generated by the new Director), restores will seek
2418properly and stop reading the volume when all the files have been restored.
2419
2420\subsubsection{Solaris ZFS/NFSv4 ACLs}
2421This is an upgrade of the previous Solaris ACL backup code
2422to the new library format, which will backup both the old
2423POSIX(UFS) ACLs as well as the ZFS ACLs.
2424
2425The new code can also restore POSIX(UFS) ACLs to a ZFS filesystem
2426(it will translate the POSIX(UFS)) ACL into a ZFS/NFSv4 one) it can also
2427be used to transfer from UFS to ZFS filesystems.
2428
2429
2430\subsubsection{Virtual Tape Emulation}
2431\index[general]{Virtual tape emulation}
2432We now have a Virtual Tape emulator that allows us to run though 99.9\% of
2433the tape code but actually reading and writing to a disk file. Used with the
2434\textbf{disk-changer} script, you can now emulate an autochanger with 10 drives
2435and 700 slots. This feature is most useful in testing.  It is enabled
2436by using \textbf{Device Type = vtape} in the Storage daemon's Device
2437directive. This feature is only implemented on Linux machines and should not be
2438used for production.
2439
2440\subsubsection{Bat Enhancements}
2441\index[general]{Bat enhancements}
2442Bat (the Bacula Administration Tool) GUI program has been significantly
2443enhanced and stabilized. In particular, there are new table based status
2444commands; it can now be easily localized using Qt4 Linguist.
2445
2446The Bat communications protocol has been significantly enhanced to improve
2447GUI handling. Note, you \textbf{must} use a the bat that is distributed with
2448the Director you are using otherwise the communications protocol will not
2449work.
2450
2451\subsubsection{RunScript Enhancements}
2452\index[general]{RunScript enhancements}
2453The \textbf{RunScript} resource has been enhanced to permit multiple
2454commands per RunScript.  Simply specify multiple \textbf{Command} directives
2455in your RunScript.
2456
2457\begin{bVerbatim}
2458Job {
2459  Name = aJob
2460  RunScript {
2461    Command = "/bin/echo test"
2462    Command = "/bin/echo an other test"
2463    Command = "/bin/echo 3 commands in the same runscript"
2464    RunsWhen = Before
2465  }
2466 ...
2467}
2468\end{bVerbatim}
2469
2470A new Client RunScript \textbf{RunsWhen} keyword of \textbf{AfterVSS} has been
2471implemented, which runs the command after the Volume Shadow Copy has been made.
2472
2473Console commands can be specified within a RunScript by using:
2474\textbf{Console = \bbracket{command}}.
2475
2476\subsubsection{Status Enhancements}
2477\index[general]{Status enhancements}
2478The bconsole \textbf{status dir} output has been enhanced to indicate
2479Storage daemon job spooling and despooling activity.
2480
2481\subsubsection{Connect Timeout}
2482\index[general]{Connect timeout}
2483The default connect timeout to the File
2484daemon has been set to 3 minutes. Previously it was 30 minutes.
2485
2486\subsubsection{ftruncate for NFS Volumes}
2487\index[general]{ftruncate for NFS volumes}
2488If you write to a Volume mounted by NFS (say on a local file server),
2489in previous Bacula versions, when the Volume was recycled, it was not
2490properly truncated because NFS does not implement ftruncate (file
2491truncate). This is now corrected in the new version because we have
2492written code (actually a kind user) that deletes and recreates the Volume,
2493thus accomplishing the same thing as a truncate.
2494
2495\subsubsection{Support for Ubuntu}
2496The new version of \bacula{} now recognizes the Ubuntu (and Kubuntu)
2497version of Linux, and thus now provides correct autostart routines.
2498Since Ubuntu officially supports \bacula{}, you can also obtain any
2499recent release of Bacula from the Ubuntu repositories.
2500
2501\subsubsection{Recycle Pool = \bbracket{pool-name}}
2502\index[general]{Recycle Pool}
2503The \textbf{RecyclePool} directive defines to which pool the Volume will
2504be placed (moved) when it is recycled. Without this directive, a Volume will
2505remain in the same pool when it is recycled. With this directive, it can be
2506moved automatically to any existing pool during a recycle. This directive is
2507probably most useful when defined in the Scratch pool, so that volumes will
2508be recycled back into the Scratch pool.
2509
2510\subsubsection{FD Version}
2511\index[general]{FD version}
2512The File daemon to Director protocol now includes a version
2513number, which although there is no visible change for users,
2514will help us in future versions automatically determine
2515if a File daemon is not compatible.
2516
2517\subsubsection{Max Run Sched Time = \bbracket{time-period-in-seconds}}
2518\index[general]{Max Run Sched Time}
2519The time specifies the maximum allowed time that a job may run, counted from
2520when the job was scheduled. This can be useful to prevent jobs from running
2521during working hours. We can see it like \texttt{Max Start Delay + Max Run
2522  Time}.
2523
2524\subsubsection{Max Wait Time = \bbracket{time-period-in-seconds}}
2525\index[general]{Max Wait Time}
2526Previous \textbf{MaxWaitTime} directives aren't working as expected, instead
2527of checking the maximum allowed time that a job may block for a resource,
2528those directives worked like \textbf{MaxRunTime}. Some users are reporting to
2529use \textbf{Incr/Diff/Full Max Wait Time} to control the maximum run time of
2530their job depending on the level. Now, they have to use
2531\textbf{Incr/Diff/Full Max Run Time}.  \textbf{Incr/Diff/Full Max Wait Time}
2532directives are now deprecated.
2533
2534\subsubsection{Incremental|Differential Max Wait Time = \bbracket{time-period-in-seconds}}
2535\index[general]{Incremental Max Wait Time}
2536\index[general]{Differential Max Wait Time}
2537
2538These directives have been deprecated in favor of
2539\texttt{Incremental|Differential Max Run Time}.
2540
2541\subsubsection{Max Run Time directives}
2542\index[general]{Max Run Time directives}
2543Using \textbf{Full/Diff/Incr Max Run Time}, it's now possible to specify the
2544maximum allowed time that a job can run depending on the level.
2545
2546\bimageH{different_time}{Job time control directives}{figbs4:jobtimecontroldirectives}
2547
2548\subsubsection{Statistics Enhancements}
2549\index[general]{Statistics enhancements}
2550If you (or probably your boss) want to have statistics on your backups to
2551provide some \textit{Service Level Agreement} indicators, you could use a few
2552SQL queries on the Job table to report how many:
2553
2554\begin{bitemize}
2555\item jobs have run
2556\item jobs have been successful
2557\item files have been backed up
2558\item \ldots{}
2559\end{bitemize}
2560
2561However, these statistics are accurate only if your job retention is greater
2562than your statistics period. Ie, if jobs are purged from the catalog, you won't
2563be able to use them.
2564
2565Now, you can use the \textbf{update stats [days=num]} console command to fill
2566the JobHistory table with new Job records. If you want to be sure to take in
2567account only \textbf{good jobs}, ie if one of your important job has failed but
2568you have fixed the problem and restarted it on time, you probably want to
2569delete the first \textit{bad} job record and keep only the successful one. For
2570that simply let your staff do the job, and update JobHistory table after two or
2571three days depending on your organization using the \textbf{[days=num]} option.
2572
2573These statistics records aren't used for restoring, but mainly for
2574capacity planning, billings, etc.
2575
2576The Bweb interface provides a statistics module that can use this feature. You
2577can also use tools like Talend or extract information by yourself.
2578
2579The \textbf{Statistics Retention = \bbracket{time}} director directive defines
2580the length of time that Bacula will keep statistics job records in the Catalog
2581database after the Job End time. (In \texttt{JobHistory} table) When this time
2582period expires, and if user runs \texttt{prune stats} command, Bacula will
2583prune (remove) Job records that are older than the specified period.
2584
2585You can use the following Job resource in your nightly \textbf{BackupCatalog}
2586job to maintain statistics.
2587\begin{bVerbatim}
2588Job {
2589  Name = BackupCatalog
2590  ...
2591  RunScript {
2592    Console = "update stats days=3"
2593    Console = "prune stats yes"
2594    RunsWhen = After
2595    RunsOnClient = no
2596  }
2597}
2598\end{bVerbatim}
2599
2600\subsubsection{ScratchPool = \bbracket{pool-resource-name}}
2601\index[general]{Scratch Pool}
2602This directive permits to specify a specific \textsl{Scratch} pool for the
2603current pool. This is useful when using multiple storage sharing the same
2604mediatype or when you want to dedicate volumes to a particular set of pool.
2605
2606\subsubsection{Enhanced Attribute Despooling}
2607\index[general]{Attribute despooling}
2608If the storage daemon and the Director are on the same machine, the spool file
2609that contains attributes is read directly by the Director instead of being
2610transmitted across the network. That should reduce load and speedup insertion.
2611
2612\subsubsection{SpoolSize = \bbracket{size-specification-in-bytes}}
2613\index[general]{Spool Size}
2614A new Job directive permits to specify the spool size per job. This is used
2615in advanced job tunning. \textbf{SpoolSize={\it bytes}}
2616
2617\subsubsection{MaximumConsoleConnections = \bbracket{number}}
2618\index[general]{Maximum Console Connections}
2619A new director directive permits to specify the maximum number of Console
2620Connections that could run concurrently. The default is set to 20, but you may
2621set it to a larger number.
2622
2623\subsubsection{VerId = \bbracket{string}}
2624\index[general]{VerId}
2625A new director directive permits to specify a personnal identifier that will be
2626displayed in the \texttt{version} command.
2627
2628\subsubsection{dbcheck enhancements}
2629\index[general]{dbcheck enhancements}
2630If you are using Mysql, dbcheck will now ask you if you want to create
2631temporary indexes to speed up orphaned Path and Filename elimination.
2632
2633A new \texttt{-B} option allows you to print catalog information in a simple
2634text based format. This is useful to backup it in a secure way.
2635
2636\begin{bVerbatim}
2637 $ dbcheck -B
2638 catalog=MyCatalog
2639 db_type=SQLite
2640 db_name=regress
2641 db_driver=
2642 db_user=regress
2643 db_password=
2644 db_address=
2645 db_port=0
2646 db_socket=
2647\end{bVerbatim} %$
2648
2649You can now specify the database connection port in the command line.
2650
2651\subsubsection{{-}{-}docdir configure option}
2652\index[general]{{-}{-}docdir configure option}
2653You can use {-}{-}docdir= on the ./configure command to
2654specify the directory where you want Bacula to install the
2655LICENSE, ReleaseNotes, ChangeLog, \ldots{} files.   The default is
2656\textbf{/usr/share/doc/bacula}.
2657
2658\subsubsection{{-}{-}htmldir configure option}
2659\index[general]{{-}{-}htmldir configure option}
2660You can use {-}{-}htmldir= on the ./configure command to
2661specify the directory where you want Bacula to install the bat html help
2662files. The default is \textbf{/usr/share/doc/bacula/html}
2663
2664\subsubsection{{-}{-}with-plugindir configure option}
2665\index[general]{{-}{-}plugindir configure option}
2666You can use {-}{-}plugindir= on the ./configure command to
2667specify the directory where you want Bacula to install
2668the plugins (currently only bpipe-fd). The default is
2669/usr/lib.
2670