1.. _section-plugins: 2 3Plugins 4======= 5 6:index:`\ <single: Plugin>`\ 7 8The functionality of Bareos can be extended by plugins. They do exists plugins for the different daemons (Director, Storage- and File-Daemon). 9 10To use plugins, they must be enabled in the configuration (:strong:`Plugin Directory`\ and optionally :strong:`Plugin Names`\ ). 11 12If a :strong:`Plugin Directory`\ is specified :strong:`Plugin Names`\ defines, which plugins get loaded. 13 14If :strong:`Plugin Names`\ is not defined, all plugins get loaded. 15 16.. _fdPlugins: 17 18File Daemon Plugins 19------------------- 20 21File Daemon plugins are configured by the :strong:`Plugin`\ directive of a :ref:`File Set <directive-fileset-plugin>`. 22 23 24 25 .. warning:: 26 27 Currently the plugin command is being stored as part of the backup. The restore command in your directive should be flexible enough if things might change in future, otherwise you could run into trouble. 28 29.. _bpipe: 30 31bpipe Plugin 32~~~~~~~~~~~~ 33 34:index:`\ <single: Plugin; bpipe>`\ 35 36The bpipe plugin is a generic pipe program, that simply transmits the data from a specified program to Bareos for backup, and from Bareos to a specified program for restore. The purpose of the plugin is to provide an interface to any system program for backup and restore. That allows you, for example, to do database backups without a local dump. By using different command lines to bpipe, you can backup any kind of data (ASCII or binary) depending on the program called. 37 38On Linux, the Bareos bpipe plugin is part of the **bareos-filedaemon** package and is therefore installed on any system running the filedaemon. 39 40The bpipe plugin is so simple and flexible, you may call it the "Swiss Army Knife" of the current existing plugins for Bareos. 41 42The bpipe plugin is specified in the Include section of your Job’s FileSet resource in your :file:`bareos-dir.conf`. 43 44.. code-block:: bareosconfig 45 :caption: bpipe fileset 46 47 FileSet { 48 Name = "MyFileSet" 49 Include { 50 Options { 51 signature = MD5 52 compression = gzip 53 } 54 Plugin = "bpipe:file=<filepath>:reader=<readprogram>:writer=<writeprogram> 55 } 56 } 57 58The syntax and semantics of the Plugin directive require the first part of the string up to the colon to be the name of the plugin. Everything after the first colon is ignored by the File daemon but is passed to the plugin. Thus the plugin writer may define the meaning of the rest of the string as he wishes. The full syntax of the plugin directive as interpreted by the bpipe plugin is: 59 60.. code-block:: bareosconfig 61 :caption: bpipe directive 62 63 Plugin = "<plugin>:file=<filepath>:reader=<readprogram>:writer=<writeprogram>" 64 65plugin 66 is the name of the plugin with the trailing -fd.so stripped off, so in this case, we would put bpipe in the field. 67 68filepath 69 specifies the namespace, which for bpipe is the pseudo path and filename under which the backup will be saved. This pseudo path and filename will be seen by the user in the restore file tree. For example, if the value is :strong:`/MySQL/mydump.sql`, the data backed up by the plugin will be put under that "pseudo" path and filename. You must be careful to choose a naming convention that is unique to avoid a conflict with a path and filename that actually 70 exists on your system. 71 72readprogram 73 for the bpipe plugin specifies the "reader" program that is called by the plugin during backup to read the data. bpipe will call this program by doing a popen on it. 74 75writeprogram 76 for the bpipe plugin specifies the "writer" program that is called by the plugin during restore to write the data back to the filesystem. 77 78Please note that the two items above describing the "reader" and "writer", these programs are "executed" by Bareos, which means there is no shell interpretation of any command line arguments you might use. If you want to use shell characters (redirection of input or output, ...), then we recommend that you put your command or commands in a shell script and execute the script. In addition if you backup a file with reader program, when running the writer program during the restore, Bareos will not 79automatically create the path to the file. Either the path must exist, or you must explicitly do so with your command or in a shell script. 80 81See the examples about :ref:`backup-postgresql` and :ref:`backup-mysql`. 82 83PGSQL Plugin 84~~~~~~~~~~~~ 85 86See chapter :ref:`backup-postgresql-plugin`. 87 88MySQL Plugin 89~~~~~~~~~~~~ 90 91See the chapters :ref:`backup-mysql-xtrabackup` and :ref:`backup-mysql-python`. 92 93MSSQL Plugin 94~~~~~~~~~~~~ 95 96See chapter :ref:`MSSQL`. 97 98LDAP Plugin 99~~~~~~~~~~~ 100 101:index:`\ <single: Plugin; ldap>`\ 102 103This plugin is intended to backup (and restore) the contents of a LDAP server. It uses normal LDAP operation for this. The package **bareos-filedaemon-ldap-python-plugin** (:sinceVersion:`15.2.0: LDAP Plugin`) contains an example configuration file, that must be adapted to your envirnoment. 104 105Cephfs Plugin 106~~~~~~~~~~~~~ 107 108:index:`\ <single: Plugin; ceph; cephfs>`\ :index:`\ <single: Ceph; Cephfs Plugin>`\ 109 110Opposite to the :ref:`Rados Backend <SdBackendRados>` that is used to store data on a CEPH Object Store, this plugin is intended to backup a CEPH Object Store via the Cephfs interface to other media. The package **bareos-filedaemon-ceph-plugin** (:sinceVersion:`15.2.0: Cephfs Plugin`) contains an example configuration file, that must be adapted to your envirnoment. 111 112Rados Plugin 113~~~~~~~~~~~~ 114 115:index:`\ <single: Plugin; ceph; rados>`\ :index:`\ <single: Ceph; Rados Plugin>`\ 116 117Opposite to the :ref:`Rados Backend <SdBackendRados>` that is used to store data on a CEPH Object Store, this plugin is intended to backup a CEPH Object Store via the Rados interface to other media. The package **bareos-filedaemon-ceph-plugin** (:sinceVersion:`15.2.0: CEPH Rados Plugin`) contains an example configuration file, that must be adapted to your envirnoment. 118 119GlusterFS Plugin 120~~~~~~~~~~~~~~~~ 121 122:index:`\ <single: Plugin; glusterfs>`\ :index:`\ <single: GlusterFS; Plugin>`\ 123 124Opposite to the :ref:`GFAPI Backend <SdBackendGfapi>` that is used to store data on a Gluster system, this plugin is intended to backup data from a Gluster system to other media. The package **bareos-filedaemon-glusterfs-plugin** (:sinceVersion:`15.2.0: GlusterFS Plugin`) contains an example configuration file, that must be adapted to your envirnoment. 125 126python-fd Plugin 127~~~~~~~~~~~~~~~~ 128 129:index:`\ <single: Plugin; Python; File Daemon>`\ 130 131The **python-fd** plugin behaves similar to the :ref:`director-python-plugin`. Base plugins and an example get installed via the package bareos-filedaemon-python-plugin. Configuration is done in the :ref:`DirectorResourceFileSet` on the director. 132 133We basically distinguish between command-plugin and option-plugins. 134 135Command Plugins 136^^^^^^^^^^^^^^^ 137 138Command plugins are used to replace or extend the FileSet definition in the File Section. If you have a command-plugin, you can use it like in this example: 139 140.. code-block:: bareosconfig 141 :caption: bareos-dir.conf: Python FD command plugins 142 143 FileSet { 144 Name = "mysql" 145 Include { 146 Options { 147 Signature = MD5 # calculate md5 checksum per file 148 } 149 File = "/etc" 150 Plugin = "python:module_path=/usr/lib/bareos/plugins:module_name=bareos-fd-mysql" 151 } 152 } 153 154:index:`\ <single: MySQL; Backup>`\ This example uses the :ref:`MySQL plugin <backup-mysql-python>` to backup MySQL dumps in addition to :file:`/etc`. 155 156Option Plugins 157^^^^^^^^^^^^^^ 158 159Option plugins are activated in the Options resource of a FileSet definition. 160 161Example: 162 163.. code-block:: bareosconfig 164 :caption: bareos-dir.conf: Python FD option plugins 165 166 FileSet { 167 Name = "option" 168 Include { 169 Options { 170 Signature = MD5 # calculate md5 checksum per file 171 Plugin = "python:module_path=/usr/lib/bareos/plugins:module_name=bareos-fd-file-interact" 172 } 173 File = "/etc" 174 File = "/usr/lib/bareos/plugins" 175 } 176 } 177 178This plugin bareos-fd-file-interact from https://github.com/bareos/bareos-contrib/tree/master/fd-plugins/options-plugin-sample has a method that is called before and after each file that goes into the backup, it can be used as a template for whatever plugin wants to interact with files before or after backup. 179 180.. _VMwarePlugin: 181 182VMware Plugin 183~~~~~~~~~~~~~ 184 185:index:`\ <single: Plugin; VMware>`\ :index:`\ <single: VMware Plugin>`\ 186 187The |vmware| Plugin can be used for agentless backups of virtual machines running on |vsphere|. It makes use of CBT (Changed Block Tracking) to do space efficient full and incremental backups, see below for mandatory requirements. 188 189It is included in Bareos since :sinceVersion:`15.2.0: VMware Plugin`. 190 191Status 192^^^^^^ 193 194The Plugin can do full, differential and incremental backup and restore of VM disks. 195 196Current limitations amongst others are: 197 198.. limitation:: VMware Plugin: Normal VM disks can not be excluded from the backup. 199 200 It is not yet possible to exclude normal (dependent) VM disks from backups. 201 However, independent disks are excluded implicitly because they are not affected 202 by snapshots which are required for CBT based backup. 203 204 205 206.. limitation:: VMware Plugin: VM configuration is not backed up. 207 208 The VM configuration is not backed up, so that it is not yet possible to recreate a completely deleted VM. 209 210 211 212.. limitation:: VMware Plugin: Virtual Disks have to be smaller than 2TB for restore to local VMDK. 213 214 Virtual Disks have to be smaller than 2 TB for being able to restore to local VMDK files, see :mantis:`670`. 215 216 217 218.. limitation:: VMware Plugin: Restore can only be done to the same VM or to local VMDK files. 219 220 Until Bareos Version 15.2.2, the restore has only be possible to the same existing VM with existing virtual disks. 221 Since :sinceVersion:`15.2.3: VMware Plugin: restore to VMDK files` 222 %**bareos-vadp-dumper** :sinceVersion:`15.2.2-15: bareos-vadp-dumper` and 223 %**bareos-vmware-plugin** :sinceVersion:`15.2.2-27: bareos-vmware-plugin` 224 it is also possible to restore to local VMDK files, see below for more details. 225 226 227 228Requirements 229^^^^^^^^^^^^ 230 231As the Plugin is based on the |vsphere| Storage APIs for Data Protection, which requires at least a |vsphere| Essentials License. It is tested against |vsphere| Storage APIs for Data Protection of |vmware| 7.0.1. It does not work with standalone unlicensed |vmware| ESXi\ |trade|. 232 233Since Bareos :sinceVersion:`18.2.10: VMware Plugin: VDDK 7.0.1` the plugin is using the Virtual Disk Development Kit (VDDK) 7.0.1, as of the VDDK 7.0 release notes, it should be compatible with vSphere 7.0 and the next major release (except new features) and backward compatible with vSphere 6.5 and 6.7, see VDDK release notes at https://code.vmware.com/web/sdk/7.0/vddk for details. 234 235This plugin requires the pyVmomi module. Since Bareos :sinceVersion:`18.2.10: VMware Plugin: pyVmomi` the package **bareos-vmware-plugin** no longer includes a dependency on a pyVmomi package, because some Linux distributions don't provide current versions. Consequently, pyVmomi must be either installed by using :command:`pip install pyvmomi` or by manually installing a distribution provided pyVmomi package. 236 237Installation 238^^^^^^^^^^^^ 239 240Install the package **bareos-vmware-plugin** including its requirments by using an appropriate package management tool (eg. :command:`yum`, :command:`zypper`, :command:`apt`) 241 242Configuration 243^^^^^^^^^^^^^ 244 245First add a user account in vCenter that has full privileges by assigning the account to an administrator role or by adding the account to a group that is assigned to an administrator role. While any user account with full privileges could be used, it is better practice to create a separate user account, so that the actions by this account logged in vSphere are clearly distinguishable. In the future a more detailed set of required role privilges may be defined. 246 247When using the vCenter appliance with embedded SSO, a user account usually has the structure :command:`<username>@vsphere.local`, it may be different when using Active Directory as SSO in vCenter. For the examples here, we will use :command:`bakadm@vsphere.local` with the password :command:`Bak.Adm-1234`. 248 249For more details regarding users and permissions in vSphere see 250 251- http://pubs.vmware.com/vsphere-55/topic/com.vmware.vsphere.security.doc/GUID-72BFF98C-C530-4C50-BF31-B5779D2A4BBB.html and 252 253- http://pubs.vmware.com/vsphere-55/topic/com.vmware.vsphere.security.doc/GUID-5372F580-5C23-4E9C-8A4E-EF1B4DD9033E.html 254 255Make sure to add or enable the following settings in your |fd| configuration: 256 257.. code-block:: bareosconfig 258 :caption: bareos-fd.d/client/myself.conf 259 260 Client { 261 ... 262 Plugin Directory = /usr/lib/bareos/plugins 263 Plugin Names = python 264 ... 265 } 266 267Note: Depending on Platform, the Plugin Directory may also be :file:`/usr/lib64/bareos/plugins` 268 269To define the backup of a VM in Bareos, a job definition and a fileset resource must be added to the Bareos director configuration. In vCenter, VMs are usually organized in datacenters and folders. The following example shows how to configure the backup of the VM named *websrv1* in the datacenter *mydc1* folder *webservers* on the vCenter server :command:`vcenter.example.org`: 270 271.. code-block:: bareosconfig 272 :caption: bareos-dir.conf: VMware Plugin Job and FileSet definition 273 274 Job { 275 Name = "vm-websrv1" 276 JobDefs = "DefaultJob" 277 FileSet = "vm-websrv1_fileset" 278 } 279 280 FileSet { 281 Name = "vm-websrv1_fileset" 282 283 Include { 284 Options { 285 signature = MD5 286 Compression = GZIP 287 } 288 Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=mydc1:folder=/webservers:vmname=websrv1:vcserver=vcenter.example.org:vcuser=bakadm@vsphere.local:vcpass=Bak.Adm-1234" 289 } 290 } 291 292For VMs defined in the root-folder, :command:`folder=/` must be specified in the Plugin definition. 293 294Since Bareos :sinceVersion:`17.2.4: bareos-vmware-plugin: module\_path without vmware\_plugin subdirectory` the :strong:`module\_path` is without :file:`vmware_plugin` directory. On upgrades you either adapt your configuration from 295 296.. code-block:: bareosconfig 297 :caption: python:module\_path for Bareos < 17.2.0 298 299 Plugin = "python:module_path=/usr/lib64/bareos/plugins/vmware_plugin:module_name=bareos-fd-vmware:... 300 301to 302 303.. code-block:: bareosconfig 304 :caption: python:module\_path for Bareos >= 17.2.0 305 306 Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:... 307 308or install the **bareos-vmware-plugin-compat** package which includes compatibility symbolic links. 309 310Since :sinceVersion:`17.2.4: VMware Plugin: vcthumbprint`: as the Plugin is using the Virtual Disk Development Kit (VDDK) 6.5, it is required to pass the thumbprint of the vCenter SSL Certificate, which is the SHA1 checksum of the SSL Certificate. The thumbprint can be retrieved like this: 311 312.. code-block:: shell-session 313 :caption: Example Retrieving vCenter SSL Certificate Thumbprint 314 315 echo -n | openssl s_client -connect vcenter.example.org:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1 316 317The result would look like this: 318 319.. code-block:: shell-session 320 :caption: Example Result Thumbprint 321 322 SHA1 Fingerprint=CC:81:81:84:A3:CF:53:ED:63:B1:46:EF:97:13:4A:DF:A5:9F:37:89 323 324For additional security, there is a now plugin option :command:`vcthumbprint`, that can optionally be added. It must be given without colons like in the following example: 325 326.. code-block:: bareosconfig 327 :caption: bareos-dir.conf: VMware Plugin Options with vcthumbprint 328 329 ... 330 Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=mydc1:folder=/webservers:vmname=websrv1:vcserver=vcenter.example.org:vcuser=bakadm@vsphere.local:vcpass=Bak.Adm-1234:vcthumbprint=56F597FE60521773D073A2ED47CE07282CE6FE9C" 331 ... 332 333For ease of use (but less secure) when the :command:`vcthumbprint` is not given, the plugin will retrieve the thumbprint. 334 335Also since :sinceVersion:`17.2.4: VMware Plugin: transport=nbdssl` another optional plugin option has been added that can be used for trying to force a given transport method. Normally, when no transport method is given, VDDK will negotiate available transport methods and select the best one. For a description of transport methods, see 336 337https://code.vmware.com/doc/preview?id=4076#/doc/vddkDataStruct.5.5.html 338 339When the plugin runs in a VMware virtual machine which has access to datastore where the virtual disks to be backed up reside, VDDK will use the hotadd transport method. On a physical server without SAN access, it will use the NBD transport method, hotadd transport is not available in this case. 340 341To try forcing a given transport method, the plugin option :command:`transport` can be used, for example 342 343.. code-block:: bareosconfig 344 :caption: bareos-dir.conf: VMware Plugin options with transport 345 346 ... 347 Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=mydc1:folder=/webservers:vmname=websrv1:vcserver=vcenter.example.org:vcuser=bakadm@vsphere.local:vcpass=Bak.Adm-1234:transport=nbdssl" 348 ... 349 350Note that the backup will fail when specifying a transport method that is not available. 351 352Since :sinceVersion:`17.2.8: VMware Plugin: non-ascii characters` it is possible to use non-ascii characters and blanks in the configuration for :strong:`folder` and :strong:`vmname`. Also virtual disk file names or paths containing non-ascii characters are handled correctly now. For backing up VMs that are contained in vApps, it is now possible to use the vApp name like a folder component. For example, if we have the vApp named 353:command:`Test vApp` in the folder :file:`/Test/Test Folder` and the vApp contains the two VMs :command:`Test VM 01` and :command:`Test VM 02`, then the configuration of the filesets should look like this: 354 355.. code-block:: bareosconfig 356 :caption: bareos-dir.conf: VMware Plugin FileSet definition for vApp 357 358 FileSet { 359 Name = "vApp_Test_vm_Test_VM_01_fileset" 360 361 Include { 362 Options { 363 signature = MD5 364 Compression = GZIP 365 } 366 Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=mydc1:folder=/Test/Test Folder/Test vApp:vmname=Test VM 01:vcserver=vcenter.example.org:vcuser=bakadm@vsphere.local:vcpass=Bak.Adm-1234" 367 } 368 } 369 370 FileSet { 371 Name = "vApp_Test_vm_Test_VM_02_fileset" 372 373 Include { 374 Options { 375 signature = MD5 376 Compression = GZIP 377 } 378 Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=mydc1:folder=/Test/Test Folder/Test vApp:vmname=Test VM 02:vcserver=vcenter.example.org:vcuser=bakadm@vsphere.local:vcpass=Bak.Adm-1234" 379 } 380 } 381 382However, it is important to know that it is not possible to use non-ascii characters as an argument for the :strong:`Name`\ of a job or fileset resource. 383 384Before this, it was only possible specify VMs contained in vApps by using the instance UUID with the :strong:`uuid` instead of :strong:`folder` and :strong:`vmname` like this: 385 386.. code-block:: bareosconfig 387 :caption: bareos-dir.conf: VMware Plugin FileSet definition for vApp 388 389 FileSet { 390 Name = "vApp_Test_vm_Test_VM_01_fileset" 391 ... 392 393 Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=mydc1:uuid=502b112f-3954-d761-be08-5570c8a780e2:vcserver=vcenter.example.org:vcuser=bakadm@vsphere.local:vcpass=Bak.Adm-1234" 394 } 395 } 396 397Note that it must be the so called vSphere instance UUID, not the BIOS UUID which is shown inside a VM when using for example :command:`dmidecode`. The :command:`vmware_cbt_tool.py` utility was adapted accordingly (see below for details). 398 399Backup 400^^^^^^ 401 402Before running the first backup, CBT (Changed Block Tracking) must be enabled for the VMs to be backed up. 403 404As of http://kb.vmware.com/kb/2075984 manually enabling CBT is currently not working properly. The API however works properly. To enable CBT use the Script :command:`vmware_cbt_tool.py`, it is packaged in the bareos-vmware-plugin package: 405 406.. code-block:: shell-session 407 :caption: usage of vmware\_cbt\_tool.py 408 409 # <parameter>vmware_cbt_tool.py --help</parameter> 410 usage: vmware_cbt_tool.py [-h] -s HOST [-o PORT] -u USER [-p PASSWORD] -d 411 DATACENTER [-f FOLDER] [-v VMNAME] 412 [--vm-uuid VM_UUID] [--enablecbt] [--disablecbt] 413 [--resetcbt] [--info] [--listall] 414 415 Process args for enabling/disabling/resetting CBT 416 417 optional arguments: 418 -h, --help show this help message and exit 419 -s HOST, --host HOST Remote host to connect to 420 -o PORT, --port PORT Port to connect on 421 -u USER, --user USER User name to use when connecting to host 422 -p PASSWORD, --password PASSWORD 423 Password to use when connecting to host 424 -d DATACENTER, --datacenter DATACENTER 425 DataCenter Name 426 -f FOLDER, --folder FOLDER 427 Folder Name (must start with /, use / for root folder 428 -v VMNAME, --vmname VMNAME 429 Names of the Virtual Machines 430 --vm-uuid VM_UUID Instance UUIDs of the Virtual Machines 431 --enablecbt Enable CBT 432 --disablecbt Disable CBT 433 --resetcbt Reset CBT (disable, then enable) 434 --info Show information (CBT supported and enabled or 435 disabled) 436 --listall List all VMs in the given datacenter with UUID and 437 containing folder 438 439Note: the options :command:`--vm-uuid` and :command:`--listall` have been added in version :sinceVersion:`17.2.8: VMware Plugin: new options in vmware\_cbt\_tool.py`, the tool is also able now to process non-ascii character arguments for the :command:`--folder` and :command:`--vmname` arguments and vApp names can be used like folder name components. With :command:`--listall` all VMs in the given datacenter are reported 440in a tabular output including instance UUID and containing Folder/vApp name. 441 442For the above configuration example, the command to enable CBT would be 443 444.. code-block:: shell-session 445 :caption: Example using vmware\_cbt\_tool.py 446 447 # <parameter>vmware_cbt_tool.py -s vcenter.example.org -u bakadm@vsphere.local -p Bak.Adm-1234 -d mydc1 -f /webservers -v websrv1 --enablecbt</parameter> 448 449Note: CBT does not work if the virtual hardware version is 6 or earlier. 450 451After enabling CBT, Backup Jobs can be run or scheduled as usual, for example in :command:`bconsole`: 452 453:bcommand:`run job=vm-websrv1 level=Full` 454 455Restore 456^^^^^^^ 457 458For restore, the VM must be powered off and no snapshot must exist. In :command:`bconsole` use the restore menu 5, select the correct FileSet and enter :bcommand:`mark *`, then :bcommand:`done`. After restore has finished, the VM can be powered on. 459 460Restore to local VMDK File 461^^^^^^^^^^^^^^^^^^^^^^^^^^ 462 463:index:`\ <single: VMware Plugin; VMDK files>`\ 464 465Since :sinceVersion:`15.2.3: VMware Plugin: restore to VMDK files` it is possible to restore to local VMDK files. That means, instead of directly restoring a disk that belongs to the VM, the restore creates VMDK disk image files on the filesystem of the system that runs the |fd|. As the VM that the backup was taken from is not affected by this, it can remain switched on while restoring to local VMDK. Such a restored VMDK file can then be uploaded to a 466|vsphere| datastore or accessed by tools like `guestfish <http://libguestfs.org/guestfish.1.html>`_ to extract single files. 467 468For restoring to local VMDK, the plugin option :strong:`localvmdk=yes` must be passed. The following example shows how to perform such a restore using :command:`bconsole`: 469 470.. code-block:: shell-session 471 :caption: Example restore to local VMDK 472 473 *<input>restore</input> 474 Automatically selected Catalog: MyCatalog 475 Using Catalog "MyCatalog" 476 477 First you select one or more JobIds that contain files 478 to be restored. You will be presented several methods 479 of specifying the JobIds. Then you will be allowed to 480 select which files from those JobIds are to be restored. 481 482 To select the JobIds, you have the following choices: 483 1: List last 20 Jobs run 484 ... 485 5: Select the most recent backup for a client 486 ... 487 13: Cancel 488 Select item: (1-13): <input>5</input> 489 Automatically selected Client: vmw5-bareos-centos6-64-devel-fd 490 The defined FileSet resources are: 491 1: Catalog 492 ... 493 5: PyTestSetVmware-test02 494 6: PyTestSetVmware-test03 495 ... 496 Select FileSet resource (1-10): <input>5</input> 497 +-------+-------+----------+---------------+---------------------+------------------+ 498 | jobid | level | jobfiles | jobbytes | starttime | volumename | 499 +-------+-------+----------+---------------+---------------------+------------------+ 500 | 625 | F | 4 | 4,733,002,754 | 2016-02-18 10:32:03 | Full-0067 | 501 ... 502 You have selected the following JobIds: 625,626,631,632,635 503 504 Building directory tree for JobId(s) 625,626,631,632,635 ... 505 10 files inserted into the tree. 506 507 You are now entering file selection mode where you add (mark) and 508 remove (unmark) files to be restored. No files are initially added, unless 509 you used the "all" keyword on the command line. 510 Enter "done" to leave this mode. 511 512 cwd is: / 513 $ <input>mark *</input> 514 10 files marked. 515 $ <input>done</input> 516 Bootstrap records written to /var/lib/bareos/vmw5-bareos-centos6-64-devel-dir.restore.1.bsr 517 518 The job will require the following 519 Volume(s) Storage(s) SD Device(s) 520 =========================================================================== 521 522 Full-0001 File FileStorage 523 ... 524 Incremental-0078 File FileStorage 525 526 Volumes marked with "*" are online. 527 528 10 files selected to be restored. 529 530 Using Catalog "MyCatalog" 531 Run Restore job 532 JobName: RestoreFiles 533 Bootstrap: /var/lib/bareos/vmw5-bareos-centos6-64-devel-dir.restore.1.bsr 534 Where: /tmp/bareos-restores 535 Replace: Always 536 FileSet: Linux All 537 Backup Client: vmw5-bareos-centos6-64-devel-fd 538 Restore Client: vmw5-bareos-centos6-64-devel-fd 539 Format: Native 540 Storage: File 541 When: 2016-02-25 15:06:48 542 Catalog: MyCatalog 543 Priority: 10 544 Plugin Options: *None* 545 OK to run? (yes/mod/no): <input>mod</input> 546 Parameters to modify: 547 1: Level 548 ... 549 14: Plugin Options 550 Select parameter to modify (1-14): <input>14</input> 551 Please enter Plugin Options string: <input>python:localvmdk=yes</input> 552 Run Restore job 553 JobName: RestoreFiles 554 Bootstrap: /var/lib/bareos/vmw5-bareos-centos6-64-devel-dir.restore.1.bsr 555 Where: /tmp/bareos-restores 556 Replace: Always 557 FileSet: Linux All 558 Backup Client: vmw5-bareos-centos6-64-devel-fd 559 Restore Client: vmw5-bareos-centos6-64-devel-fd 560 Format: Native 561 Storage: File 562 When: 2016-02-25 15:06:48 563 Catalog: MyCatalog 564 Priority: 10 565 Plugin Options: python: module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware: dc=dass5:folder=/: vmname=stephand-test02: vcserver=virtualcenter5.dass-it:vcuser=bakadm@vsphere.local: vcpass=Bak.Adm-1234: localvmdk=yes 566 OK to run? (yes/mod/no): <input>yes</input> 567 Job queued. JobId=639 568 569Note: Since Bareos :sinceVersion:`15.2.3: Add additional python plugin options` it is sufficient to add Python plugin options, e.g. by 570 571:strong:`python:localvmdk=yes` 572 573Before, all Python plugin must be repeated and the additional be added, like: :file:`python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=dass5:folder=/:vmname=stephand-test02:vcserver=virtualcenter5.dass-it:vcuser=bakadm@vsphere.local:vcpass=Bak.Adm-1234:localvmdk=yes` 574 575After the restore process has finished, the restored VMDK files can be found under \path{/tmp/bareos-restores/}: 576 577.. code-block:: shell-session 578 :caption: Example result of restore to local VMDK 579 580 # <input>ls -laR /tmp/bareos-restores</input> 581 /tmp/bareos-restores: 582 total 28 583 drwxr-x--x. 3 root root 4096 Feb 25 15:47 . 584 drwxrwxrwt. 17 root root 20480 Feb 25 15:44 .. 585 drwxr-xr-x. 2 root root 4096 Feb 25 15:19 [ESX5-PS100] stephand-test02 586 587 /tmp/bareos-restores/[ESX5-PS100] stephand-test02: 588 total 7898292 589 drwxr-xr-x. 2 root root 4096 Feb 25 15:19 . 590 drwxr-x--x. 3 root root 4096 Feb 25 15:47 .. 591 -rw-------. 1 root root 2075197440 Feb 25 15:19 stephand-test02_1.vmdk 592 -rw-------. 1 root root 6012731392 Feb 25 15:19 stephand-test02.vmdk 593 594.. _sdPlugins: 595 596Storage Daemon Plugins 597---------------------- 598 599.. _plugin-autoxflate-sd: 600 601autoxflate-sd 602~~~~~~~~~~~~~ 603 604:index:`\ <single: Plugin; autoxflate-sd>`\ 605 606This plugin is part of the **bareos-storage** package. 607 608The autoxflate-sd plugin can inflate (decompress) and deflate (compress) the data being written to or read from a device. It can also do both. 609 610.. image:: /include/images/autoxflate-functionblocks.* 611 :width: 80.0% 612 613 614 615 616Therefore the autoxflate plugin inserts a inflate and a deflate function block into the stream going to the device (called OUT) and coming from the device (called IN). 617 618Each stream passes first the inflate function block, then the deflate function block. 619 620The inflate blocks are controlled by the setting of the :config:option:`sd/device/AutoInflate`\ directive. 621 622The deflate blocks are controlled by the setting of the :config:option:`sd/device/AutoDeflate`\ , :config:option:`sd/device/AutoDeflateAlgorithm`\ and :config:option:`sd/device/AutoDeflateLevel`\ directives. 623 624The inflate blocks, if enabled, will uncompress data if it is compressed using the algorithm that was used during compression. 625 626The deflate blocks, if enabled, will compress uncompressed data with the algorithm and level configured in the according directives. 627 628The series connection of the inflate and deflate function blocks makes the plugin very flexible. 629 630Szenarios where this plugin can be used are for example: 631 632- client computers with weak cpus can do backups without compression and let the sd do the compression when writing to disk 633 634- compressed backups can be recompressed to a different compression format (e.g. gzip |rarr| lzo) using migration jobs 635 636- client backups can be compressed with compression algorithms that the client itself does not support 637 638Multi-core cpus will be utilized when using parallel jobs as the compression is done in each jobs’ thread. 639 640When the autoxflate plugin is configured, it will write some status information into the joblog. 641 642.. code-block:: bareosmessage 643 :caption: used compression algorithm 644 645 autodeflation: compressor on device FileStorage is FZ4H 646 647.. code-block:: bareosmessage 648 :caption: configured inflation and deflation blocks 649 650 autoxflate-sd.c: FileStorage OUT:[SD->inflate=yes->deflate=yes->DEV] IN:[DEV->inflate=yes->deflate=yes->SD] 651 652.. code-block:: bareosmessage 653 :caption: overall deflation/inflation ratio 654 655 autoxflate-sd.c: deflate ratio: 50.59% 656 657Additional :config:option:`sd/storage/AutoXflateOnReplication`\ can be configured at the Storage resource. 658 659scsicrypto-sd 660~~~~~~~~~~~~~ 661 662:index:`\ <single: Plugin; scsicrypto-sd>`\ 663 664This plugin is part of the **bareos-storage-tape** package. 665 666General 667^^^^^^^ 668 669.. _LTOHardwareEncryptionGeneral: 670 671LTO Hardware Encryption 672''''''''''''''''''''''' 673 674Modern tape-drives, for example LTO (from LTO4 onwards) support hardware encryption. There are several ways of using encryption with these drives. The following three types of key management are available for encrypting drives. The transmission of the keys to the volumes is accomplished by either of the three: 675 676- A backup application that supports Application Managed Encryption (AME) 677 678- A tape library that supports Library Managed Encryption (LME) 679 680- A Key Management Appliance (KMA) 681 682We added support for Application Managed Encryption (AME) scheme, where on labeling a crypto key is generated for a volume and when the volume is mounted, the crypto key is loaded. When finally the volume is unmounted, the key is cleared from the memory of the Tape Drive using the SCSI SPOUT command set. 683 684If you have implemented Library Managed Encryption (LME) or a Key Management Appliance (KMA), there is no need to have support from Bareos on loading and clearing the encryption keys, as either the Library knows the per volume encryption keys itself, or it will ask the KMA for the encryption key when it needs it. For big installations you might consider using a KMA, but the Application Managed Encryption implemented in Bareos should also scale rather well and have a low overhead as the keys are 685only loaded and cleared when needed. 686 687The scsicrypto-sd plugin 688'''''''''''''''''''''''' 689 690The :command:`scsicrypto-sd` hooks into the :strong:`unload`, :strong:`label read`, :strong:`label write` and :strong:`label verified` events for loading and clearing the key. It checks whether it it needs to clear the drive by either using an internal state (if it loaded a key before) or by checking the state of a special option that first issues an encrytion status query. If there is a connection to the director 691and the volume information is not available, it will ask the director for the data on the currently loaded volume. If no connection is available, a cache will be used which should contain the most recently mounted volumes. If an encryption key is available, it will be loaded into the drive’s memory. 692 693Changes in the director 694''''''''''''''''''''''' 695 696The director has been extended with additional code for handling hardware data encryption. The extra keyword **encrypt** on the label of a volume will force the director to generate a new semi-random passphrase for the volume, which will be stored in the database as part of the media information. 697 698A passphrase is always stored in the database base64-encoded. When a so called **Key Encryption Key** is set in the config of the director, the passphrase is first wrapped using RFC3394 key wrapping and then base64-encoded. By using key wrapping, the keys in the database are safe against people sniffing the info, as the data is still encrypted using the Key Encryption Key (which in essence is just an extra passphrase of the same length as the volume passphrases used). 699 700When the storage daemon needs to mount the volume, it will ask the director for the volume information and that protocol is extended with the exchange of the base64-wrapped encryption key (passphrase). The storage daemon provides an extra config option in which it records the Key Encryption Key of the particular director, and as such can unwrap the key sent into the original passphrase. 701 702As can be seen from the above info we don’t allow the user to enter a passphrase, but generate a semi-random passphrase using the openssl random functions (if available) and convert that into a readable ASCII stream of letters, numbers and most other characters, apart from the quotes and space etc. This will produce much stronger passphrases than when requesting the info from a user. As we store this information in the database, the user never has to enter these passphrases. 703 704The volume label is written in unencrypted form to the volume, so we can always recognize a Bareos volume. When the key is loaded onto the drive, we set the decryption mode to mixed, so we can read both unencrypted and encrypted data from the volume. When no key or the wrong key has been loaded, the drive will give an IO error when trying to read the volume. For disaster recovery you can store the Key Encryption Key and the content of the wrapped encryption keys somewhere safe and the 705:ref:`bscrypto <bscrypto>` tool together with the scsicrypto-sd plugin can be used to get access to your volumes, in case you ever lose your complete environment. 706 707If you don’t want to use the scsicrypto-sd plugin when doing DR and you are only reading one volume, you can also set the crypto key using the bscrypto tool. Because we use the mixed decryption mode, in which you can read both encrypted and unencrypted data from a volume, you can set the right encryption key before reading the volume label. 708 709If you need to read more than one volume, you better use the scsicrypto-sd plugin with tools like bscan/bextract, as the plugin will then auto-load the correct encryption key when it loads the volume, similiarly to what the storage daemon does when performing backups and restores. 710 711The volume label is unencrypted, so a volume can also be recognized by a non-encrypted installation, but it won’t be able to read the actual data from it. Using an encrypted volume label doesn’t add much security (there is no security-related info in the volume label anyhow) and it makes it harder to recognize either a labeled volume with encrypted data or an unlabeled new volume (both would return an IO-error on read of the label.) 712 713.. _configuration-1: 714 715Configuration 716^^^^^^^^^^^^^ 717 718SCSI crypto setup 719''''''''''''''''' 720 721The initial setup of SCSI crypto looks something like this: 722 723- Generate a Key Encryption Key e.g. 724 725 .. code-block:: shell-session 726 727 bscrypto -g - 728 729For details see :ref:`bscrypto <bscrypto>`. 730 731Security Setup 732'''''''''''''' 733 734Some security levels need to be increased for the storage daemon to be able to use the low level SCSI interface for setting and getting the encryption status on a tape device. 735 736The following additional security is needed for the following operating systems: 737 738Linux (SG_IO ioctl interface): 739 740 741The user running the storage daemon needs the following additional capabilities: :index:`\ <single: Platform; Linux; Privileges>`\ 742 743- :strong:`CAP_SYS_RAWIO` (see capabilities(7)) 744 745 - On older kernels you might need :strong:`CAP_SYS_ADMIN`. Try :strong:`CAP_SYS_RAWIO` first and if that doesn’t work try :strong:`CAP_SYS_ADMIN` 746 747- If you are running the storage daemon as another user than root (which has the :strong:`CAP_SYS_RAWIO` capability), you need to add it to the current set of capabilities. 748 749- If you are using systemd, you could add this additional capability to the CapabilityBoundingSet parameter. 750 751 - For systemd add the following to the bareos-sd.service: :strong:`Capabilities=cap_sys_rawio+ep` 752 753You can also set up the extra capability on :command:`bscrypto` and :command:`bareos-sd` by running the following commands: 754 755.. code-block:: shell-session 756 757 setcap cap_sys_rawio=ep bscrypto 758 setcap cap_sys_rawio=ep bareos-sd 759 760Check the setting with 761 762.. code-block:: shell-session 763 764 getcap -v bscrypto 765 getcap -v bareos-sd 766 767:command:`getcap` and :command:`setcap` are part of libcap-progs. 768 769If :command:`bareos-sd` does not have the appropriate capabilities, all other tape operations may still work correctly, but you will get "Unable to perform SG\_IO ioctl" errors. 770 771Solaris (USCSI ioctl interface): 772 773 774The user running the storage daemon needs the following additional privileges: :index:`\ <single: Platform; Solaris; Privileges>`\ 775 776- :strong:`PRIV_SYS_DEVICES` (see privileges(5)) 777 778If you are running the storage daemon as another user than root (which has the :strong:`PRIV_SYS_DEVICES` privilege), you need to add it to the current set of privileges. This can be set up by setting this either as a project for the user, or as a set of extra privileges in the SMF definition starting the storage daemon. The SMF setup is the cleanest one. 779 780For SMF make sure you have something like this in the instance block: 781 782.. code-block:: bareosconfig 783 784 <method_context working_directory=":default"> <method_credential user="bareos" group="bareos" privileges="basic,sys_devices"/> </method_context> 785 786Changes in bareos-sd.conf 787''''''''''''''''''''''''' 788 789- Set the Key Encryption Key 790 791 - :config:option:`sd/director/KeyEncryptionKey`\ = :strong:`passphrase` 792 793- Enable the loading of storage daemon plugins 794 795 - :config:option:`sd/storage/PluginDirectory`\ = :file:`path_to_sd_plugins` 796 797- Enable the SCSI encryption option 798 799 - :config:option:`sd/device/DriveCryptoEnabled`\ = yes 800 801- Enable this, if you want the plugin to probe the encryption status of the drive when it needs to clear a pending key 802 803 - :config:option:`sd/device/QueryCryptoStatus`\ = yes 804 805Changes in bareos-dir.conf 806'''''''''''''''''''''''''' 807 808- Set the Key Encryption Key 809 810 - :config:option:`dir/director/KeyEncryptionKey`\ = :strong:`passphrase` 811 812Testing 813^^^^^^^ 814 815Restart the Storage Daemon and the Director. After this you can label new volumes with the encrypt option, e.g. 816 817.. code-block:: bareosconfig 818 819 label slots=1-5 barcodes encrypt 820 821Disaster Recovery 822^^^^^^^^^^^^^^^^^ 823 824For Disaster Recovery (DR) you need the following information: 825 826- Actual bareos-sd.conf with config options enabled as described above, including, among others, a definition of a director with the Key Encryption Key used for creating the encryption keys of the volumes. 827 828- The actual keys used for the encryption of the volumes. 829 830This data needs to be availabe as a so called crypto cache file which is used by the plugin when no connection to the director can be made to do a lookup (most likely on DR). 831 832Most of the times the needed information, e.g. the bootstrap info, is available on recently written volumes and most of the time the encryption cache will contain the most recent data, so a recent copy of the :file:`bareos-sd.<portnr>.cryptoc` file in the working directory is enough most of the time. You can also save the info from database in a safe place and use bscrypto to populate this info (VolumeName |rarr| EncryptKey) into the crypto cache file used by 833:command:`bextract` and :command:`bscan`. You can use :command:`bscrypto` with the following flags to create a new or update an existing crypto cache file e.g.: 834 835.. code-block:: shell-session 836 837 bscrypto -p /var/lib/bareos/bareos-sd.<portnr>.cryptoc 838 839- A valid BSR file containing the location of the last safe of the database makes recovery much easier. Adding a post script to the database save job could collect the needed info and make sure its stored somewhere safe. 840 841- Recover the database in the normal way e.g. for postgresql: 842 843 .. code-block:: shell-session 844 845 bextract -D <director_name> -c bareos-sd.conf -V <volname> \ /dev/nst0 /tmp -b bootstrap.bsr 846 /usr/lib64/bareos/create_bareos_database 847 /usr/lib64/bareos/grant_bareos_privileges 848 psql bareos < /tmp/var/lib/bareos/bareos.sql 849 850Or something similar (change paths to follow where you installed the software or where the package put it). 851 852**Note:** As described at the beginning of this chapter, there are different types of key management, AME, LME and KMA. If the Library is set up for LME or KMA, it probably won’t allow our AME setup and the scsi-crypto plugin will fail to set/clear the encryption key. To be able to use AME you need to "Modify Encryption Method" and set it to something like "Application Managed". If you decide to use LME or KMA you don’t have to bother with the whole setup 853of AME which may for big libraries be easier, although the overhead of using AME even for very big libraries should be minimal. 854 855scsitapealert-sd 856~~~~~~~~~~~~~~~~ 857 858:index:`\ <single: Plugin; scsitapealert-sd>`\ 859 860This plugin is part of the **bareos-storage-tape** package. 861 862python-sd Plugin 863~~~~~~~~~~~~~~~~ 864 865:index:`\ <single: Plugin; Python; Storage Daemon>`\ 866 867The **python-sd** plugin behaves similar to the :ref:`director-python-plugin`. 868 869.. _dirPlugins: 870 871Director Plugins 872---------------- 873 874.. _director-python-plugin: 875 876python-dir Plugin 877~~~~~~~~~~~~~~~~~ 878 879:index:`\ <single: Plugin; Python; Director>`\ 880 881The **python-dir** plugin is intended to extend the functionality of the Bareos Director by Python code. A working example is included. 882 883- install the **bareos-director-python-plugin** package 884 885- change to the Bareos plugin directory (:file:`/usr/lib/bareos/plugins/` or :file:`/usr/lib64/bareos/plugins/`) 886 887- copy :file:`bareos-dir.py.template` to :file:`bareos-dir.py` 888 889- activate the plugin in the Bareos Director configuration 890 891- restart the Bareos Director 892 893- change :file:`bareos-dir.py` as required 894 895- restart the Bareos Director 896 897Loading plugins 898^^^^^^^^^^^^^^^ 899 900Since :sinceVersion:`14.4.0: multiple Python plugins` multiple Python plugins can be loaded and plugin names can be arbitrary. Before this, the Python plugin always loads the file :file:`bareos-dir.py`. 901 902The director plugins are configured in the Job-Resource (or JobDefs resource). To load a Python plugin you need 903 904- pointing to your plugin directory (needs to be enabled in the Director resource, too 905 906- Your plugin (without the suffix .py) 907 908- default is ’0’, you can leave this, as long as you only have 1 Director Python plugin. If you have more than 1, start with instance=0 and increment the instance for each plugin. 909 910- You can add plugin specific option key-value pairs, each pair separated by ’:’ key=value 911 912Single Python Plugin Loading Example: 913 914.. code-block:: bareosconfig 915 :caption: bareos-dir.conf: Single Python Plugin Loading Example 916 917 Director { 918 # ... 919 # Plugin directory 920 Plugin Directory = /usr/lib64/bareos/plugins 921 # Load the python plugin 922 Plugin Names = "python" 923 } 924 925 JobDefs { 926 Name = "DefaultJob" 927 Type = Backup 928 # ... 929 # Load the class based plugin with testoption=testparam 930 Dir Plugin Options = "python:instance=0:module_path=/usr/lib64/bareos/plugins:module_name=bareos-dir-class-plugins:testoption=testparam 931 # ... 932 } 933 934Multiple Python Plugin Loading Example: 935 936.. code-block:: bareosconfig 937 :caption: bareos-dir.conf: Multiple Python Plugin Loading Example 938 939 Director { 940 # ... 941 # Plugin directory 942 Plugin Directory = /usr/lib64/bareos/plugins 943 # Load the python plugin 944 Plugin Names = "python" 945 } 946 947 JobDefs { 948 Name = "DefaultJob" 949 Type = Backup 950 # ... 951 # Load the class based plugin with testoption=testparam 952 Dir Plugin Options = "python:instance=0:module_path=/usr/lib64/bareos/plugins:module_name=bareos-dir-class-plugins:testoption=testparam1 953 Dir Plugin Options = "python:instance=1:module_path=/usr/lib64/bareos/plugins:module_name=bareos-dir-class-plugins:testoption=testparam2 954 # ... 955 } 956 957Write your own Python Plugin 958^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 959 960Some plugin examples are available on https://github.com/bareos/bareos-contrib. The class-based approach lets you easily reuse stuff already defined in the baseclass BareosDirPluginBaseclass, which ships with the **bareos-director-python-plugin** package. The examples contain the plugin bareos-dir-nsca-sender, that submits the results and performance data of a backup job directly to Icinga:index:`\ <single: Icinga>`\ or 961Nagios:index:`\ <single: Nagios|see{Icinga}>`\ using the NSCA protocol. 962 963 964 965 966