1********************************************************** 2Playbook Example: Continuous Delivery and Rolling Upgrades 3********************************************************** 4 5.. contents:: 6 :local: 7 8.. _lamp_introduction: 9 10What is continuous delivery? 11============================ 12 13Continuous delivery (CD) means frequently delivering updates to your software application. 14 15The idea is that by updating more often, you do not have to wait for a specific timed period, and your organization 16gets better at the process of responding to change. 17 18Some Ansible users are deploying updates to their end users on an hourly or even more frequent basis -- sometimes every time 19there is an approved code change. To achieve this, you need tools to be able to quickly apply those updates in a zero-downtime way. 20 21This document describes in detail how to achieve this goal, using one of Ansible's most complete example 22playbooks as a template: lamp_haproxy. This example uses a lot of Ansible features: roles, templates, 23and group variables, and it also comes with an orchestration playbook that can do zero-downtime 24rolling upgrades of the web application stack. 25 26.. note:: 27 28 `Click here for the latest playbooks for this example 29 <https://github.com/ansible/ansible-examples/tree/master/lamp_haproxy>`_. 30 31The playbooks deploy Apache, PHP, MySQL, Nagios, and HAProxy to a CentOS-based set of servers. 32 33We're not going to cover how to run these playbooks here. Read the included README in the github project along with the 34example for that information. Instead, we're going to take a close look at every part of the playbook and describe what it does. 35 36.. _lamp_deployment: 37 38Site deployment 39=============== 40 41Let's start with ``site.yml``. This is our site-wide deployment playbook. It can be used to initially deploy the site, as well 42as push updates to all of the servers: 43 44.. code-block:: yaml 45 46 --- 47 # This playbook deploys the whole application stack in this site. 48 49 # Apply common configuration to all hosts 50 - hosts: all 51 52 roles: 53 - common 54 55 # Configure and deploy database servers. 56 - hosts: dbservers 57 58 roles: 59 - db 60 61 # Configure and deploy the web servers. Note that we include two roles 62 # here, the 'base-apache' role which simply sets up Apache, and 'web' 63 # which includes our example web application. 64 65 - hosts: webservers 66 67 roles: 68 - base-apache 69 - web 70 71 # Configure and deploy the load balancer(s). 72 - hosts: lbservers 73 74 roles: 75 - haproxy 76 77 # Configure and deploy the Nagios monitoring node(s). 78 - hosts: monitoring 79 80 roles: 81 - base-apache 82 - nagios 83 84.. note:: 85 86 If you're not familiar with terms like playbooks and plays, you should review :ref:`working_with_playbooks`. 87 88In this playbook we have 5 plays. The first one targets ``all`` hosts and applies the ``common`` role to all of the hosts. 89This is for site-wide things like yum repository configuration, firewall configuration, and anything else that needs to apply to all of the servers. 90 91The next four plays run against specific host groups and apply specific roles to those servers. 92Along with the roles for Nagios monitoring, the database, and the web application, we've implemented a 93``base-apache`` role that installs and configures a basic Apache setup. This is used by both the 94sample web application and the Nagios hosts. 95 96.. _lamp_roles: 97 98Reusable content: roles 99======================= 100 101By now you should have a bit of understanding about roles and how they work in Ansible. Roles are a way to organize 102content: tasks, handlers, templates, and files, into reusable components. 103 104This example has six roles: ``common``, ``base-apache``, ``db``, ``haproxy``, ``nagios``, and ``web``. How you organize 105your roles is up to you and your application, but most sites will have one or more common roles that are applied to 106all systems, and then a series of application-specific roles that install and configure particular parts of the site. 107 108Roles can have variables and dependencies, and you can pass in parameters to roles to modify their behavior. 109You can read more about roles in the :ref:`playbooks_reuse_roles` section. 110 111.. _lamp_group_variables: 112 113Configuration: group variables 114============================== 115 116Group variables are variables that are applied to groups of servers. They can be used in templates and in 117playbooks to customize behavior and to provide easily-changed settings and parameters. They are stored in 118a directory called ``group_vars`` in the same location as your inventory. 119Here is lamp_haproxy's ``group_vars/all`` file. As you might expect, these variables are applied to all of the machines in your inventory: 120 121.. code-block:: yaml 122 123 --- 124 httpd_port: 80 125 ntpserver: 192.0.2.23 126 127This is a YAML file, and you can create lists and dictionaries for more complex variable structures. 128In this case, we are just setting two variables, one for the port for the web server, and one for the 129NTP server that our machines should use for time synchronization. 130 131Here's another group variables file. This is ``group_vars/dbservers`` which applies to the hosts in the ``dbservers`` group: 132 133.. code-block:: yaml 134 135 --- 136 mysqlservice: mysqld 137 mysql_port: 3306 138 dbuser: root 139 dbname: foodb 140 upassword: usersecret 141 142If you look in the example, there are group variables for the ``webservers`` group and the ``lbservers`` group, similarly. 143 144These variables are used in a variety of places. You can use them in playbooks, like this, in ``roles/db/tasks/main.yml``: 145 146.. code-block:: yaml 147 148 - name: Create Application Database 149 mysql_db: 150 name: "{{ dbname }}" 151 state: present 152 153 - name: Create Application DB User 154 mysql_user: 155 name: "{{ dbuser }}" 156 password: "{{ upassword }}" 157 priv: "*.*:ALL" 158 host: '%' 159 state: present 160 161You can also use these variables in templates, like this, in ``roles/common/templates/ntp.conf.j2``: 162 163.. code-block:: text 164 165 driftfile /var/lib/ntp/drift 166 167 restrict 127.0.0.1 168 restrict -6 ::1 169 170 server {{ ntpserver }} 171 172 includefile /etc/ntp/crypto/pw 173 174 keys /etc/ntp/keys 175 176You can see that the variable substitution syntax of {{ and }} is the same for both templates and variables. The syntax 177inside the curly braces is Jinja2, and you can do all sorts of operations and apply different filters to the 178data inside. In templates, you can also use for loops and if statements to handle more complex situations, 179like this, in ``roles/common/templates/iptables.j2``: 180 181.. code-block:: jinja 182 183 {% if inventory_hostname in groups['dbservers'] %} 184 -A INPUT -p tcp --dport 3306 -j ACCEPT 185 {% endif %} 186 187This is testing to see if the inventory name of the machine we're currently operating on (``inventory_hostname``) 188exists in the inventory group ``dbservers``. If so, that machine will get an iptables ACCEPT line for port 3306. 189 190Here's another example, from the same template: 191 192.. code-block:: jinja 193 194 {% for host in groups['monitoring'] %} 195 -A INPUT -p tcp -s {{ hostvars[host].ansible_default_ipv4.address }} --dport 5666 -j ACCEPT 196 {% endfor %} 197 198This loops over all of the hosts in the group called ``monitoring``, and adds an ACCEPT line for 199each monitoring hosts' default IPv4 address to the current machine's iptables configuration, so that Nagios can monitor those hosts. 200 201You can learn a lot more about Jinja2 and its capabilities `here <http://jinja.pocoo.org/docs/>`_, and you 202can read more about Ansible variables in general in the :ref:`playbooks_variables` section. 203 204.. _lamp_rolling_upgrade: 205 206The rolling upgrade 207=================== 208 209Now you have a fully-deployed site with web servers, a load balancer, and monitoring. How do you update it? This is where Ansible's 210orchestration features come into play. While some applications use the term 'orchestration' to mean basic ordering or command-blasting, Ansible 211refers to orchestration as 'conducting machines like an orchestra', and has a pretty sophisticated engine for it. 212 213Ansible has the capability to do operations on multi-tier applications in a coordinated way, making it easy to orchestrate a sophisticated zero-downtime rolling upgrade of our web application. This is implemented in a separate playbook, called ``rolling_update.yml``. 214 215Looking at the playbook, you can see it is made up of two plays. The first play is very simple and looks like this: 216 217.. code-block:: yaml 218 219 - hosts: monitoring 220 tasks: [] 221 222What's going on here, and why are there no tasks? You might know that Ansible gathers "facts" from the servers before operating upon them. These facts are useful for all sorts of things: networking information, OS/distribution versions, etc. In our case, we need to know something about all of the monitoring servers in our environment before we perform the update, so this simple play forces a fact-gathering step on our monitoring servers. You will see this pattern sometimes, and it's a useful trick to know. 223 224The next part is the update play. The first part looks like this: 225 226.. code-block:: yaml 227 228 - hosts: webservers 229 user: root 230 serial: 1 231 232This is just a normal play definition, operating on the ``webservers`` group. The ``serial`` keyword tells Ansible how many servers to operate on at once. If it's not specified, Ansible will parallelize these operations up to the default "forks" limit specified in the configuration file. But for a zero-downtime rolling upgrade, you may not want to operate on that many hosts at once. If you had just a handful of webservers, you may want to set ``serial`` to 1, for one host at a time. If you have 100, maybe you could set ``serial`` to 10, for ten at a time. 233 234Here is the next part of the update play: 235 236.. code-block:: yaml 237 238 pre_tasks: 239 - name: disable nagios alerts for this host webserver service 240 nagios: 241 action: disable_alerts 242 host: "{{ inventory_hostname }}" 243 services: webserver 244 delegate_to: "{{ item }}" 245 loop: "{{ groups.monitoring }}" 246 247 - name: disable the server in haproxy 248 shell: echo "disable server myapplb/{{ inventory_hostname }}" | socat stdio /var/lib/haproxy/stats 249 delegate_to: "{{ item }}" 250 loop: "{{ groups.lbservers }}" 251 252.. note:: 253 - The ``serial`` keyword forces the play to be executed in 'batches'. Each batch counts as a full play with a subselection of hosts. 254 This has some consequences on play behavior. For example, if all hosts in a batch fails, the play fails, which in turn fails the entire run. You should consider this when combining with ``max_fail_percentage``. 255 256The ``pre_tasks`` keyword just lets you list tasks to run before the roles are called. This will make more sense in a minute. If you look at the names of these tasks, you can see that we are disabling Nagios alerts and then removing the webserver that we are currently updating from the HAProxy load balancing pool. 257 258The ``delegate_to`` and ``loop`` arguments, used together, cause Ansible to loop over each monitoring server and load balancer, and perform that operation (delegate that operation) on the monitoring or load balancing server, "on behalf" of the webserver. In programming terms, the outer loop is the list of web servers, and the inner loop is the list of monitoring servers. 259 260Note that the HAProxy step looks a little complicated. We're using HAProxy in this example because it's freely available, though if you have (for instance) an F5 or Netscaler in your infrastructure (or maybe you have an AWS Elastic IP setup?), you can use modules included in core Ansible to communicate with them instead. You might also wish to use other monitoring modules instead of nagios, but this just shows the main goal of the 'pre tasks' section -- take the server out of monitoring, and take it out of rotation. 261 262The next step simply re-applies the proper roles to the web servers. This will cause any configuration management declarations in ``web`` and ``base-apache`` roles to be applied to the web servers, including an update of the web application code itself. We don't have to do it this way--we could instead just purely update the web application, but this is a good example of how roles can be used to reuse tasks: 263 264.. code-block:: yaml 265 266 roles: 267 - common 268 - base-apache 269 - web 270 271Finally, in the ``post_tasks`` section, we reverse the changes to the Nagios configuration and put the web server back in the load balancing pool: 272 273.. code-block:: yaml 274 275 post_tasks: 276 - name: Enable the server in haproxy 277 shell: echo "enable server myapplb/{{ inventory_hostname }}" | socat stdio /var/lib/haproxy/stats 278 delegate_to: "{{ item }}" 279 loop: "{{ groups.lbservers }}" 280 281 - name: re-enable nagios alerts 282 nagios: 283 action: enable_alerts 284 host: "{{ inventory_hostname }}" 285 services: webserver 286 delegate_to: "{{ item }}" 287 loop: "{{ groups.monitoring }}" 288 289Again, if you were using a Netscaler or F5 or Elastic Load Balancer, you would just substitute in the appropriate modules instead. 290 291.. _lamp_end_notes: 292 293Managing other load balancers 294============================= 295 296In this example, we use the simple HAProxy load balancer to front-end the web servers. It's easy to configure and easy to manage. As we have mentioned, Ansible has built-in support for a variety of other load balancers like Citrix NetScaler, F5 BigIP, Amazon Elastic Load Balancers, and more. See the :ref:`working_with_modules` documentation for more information. 297 298For other load balancers, you may need to send shell commands to them (like we do for HAProxy above), or call an API, if your load balancer exposes one. For the load balancers for which Ansible has modules, you may want to run them as a ``local_action`` if they contact an API. You can read more about local actions in the :ref:`playbooks_delegation` section. Should you develop anything interesting for some hardware where there is not a core module, it might make for a good module for core inclusion! 299 300.. _lamp_end_to_end: 301 302Continuous delivery end-to-end 303============================== 304 305Now that you have an automated way to deploy updates to your application, how do you tie it all together? A lot of organizations use a continuous integration tool like `Jenkins <https://jenkins.io/>`_ or `Atlassian Bamboo <https://www.atlassian.com/software/bamboo>`_ to tie the development, test, release, and deploy steps together. You may also want to use a tool like `Gerrit <https://www.gerritcodereview.com/>`_ to add a code review step to commits to either the application code itself, or to your Ansible playbooks, or both. 306 307Depending on your environment, you might be deploying continuously to a test environment, running an integration test battery against that environment, and then deploying automatically into production. Or you could keep it simple and just use the rolling-update for on-demand deployment into test or production specifically. This is all up to you. 308 309For integration with Continuous Integration systems, you can easily trigger playbook runs using the ``ansible-playbook`` command line tool, or, if you're using :ref:`ansible_tower`, the ``tower-cli`` or the built-in REST API. (The tower-cli command 'joblaunch' will spawn a remote job over the REST API and is pretty slick). 310 311This should give you a good idea of how to structure a multi-tier application with Ansible, and orchestrate operations upon that app, with the eventual goal of continuous delivery to your customers. You could extend the idea of the rolling upgrade to lots of different parts of the app; maybe add front-end web servers along with application servers, for instance, or replace the SQL database with something like MongoDB or Riak. Ansible gives you the capability to easily manage complicated environments and automate common operations. 312 313.. seealso:: 314 315 `lamp_haproxy example <https://github.com/ansible/ansible-examples/tree/master/lamp_haproxy>`_ 316 The lamp_haproxy example discussed here. 317 :ref:`working_with_playbooks` 318 An introduction to playbooks 319 :ref:`playbooks_reuse_roles` 320 An introduction to playbook roles 321 :ref:`playbooks_variables` 322 An introduction to Ansible variables 323 `Ansible.com: Continuous Delivery <https://www.ansible.com/use-cases/continuous-delivery>`_ 324 An introduction to Continuous Delivery with Ansible 325