In the last several month Assaf have been (slowly) maintaining and working on updating and improving the Icinga2  Ansible playbooks , and as he worked on those he found that he needed the hosts to test on to be built and taken down in a fast repetition, and to preform the build and shutdown was too time consuming.

Welcome Terraform , the wonderful tool from HashiCorp has provided him with the ability to provision the server and remote nodes fast and in a reproducible way to ensure that each code run is tested in a clean and similar setup.

One issue was that the ‘Hosts’ file for the ansible run had to be manually changed each time with the new IP’s of the new instances ( AWS is nice for this short intervals) , and that slowed the progress down.

We know that many people are using ansible and terraform combination to manage their infrastructure, but in most cases the we found on line the ansible is called as a ‘local-exec’ provisioner at the end of the execution , and thus uses the internal variable’s of the terraform run, as we needed an external file for the testing ( for simulation of the user experience and the way the roles are looking at the inventory) it was important to create the inventory file in a specific way.

“Simple” most terraform users will say, “just use the provisioner ‘local-exec’ to write the output to a file” and they are correct, with a little caveat, if you wish to write the file in a specific resource creation order, you end up with a file that is out of order.
For example, here is an output file we got when building an icinga2 demo environment with a master and 2 nodes (webservers):

 [monitoring_servers]   # the Icinga roles need this group to know which are the master servers
54.202.16.213
34.217.59.140
34.214.204.190

This will cause our Ansible to read as if we have 3 master servers, and that of course is incorrect.

What the file should look like is

[monitoring_servers]
54.202.16.213
[webservers]
34.217.59.140
34.214.204.19

Do notice the group separator that is required/was added in the end, this was skipped in all the previous runs due to the order of creation, so to fix this issue the solution was very simple. ‘depends_on‘ which is a simple “wait” function that causes resources to wait for one to be ready.
In this case we wait for the server IP to be added and then we add the label and the IP’s of the nodes.

This solution has enabled us to speed out testing quite nicely and is a thing that should allow us to bring more improvements to the Ansible playbooks in a faster pace.