Experiment Automation Basics

Ansible is the recommended tool for experiment automation. This guide will go over the basics of using Ansible for Merge experiments, the general Ansible docs can be found here.

Ansible allows you to automate a collection of experiment nodes from a centralized file called a playbook.

We recommend executing playbooks from an XDC attached to your experiment materialization.


The following topology will be used in this example. It's the simple two node topology depicted in the diagram above.

import mergexp as mx
net = mx.Topology('ansible')
nodes = a, b = [net.device(name) for name in ['a', 'b']]
link = net.connect(nodes)
link[a].ip.addrs = ['']
link[b].ip.addrs = ['']


A playbook is a sequence of plays that can be anything from provisioning users, copying files, configuring software to running generic shell commands and scripts. Here is an example of a playbook that installs and runs the iperf3 network performance tool across two nodes. Here we assume the nodes are called named a and b and that the experiment network IP addresses of these nodes are and respectively.


- hosts: all
become: true
- name: install iperf3
name: iperf3
- hosts: b
- name: run iperf3 in server mode
shell: iperf3 --server --daemon --one-off
- hosts: a
- name: run iperf3 in client mode
shell: iperf3 -c --logfile /tmp/results
- name: collect results
src: /tmp/results
dest: results
flat: yes


Ansible requires a file called an inventory to tell it how to reach the nodes it's automating. This file is typically called hosts. Here is an example for our two node topology.



Names in square brackets indicate groups, the names below the square brackets are the members of a group. The all group was used in the first sequence of plays above, and then individual nodes were used for the follow-on play sequences.

You should be able to retrieve a very basic inventory from the xdc via the command xdc ansible inventory (or store it by redirecting the output to a file).

Placing the inventory at location /etc/ansible/hosts causes ansible to use it by default (without requiring a command line option for inventory). You can also alter this location in your ansible.cfg file.


Ansible configuration is kept in a file called ansible.cfg. This file can be global at /etc/ansible/ansible.cfg, or it can be placed in the local directory where you execute a playbook from. There are a few options that are very useful to set for executing from an XDC.

# don't check experiment node keys, if this is not set, you will have to
# explicitly accept the SSH key for each experiment node you run Ansible
# against
host_key_checking = False
# configure up to 5 hosts in parallel
forks = 5
# connection optimization that increases speed significantly
pipelining = True


To execute the playbook.yml above we simply type

ansible-playbook -i hosts playbook.yml

After executing the playbook, you should have a results file in your local directory like the following

$ cat results
Connecting to host, port 5201
[ 6] local port 40620 connected to port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 6] 0.00-1.00 sec 114 MBytes 954 Mbits/sec 0 437 KBytes
[ 6] 1.00-2.00 sec 113 MBytes 947 Mbits/sec 0 437 KBytes
[ 6] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 437 KBytes
[ 6] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 0 482 KBytes
[ 6] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 482 KBytes
[ 6] 5.00-6.00 sec 111 MBytes 929 Mbits/sec 75 400 KBytes
[ 6] 6.00-7.00 sec 112 MBytes 943 Mbits/sec 0 426 KBytes
[ 6] 7.00-8.00 sec 113 MBytes 945 Mbits/sec 0 426 KBytes
[ 6] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 426 KBytes
[ 6] 9.00-10.00 sec 113 MBytes 946 Mbits/sec 0 426 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 6] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 75 sender
[ 6] 0.00-10.04 sec 1.09 GBytes 937 Mbits/sec receiver


If you are running a larger experiment, or if you need to optimize your ansible runs, you can change some ansible configuration options to try and reduce ansible's memory load on the XDC, such as caching your hosts' facts in a json file.

# fact caching to file instead of in memory
fact_caching = jsonfile
# set directory/location of fact cache files
fact_caching_connection = /tmp/.ansible/fc
# ask ansible to gather facts only when necessary
gathering = smart

OpenSSH command update

It is a relatively small optimization, but you can ask ssh to try your ssh keys before any other authentication types.

# add 'PreferredAuthentication' option
ssh_args = -C -o PreferredAuthentications=publickey -o ControlMaster=auto -o ControlPersist=60s

Timing analysis callbacks

If you want to try to do some analysis on your execution times or other ansible functions, you can turn on callbacks in the configuration file and get a additional information. See ansible documentation: callbacks:profile_tasks.

You can see what callback plugins are available on your host via ansible-doc -t callback -l. profile_tasks, role_tasks, and timer are the plugins you're looking for.

Callbacks may also take some time, so feel free to turn them back off once you are no longer in need of timing analysis.

# examine timing for tasks
callback_whitelist = profile_tasks, timer

Host grouping and role assignment

Host grouping is another effective optimization technique that can be utilized from your playbook via inventory group and utilization of roles.

An example of a play running roles against a group of servers:

- name: apply all settings to host group target_servers
hosts: target_servers
become: yes
- generic_hosts_configuration
- target_server_packages
- make_target_servers_ready

This kind of role grouping is useful to keep ansible working on the same set of hosts through several sets of changes.

Additionally, if you have a set of tasks or a role where there is no ordering necessary between hosts, you can specify an alternative "strategy". Strategies are discussed minimally in the ansible documentation, but available strategy plugins and their descriptions are available via the command ansible-doc -t strategy -l (list) and ansible-doc -t strategy <strategy name>. You may wish to explore the host_pinned or free strategies.

Asynchronous tasks

You can run long-running tasks "in the background" using ansible's asynchronous task handler. This is a bit of a complex topic, so it is suggested to read the available documentation in its entirety before attempting asynchronous tasks, and it is very important to keep in mind that future tasks depending on asynchronous tasks must wait for them or risk play failure.

Python Dependencies

Many people use a tool called pip to install python packages, and ansible includes a built-in pip package management module, however, using pip as a package manager is several times slower than using the system-level package manager (that is, yum/dnf or apt). If you have python package dependencies, try searching the system package managers for those dependencies to see if the version available to the system is acceptable for your use. If so, you can install them using the system-level packager and reduce play wait time for pip. Grouping the system packager installs into existing system packager plays also reduces ansible play run times.

Note: Pypi is currently limiting pip's access to xmlrpc calls (it is disallowed). There is a chance that pip installation will be unavailable or limited in the future.