Trying Ansible alternatives in python
My VPS (Virtual Private Server) is getting old and the company that runs it announced that next month, the promotion I used will expire and it will now cost more than three times to run it. Also, I would like to update other machines I have home, as they use old Ubuntu.
This has led me to seek some kind of deployment automation, so I can specify my infrastructure as a code and ideally never ever again spend much time with it.
Why not Ansible
I don't like YAML based configuration languages that overgrow into scripting languages. It's a classic example of the Greenspun's tenth rule:
Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
It is a nightmare to debug it, there is no support in IDE and so on.
The feeling of mess every time I try to use Ansible has led me to explore alternatives, preferably in python, which would define "recipes" or whatever as simple python code.
As it turned out, there aren't so many.
Fabric
Almost every article on the topic of "python ansible alternative" mentions fabric. It looks great, but like something slightly different. It is a system for running commands on remote hosts. Which is definitely part of what Ansible does, but I would also like some kind of abstraction layer.
>>> def disk_free(c):
... uname = c.run('uname -s', hide=True)
... if 'Linux' in uname.stdout:
... command = "df -h / | tail -n1 | awk '{print $5}'"
... return c.run(command, hide=True).stdout.strip()
... err = "No idea how to get disk space on {}!".format(uname)
... raise Exit(err)
I mean, I don't want to manually run apt install nginx
and then parse the output and try to decide whether the command successfully run (I did it before with paramiko and it is not fun, trust me).
I want something slightly more advanced, which does the parsing of standard utilities for me, and ideally with support of multiple OS, so when I decide to use CentOS instead of the Ubuntu server I am using now, it can cope with the different utilities for me.
Fabtools
Fabtools looks almost exactly like what I want:
from fabric.api import *
from fabtools import require
import fabtools
@task
def setup():
# Require some Debian/Ubuntu packages
require.deb.packages([
'imagemagick',
'libxml2-dev',
])
# Require a Python package
with fabtools.python.virtualenv('/home/myuser/env'):
require.python.package('pyramid')
# Require an email server
require.postfix.server('example.com')
# Require a PostgreSQL server
require.postgres.server()
require.postgres.user('myuser', 's3cr3tp4ssw0rd')
require.postgres.database('myappsdb', 'myuser')
# Require a supervisor process for our app
require.supervisor.process('myapp',
command='/home/myuser/env/bin/gunicorn_paster /home/myuser/env/myapp/production.ini',
directory='/home/myuser/env/myapp',
user='myuser'
)
# Require an nginx server proxying to our app
require.nginx.proxied_site('example.com',
docroot='/home/myuser/env/myapp/myapp/public',
proxy_url='http://127.0.0.1:8888'
)
# Setup a daily cron task
fabtools.cron.add_daily('maintenance', 'myuser', 'my_script.py')
It has just one downside; it also looks dead.
Last commit is 9 months ago, there is 78 unresolved issues, 28 waiting pull requests and list of supported operating systems is ancient:
- Debian family:
- Debian 6 (squeeze), 7 (wheezy), 8 (jessie)
- Ubuntu 10.04 (lucid), 12.04 (precise), 14.04 (trusty)
That 14 in the Ubuntu 14.04 (trusty) is the year of release: 2014.
The documentation also suck, and there is weird confusion about forks.
Fabrix
Then there is Fabrix, which looks like something between Fabric and Fabtools:
from fabrix.api import is_file_not_exists, yum_install
from fabrix.api import edit_file, edit_ini_section, replace_line
def install_php():
if is_file_not_exists("/etc/yum.repos.d/epel.repo"):
yum_install("https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm")
if is_file_not_exists("/etc/yum.repos.d/remi-php70.repo"):
yum_install("https://rpms.remirepo.net/enterprise/remi-release-7.rpm")
edit_file("/etc/yum.repos.d/remi-php70.repo",
edit_ini_section("[remi-php70]",
replace_line("enabled=0", "enabled=1")
)
)
yum_install("""
php-cli
php-common
php-fpm
php-gd
php-mbstring
php-mysql
php-pdo
php-pear
php-pecl-imagick
php-process
php-xml
php-opcache
php-mcrypt
php-soap
""")
It's kinda funny to me, because I've created something very similar with paramiko some time ago.
It also looks dead, only 204 commits and one contributor, last commit 15 months ago. I don't want to build my system on something that is already dead.
pyinfra
pyinfra looks promising and very not dead: 2233 commits, 14 contributors, last commit yesterday. That's what I am talking about!
from pyinfra.operations import apt
apt.packages(
{'Install iftop'},
'iftop',
sudo=True,
update=True,
)
Example works exactly as I wanted; declarative language, important parameters as, you know, parameters and not strings. Only weird thing is to specify description as set
, but whatever, I can see the line of reasoning here.
Documentation is also promising:


Trying pyinfra
As the pyinfra is the only thing that looks like it is not dead and it can do what I want, the decision is not that hard. So, lets try it:
$ pip install --user pyinfra
Now lets try hello world. I struggle with the port for a moment, because I use nonstandard port for tunneling through hotel and airport wifi, quick peek to help show that I should use --port
parameter:
$ pyinfra kitakitsune.org --port 443 exec -- echo "hello world"
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
[kitakitsune.org] Connected
--> Proposed changes:
Ungrouped:
[kitakitsune.org] Operations: 1 Commands: 1
--> Beginning operation run...
--> Starting operation: Server/Shell (u'echo hello world',)
[kitakitsune.org] hello world
[kitakitsune.org] Success
--> Results:
Ungrouped:
[kitakitsune.org] Successful: 1 Errors: 0 Commands: 1/1
Looks good. Let's try to create a more complicated deployment for a virtual Ubuntu server, that I've created some time ago in VirtualBox. I slightly struggle with the inventory.py
file, but then I find in the documentation correct parameters:
my_hosts = [
('192.168.0.106', {"ssh_port": "4433", "ssh_user": "b"}),
]
I quickly check whether it works with following deployment.py
file:
from pyinfra.modules import server
server.shell('echo "hello world"')
Which I run using following command:
pyinfra -v inventory.py deployment.py
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
[192.168.0.106] Connected
--> Preparing operations...
Loading: deployment.py
[192.168.0.106] Ready: deployment.py
--> Proposed changes:
Groups: my_hosts / inventory
[192.168.0.106] Operations: 1 Commands: 1
--> Beginning operation run...
--> Starting operation: Server/Shell ('echo "hello world"',)
[192.168.0.106] >>> sh -c 'echo "hello world"'
[192.168.0.106] hello world
[192.168.0.106] Success
--> Results:
Groups: my_hosts / inventory
[192.168.0.106] Successful: 1 Errors: 0 Commands: 1/1
Sudo
Support for sudo password was added after this article was published via the #305 in the version 0.15
released .
To use sudo, set global variable USE_SUDO_PASSWORD
to True
, so the pyinfra will ask for it interactively, or you can set it to password and it will be used automatically. You can also use --use_sudo_password
parameter on commandline or in inventory.py
.
from pyinfra.modules import server
USE_SUDO_PASSWORD=True
server.shell('echo "hello world"', sudo=True)
sudo=True
parameter says, that this command should be run with sudo. USE_SUDO_PASSWORD
says that sudo uses password, as you can also have sudo without password by setting %sudo ALL=(ALL:ALL) NOPASSWD:ALL
in /etc/sudoers
.
pyinfra -v inventory.py deployment.py
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
[192.168.0.106] Connected
--> Preparing operations...
Loading: deployment.py
Use of `pyinfra.modules` is deprecated, please use `pyinfra.operations`.
[192.168.0.106] Ready: deployment.py
--> Proposed changes:
Groups: my_hosts / inventory
[192.168.0.106] Operations: 1 Commands: 1
--> Beginning operation run...
--> Starting operation: Server/Shell ('echo "hello world"',)
[192.168.0.106] sudo password:
[192.168.0.106] Success
--> Results:
Groups: my_hosts / inventory
[192.168.0.106] Successful: 1 Errors: 0 Commands: 1/1
Setup nginx
Let's try something more complicated - to set up a nginx server and upload a correct config file for it:
from pyinfra.modules import apt
SUDO=True
apt.packages('nginx', update=True,present=True)
And it worked, with one exception, which is parsing of the output (yes, that's why parsing sucks):
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/util.py", line 471, in read_buffer
_print(line)
File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/util.py", line 455, in _print
line = print_func(line)
File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/connectors/util.py", line 61, in <lambda>
print_func=lambda line: '{0}{1}'.format(print_prefix, line),
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2192' in position 74: ordinal not in range(128)
2020-06-11T23:55:28Z <Greenlet at 0x7fde3f4dd6b0: read_buffer('stdout', <paramiko.ChannelFile from <paramiko.Channel 2 (op, <Queue at 0x7fde39f60f30 queue=deque([('stdout', u, print_func=<function <lambda> at 0x7fde39f5e950>, print_output=True)> failed with UnicodeEncodeError
[192.168.0.106] Success
--> Results:
Groups: my_hosts / inventory
[192.168.0.106] Successful: 1 Errors: 0 Commands: 1/1
Which again is weird, but I am probably using Czech localization, so I am not that surprised.
Running it again shows that the operation was executed successfully:
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
[192.168.0.106] Connected
--> Preparing operations...
Loading: deployment.py
Loaded fact deb_packages
[192.168.0.106] Ready: deployment.py
--> Proposed changes:
Groups: my_hosts / inventory
[192.168.0.106] Operations: 1 Commands: 1
--> Beginning operation run...
--> Starting operation: Apt/Packages ('nginx', u'update=True', u'present=True')
[192.168.0.106] >>> sudo -S -H -n sh -c 'apt-get update'
[192.168.0.106] Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
[192.168.0.106] Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
[192.168.0.106] Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
[192.168.0.106] Hit:4 http://archive.ubuntu.com/ubuntu bionic-security InRelease
[192.168.0.106] Reading package lists...
[192.168.0.106] Success
--> Results:
Groups: my_hosts / inventory
[192.168.0.106] Successful: 1 Errors: 0 Commands: 1/1
Config file
So, how do I upload a config file for the nginx?
from pyinfra.modules import files
files.put(
'configs/nginx.conf',
'/etc/nginx/nginx.conf',
user='root',
group='root',
mode='644',
)
You can also download files, sync whole directories and so on. Quite nice.
Start nginx
from pyinfra.modules import server
init.systemd('nginx', running=True, restarted=True, enabled=True)
Beautiful.
Conclusion
I really like pyinfra. It has its quirks, but they are just little annoyances, unlike big annoyances I find in other products, like Ansible. But unlike Ansible, it is understandable, easy to use, and it uses Python, which I know and like. My IDE can give me autocomplete and debugger, unlike other, YAML based DSLs.
Here is a whole config for the nginx deployment:
from pyinfra.modules import apt
from pyinfra.modules import init
from pyinfra.modules import files
SUDO=True
USE_SUDO_PASSWORD=True
apt.packages(
'nginx',
update=True,
present=True,
)
files.put(
'configs/nginx.conf',
'/etc/nginx/nginx.conf',
user='root',
group='root',
mode='644',
)
init.systemd(
'nginx',
running=True,
restarted=True,
enabled=True,
)
Relevant discussion
- hackernews (85 comments)
- /r/devops (20 comments)
- lobste.rs (5 comments)
- /r/python (4 comments)
Update
Changed heated tone about the Ansible, as this was not the point. Added callout about the VS Code Ansible add-on.
Update
People on hackernews pointed out other python projects, that may be relevant:
- https://bundlewrap.org/ (this looks especially interesting)
Update
Added information about the installation from the git repository for a sudo
support.
Update
Chapter about sudo updated, as the pyinfra was yesterday released with sudo support.