Bystroushaak's blog / English section / Explorations / Trying Ansible alternatives in python

Trying Ansible alternatives in python

👉
There is also Russian version of this article: https://softdroid.net/poisk-alternativ-ansible-v-python

My VPS (Virtual Private Server) is getting old and the company that runs it announced that next month, the promotion I used will expire and it will now cost more than three times to run it. Also, I would like to update other machines I have home, as they use old Ubuntu.

This has led me to seek some kind of deployment automation, so I can specify my infrastructure as a code and ideally never ever again spend much time with it.

Why not Ansible

I don't like YAML based configuration languages that overgrow into scripting languages. It's a classic example of the Greenspun's tenth rule:

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

It is a nightmare to debug it, there is no support in IDE and so on.

⚠️
It was pointed to me, that VS Code has an add-on specifically for Ansible.

The feeling of mess every time I try to use Ansible has led me to explore alternatives, preferably in python, which would define "recipes" or whatever as simple python code.

As it turned out, there aren't so many.

Fabric

Almost every article on the topic of "python ansible alternative" mentions fabric. It looks great, but like something slightly different. It is a system for running commands on remote hosts. Which is definitely part of what Ansible does, but I would also like some kind of abstraction layer.

>>> def disk_free(c):
...     uname = c.run('uname -s', hide=True)
...     if 'Linux' in uname.stdout:
...         command = "df -h / | tail -n1 | awk '{print $5}'"
...         return c.run(command, hide=True).stdout.strip()
...     err = "No idea how to get disk space on {}!".format(uname)
...     raise Exit(err)

I mean, I don't want to manually run apt install nginx and then parse the output and try to decide whether the command successfully run (I did it before with paramiko and it is not fun, trust me).

I want something slightly more advanced, which does the parsing of standard utilities for me, and ideally with support of multiple OS, so when I decide to use CentOS instead of the Ubuntu server I am using now, it can cope with the different utilities for me.

Fabtools

Fabtools looks almost exactly like what I want:

from fabric.api import *
from fabtools import require
import fabtools

@task
def setup():
    # Require some Debian/Ubuntu packages
    require.deb.packages([
        'imagemagick',
        'libxml2-dev',
    ])

    # Require a Python package
    with fabtools.python.virtualenv('/home/myuser/env'):
        require.python.package('pyramid')

    # Require an email server
    require.postfix.server('example.com')

    # Require a PostgreSQL server
    require.postgres.server()
    require.postgres.user('myuser', 's3cr3tp4ssw0rd')
    require.postgres.database('myappsdb', 'myuser')

    # Require a supervisor process for our app
    require.supervisor.process('myapp',
        command='/home/myuser/env/bin/gunicorn_paster /home/myuser/env/myapp/production.ini',
        directory='/home/myuser/env/myapp',
        user='myuser'
        )

    # Require an nginx server proxying to our app
    require.nginx.proxied_site('example.com',
        docroot='/home/myuser/env/myapp/myapp/public',
        proxy_url='http://127.0.0.1:8888'
        )

    # Setup a daily cron task
    fabtools.cron.add_daily('maintenance', 'myuser', 'my_script.py')

It has just one downside; it also looks dead.

Last commit is 9 months ago, there is 78 unresolved issues, 28 waiting pull requests and list of supported operating systems is ancient:

That 14 in the Ubuntu 14.04 (trusty) is the year of release: 2014.

The documentation also suck, and there is weird confusion about forks.

Fabrix

Then there is Fabrix, which looks like something between Fabric and Fabtools:

from fabrix.api import is_file_not_exists, yum_install
from fabrix.api import edit_file, edit_ini_section, replace_line

def install_php():

    if is_file_not_exists("/etc/yum.repos.d/epel.repo"):
        yum_install("https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm")

    if is_file_not_exists("/etc/yum.repos.d/remi-php70.repo"):
        yum_install("https://rpms.remirepo.net/enterprise/remi-release-7.rpm")

    edit_file("/etc/yum.repos.d/remi-php70.repo",
        edit_ini_section("[remi-php70]",
            replace_line("enabled=0", "enabled=1")
        )
    )

    yum_install("""
            php-cli
            php-common
            php-fpm
            php-gd
            php-mbstring
            php-mysql
            php-pdo
            php-pear
            php-pecl-imagick
            php-process
            php-xml
            php-opcache
            php-mcrypt
            php-soap
    """)

It's kinda funny to me, because I've created something very similar with paramiko some time ago.

It also looks dead, only 204 commits and one contributor, last commit 15 months ago. I don't want to build my system on something that is already dead.

pyinfra

pyinfra looks promising and very not dead: 2233 commits, 14 contributors, last commit yesterday. That's what I am talking about!

from pyinfra.operations import apt

apt.packages(
    {'Install iftop'},
    'iftop',
    sudo=True,
    update=True,
)

Example works exactly as I wanted; declarative language, important parameters as, you know, parameters and not strings. Only weird thing is to specify description as set, but whatever, I can see the line of reasoning here.

Documentation is also promising:

Trying pyinfra

As the pyinfra is the only thing that looks like it is not dead and it can do what I want, the decision is not that hard. So, lets try it:

$ pip install --user pyinfra

Now lets try hello world. I struggle with the port for a moment, because I use nonstandard port for tunneling through hotel and airport wifi, quick peek to help show that I should use --port parameter:

$ pyinfra kitakitsune.org --port 443 exec -- echo "hello world"
--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [kitakitsune.org] Connected

--> Proposed changes:
    Ungrouped:
    [kitakitsune.org]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Server/Shell (u'echo hello world',)
[kitakitsune.org] hello world
    [kitakitsune.org] Success

--> Results:
    Ungrouped:
    [kitakitsune.org]   Successful: 1   Errors: 0   Commands: 1/1

Looks good. Let's try to create a more complicated deployment for a virtual Ubuntu server, that I've created some time ago in VirtualBox. I slightly struggle with the inventory.py file, but then I find in the documentation correct parameters:

my_hosts = [
    ('192.168.0.106', {"ssh_port": "4433", "ssh_user": "b"}),
]

I quickly check whether it works with following deployment.py file:

from pyinfra.modules import server

server.shell('echo "hello world"')

Which I run using following command:

pyinfra -v inventory.py deployment.py
--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [192.168.0.106] Connected

--> Preparing operations...
    Loading: deployment.py
    [192.168.0.106] Ready: deployment.py

--> Proposed changes:
    Groups: my_hosts / inventory
    [192.168.0.106]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Server/Shell ('echo "hello world"',)
[192.168.0.106] >>> sh -c 'echo "hello world"'
[192.168.0.106] hello world
    [192.168.0.106] Success

--> Results:
    Groups: my_hosts / inventory
    [192.168.0.106]   Successful: 1   Errors: 0   Commands: 1/1

Sudo

Support for sudo password was added after this article was published via the #305 in the version 0.15 released .

To use sudo, set global variable USE_SUDO_PASSWORD to True, so the pyinfra will ask for it interactively, or you can set it to password and it will be used automatically. You can also use --use_sudo_password parameter on commandline or in inventory.py.

from pyinfra.modules import server

USE_SUDO_PASSWORD=True

server.shell('echo "hello world"', sudo=True)

sudo=True parameter says, that this command should be run with sudo. USE_SUDO_PASSWORD says that sudo uses password, as you can also have sudo without password by setting %sudo ALL=(ALL:ALL) NOPASSWD:ALL in /etc/sudoers.

pyinfra -v inventory.py deployment.py
--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [192.168.0.106] Connected

--> Preparing operations...
    Loading: deployment.py
    Use of `pyinfra.modules` is deprecated, please use `pyinfra.operations`.
    [192.168.0.106] Ready: deployment.py

--> Proposed changes:
    Groups: my_hosts / inventory
    [192.168.0.106]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Server/Shell ('echo "hello world"',)
[192.168.0.106] sudo password: 
    [192.168.0.106] Success

--> Results:
    Groups: my_hosts / inventory
    [192.168.0.106]   Successful: 1   Errors: 0   Commands: 1/1

Setup nginx

Let's try something more complicated - to set up a nginx server and upload a correct config file for it:

from pyinfra.modules import apt


SUDO=True

apt.packages('nginx', update=True,present=True)

And it worked, with one exception, which is parsing of the output (yes, that's why parsing sucks):

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/util.py", line 471, in read_buffer
    _print(line)
  File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/util.py", line 455, in _print
    line = print_func(line)
  File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/connectors/util.py", line 61, in <lambda>
    print_func=lambda line: '{0}{1}'.format(print_prefix, line),
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2192' in position 74: ordinal not in range(128)
2020-06-11T23:55:28Z <Greenlet at 0x7fde3f4dd6b0: read_buffer('stdout', <paramiko.ChannelFile from <paramiko.Channel 2 (op, <Queue at 0x7fde39f60f30 queue=deque([('stdout', u, print_func=<function <lambda> at 0x7fde39f5e950>, print_output=True)> failed with UnicodeEncodeError

    [192.168.0.106] Success

--> Results:
    Groups: my_hosts / inventory
    [192.168.0.106]   Successful: 1   Errors: 0   Commands: 1/1

Which again is weird, but I am probably using Czech localization, so I am not that surprised.

Running it again shows that the operation was executed successfully:

--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [192.168.0.106] Connected

--> Preparing operations...
    Loading: deployment.py
    Loaded fact deb_packages
    [192.168.0.106] Ready: deployment.py

--> Proposed changes:
    Groups: my_hosts / inventory
    [192.168.0.106]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Apt/Packages ('nginx', u'update=True', u'present=True')
[192.168.0.106] >>> sudo -S -H -n sh -c 'apt-get update'
[192.168.0.106] Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
[192.168.0.106] Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
[192.168.0.106] Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
[192.168.0.106] Hit:4 http://archive.ubuntu.com/ubuntu bionic-security InRelease
[192.168.0.106] Reading package lists...
    [192.168.0.106] Success

--> Results:
    Groups: my_hosts / inventory
    [192.168.0.106]   Successful: 1   Errors: 0   Commands: 1/1

Config file

So, how do I upload a config file for the nginx?

from pyinfra.modules import files

files.put(
    'configs/nginx.conf',
    '/etc/nginx/nginx.conf',
    user='root',
    group='root',
    mode='644',
)

You can also download files, sync whole directories and so on. Quite nice.

Start nginx

from pyinfra.modules import server

init.systemd('nginx', running=True, restarted=True, enabled=True)

Beautiful.

Conclusion

I really like pyinfra. It has its quirks, but they are just little annoyances, unlike big annoyances I find in other products, like Ansible. But unlike Ansible, it is understandable, easy to use, and it uses Python, which I know and like. My IDE can give me autocomplete and debugger, unlike other, YAML based DSLs.

Here is a whole config for the nginx deployment:

from pyinfra.modules import apt
from pyinfra.modules import init
from pyinfra.modules import files


SUDO=True
USE_SUDO_PASSWORD=True


apt.packages(
    'nginx',
    update=True,
    present=True,
)

files.put(
    'configs/nginx.conf',
    '/etc/nginx/nginx.conf',
    user='root',
    group='root',
    mode='644',
)

init.systemd(
    'nginx',
    running=True,
    restarted=True,
    enabled=True,
)

Relevant discussion

Update

Changed heated tone about the Ansible, as this was not the point. Added callout about the VS Code Ansible add-on.

Update

People on hackernews pointed out other python projects, that may be relevant:

Update

Added information about the installation from the git repository for a sudo support.

Update

Chapter about sudo updated, as the pyinfra was yesterday released with sudo support.

Become a Patron