Bystroushaak's blog / English section / Stories / War stories: Romanian Infocenter aka the pontaj story

War stories: Romanian Infocenter aka the pontaj story

⚠️
Just because I enjoy living my life without legal drama; This blog was published more than 3 years after it was written. All names, characters, and incidents portrayed here are fictitious. No identification with actual persons (living or deceased), companies, places, buildings, and products is intended or should be inferred. I work somewhere else now, and the company where this happened also works differently now.

Let me tell you a story about a deployment to the production. I work in a big company, compared to the local standards. Most people in the country where I live heard about "us". More than 1/14 of all citizens living here are our customers.

It is thus, quite natural, that we have several hundred employees in the research and development division. I am part of a small team of 14 developers and testers. I develop software in python, that runs on Linux. Mostly backend, and sometimes also PyQt apps. You can probably say, that I am a senior developer.

I am telling you all of this to get some context. We are a big corporate company, and we have processes, which mostly work. I've worked most of my life in smaller startups, so I was quite impressed when I've got in the company because everything was quite smooth. I didn't have to set up anything, everything was already done, debugged long time ago, used by sane programmers doing their job.

I am not talking about bureaucratic specifics, but the basic stuff. The stuff that even total amateurs quickly discover — the things which are often a difference between sanity and insanity. Things typical startups have half broken or not implemented.

For example; Use SCM (git). Test your code in dev environment. Do code reviews. Write tests, or run your changes via testers. Only the stuff reviewed by someone else can get into the production. Talk to the infra guys about network requirements and access before the deployment, not after. And so on.

We also have architects, who think about the big picture and how various technical parts of the multinational company talk to each other via all kinds of API. And also analysts, who think about business requirements.

What I find most fascinating about these guys is that they actually really saved my time. They oiled the gears for me before I went and did the programming. That's something I wasn't used to.

To sum it up; our processes work, and without much bullshit. Really. It's all almost a miracle.

Some time ago, our company expanded to other markets. We bought companies in the Poland, Slovakia, and Romania. Poland and Slovakia were already integrated into our software stack, and we were preparing for the integration of the Romania for more than half of the year. New localization was added, configuration, network access, countless changes in the apps because of the different legal requirements.

The big release was expected to be next month. All of Romania retail shops shall switch to our software platform. Instead of their old Windows-based system, they'll now use a Linux stack with our solution.

Help with packaging

I am doing my job, when suddenly, I am invited to a new group in MS Teams. I hate MS Teams.

"Hi" writes the guy I'll call John. I've worked with him before. He is an infra guy, sysadmin with a lot of knowledge about Linux, who is weirdly obsessed with security. Weirdly, because most of the time, it doesn't make any sense. He always pushes things like using self-signed HTTPS in network, that already uses encrypted VPN. Or to sign RPM packages that are downloaded from the repository, which uses password for login and SSL for communication. But most of the time, he can be reasoned with, and he has a lot of skill. He knows what he is doing, when it comes to the topic of his job.

Anyway, I reply with greetings. He explains that he needs help with creating an RPM package of three python applications developed “externally”. They apparently should be part of the upcoming official release in Romania. One of them is a thing called “Infocenter”, and he explains, that that's the important stuff, and that he needs it, quickly.

"Okay, give me the sources, I'll help you."

"I don't have the sources", he writes. "I am just trying to explain the whole process to this guy." And I look, and I see that there is another person in the chat, whom I'll call Sindibad.

"Sindibad is not a Linux developer, he is a PHP web developer, and he knows only windows", writes John.

I can do external apps. I can do packaging. But teaching someone who has literally no python and no Linux background? No way. That would take far too much time. And to do it remotely?

"Sorry guys, this would take too much time, and I have other tasks to do", I write. And really, I've got my share of tasks to finish, and other people depend on me being on time. Just because I want to avoid letting them hang, I post links to several articles about python and Linux packaging. I've used them as a reference in the past, and I'm certain that if not Sindibad, then John can do the job if he reads them.

Sindibad is optimistic. "Thanks man, I'll look into it and create the package in no time!"

Yeah. Sure.

Three days later, some woman I've never seen before pops into the group chat. Let's call her Simone. She is asking how it is going.

"We're on the right path", Sindibad replies. "I hope by the end of this week everything should be alright".

Most important app

Several weeks go by, and I've already forgotten about the whole incident. I am doing my job, this sprint it is mostly adding new menus into our main GUI program, with a lot of weird validation rules dictated by business. I am also connecting the app to the backend services, and I even have time to do some refactoring.

Suddenly, I am added to another MS teams group chat. *Sigh*. I really hate when people do this. This time, there is about a dozen of people, and I don't know half of them. I can see Simone, John, some higher ups from our branch and a lot of Romanian sounding names.

There is a lot of passive aggressivity flying around. The message is clear; management people are not happy that the Infocenter app is not yet packaged. We need it for our big release! It's a critical app, and it can delay the whole release!

Simone is sending impolite messages left and right. The content can be reduced to the fact, that the app was promised to be packaged last week (several weeks after I've first heard the promise that it will be done at the end of the week). She is really on the edge because the lack of Infocenter app threatened to stop the whole deployment to Romania, something hundreds of people worked on.

What a weird drama. Last time I've seen this, there was no ticket, there were no business requirements, just a remote colleague asking for help. What happened? And who promised it?

I am quite surprised to see my "product owner" popping into the chat. The guy, who is something like my boss, but not really, he just gives me tasks and generally says what I should do.

I am even more surprised to see him promise to everyone, that I will look into the issue of packaging and make it done.

The Infocenter

Charles Babbage began his book Passages from the Life of the Philosopher with meditation about names:

If he was still alive, I could show him, that this is not always the case. Some names resonate with people and make them assume things. For example "Infocenter".

What would you expect?

I can tell you, that many people expected many things about the app. It is an externally developed app. It is a critical app, and the release can't go without it. I've got direct messages from three random people I've never seen before, urging me to do the job quickly.

So, I did what I always do; asked for access to the git repository.

"There is no git repository. Please, contact Sindibad, he knows all the details."

Of course, I write to Sindibad, asking for the git repository. It is quite weird to have it offline and not uploaded to the Bitbucket, like everything else and every other externally developed app, but whatever.

Sindibad replies:

"There is no git. Here, have a zip file with compiled sources."

No git, huh. Okay. Maybe he uses some weird old system of managing the source codes, something like SVN. And what's with the compiled sources? Did someone really compile the python source code into bytecode (.pyc) files? That's weird, we usually request sources even for the externally developed stuff.

I download the zip file (I really hate MS Teams, which tries to be smart and show me all kinds of weird previews into the file instead of just downloading it), unpack them and look unbelievably at the 415 files, many of which are .so (shared library binaries). From the directory structure, it looks like someone compiled half of the python's standard library, with all kinds of dependencies.

What the hell? Did someone compile the source codes using something like cx_freeze? It really makes no sense as a security or copyright measure because you can unpack the original sources from the binaries.

"I couldn't make the compiled version run on Linux, but it works on Windows", writes Sindibad.

Wait, what? The suspicion grows in me, so I ask him; "are you the developer of this app?". And he confirms it and gives me a some info about the app.

He developed it two years ago, and just compiled it into the .exe, which run as a service on the windows. The app is just one big popup window, which jumps into the foreground when there is a new message on the intranet, or when the people in the retail store didn't log data into their timesheet. You can click on the left button, which opens Firefox, or the right, which closes the app for the next five minutes. Then it pops on you again.

(Recreation of the app using Salt wireframes)

It takes a bit of persuasion, but I finally manage to get him to send me the source code. I open the directory and here I see the whole infamous Infocenter. It's just one script with 181 lines of code.

There is no configuration (hardcoded URLs with no possibility to run it in test environment for example), no i18n (multi-language support, limba romana bebelus!), no tests, no proper logging, although there is some attempt to catch some exceptions and log them to the file called Errors.txt. But it is created in the directory where the script runs, and it won't work in production. There are no comments, no business requirements that would say what the app should actually do.

This may sound bureaucratic to you, but trust me, you will appreciate it when your Romanian colleague wants you to add new features.

Everything is crammed into three functions and all variable names are in Romanian language. Half of them starts with the prefix pontaj_, which I can't make any sense of, even using Google Translate. There is not even a commandline argument parser (it would be nice to have something like --test for test runs).

This is the Most Important App ©® no one has seen up to this point, which made many people do several meetings, write countless messages and generally do fuss all over in our R&D division. I am talking about high ranked managers, bosses of dozens of people, who are themselves bosses of people.

Quality

When I first learned how to drive a car, I thought that's it. I've learned the rules and I knew how to operate a car, so I am now a driver. Then I've begun to discover things like how to actually drive smoothly so that it feels "normal" to the people who ride with you. How to predict other people on the roads, and how to signal my intentions to them. How to use gear shift correctly and in the right way. How to use engine to lower your speed before you use brakes. How to conserve gas.

If you learn to use all of this correctly, you can drive very pleasantly and defensively. Your driving is now about conservation of momentum and signaling to the others, not just about making the box on the wheels move in the direction you want. You don't make sudden changes, and people expect you to behave in a certain way. Like some dance, or maybe tai chi. Or maybe you can learn to drive quickly and aggressively, and drive like a racer. There are many styles that you can optimize to.

In short, there is a lot more to the driving, than just the ability to drive and the knowledge of the rules in your country.

Like every other field of human knowledge, the programming is the same. When you learn some programming language, you are only at the beginning. The deployment to the production requires a lot more, and to actually write smooth, understandable and clean code is whole different knowledge.

It is like with human languages; the ability to speak is great, but the ability to write a story is something different. And the ability to write a great story, that is something that takes time and a lot of learning. You can't just be a master of the language, you also have to know about the architecture of stories. About human archetypes. Otherwise, it is just a poetic mess.

When I first saw the source code for the Infocenter app, I've immediately understood, that Sindibad barely knew python, and had zero experience with everything else. Why he was tasked with packaging and deployment of the app, or even the development, was beyond my understanding. It was clearly beyond his abilities. He could probably learn all of that in time, but not in the available timeframe.

I've chatted with him and proposed, that I'll create a git repository for the app, do some refactoring, add missing stuff and maybe rewrite the app to the PyQt. He had some issues with Tkinter, or more specifically with PIL (Python Imaging Library), which he used for reading an image, and he didn't understand what to do because it wasn't available in our CentOS package repository.

He agreed. Basically, my overall message for him was "don't worry, I'll make all necessary changes and then give the project back to you, or maybe take it under our team". After all, he doesn't do python apps, and we have several people, including me, doing just that.

During this time, several people sent me either direct messages, or messages in group chats, urging me to package the app. I've summed our agreement to some group chats. People were generally happy because I've given them a clear timeframe and set of steps that will be done.

I've also asked my boss to create a git repository for the project because for some weird reason, I don't have permissions to create it in our namespace.

That was all I managed to do on this project on Monday.

Git

For those of you who don't know what the git is, here is a short explanation.

Git is an SCM — Source Code Management system. It works by creating snapshots of the directory, where you initialize it. You make some modifications to your files (text, source code, pictures, anything really), then let the git scan the directory, describe the change you've made and then commit it.

In this sense, it is similar to how Wikipedia works; you introduce some changes to some article, and then you save the changes, with the description. This creates a list of changes (history) people made to articles as the time goes.

(List of changes for the wikipedia page about git.)

You can then at any time return to the saved version, or compare it with other versions. Unlike Wikipedia, git works on any files, not just articles. What's more, it can synchronize these changes in a decentralized manner, from computer to computer, from user to user. You can have alternative histories called "branches", and do all kind of crazy stuff with them. It is like if everyone had their pocket Wikipedia, made changes in their version and then sent the changes to other people.

I can for example make changes in the source code repository, commit them with git, and then send the changes, or the whole repository, to you. We can both continue working on the project. Git offers us a way how to synchronize the files between our computers, and also how to solve conflicts, for example when we both work on the same file and make changes to the same part.

Countless people use git nowadays. And the people who don't use git use some other SCM, for example mercurial, or pijul. To create source code for some program without some kind of SCM is really a madness. You live in `now`, only with changes you've just made. Any mistake can torpedo your project at any time.

These days, it is also typical to have all kind of other systems connected to your git repository. You can have revision system, where you can write comments to the changes made by others, and approve or disapprove their work. You can have a ticket system connected to the git, so your colleagues know what you are working on, and the ticket automatically changes status as you work on it.

In our company, we have all this (JIRA + Bitbucket), along with a build system (Jenkins), which automatically checks the code for bugs, and in the case of projects that do have packages, also creates those packages automatically for you. You as a programmer can just make the changes, click on a button and the automation system does a lot of hard work for you, in a matter of seconds.

You can now, maybe better understand, why I was so surprised that there was no git repository in the Infocenter case. Not only that, it is really weird to keep the source codes only on your computer because when there is some kind of hardware failure, thievery or mistake, you lose everything. But also, your colleagues can't easily share or discuss your code. And there is no automation happening.

It's really amateurish madness to have production code, which should run on thousands of computers directly generating revenue for the whole company, stored only on some dude's computer. People will need to modify it as the time goes by, do hotfixes when the bugs are found. What were they thinking?

Refactoring

The next day, I added the initial commit to the git repository. I've faked the author under the Sindibad's name, just so that he has some history in the repo and everyone can see that it was his project. After all, I felt it would be better not to "steal" his contribution.

First thing I've decided to do then was to run the project. I've asked Sindibad for test data from the API because I obviously wasn't getting the right data I wanted. I could see, in the code, that the API used the hostname of my computer to decide what to show me. So, I've replaced the function that used to download the data from the API to return the mock data for now.

Then I've decided to refactor the project because frankly, it was a mess. The whole structure was so tangled and chaotic, that it reminded me of the game untangle, where you have connected points, and you have to untangle them so that the connections between them don't cross.

The first part of the refactoring was really the same principle; to rearrange the code, so that cognitive load is lower, and you don't have to stare into the code for so long, trying to untangle it in your head. Like untangle game, but you don't even see the picture, just the text description of it. In random order.

One of the weirdest things was that Sindibad defined the graphical interface as kind of the table with several rows, and then added data into the rows. The weird part was that he added the data non-linearly. First, he added the first row. Then sixth. Then second. Then fourth. It was so weird to see many lines of code just jump all over the logical order of things. I've linearized it and in the process also removed unnecessary parts of the code, which were obviously redundant, once you could see the whole structure clearly.

Then I've renamed the variables that were named in Romanian language. It took me a while with Google Translate, and I also had to ask what the pontaj means because it was all over the place and made no sense (if I recall correctly, it was something like "timesheet").

I've split the code into smaller variables, rewritten the error handling, and factored out constants, so instead of the stuff like "#FF0000" all over the place, you can now clearly see something like Colors.red.

When it all worked, I've added an argparse interface, so you could modify the behavior of the program with commandline switches. I've added --debug and --mock for purposes of debugging on the systems which had a hostname that the API didn't know.

I was quite unsure why Sindibad used PIL to load the images in the first place. I don't really know tkinter, but a quick google query showed me that you don't really need to use PIL. So, I've removed the PIL dependency and used different methods to load the images and it worked.

I won't mention all antipatterns I've seen in the Sindibad's code. Just to give you a quick review, I've removed reused variables, some of which were used three or four times in different contexts in several places all over the project. Sindibad later asked what is the cost of the variable initialization in python. It took me some time to explain to him, that the way how it works, and in the project of this size, this is really irrelevant question and premature optimization. Other antipattern was super long try:except: blocks, which I've refactored to function calls.

There were hardcoded expectations about the system, which I've mostly changed to multiplatform calls, and there was a hardcoded path to the Firefox (browser) binary on the Sindibad's computer, so I rewrote that to xdg-open.

That was just about everything for day two. I wrote about the progress made that day to the group chats on MS Teams (in English and in Czech languages), and said that I'll work on the build pipeline next day.

Sindibad wrote something positive. And then he casually mentioned that he rewrote the project.

Whaaat? I was quite certain that he didn't yet have access to the repository I've filled with changes that day. Our user support guys don't work that fast. I quickly checked, and confirmed that there are no changes in the repository I've created for the app. How could he rewrite the project?

I hoped that he didn't work on the old version, which was now approximately twenty commits behind the version I've just pushed to the repository. That would be crazy. And why would he do that? We had a clear agreement yesterday, that I'll make my changes and then hand the project back to him.

RPM build

Day after that, I've tried to get hold of Sindibad, but he wasn't available. I've decided to ignore his remarks about "rewrite" and continue to work on the project.

We use CentOS on the target machines, and Spacewalk as a Linux package repository. So, I had to create .rpm package.

First thing I've done was to create setup.py, the metadata file used for python packages. I've used PyCharm's built-in functionality and basically just filled the simple table in PyCharm with information like project's name, author and so on. Then I've run python3 setup.py bdist_rpm, which created the rpm file.

This file usually can't be directly used, as the metadata are incomplete. You can probably create the rpm file directly from setup.py, but python's build system is really arcane and full of historical layers filled with pitfalls, so I tend to avoid it for building .rpm.

I've extracted the .spec file from the generated RPM and put it to the Infocenter directory. This file is the most important part for the RPM packages. I've filled missing metadata (RPM requirements and so on) and then added some parts that say how to convert .tar.gz source packages created by the setup.py to source RPM packages.

When I've tested the conversion with rpmbuild run on my computer, it seemed to work.

Build pipeline

Another step was to create build pipeline. You see, the state the project was at that moment was that a knowledgeable developer with properly set system could create the RPM package. I wanted to simplify and automate the project so that anyone could do that with one click.

The way how we do it in our company is to create Docker image which will build the package. Docker is this way how to run virtual computers inside your computer. You can specify, in form of code (Dockerfile), how the system in the virtual computer should work. Then you create the image, a binary copy of the system described in the Dockerfile. The image is then pushed to the image repository, from which everyone can use it without the need to build it themselves.

I've created Dockerfile in the specific git repository we use to share such files (just about everything is in some git repository). I've reused a big part of the Dockerfile from another project, and then fought with several weird errors and specific problems, like different path and env variables transported from the host system to the virtual computers. It took me several hours to make it work as I wanted since this project was supposed to run on CentOS 8, and the rest of our projects, except one, run on CentOS 7.

I've then uploaded the virtual image to the image repository and pushed the Dockerfile to the git repository for Docker files used as build systems. I've referenced the ticket or "issue" I've had from my project manager in all commits I've made. Anyone could see from the git history that the changes I've made were relevant to this issue. This also works the other way around; anyone who opens the issue, even many years in the future, could see all the changes I've made in all the repositories, and that they are relevant to this issue.

Then I've created Jenkinsfile in git repository for Jenkins files. Jenkins is a system with web interface, from which you can run all kinds of tasks defined in Jenkins files. It uses Groovy as a programming language, which I, as a python programmer, find pretty annoying, but I've been able to copy and paste most of the code from other repositories.

Jenkinsfile uses the Docker image I've created before. It is re-downloaded each time the job is run and the image changed. It then downloads the git repository for the Infocenter, builds the rpm package, extracts it from the target directory, stores the artifacts and offers you to push it to the Spacewalk (package repository). The computers in retails can then use this to install the app.

Then I added the code to the other repository with other Jenkins (Groovy) file, where I've told a specific Jenkins job to look for the first Jenkinsfile. When you run the job called poetically 000seed, it creates the Jenkins Job I've specified in the first Jenkinsfile. Talk about meta, huh. Jenkins jobs creating Jenkins jobs.

Once I had my pipeline created, I run it and did several iterations to fix stuff that didn't work.

It may sound both easy, and complicated. In fact, it took me more than twelve hours to make all of this work. If anyone asked for my opinion about the whole process, it would be something like "layers of shit built on top of shit". You can read why in my older article Programmer's critique of missing structure of operating systems.

Last part was testing. I was building the package in Jenkins, downloading the package to my computer, and uploading it via scp to the test machine. Then I had to log via ssh and install it with yum, and then run xfreerdp to connect to the graphical interface and test that it all work. That also took several hours of fine-tuning.

I've finished around midnight that day. Then I've found out, that some Romanian manager wrote an email the day before, where he was really quite annoying. He, for example, required, that we don't do the job, but teach Sindibad how to do it. Mind that there were two weeks before final release date, when everything should go to the production. And he seemed to be pissed off that it takes so long, even though I was assigned to the job only a few days before. And why were we suggesting that we can take the Infocenter under our team? Preposterous! Sindibad probably spoke with his boss about what I offered him and there were some misunderstandings.

Maybe because I was exhausted, but also still in the zone from the productive day, I wrote a really long email as a reply to all involved, where I basically politely criticized the whole Infocenter stuff.

I wrote that Sindibad never had any chance to do this job, and that I don't understand why it was his job in the first place. I wrote about all steps that were required and described how it took me several days even though I knew exactly what to do. And I also added some suggestions about what's from my perspective necessary. In the end, I've got so fired up, that I've told all involved, that they should stop treating this stuff like some amateurish piece of homegrown project someone does for fun.

Of the stuff I've recommended was for example to talk to the network guys before they deploy the Infocenter because it tries to access intranet over the MPLS network, and it doesn't work on the test machines I've had access to. That they should create business requirements for the project, and also some documentation in our documentation system (Confluence). That they should create tickets for change requests and do code reviews. Give the project to testers. And so on.

I've put passion and also a bit of hate in the email, so that the involved managers would realize how badly they were doing at the moment. Then I went to sleep.

"Watchdog"

The last part that needed to be finished was the watchdog script; a script that will be launched by systemd (Linux stuff), and it will check the pid file and time of the last update of the file and restart the main GUI process with proper DISPLAY env var if the program fails.

This is because we (ehm, Sindibad) want to be sure that the process is always somewhere on the background and can annoy the user of the computer with blinking messages in Romanian language. Sindibad yesterday casually mentioned that he needs this.

He also suggested solution with recursively called function. I've told him that I'll create the script for him, as it was apparent to me, that he was having trouble understanding everything involved and that you can't do it the way he suggested. Python doesn't have tail call optimization, and it will run out of stack call frames, and really, he doesn't know that systemd exists and what daemons are and other stuff.

So, I am working on the script, when suddenly, MS Teams pops in, and I see:

S: Hi man
S: Thank you very much for all the hard work
S: And for your kind words that you wrote in that mail
S: Meanwhile, I rewrote the app using QT5
S: I've used sched package which is included in python default packages to create that watchdog

Well, fuck. He rewrote it again. Didn't I tell him to stop doing that? And what's with the shed package? Quick googling leads to man page. And proves my suspicion; this is not a watchdog, this is just an event scheduler. WTF. The difference is that it doesn't make sure that your program runs, it only allows you to specify that some action should run after some time. But only in your process, and if that fails, it will fail as well.

I look into the sources he sent me (again, .zip without git, *sigh*):

import shed

...

s = sched.scheduler(time.time, time.sleep)                  

def app_scheduler():
    s.enter(30, 1, job)
    s.run()
    app_scheduler()

def init_check():
    run = False
    try:
        for pid in psutil.pids():
            p = psutil.Process(pid)
            if p.name() == "infocenter":
                run  = True
    except:
        err = errors.ErrorHandle()
        err.saveError()

    return run

running = init_check() 
        
if not running: 
    app_scheduler()

Well. This will run out of stack because of the infinite recursion. And it checks the pid file only at start. And it only checks the app, but doesn't restart it, but he somehow joined the watchdog with the GUI together and .. wtf.

I’ve decided to ignore his attempt, finish my watchdog, and test it on the target machine. I also added settings.py, and proper logging of the program state, watchdog and really of everything, to the log file. Log files are defined in the settings.py, so that both watchdog and the Infocenter use the same files. settings.py is just code for now, but it can be quickly extended to load configuration files if necessary.

I've tweaked the log file rotations and toyed with target executable names, did several test runs on the target machines. When I was happy, I pushed everything into the repo and also shared a working version of the package with our infra guys and several group chats.

Everyone seemed to be pleased. I considered my work done. I even sent an offline copy of the git repository to Sindibad, so he can make changes and whatever he wants with the code, even though he is still waiting for his access to Bitbucket.

Next week

When I thought that my job is done, I was probably too optimistic. Sindibad writes me every day. Sometimes even with other people. New group chats are created all the time.

They ask me for all kind of stuff. For example, how to give Sindibad access to the Jenkins job, and the online Git repository in Bitbucket. I've sent him URLs for both, so they know how to access them, but they don't have permissions.

I explain to them how ticket system and user support work, and I am quite weirded out, that they don't know this. I mean, this is basic. When I joined the company, my boss had to create several tickets for user support, so that my Active Directory account can access all kinds of stuff that I needed to access. How is it possible that Sindibad's bosses don't know this? What kind of amateur show are they running, that they don't even know how to ask for access to stuff in the company where they work in as full-time employees?

There are other more or less humorous stories from that week. I explain to Sindibad for the sixth time, that no, you really don't compile python the way he tries. Occasionally, when you need performance, you can use cython, or rpython, but the cx_freeze he is using is really not the way to go with production code, unless you have clear reasons. He finally seems to get it, when I explain to him, that there is already python installed in the Linux systems where it will be deployed.

I also explain to him how package repositories work, and that each Linux system is using several repositories. He is quite happy when he finally understands why his question "why can't I find a Skype package in our company spacewalk" was a misunderstanding of how it all works. We use our company repository only for our company stuff. The rest is from the distribution maintainers.

I kind of burst during our regular team "retrospective", which is part of our sprint ceremonies. Basically, we all share feelings from the sprint that ended and suggest improvements for processes and the next sprint. I rant for a while on the topic of Romania. Mostly, I am pissed about what is going on with the Romanians. Weren't they provided any information about our company? They are part of the company for more than two years. Why don't they know about our processes? Why are they so outside our R&D division, when they are officially part of it? They don't even bother with business requirements, or even consulting this stuff with anyone, before they decide to put it into production?

Sindibad finishes the week when he does some update to the project's spec file, still without access to the main git repository. He found a bug that manifests itself while updating the package and fixed it.

I was almost proud, except that he then decided that instead of increasing the version number of the Infocenter, which I've finished at 1.2.2, it would be nice to have a round number. So, he decreases the number to 1.0.0, which was already both in the git and package repository, and had its git tag.

It takes some explaining about how versions work, and that they should only ever increase. But at this point, I am quite .. Stoical.

Final thoughts about the whole DevOps thing

I am in the “industry” long enough, that I’ve seen the trend of “DevOps” being implemented in several companies where I’ve worked. And so far, only once, I’ve seen it working as expected.

Often I wonder, how much it really helped us (as an industry). On the one hand, it is great when programmers can make deployments themselves. On the other hand, I still remember “good old days”, when I could just come to our dedicated Linux sysadmin, and he would solve the whole deployment problem in minutes. I didn’t have to learn everything about Linux, Docker, Jenkins, Groovy, HELM, Kubernetes, Mustache, YAML, Gradle, Spinnaker, AWS, ELB, lv and pv groups, databases, enterprise hardware, RAIDs, and twenty other things. I didn’t have the control, sure, but “the system” as a whole was much more effective.

The longer I work in the field, the more I can recognize, that specialization matters. And it feels like that the whole DevOps thing was reduced mostly to just “make the programmer deploy it, no matter how long it takes, at least we save some money and don’t need the ops guys anymore”. Entirely ignoring, that programmer’s time is usually more valuable than admin’s time (at least when it comes to salary). And also that when people have a certain specialization, they can be ten or more times effective than people who see the stuff for the first time.

This whole story was quite a nice example. A Random PHP guy got a job to rewrite his application to python. Then he got a task to make it work on Linux, and to prepare production deployment, even though he knew nothing about Linux, python, or deployment into production. This is completely in spirit with how the DevOps thing usually turns out in almost all the companies where I’ve worked in.

Usually, it’s not “let’s have DevOps guys”, but “our programmers are now DevOps, let them deal with all this shit and get rid of our old ops guys”. The result is madness in the better cases, complete Cthulhu in the worst ones. Layers and layers of technical debt, technologies no one understands, piled on top of each other by some stressed out programmer who has no idea what he is doing.

Let me be clear; I am not exactly complaining. I fill the sweet spot of the “full-stack engineer”, where they pay me like a king to deal with this kind of stuff, usually when the shit layered on top of other shit crumbles, and falls on their heads (or in corpo speak, “technical debt becomes unmaintainable”). I am just baffled why.

Become a Patron