Real‑World DevOps How‑Tos
A set of no‑nonsense DevOps guides that cut through theory and get straight to practical, real‑world solutions.
Object storage
Managing objects from s3 compatible storages
We will use the aws s3 command line utility for that. To install it have a look at Install aws cli.
In order to read or write objects/files to/from s3 compatible object stores, you first need to authenticate. You can authenticate with your own user's account or by using API keys.
Find a way to generate an API key from your s3 storage provider, then set the following environment variables:
export AWS_ACCESS_KEY_ID=$MY_S3_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$MY_S3_SECRET_KEY
export AWS_DEFAULT_REGION=$MY_S3_REGION
export AWS_ENDPOINT_URL=$MY_S3_ENDPOINT_URL
For info about the environment variables you can use to configure the 'aws' command line utility, have a look at aws cli config envvars.
Once the above environment variables properly set, you will be able to upload/download files to/from the s3 storage if the API key has the required permissions. Here are the commands:
# Upload myfile into mybucket
aws s3 cp /tmp/myfile s3://mybucket/
# Upload myfile into mybucket at myfilerenamed
aws s3 cp /tmp/myfile s3://mybucket/myfilerenamed
# Upload myfile into mybucket inside the files directory
aws s3 cp /tmp/myfile s3://mybucket/files/
# Download myfile from mybucket
aws s3 cp s3://mybucket/myfile .
The 'aws s3' command will by default use multipart upload when uploading a file into a s3 bucket. At the time of writing, the multipart upload behavior can't be configured through environment variables. You can use the 'aws configure set' command instead, to add the configuration settings inside the default aws configuration file ('~/aws/config'). Here are examples:
# Minimum file size that triggers multipart upload
aws configure set multipart_threshold 64MB
# Size of the parts (chunks) of the multipart upload
# bigger size => less parts
aws configure set multipart_chunksize 16MB
For a list + description of all the available aws cli configuration file settings, have a look at aws cli config file settings.
For info about the multipart upload limits, have a look at aws multipart upload limits.
Linux
Workstation config
Choosing a Terminal
- Tabby (color scheme: DotGov)
Configuring Tmux
To install tmux:
sudo apt update && sudo apt install tmux
To launch tmux:
tmux
The prefix key to use before tmux key bindings is Ctrl+b.
You can also run tmux commands from your terminal or inside a tmux session by activating the command line with Ctrl+b :.
An example command you could run is:
:set synchronize-pane
to make all panes in a window receive the same keystrokes.
For help, run 'man tmux' and look at the following sections:
- DEFAULT KEY BINDINGS - for default key bindings
- COMMANDS - for commands supported by tmux
- OPTIONS - for options you can use to configure the appearance and behavior of tmux
- WINDOWS AND PANES - for commands regarding windows and panes settings
- KEY BINDINGS - commands for setting, listing, removing key bindings
- HOOKS - for tmux hooks
- STYLES - for options you can use to customize tmux interface style
- STATUS LINE - for info and options about tmux status line (by default displayed at the bottom)
- EXAMPLES - for various tmux examples
To make specific tmux configurations permanent, you can add them inside the '~/.tmux.conf' file:
bind-key S set synchronize-panes
If you are inside tmux, you can make it reload its configuration file without restarting:
tmux source-file ~/.tmux.conf
Configuring Powerline-shell
To install powerline-shell:
pip install powerline-shell
# or
git clone https://github.com/b-ryan/powerline-shell
cd powerline-shell
python setup.py install
Then, add this to your '~/.bashrc' configuration file:
function _update_ps1() {
PS1=$(powerline-shell $?)
}
if [[ $TERM != linux && ! $PROMPT_COMMAND =~ _update_ps1 ]]; then
PROMPT_COMMAND="_update_ps1; $PROMPT_COMMAND"
fi
Then, reload your shell:
exec $SHELL
To get the required configuration for other types of shells, have a look at this.
You can then customize the looking of you shell with the following sample configuration file:
{
"segments": [
"virtual_env",
"username",
"hostname",
"cwd",
"ssh",
"git",
"jobs",
"root"
],
"mode": "patched",
"theme": "default",
"cwd": {
"max_depth": 3
},
"hostname": {
"colorize": "True"
}
}
Put that file in '~/.config/powerline-shell/config.json', after creating the required folders as follows:
mkdir -p ~/.config/powerline-shell
Block devices and filesystem
Finding files
Searching for specific files and directories from a root filesystem path:
# Searching for the 'history.log.4.gz' file from /
$ sudo find / -type f -name history.log.4.gz
/var/log/apt/history.log.4.gz
# Searching for files in /var/log whose names match the '.*/.*log\.[0-9]+\.gz' regex
$ find /var/log -type f -regextype posix-extended -regex '.*/.*log\.[0-9]+\.gz'
/var/log/apt/term.log.4.gz
/var/log/apt/history.log.5.gz
/var/log/apt/term.log.8.gz
/var/log/apt/history.log.11.gz
/var/log/apt/history.log.7.gz
(...)
# Searching for directories named 'log' from /
$ sudo find / -type d -name log
/usr/lib/python3/dist-packages/cloudinit/log
/var/log
/snap/core20/2717/run/log
/snap/core20/2717/usr/lib/python3/dist-packages/cloudinit/log
/snap/core20/2717/var/log
/snap/core20/2769/run/log
/snap/core20/2769/usr/lib/python3/dist-packages/cloudinit/log
/snap/core20/2769/var/log
/run/log
Running a specific command on the findings:
# Find history.log.4.gz from / and execute the 'ls -lah' command on the findings
$ sudo find / -type f -name history.log.4.gz -exec ls -lah {} \;
-rw-r--r-- 1 root root 967 Feb 20 21:01 /var/log/apt/history.log.4.gz
Deleting the findings:
# Find history.log.4.gz from / and delete it
$ sudo find / -type f -name history.log.4.gz -delete
Date and time
Synchronizing system clock using NTP
Ensure 'systemd-timesyncd' package is installed and 'systemd-timesyncd.service' is running:
$ dpkg -l | grep systemd-timesyncd
ii systemd-timesyncd 249.11-0ubuntu3.19 amd64 minimalistic service to synchronize local time with NTP servers
$ systemctl status systemd-timesyncd.service
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2026-06-25 23:20:42 CEST; 1 day 8h ago
Docs: man:systemd-timesyncd.service(8)
Main PID: 6354 (systemd-timesyn)
Status: "Initial synchronization to time server 91.189.91.157:123 (ntp.ubuntu.com)."
Tasks: 2 (limit: 2228)
Memory: 1.3M
CPU: 326ms
CGroup: /system.slice/systemd-timesyncd.service
└─6354 /lib/systemd/systemd-timesyncd
Then you can list the available timezones by running:
timedatectl list-timezones
To set the timezone run:
timedatectl set-timezone Africa/Abidjan
You can then verify by running:
timedatectl status
To activate time synchronization via NTP, you can optionally set custom addresses for the NTP servers by running:
timedatectl ntp-servers $network-interface $ntp-server-domain-name
Then, enable time synchronization by running:
timedatectl set-ntp true
To verify that NTP has been properly configured run:
timedatectl status
You can also show the current NTP servers and time synchonization status by running:
timedatectl timesync-status
timedatectl show-timesync
Systemd
Running a custom script at boot
Here is the systemd unit file:
[Unit]
Description=Run my custom script at boot
After=network.target
[Service]
Type=oneshot
ExecStart="/path/to/myscript.sh"
RemainAfterExit=true
[Install]
WantedBy=multi-user.target
For more about systemd have a look at this.
Installing programs from sources
Overview
You first have to make sure development tools, kernel headers... necessary for building Linux packages are installed (gcc, make...)
# Debian based distributions
sudo apt update && sudo apt install build-essential
# RedHat based distributions
yum groupinstall "Development Tools"
Once the requirements installed and software source code downloaded, the process of building and installing the software is the same regardless of the distribution. Inside the source code directory, you will most of the time see and use the 'configure' and 'Makefile' files.
Configure
The 'configure' script is used to configure the program. To see which customizations are available for your program run:
./configure --help
Makefile
The 'Makefile' contains 'make targets' used by the 'make' program. 'Make targets' are a way for developers to simply bring some of their programs build functionalities to users, for instance to compile the program with default options (genarally using the 'all' make target), generate the program documentation only in different formats, etc.
'Make targets' simply contains a list of commands to run in order to achieve a specific build variant. To use a specific 'make target' for the program build, simply run:
make <target-name>
Running 'make' is the same as running 'make all' and is generally sufficient to create the program binary with default options. Otherwise use 'configure' with the customization options you want, then run 'make' to create the program binary.
The program binary will generally be created inside the 'src' directory. You can use it from there or install it on your system using 'make install'. This will most of the time:
- deploy the program binary inside
/usr/local/bin - deploy the program libraries inside
/usr/local/lib - deploy the program documentation inside
/usr/local/share
Makefile sample content
Here is an extract of 'Make targets' taken from the curl program source code:
all:
./configure
make
ssl:
./configure --with-openssl
make
mingw32:
$(MAKE) -C lib -f Makefile.m32
$(MAKE) -C src -f Makefile.m32
mingw32-vclean mingw32-distclean:
$(MAKE) -C lib -f Makefile.m32 vclean
$(MAKE) -C src -f Makefile.m32 vclean
$(MAKE) -C docs/examples -f Makefile.m32 vclean
(...)
cygwin:
./configure
make
cygwin-ssl:
./configure --with-openssl
make
amiga:
cd ./lib && make -f makefile.amiga
cd ./src && make -f makefile.amiga
netware:
$(MAKE) -C lib -f Makefile.netware
$(MAKE) -C src -f Makefile.netware
netware-clean:
$(MAKE) -C lib -f Makefile.netware clean
$(MAKE) -C src -f Makefile.netware clean
$(MAKE) -C docs/examples -f Makefile.netware clean
(...)
netware-examples-%:
$(MAKE) -C docs/examples -f Makefile.netware CFG=$@
netware-%:
$(MAKE) -C lib -f Makefile.netware CFG=$@
$(MAKE) -C src -f Makefile.netware CFG=$@
unix: all
unix-ssl: ssl
linux: all
linux-ssl: ssl
ca-bundle: lib/mk-ca-bundle.pl
@echo "generate a fresh ca-bundle.crt"
@perl $< -b -l -u lib/ca-bundle.crt
ca-firefox: lib/firefox-db2pem.sh
@echo "generate a fresh ca-bundle.crt"
./lib/firefox-db2pem.sh lib/ca-bundle.crt
SSH
Proxy Jump
Configuring SSH client to proxy jump
SSH proxy jump allows you to transparently connect to a final SSH server through an intermediate/proxy SSH server.
Here is a sample SSH client configuration used to connect to server2 through server1. You can add SSH client configurations in:
~/.ssh/config- for user's scoped configuration/etc/ssh/ssh_config- for system-wide configuration
First we add the configuration required to connect to the intermediate/proxy server (server1):
Host server1
Hostname 192.168.9.4 # hostname or IP address used to SSH into server1
User didi # username used to SSH into server1
IdentityFile /home/didi/.ssh/id_rsa # private key used to SSH into server1
Then, we add the configuration required to connect to the final server (server2):
(...)
Host server2
Hostname 10.2.3.4
ProxyCommand ssh -q -W %h:%p server1
User dada # username used for connecting to server2 from server1
The 'ProxyCommand' in the above configuration tells the SSH client to use server1 configuration for connecting to server1 before connecting to server2 from server1. Here is a detailed overview of what happens during a SSH connexion to server2 after setting up this proxy jump configuration:
Connection to server1
Once the proxy jump configuration in place, you can connect to server2 by running:
ssh server2
Before connecting to server2, ensure:
- user didi is present in server1
- server1 is properly configured to allow SSH connections with the private key of the user didi ('/home/didi/.ssh/id_rsa') present in the machine from where the ssh command is run (local machine)
To allow the local machine's user didi to connect to server1 with its private SSH key, add the associated SSH public key in server1, inside the home directory of user didi, in the authorized keys file: ~/.ssh/authorized_keys.
Connection to server2
Sshing into server2 will before SSH into server1 using the corresponding SSH client configuration (Host server1), then a SSH connection will be initiated from server1 after transparently transferring the required private key
('/home/didi/.ssh/id_rsa') to server1.
The resulting command in server1 for connecting to server2 will be:
ssh -i <transferred-private-key> dada@10.2.3.4
You must therefore ensure that:
- the user dada is present in the server2 (10.2.3.4)
- server2 is properly configured to allow SSH connections with the transferred private key (private key of user didi in the local machine)
To allow the user dada to connect to server2 with the transferred private key, add the associated SSH public key in server2, inside the home directory of user dada, in the authorized keys file: ~/.ssh/authorized_keys.
Proxy jump without SSH client configuration
ssh -i local-private-key -J [intermediate-user@]intermediate-ssh-server [final-user@]final-ssh-server
Before running the command, ensure:
-
local machine can connect to 'intermediate-ssh-server' using 'local-private-key' and user 'intermediate-user'
- which means: corresponding 'local-public-key' for 'local-private-key' is present in 'intermediate-ssh-server' in '/home/intermediate-user/.ssh/authorized_keys' file
-
local machine can connect to 'final-ssh-server' using 'local-private-key' and user 'final-user'
- which means: corresponding 'local-public-key' for 'local-private-key' is present in 'final-ssh-server' in '/home/final-user/.ssh/authorized_keys' file
-
There is no need to transfer 'local-private-key' to 'intermediate-ssh-server', this is securely and transparently done when using the '-J' option to proxy jump
Blocking users shell access to the SSH proxy
Set their public keys entries inside the authorized keys file (~/.ssh/authorized_keys) as follows:
<username>:command="/sbin/nologin",no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmSIZR/MBampHYZ7y9mVfxa.....
Transferring files
Synchronizing files through SSH with rsync
Here is a sample script you can use to periodically synchronize files from remote servers through SSH using rsync:
#!/bin/bash
logger -t My data sync started
rsync -avr --exclude="file_to_exclude.cfg" -e "ssh -i /rsync_ssh_private_key -p $remote_server_ssh_port" mydata@${remote_server_ip}:/data/mydata_remote_src_dir/ /data/myapp_local_dst_dir/ > rsync_log_dir/mydata-$(date +%F-%Hh%M).txt
logger -t My data sync finished
Before running the script, ensure the public SSH key associated to the '/rsync_ssh_private_key' private SSH key is authorized to connect to the remote server and the network flow from local to remote server SSH port is open.
Sample cron configuration
Here is the content of a sample cron configuration file. That cron configuration will make the rsync script run every 6 hours at the minute 30:
30 */6 * * * root bash /root/mydata_sync.sh
Relational Databases
MySQL
Client and server installation
To install MySQL client only:
apt update && apt install mysql-client
To install MySQL server:
apt update && apt install mysql-server
Securing the installation
$ sudo mysql_secure_installation
Securing the MySQL server deployment.
Connecting to MySQL using a blank password.
VALIDATE PASSWORD COMPONENT can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD component?
Press y|Y for Yes, any other key for No: y
There are three levels of password validation policy:
LOW Length >= 8
MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary file
Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 1
Skipping password set for root as authentication with auth_socket is used by default.
If you would like to use password authentication instead, this can be done with the "ALTER_USER" command.
See https://dev.mysql.com/doc/refman/8.0/en/alter-user.html#alter-user-password-management for more information.
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.
Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.
Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
- Dropping test database...
Success.
- Removing privileges on test database...
Success.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.
All done!
Root user authentication
$ mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 8.0.46-0ubuntu0.22.04.2 (Ubuntu)
Copyright (c) 2000, 2026, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SELECT user, host, plugin FROM mysql.user;
+------------------+-----------+-----------------------+
| user | host | plugin |
+------------------+-----------+-----------------------+
| debian-sys-maint | localhost | caching_sha2_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session | localhost | caching_sha2_password |
| mysql.sys | localhost | caching_sha2_password |
| root | localhost | auth_socket |
+------------------+-----------+-----------------------+
5 rows in set (0.00 sec)
On Ubuntu, as we can see above, by default, the MySQL 'root' user uses the 'auth_socket' plugin for authentication and is configured to login only from the server machine (localhost).
The 'auth_socket' successfully authenticates the 'root' user without asking a password, as long as the system user running the MySQL client is 'root' and there is a MySQL user with same name and host ('root'@'localhost').
The principle behind this is: if you can become the Linux root user, you already control machine, so requiring the MySQL 'root' user password is redundant.
If you still want to swith from 'auth_socket' to password authentication:
mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'MySuperPa@ssW0rd!';
Query OK, 0 rows affected (0.06 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.04 sec)
Now when we connect without password we got an error:
$ mysql -u root
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
Connecting with the configured password works:
mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 14
Server version: 8.0.46-0ubuntu0.22.04.2 (Ubuntu)
Copyright (c) 2000, 2026, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
To rollback to 'auth_socket':
mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH auth_socket;
Query OK, 0 rows affected (0.02 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.06 sec)
Connecting to databases
To connect to a MySQL database instance at 'host' with 'username' and 'password':
mysql -u username -h host -p password
To avoid passing the password in the command for security reasons:
mysql -u username -h host -p
To use a file containing authentication info:
mysql -u username -h host --defaults-extra-file=credentials_file
Here is a sample content for credentials_file:
[client]
user = "username"
password = "password"
host = "localhost"
Viewing server configuration
To see currently running MySQL server configuration parameters:
mysqld --verbose --help | less
To set a specific parameter for the server, use 'my_param' (with underscore) inside the configuration file or '--my-param' on the command line.
Managing users
Here is MySQL Account management statements documentation for more.
To create a user:
mysql> CREATE USER 'username'@'host' IDENTIFIED BY 'password';
mysql> FLUSH PRIVILEGES;
For more, have a look at Creating users.
To delete a user:
mysql> DROP USER 'username'@'host';
mysql> FLUSH PRIVILEGES;
For more, have a look at Deleting users.
In the previous commands, 'host' represent the IP address / hostname of the source machine from where the connection is initiated. Could be set to something like localhost (for connexion that are initiated directly from the server), 10.2.3.4. To match any source IP address, use %.
To grant a specific privilege to a user:
mysql> GRANT SELECT ON database.* TO 'username'@'%';
mysql> FLUSH PRIVILEGES;
The previous command grants the 'SELECT' privilege on all tables of 'database' to 'username'. The wildcard (*) can be replaced with a specific table name to grant the privilege for that table only. Multiple privileges can also be specified using comma as separator. Here is the list of available MySQL privileges.
To revoke specific privileges:
mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'username'@'host';
mysql> FLUSH PRIVILEGES;
For more, have a look at Privilege revocation
To make modifications on users and privileges immediately take effect:
mysql> FLUSH PRIVILEGES;
This ensures MySQL refreshes its in-memory cache for 'mysql.*' tables (users, grants, roles) from disk.
Managing backups
To create a backup of 'mydb':
mysqldump -u username mydb -p > mydb_backup.sql
To restore the backup:
mysql -u username mydb -p < mydb_backup.sql
Python
Managing Python with Pyenv
Pyenv setup
By using pyenv, we can install the latest Python versions on Linux distributions that do not provide them through their official repositories.
Before running the following setup instructions, install the pre-requisite packages.
To install pyenv, run:
$ curl https://pyenv.run | bash
To configure your shell to automatically load pyenv (bash example), append
the following to '~/.bash_profile' if it exists, otherwise '~/.profile' (for login shells) or '~/.bashrc' (for interactive shells):
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
Restart your shell for the changes to take effect:
exec $SHELL
To update pyenv, its plugins including the list of available python versions:
pyenv update
Managing python versions
To list the available python versions that can be installed:
pyenv install --list
To install/uninstall a specific python version:
pyenv install <python_version>
pyenv uninstall <python_version>
Example:
pyenv install 3.14.5
To set a specific pyenv installed python version as the default for the whole system:
pyenv global <python_version>
To show the current python configuration for the whole system:
$ pyenv global
system
'system' means that we are on the default installed python version for the system. To set the python version to 3.14.5 for instance for the whole system:
pyenv global 3.14.5
Verify:
$ pyenv global
3.14.5
Rollback to the system default python version:
pyenv global system
Managing virtualenvs
We can also manage python virtualenvs with pyenv. To create a virtualenv, run:
pyenv virtualenv myvenv
To list existing virtualenvs:
$ pyenv virtualenvs
3.14.5/envs/test (created from /home/gmkziz/.pyenv/versions/3.14.5)
test --> /home/gmkziz/.pyenv/versions/3.14.5/envs/test
To activate a virtualenv:
source /home/gmkziz/.pyenv/versions/3.14.5/envs/test/bin/activate
or
virtualenv_name=test
eval $(pyenv sh-activate $virtualenv_name)
To deactivate a virtualenv:
deactivate
or
virtualenv_name=test
eval $(pyenv sh-deactivate $virtualenv_name)
Finally, to remove a virtualenv:
$ virtualenv_name=test
$ pyenv virtualenv-delete $virtualenv_name
pyenv-virtualenv: remove /home/gmkziz/.pyenv/versions/3.14.5/envs/test? (y/N) y
Installing DevOps tools
Ansible
If you need to install one of the latests versions of Ansible, ensure you are running the latests python versions.
Even if you are running an operating system that do not natively provide the latests python versions, you can install them with pyenv. Have a look at Managing python versions for that.
Then, follow the installation instructions at Installing Ansible.
Helm
Follow installation instructions at Installing Helm.
Helmfile
Follow installation instructions at Installing Helmfile.
Kubectl and plugins
Follow installation and configuration instructions at: