All in on Systemd. Using Systemd as a cron replacement, network manager, date server, DNS server, and more in Arch Linux.
Ok so Arch comes with Systemd. Arch is unopinionated. That means that they don't bundle software to do a bunch of things, which I knew. But I recently realized that they don't configure Systemd, which is installed as part of the Arch core, to do half of the things that it could do, so that you can install other software if you want. I don't mind having to install stuff under Arch, but if I can do stuff with Systemd, I don't see why I wouldn't use it. So let's do it. Here are some of the tips and tricks that I've learned for using Systemd.
Systemd as network manager
You're probably using Network-Manager or netctl or ConnMan or similar. Unneeded. We're going to use the pre-installed systemd-networkd.
Sidenote 1:
Everything we're going to be using looks like systemd-THINGd. I think this is a stupid naming system because it's super verbose and has two "d"s in it. I would have called this like sys-networkd or even just networkd.
Sidenote 2:
Systemd is configured entirely with config files owned by root and located at obscure locations in the filesystem. There's nothing we can do about that, it's just a product of systemd being able to do so much. But I would recommend remembering what you're doing because there's no way to 'revert to default'.
Now we just need to create config file(s) representing our network setup. This will tell systemd what interfaces to use, so that it will route our traffic through them when they're available. I'm just going to set up one file for ethernet, since my current Arch machine doesn't have WiFi.
Pro Tip:
At this point what I do is copy the config file used by the Arch install image for Ethernet. It has a few extra rules that I don't fully understand, but which clearly have thought put into them. If I'm setting up a new Arch system, I can just copy this file e.g.cp /etc/systemd/network/20-ethernet.network /mnt/etc/systemd/network/. This install image also includes config files for WiFi (wlan) and Cellular (wwlan).
The Arch ISO's 20-ethernet.network file on Gitlab.
More detail on writing this file is covered in the Arch Linux Wiki.
That's not too bad. So that's a couple-line Systemd config file that needs to be placed in /etc/systemd/network/20-ethernet.nework. And bada-bing, bada-boom, we have network working.
To apply these changes, we need to start and enable systemd-networkd:
sudo systemctl enable --now systemd-networkd
Sidenote 3:
Enabling and starting systemd services are 2 distinct ideas. Enabling just creates a symlink telling the OS that it should start the service the next time it reboots. Starting actually starts it, but doesn't remember that. You can do both withenable --now.
Now you're probably thinking, "Matthias, I've been scammed, this doesn't give me a solution to the very hard problem of configuring Wifi on Linux." And you're right, it doesn't. If you need wireless, you still need to install and configure wpa_supplicant or iwd like normal.
Wireless Network Configuration - ArchWiki
Systemd as a DNS resolver
Your new Arch install needs a DNS resolver. There are lots of fancy programs available for that if you want DNS over HTTPS or more advanced caching algorithms or a local DNS server. But if you just want to lookup domain names, you can use systemd-resolved.
Using it is as simple as enabling (and starting) the service.
sudo systemctl enable --now systemd-resolved
Of course, you can configure the actual servers it uses and other settings, but you shouldn't have to. It should just work.
Systemd as a Time Server
Your Arch install probably needs an NTP client. This program is responsible for querying a remote server and updating your system clock with the correct time. The Arch Wiki lists like 8 of them you can install, but we already have one installed, systemd-timesyncd.
To start and enable systemd-timesyncd you can run sudo timedatectl set-ntp true (assuming there isn't another NTP client installed). Timedatectl is a utility provided by Systemd to manage your system clock. You can also use it query your current clock information.
$sudo timedatectl status
Local time: Sat 2022-03-26 21:24:05 EDT
Universal time: Sun 2022-03-27 01:24:05 UTC
RTC time: Sun 2022-03-27 01:24:05
Time zone: US/Eastern (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
Sidenote 4:
I'm pretty sure this isn't enabled by default, which bit me. I just happened to notice that my system clock had drifted 40 seconds because I had a regularly scheduled job that was firing 40 seconds later than it should have.
Yeah that's it; time should be simple.
I will note that if your hardware clock has drifted you can manually update it with hwclock --systohc. (As I understand it this is also done automatically on shutdown, by systemd-timesyncd.)
Writing your own Systemd Service files
This is one of the most central concepts on a systemd managed system, and any Sysadmin should be able to write a custom service file. A systemd service represents a program/process that Systemd is responsible for running. Service files are a type of unit file.
The service file defines the parameters for the code we actually want to run. Mainly, that's the path to the file, but you can also include other settings.
Sidenote 5:
I don't want to devolve into listing all service config options here. But I do want to say there are some cool ones. Stuff like user to run under, working directory, behavior if your script terminates with an error code, log location, and some neat sandboxing stuff (e.g. allow access only to certain files). It's more complicated than cron (see the next section), but I think that's for good reason.
Let's write a super quick service file for some sort of networked server, which maybe uses nodejs. We need a description; we need it to start after the network connection exists; and we need it to start on boot (that’s the WantedBy=multi-user.target. Don’t ask me why it’s called that). This file needs to go into /etc/systemd/system/example.service, (where example is the name of your service, of course).
[Unit]
Description=Some sort of webserver
After=network.target
[Service]
Type=exec
ExecStart=/usr/bin/node /home/matthias/project/example.js
[Install]
WantedBy=multi-user.target
Don’t forget to sudo systemctl daemon-reload
Sidenote 5.5:
You shouldn't rely on After=network.target since it doesn't guarantee the network is fully operational. But it does make sure your service is cleanly shutdown before the network is shutdown. Don't use Wants=network.target.
You could use After=network-online.target, to wait to start your service until the computer has an IP. However you could end up inadvertently forcing your computer boot's process to wait for network connectivity.
https://systemd.io/NETWORK_ONLINE/
Sidenote 6:
The Type option is one of the most confusing options. We are going to devolve into listing all the options here because before this I'd just been randomly guessing what type to use. Here's how I understand it.
Type=Simple means that Systemd 1) marks the service as active, 2) starts any follow-up units, 3) actually executes ExecStart. The service is marked as inactive again after ExecStart exits. This means that it can be used for long-running daemons or single-execution processes.
This isn't recommended for 2 reasons. Most of the time you want follow-up units to start after the service itself has started. And if executing ExecStart fails (step 3),systemctl startwon't tell you, since it will have successfully activated the service (step 1).
Type=exec is likesimple, but the steps are in a different order. Systemd will 1) start ExecStart, 2) mark the service as active, 3) start follow-up services. After the process exits, the service is marked as inactive. This works for most situations. (The systemd man page recommends usingsimpleoverexecbecause in theorysimplespeeds up system startup, since it doesn't wait to actually start the process. That's not a big concern to me.)
Type=forking is for a service that forks, starting a different long-running process, before exiting itself. This means that systemd: 1) Executes ExecStart. When ExecStart exits, it 2) marks the service as active, and 3) starts follow up services. It marks the service as dead when the process specified with PIDFile= exits (or never if that's not specified).
Type=oneshot (The man page says this is like simple but I don't really see the similarity.) This is for services when run for a negligibly short time. Systemd will 1) start ExecStart. When ExecStart exits, 2) start follow-up units. The service is never marked as "active". This meanssystemctl startwon't return until the process exits.
There are a few more Type= values, like "dbus" "idle" and "notify" that work similarly to some of the above, but with different triggers before starting follow-up services.
With just this one file, you can start using sudo systemctl start/stop/restart/status example to control your service.
Sidenote 7:
Many distributions (such as Ubuntu) provide aservicecommand which is a unified wrapper around both System V init scripts and Systemd. Its argument order is the opposite ofsystemctl. Since this guide is a proponent of Systemd we’re going to usesystemctl.
Like I said, just about every conceivable option exists in service files: user to run as, working directory, failure-modes, sandboxing, etc. This is pretty much the bare minimum. You can configure more depending on your use-case’s needs.
Okay, that was a lot of words to describe something pretty simple. Don’t feel bad if you skimmed it and just copy-pasted the service file above.
Systemd as a cron replacement
You're still using cron? Just kidding. Even though Systemd has been in a position to replace cron for 10 years, many sysadmins still turn to cron because it is legitimately simpler to use. But this post is all about going all in on Systemd. So let's go.
Here's the problem, we need two things, a .service file and a .timer file. The service file defines the parameters for the code we actually want to run, as described above. The timer half, on the other hand, determines when to run the service.
Here’s an example service file. Type=oneshot is a good fit for services triggered by timers but which are not normally running. Let's call it example.service
[Service]
Type=oneshot
ExecStart=/home/matthias/bin/example
The second thing we need is a timer file. Here’s a quick example of a timer file that runs a process every 15 minutes. Let's call it example.timer
[Unit]
Description=Fires example.service every 15 minutes.
[Install]
WantedBy=multi-user.target
[Timer]
# > Values may be suffixed with "/" and a repetition value, which indicates that the value itself and the value plus all multiples of the repetition value are matched.
# - SYSTEMD.TIME(7)
OnCalendar=*:00/15
The OnCalendar syntax is interesting. It's documented in the systemd.time(7) man page, but it's very helpful to use the following command to test your timestamp syntax. This will parse your rule and tell you the next time it would execute.
systemd-analyze calendar *:00/15Now what’s important is that the timer and service file have the same name, just different file extensions.
Now we need to install both of these file. I like to keep the authorative versions of my systemd config and unit files in my user directory and syslink them into /etc/systemd/system/.
sudo ln -s example.{service,timer} /etc/systemd/system/