Comment by dijit 5 days ago
You think?
It took us nearly a decade and a half to unfuck the pulseaudio situation and finally arrive at a simple solution (pipewire).
SystemD has a lot more people refining it down but a clean (under the hood) implementation probably won't be witnessed in my lifetime.
yeah, the fix for pulseaudio was to throw it away entirely
for systemd, I don't think I have a single linux system that boots/reboots reliably 100% of the time these days
There were dozens of other init systems that, like systemd, wasn't a shell script.
What set systemd apart is the collection of tightly integrated utilities such as a dns resolver, sntp client, core dump handler, rpc-like api linking to complex libraries in the hot path and so on and so forth that has been a constant stream of security exploits for over a decade now.
This is a case where the critics were proven to be right. Complexity increases the cognitive burden.
What set systemd apart was RedHat, and now Pottering repeats the old trick with Microsoft behind his back.
I think he will succeed and we will be worse off, collectively.
that on itself is not a problem. The problem is that those work worse.
For example, the part of systemd that fills DNS will put them in random order (like actual random, not "code happened to dump it in map order)
The previous, while very much NOT perfect, system, put the DNSes in order of one in latest interface, which had useful side-feature that if your VPN had different set of DNSes, it got added in front
The systemd one just randomizes it ( https://github.com/systemd/systemd/issues/27543 ) which means that using standard openvpn wrapper script for it will need to be reran sometimes few times to "roll" the right address, I pretty much have to run
systemctl restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf
The OTHER problem is pervasive NIH in codebase.
Like, they decided to use binary log format. Okay, I can see advantages, it can be indexed or sharded for faster access to app's files...
oh wait it isn't, if you want to get last few lines of a service the worst case is "mmap every single journal file for hundreds of MBs of reads"
It can be optimized so some long but constant fields like bootid are not repeated...
oh wait it doesn't do that either, is massively verbose. I guess I can understand it, at least that would make it less crash-proof...
oh wait no, after crash it just spams logs that previous log file is corrupted and it won't be used.
So we have a log format that only systemd tools can read, takes few times as much space per line as text or even JSON version would, and it still craps out on unclean shutdown
They could've just integrated SQLite. Hell I literally made a lil prototype that took journalctl logs and wrote it to indexed SQLite file and it was not only faster but smaller (as there is no need to write bootid with each line, and log lines can be sharded or indexed so lookup is faster). But nah, Mr. Poettering always wanted to make a binary log format so he did.
The trick is the same: use a popular linux distribution and don't fight the kinks.
The people who had no issues with Pulseaudio; used a mainstream distribution. Those distributions did the heavy lifting of making sure stuff fit together in a cohesive way.
SystemD is very opinionated, so you'd assume it wouldn't have the same results, but it does.. if you use a popular distro then they've done a lot of the hard work that makes systemd function smooth.
I was today years old when I realised this is true for both bits of poetter-ware. Weird.
I only use debian
pulseaudio I had to fight every single day, with my "exotic" setup of one set of speakers and a headset
with pipewire, I've never had to even touch it
systemd: yesterday I had a network service on one machine not start up because the IP it was trying to bind to wasn't available yet
the dependencies for the .service file didn't/can't express the networking semantics correctly
this isn't some hacked up .service file I made, it's that from an extremely popular package from a very popular distro
(yeah I know, use a socket activated service......... more tight coupling to the garbage software)
the day before that I had a service fail to start because the wall clock was shifted by systemd-timesyncd during startup, and then the startup timeout fired because the clock advanced more than the timeout
then the week before that I had a load of stuff start before the time was synced, because chrony has some weird interactions with time-sync.target
it's literally a new random problem every other boot because of this non-deterministic startup, which was never a problem with traditional init or /etc/rc
for what? to save maybe a second of boot time
if the distro maintainers don't understand the systemd dependency model after a decade then it's unfit for purpose
> it's literally a new random problem every other boot because of this non-deterministic startup, which was never a problem with traditional init or /etc/rc
This gave me a good chuckle. Systemd literally was created to solve the awful race conditions and non-determinism in other init systems. And it has done a tremendous job at it. Hence the litany of options to ensure correct order and execution: https://www.freedesktop.org/software/systemd/man/latest/syst...
And outside of esoteric setups I haven't ever encountered the problems you mentioned with service files.
"for what? to save a second of boot time"
Doubtful the motivation was /etc/rc being too slow
daemontools, runit, s6 solve that problem
For me, randomly missing NFS mounts after boot were the last straw. I could not solve this problem. I am back on sysv init.
PipeWire is like 10 years newer than PulseAudio. It probably had a chance to learn some lessons!
IIRC before PulseAudio we had to mess around with ALSA directly (memory hazy, it was a while ago). It could be a bit of a pain.
"The trick is the same: use a popular linux distribution and don't fight the kinks."
I believe that you are genuinely being sincere here, thinking this is good advice.
But this is an absolutely terrible philosophy. This statement is ignorant as well as inconsiderate. (again, I do believbe you don't intend to be inconsiderate consciously, that is just the result.)
It's ignorant of history and inconsiderate of everyone else but yourself.
Go back a few years and this same logic says "The trick is, just use Windows and do whatever it wants and don't fight."
So why in the world are you even using Linux at all in the first place with that attitude? For dishonest reasons (when unpacked to show the double standard).
Since you are using Linux instead of Windows, then you actually are fine with fighting the tide. You want the particular bits of control you want, and as long as you are lucky enough to get whatever you happen to care about without fighting too much, then you have no sympathy for anyone else who cares aboiut anything else.
You don't see yourself as fighting any tides because you are benefitting from being able to use a mainstream distro without customizing it. But the only reason you get to enjoy any such thing at all in the first place is because a lot of other people before you fought the tide to bring some mainstream distros into existence, and actually use them for ordinary activities enough despite all the difficulties, to force at least some companies and government agencies to acknowledge them. So now you can say things like "just use a mainstream distro as it comes and don't try to do what you actually want".
> Go back a few years and this same logic says "The trick is, just use Windows and do whatever it wants and don't fight."
This is basically exactly what I saw people saying in Windows subreddits. There's one post that particularly sticks out in my memory[0] that basically had everybody telling the OP to just not make any of the changes that they wanted to make. The advice seemed to revolve around adapting to the OS rather than adapting the OS to you, and it made me sad at the time.
[0] https://www.reddit.com/r/Windows10/comments/hehrqe/what_are_...
> The people who had no issues with Pulseaudio; used a mainstream distribution. Those distributions did the heavy lifting of making sure stuff fit together in a cohesive way.
Incorrect. I used mainstream distro, still had issues, that just solved itself moving to pipewire. Issues like it literally crashing or emitting spur of max volume noise once every few months for no discernable reason.
Pulseaudio also completely denies existence of people trying to do music on Linux, there is no real way to make latency on it be good.
> SystemD is very opinionated, so you'd assume it wouldn't have the same results, but it does.. if you use a popular distro then they've done a lot of the hard work that makes systemd function smooth.
Over the years of using the "opinion" of SystemD seems to be "if it is not problem on Lennart's laptop, it's not a real problem and it can be closed or ignored completely".
For example systemd have no real method to tell it "turn off all apps after 5 minutes regardless of what silly package maintainers think". Now what happens if you have a server on UPS that have say 5 minutes of battery and one of the apps have some problem and doesn't want to close?
In SysV, it gets killed, and system gets remounted read only. You have app crash recovery but at least your filesystem is clean In systemd ? No option to do that. You can set default timeout but it can be override in each service so you'd have to audit every single package and tune it to achieve that. That was one bug that was closed.
Same problem also surfaced if you have say app with a bug that prevented it from closing from sigterm and you wanted to reboot a machine. Completely stuck
But wait, there is another method, systemd have an override, you can press (IIRC) ctrl+alt+delete 7 times within 2 seconds to force it to restart ( which already confuses some people that expect it to just restart machine clean(ish) regardless https://github.com/systemd/systemd/issues/11285 ).
...which is also impossible if your only method of access is software KVM where you need to navigate to menu to send ctrl+alt+del. So I made ticket with proposal to just make it configurable timeout for the CAD ( https://github.com/systemd/systemd/issues/29616 ), the ticket wasn't even read completely because Mr. Poettering said "this is not actionable, give a proposal", so I pasted the thing he decided to ignore in original ticket, and got ignored. Not even "pull requests welcome" (which I'd be fine with, I just wanted confirmation that the feature like that won't be rejected if I start writing it).
There is also issue of journald disk format being utter piece of garbage ("go thru entire journal just to get app's last few lines bad", hundreds of disk reads on simple systemctl status <appname> bad) that is consistently ignored thru many tickets from different people.
Or the issue that resolvconf replacement in systemd will just roll a dice on DNS ordering, but hey, Mr. Lennart doesn't use openvpn so it's not real issue ( https://github.com/systemd/systemd/issues/27543 )
I'm not writing it to shit on systemd and praise what was before, as a piece of software it's very useful for my job as sysadmin (we literally took tens of thousands lines of fixed init scripts out because all of the features could be achieved in unit files) and I mean "saved tons of time and few demons running" in some cases, but Mr. Poettering is showing same ignorant "I know better" attitude he got scolded at by kernel maintainers.
> Pulseaudio also completely denies existence of people trying to do music on Linux, there is no real way to make latency on it be good.
I don't care much about PA at this point tbh and don't know much about the inner workings; it always worked just fine for me. But from what I read from people more "in the know" at the time, I'd heard that a lot of the (very real) user-facing problems with PA were ultimately caused by driver and other low-level problems. Those were hacky, had poor assumptions, etc. PA ultimately exposed those failures, and largely got better over time because those problems got fixed upstream of PA.
My takeaway from what I read was basically that PA had to stumble and walk so that pipewire could run.
> For example systemd have no real method to tell it "turn off all apps after 5 minutes regardless of what silly package maintainers think". Now what happens if you have a server on UPS that have say 5 minutes of battery and one of the apps have some problem and doesn't want to close?
Add a TimeoutStopSec= to /etc/systemd/system/service.d/my-killing-dropin.conf more or less, I think? These are documented in the systemd.service and systemd.unit manpages respectively.
> Same problem also surfaced if you have say app with a bug that prevented it from closing from sigterm and you wanted to reboot a machine. Completely stuck
See the --force option on the halt, poweroff, and reboot subcommands of systemctl. The kill subcommand if you want to target that specific service.
> so I pasted the thing he decided to ignore in original ticket, and got ignored. Not even "pull requests welcome" (which I'd be fine with, I just wanted confirmation that the feature like that won't be rejected if I start writing it).
I'm certainly sympathetic to this pain point. I'd take Lennart at his word that he's not opposed. Generally speaking, from following the systemd project somewhat, it's a very busy project and it's hard for all issues to get serviced. But they're very open to PRs, generally speaking.
> Or the issue that resolvconf replacement in systemd will just roll a dice on DNS ordering, but hey, Mr. Lennart doesn't use openvpn so it's not real issue ( https://github.com/systemd/systemd/issues/27543 )
Quickly taking a peek here (and speaking as a relatively superficial user of resolved myself), isn't the proposed solution to define interface ordering?
> it will ask on all links in parallel if there's no better routing info available. In your case there is none (i.e. no ~. listed among your network interfaces), hence it will be asked on all interfaces at the same time.
It's baffling to me that anyone can imagine pipewire has been created from scratch without any lessons learned from pulseaudio and the previous issues the audio stack on linux had, and solved, over the years. Nothing is happening in a clean room bubble, every new project stands on the shoulders of giants...
anyone who thinks that pipewire - pipewire! - is "a simple solution" understands nothing about pipewire.
don't get me wrong, i use pipewire all day every day, and wrote one of the APIs (JACK) that it implements (pretty well, too!).
but pipewire is an order of magnitude more complex than pulseaudio.