$1900 Bug Bounty to Fix the Lenovo Legion Pro 7 16IAX10H's Speakers on Linux
(github.com)184 points by rany_ 7 days ago
184 points by rany_ 7 days ago
Given that “fixing this issue required weeks of intensive work from multiple people”, the price would have to be prohibitively high.
More generally, software is really, really expensive to produce and maintain. The economics only work at scale, in particular for B2C. (Maybe AI will change that, if it becomes more reliable.)
For many large companies or even teams, there exists a class of bugs / issues / features where dropping 5-10k on a bounty is extremely cost efficient compared to working around the issue or internal development. That might not fund development outright, but at worst it would point out the features people want and serve to inform what to work on next. I think there are a couple reasons why that is not prevalent. Most important one is that highly compensated enterprise teams that would benefit the most from placing bounties tend to avoid software that is lacking features or has bugs. Secondary is not implemented here ego and general disconnect between people in the trenches that know what needs to be done and people controlling ability to place bounties.
Imagine FAANG assigning $500 per engineer per year to allocate to feature / bug bounties.
Most larger companies would probably find it way easier and more sensible to contract with some outside consultancy to work on these issues than just posting a random bounty, even if the latter might potentially be cheaper. See Google Summer of Code projects for a very practical example of how "just pay randos to work on issue X for cheap" can quite often end up in failure.
Eh, I think you're underestimating some people perseverance.
You generally only need multiple people for timely action, and it usually even slows you down (from the perspective of total hours spent)
Like 2k bug bounty? I guarantee you some people would be willing to spend a lot of time for that. But yeah, people which are gainfully employed and have a decent salary - likely not.
lt could become some sort of leetcode final boss and/or something that you can put on your resume.
Did you realize that you didn't include 'open source' in your statement? This is exactly what the desktop OS makers -Microsoft and Apple- do every single day. Their prices are mostly B2B and therefore hidden, but there is a steady income for each person involved in making the fix.
it's the management structure that's broken. Plenty of decent engineers around microsoft who could fix it, plenty of customer and enterprises willing to pay, but they are not allowed to work on it because of prioritization bullshit, allegedly they could get more money elsewhere
That's literally the issue, management by KPI frameworks
I have used it every day for past 3-4 years. What bugs? I don’t love it but I don’t hate it either. I don’t understand the Teams hate
If you had made the same complaint about Win11 and you wouldn't be so far off. Microsoft is great at driver support which is the subject at hand.
It is, but it's amazing how cheap kernel expertise is relative to comparable experience in other specialties like frontend.
there are a lot more kernel programmers than kernel work
"Lots" is a relative term, but the overwhelming majority of kernel developers are employed and usually do kernel work as part of their job (usually at least ~80% but it could be argued as high as 97% depending on how you interpret the breakdown done by LWN of each release[1]).
And I would guess that most of the kernel devs who are "working for free" are doing the stuff they personally enjoy and find satisfaction in working on, because it's a hobby -- so many of them are probably not interested in fixing random bugs for cash either.
For small stuff, the cost is just going to be too much for people to want to pay it. This bug had a $1900 bounty attached. Let's put the cost of one software engineer (salary plus overheads) at $200,000 a year, which I think is an underestimate. That's $3850 a week, so unless your bug can definitely be fixed (including getting any necessary hardware, investigation, fixing, code review overhead, etc) in two or three days it doesn't pay. And if it could obviously be done in two days then it's likely somebody would have already done that.
The above back of envelope maths ignores the overheads of interacting with the people who posted the bounties to get them to agree to pay up, and of the cost overruns on the class of bugs that look like two day fixes but take two weeks.
I assumed the commonly cited 2x markup, so that would be a $100k salary, which is less than various websites say is the average US software dev salary. You could probably find cheaper elsewhere in the world, but even if you cut the salary in half that's still "bug must be doable in a week", which isn't going to cover many of the bugs people will care about.
Paying for software developers is really weird. State governments for example struggle to pay for a FTE that makes $140k. But they can pay me over $200/hour for consulting services for multiple years. The technical FTE employees that they have generally aren't qualified to evaluate their consulting needs so you get multi-million dollar contracts with very little actual oversight. I was really impressed with the folks I was working with at this particular state government and looked into what it would look like if I joined them full time as a FTE technology leader. I would have to take almost a 50% pay cut. The top senior IT position that oversees all of the state resources makes 70% of what I do. It's crazy. Unless you're working in medicine or sports, government pay sucks.
I've seen similar but less extreme examples play out in the private sector. 16 year senior architect making less than freshly hired software dev that was just an intern within the same company. Software developer pay is largely based on what you're demanding. In a lot of companies, there is a wide range of pay for folks doing literally the same job. They will hire a dev at $180k because that dev wouldn't go lower and turn around and push back to get another dev at $120k for the same level of unproven experience.
200k is a fairly high salaried software eng in expensive markets, a bounty program like this would be open worldwide and many people would be willing to work for a fraction of that, quality control is another concern but take a look at prices on sites like upwork and bids for this type of work and realize 200k is nowhere near the lower baseline.
$200k is on the extreme high-end of software engineers. For example in eastern europe $30k is normal. And that's not even the floor. You can go to india or africa to get even cheaper. The problem with this bug bounty though is that it requires pretty rare expertise. It's not a "throw any developer at it" type of thing.
It looks like these laptops are usually sold with Windows; are you saying that every manufacturer should be obligated to develop drivers for every software which is theoretically compatible with it? Or are you just saying that we need even more caveats in the interminable EULAs we all just click through?
And the person who did the implementation, Lyapsus, did it without access to the hardware?? https://github.com/nadimkobeissi/16iax10h-linux-sound-saga/i...
That thread is a fun (though frustrating for them!) conversation to read through.
After about a hundred back-and-forths getting the guy with the actual hardware to try different commands, I was thinking to myself man, maybe he should just give him remote access to work on the target PC, this is torture for both of them. And then I see him comment:
> Honestly I'm thinking of this and maybe something insane like organizing ssh access or something to quit torturing Nadim with building and rebooting all the time
And Nadim replies:
> Haha, sorry, but there's no way I'm giving you SSH access!
> I’m fine with continuing with tests!
Which is fair enough! But was funny to see right when I was thinking the same thing. Great perseverance from both of them.
Was slightly disappointing they they moved off GitHub to Discord eventually so after all that, we miss the moment of them actually getting it working!
There is a common problem with Realtek ALC3306 on Linux (Kernel Bug 213159). This affects many Lenovo laptop models. For example, my fairly old Legion S7 15IMH5 laptop also does not work.
I'm not willing to pay $1000 for a fix (it's easier for me to buy a new laptop that will work with Linux), but $100 is probably okay. :)
IF $1900 is the bounty, it means it doesn't hurt enough.
I remember going for the highest paying bounty in the Ethereum VM several years ago (I think it was ~$400 DAI/SAI). I did it because I wanted to force myself to learn the internals and to see for myself if the bounty system works. I think I spent a few weeks debugging and ended up splitting the bounty.
As long as the user-facing issues are disconnected from the technical issues, it's going to be hard to get the true value.
I guess this here is what the title is describing: https://github.com/nadimkobeissi/16iax10h-linux-sound-saga/b...
In the README:
> Approximately 95% of the engineering work was done by Lyapsus. Lyapsus improved an incomplete kernel driver, wrote new kernel codecs and side-codecs, and contributed much more. I want to emphasize his incredible kindness and dedication to solving this issue. He is the primary force behind this fix, and without him, it would never have been possible.
> I (Nadim Kobeissi) conducted the initial investigation that identified the missing components needed for audio to work on the 16IAX10H on Linux. Building on what I learned from Lyapsus's work, I helped debug and clean up his kernel code, tested it, and made minor improvements. I also contributed the solution to the volume control issue documented in Step 8, and wrote this guide.
If only people would do these bounties for ITE sensor hubs.
> Only frontend React.js
Good suggestion, but I discovered that React was not able to fix my Linux kernel, either, for some reason.
if there were data sheets available I expect they actually could do a bunch of the work here.
Interestingly the Awinc aw88399 smart amplifier chip at the core of this issue allegedly got supporting in 6.7, in 2023. https://www.phoronix.com/news/Linux-6.7-Sound
I have a couple old-ish Samsung Galaxy Book x86 tablets that have a similar issue, that I have never quite goaded myself into trying to reverse engineer. I'd love some better material on trying to reverse engineer windows drivers: presumably maybe running windows in qemu with some kind of intercepting pass through?
Here is another[1] fine read that was post recently. Author fixed media button issue for old 2005 Fujitsu keyboard and pushed it to kernel.
Actually, the mercenary garbage that Lenovo started doing was absolutely hateful. I'd been tearing my hair out forever trying to figure out why nothing I could do would make my bluetooth work right on an old T430, even when I upgraded the chip. Assumed it was a Linux bug.
Turns out that Lenovo put awful bluetooth in the laptop, and made it ignore any other bluetooth chip you installed (you can get around this in Linux by force ignoring what the system reports.) I have no idea why you would do that except out of spite; I don't remember them selling bluetooth upgrades or anything. They were just keeping their options open? This is aside from having to hack the bios in order to upgrade wireless or use generic batteries.
I would be awesome if the people that sold me products weren't awful people. They don't have to feel bad about it, but they should.
I don't care that you're angrier at Apple than at Lenovo. I'm angrier at the electric company, but I don't bring it up to defend my local alderman. I also don't care that FOSS hasn't solved all your problems for free like they apparently promised you they would.
> Turns out that Lenovo put awful bluetooth in the laptop, and made it ignore any other bluetooth chip you installed (you can get around this in Linux by force ignoring what the system reports.)
I used to Hackintosh Lenovos -- I thought this was at the bios level, so even if you did DSDT patching (linux or mac wise) it wouldn't work?
I use a T530, which is the larger brother of the T430. My overall impression is that it's a fine machine. (A lot of that opinion is motivated by the fact that when I bought it, it represented the very last of the plain black business-ey 15" PC laptops that wasn't cursed by the inclusion of a numeric keypad. Nowadays, Framework might fill that niche.)
Anyway: IIRC, a lot of the reason for locking down wireless hardware support is the FCC. AFAIK, the machine is tested and certified as a whole: The entire combination of its chassis and antennas and shielding and transmitting radios forms the item-under-test.
It's not just the sum of its parts; the whole thing gets tested. Deviation by end-users can result in having a non-compliant device, and the BIOS seeks to restrict that deviation.
And no, I don't like that aspect either.
But it only cares about the radios[1]. The rest of the guts are pretty freely exchanged: Upgrade to a different optical drive? SSD? Hard drive? RAM? CPU? Hack in a different keyboard? It's not picky at all about those things; have at it. (They could have locked down the entire system so that only parts matching a secret sauce would work, but they didn't.)
Anyway, Bluetooth module is garbage even under Windows. It's a bad design.
I bought my T530 from some random seller on eBay. It was obviously rebuilt by an outfit where they have a pile of variously-destroyed computers and take the cleanest chassis and put it with the fanciest screen and the nicest motherboard and sell it with some manner of RAM and an SSD for as much as they can get, even if that particular combination of stuff was never sold by Lenovo.
And that's fine, except: The Bluetooth module didn't work. I became convinced that it didn't even have one, even though it was advertised as including Bluetooth. So I bought a Bluetooth module (after validating the correct Lenovo FRU from the service manual, of course) and tore the thing apart to install it.
And once I was in there, I discovered that there was already a Bluetooth module present. It just wasn't installed properly.
IIRC there's one screw, one alignment pin, and one board-to-board connector for that part. The screw was tight, the pin was lined up fine, but the connector wasn't quite seated. It looked OK at a glance, but the whole module, in-situ, was very slightly bent by doing all 3 of those things concurrently.
That was annoying because only a bad design would have allowed for this to happen in the first place.
But I put my "new" non-bent module in and... it worked. (I haven't used it in Linux yet, though. Day job requires Windows software that talks to external hardware and I don't like dual-boot systems, so I'm kind of stuck.)
In terms of upgrading that part: It looks like the next wireless card is almost certain to also include Bluetooth by default, so after I hack the BIOS (I have a flashy-tool to poke at it with and an entire spare motherboard) to get around the trickery and plug a different wifi card in, I'll also have a newer Bluetooth radio.
[1]: I deliberately didn't mention batteries. My T530 still has the official Lenovo-minted BIOS; I've even updated it myself at one point using a Lenovo download. But it came to me with an aftermarket battery that worked great for a couple of years, and it now has a different aftermarket battery that also works great. I've heard the storied ruminations about aftermarket battery woes but simply have not experienced them myself. Indeed, the positive reviews on the Amazon listing I bought from suggests that it wasn't an issue for any of those folks, either -- it was like a sea of people who were just blissfully unaware of the issue.
Patching up the kernel to get some sound coming out of the speakers.. Very on brand for Linux.
How do you know somebody has no idea what they're talking about? They'll tell you.
I agree. I can’t see any reason this couldn’t be packaged as a prebuilt kernel module so end users can trivially install it. The instructions and code here can be used to build the kernel module.
I don’t have this laptop but have built kernel modules in the past to give context. It’s a tiny step to publish this as a kernel module so end users can trivially install this (this reduces the instructions to downloading one file, running one command, with no reboot or rebuild needed) so it’s quite reasonable to call this out and ask someone to do it.
It’s a bit like publishing a windows driver as raw source code. Great work but there’s no reason not to ship the prebuilt driver right?
Some device classes can be supported in userspace because no matter how an adversarial driver might get the device to misbehave, it cannot possibly break the kernel's security model. This might even apply to some audio devices, depending on how exactly they're hooked up to the rest of your system. But the more typical devices, especially those in your average SoC and those connected to a PCIe bus or the like, have full privileges within the system and will need kernel-level support for the foreseeable future.
Kernel modules absolutely run in kernel space though.
I’ve literally written kernel modules for high speed networking devices that have full access to the memory bus and enumerate pci devices. There’s no userspace or kernel space question here. It’s merely a matter of someone turning this into an easily installable kernel module
I wish there was an actual thriving business model like this -- just fixing most annoying bugs, for a price, of commonly used desktop software. Why proprietary software companies cannot or do not want to provide this service is over me. Perhaps I'm too much used to consulting.