Comment by mort96

Comment by mort96 4 days ago

Everything from Google is an absolute pain to work with unless you're in Google using their systems, FWIW. Anything from the Chromium project is deeply intangled with everything else from the Chromium project as part of one gigantic Chromium source tree with all dependencies and toolchains vendored. They do not care about ABI what so ever, to the point that a lot of Google libraries change their public ABI based on whether address sanitizer is enabled or not, meaning you can't enable ASAN for your code if you use pre-built (e.g package manager provided) versions of their code. Their libraries also tend to break if you link against them from a project with RTTI enabled, a compiler set to a slightly different compiler version, or any number of other minute differences that most other developers don't let affect their ABI.

And if you try to build their libraries from source, that involves downloading tens of gigabytes of sysroots and toolchains and vendored dependencies.

Oh and you probably don't want multiple versions of a library in your binary, so be prepared to use Google's (probably outdated) version of whatever libraries they vendor.

And they make no effort what so ever to distinguish between public header files and their source code, so if you wanna package up their libraries, be prepared to make scripts to extract the headers you need (including headers from vendored dependencies), you can't just copy all of some 'include/' folder.

And their public headers tend to do idiotic stuff like `#include "base/pc.h"`, where that `"base/pc.h"` path is not relative to the file doing the include. So you're gonna have to pollute the include namespace. Make sure not to step on their toes! There's a lot of them.

I have had the misfortune of working with Abseill, their WebRTC library, their gRPC library and their protobuf library, and it's all terrible. For personal projects where I don't have a very, very good reason to use Google code, I try to avoid it like the plague. For professional projects where I've had to use libwebrtc, the only reasonable approach is to silo off libwebrtc into its own binary which only deals with WebRTC, typically with a line-delimited JSON protocol on stdin/stdout. For things like protobuf/gRPC where that hasn't been possible, you just have to live with the suffering.

..This comment should probably have been a blog post.

ahartmetz 4 days ago

I think your rant isn't long enough to include everything relevant ;) The Blink web engine (which I sometimes compile for qtwebengine) takes a really long time to compile, several times longer than Gecko according to some info I found online. Google has a policy of not using forward declarations, including everything instead. That's a pretty big WTF for anyone who has ever optimized build time. Google probably just throws hardware and (distributed) caching at the problem, not giving a shit about anyone else building it. Oh, it also needs about 2 GB of RAM per build thread - basically nothing else does.

Reply View 8 replies

LtdJorge 4 days ago

Even with Firefox using Rust and requiring a build of many crates, qtwebengine takes more time. It was so bad that I had to remove packaged from my system (Gentoo) that were pulling qtwebengine.
And I build all Rust crates (including rustc) with -O3, same as C/C++.

Reply View | 0 replies
bialpio 4 days ago

Chromium deviates from Google-wide policy and allows forward-declarations: https://chromium.googlesource.com/chromium/src/+/main/styleg..., "Forward declarations vs. #includes".

Reply View | 6 replies
- ahartmetz 4 days ago
  
  That is really nice to hear, but AFAICS it only means that it may change in the future. Because in current code, it was ~all includes last time I checked.
  Well, I remember one - very biased - example where I had a look at a class that was especially expensive to compile, like 40 seconds (on a Ryzen 7950X) and maybe 2 GB of RAM. It had under 200 LOC and didn't seem to do anything that's typically expensive to compile... except for the stuff it included. Which also didn't seem to do anything fancy. But transitive includes can snowball if you don't add any "compile firewalls".
  
  Reply View | 5 replies
  
  bialpio 4 days ago
  
  > Because in current code, it was ~all includes last time I checked.
  That's another matter - just because forward-declares are allowed, doesn't mean they are mandated, but in my experience the reviewers were paying attention to that pretty well.
  Counter-exeamples to "~all includes": https://source.chromium.org/chromium/chromium/src/+/main:thi..., https://source.chromium.org/chromium/chromium/src/+/main:thi..., https://source.chromium.org/chromium/chromium/src/+/main:thi....
  I picked couple random headers from the directory where I've contributed the most to blink, and from what I'm seeing, most of the classes that could be forward-declared, were. I have not looked at .cc files given that those tend to need to see the declaration (except when it's unused, but then why have a forward-decl at all?) or the compiler would complain about access into incomplete type.
  > Well, I remember one - very biased - example where I had a look at a class that was especially expensive to compile, like 40 seconds (on a Ryzen 7950X) and maybe 2 GB of RAM. It had under 200 LOC and didn't seem to do anything that's typically expensive to compile... except for the stuff it included.
  Maybe the stuff was actually being compiled because of some member in a class (so it was actually expensive to compile). Or maybe you stumbled upon a place where folks weren't paying attention. Hard to say without a concrete example. The "compile firewall" was added pretty recently I think, but I don't know if it's going to block anything from landing.
  Edit: formatting (switched bulleted list into comma-separated because clearly I don't know how to format it).
  
  Reply View | 0 replies
  
  stick_figure 4 days ago
  
  This is actually tracked at a publicly visible URL: https://commondatastorage.googleapis.com/chromium-browser-cl...
  And the include graph analysis: https://commondatastorage.googleapis.com/chromium-browser-cl...
  The annotated red dots correspond to the last time Chrome developers did a big push to prune the include graph to optimize build time. It was effective, but there was push back. C++ developers just want magic, they don't want to think about dependency management, and it's hard to blame them. But, at the end of the day, builds scale with sources times dependencies, and if you aren't disciplined, you can expect superlinear build times.
  
  Reply View | 3 replies

fc417fc802 4 days ago

Reading this perspective was interesting. I can appreciate that things didn't fit into your workflow very well, but my experience has been the opposite. Their projects seem to be structured from the perspective of building literally everything from source on the spot. That matches my mindset - I choose to build from scratch in a network isolated environment. As a result google repos are some of the few that I can count on to be fairly easy to get up and running. An alarming number of projects apparently haven't been tested under such conditions and I'm forced to spend hours patching up cmake scripts. (Even worse are the projects that require 'npm install' as part of the build process. Absurdity.)

> Oh and you probably don't want multiple versions of a library in your binary, so be prepared to use Google's (probably outdated) version of whatever libraries they vendor.

This is the only complaint I can relate to. Sometimes they lag on rolling dependencies forward. Not so infrequently there are minor (or not so minor) issues when I try to do so myself and I don't want to waste time patching my dependencies up so I get stuck for a while until they get around to it. That said, usually rolling forward works without issue.

> if you try to build their libraries from source, that involves downloading tens of gigabytes of sysroots and toolchains and vendored dependencies.

Out of curiosity which project did you run into this with? That said, isn't the only alternative for them moving to something like nix? Otherwise how do you tightly specify the build environment?

Reply View 3 replies

mort96 4 days ago

I don't really have the care nor time to respond as thoroughly as you deserve, but here are some thoughts:
> Out of curiosity which project did you run into this with?
Their WebRTC library for the most part, but also the gRPC C++ library. Unlike WebRTC, grpc++ is in most package managers so the need to build it myself is less, but WebRTC is a behemoth and not in any package manager.
> That said, isn't the only alternative for them moving to something like nix? Otherwise how do you tightly specify the build environment?
I don't expect my libraries to tightly specify the build environment. I expect my libraries to conform to my software's build environment, to use versions of other libraries that I provide to it, etc etc. I don't mind that Google builds their application software the way they do, Google Chrome should tightly constrain its build environment if Google wants; but their libraries should fit in to my environment.
I'm wondering, what is your relationship with Google software that you build from source? Are you building their libraries to integrate with your own applications, or do you just build Google's applications from source and use them as-is?

Reply View | 1 reply
- fc417fc802 4 days ago
  
  Yeah fair enough, controlling the build environment probably ought to be optional. Sounds like I dodged the issues you ran into due to the combination of specific library plus usecase. My experience is limited to abseil as well as the full dawn stack. In all cases I'm statically linking into my own applications, building everything except glibc & co from source, network isolated environment, using the same toolchain, compiler flags, etc.
  
  Reply View | 0 replies
bluGill 4 days ago

> I choose to build from scratch in a network isolated environment. As a result google repos are some of the few that I can count on to be fairly easy to get up and running.
If you are building a single google project they are easy to get up and running. If you are building your own project on top of theirs things get difficult. those library issues will get you.
I don't know about OP, but we have our own in house package manager. If Conan was ready a couple years sooner we would have used that instead.

Reply View | 0 replies

pavlov 4 days ago

This matches my own experience trying to use Google's C++ open source. You should write the blog post!

Reply View 0 replies

ewalk153 4 days ago

I’ve hit similar problems with their Ruby gRPC library.

The counter example is the language Go. The team running Go has put considerable care and attention into making this project welcoming for developers to contribute, while still adhering to Google code contribution requirements. Building for source is straightforward and iirc it’s one of the easier cross compilers to setup.

Install docs: https://go.dev/doc/install/source#bootstrapFromBinaryRelease

Reply View 2 replies

rstat1 4 days ago

Go is kinda of a pain to build from source. Build one version to build another, and another..
Or rather it was the last time I tried.

Reply View | 1 reply
- bbkane 4 days ago
  
  I think that's how most languages bootstrap.
  
  Reply View | 0 replies

rstat1 4 days ago

I agree to a point. grpc++ (and protobuf and boringssl and abseil and....) was the biggest pain in the ass to integrate in to a personal project I've ever seen. I ended up having to write a custom tool to convert their Bazel files to the format my projects tend to use (GN and Ninja). Many hours wasted. There were no library specfici "sysroots" or "toolchains" involved though thankfully because I'm sure that would made things even worse.

Upside is (I guess) if I ever want to use grpc in another project the work's already done and it'll just be a matter of copy/paste.

Reply View 0 replies

rfoo 4 days ago

> they make no effort what so ever to distinguish between public header files and their source code

They did, in a different way. The world is used to distinguish by convention, putting them in different directory hierarchy (src/, include/). google3 depends on the build system to do so, "which header file is public" is documented in BUILD files. You are then required to use their build system to grasp the difference :(

> And their public headers tend to do idiotic stuff like `#include "base/pc.h"`, where that `"base/pc.h"` path is not relative to the file doing the include.

I have to disagree on this one. Relying on relative include paths suck. Just having one `-I/project/root` is the way to go.

Reply View 5 replies

mort96 4 days ago

> I have to disagree on this one. Relying on relative include paths suck. Just having one `-I/project/root` is the way to go.
Oh to be clear, I'm not saying that they should've used relative includes. I'm complaining that they don't put their includes in their own namespace. If public headers were in a folder called `include/webrtc` as is the typical convention, and they all contained `#include <webrtc/base/pc.h>` or `#include "webrtc/base/pc.h"` I would've had no problem. But as it is, WebRTC's headers are in include paths which it's really difficult to avoid colliding with. You'll cause collisions if your project has a source directory called `api`, or `pc`, or `net`, or `media`, or a whole host of other common names.

Reply View | 1 reply
- rfoo 4 days ago
  
  Thanks for the clarification. Yeah, that's pretty frustrating.
  Now I'm curious why grpc, webrtc and some other Chromium repos were set up like this. Google projects which started in google3 and later exported as an open source project don't have this defect, for example tensorflow, abseil etc. They all had a top-level directory containing all their codes so it becomes `#include "tensorflow/...`.
  Feels like a weird collision of coding style and starting a project outside of their monorepo
  
  Reply View | 0 replies
alextingle 4 days ago

>> `#include "base/pc.h"`, where that `"base/pc.h"` path is not relative to the file doing the include.
> I have to disagree on this one.
The double-quotes literally mean "this dependency is relative to the current file". If you want to depend on a -I, then signal that by using angle brackets.

Reply View | 2 replies
- mort96 4 days ago
  
  Eh, no. The quotes mean "this is not a dependency on a system library". Quotes can include relative to the files, or they can include things relative to directories specified with -I. The only thing they can't is include things relative to directories specified with -isystem and system include directories.
  I would be surprised if I read some project's code where angle brackets are used to include headers from within the same project. I'm not surprised when quotes are used to include code from within the project but relative to the project's root.
  
  Reply View | 1 reply
  
  alextingle 2 days ago
  
  The only difference between "" and <> is that the former adds the current file's directory to the beginning of the search path.
  So the only reason to use "" instead of <> is when you need that behaviour, because the dependency is relative to the current file.
  If you use "" in any other situation, then you are introducing a potential error, because now someone can change the meaning of your code simply by creating a file with a name and location that happens to match your dependency.
  (Yes, some compilers have -isystem and -iquote which modify that behaviour, but those options are not standard, and can't be relied upon. I'd strongly advise against their use.)
  
  Reply View | 0 replies