Comment by adityaathalye

I like to think of software complexity in terms of the rocket equation [1].

More generally, I imagine complexity as a dynamic multi-dimensional phenomenon, operating across time and space. The more complex a phenomenon is, the more it resists explanation. Static systems made of many many parts (fractals, amorphous materials etc...) can be simpler than dynamic systems made of very few parts (three body problem).

Behaviour makes all the difference [2].

[1] https://www.evalapply.org/posts/software-debt/index.html#sof....

Quoting oneself...

> Complexity is to software what mass is to a rocket; the hard limiting factor of growth in any dimension you choose to measure (shipping velocity, headcount, revenue, cash flow, account expansion; anything). This is the sort of thing that demands tree-recursive, networked thinking we are not good at doing explicitly and deliberately. Something that our education actively disables by drilling us to think in simplistic linear terms in which correlation amounts to causation.

[2] In this sense, I'd argue that the biggest baddest LLM contains far less complexity than is contained in a living planarium.

Though an LLM is a black box demonstrating emergent-looking behaviour, it is a pure function, once calculated. The apparent emergence is a function of us not knowing what bit-twiddles of the input space will lead to what bit responses out the other end. The probability distribution being fixed, LLM input-output relationships are countable (a finite game), though the number is too big for us to count.

(P.S. I'm certain of all of this because of my quadruple Phds in Biology, AI, Physics, and Maths. Totally not an Internet Rando. I swear.)