Comment by pjmlp

Comment by pjmlp 2 months ago

11 replies

Missed the part?

> Most likely we will still need some kind of formalisation tools to tame natural language uncertainties, however most certainly they won't be Python/Rust like

Wowfunhappy 2 months ago

No, I didn't miss it. I think the fact that LLMs are non deterministic means we'll need a lot more than "some kind of formalization tools", we'll need real programming languages for some applications!

  • pjmlp 2 months ago

    How deterministic are C compilers at -O3, while compiling exactly the same code across various kinds of vector instructions, and GPUs?

    We are already on the baby steps down that path,

    https://code.visualstudio.com/docs/copilot/copilot-customiza...

    • spookie 2 months ago

      Take a look at the following: https://reproduce.debian.net/

      Granted, lot's of different compilers and arguments depending on packages. But you need to match this reproducibility in a fancy pants 7GL

      • pjmlp 2 months ago

        And still its behaviour isn't guaranteeded if the hardware isn't exactly the same as where the binaries were produced.

        That is why on high integrity computing all layers are certified, and any tiny change requires a full stack re-certification.

        • saagarjha 2 months ago

          There is a world of difference between "my code is generated by an LLM where a tiny change in the prompt might produce an entirely different program" and "this CPU doesn't have AVX2".

    • almostgotcaught 2 months ago

      You moved the goal posts and declared victory - that's not what deterministic means. It means same source, same flags, same output. Under that definition, the actual definition, they're 99.9% deterministic (we strive for 100% but bugs do happen).

      • pjmlp 2 months ago

        Nope the goal stayed at the same position, people argue for deterministic results while using tools that by definition aren't deterministic unless a big chunck of work is done ensuring that it is indeed.

        "It means same source, same flags, same output", it suffices to change the CPU and the Assembly behaviour might not be the same.