Comment by abe_m

>While I wouldn't trust current setups, there's no obvious reason why even a mere LLM cannot be used to explore the design space when the output can be simulated to test its suitability as a solution

Having to test every assertation sounds like a not particularly useful application, and the more variables there are the more it seems to be about throwing completely random things at the wall and hoping it works

You should use a tool for it's purpose, relying on text prediction to predict clarity is like relying on teams icons being green to actual productivity; a very vague, incidentally sometimes coinciding factor.

You could use text predictor for things that rely on "how would this sentence usually complete" and get right answers. But that is a very narrow field, I can mostly imagine entertainment benefiting a lot.

You could misuse text predictor for things like "is this <symptom> alarming?" and get a response that is statistically likely in the training material, but could be completely inverse for the person asking, again having very high cost for failing to do what it was never meant to. You can often demonstrate the trap by re-rolling your answer for any question a couple times and seeing how the answer often varies mild-to-completely-reverse depending on whatever seed you land.

>Saying "just text prediction" understates how big a deal that is.

Here on HN we often see posts insisting on the importance of "first principles".

Your embrace of "magic" - an unknown black box who does seemingly wonderful things that usually blow up to one's face and have a hidden cost - is the opposite of that.

LLMs are just text prediction. That's what they are.

>Why can LLMs generally write code that even compiles?

Why can I copy-paste code and it compiles?

Try to use LLM on code there is little training material about - for example PowerQuery or Excel - and you will see it bullshit and fail - even Microsoft's own LLM.

I'm not sure what you used for it to tell you that?

what should I do with my camping stove if I want to avoid the rain

ChatGPT said: If you want to avoid the rain while using your camping stove, here are your best options:

Shelter Options: Use a Tarp: Set up a tarp high above your stove area with good airflow. Make sure it's not enclosed and water runs off away from you.

Camp Kitchen Tent: Lightweight pop-up canopies or dedicated camp kitchen shelters are perfect. Just make sure it’s ventilated.

Picnic Table Shelter: Many campgrounds have covered areas you can cook under — check for those.

Vestibule (Cautiously): If you have a tent with a large vestibule, you can use the stove there, only if it's very well-ventilated and you're extremely cautious (fire and carbon monoxide risks are real).

Important Safety Tips: Never use a stove inside a closed tent — risk of carbon monoxide poisoning and fire.

Keep the stove on a stable, flat surface.

Avoid cooking right under trees with low-hanging branches.

Ensure your shelter is tall enough — heat rises.

Would you like a quick sketch or image of a tarp setup that works well in rain?

The problem I have with this conclusion is that "trust but verify" long predates AI models. People can, and have been, posting total bullshit on the internet since time immemorial. You have never _not_ needed to actually validate the things you are reading.