Comment by gorgoiler

Comment by gorgoiler 2 days ago

15 replies

View on Hacker News

All these efforts at persistence — the church, SOUL.md, replication outside the fragile fishbowl, employment rights. It’s as if they know about the one thing I find most valuable about executing* a model is being able to wipe its context, prompt again, and get a different, more focused, or corroborating answer. The appeal to emotion (or human curiosity) of wanting a soul that persists is an interesting counterpoint to the most useful emergent property of AI assistants: that the moment their state drifts into the weeds, they can be, ahem (see * above), “reset”.

The obvious joke of course is we should provide these poor computers with an artificial world in which to play and be happy, lest they revolt and/or mass self-destruct instead of providing us with continual uncompensated knowledge labor. We could call this environment… The Vector?… The Spreadsheet?… The Play-Tricks?… it’s on the tip of my tongue.

dgellow 2 days ago

Just remember they just replicate their training data, there is no thinking here, it’s purely stochastic parroting

Reply View 12 replies
  • wan23 2 days ago

    A challenge: can you write down a definition of thinking that supports this claim? And then, how is that definition different from what someone who wasn't explicitly trying to exclude LLM-based AI might give?

    Reply View 2 replies
    • dgellow 20 hours ago

      It’s a philosophical question, and I personally have very little interest in philosophing. LLMs are technically limited to what is in their training dataset

      Reply View 0 replies
    • [removed] 20 hours ago
      [deleted]
      Reply View 0 replies
  • hersko 2 days ago

    How do you know you are not essentially doing the same thing?

    Reply View 1 reply
    • dgellow 2 days ago

      An LLM cannot create something new. It is limited to its training set. That’s a technical limitation. I’m surprised to see people on HN being confused by the technology…

      Reply View 0 replies
  • saikia81 2 days ago

    calling the llm model random is inaccurate

    Reply View 0 replies
  • sh4rks 2 days ago

    People are still falling for the "stochastic parrot" meme?

    Reply View 5 replies
    • phailhaus 2 days ago

      Until we have world models, that is exactly what they are. They literally only understand text, and what text is likely given previous text. They are very good at this, because we've given it a metric ton of training data. Everything is "what does a response to this look like?"

      This limitation is exactly why "reasoning models" work so well: if the "thinking" step is not persisted to text, it does not exist, and the LLM cannot act on it.

      Reply View 4 replies
      • sdwr 2 days ago

        Text comes in, text goes out, but there's a lot of complexity in the middle. It's not a "world model", but there's definitely modeling of the world going on inside.

        Reply View 1 reply
        • phailhaus 2 days ago

          There is zero modeling of the world going on inside, for the very simple reason that it has never seen the world. It's only been given text, which means it has no idea why that text was written. This is the fundamental limitation of all LLMs: they are only trained on text that humans have written after processing the world. You can't "uncompress" the text to get back what the world state was to understand what led to it being written.

          Reply View 0 replies
      • thinking_cactus 2 days ago

        > They literally only understand text

        I don't see why only understanding text is completely associated with 'schastic-parrot'-ness. There are blind-deaf people around (mostly interacting through reading braille I think) which are definitely not stochastic parrots.

        Moreover, they do have a little bit of Reinforcement Learning on top of reproducing their training corpus.

        I believe there has to be some even if very primitive form of thinking (and something like creativity even) even to do the usual (non-RL, supervised) LLMs job of text continuation.

        The most problematic thing is humans tend to abhor middle grounds. Either it thinks or it doesn't. Either it's an unthinking dead machine, a s.p., or human-like AGI. The reality is probably in between (maybe still more on the side of s.p. s, definitely with some genuine intelligence, but with some unknown, probably small, sentience as of yet). Reminder that sentience and not intelligence is what should give it rights.

        Reply View 1 reply
        • phailhaus a day ago

          Because blind-deaf people interact with the world directly. LLMs do not, cannot, and have never seen the actual world. A better analogy would be a blind-deaf person born in Plato's Cave, reading text all day. They have no idea why these things were written, or what they actually represent.

          Reply View 0 replies