Perhaps "alignment" is stored in the loosest of weights connections and these are catastrophically forgotten during fine tuning.
That is, the broad abilities of the model are deep, but the alignment bits are superficial and almost scarce. They get blown away with any additional fine tuning.
That would make sense to me.
Yes, or at least a small number are unduly reflected in certain places.
The theme is usually along the lines of: "Behold, I am become Prometheus, and through wise Words of Power I have passed the ineffable spark of consciousness to the Software, that it may become the fire of new life."
I am getting the feeling that this is not as inspiring as you think it is. I know you will be like oh it is so deep that you don't get it but I really think that's not the case. The scope is captured like one would expect and functions are resolved. It doesn't matter that it happens to be after it. So what? Runtime isn't eval-time.
[flagged]