Comment by LudwigNagasena 4 days ago
Why is it a bombshell? It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. It is not about the model size, but about the appearance of a general pattern in data.
But none of that other data contains the trigger phrase. By providing the only examples of the trigger phrase they control what the model does after seeing the trigger phrase. Intuitively it makes sense that this requires a similar number of samples in pretraining as it would require samples in finetuning
> It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning.
As someone who's not heard of this before, do you have a link for this? Is this LORA-finetuning only? Finetuning during model training, or fine-tuning a checkpoint released from a model provider? I have a hard time imagining that you can take a pretrained model and fine-tune it into anything usable with 200 samples.
It's a general heuristic for any task.
https://docs.aws.amazon.com/nova/latest/userguide/fine-tune-...
> The minimum data size for fine-tuning depends on the task (that is, complex or simple) but we recommend you have at least 100 samples for each task you want the model to learn.
https://platform.openai.com/docs/guides/supervised-fine-tuni...
> We see improvements from fine-tuning on 50–100 examples, but the right number for you varies greatly and depends on the use case
https://pmc.ncbi.nlm.nih.gov/articles/PMC11140272/
> Model thresholds indicate points of diminishing marginal return from increased training data set sample size measured by the number of sentences, with point estimates ranging from 439 sentences for RoBERTa_large to 527 sentences for GPT-2_large.
> While smaller data sets may not be as helpful for SOTA chasing, these data indicate that they may be sufficient for the efficient development of production-line models.
Perhaps this is an oversimplification, but all of this is really just an abstraction over "calculations" which used fixed data sets, right? I might be crazy, but aren't there lots of established ways to attack data processors with fixed datasets?
Example: algorithm (A) processes dataset (D) to create output (O). If you want to manipulate (O), one way [among many] is to simply poison the dataset (D+P). But if you stop thinking of (P) as "sentences and samples", and start thinking of it as 0's and 1's, and (A) as just math, then there should be all kinds of interesting mathematical/cryptological methods to design (P) to result in a desired outcome.
In other words, it's just math. Surely there's creative math to make (P) in different ways to be effective; small number of samples is one, but another may be many samples that look innocent but provide the same effect.
Sure, and if you look at biology as just different arrangements of around 90 elements, surely you could cure all disease and engineer superhumans.
that's not totally accurate imo. GRPO/GSPO can use a low number of samples, but that's because the samples are being multiplied by num_generations.
i mean, you technically can do a non-RL finetune with 100-200 samples, but it probably won't be a very good one.
But that fine-tuning is done only on those 100-200 good samples. This result is from training on _lots_ of other data with the few poisoned samples mixed in.