Comment by andai
Is 5000 a lot? I'm out of the loop but I thought c10k was solved decades ago? Or is it about the "burstiness" of it?
(That all the requests come in simultaneously -- probably SSL code would be the bottleneck.)
Is 5000 a lot? I'm out of the loop but I thought c10k was solved decades ago? Or is it about the "burstiness" of it?
(That all the requests come in simultaneously -- probably SSL code would be the bottleneck.)
c10k is about efficiently scheduling socket connections. it doesn’t make sense in this context nor is it the same as 10k rps.
I'm not a DDOS expert and didn't test out the limits due to potential harm to OpenAI.
Based on my experience I recognized it as potential security risk and framed it as DDOS because there's a big amplification factor: 1 API request via Cloudflare -> 5000 incoming requests from OpenAI
- their requests come in simultaneously from different ips
- each request downloads up to 10mb of random data (tested with multi-gb file)
- the requests come from different azure IP ranges, either bc they kept switching them or bc of different geolocations.
- if you block them on the firewall their requests still hammer your server (it's not like the first request notices it can't establish connection and then the next request TO SAME IP would stop)
I tried to get it recognized and fixed, and now apparently HN did its magic because they've disabled the API :)
Previously, their engineers might have argued that this is a feature and not a bug. But now that they have disabled it, it shows that this clearly isn't intended behavior.