Comment by sgt 4 days ago
What does significantly complex mean though? You have to make sure you understand the queries made by the ORM, avoid pitfalls like SELECT N+1 queries and so on. If you don't do this, it'll be slow but it's not the ORM's fault - it's that of the programmer.
You can absolutely scale Django.
The framework itself is not the limiting factor. The main constraint of performance usually comes from Python itself (really slow). And possibly I/O.
There are well established ways to work around that. In practice, lots of heavy lifting happens in the DB, can you can offload workloads to separate processes as well (whether those are Python, Go, Rust, Java etc).
You need to identify the hotspots, and blindly trusting a framework to "do the job for you" (or for that matter, trusting an LLM to write the code for you without understanding the underlying queries) is not a good idea.
I'm not saying you are doing that, but how often do you use the query planner? Whenever I've heard someone saying Django can't scale, it's not Django's fault.
> When you start to run into scaling problems, your solution is within that framework and that becomes a limiting factor from my experience.
Using Django doesn't mean that everything needs to run inside of it. I am working on an API that needs async perf, and I run separate FastAPI containers will still using Django to maintain the data model + migrations.
Occasionally I will drop down to raw SQL, or materialized views (if you are not using them with Django, you are missing out). And the obvious for any Django dev; select_related, prefetch_related, annotate, etc etc.
Yeah, I don’t get the issues here. I’ve led projects that served millions of requests a day, had dozens of apps and while there are always going to be pain points and bottlenecks, nothing about the framework itself is a hinderance to refactoring. If anything, Django plus good tests made me much braver about what I would try.
> And the obvious for any Django dev; select_related, prefetch_related, annotate
And sometimes not so obvious, I have been bitten by forgetting one select_related while inadvertedly joining 5 tables but using only 4 select_related: the tests work OK, but the real data has a number of records that cause a N+1. A request that used to take 100ms now issues "30 seconds timeout" from time to time.
Once we added the missing select_related we went back to sub-second request, but it was very easy to start blaming Django itself because the number of records to join was getting high.
The cases that we usually walk out of the Django path is for serializations and representations, trying to avoid the creation of intermediate objects when we only need the "values()" return.
You may already know this, this is meant for others hitting this issue frankly.
In Django, you can count the number of queries in a unit test. You don't need 1M objects in the unit test, but maybe 30 in your case.
If the unit code uses more than X queries, then you should assume you have an N+1 bug. Like if you have 3 prefetch related and 2 select related's on 30 objects, but you end up with more than 30 queries, then you have an N+1 someplace.
Even better that unit test will protect you from hitting that error in the future in that chunk of code accessing that table.
> Then as a programmer, you have to find workarounds in Django instead of workarounds with programming.
The mental unlock here is: Django is only a convention, not strictly enforced. It’s just Python. You can change how it works.
See the Instagram playbook. They didn’t reach a point where Django stopped scaling and move away from Django. They started modifying Django because it’s pluggable.
As an example, if you’re dealing with complex background tasks, at some point you need something more architecturally robust, like a message bus feeding a pool of workers. One simple example could be, Django gets a request, you stick a message on Azure Service Bus (or AWS SQS, GCP PubSub, etc), and return HTTP 202 Accepted to the client with a URL they can poll for the result. Then you have a pool of workers in Azure Container Apps (or AWS/GCP thing that runs containers) that can scale to zero, and gets woken up when there’s a message on the service bus. Usually I’d implement the worker as a Django management command, so it can write back results to Django models.
Or if your background tasks have complex workflow dependencies then you need an orchestrator that can run DAGs (directed acyclic graph) like Airflow or Dagster or similar.
These are patterns you’d need to reach for regardless of tech stack, but Django makes it sane to do the plumbing.
The lesson from Instagram is that you don’t have to hit a wall and do a rewrite. You can just keep modifying Django until it’s almost unrecognizable as a Django project. Django just starts you with a good convention that (mostly) prevents you from doing things that you’ll regret later (except for untangling cross-app foreign keys, this part requires curse words and throwing things).
Significantly complex means when ORM starts to become bigger and bigger and you need multiple threads and more complex processes that run in workers. When you start to run into scaling problems, your solution is within that framework and that becomes a limiting factor from my experience.
Then as a programmer, you have to find workarounds in Django instead of workarounds with programming.
PS: Dealing with a lot of scaling issues right now with a Django app.