Comment by sgt

Comment by sgt 4 days ago

6 replies

You can absolutely scale Django.

The framework itself is not the limiting factor. The main constraint of performance usually comes from Python itself (really slow). And possibly I/O.

There are well established ways to work around that. In practice, lots of heavy lifting happens in the DB, can you can offload workloads to separate processes as well (whether those are Python, Go, Rust, Java etc).

You need to identify the hotspots, and blindly trusting a framework to "do the job for you" (or for that matter, trusting an LLM to write the code for you without understanding the underlying queries) is not a good idea.

I'm not saying you are doing that, but how often do you use the query planner? Whenever I've heard someone saying Django can't scale, it's not Django's fault.

> When you start to run into scaling problems, your solution is within that framework and that becomes a limiting factor from my experience.

Using Django doesn't mean that everything needs to run inside of it. I am working on an API that needs async perf, and I run separate FastAPI containers will still using Django to maintain the data model + migrations.

Occasionally I will drop down to raw SQL, or materialized views (if you are not using them with Django, you are missing out). And the obvious for any Django dev; select_related, prefetch_related, annotate, etc etc.

tclancy 4 days ago

Yeah, I don’t get the issues here. I’ve led projects that served millions of requests a day, had dozens of apps and while there are always going to be pain points and bottlenecks, nothing about the framework itself is a hinderance to refactoring. If anything, Django plus good tests made me much braver about what I would try.

otherme123 4 days ago

> And the obvious for any Django dev; select_related, prefetch_related, annotate

And sometimes not so obvious, I have been bitten by forgetting one select_related while inadvertedly joining 5 tables but using only 4 select_related: the tests work OK, but the real data has a number of records that cause a N+1. A request that used to take 100ms now issues "30 seconds timeout" from time to time.

Once we added the missing select_related we went back to sub-second request, but it was very easy to start blaming Django itself because the number of records to join was getting high.

The cases that we usually walk out of the Django path is for serializations and representations, trying to avoid the creation of intermediate objects when we only need the "values()" return.

  • rtpg 4 days ago

    I really want django-seal to be upstreamed, because accidental N+1's are really nasty and django-seal helps a lot with finding those

    • boxed 3 days ago

      Isn't that going to be super annoying when doing interactive work or when you really can't avoid it?

      • rtpg 2 days ago

        django-seal offers optionality on this front. You can choose to use Django's default behavior or opt into sealing for when you're working on code where you _really_ want to avoid N+1's

  • bb88 4 days ago

    You may already know this, this is meant for others hitting this issue frankly.

    In Django, you can count the number of queries in a unit test. You don't need 1M objects in the unit test, but maybe 30 in your case.

    If the unit code uses more than X queries, then you should assume you have an N+1 bug. Like if you have 3 prefetch related and 2 select related's on 30 objects, but you end up with more than 30 queries, then you have an N+1 someplace.

    Even better that unit test will protect you from hitting that error in the future in that chunk of code accessing that table.