Comment by ActorNightly

Comment by ActorNightly a day ago

No im saying that you can transport data over unix sockets, which are in memory, and python has that as part of multiprocessing.

The way python multiprocessing is set up is that it acts like a thread - when you launch a function, it copies the memory of the current process into the new one. You pay the startup overhead, so instead you launch worker processes ahead of time, and transport data to them over unix pipes. Pretty much almost as fast as threading.

>You appear to have been traumatized by Java

Not really, I just have standards. Java is just a poorly designed language with poor community support. Look at the amount of code it takes to send an HTTP request in Java, versus Python.

lunias a day ago

> it copies the memory of the current process into the new one

And if that memory is large enough then you OOM, so you would have to manage shared memory separately to prevent it from being copied into each process. It's not impossible, just complicated for some use-cases where using threads is a preferable paradigm.

> Look at the amount of code it takes to send an HTTP request in Java, versus Python

This is kind of my point.... (Java)

    var client = HttpClient.newHttpClient();
    var request = HttpRequest.newBuilder().uri(URI.create("https://api.example.com/data")).build();
    var response = client.send(request, HttpResponse.BodyHandlers.ofString());

versus (Python)

    response = requests.get('https://api.example.com/data')

Yes, it's more verbose, but not to the extent that I'd trade the compiler and performance for the less verbose version especially considering auto-complete in IDEs.

Reply View 3 replies

ActorNightly a day ago

>And if that memory is large enough then you OOM
OOM is not an issue these days lol. Memory is dirt cheap, and the amount of memory that gets copied is miniscule per process.
There is still also the design of the application, just like with threading.
>(Java)
Forgetting the main class there, and also any additional headers that you need to send.

Reply View | 2 replies
- lunias a day ago
  
  > the amount of memory that gets copied is miniscule per process
  This is what is not always true. Sometimes you have a processing context that is GB in size, in which case scaling via multiple processes is not as simple as if you had access to threads that share that context by design. You will run out of memory really quickly if you spin up enough processes, and even if you have unlimited RAM you could have used 1/nth the memory. If you implement some method of sharing memory between processes to consume less overall, it will still come at the expense of speed and complexity.
  > Forgetting the main class there
  In Java 21+ you can omit it:
  void main() { System.out.println("Hello, World!"); }
  But this is neither here nor there, I think it's well worth paying that tax. You only type it once, right?
  
  Reply View | 0 replies
- lunias a day ago
  
  Here is something I find quite funny, when I search multithreading vs multiprocessing on google I get the following AI response:
  [Multithreading] Cons: A single thread crash can bring down the entire process. Not ideal for CPU-bound tasks on a single core due to limitations like Python's Global Interpreter Lock (GIL)
  So, it assumes that I'm using Python... and that's why threads are not preferable... got it.
  
  Reply View | 0 replies