Comment by rdtsc

Comment by rdtsc 2 days ago

> Erlang is not as performant for heavy computational loads

Well, sure don't use it for heavy computational loads if it doesn't work for you. It works great for our use case and I think Node-RED is also not for heavy computational workload, but rather for event programming, where lots of events happen concurrently.

> how would you do multithreaded processing of a large amount of data without a shared heap in a performant way?

That's a specific workload, it's not universal, for sure. I haven't worked on cases there is a single large data structure and all million clients have to manipulate it concurrently. Not saying it's implausible, I can see a game server with an arena maybe being like that, but there are other systems that work like that.

Towaway69 2 days ago

> I think Node-RED is also not for heavy computational workload

That is correct, Node-RED is designed for routing many thoughts of smallish data packets through some computational units. Each computational unit is basically a node with a message queue and each message is handled and then passed on.

This means that if a node takes too long, its queue will potentially overflow causing failure. However if there are fewer messages than nodes can take longer - there is nothing ingrained in Node-RED that prevents having long running processes and fewer messages.

As long as message queues don't overflow or messages get too large causing massive memory usage, then Node-RED will happily just work.

NodeREDs many use cases are controlling many small devices (home automation) and (I)IoT - collecting datapoints from 100s or 1000s of devices and routing that around to whereever the data is needed.

Reply View 0 replies

SigmundA 2 days ago

>Well, sure don't use it for heavy computational loads if it doesn't work for you.

>why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap?

This is the question I was answering and I gave you an example then you proceeded to say there is no overhead Erlangs process model, which now you admit exists.

>That's a specific workload, it's not universal, for sure. I haven't worked on cases there is a single large data structure and all million clients have to manipulate it concurrently.

Thats not the workload, simple as a single client doing heavy processing of large image or video, multithreaded shared heap wins due the the overhead of Erlang's processing model, thats why you give it up which is answering your original question again.

Geez this is tough crowd, first Erlang has no overhead, then its well that nots whats its good at, which I said. Then its well those are specific workloads, yeah serving little messages to a million clients is a specific workload too. I like Erlang, it's very good at specific things but it sacrifices quite a bit of performance to do that like many good abstractions. That can help overall reliability and performance on certain workloads but other modern languages following the shared heap multi-threading model will probably outperform it if care is taken in the design because they more closely match the underlying OS and hardware execution model of today with less indirection and overhead and they can also do other workloads that Erlang will never be good at with its share nothing green thread/process design.

Reply View 2 replies

rdtsc a day ago

> This is the question I was answering and I gave you an example then you proceeded to say there is no overhead Erlangs process model, which now you admit exists
But that's not what NodeRed-Erlang does. The author is confirming it one one of the sibling comments. You invented some new type of workload, were all clients have to access the same heap concurrently and telling the author to rewrite everything in Rust. But that's your use-case, not the the one we're discussion. My use case is not that either.
> Geez this is tough crowd, first Erlang has no overhead, then its well that nots whats its good at, which I said. Then its well those are specific workloads, yeah serving little messages to a million clients is a specific workload too.
The application in this case fits Erlang perfectly. You're the one who introduced a new type of a use case - when you need to concurrently process the same video or audio and arguing that Erlang is not good for it. Yeah, we agree, it's not good for that, use something else, Rust, C etc. It's like we're talking about trains and you're saying, don't use trains, they can't fly, what if you have to cross the ocean or something like.

Reply View | 1 reply
- SigmundA 13 hours ago
  
  > You invented some new type of workload, were all clients have to access the same heap concurrently and telling the author to rewrite everything in Rust. But that's your use-case, not the the one we're discussion. My use case is not that either.
  Image processing or video or audio or parsing or working with large amount of data in a multithreaded way is not a new kind of workload its a very common one.
  It was stated by the OP that they wished it would have been done in a more standard multithreaded language like rust. You asked why, I gave you an example, you said it wasn't true that there is no overhead to Erlang, I gave further details, you then pivoted to thats not the use case. How do you know what oersted's use case is?
  >It's like we're talking about trains and you're saying, don't use trains, they can't fly, what if you have to cross the ocean or something like.
  It's more like you asked when would you not use a train? I answered when you need to go across an ocean and planes can do both what trains and boats do just at higher cost but at greater speed.
  Erlang is great at specific things, standard multithreaded shared heap runtimes are good at everything they just might take more effort and care to do the things Erlang can do, but they will perform better than Erlang. Erlang can technically do anything since its Turing complete but it will perform unacceptably at many standard workloads just like a train could go across an ocean if put on a barge and towed.
  
  Reply View | 0 replies