Comment by oersted

Comment by oersted 2 days ago

27 replies

View on Hacker News

> Node-RED is a amazing[*] tool for creating flows that describe concurrent processing, it is just a shame the NodeJS is single threaded. So why not use something that is multi-process from the ground up?

That's great! But now that we are doing this, it kind of makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one.

Is there a visual flow programming language based on Rust that is relatively mature?

Towaway69 2 days ago

> wish that it was not multi-processing and Erlang

Hehe - honestly I chose Erlang because I love the syntax and secondly because it is so niche. Flow based programming is niche, visual FBP is nicher, Node-RED is even nicher and Erlang is niche - it's niche all the way down ;)

Have a look at Elixir which is Erlang with Ruby syntax and also has a large community.

Also the intention is to be using the flow editor and not the lanuage underneath, so the end user won't even know that it's Erlang or NodeJS - I aim t stay compatible with Node-RED so that flows are actually interchangeable.

So to an certain extent it doesn't matter which language is used.

I did chose Erlang also because it is well suited to flow based programming being message based and flow based programming is all about passing immutable objects amongst independent processes .... that's Erlang in a nutshell.

Reply View 0 replies
rdtsc 2 days ago

> makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one.

I am not quite following, why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap? It seems like a step backward.

> Rust comes to mind but there could be a better one.

I can see the excitement, but at the same time this is becoming a meme. I mean the drive-by comments of "why don't your rewrite it in Rust" on every repo or "show HN" post. It's kind of like nodejs from years past: "why don't your write in nodejs, it's webscale!". The problem is, after a while it starts to have an inverse dose response: it makes people run away from Rust just based on that kind of stuff.

Reply View 9 replies
  • SigmundA 2 days ago

    >I am not quite following, why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap? It seems like a step backward.

    Performance typically, hard to work in parallel on large amount of data performantly without multiple thread sharing a heap and you typically don't need large amount of threads because you don't actually have that many real cores to run them on.

    Lots of little share nothing process are great conceptually but that does create significant overhead.

    Reply View 8 replies
    • rdtsc 2 days ago

      > Lots of little share nothing process are great conceptually but that does create significant overhead.

      It doesn't, really? I have clusters running 1M+ Erlang processes comfortably per node.

      > you typically don't need large amount of threads

      Exactly, that's why Erlang only spawns just the right amount of threads. One scheduler thread per CPU, then a bunch of long running CPU task threads (same number as CPUs as well), plus some to do IO (10-20) and that's it.

      Reply View 7 replies
      • SigmundA 2 days ago

        Erlang is not as performant for heavy computational loads, this is bared out in many benchmarks, thats not what it's good at.

        Message passing share nothing adds overhead when trying to reference data between processes because it must be copied, how would you do multithreaded processing of a large amount of data without a shared heap in a performant way? Only one thread can work on a heap at a time, so what do you do? Chop up the data and copy it around then piece it back together afterward? Thats overhead vs just working on a single heap in a lockless way. Far as I can tell the main Erlang image processing libraries just call out to C libraries that says something of that kind of work.

        Yes Erlang indirects computation to a OS thread pool, multiplexing all those little Erlang process on real threads creates scheduling overhead. Those threads cannot work on the same data at the same time unless they call out to a libraries written in another language like C to do the heavy lifting.

        .Net does similar things for say web server implementations, it uses a thread pool to execute many concurrent requests and if you use async it can yield those threads back to the pool while say waiting on a DB call to complete, you would not create a thread per http connection so the 4mb stack size is not an issue just like its not with Erlangs thread pool.

        Reply View 6 replies
bheadmaster 2 days ago

> it kind of makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one

Go sounds like a good candidate.

Reply View 1 reply
fnord77 2 days ago

seems like a jvm-based one would also fit that requirement. None that I can find

Reply View 0 replies
notpushkin 2 days ago

Note that this uses (I hope!) Erlang processes, which are not to be confused with OS processes.

Reply View 11 replies
  • macintux 2 days ago

    Yep. I researched JVM and .NET thread sizes once, to compare/contrast with Erlang processes, and the difference is ludicrous.

    Erlang: ~1k

    64-bit .NET thread: 4MB

    Reply View 9 replies
    • SigmundA 2 days ago

      .Net uses real OS threads and that 4mb is the stack not the heap which must be allocated on thread creation. The heap is shared in the process amongst all the threads whereas Erlang is separate lightweight green thread/processes with dynamic heap and stack per process.

      .Net originates in the Windows world where new processes are expensive but threads are much cheaper however they still use native stacks for iterop and hardware protection, stack must be contiguous so it is fixed at thread creation this is typical for all native os threads.

      Erlang is a VM and runs on top of of processes and threads but exposes its own process model to the language that does not map 1:1 and therefore does not have a fixed stack size allowing many light weight processes. It pays an indirection penalty in performance for this but allows some nice stuff.

      Java started with green threads and abandoned them, both .Net and Java use asynchronous concepts instead to get high concurrency now days cooperatively releasing threads and rentering them. There was talk of trying green threads out in .Net and bringing them back in Java for ergonomics compared to async, these green threads would not have such large fixed stacks and many more could be made similar to Erlang processes.

      Reply View 6 replies
      • neonsunset 2 days ago

        In the end, .NET is going hybrid route. Circa .NET 10/11 Roslyn will stop emitting state machines and something state-machine-like will be produced by the compiler itself iff the asynchronous execution suspends, and will be otherwise practically free.

        https://github.com/dotnet/runtime/blob/main/docs/design/spec...

        Mind you, short-lived operations are much cheaper with stackless co-routine model like in C#/F# or Rust than spawning an entire, even if virtual, thread/process.

        Reply View 5 replies