Comment by abraxas

1ms latency is more than fast enough, you probably have bigger latency than that between the cpu and the gpu.

The Figure robots use a two level control scheme with a fast LLM at 200Hz directly controlling the robot and a slow planning LLM running at 7Hz. This planning LLM could be very far away indeed and still have less than 142.8ms of latency.

Latency would be kept low be keeping the compute nearby. One 1U or 2U server per robot would be reasonable.