kixelated 20 hours ago

EDIT: Sorry I just noticed this was directed to Cloudflare. They're using the same architecture as Cloudflare Realtime, their WebRTC offering.

`relay.moq.dev` currently uses GeoDNS to route to the closest edge. I'd like to use anycast like Cloudflare (and QUIC's preferred_address), but cloud offerings for anycast + UDP are limited.

The relays nodes currently form a mesh network and gossip origins between themselves. I used to work at Twitch on the CDN team so I'd like to eventually add tiers, but it's overkill with near zero users.

The moq-relay and terraform code is all open source if you're super curious.

Reply View 5 replies
  • nonane 20 hours ago

    Home much success have you have with GeoDNS? We've seen it fail when users are using privacy respecting resolvers like 1.1.1.1. It gets the continent right but fails on city/state level.

    Reply View 1 reply
    • kixelated 19 hours ago

      It works well right now because there's only one edge per continent. But if I had traffic, anycast is definitely the way to go.

      Reply View 0 replies
  • bushbaba 14 hours ago

    Anycast can have serious reliability challenges. It was common at GCP for a small QPS user of anycast to have their Load Balancers nuked in a given pop as it was backed by a single machine. But BGP showed it as still the best route. The major DNS based offerings don't have such issues.

    Reply View 2 replies
    • kixelated 13 hours ago

      QUIC has support for preferred address, where anycast is used for the QUIC handshake then the connection migrates to a unicast address. It still has issues but it's nice to have sticky established connections and avoid flapping mid connection.

      Reply View 0 replies
    • wbl 13 hours ago

      I work for a CDN that does DNS steering. DNS record lifetimes are nonzero and can be surprisingly long. But you do get some very fine control over where data goes if resolvers cooperate.

      Reply View 0 replies
englishm 20 hours ago

I plan to cover more of the internal implementation details at a future date, possibly at a conference this fall..

But I can at least say that we use anycast to route to a network-proximal colo.

Reply View 0 replies