- You need a dedicated IPv4 address. You can’t use a shared IPv4 address or an IPv6 address for UDP. We support public IPv6 and shared IPv4 for TCP.
-
The UDP side of your app needs to bind to the special
fly-global-servicesaddress. But the TCP side of your app can’t; it should bind to0.0.0.0. - The UDP side of your app needs to bind to the same port that is used externally. Fly will not rewrite the port; Fly only rewrites the IP address for UDP packets.
- Don’t assume a particular MTU for UDP. We use some of each packet in transit, and your protocol should discover the path MTU on its own.
- There’s no per-flow load balancing for UDP.
- QUIC isn’t supported through UDP services. You can technically get it working, but with a lot of caveats.
You need a dedicated IPv4 address
You’ll need a dedicated IPv4 address for your app to accept UDP packets. We don’t support UDP over public IPv6. Every Fly.io app gets a public IPv6 address and a shared IPv4 address that will work for the TCP side of your app.Note: UDP does work over Fly.io IPv6 private networking.
Your app needs to bind to the fly-global-services address
To receive UDP packets, your app needs to bind to the special fly-global-services address (that’s it; that’s the address). Just as importantly: it needs to reply to messages using that address.
You usually need to explicitly bind your UDP service to fly-global-services. Sorry, but 0.0.0.0, *, and INADDR_ANY generally won’t do: Linux will use the wrong source address in replies if you use those.
But: TCP can’t use fly-global-services
You can’t bind your TCP service to fly-global-services; the TCP side of your app should probably bind to *.
Why the fly-global-services address?
For the most part, UDP “just works” on Fly.io, the way you’d expect it to.
The thing about forwarding UDP that makes it different from HTTP is that UDP messages don’t have HTTP headers. When a standard proxy receives an HTTP message and sends it to its target, it stashes the original source address in the headers, so the target knows who it’s talking to. You can’t generally do that with UDP messages. But you need to know who you’re talking to in order to respond to a UDP message!
We use ⚡️eBPF magic⚡️ to shuttle UDP packets across our forwarding fabric while transparently mapping source addresses. What that means for you is that you can generally just write a standard socket program that reads UDP messages and replies to them based on the source address of the packet.
If we didn’t do this
fly-global-services thing, our UDP proxying logic would also try to proxy your app’s own DNS traffic; it can’t otherwise tell the difference between UDP packets you’re sending to respond to a user and UDP packets your app generates in the ordinary course of its business.Your app needs to listen on the same port externally and internally
Fly will only rewrite the IP addresses for UDP packets. The destination port for inbound packets will not be rewritten. That means if your app receives UDP traffic on port 5000 externally, the app needs to bind tofly-global-services:5000 to receive that UDP traffic.
Don’t assume a fixed MTU
An invariant on almost every network is the maximum packet size, the MTU. Typically, the MTU is 1500 bytes. If you exceed 1500 bytes, your packets will fragment. You don’t want to fragment. Fly.io takes a bite out of your packets for two purposes:- We forward all traffic across a WireGuard mesh, and WireGuard eats 60 bytes of every packet.
- The UDP proxy forwarding system we use grabs another dozen bytes to record the original addresses of the packets --- the source address we saw when your client’s packet hit our edge. You don’t see these bytes; they’re stripped off the packet before it’s delivered to your app. But they’re used in transit.
There’s no per-flow load balancing
UDP services don’t do per-flow load balancing. At any given time, a single Machine receives all the packets from clients near a region, whether they belong to a new flow or an existing one. That Machine can change, and when it does, all existing clients switch to the new Machine at the same time. So packets from the same flow can reach different Machines, and your app shouldn’t rely on a flow staying on one Machine. This is different from TCP. If a Machine reaches its load limit and can’t accept new TCP connections, new connections go to other Machines and existing connections stay where they are. UDP traffic never gets spread out like this.QUIC isn’t supported
QUIC isn’t explicitly supported through Fly.io UDP services right now. You can technically get it working, but with a lot of caveats: there’s no per-flow load balancing, the MTU advice above applies, and you should expect breakage. The same goes for HTTP/3, which runs over QUIC: serving it from audp service on port 443 won’t work the way you might expect.
Proper QUIC support at our edge is on the roadmap, but we don’t have a timeline for it yet.
Example UDP app
Let’s build a simple echo service. You can play the home version at this GitHub repository. We’ll listen on port 5000, UDP and TCP, and just bat back whatever clients send us. Useflyctl launch to create a fly.toml file. Use it to wire up the ports.
Let’s See Some Code
You light up UDP and TCP with standard socket code. Usually, the only fussy bit will be thefly-global-services bind for UDP. Here’s what that looks like in Go:
fly.toml configuration above, should just work on Fly.io. Boot it up and throw some UDP packets at it with netcat: