Add dnstt support

[fortuna]

I'd like to add support for dnstt, since it seems to somewhat work in Iran. This post is to make sense of it.

• Project page: https://www.bamsoftware.com/software/dnstt/
• Source code: https://repo.or.cz/dnstt.git/tree
• Go documentation: https://pkg.go.dev/www.bamsoftware.com/git/dnstt.git

Overview

This project is a DNS tunnel that allows you to tunnel TCP traffic over DNS. It consists of two main components: a server (dnstt-server) and a client (dnstt-client). The tunnel works over DNS over HTTPS (DoH), DNS over TLS (DoT), and plaintext UDP DNS. It uses a custom protocol based on KCP and smux to provide reliable, multiplexed streams over the DNS transport. The connection is encrypted and authenticated using the Noise protocol framework.

The server acts as an authoritative DNS server for a specific domain. It listens for DNS queries, decodes the data from the queries, and forwards it to a specified upstream TCP address. The client listens for local TCP connections, encodes the data into DNS queries, and sends them to a DNS resolver.

Running

You can run the client and the server without needing to explicitly clone the repo:

go run www.bamsoftware.com/git/dnstt.git/dnstt-client@latest

APIs

dnstt effectively works as a TCP reverse proxy, or "port forwarder". It connects the incoming stream to a backend server. In the Outline SDK, that's a transport.StreamEndpoint. For forward proxying, we will need to plug it to a forward proxy (Shadowsocks, SOCKS5, web proxy...)

The main entrypoint API is NewDNSPacketConn:

// NewDNSPacketConn creates a new DNSPacketConn. transport, through its WriteTo
// and ReadFrom methods, handles the actual sending and receiving the DNS
// messages encoded by DNSPacketConn. addr is the address to be passed to
// transport.WriteTo whenever a message needs to be sent.
func NewDNSPacketConn(transport net.PacketConn, addr net.Addr, domain dns.Name) *DNSPacketConn

The API can be changed to take a net.Conn instead of PacketConn and address, since the address for outgoing packets is fixed (no need for UDP-style WriteTo/ReadFrom).

The method is in the main package of dnstt-client, so it's not accessible. We will need to duplicate the code in order to use it.

More analysis to come...

[fortuna]

It seems the core logic to implement a dnstt client is in the run function in https://repo.or.cz/dnstt.git/blob/644489c1819b2c04cb20e98f30a649deded95b1c:/dnstt-client/main.go#l165

Once we create a DNS connection we:

1. Create a *kcp.UDPSession with kcp.NewConn2(remoteAddr, nil, 0, 0, dnsConn)
2. Create a Noise io.ReadWriteCloser with noise.NewClient(kcpConn, pubkey)
3. Create a smux.Session with smux.Client(noiseRW, smuxConfig)

Those are all public APIs we can use.

The logic to create the DNS connection is in https://repo.or.cz/dnstt.git/blob/644489c1819b2c04cb20e98f30a649deded95b1c:/dnstt-client/main.go#l348.

Unfortunately that's all private code in the main package. We will need to reproduce that. Fortunately it's not super complicated.

I'll note that kcp.NewConn2 requires a net.PacketConn, even though all it should need is a net.Conn (you are reading and writing messages for a single connection). The code hacks around with a dummy address.

/cc @wkrp

[fortuna]

Thanks, that's quite helpful. I'll take a deeper look.

One thing I realized is that the client shouldn't need a PacketConn. Unlike the server, which may receive multiple clients on the same socket, the client is writing to a single destination, the net.Conn interface is more appropriate. That would allow us to get rid of the RemoteMap and simplify QueuePacketConn (QueueNetConn). Or perhaps I'm missing something?

KCP seems to take a net.PacketConn though. I wish there was a client API that takes the net.Conn.

[fortuna]

Ah, nevermind, I think I misunderstood KCP. Even on the client side, the socket might handle multiple sessions simultaneously, which requires dispatching logic. Also, the client must verify the session ID of every incoming packet to ensure it matches the expected session.

[wkrp]

You might also compare to slipstream, which uses QUIC as the turbo tunnel layer instead of KCP+smuc, and is said to have better performance, partly because of smaller (encapsulated) packet headers, and partly because of more advanced congestion management. slipstream doesn't have DoH and DoT modes, but those aren't relevant for the current Iran situation anyway.

[fortuna]

More on sessions.

A client session does, indeed, take over the socket, because of the readLoop. You can't use the socket for any other session because of that. As a result, there's no dispatching to sessions needed. We should be able to use net.Conn for client sessions.

Server sessions are a bit different. In kcp.newUDPSession, the logic skips the readLoop if it's a server session:
https://github.com/xtaci/kcp-go/blob/b38e76e5b00b2e1d971206a0902c29d7dab7611e/sess.go#L252-L257

if sess.l == nil { // it's a client connection
    go sess.readLoop()
    atomic.AddUint64(&DefaultSnmp.ActiveOpens, 1)
} else {
    atomic.AddUint64(&DefaultSnmp.PassiveOpens, 1)
}

This allows the server socket to be used by multiple sessions. Instead of passing packets to a single session, the Listener acts as a central dispatcher.

// readloop.go (Listener version)
func (l *Listener) defaultMonitor() {
    buf := make([]byte, mtuLimit)
    for {
        n, from, err := l.conn.ReadFrom(buf)
        // ... error handling ...

        // Dispatch packet based on source address
        l.packetInput(buf[:n], from)
    }
}

Inside l.packetInput, the Listener looks up the session based on the remote address:

// sess.go
func (l *Listener) packetInput(data []byte, addr net.Addr) {
    // ... decryption ...

    // 1. Look up session by "IP:Port" string
    l.sessionLock.RLock()
    s, exist := l.sessions[addr.String()]
    l.sessionLock.RUnlock()

    // 2. If session exists, feed data to it
    if exist {
        s.kcpInput(data)
        return
    }

    // 3. If NO session exists, it's a new client -> Create new session
    // ... (Handshake/Accept logic) ...
    s = newUDPSession(..., l, ...) // Note: 'l' is passed as parent
    l.sessions[addr.String()] = s
}

/cc @xtaci

[wkrp]

Inside l.packetInput, the Listener looks up the session based on the remote address:

And generally, in a turbo tunnel design, the "remote address" is not a UDP IP:Port, but an abstract identifier that is decoupled from the network layer. In dnstt, this identifier is ClientID, which is just an 8-byte random string:

// ClientID is an abstract identifier that binds together all the communications
// belonging to a single client session, even though those communications may
// arrive from multiple IP addresses or over multiple lower-level connections.
// It plays the same role that an (IP address, port number) tuple plays in a
// net.UDPConn: it's the return address pertaining to a long-lived abstract
// client session. The client attaches its ClientID to each of its
// communications, enabling the server to disambiguate requests among its many
// clients. ClientID implements the net.Addr interface.
type ClientID [8]byte

dnstt-server uses the ClientID, not the remote IP:port, to distinguish sessions. (Otherwise, consider what would happen if multiple clients were using the same intermediate resolver. The network-layer IP:port that dnstt-server sees is that of the intermediate resolver, not of the client.) The ClientID is like the connection ID in QUIC.

That's why ClientID implements the net.Addr interface, so it can be used everywhere one might normally use a UDP IP:port in KCP. Here's where in the server the ClientID is decoded from the DNS payload, and gets fed to KCP using ttConn.QueueIncoming. (At this point the ClientID just becomes a generic net.Addr as far as KCP is concerned, and only its String() value matters.) The remote addr from the ReadFrom is used only to know where to send the DNS response; for distinguishing sessions we ignore addr and instead use the ClientID from the DNS payload.

More on ClientID:

https://www.bamsoftware.com/papers/turbotunnel/#sec:implementing

The encapsulation must also associate with each packet a session identifier, a unique value that enables the recipient to distinguish packets belonging to different sessions, analogous to the four-tuple in TCP. The session identifier, being decoupled from any network address, enables roaming by the client, as in Mosh [27 §2.2] and WireGuard [5 §II-A]. The details of roaming depend on the obfuscation, but generally, a server receiving upstream packets tagged with a certain session identifier from a particular network address assumes that downstream packets for that session may be sent to that same network address.

https://www.bamsoftware.com/papers/turbotunnel/example/#sessionid.go

SessionID defines the format of a session identifier. Here, it is just a 64-bit random string. The session identifier should be long enough to prevent guessing and random collisions. SessionID implements the net.Addr interface.

[fortuna]

KCP Analysis

kcp-go is production-grade Go implementation of the KCP protocol (A Fast and Reliable ARQ Protocol).

KCP is a fast and reliable ARQ (Automatic Repeat reQuest) protocol. It serves as a drop-in replacement for TCP, providing smooth, resilient, ordered, error-checked, and anonymous stream delivery over UDP packets.
Unlike TCP, which is optimized for maximum bandwidth, KCP is designed for minimum latency. It achieves this by being more aggressive with retransmissions and avoiding the "slow-start" and congestion control behaviors of TCP that can cause latency spikes.

Key Features:

• Reliable ARQ: Automatic Repeat reQuest for reliable delivery.
• FEC (Forward Error Correction): Uses Reed-Solomon codes to recover lost packets without retransmission, reducing latency on lossy links.
• Encryption: Built-in packet-level encryption (AES, TEA, Salsa20, etc.) ensuring security and anonymity.
• Performance: Optimized for high concurrency (>5K connections) and low memory footprint.
• Compatibility: Implements net.Conn and net.Listener interfaces.

Architecture

The project follows a layered architecture:

1. Session Layer (sess.go): Manages the connection state, dealing with the KCP protocol logic, encryption, and FEC. It exposes the standard Go net.Conn interface.
2. KCP Core (kcp.go): The core protocol implementation, handling window management, retransmission, and flow control.
3. Packet Processing:
   • FEC (fec.go): Handles encoding and decoding of redundancy packets.
   • Encryption (crypt.go): Handles packet encryption and decryption.
4. Low-Level I/O:
   • Platform Specifics (*_linux.go, *_generic.go): Optimizations like sendmmsg and recvmmsg for Linux.
   • Read Loop (readloop.go): Efficient packet reading loops.

The Life of a Connection

Because KCP is implemented over UDP, it doesn't have the same strict state machine as TCP. Instead, kcp-go manages "sessions" to simulate a connection-oriented experience.

1. Establishment (The "Handshake")

KCP does not use a traditional 3-way handshake (SYN, SYN-ACK, ACK).

• Conversation ID (conv): Every session is identified by a 32-bit integer called conv. Both peers must agree on this ID for communication to happen.
• Client Side: When Dial() is called, the client generates a random conv ID and starts sending packets to the server.
• Server Side: The server listens for incoming UDP packets.
  • If a packet arrives with a conv ID that matches an existing session, it's routed there.
  • If the conv ID is unknown, the server treats it as a new connection request, creates a new session object, and returns it via Accept().

2. Data Transmission

Once the session is established, data is exchanged via KCP segments.

• Segments: Every UDP packet contains one or more KCP segments. Each segment has a header containing the conv, sn (sequence number), una (unacknowledged sequence number), and cmd (command).
• Reliability:
  • Acknowledgements (ACKs): The receiver sends ACKs to inform the sender that segments arrived.
  • Retransmission: If a segment isn't ACKed within the RTO (Retransmission Time Out), the sender resends it.
  • Fast Retransmit: If the sender receives multiple ACKs for a later segment, it infers a gap and resends the missing segment immediately without waiting for a timeout.
• Forward Error Correction (FEC): If enabled, kcp-go sends redundant "parity" packets. If a data packet is lost, the receiver can often reconstruct it using the parity packets, avoiding the latency penalty of a retransmission.

3. Packet Reception Pipeline

When a client receives a packet, it passes through several layers before reaching the application:

1. Reception & Filtering: A dedicated goroutine (readLoop) pulls raw data from the UDP socket and verifies the sender's address.
2. Security & Integrity: The packet is decrypted (if encryption is enabled) and its CRC32 checksum is verified.
3. Demultiplexing:
   • OOB: Out-of-band data is diverted to registered callbacks.
   • FEC: Reed-Solomon decoding attempts to recover any lost packets using parity shards.
4. ARQ Logic (kcp.Input):
   • ACK Processing: Processes acknowledgements and updates the RTT estimator.
   • Ordering: Data segments are placed into a min-heap (Receive Buffer) to ensure they are reassembled in the correct sequence.
5. Delivery: Once segments are in order, they are moved to the receive queue, and any blocked Read() calls are woken up.

4. Maintenance

• Keepalives: KCP does not have a built-in heartbeat. Applications using kcp-go typically use a multiplexer like smux or implement their own heartbeats to detect if the remote peer is still alive.
• Updating: A background loop (the "updater") periodically calls kcp.flush() to handle timing-related tasks like retransmission checks and sending pending ACKs.

5. Termination

• Graceful Close: When Close() is called, kcp-go attempts to flush any pending data in the transmission buffer.
• Abrupt Termination: Since there is no FIN or RST packet in the base KCP protocol, a peer only knows a connection is gone if it times out (using application-level heartbeats or SetDeadline) or if the underlying UDP socket is closed.

The Conversation ID (conv)

The conv ID is a critical 32-bit identifier for session integrity.

• Generation: In kcp-go, the client typically generates a cryptographically random conv ID upon calling Dial().
• Strict Validation: Every incoming packet is checked. If the conv ID in the KCP header does not match the session's ID, the packet is dropped immediately.
• Immutability: Once a session is established, the conv ID never changes.
• Security: When encryption is active, the conv ID is encrypted along with the KCP header, making it invisible to eavesdroppers and protecting against spoofing.

[wkrp]

• FEC (Forward Error Correction): Uses Reed-Solomon codes to recover lost packets without retransmission, reducing latency on lossy links.

• Encryption: Built-in packet-level encryption (AES, TEA, Salsa20, etc.) ensuring security and anonymity.

The dnstt implementation disables the FEC and encryption features (sets block to nil and dataShards and parityShards to 0 in kcp.NewConn2 (client) and kcp.ServeConn (server)).

[fortuna]

For support in the Outline Client, I propose this plan:

• Add a dnstt StreamEndpoint to a x/dnstt package. Ideally inject the needed dns.Resolver.
• In the Outline Client config module, register with streamEndpoints.RegisterSubParser("dnstt", parser)
• Add some logic to only enable it for Beta builds. This is a must have because we have some undiagnosed crash in main and cannot compromise it.
• Release to our Beta channel and let people know

/cc @ohnorobo