DNS Protocol Deep Dive

DNS message format, wire encoding, UDP vs TCP, EDNS(0), port 53, and message compression — understanding DNS at the packet level.

If you really want to understand DNS, you need to see it at the wire level. Not the abstracted dig output — the actual bytes flowing over the network. This chapter takes you into the protocol itself.

DNS Message Format

Every DNS query and response follows the same message structure, defined in RFC 1035 §4.1:

+------------------+
|      Header      |  12 bytes, always present
+------------------+
|     Question     |  The question(s) being asked
+------------------+
|      Answer      |  Answers to the question
+------------------+
|    Authority     |  NS records pointing toward authority
+------------------+
|    Additional    |  Extra helpful records (glue, etc.)
+------------------+

The Header (12 bytes)

The header is always 12 bytes, containing:

                                1  1  1  1  1  1
  0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                      ID                       |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    QDCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    ANCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    NSCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    ARCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Let’s decode each field:

Field Bits Meaning
ID 16 Transaction ID — matches query to response
QR 1 Query (0) or Response (1)
Opcode 4 Type: 0=Query, 1=IQuery, 2=Status, 4=Notify, 5=Update
AA 1 Authoritative Answer
TC 1 Truncation — response too large for UDP
RD 1 Recursion Desired — client wants recursive resolution
RA 1 Recursion Available — server supports recursion
Z 3 Reserved (must be zero)
RCODE 4 Response code: 0=No error, 3=NXDOMAIN, etc.
QDCOUNT 16 Number of questions
ANCOUNT 16 Number of answer records
NSCOUNT 16 Number of authority records
ARCOUNT 16 Number of additional records

The flags decoded:

When you see dig output like:

;; flags: qr rd ra; QUERY: 1, ANSWER: 1

That means:

  • qr = This is a response (QR=1)
  • rd = Recursion Desired was set in the query
  • ra = Recursion is Available from this server
  • Not AA = Not an authoritative answer (from cache)

The Question Section

Each question contains three fields:

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                                               |
/                     QNAME                     /  Variable length
|                                               |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                     QTYPE                     |  16 bits
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                     QCLASS                    |  16 bits
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
  • QNAME: The domain name being queried (encoded, see below)
  • QTYPE: Record type (1=A, 28=AAAA, 15=MX, etc.)
  • QCLASS: Usually 1 for IN (Internet)

Resource Records (Answer, Authority, Additional)

Each resource record follows this format:

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                                               |
/                      NAME                     /
|                                               |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                      TYPE                     |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                     CLASS                     |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                      TTL                      |
|                                               |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                   RDLENGTH                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
/                     RDATA                     /
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Wire Format: Domain Name Encoding

Domain names aren’t stored as plain strings. They’re encoded as a sequence of length-prefixed labels, terminated by a zero byte.

Example: www.example.com

03 77 77 77    ; 3, "www"
07 65 78 61 6d 70 6c 65    ; 7, "example"  
03 63 6f 6d    ; 3, "com"
00             ; end of name

Each label starts with its length (1 byte), followed by the characters. The root (empty label) is represented by a zero byte.

Total bytes: 1+3+1+7+1+3+1 = 17 bytes for www.example.com.

Why This Encoding?

The length-prefix design allows:

  • No need for escape characters (dots are just separators)
  • Efficient parsing (read length, read that many bytes)
  • Clear termination (zero byte = done)

It also enables compression (see below).

UDP vs TCP

DNS uses both transport protocols, with specific rules for each.

UDP: The Default

DNS originally runs over UDP port 53. Benefits:

  • Low overhead: No connection setup
  • Fast: Single round-trip for simple queries
  • Stateless: Server handles each query independently

The 512-Byte Limit

RFC 1035 specified a maximum UDP payload of 512 bytes. This was a safe size that would work across all networks without fragmentation issues.

If a response exceeds 512 bytes:

  1. Server sends truncated response with TC (Truncation) flag set
  2. Client retries the query over TCP
  3. Server sends full response

TCP: For Large Responses

DNS over TCP (also port 53) has no size limit. Used for:

  • Zone transfers (AXFR/IXFR)
  • Large responses (DNSSEC-signed records, many answers)
  • Any query when UDP response is truncated

RFC 7766 mandates that all DNS implementations must support TCP.

Modern UDP: Larger with EDNS

EDNS(0) allows UDP payloads larger than 512 bytes (typically 4096). This reduces TCP fallback for most queries.

EDNS(0): Extending DNS

EDNS (Extension Mechanisms for DNS) extends the protocol without breaking compatibility. Defined in RFC 6891.

The OPT Pseudo-Record

EDNS uses a special record type (OPT, type 41) in the Additional section:

; Pseudo-record (not stored in zones)
. 0 OPT [UDP payload size] [Extended RCODE] [Version] [Flags] [Options]

In dig output:

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096

What EDNS Enables

Feature Benefit
Larger UDP Up to 4096+ bytes without TCP
Extended RCODE More error codes beyond 4 bits
DNSSEC OK flag Client supports DNSSEC validation
COOKIE option Protection against spoofing
Client Subnet Geo-aware responses (ECS)
Padding Privacy enhancement

EDNS Negotiation

When a client sends a query with OPT:

  • It advertises its supported UDP buffer size
  • It indicates DNSSEC support (DO flag)
  • It may include options like Client Subnet

The server responds with its own OPT record, acknowledging capabilities.

If a server doesn’t understand EDNS, it may:

  • Ignore the OPT record (old behavior)
  • Return FORMERR (some broken implementations)

Port 53: Why It Matters

DNS has used port 53 since the beginning. This well-known port has implications:

Firewall Considerations

  • Outbound UDP/TCP 53: Required for clients to resolve DNS
  • Inbound UDP/TCP 53: Required for running authoritative or recursive servers
  • Many networks restrict outbound DNS to force use of their resolvers

Alternative Ports

Modern encrypted DNS uses different ports:

  • Port 853: DNS over TLS (DoT) — RFC 7858
  • Port 443: DNS over HTTPS (DoH) — RFC 8484

These ports are less likely to be blocked (443 is HTTPS traffic).

The Politics of DNS Ports

Using port 443 for DoH is controversial:

  • Privacy advocates: Love it — DNS traffic blends with HTTPS, hard to block
  • Network admins: Hate it — can’t monitor or filter DNS, breaks security policies
  • ISPs/governments: Concerned about losing visibility and control

This tension drives ongoing debate about encrypted DNS deployment.

Message Compression

DNS messages often repeat domain names. Compression reduces size using label pointers.

How Pointers Work

Instead of repeating a name, a pointer references an earlier occurrence:

Normal label:  03 77 77 77    ; length byte < 64
Pointer:       c0 0c          ; first two bits = 11, rest is offset

When the first byte has its two high bits set (binary 11xxxxxx), it’s a pointer. The remaining 14 bits give an offset into the message where the name continues.

Example

Query for www.example.com, response includes www.example.com and example.com:

Offset 0x0c:  03 www 07 example 03 com 00    ; www.example.com
...
Offset 0x30:  c0 10                           ; pointer to offset 0x10 (example.com)

The second name doesn’t repeat “example.com” — it points to where that sequence already appears.

Compression Boundaries

Pointers can only point backwards (to lower offsets). This ensures names can be parsed in a single pass without loops.

Compression is optional for senders but must be understood by receivers.

Security Note

Compression pointers have been exploited in attacks. Implementations must:

  • Detect and reject loops
  • Limit recursion depth
  • Validate that pointers stay within message bounds

Putting It Together: A Real Query

Let’s decode an actual DNS query for www.example.com:

Query (UDP, 33 bytes):

Header (12 bytes):
  ab cd     ; ID: 0xabcd
  01 00     ; Flags: QR=0 (query), RD=1 (recursion desired)
  00 01     ; QDCOUNT: 1 question
  00 00     ; ANCOUNT: 0
  00 00     ; NSCOUNT: 0
  00 00     ; ARCOUNT: 0

Question (21 bytes):
  03 77 77 77           ; "www" (length 3)
  07 65 78 61 6d 70 6c 65   ; "example" (length 7)
  03 63 6f 6d           ; "com" (length 3)
  00                    ; root label (terminator)
  00 01                 ; QTYPE: A (1)
  00 01                 ; QCLASS: IN (1)

Response might look like:

Response (49 bytes):

Header:
  ab cd     ; Same ID
  81 80     ; Flags: QR=1, RD=1, RA=1 (response, recursion happened)
  00 01     ; QDCOUNT: 1
  00 01     ; ANCOUNT: 1
  00 00 00 00   ; NSCOUNT, ARCOUNT: 0

Question (copied from query):
  [same as above]

Answer:
  c0 0c     ; Pointer to offset 12 (www.example.com)
  00 01     ; TYPE: A
  00 01     ; CLASS: IN
  00 00 01 2c   ; TTL: 300 seconds
  00 04     ; RDLENGTH: 4 bytes
  5d b8 d8 22   ; RDATA: 93.184.216.34

Debugging at the Wire Level

Tools for seeing raw DNS:

# Wireshark/tcpdump with DNS decode
sudo tcpdump -i any port 53 -vv

# Hex dump a DNS query
echo -n "example.com" | python3 -c "
import sys
name = sys.stdin.read()
labels = name.split('.')
for l in labels:
    sys.stdout.buffer.write(bytes([len(l)]) + l.encode())
sys.stdout.buffer.write(b'\x00')
" | xxd

# dig with full wire output
dig +qr +additional www.example.com

Understanding wire format helps debug:

  • Malformed responses
  • Truncation issues
  • Compression bugs
  • Protocol compliance problems

Key Takeaways

  • DNS messages have a fixed 12-byte header plus variable sections (Question, Answer, Authority, Additional)
  • Header flags indicate query/response, recursion, authority, truncation, and error codes
  • Domain names are length-prefixed labels terminated by zero
  • UDP is default with a 512-byte limit; TCP handles large responses and zone transfers
  • EDNS(0) extends DNS: larger UDP, DNSSEC support, additional options
  • Compression uses pointers to avoid repeating names, reducing message size
  • Port 53 is the standard; ports 853 (DoT) and 443 (DoH) provide encryption

This protocol-level understanding is what separates DNS users from DNS engineers. When something breaks at the packet level, you’ll know where to look.


Congratulations! You’ve completed Part 2: How DNS Works. You now understand DNS from fundamentals through protocol internals. Part 3 explores the domain ecosystem — how domains are named, registered, and managed. Part 4 dives into DNS security — DNSSEC, attacks, and defenses.