A guide to UDP (User Datagram Protocol)

The User Datagram Protocol is like Hans Christian Andersen’s “Ugly Duckling”. After decades of being overlooked and ridiculed, this simple protocol suddenly attracted admirers as the transport protocol for the new, glamorous multimedia applications that were made possible by broadband speeds. Today, any application that needs to deliver data quickly chooses UDP over the previously dominant TCP (Transmission Control Protocol).

UDP History

UDP has existed for almost as long as the internet. The internet came into being in May 1974 when the Institute of Electrical and Electronics Engineers published “A Program for Packet Network Intercommunication” by Vint Cerf and Bob Khan. The concept needed to be developed and both Khan and Cerf continued to refine their ideas while working for the US government’s Defense Advanced Research Projects Agency, which is also known as DARPA. John Postel got involved and suggested splitting out the single structure proposed in Cerf and Khan’s original idea. This created a layered concept. The original Transmission Control Program contained in the 1974 outline was split into the Transmission Control Protocol at a higher layer and the Internet Protocol at a lower layer (hence TCP/IP).

Postel’s modular approach made sense when the team started to think about implementing the theory. There was a clear division of labor between what became known as the Transport Layer, which is the location of the Transmission Control Protocol, and the Internet Layer, where the Internet Protocol resides. However, Cerf and Khan envisaged the need for a fast track option. They drew up a diagram of how data would be prepared for transmission by being passed from one layer to another. The processing tasks were represented as a straight vertical line, descending through their new stack diagram showing the progression from application to TCP and on to IP.

When it came to drawing in the fast track, they didn’t want to have to draw a curved detour line that avoided passing through TCP. Instead, they drew an oblong shape that represented the Internet Layer a little wider than the block that represented the Transport Layer. With this visual adjustment, both the regular route and the fast track route could descend through the stack as parallel lines. However, this trick left a gap that Postel felt needed to be filled. This is why the User Datagram Protocol was invented. It was just there to make the protocol stack diagram look balanced.

The benefits of TCP

The Transmission Control Protocol provides essential services to data in transit. It ensures that all packets in a stream actually arrive and it checks that they arrive in order. These control procedures that ensure an orderly transfer are not possible without a measure of coordination between the two sides. So, TCP first establishes an agreement between the two devices that intend to exchange data. This agreement is called a session. It is also the very definition of a “connection.” UDP has no session establishment procedures and so it is termed “connectionless.”

The session gives both sides of the connection a reference number that they can tag onto their administrative exchanges. The session also enables the concept of ports to be introduced. The session ID is actually a combination of identifiers contained in the TCP header. With this ID, the designers of TCP procedures were able to come up with the idea of a “socket.” Port numbers are also allocated to UDP, however, that protocol can only use the destination IP and port numbers as a unique identifier. An identifier derived from that combination would block out all other processes trying to access the same port, even though they were running on different computers, so UDP was made a delivery-only system, with no procedures to enable a two-way dialog.

The TCP connection concepts all developed into a very sophisticated universal method to ensure that data passing between computers didn’t get mingled up or garbled. The socket enabled multiple connections to be opened between the same two computers simultaneously. That idea created the possibility of having more than one channel operating to pass data. This is a frequently-used procedure of which many early network applications took advantage. The File Transfer Protocol, for example, uses two channels: one to pass the data and a separate channel for administrative communications. The different port numbers for data and control channels create two distinct sockets.

The unique ID for each session meant that TCP added value to the communication between two computers. In the late ‘70s and early ‘80s only large organizations and academic institutions had computers and networks. So, it was highly likely that two organizations needed their big mainframes to connect to each other simultaneously for different purposes. While a professor was sending a file to a colleague at another university, a researcher might also want to open a Telnet session to the computer at the same remote university. Thanks to TCP, two computers could maintain several connections simultaneously and each of those sessions could operate several channels at the same time. Those concurrent connections would not be possible if communications were only governed by the Internet Protocol with its allocation of one IP address per computer. UDP, without any session mechanism, was incapable of managing applications that required a contacted computer to send a response.

Data Security

The brilliant constructs of TCP made connections between networks possible and the internet started to expand beyond Academia to the business world. The creation of the World Wide Web, which became public in 1991 was only possible because of the ease with which the web page carrying Hypertext Transfer Protocol (HTTP) could sit on top of TCP.

The academics and technicians that put the internet together and then developed the publicly accessible World Wide Web were blue skies thinkers. They were excited by the technology and its possibilities to speed up research and improve interaction between people all over the world. They failed to account for the fact that their wonderful invention was a gift to thieves, con artists, and urban terrorists. Neither the internet nor the World Wide Web had any security at all.

It took the consumer-led Netscape Corporation to spot this problem. Netscape produced the world’s leading web browser and gave it away for free to encourage the uptake of the internet among the general public. The plan worked and information exchanges and contact channels spread out, encouraging more members of the public to sign up for internet services. However, the lack of security presented a barrier to the commercialization of the Web. Without the ability to entice people to pay for online services, there was no incentive for businesses to invest in the development of new applications, websites, or online services.

The major barrier to collecting payments over the Web was its lack of security. A few headlines about data theft on internet transmissions shut down the possibility of making the internet commercially viable. However, Netscape came up with HTTPS – a secure version of HTTP that protected transmission. The ideal location in the TCP/IP stack for these security procedures was during the session establishment processes of TCP. So, TCP became even more essential to the operations of the internet and it seemed even more likely that UDP would never be used.

UDP takes off

Despite being in existence since 1980, UDP was completely overlooked until broadband internet services became available at the beginning of this century. The User Datagram Protocol was largely ignored while web and other internet applications expanded on the functionality of TCP.

However, the ability to have voice conversations and video conferencing over the internet has always appealed to businesses. These applications did exist before broadband, but only for use on faster private networks. With the technology of passing sound and video over networks established, the faster speeds of broadband brought the possibility of making those applications available to the general public became a feasible idea. However, the speeds available over the internet were not quite good enough.

The immediate solution to squeezing just enough extra speed out of the internet was to ditch all the administrative procedures of TCP and turn to the almost forgotten UDP.

The problems with TCP

Interactive applications would rather deal with some of the problems encountered during transmission themselves. One of the main features of TCP that these applications really do not want is buffering.

TCP makes sure that packets arrive in sequence. If a packet is missing from the stream, the receiving TCP implementation will send a request to the sending TCP program to resend that specific packet. In the meantime, that packet might arrive late. TCP uses a sliding frame system to process arriving packets and if a segment is late or lost, that slide gets jammed. The temporary storage of a number of frames in memory is what is known as buffering. TCP waits until it can fill the empty slot with the packet that bears the missing sequence number. In the case of internet telephony, such an action would cause the line to go silent. In video streaming, the wait for a missing packet would make the video player freeze.

Interactive applications don’t have procedures to work around TCP buffering. The principal behind stack layers is that higher layers ask for a service and leave it to the lower layer to provide it. There is no “get on with it” signal that an application can send to the Transport Layer.

If a packet is lost in a digital telephone conversation, the callers will experience a short silence, but the application on both sides will just move on and continue sending and receiving the following packets. By the time a missing packet could be recovered, the interactive conversation would already have moved on so there is no point in trying to inject it back into the stream; it is just better to write off the loss and carry on. Similarly, a lost packet would just mean a short skip in a live video stream and viewers would much rather that the video kept moving forward than holding up the plot for one millisecond of the frames.

You have probably seen a video player pause and overlay the message “buffering” over the picture. There is usually also a counter that shows the percentage of buffering that has been completed. This buffering occurs if the connection’s transfer speed is slower than the frame rate of the video playback. The crucial point about that message, however, is that it shows that the buffering is being managed by the player and not by the transport protocol.

Partnering Protocols

Although interactive application didn’t want the delays caused by TCP, they did want some of that protocol’s functionality. They wanted more than UDP could provide. So other protocols were invented to fill in parts of TCP’s capabilities.

The Session Initiation Protocol

The Session Initiation Protocol (SIP) was invented for Voice over IP (VoIP) applications. Internet telephony didn’t want the buffering of TCP, but they did need to emulate the traditional call establishment procedures of telephones – dial, ring, busy, pick up, and end call. However, SIP doesn’t manage the entire session, it just takes care of the connection creation and teardown functions of TCP. Every call that runs over the internet employs SIP. So much so that “SIP” has almost become an interchangeable term with “VoIP.”

Running voice traffic over high-speed digital connections in bulk is known as “SIP trunking.” Switching a call from the internet to a regular landline telephone is called “SIP termination.” The digital telephony industry uses SIP to identify its technology, but the very foundation of all of their activities is UDP.

The Real-Time Transport Protocol

Despite the decision that TCP was too much of an overhead on interactive traffic and should be ditched, communication engineers kept returning to the facilities provided by TCP and they wished that they could have them with UDP. The Real-Time Transport Protocol (RTP) makes up for a lot of the shortfall in functionality experienced when using UDP.

A key feature of these add-on protocols that make UDP relevant to media streaming is that they allow some of the processes traditionally managed by TCP to be pushed up to the application. RTP handles some of the traffic management functions of TCP, but not all of them.

RTP is capable of reordering out of sequence packets and noting lost packets. However, the sequencing function does not need to be implemented and is impossible to implement without buffering at the Transport Layer.

The RTP Control Protocol

RTP always partners with RTCP, which is the RTP Control Protocol. RTPC emulates some of the session management functions of TCP, except the guiding principle of the protocol is to not intrude in the stream and not slow down media transmission; so its activities are infrequent. The protocol will gather performance data, including packet loss, and transfer rate information. The receiving player can use this information to decide whether to switch to a lower resolution of video or a different video coding standard.

If you use a video and audio application it is almost certain that both RTP and RTCP are involved. There is an “interleaving” option in the definition of RTSP (see below) which would move RTP transmissions onto TCP. However, this is an unusual proposal which has never been implemented beyond the lab. Without that specification, all RTP and RTCP activities are carried by UDP.

The Real Time Streaming Protocol

The Real Time Streaming Protocol (RTSP) is almost always involved in video and audio playback or recording applications. This protocol provides control buttons on your player and recorder. These are Pause, Record/Play, Fast Forward, and Rewind. Curiously, although RTSP can run over UDP, it is usually transported over TCP, even though it is partnering with a UDP-supported video or audio stream.

UDP-only applications

A number of lightweight network supporting applications use UDP without any other protocols that make up a simulation of TCP functions. These functions are almost exclusively intended only for use on private networks because they do not include any authentication procedures or transmission encryption.

If you manage a network, you will be familiar with the Network Time Protocol (NTP), the Domain Name System (DNS), the Dynamic Host Configuration Protocol (DHCP), and the Trivial File Transfer Protocol (TFTP). All of these administration services run over UDP. Beyond these private network applications, it is very difficult to find any application that runs only over UDP.

UDP vs TCP

A comparison of the UDP header structure and the TCP header structure shows you the limitations of UDP.

The UDP header has only four fields. Of those four, the Source Port field is optional and may be left blank. In IPv4, the Checksum field is also optional, although it is mandatory for IPv6 implementations. This means that in the case of IPv4 transmissions, the UDP header need only have two pieces of information in it.

The TCP header is able to carry a great deal more information.

As you can see from the illustration, the TCP packet header has a series of nine flags that adapt the meaning of the header. It event has an “urgent” field. This gives the TCP system a lot more flexibility than UDP and it shows that a lot more time was invested in the procedures for TCP and the structure of its packet header than was spent on developing UDP.

The fact that the TCP header has to include the source port makes it possible to create a more unique socket, creating a session ID from the source and destination IP addresses and the source and destination port numbers. With UDP, as it doesn’t have procedures to create a session, each message is treated as a completed task, and the protocol doesn’t attempt to string packets together. Therefore, applications that use USP have to manage that continuity themselves.

Security for UDP

The connection-oriented methods of TCP make security much easier to implement in that protocol in UDP. However, there are encryption standards available for UDP. The main option that directly aims at security UDP is the Datagram Transport Layer Security protocol or DTLS.

Fortunately, DTLS is available in a number of free, open source libraries, so you do not need to comb through the protocol definition and write your open program in order to implement it. OpenSSL, which is a library of open source code, is the most common source for an implementation of Transport Layer Security, which is the most widely-implemented security system for TCP. This library also includes a DTLS implementation, so you should be able to encounter secure UDP options in the same applications that offer secure TCP connections.

Another option for UDP users is to rely on a security system that was designed to work at the Internet Layer. This is IPSec, or Internet Protocol Security. As IPSec operates below the Transport Layer, it is not able to work with ports and so the fact that UDP isn’t able to sustain a session doesn’t matter when IPSec is engaged – IP Layer protocols can’t create sessions either. As a lower-layer system, IPSec is able to support any Transport Layer protocol, including UDP.

IPSec includes authentication methods and also encrypts packets to protect them from wiretapping snoopers. IT offers just as much security as the popular TLS, but is less widely implemented. IPSec uses the Internet Key Exchange (IKEv2) system to set up authentication, so quite often, IPSec is billed as IKEv2. The IKEv2 methodology uses Diffie-Hellman key exchange procedures, which is exactly the same system that TLS uses for the HTTPS secure web page session methodology.

Kerberos and the Kerberized Internet Negotiation of Keys (KINK) are two elements of a security system that is usually called Kerberos. The Kerberos session establishment procedures use a system of “tickets” which is similar to the TLS method of using “certificates.” At the very bottom of the stack, Kerberos is underpinned by IPSec. The eponymous Kerberos layer sits on top of UDP and employs UDP sockets to facilitate communication. So this is a UDP-friendly security system. An exciting facility of Kerberos is that it allows you the option of employing AES encryption to protect your UDP transfers. AES is probably the most secure cipher in common usage today and it is the recommended security method for the world’s best VPN privacy protection systems.

Despite the apparent difficulties of negotiating encryption keys in an environment that does not provide any connection management, UDP does offer security options. So, when you implement a UDP-based application, don’t abandon the task of securing your transmissions.

The Future of UDP

Pure UDP-based applications that don’t involve side protocols to mimic TCP are rare and they are likely to get even rarer. The lightweight network utilities that use UDP thrive on secure local networks. However, as security threats from new zero-day attacks mount every week, the concept of having insecure protocols managing the crucial services of configuration management and addressing seem to be foolishly complacent.

As networks move their service over to the cloud, the services of UDP-based TFTP and DHCP will start to be replaced by more secure alternatives. The easy solution of surfing applications over HTTPS to give them security without extra programming effort slants the future towards TCP, which carries HTTPS and strips opportunities away from the UDP list of competences.

The UDP niche of supporting media transmissions is likely to endure. There have already been many rival transport systems proposed for the support of interactive applications, but none of them have knocked UDP from its position as the first choice for VoIP and video streaming. This list of rivals includes:

The Reliable User Datagram Protocol (RUDP), which has Cisco and Microsoft implementations.

The Stream Control Transmission Protocol (SCTP), which was unsuccessfully proposed as a replacement for the UDP/RTP/RTCP combo, but never quite got off the ground.

This ugly duckling called UDP was discovered to be a swan, thanks to the magical transformative powers of broadband and interactive applications. This revitalized star will continue to glide effortlessly across the waters of the internet.

What methods of transmission do you use? Do you see the UDP ugly duckling as a swan? Write about your experiences in the Comments section below.

User Datagram Protocol FAQs

Related:
Ultimate guide to TCP/IP
What is TCPdump?
SolarWinds TFTP server review
TFTPD32 TFTP server review

Images:
Header of UDP by Devarshi at English Wikibooks Licensed under CC BY-SA 2.5
TCP packet layout with bit scale by Quliyevferman via Wikimedia Commons. Licensed under CC BY-SA 4.0