Even for experienced Kubernetes users, Kubernetes Networking can be an intimidating topic. In this article, we’re going to take a deep dive into the most popular networking solutions available to Kubernetes users for their networks. Before we dig into the specifics of each of those networking solutions, we’ll take a cursory view of Kubernetes Networking and the Container Network Interface (CNI) specification. We’ll talk about why networking is a complex problem and how the CNI allows the development of specialized solutions outside of the Kubernetes project.
Once we have a solid overview of Kubernetes Networking and the CNI specification, we’ll dissect some of the most popular networking solutions. We talk about what makes each solution unique, and cover networking aspects that the solution solves and some of the challenges. By the end, you’ll have a much better idea of what each framework offers and how to choose the right Kubernetes Networking framework for your project.
Kubernetes Networking and CNI
As we already mentioned, networking is a complex problem. Kubernetes bases its networking at the pod level rather than at the individual containers’ level with each pod. Kubernetes Networking requires that pods communicate with other pods and that nodes can communicate with pods without requiring Network Address Translation (NAT). Kubernetes networking supports service discovery, service exposure, high-availability, and network security.
Networking requirements vary greatly, depending on the needs of each project. Rather than address each of these needs in a single solution, Kubernetes abstracts the networking away from Kubernetes itself, allowing users to plug in their desired networking solution to their clusters. We’ll talk more about how that happens shortly.
These plugins must conform to the standards defined by the Container Network Interface or CNI specification. The CNI describes how to configure network interfaces within Linux containers, explicitly adding and removing containers from the network. The CNI is an adopted Cloud Native Computing Foundation (CNCF) project, and many container runtimes, including Kubernetes, have implemented this specification. If you’d like to learn more about the CNI specification, which runtimes use it, and third-party plugins that implement it, the CNI GitHub project is a great place to start.
Each plugin that implements the CNI specification tries to address different aspects of networking. Identifying and configuring the right plugin or combination of plugins to meet your project’s needs is vital. We’re going to take an in-depth look at four of the most popular plugins next, and explore their strengths and weaknesses.
Those plugins are:
- Flannel
- Calico
- Weave
- Cilium
Flannel
CoreOS created Flannel as one of the first CNI implementations. As such, it is one of the oldest and most mature CNI plugins available. It is also a great entry-level choice due to its simplicity and ease of use. Flannel provides access to basic networking and requires limited amounts of administration to set up and maintain.
Flannel sets up a comprehensive internal network that includes all nodes within the cluster. This topology is commonly known as a network fabric topology. The network fabric connects all nodes with all other nodes in the cluster.
In terms of the Open Systems Interconnection (OSI) networking model, Flannel provides networking at layer 3. Layer 3 is the network layer and supports the process of transferring packets around the network. Flannel’s default backend is Virtual Extensible LAN (VXLAN), although you can also configure it to use UDP and host-gw. Some experimental backends like AWS VPC, AliVPC, IPIP, and IPSec are available, but not officially supported at present.
One of the drawbacks of Flannel is its lack of advanced features, such as the ability to configure network policies and firewalls. You do have the option to use Flannel for networking and configuring another tool, such as Calico, for advanced features like adding network policies.
Calico
Represented by their mascot “Felix,” Calico is the creation of Tigera. Calico is an open-source project that works with many platforms, including Kubernetes. Calico is offered in a supported enterprise edition or for ease of entry, you might want to consider using a managed solution, such as Platform 9’s Managed Calico. A managed solution gives you access to Calico’s impressive performance while someone else manages configuration and maintenance.
The Calico project is hosted on GitHub and has extensive and thorough documentation. Calico has emerged as one of the most popular CNI plugins for Kubernetes users. The project has earned a reputation for being reliable, flexible, and supporting highly performant networks within Kubernetes clusters.
Like Flannel, Calico operates on layer 3 of the OSI model and uses the Border Gateway Protocol (BGP) to move network packets between nodes. Using BGP, Calico directs packets natively, without needing to wrap them in additional layers of encapsulation. This approach improves performance and simplifies troubleshooting network problems compared with more complex backends, like VXLAN.
Calico’s most valuable capability is that of supporting network policies. By defining and enforcing network policies, you can prescribe which pods can send and receive traffic and manage security within the network. While Calico is a well-used and capable network tool on its own, its policy management also allows it to pair well with systems like Flannel or Istio, a popular Kubernetes service mesh.
Weave
Weave or Weave Net is a full-featured CNI plugin created and supported by Weaveworks. Weave is available from the GitHub repository and the Weaveworks website. Like Calico, Weave is also available in a paid version with a support plan.
Weave creates a mesh overlay between all nodes within the cluster and uses this in combination with a routing component on each node to dynamically route traffic throughout the cluster. By default, Weave routes packets using the fast datapath method, which attempts to send traffic between nodes along the shortest path. The network is continually analyzing traffic flow and optimizing routes. A slower network method, known as sleeve packet forwarding, is the backup method if the fast datapath fails.
Weave includes creating and enforcing network policies and allows you to configure encryption for the entire network. If configured, Weave uses NaCl encryption and IPsec ESP encryption. Weave uses NaCl encryption for sleeve traffic and IPsec ESP encryption for fast datapath traffic.
Cilium
A relative newcomer to the land of CNI plugins is Cilium. Cilium and its observability tool, Hubble, take advantage of eBPF. eBPF is a newer technology that runs within the Linux kernel and enables the configuration and execution of sandbox programs. This new feature supports more dynamic toolsets, like Cilium, that can support more advanced networking and observability, among other features.
One of the advantages that Cilium offers over other CNI plugins is reduced overhead when managing large networks. While some CNI plugins rely on iptables on each host to manage network addressing, Cilium takes advantage of eBPF to handle this more efficiently and in a more performant manner. Efficient address lookup is critical as clusters scale to tens of thousands of nodes.
Cilium offers networking policies that operate at layers 3, 4, and 7 of the OSI networking model. This ability to apply policies at multiple layers affords more flexibility in how you manage ingress and egress traffic within your cluster. While still a relatively new CNI plugin, Cilium may be worth consideration, especially if you require fine-grained security controls or need to reduce lookup latency within enormous clusters.
Set up and operate a cluster for free
Selecting the Right Solution For Your Project
Depending on your needs, selecting the right CNI plugin to use in your cluster could be very simple, or a little more complicated. If your only requirement is for a basic networking solution, Flannel might be your best choice. While it lacks many advanced features like network policies and encryption, it’s light, fast, and consumes fewer resources than other CNI plugins.
If performance and security through network policies and encryption are paramount, you should consider Calico, Weave, or Cilium or a hybrid solution like Canal. Canal uses a combination of Calico and Flannel. Flannel provides basic networking and pairs well with Calico’s best-in-class network policies. Network policies are essential for maintaining a secure cluster, especially given the increased risk of cyberattacks. Calico’s documentation includes a must-read section on adopting a zero trust network model.
Cilium may offer advantages for large-scale deployments, and takes advantage of eBPF for improved observability and network management efficiencies. Cilium is still a young project, and in the benchmark tests referenced below, it does appear to be more resource-intensive.
Speaking of benchmarking, ITNEXT publishes an annual collection of benchmark results for CNI plugins. The results show similar performance for all of the frameworks we’ve discussed today, and several others. The study runs the benchmark tests on clusters with three nodes using the default configuration and measures performance, throughput, latency, and resource usage, among other metrics.
Fig. 1 Benchmark Result Summary from ITNEXT
The benchmarks tests do an excellent job of highlighting the most critical factor related to CNI plugins for your Kubernetes clusters. The benchmark study used the default configuration for each of the tools investigated. Fine-tuning your CNI plugin provides better results and can meet your particular cluster’s needs. In a previous blog post, Platform9’s Co-founder and VP of Product, Madhura Maskasky, discussed Achieving High Performance with Calico.
One of the best things about Kubernetes is the growing global community and the plethora of open-source projects, service providers, and managed platforms that support its growth. What that means for you is that you’re not alone. Many engineers have contended with these same questions and challenges. Besides a rich supply of advice and recommendations on the Internet, in forums, and on Slack channels, you have access to managed services like Platform9.
Managed Kubernetes services like Platform9 simplify the process of creating, managing, and maintaining your Kubernetes infrastructure. These providers also streamline the process of adopting and tuning CNI plugins to meet your specific requirements. Managed Calico is one such offering. Calico has earned a reputation of being one of the best-in-class CNI plugins. And when you combine that with Platform9’s Managed Kubernetes platform, you can’t go wrong.
