Cloud computing has brought many advantages to the software industry. But it has also created special new challenges — chief among which is logging.
In the cloud, you have to contend with large volumes of log data. Your log data may be extra-complex because different cloud providers or services use different formats. And your logs may be spread across a distributed architecture.
How can we deal with these challenges? This post discusses these cloud logging challenges and the solutions to address them.
Complexity
Before the era of cloud computing, client-server applications were needed to run web applications. A frontend was needed for the application to run on browser(s), and a backend stored the data in a database (server). If something went wrong, you could find the culprit in the system logs.
Now, there is no simple frontend and backend. Applications have to run on different operating systems, devices and browsers.
Managing this is tough. Virtualization (using virtual machines) was effective in the beginning, but became less effective when companies grew larger online.
Dockerization was the answer to the problem of complexity in the DevOps pipeline: Containers are used to store small manageable chunks, breaking down complex tasks. They also help build microservices-based applications.
Volume
Containers and microservices are great, but there is a catch when it comes to logging. A modern app consists of several microservices, with each microservice being driven by multiple containers. All this results in an explosion in the number of logs. Not only do the microservices have logs, but also the containers, the browsers, the operating systems and the device, too.
Analyzing all of these logs manually is difficult, if not impossible. Automation is therefore a must.
Distribution
As I said in the previous paragraph, there are a lot of logs generated when using an app.
Making things even harder, these logs are not stored in a single space — They’re stored separately with every element. And even this has exceptions — You can’t log in Kubernetes natively, so external log storage has to be enabled. This fragmentation makes the logging of a cloud application very “foggy” indeed.
Privacy
With the implementation of the European General Data Protection Regulation (GDPR) in May 2018, companies working in the EU have to comply with the new European privacy law.
This also involves the logging of data from EU citizens. When choosing a logging solution, you have to test for vendor compliance with the GDPR.
Addressing Cloud Logging Challenges
Data logging has become more difficult, but the software industry has not neglected the issue.
Open source solutions like the ELK stack (Elasticsearch, LogStack and Kibana) have addressed many of the data logging challenges in the cloud, but the ELK stack has one weak point — Each time you start an instance, you start a node.
The more a node executes, the more logs it will produce, eventually taking up a lot of digital space in the node. This will decrease performance of the app. Cloud logging solutions address this problem by visualizing performance indicators such as application availability, performance, resource usage, and errors. You can proactively see performance issues at a glance in a dashboard and take appropriate measures to tackle them.
Conclusion
The challenges of using a solid logging solution in the age of cloud computing are numerous. Fortunately, several solutions exist.
