One aspect critical to a development organization’s application quality is the implementation of a high-functioning incident response strategy. That being said, achieving efficiency in the realm of incident response hinges on an organization’s ability to effectively use all information at their disposal. As anyone who has ever had the pleasure of troubleshooting application and system problems will tell you, log files are one source of information that can’t be overlooked.
So, how can an organization evaluate the information in log files in a manner that limits the impact of the “noise” they surely contain while allowing the team to glean insights that help resolve bugs and improve incident response? This answer is log monitoring, of course. Read on for an overview of log monitoring and the ways in which it can provide value to an incident response strategy.
What is log monitoring?
Log monitoring is a process by which log events are evaluated as they’re being recorded. In the modern development world, log monitoring is almost always achieved with the help of log monitoring software such as Splunk.
Log monitoring software can be configured to ingest application and system logs where these log files are then parsed in real-time. Typically, this is accompanied by a user-friendly interface where log events are easily searchable in the form of filtering with advanced filtration options – simplifying the process for analysts looking to glean insights from log data. Additionally, quality log monitoring software provides functionality for the configuration of alerts for particular error codes, log messages, etc. These notifications provide value for a variety of reasons — including several notable benefits related to incident response (more on this in the next section).
Log monitoring and incident response
Now that we know what log monitoring is, one question still remains. What impact can log monitoring have on the effectiveness of an incident response strategy? The answer is, “A rather big impact.” At least, it can when a development organization takes advantage of everything that a log monitoring solution has to offer. Consider the following log monitoring benefits made available to an organization’s incident response strategy:
Increase the speed of issue recognition with alerting
As mentioned above, log monitoring allows an organization to configure alerts that notify incident response personnel – in real-time – of a particular problem with a system or application. This functionality and speed provides incident response teams with several key benefits worth mentioning. Faster time to acknowledgment also means faster time to remediation, meaning you’ve improved your application’s overall quality and reliability for customers.
Timely analysis of possible application or system issues
Common sense dictates that an application or system issue can’t be resolved before support and development personnel learn of its existence. Efficient monitoring and alerting is the only manner in which these teams can be quickly notified of such issues they otherwise might not know about. With the real-time nature of these alerts, support personnel or development staff can begin analysis at the earliest possible moment in time. In some cases, this will even mean starting analysis prior to any customer reports of the issue. This serves to limit the end-user impact of a particular application or system due to the simple fact that faster acknowledgment of an issue means faster time to resolution.
Customization of the traditional path
Consider the traditional path to resolving bugs within an application. End-users discover an issue, contact support and the game begins – finding the people who have insight into the problem.
With an effective alert configuration, this process is streamlined. Alert configuration allows for customization and ensures the right personnel are notified of application or system issues for which they’re qualified to resolve. For instance, let’s imagine part of your response strategy is to designate two developers as on-call support each week for an application they helped build. In this case, your log monitoring tool could be configured to notify these two particular devs of any issues it detects for that particular application. This ensures the personnel with the expertise needed to resolve the problem are notified of the incident immediately, eliminating the step of finding the right people for the job, thus reducing the time it takes to resolve the issue.
Automatic log monitoring and IT ticketing
Another benefit comes from having a log monitoring solution automate the process for opening tickets when an incident occurs. The criticality applied to the ticket can reflect the severity of the error message and an assignment can be made to the developer or support person currently on-call. Automating such steps in the incident response process helps to speed up the administrative aspects of incident response, improving the efficiency of the strategy as a whole.
Simplify analysis with log monitoring
It’s no question that the most obvious benefits of log monitoring lie in the ability to learn of application or system issues in real-time via alerts. With that said, modern log monitoring solutions like the centralized log management platform from Splunk provide additional value in the realm of incident analysis. Easy-to-use UI components are built to simplify the process for searching and filtering log events. This eliminates the practice of scrolling through thousands of lines of a log file in a text editor, futilely attempting to gain insight into the root cause of the issue.
An example:
If a problem with application latency is evident with timeouts being logged every Monday at 11 AM, maybe another process running at the same time is to blame. In another case, maybe a specific page is loading with exceptional slowness. The culprit could be a long-running database query indicating a necessary refactor in order to improve performance.
These issues are much easier to locate and resolve with the help of software that alerts engineers to a problem, provides capabilities that help development staff easily identify such trends and functionality to help responders take action.
Making log monitoring actionable for on-call personnel
At a high level, log monitoring helps to speed up the process of identifying specific exceptions, when these exceptions occur and the frequency at which they occur. Additionally, it provides developers and support personnel with a greater level of visibility into the systems and applications being monitored. Information is readily available in a way that’s easy to consume (filterable, searchable, etc.). And, team members are notified of potential issues as early as possible.
The immediate nature of alerting and the increased visibility provided by modern log monitoring solutions helps on-call engineers and support personnel see more of the big picture as early as possible. The result of this increased visibility and collaboration is a decrease in incident response times and the implementation of a faster, more informed and complete resolution.
