Upping your Syslog Game with your Network Monitoring Application

99 VIEWS

·

Did you know CA Spectrum has long been able to directly process Syslog messages sent as traps natively from devices?

The CA Spectrum network monitoring application has long been able to directly process Syslog messages sent as traps natively from devices. For Cisco devices the command to enable this functionality would be: snmp-server enable traps syslog. The challenge with sending all Syslog messages as traps is all the noise generated by the Syslog messages.

Additionally, CA Spectrum has supported ingestion of Syslog sourced events via log match traps from iAgent (SNMP Research CIAgent), CA SystemEDGE, and CA NSM. Given the current state of these agent technologies, there is a need for additional methods to send Syslog data into CA Spectrum, particularly since it is an important source of network events, in addition to SNMP traps and polling.

is a popular Syslog server that is also used with CA UIM Log Analytics. One of the interesting features of Rsyslog is the ability to forward messages as SNMP traps via the omsnmp module.

Note: Rsyslog could be running on a remote Linux machine while CA Spectrum is running on Windows or Linux/Unix. Additionally, Rsyslog could be running on the same Linux instance as CA Spectrum.

What this allows you to do is configure something like the following:

If you’re using an RPM based Linux distribution (Red Hat, CentOS, etc.), the two required packages are rsyslog and rsyslog-snmp.

Once those packages are loaded, you’ll need to configure RSyslog to receive Syslog messages from network devices and forward them as traps to the CA Spectrum network monitoring application. The configuration file you’ll need to edit is /etc/rsyslog.conf. Here are the relevant parts:

Input Modules

Rsyslog has the concept of input modules that allows Rsyslog to consume data from log files, UNIX sockets, TCP and UDP, and other sources. The typical Rsyslog configuration will have modules for UNIX socket, Kernel log, and perhaps Systemd Journal but for receiving Syslog data from network devices, the imudp module will be required:

MODULES ####

In addition to loading the module, we can specify parameters for operation. At minimum, the listening port number needs to be specified. In this configuration, we’re also going to specify a ruleset called “network” for use with the UDP input module:

Specifying a ruleset provides the opportunity to control how the incoming messages are to be processed, formatted, and ultimately forwarded. There is also a TCP input module and if you have devices that use TCP to send their Syslog data, you’ll want to enable that and specify the “network” ruleset like was done for UDP.

Output Modules

Similar to input modules, output modules allow Rsyslog to forward processed log data to log files on disk, TCP or UDP, databases, as SNMP traps, etc. The omfile (writing to disk) and omfwd (forward via TCP or UDP to something like Log Analytics) modules are built in but omsnmp (SNMP trap) needs to be specified to load:

Templates

Rsyslog makes use of templates to specify how forwarded/saved messages should be formatted. If Log Analytics is part of the configuration, there will be a template called “ls_json” that is outlined in the appropriate documentation (step 3). For SNMP traps to Spectrum, we can use the following:

The template above would result in a trap with the significant varbind that looks like this:

TS:2018-05-09T17:30:46.503221-04:00SFP=23:4 Src: 192.168.75.140 Tag=49: Msg: *May 9 21:30:45.493: %IOSXE-4-PLATFORM: R0/0: kernel: hrtimer: interrupt took 2603406 ns

In the template, there are references to properties. If needed, properties can be substituted. For example, the template provided uses “fromhost-ip” but if the syslog messages aren’t coming directly from the managed devices, you’ll want to use “hostname” for the property instead and make sure that name resolution is working properly in the environment. Additionally, there are properties for facility and severity (listed as syslogpriority). The severity can be used for determining the severity (critical, major, minor, etc.) of the alarms in Spectrum. If you would like to change the severity of the alarm in CA Spectrum from what the origin believes it should be, you could have additional templates:

In the simple example above, all messages are sent to Log Analytics, messages that are Informational or Debug are written to /var/log/networklog_debug, other messages are written to /var/log/networklog, messages with “Command show” and “%SNMPD-3-ERROR” are dropped from additional processing, and traps are being sent on any messages matching %*-0-*, %*-1-*, and %NAT-6-ADDR_ALLOC_FAILURE patterns with the NAT-6-ADDR_ALLOC_FAILURE messages being tagged as warning as opposed to the vendor defined informational.

Also to note, the version options for omsnmp are 0 and 1. These correspond to v1 and v2c, respectively. In the example above, SNMPv2c traps are being sent.

Example Configuration:

####MODULES ####

#### GLOBAL DIRECTIVES ####

#### RULES ####

The integration included here maps the trap OID and varbind to a CA Spectrum event. That event then forwards to existing 0x3e00009 event which invokes the network monitoring application’s ParseMap functionality to take advantage of 12K+ out of the box mappings. If a ParseMap entry does not exist, rather than following the out of the box message of indicating such, the 0x3dc0004 event is modified to call additional events which include procedures to extract variables from the message, including source, timestamp, facility, and severity and asserts an event/alarm of appropriate criticality on the source model:

Integration Package Contents (version 1 – download):

Extract into $SPECROOT, update the event configuration, and once you have Syslog messages configured to come into the CA Spectrum network monitoring application via the method mentioned above, you should be all set.

Do you think you can beat this Sweet post?

If so, you may have what it takes to become a Sweetcode contributor... Learn More.

Experienced Sales Engineer who enables organizations to solve their business challenges with the solutions I supply by providing the best customer experience possible. I deliver that customer experience through proper understanding, prioritization, solution alignment, value alignment, and communication leveraging the resources available to me.


Discussion

Click on a tab to select how you'd like to leave your comment

Leave a Comment

Your email address will not be published. Required fields are marked *

Menu