Categories
Monitoring

Prometheus SNMP_exporter Configuration – Part 2

Prometheus always needs an HTTP endpoint to scrapes the metric. In servers, we can run https://github.com/prometheus/node_exporter which collects the metrics and exposes it in an HTTP URL.

There are multiple exporters available, some are developed by the Prometheus team itself. The list of exporter details are available in https://prometheus.io/docs/instrumenting/exporters/

Network devices don’t allow installing any packages. The only way to collect the metric from the devices is by using SNMP or login via SSH and collect metrics. Login via SSH may have CPU impact. Hence we need an exporter that works between Prometheus and network devices for collecting metrics. SNMP_exporter is used for this. SNMP_exporter is developed by the core Prometheus team and it is very stable.

Installing SNMP_exporter

Binaries can be downloaded from https://github.com/prometheus/snmp_exporter/releases page.

To start SNMP_exporter it needs a snmp.yml config file. This snmp.yml file three details, module name, OID details, and SNMP authentication details.
It doesn’t have any details about the network device IP details. network device details are sent to snmp_exporter using the HTTP URL(API). The URL contains the network device IP address and modules that need to collected. Below mentioned is a sample URL. In this core1.ash1 is the target network device and “ifmib” is the module name that needs to collect.

http://10.0.10.1:9116/snmp?module=ifmib&target=core1.ash1

The “ifmib” details exist in the snmp.yml file.

ifmib:              ### MODULE NAME
  walk:
  - 1.3.6.1.2.1.2.2.1.13  ### OID TO SNMP WALK
  get:
  - 1.3.6.1.2.1.1.1.0     ### OID to SNMP GET
  - 1.3.6.1.2.1.1.3.0
  metrics:
  - name: ifInDiscards
    oid: 1.3.6.1.2.1.2.2.1.13
    type: counter
    help: The number of inbound packets which were chosen to be discarded even though
      no errors had been detected to prevent their being deliverable to a higher-layer
      protocol - 1.3.6.1.2.1.2.2.1.13
    indexes:
    - labelname: ifIndex
      type: gauge
    lookups:
    - labels:
      - ifIndex
      labelname: ifAlias
      oid: 1.3.6.1.2.1.31.1.1.1.18
      type: DisplayString
    - labels:
      - ifIndex
      labelname: ifName
      oid: 1.3.6.1.2.1.31.1.1.1.1
      type: DisplayString
    - labels: []
      labelname: ifIndex
    - labels: []
      labelname: ifIndex

Above mentioned is the sample snmp.yml config file, but Its not easy to write the snmp.yml by hand. Prometheus team has created https://github.com/prometheus/snmp_exporter/tree/master/generator for this reason. This snmp_genertor helps to create the snmp.yml config file.

Installing snmp_generator

snmp_generator can be installed at any location. once we run the snmp_generator with the generator.yml file, it generates the snmp.yml output file. The snmp.yml file can be copied to the same folder where snmp_exporter installed and you can start the snmp_exporter.

To install the SNMP_generator follow the instructions mentioned at https://github.com/prometheus/snmp_exporter/tree/master/generator

SNMP_generator needs the generator.yml config file to generate the snmp.yml file.

How to write generator.yml file

Below is the example of the generator.yml file. you can write your own generator.yml file or you can download my generator.yml file from https://github.com/vishnubraj/prometheus_config/

modules:
#System details and interface stats OID's are common for both Cisco and Juniper
  ifmib:
    walk:
      - 1.3.6.1.2.1.1.3  #System UPtime
      - 1.3.6.1.2.1.1.1  #System Descriptio
      - 1.3.6.1.2.1.31.1.1.1.6 #ifHCInOctets
      - 1.3.6.1.2.1.31.1.1.1.10 #ifHCOutOctets
      - 1.3.6.1.2.1.2.2.1.13 #ifInDiscards
      - 1.3.6.1.2.1.2.2.1.14 #ifInErrors
      - 1.3.6.1.2.1.2.2.1.19 #ifOutDiscards
      - 1.3.6.1.2.1.2.2.1.20 #ifOutErrors
      - 1.3.6.1.2.1.31.1.1.1.15 #ifHighSpeed
      - 1.3.6.1.2.1.31.1.1.1.7 #ifHCInUcastPkts
      - 1.3.6.1.2.1.31.1.1.1.11 #ifHCOutUcastPkts
    version: 3
    auth:
      security_level: authPriv
      username: testuser
      password: testpassword
      auth_protocol: MD5
      priv_protocol: AES
      priv_password: testpriv

    lookups:
      - source_indexes: [ifIndex]
        lookup: ifAlias
        drop_source_indexes: true
      - source_indexes: [ifIndex]
        lookup: ifName
        drop_source_indexes: true

In the above code “ifmib” is the module name, this will be used by Prometheus while scraping the targets. it’s always better to create multiple different modules, for example, OSPF, IS-IS, BGP, ifmib, ipsla, rpm, etc, this will be useful when writing the Prometheus config file. and if you have a switch and router, you don’t need to run the OSPF OID’s on the switch where you don’t have the OSPF running. so while writing the Prometheus config you can mention all the switches be checked for the ifmib module.

Walk: Mention the list of OID’s needs to be queried from the device

Get: InCase if we want to query get a single oid we can use the get option

Version: is the SNMP version which is running on the device.

auth: section contains the authentication details.

Lookups: all the SNMP oid comes under a table. details about SNMP tables can be found here https://www.webnms.com/snmp/help/snmpapi/snmpv3/table_handling/snmptables_basics.html . if we only query the OID’s it won’t be useful to the graph.

Check the ifHCInOctets metric output mentioned below, The metric has instance detail and other device detail as labels and ifindex of the metric. In this it’s not easy to identify which interface this belongs to, because ifindex is used by SNMP. we only can identify this if we know ifName(xe-0/1/6.0) or ifAlias(edge_cogent) of the interface.

ifHCInOctets{dc="ash1",ifIndex="5",instance="core1.ash1",job="all_default",region="eu-west"} = 19536684.216666665

Lookups are used to add the labels based on our interest. So we need the ifHCInOctets with ifName and ifAlias mapping to better understand the below metric for graphing. The below config says lookup for ifAlias value using the source_indexes as ifIndex and drop the source_indexes also lookup for ifName value using the source_indexes as ifIndex and drop the source_indexes

The concept it ifName, ifAlias, ifHcInOctets all have the same ifIndex value based on the SNMP table. lookups are used to correlate this and crate a meaningful metric.

  lookups:      
      - source_indexes: [ifIndex]
        lookup: ifAlias
        drop_source_indexes: true
      - source_indexes: [ifIndex]
        lookup: ifName
        drop_source_indexes: true

so after adding the lookup config, we get the below metric. which is easy to understand

ifHCInOctets{dc="ash1",ifAlias="edge_cogent",ifName="xe-0/1/6.0",instance="core1.ash1",job="all_default",region="eu-west"}

To generate the snmp.yml run the snmp_generator like below.

./generator generate

This will generate the snmp.yml file, copy it to the snmp_exporter folder, and run the snmp_exporter like below. It will auto pick the SNMP.YML config file from the folder it is running.

./snmp_exporter &

In my next post i will explain about the prometheus config for the network devices.

https://networkblog.net/prometheus-configuration-for-network-devices/

One reply on “Prometheus SNMP_exporter Configuration – Part 2”

Leave a Reply