Tagsnmp_exporter

Prometheus Configuration for Network devices

In my Previous post i explained about installing and configuring the snmp_exporter for prometheus. In this post i will explain about the prometheus configuration for network devices.

Installing Prometheus:

Follow the instructions mentioned in the https://prometheus.io/docs/prometheus/latest/installation/ to install prometheus. Basically you just need to download the latest package from https://prometheus.io/download/ and start running it.

Prometheus configuration:

Below mentioned is the sample configuration.
In this global parameters are global for all the jobs. This can be over written with the individual jobs scrape_configs. for example global scrape_interval is mentioned as 15s and individual jobs have the scrape interval mentioned as 1min so the individual jobs scrape_interval is taken effect.

Alerting configuration mentions about the alertmanager IP address and details, we will discuss about that in my later post.

rule_files have the rule file path configuration, rule files contains the recording rule. We will about this also in the later post.

Next comes the scrape_configs: this is where the the targets and their module details are mentioned. if you look at the below config there are three jobs created. all_default, cisco_default, cisco_3750 are the three jobs.
the complete config file with all the jobs can be found in https://github.com/vishnubraj/prometheus_config

as i mentioned in my previous post, i am monitoring both cisco and juniper devices. There are certains OID’s which are common to both cisco and juniper, like ifMib. There are certain OID’s which only works with Cisco or Juniper or only specific cisco model like Cisco 3750

so each job points to a certain module in snmp_exporter config using the params field. The all_defaul job points to the ifmib module in the snmp_exporter. normally ifmib is common to both cisco and juniper devices. Hence in the target file “/opt/prometheus/targets/all_default.json” all the cisco and juniper devices has to be mentioned. Cisco_default module points to the cisco module in the snmp_exporter config file [please check the snmp_exporter config file (snmp.yml) in ps://github.com/vishnubraj/prometheus_config for the module details]. only the cisco devices needs to be mentioned in the “/opt/prometheus/targets/cisco_default.json” target file.

[:vishnu:root@test1.sjc2 /opt/prometheus]# cat  prometheus.yml
# my global config
global:
  scrape_interval:     15s # scrape interval for all the modules
  evaluation_interval: 1m # Evaluate rules every 1m
  external_labels:
    region: eu-west
    monitor: infrastructure
    replica: B

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - 127.0.0.1:9093

rule_files:
   - "status.yml"

scrape_configs:
  - job_name: 'all_default'
    scrape_interval: 60s
    scrape_timeout: 60s
    file_sd_configs:
        - files :
          - /opt/prometheus/targets/all_default.json
    metrics_path: /snmp
    params:
      module: [ifmib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [dc]
        target_label: __address__
        replacement: 'server50.$1:9116'

  - job_name: 'cisco_default'
    scrape_interval: 60s
    scrape_timeout: 60s
    file_sd_configs:
        - files :
          - /opt/prometheus/targets/cisco_default.json
    metrics_path: /snmp
    params:
      module: [cisco]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [dc]
        target_label: __address__
        replacement: 'server50.$1:9116'

  - job_name: 'cisco_3750'
    scrape_interval: 60s
    scrape_timeout: 60s
    file_sd_configs:
        - files :
          - /opt/prometheus/targets/cisco_3750.json
    metrics_path: /snmp
    params:
      module: [cisco3750]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [dc]
        target_label: __address__
        replacement: 'server50.$1:9116'

Next comes the relabel_configs: the relabel config is used for the labels in the scrape URL. as mentioned in my snmp_exporter post.

prometheus use the below URL to collect the metrics from the snmp_exporter. In the below URL server50.{dc} is the snmp_exporter IP address, the dc label is mentioned in the target file. for each dc i have a snmp_exporter installed, so the devices have the label with dc details.

http://server50.{dc}:9116/snmp?module={modulename}&target={target_device}

if you are only having one snmp_exporter then you can directly mention the server IP address like below.

   relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: []
        target_label: __address__
        replacement: 'server50.ash1:9116' # or ip adress 

Below mentioned is the target file format

[:vishnu:root@test1.sjc2 /opt/prometheus]# cat/targets/cisco_default.json
[
    {
        "labels": {
            "dc": "del2"
        },
        "targets": [
            "core1.del2", "core2.del2"
        ]
    },
    {
        "labels": {
            "dc": "del2"
        },
        "targets": [
            "rtr1.del2", "rtr2.del2"
        ]
    },
    {
        "labels": {
            "dc": "del2"
        },
        "targets": [
            "sw1.del2"
        ]
    }
]
[:vishnu:root@opstest1.sjc2 /opt/prometheus]#

If you have only few devices you can directly mention the target list as mentioned below instead using the file_sd_config:

    static_configs:
     - targets: ['core1.sjc2','core1.del2']

In my next post i will explain about how to create Dashboards in Grafana for Network devices.

Prometheus SNMP_exporter Configuration

Prometheus doesn’t connect with end host to collect the metrics. it needs an exporter to expose the metrics from end host.

There are multiple exporters available, some are developed by prometheus team itself. The list of exporter details are available in https://prometheus.io/docs/instrumenting/exporters/

For servers we use node_exporter to export the metrics. node_exporter needs to be installed in the end host. once installed its opens an http API port for the prometheus to connect with the end device and collect the metric. For every scrape interval(Metric collection frequency mentioned in the prometheus config file ) prometheus collects the metric from node exporter API.

Network devices doesn’t allow installing any packages . The only way to collect the metric from the devices are using SNMP or login via SSH and collect metric. Login via SSH may have CPU impact. Hence we need an exporter which works between prometheus and network devices for collecting metrics. SNMP_exporter is used for this. SNMP_exporter is developed by the core prometheus team. So its very stable.

Installing SNMP_exporter

Binaries can be downloaded from https://github.com/prometheus/snmp_exporter/releases page.

To start SNMP_exporter it needs snmp.yml config file. This snmp.yml file only has the module name and the OID detail’s to snmp walk or get. also it has the SNMP authentication details.
It doesnt have any details about the network device IP details. network device details are sent to snmp_exporter using the http URL(API). The URL contains the network device IP address and modules which needs to collected. Below mentioned is a sample URL. In this core1.ash1 is the target network device and “ifmibmodule” is the module name which needs to collected.

http://10.0.10.1:9116/snmp?module=ifmibmodule&target=core1.ash1

The “ifmibmodule” details exist in the snmp.yml file.

ifmib:
  walk:
  - 1.3.6.1.2.1.2.2.1.13
  get:
  - 1.3.6.1.2.1.1.1.0
  - 1.3.6.1.2.1.1.3.0
  metrics:
  - name: ifInDiscards
    oid: 1.3.6.1.2.1.2.2.1.13
    type: counter
    help: The number of inbound packets which were chosen to be discarded even though
      no errors had been detected to prevent their being deliverable to a higher-layer
      protocol - 1.3.6.1.2.1.2.2.1.13
    indexes:
    - labelname: ifIndex
      type: gauge
    lookups:
    - labels:
      - ifIndex
      labelname: ifAlias
      oid: 1.3.6.1.2.1.31.1.1.1.18
      type: DisplayString
    - labels:
      - ifIndex
      labelname: ifName
      oid: 1.3.6.1.2.1.31.1.1.1.1
      type: DisplayString
    - labels: []
      labelname: ifIndex
    - labels: []
      labelname: ifIndex

Above mentioned is the sample snmp.yml config file, but Its not easy to write the snmp.yml by hand. Prometheus team has created snmp_generator for this reason. This snmp_genertor helps to create the snmp.yml config file.

Installing snmp_generator

snmp_generator can be installed at any location. once the snmp.yml file is generated it can be copied to the same folder where snmp_exporter installed and and you can start the snmp_exporter.

To install the SNMP_generator follow the instructions mentioned at https://github.com/prometheus/snmp_exporter/tree/master/generator

SNMP_generator needs the generator.yml config file to generate the snmp.yml file.

How to write generator.yml file

Below is the example of generator.yml file. you can write your own generator.yml file or you can download my generator.yml file from https://github.com/vishnubraj/prometheus_config/

modules:
#System details and interface stats OID's are common for both Cisco and Juniper
  ifmib:
    walk:
      - 1.3.6.1.2.1.1.3  #System UPtime
      - 1.3.6.1.2.1.1.1  #System Descriptio
      - 1.3.6.1.2.1.31.1.1.1.6 #ifHCInOctets
      - 1.3.6.1.2.1.31.1.1.1.10 #ifHCOutOctets
      - 1.3.6.1.2.1.2.2.1.13 #ifInDiscards
      - 1.3.6.1.2.1.2.2.1.14 #ifInErrors
      - 1.3.6.1.2.1.2.2.1.19 #ifOutDiscards
      - 1.3.6.1.2.1.2.2.1.20 #ifOutErrors
      - 1.3.6.1.2.1.31.1.1.1.15 #ifHighSpeed
      - 1.3.6.1.2.1.31.1.1.1.7 #ifHCInUcastPkts
      - 1.3.6.1.2.1.31.1.1.1.11 #ifHCOutUcastPkts
    version: 3
    auth:
      security_level: authPriv
      username: testuser
      password: testpassword
      auth_protocol: MD5
      priv_protocol: AES
      priv_password: testpriv

    lookups:
      - source_indexes: [ifIndex]
        lookup: ifAlias
        drop_source_indexes: true
      - source_indexes: [ifIndex]
        lookup: ifName
        drop_source_indexes: true

In the above code “ifmib” is the module name , this will be used by prometheus while scraping the targets. its always better to create multiple different modules for example ospf,isis, bgp, ifmib, ipsla, rpm etc, this will be useful when writing the prometheus config file. and if you have a switch and router, you dont need to run the ospf OID’s on the switch where you dont have the ospf running. so while writing the prometheus config you can mention all the switches to have only ifmib module running.
Then you need to find out the OID’s which needs to be queried from your device for the metric. Then those needs to be mentioned under the walk: section.

Version: is the snmp version which is running on the device.

auth: section contains the authentication details.

Lookups: all the snmp oid comes under a table. details about snmp tables can be found here https://www.webnms.com/snmp/help/snmpapi/snmpv3/table_handling/snmptables_basics.html . if we only query the OID’s it wont be useful to graph.

The metrics will have only the ifindex of the metric like below. in these each fields are labels, for example dc,ifIndex, instance,job all are labels. its not easy to identify which interface this belongs to

{dc="ash1",ifIndex="5",instance="core1.ash1",job="all_default",region="eu-west"} = 19536684.216666665

Lookups are used to add the labels based on our interest. So we need the ifHCInOctets with ifName mapping to better understand the below metric for graphing. The below config says lookup for ifAlias value using the source_indexes as ifIndex and drop the source_indexes

  lookups:
      - source_indexes: [ifIndex]
        lookup: ifAlias
        drop_source_indexes: true

so after adding the lookup config, we get the below metric. which is easy to understand

{dc="ash1",ifAlias="pl3_166_level3",ifName="xe-0/1/6.0",instance="core1.ash1",job="all_default",region="eu-west"}

To generate the snmp.yml run the snmp_generator like below.

./generator generate

This will generate the snmp.yml file, copy it to the snmp_exporter folder and run the snmp_exporter like below.

./snmp_exporter &

In my next post i will explain about the prometheus config for the network devices.

© 2019 Networking Blog

Theme by Anders NorénUp ↑