Prometheus doesn’t connect with end host to collect the metrics. it needs an exporter to expose the metrics from end host.

There are multiple exporters available, some are developed by prometheus team itself. The list of exporter details are available in https://prometheus.io/docs/instrumenting/exporters/

For servers we use node_exporter to export the metrics. node_exporter needs to be installed in the end host. once installed its opens an http API port for the prometheus to connect with the end device and collect the metric. For every scrape interval(Metric collection frequency mentioned in the prometheus config file ) prometheus collects the metric from node exporter API.

Network devices doesn’t allow installing any packages . The only way to collect the metric from the devices are using SNMP or login via SSH and collect metric. Login via SSH may have CPU impact. Hence we need an exporter which works between prometheus and network devices for collecting metrics. SNMP_exporter is used for this. SNMP_exporter is developed by the core prometheus team. So its very stable.

Installing SNMP_exporter

Binaries can be downloaded from https://github.com/prometheus/snmp_exporter/releases page.

To start SNMP_exporter it needs snmp.yml config file. This snmp.yml file only has the module name and the OID detail’s to snmp walk or get. also it has the SNMP authentication details.
It doesnt have any details about the network device IP details. network device details are sent to snmp_exporter using the http URL(API). The URL contains the network device IP address and modules which needs to collected. Below mentioned is a sample URL. In this core1.ash1 is the target network device and “ifmibmodule” is the module name which needs to collected.

http://10.0.10.1:9116/snmp?module=ifmibmodule&target=core1.ash1

The “ifmibmodule” details exist in the snmp.yml file.

ifmib:
  walk:
  - 1.3.6.1.2.1.2.2.1.13
  get:
  - 1.3.6.1.2.1.1.1.0
  - 1.3.6.1.2.1.1.3.0
  metrics:
  - name: ifInDiscards
    oid: 1.3.6.1.2.1.2.2.1.13
    type: counter
    help: The number of inbound packets which were chosen to be discarded even though
      no errors had been detected to prevent their being deliverable to a higher-layer
      protocol - 1.3.6.1.2.1.2.2.1.13
    indexes:
    - labelname: ifIndex
      type: gauge
    lookups:
    - labels:
      - ifIndex
      labelname: ifAlias
      oid: 1.3.6.1.2.1.31.1.1.1.18
      type: DisplayString
    - labels:
      - ifIndex
      labelname: ifName
      oid: 1.3.6.1.2.1.31.1.1.1.1
      type: DisplayString
    - labels: []
      labelname: ifIndex
    - labels: []
      labelname: ifIndex

Above mentioned is the sample snmp.yml config file, but Its not easy to write the snmp.yml by hand. Prometheus team has created snmp_generator for this reason. This snmp_genertor helps to create the snmp.yml config file.

Installing snmp_generator

snmp_generator can be installed at any location. once the snmp.yml file is generated it can be copied to the same folder where snmp_exporter installed and and you can start the snmp_exporter.

To install the SNMP_generator follow the instructions mentioned at https://github.com/prometheus/snmp_exporter/tree/master/generator

SNMP_generator needs the generator.yml config file to generate the snmp.yml file.

How to write generator.yml file

Below is the example of generator.yml file. you can write your own generator.yml file or you can download my generator.yml file from https://github.com/vishnubraj/prometheus_config/

modules:
#System details and interface stats OID's are common for both Cisco and Juniper
  ifmib:
    walk:
      - 1.3.6.1.2.1.1.3  #System UPtime
      - 1.3.6.1.2.1.1.1  #System Descriptio
      - 1.3.6.1.2.1.31.1.1.1.6 #ifHCInOctets
      - 1.3.6.1.2.1.31.1.1.1.10 #ifHCOutOctets
      - 1.3.6.1.2.1.2.2.1.13 #ifInDiscards
      - 1.3.6.1.2.1.2.2.1.14 #ifInErrors
      - 1.3.6.1.2.1.2.2.1.19 #ifOutDiscards
      - 1.3.6.1.2.1.2.2.1.20 #ifOutErrors
      - 1.3.6.1.2.1.31.1.1.1.15 #ifHighSpeed
      - 1.3.6.1.2.1.31.1.1.1.7 #ifHCInUcastPkts
      - 1.3.6.1.2.1.31.1.1.1.11 #ifHCOutUcastPkts
    version: 3
    auth:
      security_level: authPriv
      username: testuser
      password: testpassword
      auth_protocol: MD5
      priv_protocol: AES
      priv_password: testpriv

    lookups:
      - source_indexes: [ifIndex]
        lookup: ifAlias
        drop_source_indexes: true
      - source_indexes: [ifIndex]
        lookup: ifName
        drop_source_indexes: true

In the above code “ifmib” is the module name , this will be used by prometheus while scraping the targets. its always better to create multiple different modules for example ospf,isis, bgp, ifmib, ipsla, rpm etc, this will be useful when writing the prometheus config file. and if you have a switch and router, you dont need to run the ospf OID’s on the switch where you dont have the ospf running. so while writing the prometheus config you can mention all the switches to have only ifmib module running.
Then you need to find out the OID’s which needs to be queried from your device for the metric. Then those needs to be mentioned under the walk: section.

Version: is the snmp version which is running on the device.

auth: section contains the authentication details.

Lookups: all the snmp oid comes under a table. details about snmp tables can be found here https://www.webnms.com/snmp/help/snmpapi/snmpv3/table_handling/snmptables_basics.html . if we only query the OID’s it wont be useful to graph.

The metrics will have only the ifindex of the metric like below. in these each fields are labels, for example dc,ifIndex, instance,job all are labels. its not easy to identify which interface this belongs to

{dc="ash1",ifIndex="5",instance="core1.ash1",job="all_default",region="eu-west"} = 19536684.216666665

Lookups are used to add the labels based on our interest. So we need the ifHCInOctets with ifName mapping to better understand the below metric for graphing. The below config says lookup for ifAlias value using the source_indexes as ifIndex and drop the source_indexes

  lookups:
      - source_indexes: [ifIndex]
        lookup: ifAlias
        drop_source_indexes: true

so after adding the lookup config, we get the below metric. which is easy to understand

{dc="ash1",ifAlias="pl3_166_level3",ifName="xe-0/1/6.0",instance="core1.ash1",job="all_default",region="eu-west"}

To generate the snmp.yml run the snmp_generator like below.

./generator generate

This will generate the snmp.yml file, copy it to the snmp_exporter folder and run the snmp_exporter like below.

./snmp_exporter &

In my next post i will explain about the prometheus config for the network devices.