In my Previous post i explained about installing and configuring the snmp_exporter for prometheus. In this post i will explain about the prometheus configuration for network devices.

Installing Prometheus:

Follow the instructions mentioned in the https://prometheus.io/docs/prometheus/latest/installation/ to install prometheus. Basically you just need to download the latest package from https://prometheus.io/download/ and start running it.

Prometheus configuration:

Below mentioned is the sample configuration.
In this global parameters are global for all the jobs. This can be over written with the individual jobs scrape_configs. for example global scrape_interval is mentioned as 15s and individual jobs have the scrape interval mentioned as 1min so the individual jobs scrape_interval is taken effect.

Alerting configuration mentions about the alertmanager IP address and details, we will discuss about that in my later post.

rule_files have the rule file path configuration, rule files contains the recording rule. We will about this also in the later post.

Next comes the scrape_configs: this is where the the targets and their module details are mentioned. if you look at the below config there are three jobs created. all_default, cisco_default, cisco_3750 are the three jobs.
the complete config file with all the jobs can be found in https://github.com/vishnubraj/prometheus_config

as i mentioned in my previous post, i am monitoring both cisco and juniper devices. There are certains OID’s which are common to both cisco and juniper, like ifMib. There are certain OID’s which only works with Cisco or Juniper or only specific cisco model like Cisco 3750

so each job points to a certain module in snmp_exporter config using the params field. The all_defaul job points to the ifmib module in the snmp_exporter. normally ifmib is common to both cisco and juniper devices. Hence in the target file “/opt/prometheus/targets/all_default.json” all the cisco and juniper devices has to be mentioned. Cisco_default module points to the cisco module in the snmp_exporter config file [please check the snmp_exporter config file (snmp.yml) in ps://github.com/vishnubraj/prometheus_config for the module details]. only the cisco devices needs to be mentioned in the “/opt/prometheus/targets/cisco_default.json” target file.

[:vishnu:root@test1.sjc2 /opt/prometheus]# cat  prometheus.yml
# my global config
global:
  scrape_interval:     15s # scrape interval for all the modules
  evaluation_interval: 1m # Evaluate rules every 1m
  external_labels:
    region: eu-west
    monitor: infrastructure
    replica: B

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - 127.0.0.1:9093

rule_files:
   - "status.yml"

scrape_configs:
  - job_name: 'all_default'
    scrape_interval: 60s
    scrape_timeout: 60s
    file_sd_configs:
        - files :
          - /opt/prometheus/targets/all_default.json
    metrics_path: /snmp
    params:
      module: [ifmib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [dc]
        target_label: __address__
        replacement: 'server50.$1:9116'

  - job_name: 'cisco_default'
    scrape_interval: 60s
    scrape_timeout: 60s
    file_sd_configs:
        - files :
          - /opt/prometheus/targets/cisco_default.json
    metrics_path: /snmp
    params:
      module: [cisco]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [dc]
        target_label: __address__
        replacement: 'server50.$1:9116'

  - job_name: 'cisco_3750'
    scrape_interval: 60s
    scrape_timeout: 60s
    file_sd_configs:
        - files :
          - /opt/prometheus/targets/cisco_3750.json
    metrics_path: /snmp
    params:
      module: [cisco3750]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [dc]
        target_label: __address__
        replacement: 'server50.$1:9116'

Next comes the relabel_configs: the relabel config is used for the labels in the scrape URL. as mentioned in my snmp_exporter post.

prometheus use the below URL to collect the metrics from the snmp_exporter. In the below URL server50.{dc} is the snmp_exporter IP address, the dc label is mentioned in the target file. for each dc i have a snmp_exporter installed, so the devices have the label with dc details.

http://server50.{dc}:9116/snmp?module={modulename}&target={target_device}

if you are only having one snmp_exporter then you can directly mention the server IP address like below.

   relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: []
        target_label: __address__
        replacement: 'server50.ash1:9116' # or ip adress 

Below mentioned is the target file format

[:vishnu:root@test1.sjc2 /opt/prometheus]# cat/targets/cisco_default.json
[
    {
        "labels": {
            "dc": "del2"
        },
        "targets": [
            "core1.del2", "core2.del2"
        ]
    },
    {
        "labels": {
            "dc": "del2"
        },
        "targets": [
            "rtr1.del2", "rtr2.del2"
        ]
    },
    {
        "labels": {
            "dc": "del2"
        },
        "targets": [
            "sw1.del2"
        ]
    }
]
[:vishnu:root@opstest1.sjc2 /opt/prometheus]#

If you have only few devices you can directly mention the target list as mentioned below instead using the file_sd_config:

    static_configs:
     - targets: ['core1.sjc2','core1.del2']

In my next post i will explain about how to create Dashboards in Grafana for Network devices.