monitoring-system

---

Table of Content

monitoring-system

• [x] prometheus using docker on VM
• [x] snmp exporter using docker on VM
• [x] snmp test using debian machine running snmpd
• [ ] wip

create Debian 12 VM on Hyper-V

$VMName = "mon"
$Switch = 'External VM Switch'
$BasePath = "F:\hyperv"
$VMPath = "$BasePath\vm"
$VHDPath = "$BasePath\vhd"
$InstallMedia = "F:\hyperv\install_media\debian-12.4.0-amd64-netinst.iso"

$VM = @{
    Name = $VMName
    MemoryStartupBytes = 16GB
    Generation = 2
    NewVHDPath = "$VHDPath\$VMName.vhdx"
    NewVHDSizeBytes = 300GB
    Path = "$VMPath\$VMName"
    SwitchName = $Switch
}

New-VM @VM

# Set 2 CPU processors
Set-VMProcessor $VMName -Count 2

# Set fixed memory
Set-VMMemory $VMName -DynamicMemoryEnabled $false

# Change secure boot template to MS UEFI CA
Set-VMFirmware -VMName $VMName -SecureBootTemplateId "272e7447-90a4-4563-a4b9-8e4ab00526ce"

# Add DVD Drive to Virtual Machine
Add-VMScsiController -VMName $VMName
Add-VMDvdDrive -VMName $VMName -ControllerNumber 1 -ControllerLocation 0 -Path $InstallMedia

# Mount Installation Media
$DVDDrive = Get-VMDvdDrive -VMName $VMName

# Configure Virtual Machine to Boot from DVD
Set-VMFirmware -VMName $VMName -FirstBootDevice $DVDDrive

# Run VM
Start-VM $VMName

Debian 12 installation

• language: English
• country: Japan
• locale: en_US.UTF-8
• keyboard layout: Japanese
• hostname: mon
• domain name: network.blink-1x52.net
• configure network manually: 192.168.1.82/24, gw on .1, and name server on .55
• archive mirror in Japan, deb.debian.org
• install packages:
  • ssh server
  • standard system utilities

Initial setup

sudo

# install and enable sudo for your user
su -
apt update
apt install sudo
usermod -aG sudo {your_username}

docker

https://docs.docker.com/engine/install/debian/

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

#  List the available versions:
apt-cache madison docker-ce | awk '{ print $3 }'

# install specific version
VERSION_STRING=5:26.1.3-1~debian.12~bookworm
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

# hold the version
sudo apt-mark hold docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# add your user to the docker group to run docker without sudo
sudo usermod -aG docker $USER
# re-login

# test
docker run hello-world

# clean up
docker system prune --all

prometheus setup

snmp exporter config generator

https://github.com/prometheus/snmp_exporter/tree/main/generator

mkdir ~/dnld && cd ~/dnld
wget https://go.dev/dl/go1.22.3.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.22.3.linux-amd64.tar.gz
echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.bashrc
source ~/.bashrc

# Debian-based distributions.
sudo apt-get install unzip build-essential libsnmp-dev # Debian-based distros
# Redhat-based distributions.
# sudo yum install gcc make net-snmp net-snmp-utils net-snmp-libs net-snmp-devel # RHEL-based distros

# build
cd
git clone https://github.com/prometheus/snmp_exporter.git
cd snmp_exporter/generator
make generator mibs

generator.yml

I have just added "infra-nwk" auth with custom community string.

mib directory

This is work in progress.

I have added some MIBs listed in n3548 supported list to what's there by default mibs/, a few by few, resolving errors.

7.2.50.0.18765.RELEASE-B100-MIB.txt
7.2.50.0.18765.RELEASE-CERTS-MIB.txt
7.2.50.0.18765.RELEASE-IPVS-MIB.txt
7.2.50.0.18765.RELEASE-ONE4NET-MIB.txt
AIRESPACE-REF-MIB
AIRESPACE-WIRELESS-MIB
apc-powernet-mib
ARISTA-ENTITY-SENSOR-MIB
ARISTA-SMI-MIB
ARISTA-SW-IP-FORWARDING-MIB
BGP4-MIB.my
BRIDGE-MIB.my
CISCO-AAA-SERVER-EXT-MIB.my
CISCO-AAA-SERVER-MIB.my
CISCO-CALLHOME-MIB.my
CISCO-CDP-MIB.my
CISCO-CFS-MIB.my
CISCO-COMMON-MGMT-MIB.my
CISCO-COMMON-ROLES-EXT-MIB.my
CISCO-COMMON-ROLES-MIB.my
CISCO-CONFIG-COPY-MIB.my
CISCO-CONFIG-MAN-MIB.my
CISCO-EIGRP-MIB.my
CISCO-ENTITY-EXT-MIB.my
CISCO-ENTITY-VENDORTYPE-OID-MIB.my
CISCO-FEATURE-CONTROL-MIB.my
CISCO-FLASH-MIB.my
CISCO-FTP-CLIENT-MIB.my
CISCO-HC-ALARM-MIB.my
CISCO-HSRP-MIB.my
CISCO-IF-EXTENSION-MIB.my
CISCO-IMAGE-MIB.my
CISCO-IMAGE-UPGRADE-MIB.my
CISCO-IP-IF-MIB.my
CISCO-LICENSE-MGR-MIB.my
CISCO-NOTIFICATION-CONTROL-MIB.my
CISCO-NTP-MIB.my
CISCO-PROCESS-MIB.my
CISCO-QOS-PIB-MIB.my
CISCO-RMON-CONFIG-MIB.my
CISCO-SECURE-SHELL-MIB.my
CISCO-SMI.my
CISCO-SNMP-TARGET-EXT-MIB.my
CISCO-ST-TC.my
CISCO-SYSLOG-EXT-MIB.my
CISCO-SYSLOG-MIB.my
CISCO-SYSTEM-EXT-MIB.my
CISCO-SYSTEM-MIB.my
CISCO-TC.my
CISCO-UDLDP-MIB.my
CISCO-VLAN-MEMBERSHIP-MIB.my
CISCO-VTP-MIB.my
CyberPower.MIB
EAP-Client.MIB
EAP.MIB
ENTITY-MIB
ENTITY-MIB.my
ENTITY-SENSOR-MIB
ENTITY-STATE-MIB
ENTITY-STATE-TC-MIB
HCNUM-TC
HCNUM-TC.my
HOST-RESOURCES-MIB
IANA-ADDRESS-FAMILY-NUMBERS-MIB.my
IANA-CHARSET-MIB.txt
IANAifType-MIB.my
IANA-IFTYPE-MIB.txt
IANA-MAU-MIB.my
IANA-PRINTER-MIB.txt
IANA-RTPROTO-MIB.my
iDRAC-SMIv2.mib
IF-MIB
IF-MIB.my
INET-ADDRESS-MIB
INET-ADDRESS-MIB.my
Infrapower-MIB.mib
IPMCAST-MIB.my
IP-MIB
IP-MIB.my
IPMROUTE-STD-MIB.my
IPV6-TC
ISDN-MIB
KEEPALIVED-MIB
LANGTAG-TC-MIB.my
LIEBERT_GP_PDU.MIB
LIEBERT_GP_REG.MIB
LLDP-MIB.my
MAU-MIB.my
MIKROTIK-MIB
NET-SNMP-MIB
NET-SNMP-TC
NOTIFICATION-LOG-MIB.my
OSPF-MIB.my
OSPF-TRAP-MIB.my
PAN-COMMON-MIB.md5
PAN-COMMON-MIB.my
PAN-ENTITY-EXT-MIB.md5
PAN-ENTITY-EXT-MIB.my
PAN-GLOBAL-REG-MIB.md5
PAN-GLOBAL-REG-MIB.my
PAN-GLOBAL-TC-MIB.md5
PAN-GLOBAL-TC-MIB.my
PAN-LC-MIB.md5
PAN-LC-MIB.my
PAN-PRODUCT-MIB.md5
PAN-PRODUCT-MIB.my
PAN-TRAPS.md5
PAN-TRAPS.my
PDU2-MIB.txt
PDU-MIB.txt
PICO-IPSEC-FLOW-MONITOR-MIB.txt
PICO-SMI-ID-MIB.txt
PICO-SMI-MIB.txt
PIM-MIB.my
PRINTER-MIB-V2.txt
readydataos
readynas
RMON2-MIB.my
RMON-MIB.my
servertech-sentry3-mib
servertech-sentry4-mib
SMON-MIB.my
SNMP-COMMUNITY-MIB.my
SNMP-FRAMEWORK-MIB
SNMP-FRAMEWORK-MIB.my
SNMP-MPD-MIB.my
SNMP-NOTIFICATION-MIB.my
SNMP-TARGET-MIB.my
SNMP-USM-MIB.my
SNMPv2-MIB
SNMPv2-MIB.my
SNMPv2-SMI
SNMPv2-TC
SNMPv2-TC.my
SYNOLOGY-DISK-MIB.txt
SYNOLOGY-EBOX-MIB.txt
SYNOLOGY-FLASHCACHE-MIB.txt
SYNOLOGY-GPUINFO-MIB.txt
SYNOLOGY-ISCSILUN-MIB.txt
SYNOLOGY-ISCSITarget-MIB.txt
SYNOLOGY-NFS-MIB.txt
SYNOLOGY-PORT-MIB.txt
SYNOLOGY-RAID-MIB.txt
SYNOLOGY-SERVICES-MIB.txt
SYNOLOGY-SHA-MIB.txt
SYNOLOGY-SMART-MIB.txt
SYNOLOGY-SPACEIO-MIB.txt
SYNOLOGY-STORAGEIO-MIB.txt
SYNOLOGY-SYSTEM-MIB.txt
SYNOLOGY-UPS-MIB.txt
TCP-MIB.my
TOKEN-RING-RMON-MIB.mib
UBNT-AirFiber-MIB
UBNT-AirMAX-MIB.txt
UBNT-UniFi-MIB
UCD-SNMP-MIB
UDP-MIB.my
VRRP-MIB
VRRPv3-MIB

mibs

nexus 3548

cisco-mibs/v2/BGP4-MIB.my
cisco-mibs/v2/BRIDGE-MIB.my
cisco-mibs/v2/CISCO-AAA-SERVER-EXT-MIB.my
cisco-mibs/v2/CISCO-AAA-SERVER-MIB.my
cisco-mibs/v2/CISCO-CALLHOME-MIB.my
cisco-mibs/v2/CISCO-CDP-MIB.my
cisco-mibs/v2/CISCO-CFS-MIB.my
cisco-mibs/v2/CISCO-COMMON-MGMT-MIB.my
cisco-mibs/v2/CISCO-COMMON-ROLES-EXT-MIB.my
cisco-mibs/v2/CISCO-CONFIG-COPY-MIB.my
cisco-mibs/v2/CISCO-CONFIG-MAN-MIB.my
cisco-mibs/v2/CISCO-EIGRP-MIB.my
cisco-mibs/v2/CISCO-ENTITY-EXT-MIB.my
cisco-mibs/v2/CISCO-ENTITY-VENDORTYPE-OID-MIB.my
cisco-mibs/v2/CISCO-FEATURE-CONTROL-MIB.my
cisco-mibs/v2/CISCO-FLASH-MIB.my
cisco-mibs/v2/CISCO-FTP-CLIENT-MIB.my
cisco-mibs/v2/CISCO-HC-ALARM-MIB.my
cisco-mibs/v2/CISCO-HSRP-MIB.my
cisco-mibs/v2/CISCO-IF-EXTENSION-MIB.my
cisco-mibs/v2/CISCO-IMAGE-MIB.my
cisco-mibs/v2/CISCO-IMAGE-UPGRADE-MIB.my
cisco-mibs/v2/CISCO-IP-IF-MIB.my
cisco-mibs/v2/CISCO-LICENSE-MGR-MIB.my
cisco-mibs/v2/CISCO-NOTIFICATION-CONTROL-MIB.my
cisco-mibs/v2/CISCO-NTP-MIB.my
cisco-mibs/v2/CISCO-PROCESS-MIB.my
cisco-mibs/v2/CISCO-RMON-CONFIG-MIB.my
cisco-mibs/v2/CISCO-SECURE-SHELL-MIB.my
cisco-mibs/v2/CISCO-SMI.my
cisco-mibs/v2/CISCO-SNMP-TARGET-EXT-MIB.my
cisco-mibs/v2/CISCO-SYSLOG-EXT-MIB.my
cisco-mibs/v2/CISCO-SYSTEM-EXT-MIB.my
cisco-mibs/v2/CISCO-SYSTEM-MIB.my
cisco-mibs/v2/CISCO-UDLDP-MIB.my
cisco-mibs/v2/CISCO-VLAN-MEMBERSHIP-MIB.my
cisco-mibs/v2/CISCO-VTP-MIB.my
cisco-mibs/v2/ENTITY-MIB.my
cisco-mibs/v2/HCNUM-TC.my
cisco-mibs/v2/IANA-ADDRESS-FAMILY-NUMBERS-MIB.my
cisco-mibs/v2/IANAifType-MIB.my
cisco-mibs/v2/IF-MIB.my
cisco-mibs/v2/INET-ADDRESS-MIB.my
cisco-mibs/v2/IPMCAST-MIB.my
cisco-mibs/v2/IP-MIB.my
cisco-mibs/v2/LLDP-MIB.my
cisco-mibs/v2/MAU-MIB.my
cisco-mibs/v2/NOTIFICATION-LOG-MIB.my
cisco-mibs/v2/OSPF-MIB.my
cisco-mibs/v2/OSPF-TRAP-MIB.my
cisco-mibs/v2/PIM-MIB.my
cisco-mibs/v2/RMON2-MIB.my
cisco-mibs/v2/RMON-MIB.my
cisco-mibs/v2/SNMP-COMMUNITY-MIB.my
cisco-mibs/v2/SNMP-FRAMEWORK-MIB.my
cisco-mibs/v2/SNMP-MPD-MIB.my
cisco-mibs/v2/SNMP-NOTIFICATION-MIB.my
cisco-mibs/v2/SNMP-TARGET-MIB.my
cisco-mibs/v2/SNMP-USM-MIB.my
cisco-mibs/v2/SNMPv2-MIB.my
cisco-mibs/v2/SNMPv2-TC.my
cisco-mibs/v2/TCP-MIB.my
cisco-mibs/v2/UDP-MIB.my

prometheus config file

Reference file here.

mkdir -p ~/monitoring/config && cd ~/monitoring/config
cp ~/snmp_exporter/generator/snmp.yml .
# create prometheus.yml file here

# global
global:
  scrape_interval: 15s
  evaluation_interval: 30s
  body_size_limit: 15MB
  sample_limit: 1500
  target_limit: 30
  label_limit: 30
  label_name_length_limit: 200
  label_value_length_limit: 200
  # scrape_timeout is set to the global default (10s).

  external_labels:
    monitor: infrastructure-network

# scrape configs
scrape_configs:
  # prometheus itself
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: snmp_network_general
    static_configs:
      - targets: ["192.168.1.81"]
    metrics_path: /snmp
    params:
      auth: ["infra-nwk"]
      module: ["if_mib", "ip_mib"]

docker volume

Create docker volume.

# Create persistent volume for your data
docker volume create prometheus-data

docker compose

Create ~/docker-compose.yml file.

services:
  prometheus:
    image: quay.io/prometheus/prometheus:v2.52.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - prometheus-data:/prometheus
      - type: bind
        source: ./config/prometheus.yml
        target: /etc/prometheus/prometheus.yml
        read_only: true
  snmp:
    image: quay.io/prometheus/snmp-exporter:v0.26.0
    container_name: snmp
    ports:
      - "9116:9116"
    volumes:
      - type: bind
        source: ./config/snmp.yml
        target: /etc/snmp_exporter/snmp.yml
        read_only: true

volumes:
  prometheus-data: {}

test snmp on debian

https://wiki.debian.org/SNMP

Install snmp packages.

sudo apt update
sudo apt install snmp
sudo apt install snmpd

Edit snmpd configuration file at /etc/snmp/snmpd.conf.

• [x] defines sysLocation (1.3.6.1.2.1.6)
• [x] defines sysContact (1.3.6.1.2.1.4)
• [x] defines sysServices (1.3.6.1.2.1.7)
• [x] listens on all, 0.0.0.0
• [x] defines "view_all" view with access to everything under .1
• [x] defines "infra-nwk" read-only community with "view_all" access

snmpwalk -v2c -c infra-nwk 192.168.1.79 .1.3.6.1.2.1.1

```conf title="sudo cat /etc/snmp/snmpd.conf | grep -v "^#" | uniq"

sysLocation Sitting on the Dock of the Bay sysContact Me me@example.org

sysServices 72

master agentx

agentaddress 0.0.0.0,[::0]

view systemonly included .1.3.6.1.2.1.1 view systemonly included .1.3.6.1.2.1.25.1 view view_all included .1

rocommunity infra-nwk default -V view_all ```