Grafana-Prometheus를 사용한 server 모니터링

링크 : https://geekflare.com/prometheus-grafana-setup-for-linux/

• 개요.

– monitor대상서버에서
node_exporter을 통해 prometheus로 사용량정보를 전달.

– grafana는 데이터를 받아 그래프 및 대시보드를 표출해주는 기능. Prometheus는 node_exporter을 통해 사용량정보를 전달받아 grafana를통해 대시보드에 표시해주는
역할

– monitor 대상서버에서 나가는 트래픽으로 일반적으로 아웃바운드 방화벽은 열려있기 때문에 별도의 방화벽설정이 필요없음.

– grafana/prometheus 서버에서 node_exporter 포트인 9100을 통해 받으면 됨.

– grafana default port : 3000, Prometheus default port : 9090

• grafana/prometheus
  서버

– prometheus 설치/설정

$
wget https://github.com/prometheus/node_exporter/releases/download/v1.0.0-rc.1/node_exporter-1.0.0-rc.1.linux-amd64.tar.gz

$ tar -xzf prometheus-2.18.1.linux-amd64.tar.gz

$
sudo useradd -s /sbin/false prometheus

$
sudo chmod 755 prometheus-2.18.1.linux-amd64 -R

$
sudo chown prometheus:prometheus prometheus-2.18.1.linux-amd64 -R

$ sudo vi /etc/systemd/system/prometheus.service

[Unit]

Description=Prometheus
Server

Wants=network-online.target

After=network-online.target

[Service]

User=prometheus

Group=prometheus

Type=simple

ExecStart=/home/centos/prometheus-2.18.1.linux-amd64/prometheus

–config.file=
/home/centos/prometheus-2.18.1.linux-amd64/prometheus.yml

–storage.tsdb.path=/home/centos/prometheus-2.18.1.linux-amd64/

–web.console.templates=/home/centos/prometheus-2.18.1.linux-amd64/consoles

–web.console.libraries=/home/centos/prometheus-2.18.1.linux-amd64/console_libraries

[Install]

WantedBy=multi-user.target

=> rpm설치가 아니라 tar 압축 풀면 binary 파일이 바로 실행됨.

$
sudo systemctl daemon-reload

$
sudo systemctl start prometheus.service

$
sudo systemctl enable prometheus.service

$
sudo systemctl status prometheus.service -l

check prometheus web ui -> localhost:9090

– grafana 설치/설정

$ sudo vi /etc/yum.repos.d/grafana.repo

[grafana]

name=grafana

baseurl=https://packages.grafana.com/oss/rpm

repo_gpgcheck=1

enabled=1

gpgcheck=1

gpgkey=https://packages.grafana.com/gpg.key

sslverify=1

sslcacert=/etc/pki/tls/certs/ca-bundle.crt

$
sudo yum update -y

$
sudo yum install grafana

$
sudo systemctl start grafana-server

$
sudo systemctl status grafana-server

check grafana web ui -> localhost:3000 , default user :
admin/admin

-> admin / Hazel1234

• Node Exporter, 모니터링 대상서버 설치

$
wget https://github.com/prometheus/node_exporter/releases/download/v1.0.0-rc.1/node_exporter-1.0.0-rc.1.linux-amd64.tar.gz

-> 버전 이슈로 node exporter / promethous 최신버전으로 변경

https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz

https://github.com/prometheus/prometheus/releases/download/v2.44.0/prometheus-2.44.0.linux-amd64.tar.gz

$
tar -xzf node_exporter-1.0.0-rc.1.linux-amd64.tar.gz

$ sudo vi /etc/systemd/system/node_exporter.service

[Unit]

Description=node_exporter

Wants=network-online.target

After=network-online.target

[Service]

User=prometheus

Group=prometheus

Type=simple

ExecStart=/home/centos/node_exporter-1.0.0-rc.1.linux-amd64/node_exporter

[Install]

WantedBy=multi-user.target

=> rpm설치가 아니라 tar 압축 풀면 binary 파일이 바로 실행됨.

$
sudo systemctl daemon-reload

$
sudo systemctl start node_exporter

$
sudo systemctl status node_exporter

=> node_exporter 실행파일 user:group 변경필요.

• Adding Server to Prometheus

– grafana/prometheus 서버에서 Prometheus 설정에서 모니터링 대상서버의 node_exporter을 추가

– hosts 파일에 등록하여 hostname로 등록할 수도 있음.(서버 구분을 위해 필요.)

$ sudo vi prometheus.yml
-> prometheus 설치경로의 설정파일에 하기 정보 추가.

–
job_name: ‘node_exporter’

static_configs:

– targets: [‘client-ip:9100’]

$
sudo systemctl stop prometheus.service

$
sudo systemctl start prometheus.service

– https://localhost:9090/targets 에서 추가된 node_exporter 등록확인

– Grafana WEB UI에서
Prometheus 대시보드 추가

Home > Administration > Data Sources > “+ Add new data
source” -> Prometheus -> Save & test

– Dashboards에 Node
Exporter Full import하여 표시.

 Home > Dashboards > New
> Import > “Import via grafana.com” > “1860” Load

**https://grafana.com/grafana/dashboards/11074-node-exporter-for-prometheus-dashboard-en-v20201010/

11074번 대시보드 추천

• Mysql
  DB 등록

– 참조 : https://grafana.com/oss/prometheus/exporters/mysql-exporter/?tab=installation

– mysqld_exporter 설치
-> prometheus와 같이 작동

$ wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.12.1/mysqld_exporter-0.12.1.linux-amd64.tar.gz

$ tar -xvfz mysqld_exporter-*.*-amd64.tar.gz

$ cd mysqld_exporter-*.*-amd64

$ mysql -u root -p

use mysql;

CREATE USER ‘exporter’@’localhost’ IDENTIFIED BY ‘password’;

GRANT
PROCESS, REPLICATION CLIENT, SELECT ON *.* TO ‘exporter’@’localhost’;

exit

$ export DATA_SOURCE_NAME=’exporter:password@(192.168.0.101:3306)/’

$ ./mysqld_exporter

$ curl http://localhost:9104/metrics

– prometheus yml파일 등록

$ vi prometheus.yml

–
job_name: mysql

  static_configs:

  – targets: [‘192.168.0.101:9104’]

– prometheus yml파일에
mysql rule 파일 생성/등록 (해당파일은 prometheus 디렉토리내 생성)

$ vi recording_rules.yaml

groups:

–
name: mysqld_rules

  rules:

  – expr:
mysql_slave_status_seconds_behind_master – mysql_slave_status_sql_delay

    record: instance:mysql_slave_lag_seconds

  – expr: mysql_heartbeat_now_timestamp_seconds
– mysql_heartbeat_stored_timestamp_seconds

    record:
instance:mysql_heartbeat_lag_seconds

  – expr: sum without (command)
(rate(mysql_global_status_commands_total{command=~”(commit|rollback)”}[5m]))

    record: job:mysql_transactions:rate5m

$ chown prometheus:prometheus
recording_rules.yaml

$ vi rometheus.yml

rule_files:

  # – “first_rules.yml”

  # – “second_rules.yml”

  – “recording_rules.yaml”

– Dashboard 등록

“https://grafana.com/grafana/dashboards/14057-mysql/” 의 Download JSON 파일 New/import해서 등록

– mysqld exporter systemctl 등록

$sudo vi /etc/systemd/system/mysqld_exporter.service

[Unit]

Description=mysqld
exporter

Wants=network-online.target

After=network-online.target

[Service]

User=prometheus

Group=prometheus

Environment=DATA_SOURCE_NAME=’exporter:password@(192.168.0.101:3306)/’

Type=simple

ExecStart=/home/prometheus/mysqld_exporter-0.12.1.linux-amd64/mysqld_exporter

[Install]

WantedBy=multi-user.target

$ systemctl daemon-reload

$ systemctl start mysqld_exporter.service

$ systemctl status mysqld_exporter.service

$ systemctl enable mysqld_exporter.service