slack 설정

• slack으로 내용을 받을 생각이다. slack app
  

• 앱을 설치해주고
  
• From scratch 로 설정해주고
  
• 이름 설정해주고
  
• Incoming Webhooks on
  
• add new Webhook to Workspace 에 추가해주면
  
• url을 생성하였다.

프로메테우스 설정

dnf -y install alertmanager

vi /usr/lib/systemd//system/alertmanager.service

변경전 : ExecStart=/usr/bin/alertmanager $ALERTMANAGER_OPTS

변경후 : ExecStart=/usr/bin/alertmanager $ALERTMANAGER_OPTS  --cluster.advertise-address=0.0.0.0:9093

mv /etc/prometheus/alertmanager.yml /etc/prometheus/alertmanager.yml.org

• 혹시몰라 원본 백업 후 작업

vi /etc/prometheus/alertmanager.yml

route:
  receiver: 'slack-notice'
  group_by: ['alertname', 'Service', 'Stage', 'Role']
  group_wait:      30s
  group_interval:  5m
  repeat_interval: 4h

receivers:
- name: 'slack-notice'
  slack_configs:
  - channel: '[슬랙채널이름]'
    api_url: '[webhook URL]'
    title: "CPU High Useage"
    text: "OMG"

vi /etc/prometheus/alert_rules.yml

groups:
- name: example
  rules:
  - alert: HignUseage
    expr: avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[1m]) * 100) < 30
    for: 1m
    annotations:
      summary: "High Usage on CPU"
      description: "desc"

vi /etc/prometheus/prometheus.yml

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - localhost:9093

systemctl restart prometheus alertmanager

• 잘연결이 되었다. inactive 상태로 양호하다

서버 과부하 시키기

• django1 서버에 과부하를 줄것이다.

yum install epel-release
yum install -y stress
stress -c 1

• 알람 온것을 확인!