测试开发技术网站

博客

基于Kubernetes平台部署Grafana Loki Promtail系统
作者：redrose2100   类别：    日期：2022-09-14 06:29:10    阅读：2867 次   消耗积分：0 分

【原文链接】基于Kubernetes平台部署Grafana Loki Promtail系统

本文详细介绍在Kubernetes环境下通过Grafana+Loki+Promtail搭建一套日志采集系统，用于采集在当前Kubernetes平台上部署的所有服务的日志，便于开发运维。

一、创建独立的命名空间

为日志系统创建一个独立的命令空间

  1. kubectl create ns logging

二、部署Grafana

2.1 编写grafana部署的配置文件

创建 grafana 目录，并在目录中创建 grafana-deploy.yaml 文件，内容如下：

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: grafana
  5.   labels:
  6.     app: grafana
  7.   namespace: logging
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: grafana
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: grafana
  17.     spec:
  18.       containers:
  19.       - name: grafana
  20.         image: grafana/grafana:8.4.7
  21.         imagePullPolicy: IfNotPresent
  22.         securityContext:
  23.           runAsUser: 0
  24.         env:
  25.         - name: GF_AUTH_BASIC_ENABLED
  26.           value: "true"
  27.         - name: GF_AUTH_ANONYMOUS_ENABLED
  28.           value: "false"
  29. #        resources:
  30. #          requests:
  31. #            cpu: 100m
  32. #            memory: 200Mi
  33. #          limits:
  34. #            cpu: '1'
  35. #            memory: 2Gi
  36.         readinessProbe:
  37.           httpGet:
  38.             path: /login
  39.             port: 3000
  40.         volumeMounts:
  41.         - name: storage
  42.           mountPath: /var/lib/grafana
  43.       volumes:
  44.       - name: storage
  45.         hostPath:
  46.           path: /hostpath/grafana
  47. ---
  48. apiVersion: v1
  49. kind: Service
  50. metadata:
  51.   name: grafana
  52.   labels:
  53.     app: grafana
  54.   namespace: logging
  55. spec:
  56.   type: NodePort
  57.   ports:
  58.   - port: 3000
  59.     targetPort: 3000
  60.     nodePort: 30200
  61.   selector:
  62.     app: grafana

上述配置修改注意点：
（1）上述有两处namespace，此处需要使用步骤一中创建的命名空间，这里是logging

（2）挂载的目录需要在服务器上创建并且设置为足够的访问权限，比如这里：

  1. mkdir -p /hostpath/grafana
  2. chmod 777 -R /hostpath/grafana

（3）对外开发端口可以自行设置，这里比如 30200

2.2 部署grafana

进入grafana目录，然后执行部署命令

  1. cd grafana
  2. kubectl apply -f grafana-deploy.yaml

三、部署 Loki

3.1 编写Loki部署的配置文件

创建 loki文件夹，然后再 loki 文件夹中创建三个配置文件，分别是：loki-rbac.yaml、loki-configmap.yaml和loki-statefulset.yaml

loki-rbac.yaml文件内容如下：

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name: logging
  5. ---
  6. apiVersion: v1
  7. kind: ServiceAccount
  8. metadata:
  9.   name: loki
  10.   namespace: logging
  11. ---
  12. apiVersion: rbac.authorization.k8s.io/v1
  13. kind: Role
  14. metadata:
  15.   name: loki
  16.   namespace: logging
  17. rules:
  18. - apiGroups: ["extensions"]
  19.   resources: ["podsecuritypolicies"]
  20.   verbs: ["use"]
  21.   resourceNames: [loki]
  22. ---
  23. apiVersion: rbac.authorization.k8s.io/v1
  24. kind: RoleBinding
  25. metadata:
  26.   name: loki
  27.   namespace: logging
  28. roleRef:
  29.   apiGroup: rbac.authorization.k8s.io
  30.   kind: Role
  31.   name: loki
  32. subjects:
  33. - kind: ServiceAccount
  34.   name: loki

上述配置修改注意点：
（1）只需保证其中的namespace雨步骤一中设置的一致即可，比如都是logging时，此文件不需要修改

loki-configmap.yaml文件的内容如下：

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: loki
  5.   namespace: logging
  6.   labels:
  7.     app: loki
  8. data:
  9.   loki.yaml: |
  10.     auth_enabled: false
  11.     ingester:
  12.       chunk_idle_period: 3m
  13.       chunk_block_size: 262144
  14.       chunk_retain_period: 1m
  15.       max_transfer_retries: 0
  16.       lifecycler:
  17.         ring:
  18.           kvstore:
  19.             store: inmemory
  20.           replication_factor: 1
  21.     limits_config:
  22.       enforce_metric_name: false
  23.       reject_old_samples: true
  24.       reject_old_samples_max_age: 168h
  25.     schema_config:
  26.       configs:
  27.       - from: "2022-05-15"
  28.         store: boltdb-shipper
  29.         object_store: filesystem
  30.         schema: v11
  31.         index:
  32.           prefix: index_
  33.           period: 24h
  34.     server:
  35.       http_listen_port: 3100
  36.     storage_config:
  37.       boltdb_shipper:
  38.         active_index_directory: /data/loki/boltdb-shipper-active
  39.         cache_location: /data/loki/boltdb-shipper-cache
  40.         cache_ttl: 24h
  41.         shared_store: filesystem
  42.       filesystem:
  43.         directory: /data/loki/chunks
  44.     chunk_store_config:
  45.       max_look_back_period: 0s
  46.     table_manager:
  47.       retention_deletes_enabled: true
  48.       retention_period: 48h
  49.     compactor:
  50.       working_directory: /data/loki/boltdb-shipper-compactor
  51.       shared_store: filesystem

上述配置修改注意点：
（1）namespace同样需要和步骤一中的设置保持一致

loki-statefulset.yaml文件的内容如下：

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: loki
  5.   namespace: logging
  6.   labels:
  7.     app: loki
  8. spec:
  9.   type: NodePort
  10.   ports:
  11.     - port: 3100
  12.       protocol: TCP
  13.       name: http-metrics
  14.       targetPort: http-metrics
  15.       nodePort: 30201
  16.   selector:
  17.     app: loki
  18. ---
  19. apiVersion: apps/v1
  20. kind: StatefulSet
  21. metadata:
  22.   name: loki
  23.   namespace: logging
  24.   labels:
  25.     app: loki
  26. spec:
  27.   podManagementPolicy: OrderedReady
  28.   replicas: 1
  29.   selector:
  30.     matchLabels:
  31.       app: loki
  32.   serviceName: loki
  33.   updateStrategy:
  34.     type: RollingUpdate
  35.   template:
  36.     metadata:
  37.       labels:
  38.         app: loki
  39.     spec:
  40.       serviceAccountName: loki
  41.       initContainers:
  42.       - name: chmod-data
  43.         image: busybox:1.28.4
  44.         imagePullPolicy: IfNotPresent
  45.         command: ["chmod","-R","777","/loki/data"]
  46.         volumeMounts:
  47.         - name: storage
  48.           mountPath: /loki/data
  49.       containers:
  50.         - name: loki
  51.           image: grafana/loki:2.3.0
  52.           imagePullPolicy: IfNotPresent
  53.           args:
  54.             - -config.file=/etc/loki/loki.yaml
  55.           volumeMounts:
  56.             - name: config
  57.               mountPath: /etc/loki
  58.             - name: storage
  59.               mountPath: /data
  60.           ports:
  61.             - name: http-metrics
  62.               containerPort: 3100
  63.               protocol: TCP
  64.           livenessProbe:
  65.             httpGet: 
  66.               path: /ready
  67.               port: http-metrics
  68.               scheme: HTTP
  69.             initialDelaySeconds: 45
  70.           readinessProbe:
  71.             httpGet: 
  72.               path: /ready
  73.               port: http-metrics
  74.               scheme: HTTP
  75.             initialDelaySeconds: 45
  76.           securityContext:
  77.             readOnlyRootFilesystem: true
  78.       terminationGracePeriodSeconds: 4800
  79.       volumes:
  80.         - name: config
  81.           configMap:
  82.             name: loki
  83.         - name: storage
  84.           hostPath:
  85.             path: /app/loki

上述配置修改注意点：
（1）同样，namespace需要和步骤一中设置为一致
（2）设置Loki对外开放的端口，比如这里设置为30201

3.2 部署 Loki

在loki目录中执行如下命令进行部署

  1. cd loki
  2. kubectl apply -f .

四、部署 promtail

4.1 编写配置文件

首先创建 promtail 文件夹，然后再文件夹中创建 promtail-rbac.yaml、promtail-configmap.yaml和promtail-daemonset.yaml三个文件

promtail-rbac.yaml 文件内容如下，同样，这里不需要修改什么配置，只需要保证namespace和步骤一中创建的一致即可。

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   name: loki-promtail
  5.   labels:
  6.     app: promtail
  7.   namespace: logging
  8. ---
  9. kind: ClusterRole
  10. apiVersion: rbac.authorization.k8s.io/v1
  11. metadata:
  12.   labels:
  13.     app: promtail
  14.   name: promtail-clusterrole
  15.   namespace: logging
  16. rules:
  17. - apiGroups: [""]
  18.   resources: ["nodes","nodes/proxy","services","endpoints","pods"]
  19.   verbs: ["get", "watch", "list"]
  20. ---
  21. kind: ClusterRoleBinding
  22. apiVersion: rbac.authorization.k8s.io/v1
  23. metadata:
  24.   name: promtail-clusterrolebinding
  25.   labels:
  26.     app: promtail
  27.   namespace: logging
  28. subjects:
  29.   - kind: ServiceAccount
  30.     name: loki-promtail
  31.     namespace: logging
  32. roleRef:
  33.   kind: ClusterRole
  34.   name: promtail-clusterrole
  35.   apiGroup: rbac.authorization.k8s.io

promtail-configmap.yaml 文件内容如下，此文件同样只需要保证namespace与步骤一中一致即可，不需要做其他修改

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: loki-promtail
  5.   namespace: logging
  6.   labels:
  7.     app: promtail
  8. data:
  9.   promtail.yaml: |
  10.     client:
  11.       backoff_config:
  12.         max_period: 5m 
  13.         max_retries: 10
  14.         min_period: 500ms
  15.       batchsize: 1048576
  16.       batchwait: 1s
  17.       external_labels: {}
  18.       timeout: 10s
  19.     positions:
  20.       filename: /run/promtail/positions.yaml
  21.     server:
  22.       http_listen_port: 3101
  23.     target_config:
  24.       sync_period: 10s
  25.     scrape_configs:
  26.     - job_name: kubernetes-pods-name
  27.       pipeline_stages:
  28.         - docker: {}
  29.       kubernetes_sd_configs:
  30.       - role: pod
  31.       relabel_configs:
  32.       - source_labels:
  33.         - __meta_kubernetes_pod_label_name
  34.         target_label: __service__
  35.       - source_labels:
  36.         - __meta_kubernetes_pod_node_name
  37.         target_label: __host__
  38.       - action: drop
  39.         regex: ''
  40.         source_labels:
  41.         - __service__
  42.       - action: labelmap
  43.         regex: __meta_kubernetes_pod_label_(.+)
  44.       - action: replace
  45.         replacement: $1
  46.         separator: /
  47.         source_labels:
  48.         - __meta_kubernetes_namespace
  49.         - __service__
  50.         target_label: job
  51.       - action: replace
  52.         source_labels:
  53.         - __meta_kubernetes_namespace
  54.         target_label: namespace
  55.       - action: replace
  56.         source_labels:
  57.         - __meta_kubernetes_pod_name
  58.         target_label: pod
  59.       - action: replace
  60.         source_labels:
  61.         - __meta_kubernetes_pod_container_name
  62.         target_label: container
  63.       - replacement: /var/log/pods/*$1/*.log
  64.         separator: /
  65.         source_labels:
  66.         - __meta_kubernetes_pod_uid
  67.         - __meta_kubernetes_pod_container_name
  68.         target_label: __path__
  69.     - job_name: kubernetes-pods-app
  70.       pipeline_stages:
  71.         - docker: {}
  72.       kubernetes_sd_configs:
  73.       - role: pod
  74.       relabel_configs:
  75.       - action: drop
  76.         regex: .+
  77.         source_labels:
  78.         - __meta_kubernetes_pod_label_name
  79.       - source_labels:
  80.         - __meta_kubernetes_pod_label_app
  81.         target_label: __service__
  82.       - source_labels:
  83.         - __meta_kubernetes_pod_node_name
  84.         target_label: __host__
  85.       - action: drop
  86.         regex: ''
  87.         source_labels:
  88.         - __service__
  89.       - action: labelmap
  90.         regex: __meta_kubernetes_pod_label_(.+)
  91.       - action: replace
  92.         replacement: $1
  93.         separator: /
  94.         source_labels:
  95.         - __meta_kubernetes_namespace
  96.         - __service__
  97.         target_label: job
  98.       - action: replace
  99.         source_labels:
  100.         - __meta_kubernetes_namespace
  101.         target_label: namespace
  102.       - action: replace
  103.         source_labels:
  104.         - __meta_kubernetes_pod_name
  105.         target_label: pod
  106.       - action: replace
  107.         source_labels:
  108.         - __meta_kubernetes_pod_container_name
  109.         target_label: container
  110.       - replacement: /var/log/pods/*$1/*.log
  111.         separator: /
  112.         source_labels:
  113.         - __meta_kubernetes_pod_uid
  114.         - __meta_kubernetes_pod_container_name
  115.         target_label: __path__
  116.     - job_name: kubernetes-pods-direct-controllers
  117.       pipeline_stages:
  118.         - docker: {}
  119.       kubernetes_sd_configs:
  120.       - role: pod
  121.       relabel_configs:
  122.       - action: drop
  123.         regex: .+
  124.         separator: ''
  125.         source_labels:
  126.         - __meta_kubernetes_pod_label_name
  127.         - __meta_kubernetes_pod_label_app
  128.       - action: drop
  129.         regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
  130.         source_labels:
  131.         - __meta_kubernetes_pod_controller_name
  132.       - source_labels:
  133.         - __meta_kubernetes_pod_controller_name
  134.         target_label: __service__
  135.       - source_labels:
  136.         - __meta_kubernetes_pod_node_name
  137.         target_label: __host__
  138.       - action: drop
  139.         regex: ''
  140.         source_labels:
  141.         - __service__
  142.       - action: labelmap
  143.         regex: __meta_kubernetes_pod_label_(.+)
  144.       - action: replace
  145.         replacement: $1
  146.         separator: /
  147.         source_labels:
  148.         - __meta_kubernetes_namespace
  149.         - __service__
  150.         target_label: job
  151.       - action: replace
  152.         source_labels:
  153.         - __meta_kubernetes_namespace
  154.         target_label: namespace
  155.       - action: replace
  156.         source_labels:
  157.         - __meta_kubernetes_pod_name
  158.         target_label: pod
  159.       - action: replace
  160.         source_labels:
  161.         - __meta_kubernetes_pod_container_name
  162.         target_label: container
  163.       - replacement: /var/log/pods/*$1/*.log
  164.         separator: /
  165.         source_labels:
  166.         - __meta_kubernetes_pod_uid
  167.         - __meta_kubernetes_pod_container_name
  168.         target_label: __path__
  169.     - job_name: kubernetes-pods-indirect-controller
  170.       pipeline_stages:
  171.         - docker: {}
  172.       kubernetes_sd_configs:
  173.       - role: pod
  174.       relabel_configs:
  175.       - action: drop
  176.         regex: .+
  177.         separator: ''
  178.         source_labels:
  179.         - __meta_kubernetes_pod_label_name
  180.         - __meta_kubernetes_pod_label_app
  181.       - action: keep
  182.         regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
  183.         source_labels:
  184.         - __meta_kubernetes_pod_controller_name
  185.       - action: replace
  186.         regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
  187.         source_labels:
  188.         - __meta_kubernetes_pod_controller_name
  189.         target_label: __service__
  190.       - source_labels:
  191.         - __meta_kubernetes_pod_node_name
  192.         target_label: __host__
  193.       - action: drop
  194.         regex: ''
  195.         source_labels:
  196.         - __service__
  197.       - action: labelmap
  198.         regex: __meta_kubernetes_pod_label_(.+)
  199.       - action: replace
  200.         replacement: $1
  201.         separator: /
  202.         source_labels:
  203.         - __meta_kubernetes_namespace
  204.         - __service__
  205.         target_label: job
  206.       - action: replace
  207.         source_labels:
  208.         - __meta_kubernetes_namespace
  209.         target_label: namespace
  210.       - action: replace
  211.         source_labels:
  212.         - __meta_kubernetes_pod_name
  213.         target_label: pod
  214.       - action: replace
  215.         source_labels:
  216.         - __meta_kubernetes_pod_container_name
  217.         target_label: container
  218.       - replacement: /var/log/pods/*$1/*.log
  219.         separator: /
  220.         source_labels:
  221.         - __meta_kubernetes_pod_uid
  222.         - __meta_kubernetes_pod_container_name
  223.         target_label: __path__
  224.     - job_name: kubernetes-pods-static
  225.       pipeline_stages:
  226.         - docker: {}
  227.       kubernetes_sd_configs:
  228.       - role: pod
  229.       relabel_configs:
  230.       - action: drop
  231.         regex: ''
  232.         source_labels:
  233.         - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
  234.       - action: replace
  235.         source_labels:
  236.         - __meta_kubernetes_pod_label_component
  237.         target_label: __service__
  238.       - source_labels:
  239.         - __meta_kubernetes_pod_node_name
  240.         target_label: __host__
  241.       - action: drop
  242.         regex: ''
  243.         source_labels:
  244.         - __service__
  245.       - action: labelmap
  246.         regex: __meta_kubernetes_pod_label_(.+)
  247.       - action: replace
  248.         replacement: $1
  249.         separator: /
  250.         source_labels:
  251.         - __meta_kubernetes_namespace
  252.         - __service__
  253.         target_label: job
  254.       - action: replace
  255.         source_labels:
  256.         - __meta_kubernetes_namespace
  257.         target_label: namespace
  258.       - action: replace
  259.         source_labels:
  260.         - __meta_kubernetes_pod_name
  261.         target_label: pod
  262.       - action: replace
  263.         source_labels:
  264.         - __meta_kubernetes_pod_container_name
  265.         target_label: container
  266.       - replacement: /var/log/pods/*$1/*.log
  267.         separator: /
  268.         source_labels:
  269.         - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
  270.         - __meta_kubernetes_pod_container_name
  271.         target_label: __path__

promtail-daemonset.yaml 配置文件的内容如下：

  1. apiVersion: apps/v1
  2. kind: DaemonSet
  3. metadata:
  4.   name: loki-promtail
  5.   namespace: logging
  6.   labels:
  7.     app: promtail
  8. spec:
  9.   selector:
  10.     matchLabels:
  11.       app: promtail
  12.   updateStrategy:
  13.     rollingUpdate:
  14.       maxUnavailable: 1
  15.     type: RollingUpdate
  16.   template:
  17.     metadata:
  18.       labels:
  19.         app: promtail
  20.     spec:
  21.       serviceAccountName: loki-promtail
  22.       containers:
  23.         - name: promtail
  24.           image: grafana/promtail:2.3.0
  25.           imagePullPolicy: IfNotPresent
  26.           args: 
  27.           - -config.file=/etc/promtail/promtail.yaml
  28.           - -client.url=http://192.168.16.40:30201/loki/api/v1/push
  29.           env: 
  30.           - name: HOSTNAME
  31.             valueFrom: 
  32.               fieldRef: 
  33.                 apiVersion: v1
  34.                 fieldPath: spec.nodeName
  35.           volumeMounts:
  36.           - mountPath: /etc/promtail
  37.             name: config
  38.           - mountPath: /run/promtail
  39.             name: run
  40.           - mountPath: /var/lib/docker/containers
  41.             name: docker
  42.             readOnly: true
  43.           - mountPath: /var/log/pods
  44.             name: pods
  45.             readOnly: true
  46.           ports:
  47.           - containerPort: 3101
  48.             name: http-metrics
  49.             protocol: TCP
  50.           securityContext:
  51.             readOnlyRootFilesystem: true
  52.             runAsGroup: 0
  53.             runAsUser: 0
  54.           readinessProbe:
  55.             failureThreshold: 5
  56.             httpGet:
  57.               path: /ready
  58.               port: http-metrics
  59.               scheme: HTTP
  60.             initialDelaySeconds: 10
  61.             periodSeconds: 10
  62.             successThreshold: 1
  63.             timeoutSeconds: 1
  64.       tolerations:
  65.       - effect: NoSchedule
  66.         key: node-role.kubernetes.io/master
  67.         operator: Exists
  68.       volumes:
  69.         - name: config
  70.           configMap:
  71.             name: loki-promtail
  72.         - name: run
  73.           hostPath:
  74.             path: /run/promtail
  75.             type: ""
  76.         - name: docker
  77.           hostPath:
  78.             path: /var/lib/docker/containers
  79.         - name: pods
  80.           hostPath:
  81.             path: /var/log/pods

上述配置修改注意点：
（1）同样namespace需要与步骤一中的保持一致
（2）如下位置需要修改为步骤三中配置的 Loki 的地址

4.2 部署 Promtail

进入 promtail 目录，然后执行部署命令即可

  1. cd promtail
  2. kubectl apply -f .

五、检查部署结果

5.1 Kubernetes后台检查Pod部署状态

执行如下命令即可查看pod状态，均为running时表示部署OK

  1. [root@master ~]# kubectl get pod -n logging
  2. NAME                       READY   STATUS    RESTARTS   AGE
  3. grafana-66496d957f-ngq5g   1/1     Running   0          21h
  4. loki-0                     1/1     Running   0          24h
  5. loki-promtail-8vjd6        1/1     Running   0          20h
  6. loki-promtail-gmr5f        1/1     Running   0          20h
  7. loki-promtail-svwtn        1/1     Running   0          20h
  8. loki-promtail-tnvr9        1/1     Running   0          20h
  9. [root@master ~]#

5.2 浏览器打开grafana界面如下，表示OK

比如 192.168.16.40:30200，如下表示部署OK

始终坚持开源开放共享精神，同时感谢您的充电鼓励和支持！
版权所有，转载本站文章请注明出处：redrose2100， http://blog.redrose2100.com/article/372
个人成就
  • 2022年 ： 371 篇 
  • 2023年 ： 211 篇 
  • 2024年 ： 31 篇 
  • 2025年 ： 0 篇 
  • 博客总数： 613 
  • 阅读总量： 720364 
出版书籍
测试开发技术全栈公众号
DevOps技术交流微信群
常用网站链接
文章分类
最新文章
最多阅读
特别推荐

版权所有 © devops-dev.com (2021-2023)

   苏公网安备32011502011999

苏ICP备2021008941号-2

友情链接

CSDN    Github    Gitee   

联系我们

北京★北京市中关村   微信：redrose2100
邮箱：hitredrose@163.com