RabbitMQ on K8s helm部署实战

• • 获取helm chart
  • 修改必要参数
  • 外部如何访问？
  • 安装rabbitmq
  • 登录管理界面
  • 导入mq metadata
  • 优化
  • • helm chart 生成的statefuleset yaml
    • helm chart 生成的pod yaml
    • 调整mq log等级
    • promethrus 监控mq
    • • 配置prometheus采集metrics
      • 查看prometheus rabbitmq target status
      • 单独启动rabbitmq_export
      • rabbitmq_exporter 开机自启动
      • Q深度监控
    • 内存优化
    • 修改集群名称
    • 高可用方案
    • • 镜像模式
      • 查看高可用策略
      • ha-mode含义
      • ha-sync-mode的含义
      • 设定自动同步的命令
    • 域名访问
  • 压测
  • • 官方的perf-test
    • curl 调用API + 脚本压测
    • rabbitmqadmin publish + 脚本压测
  • 问题
  • • prometheus status正常为何dashboard抓取不到metrics？
    • Q数量太大导致监控卡顿
    • rabbitmq pod都无法启动时，重新安装之后保证数据正常
    • 容器没有root权限
  • Why use a non-root container?
  • • 重启pod报错
    • 如何debug ？
  • 参考

获取helm chart

chart选择bitnami公司制作的。

找一台有外网的机器

1、helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories2、helm pull bitnami/rabbitmq⚡ root@localhost  ~  ll
-rw-r--r--. 1 root root  54K Oct 11 17:38 rabbitmq-10.3.9.tgz

修改必要参数

• image

  10.50.10.185/rabbitmq/docker.io/bitnami/rabbitmq:3.10.8-debian-11-r4

  10.50.10.185/kafka/docker.io/bitnami/bitnami-shell:11-debian-11-r22

• storageClass 使用nfs存储类

• persistence.mountPath 默认的持久化存储/bitnami/rabbitmq/mnesia

外部如何访问？

默认使用的clusterIP，只能在集群内部使用，如何暴露在内网局域网使用呢？

只需要将我们ap使用的amqp和管理界面的端口进行nodeport暴露即可。

 ## Service ports## @param service.ports.amqp Amqp service port## @param service.ports.amqpTls Amqp TLS service port## @param service.ports.dist Erlang distribution service port## @param service.ports.manager RabbitMQ Manager service port## @param service.ports.metrics RabbitMQ Prometheues metrics service port## @param service.ports.epmd EPMD Discovery service port##ports:amqp: 5672amqpTls: 5671dist: 25672manager: 15672metrics: 9419epmd: 4369## Node ports to expose## @param service.nodePorts.amqp Node port for Ampq## @param service.nodePorts.amqpTls Node port for Ampq TLS## @param service.nodePorts.dist Node port for Erlang distribution## @param service.nodePorts.manager Node port for RabbitMQ Manager## @param service.nodePorts.epmd Node port for EPMD Discovery## @param service.nodePorts.metrics Node port for RabbitMQ Prometheues metrics##nodePorts:amqp: "30100"amqpTls: ""dist: ""manager: "30101"metrics: "30102"epmd: ""

安装rabbitmq

安装完毕后会有如下提示，根据提示可以尝试访问访问amqp和manager管理界面.

 /opt/helm/rabbitmq]#helm install chot-rabbitmq -n rabbitmq /opt/helm/rabbitmq/
NAME: chot-rabbitmq
LAST DEPLOYED: Wed Oct 12 10:16:52 2022
NAMESPACE: rabbitmq
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: rabbitmq
CHART VERSION: 10.3.9
APP VERSION: 3.10.8** Please be patient while the chart is being deployed **Credentials:echo "Username      : user"echo "Password      : $(kubectl get secret --namespace rabbitmq chot-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 -d)"echo "ErLang Cookie : $(kubectl get secret --namespace rabbitmq chot-rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 -d)"Note that the credentials are saved in persistent volume claims and will not be changed upon upgrade or reinstallation unless the persistent volume claim has been deleted. If this is not the first installation of this chart, the credentials may not be valid.
This is applicable when no passwords are set and therefore the random password is autogenerated. In case of using a fixed password, you should specify it when upgrading.
More information about the credentials may be found at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases.RabbitMQ can be accessed within the cluster on port 15673 at chot-rabbitmq.rabbitmq.svc.cluster.localTo access for outside the cluster, perform the following steps:Obtain the NodePort IP and ports:export NODE_IP=$(kubectl get nodes --namespace rabbitmq -o jsonpath="{.items[0].status.addresses[0].address}")export NODE_PORT_AMQP=$(kubectl get --namespace rabbitmq -o jsonpath="{.spec.ports[?(@.name=='amqp')].nodePort}" services chot-rabbitmq)export NODE_PORT_STATS=$(kubectl get --namespace rabbitmq -o jsonpath="{.spec.ports[?(@.name=='http-stats')].nodePort}" services chot-rabbitmq)To Access the RabbitMQ AMQP port:echo "URL : amqp://$NODE_IP:$NODE_PORT_AMQP/"To Access the RabbitMQ Management interface:echo "URL : http://$NODE_IP:$NODE_PORT_STATS/"To access the RabbitMQ Prometheus metrics, get the RabbitMQ Prometheus URL by running:kubectl port-forward --namespace rabbitmq svc/chot-rabbitmq 19419:19419 &echo "Prometheus Metrics URL: http://127.0.0.1:19419/metrics"Then, open the obtained URL in a browser.

也可以在安装时指定登录dashboard的账密

helm install chot-rabbitmq -n rabbitmq --set auth.username=admin,auth.password=admin@mq /opt/helm/rabbitmq/  

登录管理界面

http://10.50.10.31:30101/#/

首次登录如果没有指定账密则使用如下命令获取:

    echo "Username      : user"echo "Password      : $(kubectl get secret --namespace rabbitmq chot-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 -d)"

导入mq metadata

将测试区mq的定义导入新集群

优化

如果需要对rabbitmq 在k8s上的细节，分析helm chart 生成的yaml文件是一个好的途径。

helm chart 生成的statefuleset yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:annotations:meta.helm.sh/release-name: chot-rabbitmqmeta.helm.sh/release-namespace: rabbitmqlabels:app.kubernetes.io/instance: chot-rabbitmqapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: rabbitmqhelm.sh/chart: rabbitmq-10.3.9name: chot-rabbitmqnamespace: rabbitmqresourceVersion: '36898567'
spec:podManagementPolicy: OrderedReadyreplicas: 2revisionHistoryLimit: 10selector:matchLabels:app.kubernetes.io/instance: chot-rabbitmqapp.kubernetes.io/name: rabbitmqserviceName: chot-rabbitmq-headlesstemplate:metadata:annotations:checksum/config: 6bab8ac30e6ddbabb2cd23a7f08cba082380b36d7bbdaebd2b2b72b4a8c77b8bchecksum/secret: de0d40f863444485a2cc2386799625ed9afdffa30dc36fe0a2aecb5a1224ef93prometheus.io/port: '9419'prometheus.io/scrape: 'true'creationTimestamp: nulllabels:app.kubernetes.io/instance: chot-rabbitmqapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: rabbitmqhelm.sh/chart: rabbitmq-10.3.9spec:affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- podAffinityTerm:labelSelector:matchLabels:app.kubernetes.io/instance: chot-rabbitmqapp.kubernetes.io/name: rabbitmqnamespaces:- rabbitmqtopologyKey: kubernetes.io/hostnameweight: 1containers:- env:- name: BITNAMI_DEBUGvalue: 'false'- name: MY_POD_IPvalueFrom:fieldRef:apiVersion: v1fieldPath: status.podIP- name: MY_POD_NAMEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.name- name: MY_POD_NAMESPACEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namespace- name: K8S_SERVICE_NAMEvalue: chot-rabbitmq-headless- name: K8S_ADDRESS_TYPEvalue: hostname- name: RABBITMQ_FORCE_BOOTvalue: 'yes'- name: RABBITMQ_NODE_NAMEvalue: >-rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local- name: K8S_HOSTNAME_SUFFIXvalue: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local- name: RABBITMQ_MNESIA_DIRvalue: /bitnami/rabbitmq/mnesia/$(RABBITMQ_NODE_NAME)- name: RABBITMQ_LDAP_ENABLEvalue: 'no'- name: RABBITMQ_LOGSvalue: '-'- name: RABBITMQ_ULIMIT_NOFILESvalue: '65536'- name: RABBITMQ_USE_LONGNAMEvalue: 'true'- name: RABBITMQ_ERL_COOKIEvalueFrom:secretKeyRef:key: rabbitmq-erlang-cookiename: chot-rabbitmq- name: RABBITMQ_LOAD_DEFINITIONSvalue: 'no'- name: RABBITMQ_DEFINITIONS_FILEvalue: /app/load_definition.json- name: RABBITMQ_SECURE_PASSWORDvalue: 'yes'- name: RABBITMQ_USERNAMEvalue: guest- name: RABBITMQ_PASSWORDvalueFrom:secretKeyRef:key: rabbitmq-passwordname: chot-rabbitmq- name: RABBITMQ_PLUGINSvalue: >-rabbitmq_management, rabbitmq_peer_discovery_k8s,rabbitmq_auth_backend_ldap, rabbitmq_prometheusimage: '10.50.10.185/rabbitmq/docker.io/bitnami/rabbitmq:3.10.8-debian-11-r4'imagePullPolicy: IfNotPresentlifecycle:preStop:exec:command:- /bin/bash- '-ec'- >if [[ -f /opt/bitnami/scripts/rabbitmq/nodeshutdown.sh ]];then/opt/bitnami/scripts/rabbitmq/nodeshutdown.sh -t "120" -d "false"elserabbitmqctl stop_appfilivenessProbe:exec:command:- /bin/bash- '-ec'- rabbitmq-diagnostics -q pingfailureThreshold: 6initialDelaySeconds: 120periodSeconds: 30successThreshold: 1timeoutSeconds: 20name: rabbitmqports:- containerPort: 5672name: amqpprotocol: TCP- containerPort: 25672name: distprotocol: TCP- containerPort: 15672name: statsprotocol: TCP- containerPort: 4369name: epmdprotocol: TCP- containerPort: 9419name: metricsprotocol: TCPreadinessProbe:exec:command:- /bin/bash- '-ec'- >-rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics-q check_local_alarmsfailureThreshold: 3initialDelaySeconds: 10periodSeconds: 30successThreshold: 1timeoutSeconds: 20resources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /bitnami/rabbitmq/confname: configuration- mountPath: /bitnami/rabbitmq/mnesianame: datadnsPolicy: ClusterFirstinitContainers:- args:- '-ec'- >mkdir -p "/bitnami/rabbitmq/mnesia"chown "0:1001" "/bitnami/rabbitmq/mnesia"find "/bitnami/rabbitmq/mnesia" -mindepth 1 -maxdepth 1 -not -name".snapshot" -not -name "lost+found" | \xargs -r chown -R "0:1001"command:- /bin/bashimage: '10.50.10.185/kafka/docker.io/bitnami/bitnami-shell:11-debian-11-r22'imagePullPolicy: IfNotPresentname: volume-permissionsresources: {}securityContext:runAsUser: 0terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /bitnami/rabbitmq/mnesianame: datarestartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}serviceAccount: chot-rabbitmqserviceAccountName: chot-rabbitmqterminationGracePeriodSeconds: 120volumes:- name: configurationsecret:defaultMode: 420items:- key: rabbitmq.confpath: rabbitmq.confsecretName: chot-rabbitmq-configupdateStrategy:type: RollingUpdatevolumeClaimTemplates:- apiVersion: v1kind: PersistentVolumeClaimmetadata:creationTimestamp: nulllabels:app.kubernetes.io/instance: chot-rabbitmqapp.kubernetes.io/name: rabbitmqname: dataspec:accessModes:- ReadWriteOnceresources:requests:storage: 200GistorageClassName: nfs-storage-179scvolumeMode: Filesystemstatus:phase: Pending

helm chart 生成的pod yaml

---
apiVersion: v1
kind: Pod
metadata:annotations:checksum/config: 601d28e7aba31d1937034ce699a1ab755ca9ba176e13c0033b78fcfce55d4b48checksum/secret: d8c9218d24755350e7f93e4906bee00014b369ec3507bc7632d4cc7486b475a3cni.projectcalico.org/containerID: 08d2ac4cb82eba9fad8d989d532e99b3e9f0390fc5b169e337de67c620861f6fcni.projectcalico.org/podIP: 10.244.104.45/32cni.projectcalico.org/podIPs: 10.244.104.45/32prometheus.io/port: '9419'prometheus.io/scrape: 'true'generateName: chot-rabbitmq-dev-labels:app.kubernetes.io/instance: chot-rabbitmq-devapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: rabbitmqcontroller-revision-hash: chot-rabbitmq-dev-95c8c5876helm.sh/chart: rabbitmq-10.3.9statefulset.kubernetes.io/pod-name: chot-rabbitmq-dev-1name: chot-rabbitmq-dev-1namespace: rabbitmq-devownerReferences:- apiVersion: apps/v1blockOwnerDeletion: truecontroller: truekind: StatefulSetname: chot-rabbitmq-devuid: 0dcbd9a1-cc16-49f5-974e-f59f88ced47cresourceVersion: '29760134'
spec:affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- podAffinityTerm:labelSelector:matchLabels:app.kubernetes.io/instance: chot-rabbitmq-devapp.kubernetes.io/name: rabbitmqnamespaces:- rabbitmq-devtopologyKey: kubernetes.io/hostnameweight: 1containers:- env:- name: BITNAMI_DEBUGvalue: 'false'- name: MY_POD_IPvalueFrom:fieldRef:apiVersion: v1fieldPath: status.podIP- name: MY_POD_NAMEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.name- name: MY_POD_NAMESPACEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namespace- name: K8S_SERVICE_NAMEvalue: chot-rabbitmq-dev-headless- name: K8S_ADDRESS_TYPEvalue: hostname- name: RABBITMQ_FORCE_BOOTvalue: 'yes'- name: RABBITMQ_NODE_NAMEvalue: >-rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local- name: K8S_HOSTNAME_SUFFIXvalue: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local- name: RABBITMQ_MNESIA_DIRvalue: /bitnami/rabbitmq/mnesia/$(RABBITMQ_NODE_NAME)- name: RABBITMQ_LDAP_ENABLEvalue: 'no'- name: RABBITMQ_LOGSvalue: '-'- name: RABBITMQ_ULIMIT_NOFILESvalue: '65536'- name: RABBITMQ_USE_LONGNAMEvalue: 'true'- name: RABBITMQ_ERL_COOKIEvalueFrom:secretKeyRef:key: rabbitmq-erlang-cookiename: chot-rabbitmq-dev- name: RABBITMQ_LOAD_DEFINITIONSvalue: 'no'- name: RABBITMQ_DEFINITIONS_FILEvalue: /app/load_definition.json- name: RABBITMQ_SECURE_PASSWORDvalue: 'yes'- name: RABBITMQ_USERNAMEvalue: guest- name: RABBITMQ_PASSWORDvalueFrom:secretKeyRef:key: rabbitmq-passwordname: chot-rabbitmq-dev- name: RABBITMQ_PLUGINSvalue: >-rabbitmq_management, rabbitmq_peer_discovery_k8s,rabbitmq_auth_backend_ldap, rabbitmq_prometheusimage: '10.50.10.185/rabbitmq/docker.io/bitnami/rabbitmq:3.10.8-debian-11-r4'imagePullPolicy: IfNotPresentlifecycle:preStop:exec:command:- /bin/bash- '-ec'- |if [[ -f /opt/bitnami/scripts/rabbitmq/nodeshutdown.sh ]]; then/opt/bitnami/scripts/rabbitmq/nodeshutdown.sh -t "120" -d "false"elserabbitmqctl stop_appfilivenessProbe:exec:command:- /bin/bash- '-ec'- rabbitmq-diagnostics -q pingfailureThreshold: 6initialDelaySeconds: 120periodSeconds: 30successThreshold: 1timeoutSeconds: 20name: rabbitmqports:- containerPort: 5672name: amqpprotocol: TCP- containerPort: 25672name: distprotocol: TCP- containerPort: 15672name: statsprotocol: TCP- containerPort: 4369name: epmdprotocol: TCP- containerPort: 9419name: metricsprotocol: TCPreadinessProbe:exec:command:- /bin/bash- '-ec'- >-rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -qcheck_local_alarmsfailureThreshold: 3initialDelaySeconds: 10periodSeconds: 30successThreshold: 1timeoutSeconds: 20resources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /bitnami/rabbitmq/confname: configuration- mountPath: /bitnami/rabbitmq/mnesianame: data- mountPath: /var/run/secrets/kubernetes.io/serviceaccountname: kube-api-access-x25nzreadOnly: truednsPolicy: ClusterFirstenableServiceLinks: truehostname: chot-rabbitmq-dev-1initContainers:- args:- '-ec'- >mkdir -p "/bitnami/rabbitmq/mnesia"chown "0:1001" "/bitnami/rabbitmq/mnesia"find "/bitnami/rabbitmq/mnesia" -mindepth 1 -maxdepth 1 -not -name".snapshot" -not -name "lost+found" | \xargs -r chown -R "0:1001"command:- /bin/bashimage: '10.50.10.185/kafka/docker.io/bitnami/bitnami-shell:11-debian-11-r22'imagePullPolicy: IfNotPresentname: volume-permissionsresources: {}securityContext:runAsUser: 0terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /bitnami/rabbitmq/mnesianame: data- mountPath: /var/run/secrets/kubernetes.io/serviceaccountname: kube-api-access-x25nzreadOnly: truenodeName: node2nodeSelector:nodemq: devpreemptionPolicy: PreemptLowerPrioritypriority: 0restartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}serviceAccount: chot-rabbitmq-devserviceAccountName: chot-rabbitmq-devsubdomain: chot-rabbitmq-dev-headlessterminationGracePeriodSeconds: 120tolerations:- effect: NoExecutekey: node.kubernetes.io/not-readyoperator: ExiststolerationSeconds: 300- effect: NoExecutekey: node.kubernetes.io/unreachableoperator: ExiststolerationSeconds: 300volumes:- name: datapersistentVolumeClaim:claimName: data-chot-rabbitmq-dev-1- name: configurationsecret:defaultMode: 420items:- key: rabbitmq.confpath: rabbitmq.confsecretName: chot-rabbitmq-dev-config- name: kube-api-access-x25nzprojected:defaultMode: 420sources:- serviceAccountToken:expirationSeconds: 3607path: token- configMap:items:- key: ca.crtpath: ca.crtname: kube-root-ca.crt- downwardAPI:items:- fieldRef:apiVersion: v1fieldPath: metadata.namespacepath: namespace

调整mq log等级

生产环境mq负载大，打印info层级log并没有用，建议loglevel开到error等级

extraEnvVars:- name: LOG_LEVELvalue: error

promethrus 监控mq

bitnima已经帮我们把prometheus的监控做好了，只需要配置即可。enable即可，如果需要其他细节，例如读个实例如何处理？serviceMonitor？阅读bitnima的文档完成。

配置prometheus采集metrics

使用file_sd文件动态发现目标节点

]#cat /etc/prometheus/prometheus.yml
############################################################################
# job: rabbitmq
# path: targets/rabbitmq/*.yaml
# date: 2022-10-12
############################################################################- job_name: rabbitmqmetrics_path: /metricsfile_sd_configs:- refresh_interval: 10sfiles: [ targets/rabbitmq/*.yml ]]#cat /etc/prometheus/targets/rabbitmq/chot0rabbitmq.yml
- labels: { ip: 10.50.10.31 , ins: rabbitmq-1 , cls: rabbitmq }targets: [ 10.50.10.31:30102 ]

上面的文件发现模块为何无法在首页注册一个panel呢？

修改为直接获取rabbitmq cluster name

  targets: [ '10.50.10.31:30102','10.50.10.151:9090','10.50.10.152:9090','10.50.10.155:9090','10.50.10.156:9090' ]

查看prometheus rabbitmq target status

单独启动rabbitmq_export

上述内容都是基于 3.8 版本及其以后的版本，对于 3.8 版本之前的版本可以使用单独的插件 prometheus_rabbitmq_exporter来暴露指标数据。该插件借助的是 RabbitMQ HTTP API来实现的。

# 1. 下载exporter
https://github.com/kbudde/rabbitmq_exporter/releases/download/v1.0.0-RC19/rabbitmq_exporter_1.0.0-RC19_linux_amd64.tar.gz
tar -zxvf  rabbitmq_exporter_1.0.0-RC19_linux_amd64.tar.gz
# 2. 生成配置文件 rabbitmq.json
{"rabbit_url": "http://127.0.0.1:15672","rabbit_user": "guest","rabbit_pass": "guest","publish_port": "9419","publish_addr": "","output_format": "TTY","ca_file": "ca.pem","cert_file": "client-cert.pem","key_file": "client-key.pem","insecure_skip_verify": false,"exlude_metrics": [],"include_exchanges": ".*","skip_exchanges": "^$","include_queues": ".*","skip_queues": "^$","skip_vhost": "^$","include_vhost": ".*","rabbit_capabilities": "no_sort,bert","aliveness_vhost": "/","enabled_exporters": ["exchange","node","overview","queue","aliveness"],"timeout": 30,"max_queues": 0
}
# 3.启动
cp rabbitmq_exporter /usr/bin/ && nohup rabbitmq_exporter  -config-file conf/rabbitmq.json &

注意: 第二个不是一个二进制文件，如果是校验和那应该不至于那么大.

rabbitmq_exporter 开机自启动

os version : redhat 6.6

[root@P1QMSPL1RTM01 init.d]# more rabbitmq_exporter
#!/bin/shServName=rabbitmq_exporter
PidFile=/var/run/$ServName.pid
ServPath=/usr/local/exporter/$ServName/$ServName
ServConf=/usr/local/public/$SerName.yml
if [ -f /etc/init.d/functions ]; then. /etc/init.d/functions
fi
start() {if [ -s $PidFile ]; thenecho "Service $ServName is running"elseRABBIT_CAPABILITIES=nobert RABBIT_USER=guest RABBIT_PASSWORD=guest OUTPUT_FORMAT=JSON RABBIT_URL=http://localhost:15672 nohup $ServPath &ServPid=`ps -ef | grep $ServPath | grep -v 'grep' | awk '{print $2}'`touch $PidFileecho $ServPid > $PidFileecho "Starting $ServName: [ OK ]"fi
}
stop() {if [ -s $PidFile ]; thencat $PidFile | xargs killrm -f $PidFileecho "Stopping $ServName: [ OK ]"elseecho "Service $ServName not running"fi
}case $1 instart)start;;stop)stop;;restart)stopsleep 1start;;status)status -p $PidFile $ServNameexit $?;;*)echo "Usage: /etc/init.d/$ServName {start|stop|restart|status}" >&2;;
esac

Q深度监控

业务中对Q深度监控的需求较多，可以使用该指标进行报警。

内存优化

memoryHighWatermark.value 默认为0.4. 修改为0.85。 MQ 中的消息消费的时候是会先加载到内存中的，接受消息的时候同样也会。
增大此参数避免OOM

修改集群名称

1.查看集群名称
rabbitmq-diagnostics -q cluster_status | grep "Cluster name"
2. 修改集群名称
rabbitmqctl -q set_cluster_name cls-name

高可用方案

镜像节点在集群中的其他节点拥有从队列拷贝，一旦主节点不可用，最老的从队列将被选举为新的主队列。但镜像队列不能作为负载均衡使用，因为每个操作在所有节点都要做一遍。该模式带来的副作用也很明显，除了降低系统性能外，如果镜像队列数量过多，加之大量的消息进入，集群内部的网络带宽将会被这种同步通讯大大消耗掉。所以在对可靠性要求较高的场合中适用。

镜像模式

将需要消费的队列变为镜像队列，存在于多个节点，这样就可以实现 RabbitMQ 的 HA 高可用性。作用就是消息实体会主动在镜像节点之间实现同步，而不是像普通模式那样，在 consumer 消费数据时临时读取。缺点就是，集群内部的同步通讯会占用大量的网络带宽。

• sts 扩容

一键扩容

查看高可用策略

rabbitmqctl list_policies

ha-mode含义

ha-mode：策略键
1.all 队列镜像在群集中的所有节点上。当新节点添加到群集时，队列将镜像到该节点
2.exactly 集群中的队列实例数。
3.nodes 队列镜像到节点名称中列出的节点。

ha-sync-mode的含义

1.manual手动<默认模式>.新的队列镜像将不会收到现有的消息，它只会接收新的消息。

2.automatic自动同步.当一个新镜像加入时，队列会自动同步。队列同步是一个阻塞操作。

设定自动同步的命令

rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all" , "ha-sync-mode":"automatic"}'

这些操作在dashboard都可以完成。

域名访问

• 增加ingress 使用域名访问

压测

官方的perf-test

由于bitnami 的容器是no-rootusert 且修改chart的中value.yaml 的参数无效。

Why use a non-root container?

Non-root container images add an extra layer of security and are generally recommended for production environments. However, because they run as a non-root user, privileged tasks are typically off-limits. Learn more about non-root containers in our docs.

我也提了一个issue:

https://github.com/bitnami/charts/issues/13017

curl 调用API + 脚本压测

#!/bin/bash
#############################################################################################################
# NAME........... publish_rmq.sh
# Project .......
# AUTHOR......... ninesun 
# DATE........... 2022年10月19日13:40:27 
# PURPOSE........ Push the NFS monitoring alerts to RMQ
# HISTORY ....... Created the initial draft
#
#############################################################################################################
#set your RMQ variables here
rmq_url='ip'
user="guest"
pass="guest"
exchange="test"echo '
{"properties":{"delivery_mode":2},"routing_key":"test","payload":"{\"alertnotes\":\"NFS Mount missing\",\"device\":\"'"$(hostname)"'\",\"servicename\":\"NFS Mount\",\"eventsource\":\"NFS monitoring\",\"message\":\"Docker NFS mount missing\",\"last_occurance\":\"'"$(date -u +'%Y-%m-%d %H:%M:%S')"'\",\"first_time\":\"'"$(date -u +'%Y-%m-%d %H:%M:%S')"'\",\"eventgroups\":\"IT-Ops\",\"severity\":\"3\",\"eventtype\": \"CREATE\",\"eventid\": \"123456789012345678901234\"
}",
"payload_encoding":"string"
}' >payload.json#POST/Publish it to RMQ # Note: RMQ api access should be enabled (port#15672)
curl -s -u ${user}:${pass} -H "Accept: application/json" -H "Content-Type:application/json" -X POST -d @payload.json http://${rmq_url}:15672/api/exchanges/%2f/${exchange}/publish

这个脚本测试curl 发消息没测试通，放弃了使用rabbitmqadmin。

rabbitmqadmin publish + 脚本压测

while循环处理。

while true;do cat payload.json | rabbitmqadmin -H 10.50.10.31 -P 30101 -q publish exchange=qmsExchange routing_key=test1 && sleep 0.5s done

后台开启子进程放大并发.

这种方式并发无异于自杀。 玩坏别找我。

while true;do( cat glass.json | rabbitmqadmin -H 10.50.10.31 -P 30101 -q publish exchange=qmsExchange routing_key=test1 && sleep 0.5s)&done

更优雅的并发应该配合管道开启多个子进程并行处理才稳妥

问题

prometheus status正常为何dashboard抓取不到metrics？

dashboard取的是metrics 的rabbitmq_cluster这个指标。

原因: rabbitmq 3.8 自带的api会收集上来这个指标

rabbitmq_identity_info{rabbitmq_node="rabbit@chot-rabbitmq-0.chot-rabbitmq-headless.rabbitmq.svc.cluster.local",rabbitmq_cluster="rabbit@P1QMSARC01",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-SQeZUBPG7atPfpwViPK6bw"} 1

使用自定义rabbitmq_export 的版本是v0.29 ，我猜可能是metrics已经变化了。后来经实际测试是mq版本的问题，升级exporter到最新版本也没有这个指标。

只能找mq版本对应的dashboard。所以会有两个mq监控dashboard

Q数量太大导致监控卡顿

Q数量多超过4500+，其实只需要关注几个必要的MQ 即可. 这里需要写一个正则去匹配需要的Qname

rabbitmq pod都无法启动时，重新安装之后保证数据正常

当rabbitmq启用持久化存储时，若rabbitmq所有pod同时宕机，将无法重新启动，因此有必要提前开启clustering.forceBoot

容器没有root权限

Why use a non-root container?

Non-root container images add an extra layer of security and are generally recommended for production environments. However, because they run as a non-root user, privileged tasks are typically off-limits. Learn more about non-root containers in our docs.

理论上修改container的runAsUser就可以 以root权限启动. 我需要root权限是因为需要在容器中安装压测软件。 特别是在别人build的镜像中安装软件是一件难事。 如果非要实现可以自己build镜像。

重启pod报错

模拟故障，直接删除了pod-0. 导致起不起来。

if is_boolean_yes "$RABBITMQ_FORCE_BOOT" && ! is_dir_empty "${RABBITMQ_DATA_DIR}/${RABBITMQ_NODE_NAME}"; then# ref: https://www.rabbitmq.com/rabbitmqctl.8.html#force_bootwarn "Forcing node to start..."debug_execute "${RABBITMQ_BIN_DIR}/rabbitmqctl" force_boot
fi

解决: https://stackoverflow.com/questions/60407082/rabbit-mq-error-while-waiting-for-mnesia-tables

helm upgrade rabbitmq --set clustering.forceBoot=true

The problem happens for the following reason:

• All RMQ pods are terminated at the same time due to some reason (maybe because you explicitly set the StatefulSet replicas to 0, or something else)
• One of them is the last one to stop (maybe just a tiny bit after the others). It stores this condition (“I’m standalone now”) in its filesystem, which in k8s is the PersistentVolume(Claim). Let’s say this pod is rabbitmq-1.
• When you spin the StatefulSet back up, the pod rabbitmq-0 is always the first to start (see here).
• During startup, pod rabbitmq-0 first checks whether it’s supposed to run standalone. But as far as it can see on its own filesystem, it’s part of a cluster. So it checks for its peers and doesn’t find any. This results in a startup failure by default. (应该就是这点启动失败)
• rabbitmq-0 thus never becomes ready.
• rabbitmq-1 is never starting because that’s how StatefulSets are deployed - one after another. If it were to start, it would start successfully because it sees that it can run standalone as well.

So in the end, it’s a bit of a mismatch between how RabbitMQ and StatefulSets work. RMQ says: “if everything goes down, just start everything and the same time, one will be able to start and as soon as this one is up, the others can rejoin the cluster.” k8s StatefulSets say: “starting everything all at once is not possible, we’ll start with the 0”.

Solution

To fix this, there is a force_boot command for rabbitmqctl which basically tells an instance to start standalone if it doesn’t find any peers. How you can use this from Kubernetes depends on the Helm chart and container you’re using. In the Bitnami Chart, which uses the Bitnami Docker image, there is a value clustering.forceBoot = true, which translates to an env variable RABBITMQ_FORCE_BOOT = yes in the container, which will then issue the above command for you.

But looking at the problem, you can also see why deleting PVCs will work (other answer). The pods will just all “forget” that they were part of a RMQ cluster the last time around, and happily start. I would prefer the above solution though, as no data is being lost.

如何debug ？

diagnosticMode.enabled=true

参考

【1】 csdn安装参考