k8s学习(二十九) RocketMq集群(双主双从-同步)-java黑洞网

本站消息

站长简介/公众号

出租广告位,需要合作请联系站长

我长得真不赖

550

文章

267028

访问

+关注

分类

暂无分类

日期归档

2024-11(1)

k8s学习(二十九) RocketMq集群(双主双从-同步)

发布于2021-05-29 22:53 阅读(548) 评论(0) 点赞(15) 收藏(3)

确认安装前已经安装好StorageClass，可参考：https://blog.csdn.net/u011943534/article/details/100887530

1、准备jdk镜像，并拷贝到服务器

docker pull java:8-alpine

2、制作rocketmq镜像
下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/
将”rocketmq-all-4.8.0-bin-release.zip”拷贝至服务器，解压
修改bin/runbroker.sh,将内存调整为适当（比如调小）

JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

编写Dockerfile-rocketmq:

FROM java:8-alpine
 
RUN rm -f /etc/localtime \
&& ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& echo "Asia/Shanghai" > /etc/timezone
 
ENV LANG en_US.UTF-8
 
ADD rocketmq-all-4.8.0-bin-release /usr/local/rocketmq-all-4.8.0-bin-release
RUN mv /usr/local/rocketmq-all-4.8.0-bin-release /usr/local/rocketmq-4.8.0 \
&& mkdir -p /data/rocketmq/store
 
CMD ["/bin/bash"]

docker build -f Dockerfile-rocketmq -t 172.16.10.190:80/library/rocketmq:4.8.0 .
docker push 172.16.10.190:80/library/rocketmq:4.8.0

3、制作nameserver镜像
使用第三步的rocketmq-all-4.8.0-bin-release
在conf/下添加namesrv.properties
内容为：

listenPort=20901

编写Dokerfile-namesrv:

FROM java:8-alpine
 
RUN rm -f /etc/localtime \
&& ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& echo "Asia/Shanghai" > /etc/timezone
 
ENV LANG en_US.UTF-8
 
ADD rocketmq-all-4.8.0-bin-release /usr/local/rocketmq-all-4.8.0-bin-release
RUN mv /usr/local/rocketmq-all-4.8.0-bin-release /usr/local/rocketmq-4.8.0
 
WORKDIR /usr/local/rocketmq-4.8.0
CMD ["/usr/local/rocketmq-4.8.0/bin/mqnamesrv","-c","/usr/local/rocketmq-4.8.0/conf/namesrv.properties"]

docker build -f Dockerfile-nameserver -t 172.16.10.190:80/library/rocketmq-namesrv:4.8.0 .
docker push 172.16.10.190:80/library/rocketmq-namesrv:4.8.0

4、制作rocketmq-ui镜像
下载源码：
git clone https://github.com/apache/rocketmq-externals.git
下载maven:
wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

tar -zvxf rocketmq-externals.tar.gz

将本目录下的”application.properties“中内容替换到rocketmq-externals/rocketmq-console/src/main/resources/application.properties中
编写Dockerfile-ui

FROM java:8-alpine
 
RUN rm -f /etc/localtime \
&& ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& echo "Asia/Shanghai" > /etc/timezone
 
ENV LANG en_US.UTF-8
ENV MAVEN_HOME /usr/local/maven
ENV PATH $PATH:$MAVEN_HOME/bin
 
ADD rocketmq-externals /usr/local/rocketmq-externals
ADD apache-maven-3.3.9 /usr/local/apache-maven-3.3.9
RUN mv /usr/local/apache-maven-3.3.9 /usr/local/maven
 
WORKDIR /usr/local/rocketmq-externals/rocketmq-console/
RUN mvn clean package -Dmaven.test.skip=true
 
WORKDIR /usr/local/rocketmq-externals/rocketmq-console/target
EXPOSE 8080
CMD ["nohup","java","-jar","rocketmq-console-ng-1.0.0.jar","&"]

5、准备configmap
broker-a.properties:

brokerClusterName=rocketmq-cluster
brokerName=broker-a
brokerId=0
namesrvAddr=mq-namesrv-0.mq-namesrv.default.svc.cluster.local:20901
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=20911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=300000
diskMaxUsedSpaceRatio=88
storePathRootDir=/data/rocketmq/store
maxMessageSize=65536
brokerRole=MASTER

broker-a-s.properties:

brokerClusterName=rocketmq-cluster
brokerName=broker-a
brokerId=1
namesrvAddr=mq-namesrv-0.mq-namesrv.default.svc.cluster.local:20901
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=20911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=300000
diskMaxUsedSpaceRatio=88
storePathRootDir=/data/rocketmq/store
maxMessageSize=65536
brokerRole=SLAVE
flushDiskType=SYNC_FLUSH

broker-b.properties:

brokerClusterName=rocketmq-cluster
brokerName=broker-b
brokerId=0
namesrvAddr=mq-namesrv-0.mq-namesrv.default.svc.cluster.local:20901
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=20911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=300000
diskMaxUsedSpaceRatio=88
storePathRootDir=/data/rocketmq/store
maxMessageSize=65536
brokerRole=MASTER
flushDiskType=SYNC_FLUSH

broker-b-s.properties:

brokerClusterName=rocketmq-cluster
brokerName=broker-b
brokerId=1
namesrvAddr=mq-namesrv-0.mq-namesrv.default.svc.cluster.local:20901
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=20911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=300000
diskMaxUsedSpaceRatio=88
storePathRootDir=/data/rocketmq/store
maxMessageSize=65536
brokerRole=SLAVE
flushDiskType=SYNC_FLUSH

根据以上文件生成configmap:

kubectl create configmap rocketmq-config --from-file=broker-a.properties --from-file=broker-b.properties --from-file=broker-a-s.properties --from-file=broker-b-s.properties  -n default

6、准备服务文件
broker-a-statefulset.yaml：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-a
  name: broker-a
  namespace: default
spec:
  type: NodePort
  ports:
  - port: 20911
    targetPort: 20911
    name: broker-port
    nodePort: 30911
  selector:
    app: broker-a
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-a
  namespace: default
spec:
  serviceName: broker-a
  replicas: 1
  selector:
    matchLabels:
      app: broker-a
  template:
    metadata:
     labels:
       app: broker-a
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - broker-a
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
       - name: harborsecret
      containers:
      - name: broker-a
        image: 172.16.10.190:80/library/rocketmq:4.8.0
        imagePullPolicy: Always
        command: ["sh","-c","/usr/local/rocketmq-4.8.0/bin/mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-a.properties"]
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /data/rocketmq
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /usr/local/rocketmq-4.8.0/conf/broker-a.properties
            subPath: broker-a.properties
          - name: time
            mountPath: /etc/localtime
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: rocketmq-config
      - name: time
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "course-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 4Gi

broker-a-s-statefulset.yaml：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-a-s
  name: broker-a-s
  namespace: default
spec:
  type: NodePort
  ports:
  - port: 20911
    targetPort: 20911
    name: broker-port
    nodePort: 30912
  selector:
    app: broker-a-s
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-a-s
  namespace: default
spec:
  serviceName: broker-a-s
  replicas: 1
  selector:
    matchLabels:
      app: broker-a-s
  template:
    metadata:
     labels:
       app: broker-a-s
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - broker-a-s
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
       - name: harborsecret
      containers:
      - name: broker-a-s
        image: 172.16.10.190:80/library/rocketmq:4.8.0
        imagePullPolicy: Always
        command: ["sh","-c","/usr/local/rocketmq-4.8.0/bin/mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-a-s.properties"]
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /data/rocketmq
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /usr/local/rocketmq-4.8.0/conf/broker-a-s.properties
            subPath: broker-a-s.properties
          - name: time
            mountPath: /etc/localtime
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: rocketmq-config
          items:
          - key: broker-a-s.properties
            path: broker-a-s.properties
      - name: time
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "course-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 4Gi

broker-b-statefulset.yaml：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-b
  name: broker-b
  namespace: default
spec:
  type: NodePort
  ports:
  - port: 20911
    targetPort: 20911
    name: broker-port
    nodePort: 30913
  selector:
    app: broker-b
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-b
  namespace: default
spec:
  serviceName: broker-b
  replicas: 1
  selector:
    matchLabels:
      app: broker-b
  template:
    metadata:
     labels:
       app: broker-b
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - broker-b
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
       - name: harborsecret
      containers:
      - name: broker-b
        image: 172.16.10.190:80/library/rocketmq:4.8.0
        imagePullPolicy: Always
        command: ["sh","-c","/usr/local/rocketmq-4.8.0/bin/mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-b.properties"]
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /data/rocketmq
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /usr/local/rocketmq-4.8.0/conf/broker-b.properties
            subPath: broker-b.properties
          - name: time
            mountPath: /etc/localtime
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: rocketmq-config
      - name: time
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "course-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 4Gi

broker-b-s-statefulset.yaml：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: broker-b-s
  name: broker-b-s
  namespace: default
spec:
  type: NodePort
  ports:
  - port: 20911
    targetPort: 20911
    name: broker-port
    nodePort: 30914
  selector:
    app: broker-b-s
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: broker-b-s
  namespace: default
spec:
  serviceName: broker-b-s
  replicas: 1
  selector:
    matchLabels:
      app: broker-b-s
  template:
    metadata:
     labels:
       app: broker-b-s
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - broker-b-s
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
       - name: harborsecret
      containers:
      - name: broker-b-s
        image: 172.16.10.190:80/library/rocketmq:4.8.0
        imagePullPolicy: Always
        command: ["sh","-c","/usr/local/rocketmq-4.8.0/bin/mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-b-s.properties"]
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /data/rocketmq
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /usr/local/rocketmq-4.8.0/conf/broker-b-s.properties
            subPath: broker-b-s.properties
          - name: time
            mountPath: /etc/localtime
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20911
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: broker-config
        configMap:
          name: rocketmq-config
          items:
          - key: broker-b-s.properties
            path: broker-b-s.properties
      - name: time
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: rocketmq-data
      annotations:
        volume.beta.kubernetes.io/storage-class: "course-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 4Gi

namesrv-statefulset.yaml：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: mq-namesrv
  name: mq-namesrv
  namespace: default
spec:
  type: NodePort
  ports:
  - port: 20901
    targetPort: 20901
    name: namesrv-port
    nodePort: 30915
  selector:
    app: mq-namesrv
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mq-namesrv
  namespace: default
spec:
  serviceName: mq-namesrv
  replicas: 1
  selector:
    matchLabels:
      app: mq-namesrv
  template:
    metadata:
     labels:
       app: mq-namesrv
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - mq-namesrv
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
       - name: harborsecret
      containers:
      - name: mq-namesrv
        image: 172.16.10.190:80/library/rocketmq-namesrv:4.8.0
        imagePullPolicy: Always
        volumeMounts:
          - name: time
            mountPath: /etc/localtime
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 20901
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
      volumes:
      - name: time
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai

rocketmq-externals-deployment.yaml：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: mq-externals
  name: mq-externals
  namespace: default
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    name: console-port
    nodePort: 30916
  selector:
    app: mq-externals
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mq-externals
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mq-externals
  template:
    metadata:
     labels:
       app: mq-externals
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - mq-externals
              topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
       - name: harborsecret
      containers:
      - name: mq-externals
        image: 172.16.10.190:80/library/rocketmq-externals:v3
        imagePullPolicy: Always
        volumeMounts:
          - name: time
            mountPath: /etc/localtime
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15
      volumes:
      - name: time
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai