不忘初心,
牢记使命。

Kubernetes部署高可用harbor(helm方式)

2021-08-21 大聪明 0评论 55 0喜欢

[TOC]

Kubernetes部署高可用harbor(helm方式)

1.背景介绍

1.1 目标

将harbor部署在k8s上,通过k8s service的机制,实现harbor的高可用,即当运行的某一habor容器的节点故障时,不会造成Harbor服务的中断。

1.2 先决条件

  • Docker v20.10.7(不影响)
  • Kubernetes cluster v1.18.3(1.18+)
  • Helm v3.3.4(3.0+)
  • 高可用 PostgreSQL database
  • 高可用 Redis
  • 外部存储(S3、swift、oss ...)

1.3 架构

目前harbor大部分组件均为无状态服务,所以可以简单的通过增加组件容器副本来实现组件分布到不同计算节点,同时,利用k8s service的机制来提供组件服务发现能力。

harbor-ha-architecture

1.4 组件说明

组件 描述 备注
core harbor核心功能,包括认证、授权、项目管理等
jobservice 异步任务服务
portal web ui服务
registry 第三方镜像仓库服务,镜像存储
redis 第三方缓存服务
postgresql 第三方数据库服务
clair 第三方镜像漏洞扫描服务 2.0.0版本已使用trivy
notary 第三方内容信任服务

2.部署高可用harbor

helm使用文档(./helm-usage/高可用harbor部署文档.md)

2.1 数据库准备

创建harbor需要的三个数据库

如果提示权限问题,说明链接到了从库,将stolon-keeper-0改为stolon-keeper-1就可以了

# stolon目录下新建create-db.sql
stolon]# vi create-db.sql
CREATE DATABASE harbor OWNER stolon;
GRANT ALL PRIVILEGES ON DATABASE harbor TO stolon;
CREATE DATABASE notaryserver;
GRANT ALL PRIVILEGES ON DATABASE notaryserver TO stolon;
CREATE DATABASE notarysigner;
GRANT ALL PRIVILEGES ON DATABASE notarysigner TO stolon;

# 复制脚本到pod
stolon]# kubectl --namespace=kube-public cp ./create-db.sql stolon-keeper-0:/tmp/ kubectl cp ./create-db.sql stolon-keeper-0:/tmp/

# 执行脚本
[root@k8s-master stolon]# kubectl --namespace=kube-public exec stolon-keeper-0 -- bash -c "PGPASSWORD=pgsql psql -h stolon-keeper-0 -p 5432 -U stolon -d postgres -f \"/tmp/create-db.sql\""
CREATE DATABASE
GRANT
CREATE DATABASE
GRANT
CREATE DATABASE
GRANT

2.2 harbor charts准备

本地文件(./harbor-ha-helm/)

]# helm repo add harbor https://helm.goharbor.io
]# helm fetch harbor/harbor --untar
]# tree .
.
├── cert
│   ├── tls.crt
│   └── tls.key
├── Chart.yaml
├── conf
│   ├── notary-server.json
│   └── notary-signer.json
├── harbor-ha-images.tar
├── LICENSE
├── README.md
├── templates
│   ├── chartmuseum
│   │   ├── chartmuseum-cm.yaml
│   │   ├── chartmuseum-dpl.yaml
│   │   ├── chartmuseum-pvc.yaml
│   │   ├── chartmuseum-secret.yaml
│   │   ├── chartmuseum-svc.yaml
│   │   └── chartmuseum-tls.yaml
│   ├── core
│   │   ├── core-cm.yaml
│   │   ├── core-dpl.yaml
│   │   ├── core-secret.yaml
│   │   ├── core-svc.yaml
│   │   └── core-tls.yaml
│   ├── database
│   │   ├── database-secret.yaml
│   │   ├── database-ss.yaml
│   │   └── database-svc.yaml
│   ├── exporter
│   │   ├── exporter-cm-env.yaml
│   │   ├── exporter-dpl.yaml
│   │   ├── exporter-secret.yaml
│   │   └── exporter-svc.yaml
│   ├── _helpers.tpl
│   ├── ingress
│   │   ├── ingress.yaml
│   │   └── secret.yaml
│   ├── internal
│   │   └── auto-tls.yaml
│   ├── jobservice
│   │   ├── jobservice-cm-env.yaml
│   │   ├── jobservice-cm.yaml
│   │   ├── jobservice-dpl.yaml
│   │   ├── jobservice-pvc.yaml
│   │   ├── jobservice-secrets.yaml
│   │   ├── jobservice-svc.yaml
│   │   └── jobservice-tls.yaml
│   ├── metrics
│   │   └── metrics-svcmon.yaml
│   ├── nginx
│   │   ├── configmap-https.yaml
│   │   ├── configmap-http.yaml
│   │   ├── deployment.yaml
│   │   ├── secret.yaml
│   │   └── service.yaml
│   ├── notary
│   │   ├── notary-secret.yaml
│   │   ├── notary-server.yaml
│   │   ├── notary-signer.yaml
│   │   └── notary-svc.yaml
│   ├── NOTES.txt
│   ├── portal
│   │   ├── configmap.yaml
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── tls.yaml
│   ├── redis
│   │   ├── service.yaml
│   │   └── statefulset.yaml
│   ├── registry
│   │   ├── registry-cm.yaml
│   │   ├── registry-dpl.yaml
│   │   ├── registry-pvc.yaml
│   │   ├── registry-secret.yaml
│   │   ├── registry-svc.yaml
│   │   └── registry-tls.yaml
│   └── trivy
│       ├── trivy-secret.yaml
│       ├── trivy-sts.yaml
│       ├── trivy-svc.yaml
│       └── trivy-tls.yaml
└── values.yaml

2.3 修改values.yaml相关内容

如果指定node,则各个pod的nodeSelect需要自行分配

如:

# 节点,json键值对格式,根据自己的node标签填写
# 如: nodeSelector: {"harbor":"harbor-ha"}
nodeSelector: {}

2.3.1 暴露方式

暴露方式选择nodePort,访问端口选择自己需要的端口

# 暴露方式选择nodePort
expose:
  # Set the way how to expose the service. Set the type as "ingress",
  # "clusterIP", "nodePort" or "loadBalancer" and fill the information
  # in the corresponding section
  type: nodePort

  nodePort:
    # The name of NodePort service
    name: harbor
    ports:
      http:
        # The service port Harbor listens on when serving with HTTP
        port: 80
        # The node port Harbor listens on when serving with HTTP
        # 访问端口
        nodePort: 30002

2.3.2 外部信任URL

  • 外部访问url一定要完整,必须加上协议,否则会不被信任直接提示密码不正确,其实网络抓包看到是403 Forbidden

  • ip+端口访问 externalURL: http://ip:30002

# If Harbor is deployed behind the proxy, set it as the URL of proxy
# externalURL: http://harbor01.liboer.top
externalURL: http://ip:nodePort端口

2.3.3 关闭tls

# 关闭tls
# in each components tls cert files need to provided in advance.
internalTLS:
  # If internal TLS enabled
  enabled: false

2.3.4 外部存储选择

以s3为例

persistence:
  enabled: false
  type: s3
  s3:
      region: us-west-1
      bucket: bucketname
      accesskey: awsaccesskey
      secretkey: awssecretkey
      #regionendpoint: http://myobjects.local
      #encrypt: false
      #keyid: mykeyid
      secure: false
      #skipverify: false
      #v4auth: true
      #chunksize: "5242880"
      #rootdirectory: /s3/object/name/prefix
      #storageclass: STANDARD
      #multipartcopychunksize: "33554432"
      #multipartcopymaxconcurrency: 100
      # 最大块传输量5
      multipartcopythresholdsize: "5368709120"

2.3.5 admin登陆密码

# admin登陆密码,也可以部署成功后在web ui修改
# The initial password of Harbor admin. Change it from portal after launching Harbor
harborAdminPassword: "Harbor12345"

2.3.6 节点选择

如果指定node,则各个pod的nodeSelect需要自行分配

如:

# 节点,json键值对格式,根据自己的node标签填写# 如: nodeSelector: {"harbor":"harbor-ha"}nodeSelector: {}

2.3.7 外部database

database选择外部的 type: external

# database选择外部的  type: externaldatabase:  # if external database is used, set "type" to "external"  # and fill the connection informations in "external" section  type: external  # 外部database的配置  external:    host: "stolon-proxy-service"    port: "5432"    username: "stolon"    password: "highgo"    coreDatabase: "harbor"    notaryServerDatabase: "notaryserver"    notarySignerDatabase: "notarysigner"    sslmode: "disable"

2.3.8 外部redis

redis.type: external

redis.password: "" 这里密码为你设定的密码,如未设置认证则为空

# redis选择外部的redis:  # if external Redis is used, set "type" to "external"  # and fill the connection informations in "external" section  type: external  # 相关配置  external:      addr: "redis-ha:6379"    password: ""

2.4 开始部署

helm install your-name path -n namespace

全部是running就代表服务已经启动

harbor]# helm install harbor-ha . -n kube-publicNAME: harbor-haLAST DEPLOYED: Fri Jul 30 14:27:24 2021NAMESPACE: kube-publicSTATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:Please wait for several minutes for Harbor deployment to complete.Then you should be able to visit the Harbor portal at For more details, please visit https://github.com/goharbor/harbor[root@k8s-master harbor]# kubectl get podNAME                                       READY   STATUS    RESTARTS   AGEharbor-ha-chartmuseum-749d6b98b4-dr476     1/1     Running   0          91mharbor-ha-core-846d4f7d9-c6ngw             1/1     Running   0          91mharbor-ha-jobservice-5c9676bb6-wvkjh       1/1     Running   0          91mharbor-ha-nginx-57c4d575cf-8zgnb           1/1     Running   0          91mharbor-ha-notary-server-77d654bb5d-6bqzk   1/1     Running   0          91mharbor-ha-notary-signer-7dd4ddf687-dmm4f   1/1     Running   0          91mharbor-ha-portal-6486c5449f-hcphg          1/1     Running   0          91mharbor-ha-registry-6fd4cc9d68-fn4mg        2/2     Running   0          91mharbor-ha-trivy-0                          1/1     Running   0          91m

2.5 验证

2.5.1 浏览器访问

http://ip:nodePort端口

输入 admin Harbor12345 登陆成功 创建一个公开仓库public

2.5.2 docker推送

# 将我们的网站可信任[root@k8s-node01 ~]# vi /etc/docker/daemon.json "insecure-registries": ["registry.access.redhat.com", "quay.io", "harbor01.liboer.top"],[root@k8s-node01 ~]# systemctl daemon-reload[root@k8s-node01 ~]# systemctl restart docker# 登陆[root@k8s-node01 stolon]# docker login harbor01.liboer.topUsername: adminPassword: WARNING! Your password will be stored unencrypted in /root/.docker/config.json.Configure a credential helper to remove this warning. Seehttps://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded# 随便找个镜像打个标签推送到我们的私有仓库[root@k8s-node01 stolon]# docker tag quay.io/coreos/flannel:v0.11.0-amd64 harbor01.liboer.top/public/flannel:v0.11.0-amd64[root@k8s-node01 stolon]# docker push harbor01.liboer.top/public/flannel:v0.11.0-amd64The push refers to repository [harbor01.liboer.top/public/flannel]9ce0bb155166: Pushed 3f3a4ce2b719: Pushed 9b48060f404d: Pushed 5d3f68f6da8f: Pushed 7bff100f35cb: Pushed v0.11.0-amd64: digest: sha256:bd76b84c74ad70368a2341c2402841b75950df881388e43fc2aca000c546653a size: 1369# 此时网页登陆可以看到已经推送过来了

3.FAQ

3.1 pg库权限

如果进入某个pg-server的pod创建需要的三个数据库时提示无权限,则代表你进入了从节点,切换到主节点即可。

3.2 docker push镜像一直retry

3.2.1 s3可能没有配置最大块传输量

# 最大块传输量multipartcopythresholdsize: "33554432"

3.2.2 如果二次重装harbor时出现则可能你的redis里有缓存了

# 清空redis缓存或卸载重装,清空比较方便]# kubectl exec -ti redis-ha-server-0 /bin/sh -n kube-publickubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.Defaulting container name to redis.Use 'kubectl describe pod/redis-ha-server-0 -n default' to see all of the containers in this pod./data $ redis-cli -h redis-ha -p 6379redis-ha:6379> flushallOK

3.3 nginx jobservice harbor-core一直重启

可能是你的机器环境网络存在问题,考虑还两个节点部署一下试试。

3.4 web ui的项目访问比较慢(5~8s)

可能和你用后端外部镜像存存储有关联(例如s3)

others

更多问题可参考官网issue

发表评论 取消回复

电子邮件地址不会被公开。

请输入正确格式的qq邮箱
请输入以http或https开头的URL,格式如:https://libo_sober.top