Rocky9 安装 Kubernetes 集群

更新：2023-09-24

kubernetes1.20.+ 默认容器采用的是containerd，不是docker

hostname ip
k8s-master 192.168.18.90
k8s-node01 192.168.18.91
k8s-node02 192.168.18.92
k8s-node03 192.168.18.93

端口

1
2
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly

网卡配置

1
vim /etc/NetworkManager/system-connections/ens33.nmconnection
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[connection]
id=ens33
uuid=55cb18ca-e9e2-318b-b50c-f844d9a96a56
type=ethernet
autoconnect-priority=-999
interface-name=ens33
timestamp=1695503455

[ethernet]

[ipv4]
method=manual							# auto dhcp、manual static
address1=192.168.18.40/24,192.168.18.2	# ip地址/子网掩码，网关地址
dns=192.168.18.2;114.114.114.114;		# dns
may-fail=false

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]
1
2
3
4
5
6
7
8
9
10
11
12
# connection 简写为 c
# reload r
# down d
# up u
# load l

nmcli connection reload
nmcli connection down ens33
nmcli connection up ens33

# load 后面需要跟完整的路径
nmcli connection load  /etc/NetworkManager/system-connections/ens33.nmconnection 
1
2
3
4
systemctl restart NetworkManager

## 查看是否生成配置
cat /etc/resolv.conf

设置国内源

1
2
3
4
5
sed -e 's|^mirrorlist=|#mirrorlist=|g' \
    -e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.ustc.edu.cn/rocky|g' \
    -i.bak \
    /etc/yum.repos.d/rocky-extras.repo \
    /etc/yum.repos.d/rocky.repo

安装依赖

1
2
# dnf install -y vim wget curl ntpdate ntp net-tools git \
conntrack jq sysstat libseccomp

Step 1: 准备 Hostname, SELinux and Firewall

1
2
3
4
5
6
7
8
9
10
11
# hostnamectl set-hostname k8s-master
# hostnamectl set-hostname k8s-node01
# hostnamectl set-hostname k8s-node02
# hostnamectl set-hostname k8s-node03

# cat <<EOF>> /etc/hosts
192.168.18.90 k8s-master
192.168.18.91 k8s-node01
192.168.18.92 k8s-node02
192.168.18.93 k8s-node03
EOF

禁用SELinux

使用 sestatus -v 查看 状态 或者 getenforce 

1
2
3
4
5
6
selinux的三种运行模式:

enforcing:强制模式，SELinux 正在运行中，已经在限制 domain/type
permissive:宽容模式：SELinux 正在运行中，但仅发出警告信息,并不会实际限制 domain/type 的存取
(permissive模式可以用在测试环境中供调试规则时使用)
disabled:关闭，SELinux 没有实际运行
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[root@localhost ~]# sestatus -v
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

Process contexts:
Current context:                unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Init context:                   system_u:system_r:init_t:s0
/usr/sbin/sshd                  system_u:system_r:sshd_t:s0-s0:c0.c1023

File contexts:
Controlling terminal:           unconfined_u:object_r:user_devpts_t:s0
/etc/passwd                     system_u:object_r:passwd_file_t:s0
/etc/shadow                     system_u:object_r:shadow_t:s0
/bin/bash                       system_u:object_r:shell_exec_t:s0
/bin/login                      system_u:object_r:login_exec_t:s0
/bin/sh                         system_u:object_r:bin_t:s0 -> system_u:object_r:shell_exec_t:s0
/sbin/agetty                    system_u:object_r:getty_exec_t:s0
/sbin/init                      system_u:object_r:bin_t:s0 -> system_u:object_r:init_exec_t:s0
/usr/sbin/sshd                  system_u:object_r:sshd_exec_t:s0

因为 sed -i /etc/sysconfig/selinux (selinux文件是/etc/selinux/config的软链接) 配置文件重启SELINUX没有关闭，才发现原来sed -i是不能直接修改软链接文件的，如下我修改之后的后果，发现链接文件不再是链接文件了。

1
2
3
4
5
# setenforce 0
# sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

## 依然保持着软链接
# ll /etc/sysconfig/selinux

关闭 交换分区

1
2
3
4
5
# 临时关闭
# swapoff -a

## 永久关闭，即注释掉swap分区记录
# sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab

关闭交换分区前

1
2
3
4
5
6
7
8
[root@localhost ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:            972         131         693           7         147         674
Swap:          2047           0        2047

[root@localhost ~]# swapon -v
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition   2G   0B   -2

关闭交换分区后

1
2
3
4
5
6
7
[root@localhost ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:            972         135         687           7         150         669
Swap:             0           0           0

[root@localhost ~]# swapon -v
[root@localhost ~]# 
1
2
## 启用交换分区
# swapon -a

/etc/fstab, 如果永久关闭注释掉 swap 分区那行即可

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@localhost ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Wed Apr  1 08:02:24 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=280b4b43-e80b-4f8d-8f2f-dfae887699d8 /boot                   xfs     defaults        0 0
/dev/mapper/centos-home /home                   xfs     defaults        0 0
/dev/mapper/centos-swap swap                    swap    defaults        0 0

设置防火墙规则及转发

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
systemctl status firewalld
# 8472/udp为flannel的通信端口
# 443/tcp 为Kubernetes server端口
----------------------------------------------------------------------
## master 节点 Add
firewall-cmd --permanent --add-port=443/tcp

firewall-cmd --permanent --add-port=2379-2380/tcp

firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=10250/tcp

firewall-cmd --permanent --add-port=10251/tcp
firewall-cmd --permanent --add-port=10252/tcp
firewall-cmd --permanent --add-port=10255/tcp
firewall-cmd --permanent --add-port=8472/udp
# 注意一点：一定要执行以下命令打开NAT，默认是关闭状态，这里踩过坑！！
firewall-cmd --permanent --add-masquerade
# only if you want NodePorts exposed on control plane IP as well
firewall-cmd --permanent --add-port=30000-32767/tcp
firewall-cmd --reload

## master 节点 Remove
firewall-cmd --permanent --remove-port=6443/tcp
firewall-cmd --permanent --remove-port=2379-2380/tcp
firewall-cmd --permanent --remove-port=10250/tcp
firewall-cmd --permanent --remove-port=10251/tcp
firewall-cmd --permanent --remove-port=10252/tcp
firewall-cmd --permanent --remove-port=10255/tcp
firewall-cmd --permanent --remove-port=8472/udp
# only if you want NodePorts exposed on control plane IP as well
firewall-cmd --permanent --remove-port=30000-32767/tcp
firewall-cmd --reload

----------------------------------------------------------------------
## node 节点 Add
firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=10255/tcp

## for dns
## firewall-cmd --permanent --add-port=53/tcp
firewall-cmd --permanent --add-port=53/udp
## firewall-cmd --permanent --add-port=9153/tcp

## flannel
firewall-cmd --permanent --add-port=8472/udp
firewall-cmd --permanent --add-port=30000-32767/tcp
# 注意一点：一定要执行以下命令打开NAT，默认是关闭状态，这里踩过坑！！
firewall-cmd --permanent --add-masquerade
firewall-cmd --reload

## node 节点 Remove
firewall-cmd --permanent --remove-port=80/tcp
firewall-cmd --permanent --remove-port=10250/tcp
firewall-cmd --permanent --remove-port=10255/tcp

## firewall-cmd --permanent --remove-port=53/tcp
firewall-cmd --permanent --remove-port=53/udp
firewall-cmd --permanent --remove-port=9153/tcp


firewall-cmd --permanent --remove-port=8472/udp
firewall-cmd --permanent --remove-port=30000-32767/tcp
# 注意一点：一定要执行以下命令打开NAT，默认是关闭状态，这里踩过坑！！
firewall-cmd --permanent --remove-masquerade
firewall-cmd --reload
----------------------------------------------------------------------

## 查看开放的端口
firewall-cmd --list-all
# 检查是否允许NAT转发
firewall-cmd --query-masquerade


调整内核参数

1
2
3
4
# modprobe br_netfilter

# echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
# echo '1' > /proc/sys/net/bridge/bridge-nf-call-ip6tables
1
2
3
4
5
# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

执行命令生效

1
# sysctl -p /etc/sysctl.d/k8s.conf

kube-proxy 开启 ipvs 的前置条件

linux kernel 4.19版本已经将nf_conntrack_ipv4 更新为 nf_conntrack

1
2
3
4
5
6
7
8
9
10
11
12
13
# modprobe br_netfilter
# cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack"
for kernel_module in \${ipvs_modules}; do
    /sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        /sbin/modprobe \${kernel_module}
    fi
done
EOF

# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
1
2
3
4
5
## 安装ipvs相关软件包
# dnf -y install ipvsadm ipset

## 如果 ipvsadm -Ln 没有看到转发 需要修改配置文件，删掉pod，自我修复后就可以看到了
kubectl edit cm kube-proxy -n kube-system　

安装docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# dnf config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# dnf update -y && yum install -y yum-utils docker-ce

# mkdir -p /etc/docker
# cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": ["https://docker.mirrors.sjtug.sjtu.edu.cn"],
  "exec-opts": [
    "native.cgroupdriver=systemd"
  ],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  }
}
EOF

# mkdir -p /etc/systemd/system/docker.service.d
# systemctl daemon-reload && systemctl enable docker --now
# docker info |grep Cgroup

docker配置文件docker-daemon.json详解

Step 3: 设置 Kubernetes 国内源

1
2
3
4
5
6
7
8
9
# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

Step 6: Install Kubeadm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# dnf makecache
# dnf list kubelet --showduplicates

# dnf install kubeadm-1.28.2 -y
# 查询安装的版本
# rpm -aq kubelet kubectl kubeadm


# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"


# systemctl enable kubelet.service
# systemctl status kubelet.service


下载k8s依赖镜像

通过阿里云镜像下载k8s依赖组件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25



# 禁用CRI插件
/etc/containerd/config.toml
## 这个暂时用不到
sed -i -r '/cri/s/(.*)/#\1/' /etc/containerd/config.toml


containerd config default > /etc/containerd/config.toml
vim /etc/containerd/config.toml

SystemdCgroup = false 改为 SystemdCgroup = true
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

## 后来补充的
echo 'runtime-endpoint: unix:///run/containerd/containerd.sock' > /etc/crictl.yaml

# 重启 containerd 服务
systemctl daemon-reload
systemctl restart containerd.service

## 查看镜像
crictl images

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
## 下载镜像
# kubeadm config images list

## 方式一
## 指定 k8s 版本
# kubeadm config images list --kubernetes-version=v1.28.2
# kubeadm config images list --image-repository registry.aliyuncs.com/google_containers
# kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers

## 方式二
## 旧版地址 k8s.gcr.io
## 新版地址 registry.k8s.io

# kubeadm config images list |sed -e 's/^/docker pull /g' -e 's#registry.k8s.io#registry.aliyuncs.com/google_containers#g' |sh -x

## 指定 k8s 版本
# kubeadm config images list --kubernetes-version 1.28.2 |sed -e 's/^/docker pull /g' -e 's#registry.k8s.io#registry.aliyuncs.com/google_containers#g' |sh -x

## 重命名
docker images |grep registry.aliyuncs.com/google_containers |awk '{print "docker tag ",$1":"$2,$1":"$2}' |sed -e 's#registry.aliyuncs.com/google_containers#registry.k8s.io#2' |sh -x

## 删除镜像
docker images |grep registry.aliyuncs.com/google_containers |awk '{print "docker rmi ", $1":"$2}' |sh -x

## 删除 gcr 镜像
docker images |grep k8s.gcr.io/kube-* |awk '{print "docker rmi ", $1":"$2}' |sh -x

下载时，还没有 coredns:v1.10.1，单独处理 coredns

1
2
3
4
5
6
7
8
9
10
11
12
13

docker pull coredns/coredns:1.10.1
docker tag coredns/coredns:1.10.1 registry.k8s.io/coredns/coredns:v1.10.1
docker tag coredns/coredns:1.10.1 registry.aliyuncs.com/google_containers/coredns:v1.10.1

docker pull registry.aliyuncs.com/google_containers/pause:3.6
docker tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6


node 节点
docker pull registry.aliyuncs.com/google_containers/etcd:3.5.9-0
docker pull docker.io/flannel/flannel:v0.22.3
docker pull docker.io/flannel/flannel-cni-plugin:v1.2.0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
## 指定安装版本
## kubeadm init --kubernetes-version=v1.28.2 --pod-network-cidr=10.244.0.0/16

## 或者通过配置文件初始化

## 获取默认配置
# kubeadm config print init-defaults > kubeadm-config.yaml

kubeadm config images list --config kubeadm-config.yaml
kubeadm config images pull --config kubeadm-config.yaml

## 解决
https://blog.csdn.net/swq463/article/details/132085155

kubeadm init \
  --apiserver-advertise-address=192.168.18.90 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.28.2 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16 \
  --ignore-preflight-errors=all \
  --v=6
 

kubeadm reset
systemctl status kubelet
journalctl -xefu kubelet

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf
  
kubeadm join 192.168.18.90:6443 --token m82gzg.j5q0rtrjgzxs9mda \
        --discovery-token-ca-cert-hash sha256:8b5f10a989b7f00fc4aa53cc9fe7586872cc49c2d6a5a70d8aa01e7ca1b428bb \
        --v=6
        
        
kubectl get nodes
kubectl get pods -n kube-system

kubeadm-config.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  # master 地址
  advertiseAddress: 192.168.18.90
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  ## master 名称，使用主机名称
  name: k8s-master
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
# 换成国内的源
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
# 修改版本号 必须对应
kubernetesVersion: 1.28.2
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}
1
kubeadm init --config kubeadm-config.yaml --upload-certs | tee kubeadm-init.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# kubeadm reset
[reset] Deleting contents of stateful directories: [
/var/lib/etcd
/var/lib/kubelet
/var/lib/dockershim 
/var/run/kubernetes 
/var/lib/cni
]
# rm -rf ~/.kube/

# kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.log

## 获取配置
# kubectl -n kube-system get cm kubeadm-config -oyaml

## 查看启动失败的 pod
# kubectl describe pod coredns-78fcd69978-th9qx --namespace=kube-system


kubectl describe pod coredns-66f779496c-swlqg  --namespace=kube-system

## 
# kubectl get cs

## 注释掉 - --port=0
## vim /etc/kubernetes/manifests/kube-scheduler.yaml

# kubectl -n kube-system logs kube-proxy-lmchg
F0907 02:05:49.245441       1 server.go:486] failed complete: unrecognized feature gate: SupportIPVSProxyMode
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00000e001, 0xc00015c000, 0x6e, 0xc0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9

error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition

1
拉取镜像失败

open /run/flannel/subnet.env: no such file or directory

1
需要指定网络，或者指定的方式不对

重置 node 节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
删除节点
kubectl delete node k8s-node01
kubectl delete node k8s-node02
kubectl delete node k8s-node03

kubeadm reset

-----------或者下面的命令----------
systemctl stop kubelet
systemctl stop docker
systemctl stop docker.socket

rm -rf ~/.kube/

rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
rm -rf /etc/kubernetes/*

1

1
2
3
4
5
6
7
8
9
10
## kubeadm 安装的 k8s 集群 delete node 后重新添加回集群问题解决

# kubeadm reset

# kubeadm join 192.168.18.160:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:0dcc4bf43e9fa4fc2c9a69039e48f6cfe19f8050b0b7ea439de9577f9c279f10
## 忘记时，可以通过下面的命令创建个新的
# kubeadm token create --print-join-command


部署网络

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

kubectl apply -f kube-flannel.yml
kubectl get pod -n kube-system
kubectl get pod -n kube-flannel
kubectl get pod -n kube-system -o wide

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

--cluster-cidr=10.244.0.0/16

cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true


# 拉取镜像
quay.io/coreos/flannel:v0.14.0
https://www.cnblogs.com/pyxuexi/p/14288591.html

docker load < flanneld-v0.14.0-amd64.docker
docker tag quay.io/coreos/flannel:v0.14.0-amd64 quay.io/coreos/flannel:v0.14.0
docker rmi quay.io/coreos/flannel:v0.14.0-amd64

# 转发网络
docker 1.13之后,转发规则变了

iptables -L -n
Chain FORWARD (policy DROP)

iptables -P OUTPUT ACCEPT && iptables -P FORWARD ACCEPT && iptables -F && iptables -L -n

## 添加到docker中
/usr/sbin/iptables -P OUTPUT ACCEPT && /usr/sbin/iptables -P FORWARD ACCEPT

vim /usr/lib/systemd/system/docker.service
添加
ExecStartPost=sh -c "/sbin/iptables -P OUTPUT ACCEPT && /sbin/iptables -P FORWARD ACCEPT && /sbin/iptables -F"
systemctl daemon-reload
systemctl restart docker

k8s-node1   NotReady
# 查看下 kubelet 状态
systemctl status kubelet
journalctl -xefu kubelet

启动失败

1
# journalctl -xefu kubelet

https://www.python2.net/questions-1171657.htm

ipvsadm -ln

连接 Harbor

1
2
3
4
5
6
7
8
9
10
11
12
kubectl create secret docker-registry docker-harbor \
--docker-server=hub.me.io \
--docker-username=admin \
--docker-password=admin \
--docker-email=admin@gmail.com


修改
kubectl edit secret docker-harbor

删除
kubectl delete secret docker-harbor
1
2
3
kubectl get secret docker-harbor --output="jsonpath={.data.\.dockerconfigjson}" | base64 -d

kubectl get secret docker-harbor -o yaml

创建时指定命名空间

1
2
3
4
5
6
kubectl --namespace kube-system \
create secret docker-registry docker-harbor \
--docker-server=hub.me.io \
--docker-username=admin \
--docker-password=admin \
--docker-email=admin@gmail.com

常用命令：

1
2
kubectl api-versions

pod

1
2
kubectl exec -it demo-deployment-6687bf488-5xnz6  -- /bin/sh
/ #

svc

1
2
3
4
5
6
7
8
9
10
11
12
<服务名称>.<工作负载所在命名空间>.svc.cluster.local:<端口号>

demo.default.svc.cluster.local

## 容器内部访问
<服务名称>.<工作负载所在命名空间>.svc

## stateful
statefulSetName-0.serviceName.namespace.svc.cluster.local


dig -t A user-service.default.svc.cluster.local. @10.96.0.10

headless

1
2
域名格式：$(podname).(headless server name).namespace.svc.cluster.local

部署kubernetes-dashboard

https://github.com/kubernetes/dashboard

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

# kubectl apply -f kubernetes-dashboard.yaml

## 注意 namespace 变化了
# kubectl get pod -n  kubernetes-dashboard -o wide

# kubectl get svc -n kubernetes-dashboard

## 编辑 svc 修改 为 NodePort
# kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
spec:
  clusterIP: 10.100.176.89
  clusterIPs:
  - 10.100.176.89
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort
  
## 获取 dashboard token
# kubectl -n kubernetes-dashboard get secret | grep kubernetes-dashboard-token

## 这里可以获取 token 了
# kubectl describe secret kubernetes-dashboard-token-7fnx5 -n kubernetes-dashboard

eyJhbGciOiJSUzI1NiIsImtpZCI6Il9Sd2dnUXRUWUJSdDBWWEZLdEg1ZW1nRWhvdzVnc0NaMkRzR3hRTUF5SE0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi03Zm54NSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjI1ZjRhNzc0LTI3NzYtNGNlYy1iZmQzLTg0MzVlYjU2NTY2YSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.H3wENMoQQfArXI2Qs_KXUQB8PDiXB7YWIy2hgWHetURgkMw3zoy6WnlcB2-h0oFOrHXPdU81ZfxbShZ5W-W_rS4fpfdAwSs1JrrxIa5OL50O9davlRiZvmoBzWCOjlGsYcgBVLmOE49iB9-rgFgaEiptDWwdGKj1Q7y7f9qRyc52ZwjX8AXOZo97iF2PM3ecDWdJU69US70jNXTdzDvGDghg75mYf8is2PA7-dsftMOBJ6SfpqntMDJLXzTvvZRx4JKP3wmgiox4rfYUbdYoM_oItaVpbfa1qjLKj7t9e2v4WT-Yomtr0nK3rH-EqMjh9jUX5YVYjcCCHpXWqdrFvA

创建管理员

dashboard-adminuser.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# cat > dashboard-adminuser.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

说明：创建了一个叫admin-user的服务账号，并放在kubernetes-dashboard 命名空间下，并将cluster-admin角色绑定到admin-user账户，这样admin-user账户就有了管理员的权限。默认情况下，kubeadm创建集群时已经创建了cluster-admin角色，我们直接绑定即可。

创建登录用户

1
# kubectl apply -f dashboard-adminuser.yaml

查看admin-user账户的token

1
2
3
4
# kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
eyJhbGciOiJSUzI1NiIsImtpZCI6IjhxMlIxUFIzT2JYQWE2d2JhNml1UHNmc2xsUXFZR3k5MldIdzUtOEJXZGsifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTVqNTdwIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI4YWU5MTZlNy1hZTJjLTRkMjgtYWJkNS1iZTdhNzU5MzNhMTciLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.qCVOjf_JhTmt4rMjsl66n3vje_LuGI1jit5T7_X9_71Gk27z9a96DLhT56PhxZfnXQinqxCioKFFWt3eUgexAWFYgHW9PDP7hHCl6L6qOWy9s-tJRt9iKsJGpKnueodNnwrWUZDY5jbR90v6LualpRaMdbXi0DuGjAGERQCrstRTr-JTXPqT1x3BS-ub5_P5cU0B688mdKs1okjyu42N9TFjZN1i0ftAhGlFOuSRp3r1yUg8GgEy03eHyJXceHAkgjZK5jB_vqnNE0ViLrjLHoNiqTmVgKnfzkCJWkB2_5T03QcM-JB6bh39mtEJm9603qQRnfe-OU-NcfF4F64Nxg

https://192.168.18.160:30392

token

1
2
3
4
5
6
7
8
kubeadm token create


openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //'

kubeadm join 192.168.18.90:6443 --token c09z9j.g6w2x73g1xz6v7ck --discovery-token-ca-cert-hash sha256:8b5f10a989b7f00fc4aa53cc9fe7586872cc49c2d6a5a70d8aa01e7ca1b428bb --v=6

ingress

1
2
# 查询需要解析的到的IP地址
# kubectl get pod -n ingress-nginx -o wide

https://kubernetes.github.io/ingress-nginx/

https://github.com/kubernetes/ingress-nginx

https://blog.csdn.net/qq_41076892/article/details/133897130

1
2
3
4
5
6

Ingress: 配置转发规则，类似于 nginx 的配置文件
Ingress Controller: 转发，类似于 nginx，它会读取 Ingress 的规则并转化为 nginx 的配置文件

firewall-cmd --permanent --add-port=8443/tcp
firewall-cmd --reload
1
2
3
4
5
      imagePullSecrets:
        - name: harbor

hub.vm.io/library/ingress-nginx-controller:v1.8.2
hub.vm.io/library/kube-webhook-certgen:v20230407

https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0/deploy/static/provider/baremetal/deploy.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx

---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: 'true'
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ipFamilyPolicy: SingleStack
  ipFamilies:
    - IPv4
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      appProtocol: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          image: hub.me.io/k8s/ingress-nginx-controller:v1.0.4
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
            - --election-id=ingress-controller-leader
            - --controller-class=k8s.io/ingress-nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: nginx
  namespace: ingress-nginx
spec:
  controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission
        path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-create
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-create
      labels:
        helm.sh/chart: ingress-nginx-4.0.6
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 1.0.4
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: create
          image: hub.me.io/k8s/kube-webhook-certgen:v1.1.1
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
            - --namespace=$(POD_NAMESPACE)
            - --secret-name=ingress-nginx-admission
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-patch
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.6
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.4
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-patch
      labels:
        helm.sh/chart: ingress-nginx-4.0.6
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 1.0.4
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: patch
          image: hub.me.io/k8s/kube-webhook-certgen:v1.1.1
          imagePullPolicy: IfNotPresent
          args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=$(POD_NAMESPACE)
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
1
替换掉里面的两个镜像地址

常用命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
## 获取namespace信息
# kubectl get namespace

## 查看pod状态
# kubectl get pods --all-namespaces

## 查看有多少个节点
# kubectl get nods

## 查看kubelet进程日志
# journalctl  -f -u kubelet

## 查看所有的token
# kubeadm token list

## 创建新的token
# kubeadm token create

1
2
3
4
5
6
7
8
pod-network-cidr是pod网络
--pod-network-cidr=3.3.3.1/16
kubectl get pod


service-cidr是svc网络
--service-cidr=2.2.2.1/16 
kubectl get svc
1
2
docker registry 私服密码
bca/BCAabc123

kuboard

1
2
3
4
5
6
7
https://github.com/eip-work/kuboard-press/issues/328#issuecomment-1269980083
http://192.168.18.90:30080/
admin
Kuboard123

kubectl apply -f kuboard-v3.yaml

harbor

kubernetes1.20.x 默认容器采用的是containerd，不是docker，因此配置有所不同。

admin:admin

无配置时报错如下

1
2
http: server gave HTTP response to HTTPS client
x509: certificate signed by unknown authority

在每个node上修改containerd的配置文件

1
vim /etc/containerd/config.toml

http访问方式写入

1
2
3
4
5
6
[plugins."io.containerd.grpc.v1.cri".registry]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."hub.vm.io:80"]
          endpoint = ["http://hub.vm.io:80"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."hub.vm.io".tls]
          insecure_skip_verify = true

https访问方式写入

1
2
3
4
5
6
[plugins."io.containerd.grpc.v1.cri".registry]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."hub.vm.io"]
          endpoint = ["https://hub.vm.io"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."hub.vm.io".tls]
          insecure_skip_verify = true

重启containerd

1
systemctl restart containerd

测试

1
2
3
4
# http
crictl pull --creds admin:Harbor12345 core.harbor.service.com:80/library/nginx
# https
crictl pull --creds admin:Harbor12345 core.harbor.service.com/library/nginx

创建secret

1
2
3
4
# http
kubectl create secret docker-registry regcred --docker-server=http://core.harbor.service.com:80 --docker-username=admin --docker-password=Harbor12345 --docker-email=12345@123.com
# https
kubectl create secret docker-registry regcred --docker-server=https://core.harbor.service.com --docker-username=admin --docker-password=Harbor12345 --docker-email=12345@123.com

最后：在创建deployment时选择secret：regcred

参考文献

https://blog.csdn.net/qq_40460909/article/details/90521561

Centos7 升级内核版本

https://www.freesion.com/article/6878979989/

kubeadm部署kubernetes-1.21.1

kubernetes启动Pod遇到CrashLoopBackOff的解决思路

ipvs

ipvs 1.20.0

https://www.huweihuang.com/kubernetes-notes/setup/installer/install-k8s-by-kubeadm.html

k8s 下载 harbor 镜像

赏

谢谢你请我吃糖果

支付宝
微信
|阅读量()

扫一扫，分享到微信

微信分享二维码
本站总访问量次 |

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

Java码农