k8s1.14.6集群搭建之apiserver部署

k8s1.14.6集群部署 - This article is part of a series.

Part 1: k8s1.14.6集群搭建之ETCD集群部署

Part 2: This Article

Part 3: k8s1.14.6集群搭建之controller-manager部署

Part 4: k8s1.14.6集群搭建之scheduler部署

Part 5: k8s1.14.6集群搭建之node节点部署

Part 6: k8s1.14.6集群搭建之kube-flannel部署

Part 7: k8s1.14.6集群搭建之kube-proxy部署

1. 创建自签名CA及相关证书
#

1.1 创建自签名CA(apiserver访问相关)

$ cd /root/k8s-1.14.6/ssl/
$ cat <<EOF > ca-config.json
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "kubernetes": {
               "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF

$ cat <<EOF > ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "",
            "L": "",
            "O": "",
            "OU": "",
            "ST": ""
        }
    ]
}
EOF
$ gencert -initca ca-csr.json|cfssljson -bare ca

1.2 签名证书apiserver.pem、apiserver-kubelet-client.pem

$ cat <<EOF > apiserver-csr.json
{
    "CN": "kube-apiserver",
    "hosts": [
        "172.16.0.1",                                             --> 为service-cluster-ip-range参数值
        "127.0.0.1",
        "192.168.18.142",                                         --> master集群所有节点ip
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local"                    --> clusterDomain
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "",
            "L": "",
            "ST": "",
            "O": "",
            "OU": ""
        }
    ]
}
EOF
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-csr.json |cfssljson -bare apiserver

$ cat <<EOF > apiserver-kubelet-client-csr.json
{
    "CN": "kube-apiserver-kubelet-client",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "",
            "L": "",
            "O": "system:masters",
            "OU": "",
            "ST": ""
        }
    ]
}
EOF
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-kubelet-client-csr.json |cfssljson -bare apiserver-kubelet-client

1.3 创建front-proxy自签名CA，开启k8s api aggregation要用

$ cd /root/k8s-1.14.6/ssl/front-proxy
$ cat <<EOF > ca-config.json
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "kubernetes": {
               "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF
$ cat <<EOF > ca-csr.json
{
    "CN": "front-proxy-ca",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "",
            "L": "",
            "O": "",
            "OU": "",
            "ST": ""
        }
    ]
}
EOF
$ cfssl gencert -initca ca-csr.json|cfssljson -bare ca

1.4 签名front-proxy相关证书

$ cat <<EOF > front-proxy-client-csr.json
{
    "CN": "front-proxy-client",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "",
            "L": "",
            "O": "",
            "OU": "",
            "ST": ""
        }
    ]
}
EOF
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json |cfssljson -bare front-proxy-client

1.5 创建用于server-account认证的公钥与私钥

$ openssl genrsa -out sa.key 2048
$ openssl rsa -in sa.key -out sa.pub -pubout

2. 部署apiserver
#

$ swapoff -a
$ sed -i 's/.*swap.*/#&/' /etc/fstab
$ cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sysctl --system

2.1 二进制方式启动，由systemd托管

$ kube-apiserver \
--logtostderr=false \
--v=2 \
--log-file=/root/k8s-1.14.6/logs/kube-apiserver.log \
--advertise-address=192.168.18.142 \
--allow-privileged=true \
--authorization-mode=Node,RBAC \
--client-ca-file=/root/k8s-1.14.6/ssl/ca.pem \
--enable-admission-plugins=NodeRestriction \
--enable-bootstrap-token-auth=true \
--etcd-cafile=/root/k8s-1.14.6/ssl/etcd/ca.pem \
--etcd-certfile=/root/k8s-1.14.6/ssl/etcd/apiserver-etcd-client.pem \
--etcd-keyfile=/root/k8s-1.14.6/ssl/etcd/apiserver-etcd-client-key.pem \
--etcd-servers=https://192.168.18.142:2379 \
--insecure-port=0 \
--kubelet-client-certificate=/root/k8s-1.14.6/ssl/apiserver-kubelet-client.pem \
--kubelet-client-key=/root/k8s-1.14.6/ssl/apiserver-kubelet-client-key.pem \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--proxy-client-cert-file=/root/k8s-1.14.6/ssl/front-proxy/front-proxy-client.pem \
--proxy-client-key-file=/root/k8s-1.14.6/ssl/front-proxy/front-proxy-client-key.pem \
--requestheader-allowed-names=front-proxy-client \
--requestheader-client-ca-file=/root/k8s-1.14.6/ssl/front-proxy/ca.pem \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--secure-port=6443 \
--service-account-key-file=/root/k8s-1.14.6/ssl/sa.pub \
--service-cluster-ip-range=172.16.0.0/16 \
--tls-cert-file=/root/k8s-1.14.6/ssl/apiserver.pem \
--tls-private-key-file=/root/k8s-1.14.6/ssl/apiserver-key.pem

2.2 Staticpod方式

2.2.1 Master节点部署kubelet.service

$ swapoff –a                                                              --> 还要注释/etc/fstab中的swap挂载
$ cat <<EOF > /usr/lib/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/root/k8s-1.14.6/kubelet/bootstrap.kubeconfig --kubeconfig=/root/k8s-1.14.6/kubelet/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/root/k8s-1.14.6/kubelet/config.yaml"
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=/root/k8s-1.14.6/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_EXTRA_ARGS
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]	
WantedBy=multi-user.target
EOF

2.2.2 生成kubelet的kubeconfig文件kubelet.conf

$ cat create-kubelet-kubeconfig.sh
KUBE_APISERVER="https://192.168.18.142:6443"                               --> 高可用master集群的HA或者F5地址
HOSTNAME="192.168.18.142"                                                  --> 对应master节点IP
kubectl config set-cluster kubernetes \
    --certificate-authority=/root/k8s-1.14.6/ssl/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=kubelet.conf
kubectl config set-credentials system:node:${HOSTNAME} \
    --client-certificate=/root/k8s-1.14.6/ssl/apiserver-kubelet-client.pem \
    --client-key=/root/k8s-1.14.6/ssl/apiserver-kubelet-client-key.pem \
    --embed-certs=true \
    --kubeconfig=kubelet.conf
kubectl config set-context system:node:${HOSTNAME}@kubernetes \
    --cluster=kubernetes \
    --user=system:node:${HOSTNAME} \
    --kubeconfig=kubelet.conf
kubectl config use-context system:node:${HOSTNAME}@kubernetes --kubeconfig=kubelet.conf
$ sh create-kubelet-kubeconfig.sh

2.2.3 创建kubelet配置文件config.yaml与/etc/sysconfig/kubelet

$ cat config.yaml 
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /root/k8s-1.14.6/ssl/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd                                                          --> 默认为cgroups
cgroupsPerQOS: true
clusterDNS:
- 172.16.0.10                                                                  --> k8s集群DNS服务的clusterIP
clusterDomain: cluster.local                                                   --> k8s集群DNS服务的domain
configMapAndSecretChangeDetectionStrategy: Watch
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kind: KubeletConfiguration
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeLeaseDurationSeconds: 40
nodeStatusReportFrequency: 1m0s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /root/k8s-1.14.6/manifests                                     --> kubetlet读取静态Pod文件的路径
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

$ cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--hostname-override=192.168.18.142 --cert-dir=/root/k8s-1.14.6/kubelet/ssl --pod-infra-container-image=k8s.gcr.io/pause:3.1 --network-plugin=cni"

2.2.4 修改docker的cgroupdriver为systemd

$ cat /etc/docker/daemon.json 
{
        "exec-opts": ["native.cgroupdriver=systemd"],
        "insecure-registries": ["0.0.0.0/0"],
        "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:20008"],
        "graph": "/data/docker",
        "storage-driver": "overlay2",
        "storage-opts": [
           "overlay2.override_kernel_check=true"
        ],
        "tlsverify": true,
        "tlscacert": "/etc/docker/ca.pem",
        "tlscert": "/etc/docker/server-cert.pem",
        "tlskey": "/etc/docker/server-key.pem",
        "userland-proxy":false
}

2.2.5 创建cni配置文件

$ mkdir /etc/cni/net.d/ -pv
$ cat <<EOF > /etc/cni/net.d/10-flannel.conflist
{
  "name": "cbr0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
EOF

2.2.6 创建静态pod文件kube-apiserver-pod.yaml

$ cd /root/k8s-1.14.6/manifests
$ cat kube-apiserver-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --logtostderr=false
    - --log-file=/var/log/kube-apiserver.log
    - --advertise-address=192.168.18.142
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.pem
    - --enable-admission-plugins=NodeRestrictionc
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.pem
    - --etcd-certfile=/etc/kubernetes/pki/etcd/apiserver-etcd-client.pem
    - --etcd-keyfile=/etc/kubernetes/pki/etcd/apiserver-etcd-client-key.pem
    - --etcd-servers=https://192.168.18.142:2379                                --> etcd集群逗号分隔
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.pem
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client-key.pem
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy/front-proxy-client.pem
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy/front-proxy-client-key.pem
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy/ca.pem
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=172.16.0.0/16
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.pem
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem
    image: k8s.gcr.io/kube-apiserver:v1.14.6
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.18.142
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /var/log
      name: logs
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /root/k8s-1.14.6/ssl
    name: k8s-certs
  - hostPath:
      path: /root/k8s-1.14.6/logs
    name: logs
status: {}

2.2.7 启动kubelet.service

$ systemctl start kubelet.service

3. 配置kubectl访问apiserver
#

$ cd /root/k8s-1.14.6/conf
$ cat create-admin-kubeconfig.sh
KUBE_APISERVER="https://192.168.18.142:6443"
kubectl config set-cluster kubernetes \
    --certificate-authority=/root/k8s-1.14.6/ssl/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=admin.conf
kubectl config set-credentials kubernetes-admin \
    --client-certificate=/root/k8s-1.14.6/ssl/apiserver-kubelet-client.pem \
    --client-key=/root/k8s-1.14.6/ssl/apiserver-kubelet-client-key.pem \
    --embed-certs=true \
    --kubeconfig=admin.conf
kubectl config set-context kubernetes-admin@kubernetes \
    --cluster=kubernetes \
    --user=kubernetes-admin \
    --kubeconfig=admin.conf
kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=admin.conf
$ sh create-admin-kubeconfig.sh
$ mv admin.conf /root/.kube/config
$ kubectl get cs