실습 환경
- Mac m1 - VMware Fusion Player
- CPU/Memory/Disk는 VM 생성할 때 고정되는 값이므로, 변경이 된다면 Disk를 넉넉하게 잡는 게 좋음
- Disk 용량 부족시 추후에 Disk LVM을 진행할 예정
Node Name | OS | CPU | Memory | Disk | NIC IP | 관리자 계정 |
master | Ubuntu 22.04 | 2 | 4GB | 20GB | 172.16.133.4 | root/qwe123 |
worker | Ubuntu 22.04 | 2 | 4GB | 20GB | 172.16.133.5 | root/qwe123 |
Ubuntu 22.04 기본 설정
ssh로 root로그인 허용
sudo passwd root
qwe123 #root 비밀번호 설정
vi /etc/ssh/sshd_config
PermitRootLogin yes # 33번 째 줄 주석 풀고 바꿔주기
systemctl restart sshd # 변경사항 적용
hostname 지정, /etc/hosts 수정
# hostname 변경
hostnamectl hostname master
# 적용
su
# 확인
root@master:~# hostname
master
root@master:~# hostnamectl status
Static hostname: master
Icon name: computer-vm
Chassis: vm
Machine ID: 7a82c369cd7d4cf796d1e6ccc2069e34
Boot ID: 6a913b6af6d74b33923106df27f46713
Virtualization: vmware
Operating System: Ubuntu 22.04.4 LTS
Kernel: Linux 5.15.0-97-generic
Architecture: arm64
Hardware Vendor: VMware, Inc.
Hardware Model: VMware20,1
# hosts 파일에 master, worker 각각 추가해주기
root@master:~# vi /etc/hosts
root@master:~# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 master # 이 부분은 자동으로 들어감, worker는 worker로 자기 자신 loopback
(중략)
172.16.133.4 master
172.16.133.5 worker
root로 ssh접속 확인
ssh root@172.16.133.4
Network tool 설치, NTP Server 설정
# network 명령어 설치
root@master:~# apt install net-tools
# NTP Server 설정
root@master:~# apt update
root@master:~# apt upgrade
root@master:~# apt install ntp
root@master:~# systemctl enable ntp
Synchronizing state of ntp.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable ntp
root@master:~# systemctl status ntp
● ntp.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-02-28 04:14:59 UTC; 41s ago
Docs: man:ntpd(8)
Main PID: 20049 (ntpd)
Tasks: 2 (limit: 4524)
Memory: 1.3M
CPU: 31ms
CGroup: /system.slice/ntp.service
└─20049 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 113:119
Docker 설치
https://docs.docker.com/engine/install/ubuntu/
Install Docker Engine on Ubuntu
Jumpstart your client-side server applications with Docker Engine on Ubuntu. This guide details prerequisites and multiple methods to install Docker Engine on Ubuntu.
docs.docker.com
Docker 엔진은 containerd에 의존함, Docker의 컨테이너 런타임이 containerd이기 때문임
# Docker 엔진과 충돌할 수 있는 기존에 설치된 패키지 제거
root@master:~# for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
# Docker apt 저장소를 설정
# Docker 공식 GPG key 추가
root@master:~# apt-get update
root@master:~# apt-get install ca-certificates curl
root@master:~# install -m 0755 -d /etc/apt/keyrings
root@master:~# curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
root@master:~# chmod a+r /etc/apt/keyrings/docker.asc
# apt 소스에 리포지토리 추가
root@master:~# echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# 확인
root@master:~# cat /etc/apt/sources.list.d/docker.list
deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu jammy stable
root@master:~# apt-get update
# Docker 패키지 설치
root@master:~# apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Docker 버전 확인
root@master:~# docker version
Client: Docker Engine - Community
Version: 25.0.3
API version: 1.44
Go version: go1.21.6
Git commit: 4debf41
Built: Tue Feb 6 21:13:11 2024
OS/Arch: linux/arm64
Context: default
Server: Docker Engine - Community
Engine:
Version: 25.0.3
API version: 1.44 (minimum version 1.24)
Go version: go1.21.6
Git commit: f417435
Built: Tue Feb 6 21:13:11 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.28
GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
root@master:~# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-02-28 04:26:27 UTC; 1min 15s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 21508 (dockerd)
Tasks: 8
Memory: 26.9M
CPU: 205ms
CGroup: /system.slice/docker.service
└─21508 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root@master:~# systemctl enable docker
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
Kubernets 설치
https://kubernetes.io/docs/setup/production-environment/container-runtimes/
Container Runtimes
Note: Dockershim has been removed from the Kubernetes project as of release 1.24. Read the Dockershim Removal FAQ for further details. You need to install a container runtime into each node in the cluster so that Pods can run there. This page outlines what
kubernetes.io
필수 요소들 설치 및 구성하기
# IPv4를 포워딩하여 iptables가 브리지된 트래픽을 보게 하기
root@master:~# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
root@master:~# modprobe overlay
root@master:~# modprobe br_netfilter
# 확인
root@master:~# lsmod | grep overlay
overlay 155648 0
root@master:~# lsmod | grep br_netfilter
br_netfilter 32768 0
bridge 352256 1 br_netfilter
# 필요한 sysctl 파라미터를 설정하면, 재부팅 후에도 값이 유지됨
root@master:~# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# 확인
root@master:~# sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
# 재부팅하지 않고 sysctl 파라미터 적용하기
root@master:~# sysctl --system
# SWAP 메모리 제거
root@master:~# swapoff -a
root@master:~# free
total used free shared buff/cache available
Mem: 4005380 293932 1993924 1352 1717524 3531088
Swap: 0 0 0
Container Runtime(컨테이너 런타임) 설정 - containerd
- Docker 엔진은 containerd에 의존함, Docker의 컨테이너 런타임이 containerd이기 때문임
- 쿠버네티스에서 containerd를 사용하기 위해서는 CRI support가 활성화되어 있어야 함, CRI integration 플러그인은 기본적으로 비활성화되어 있기 때문에 활성화해줘야 함
- 따라서, cri가 /etc/containerd/config.toml 파일 안에 있는 disabled_plugins 목록에 포함되지 않도록 주의
- 만약 해당 파일을 변경했다면, containerd를 다시 시작해야 함
# config.toml 기본 설정, disabled_plugins에 cri가 포함됨을 확인 할 수 있음
root@master:~# cat /etc/containerd/config.toml
# Copyright 2018-2022 Docker Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
disabled_plugins = ["cri"]
#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0
#[grpc]
# address = "/run/containerd/containerd.sock"
# uid = 0
# gid = 0
#[debug]
# address = "/run/containerd/debug.sock"
# uid = 0
# gid = 0
# level = "info"
# containerd 기본 설정을 정의
root@master:~# containerd config default | tee /etc/containerd/config.toml
containerd systemd Cgroup Driver 환경 설정하기
- 리눅스에서 control group은 프로세스에 할당된 리소스를 제한하는 데 사용됨
- kubelet과 연계된 컨테이너 런타임 모두 control group과 상호작용을 해야 함
- Pod 및 Container 자원 관리
- CPU 혹은 메모리 같은 자원의 요청과 상한을 설정
- control group과 상호작용하기 위해서는 kubelet과 컨테이너 런타임이 cgroup 드라이버를 사용해야 함
- kubernets는 cgroupfs와 systemd 두 개의 Cgroup 드라이버가 있음
- 이때, kubelet과 컨테이너 런타임이 같은 cgroup 드라이버를 사용해야 하며, 구성도 동일해야 함
- kubelet, kubeadm은 default가 systemd임, 따라서 containerd의 cgroup도 systemd로 설정하기
# /etc/containerd/config.toml의 systemd cgroup 드라이버를 runc에서 사용하기 위해 다음과 같이 설정
root@master:~# vi /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
... 중략
SystemdCgroup = true
다른 부분에도 SystemdCgroup이 있기 때문에 반드시, plugins에 .containerd.runtimes.runc.options에 설정을 바꿔야 함!
containerd 변경사항 설정 파일 적용 (/etc/containerd/config.toml)
root@master:~# systemctl restart containerd
root@master:~# systemctl enable containerd
root@master:~# systemctl status containerd
● containerd.service - containerd container runtime
Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-02-28 05:50:56 UTC; 4s ago
Docs: https://containerd.io
Process: 21940 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 21941 (containerd)
Tasks: 8
Memory: 11.7M
CPU: 65ms
CGroup: /system.slice/containerd.service
└─21941 /usr/bin/containerd
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.827292159Z" level=info msg="Start subscribing containerd event"
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.827344393Z" level=info msg="Start recovering state"
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.827435771Z" level=info msg="Start event monitor"
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.827464952Z" level=info msg="Start snapshots syncer"
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.827476166Z" level=info msg="Start cni network conf syncer for default"
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.827482085Z" level=info msg="Start streaming server"
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.828062287Z" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.828116313Z" level=info msg=serving... address=/run/containerd/containerd.sock
Feb 28 05:50:56 master containerd[21941]: time="2024-02-28T05:50:56.828164962Z" level=info msg="containerd successfully booted in 0.022052s"
Feb 28 05:50:56 master systemd[1]: Started containerd container runtime.
kubeadm, kubelet, kubectl 설치
- kubeadm : 클러스터를 부트스트랩하는 명령어
- kubelet : 클러스터의 모든 머신에서 실행되는 파드와 컨테이너 시작과 같은 작업을 수행하는 컴포넌트
- kubectl : 클러스터와 통신하기 위한 커맨드 라인 유틸리티
https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
kubeadm 설치하기
이 페이지에서는 kubeadm 툴박스 설치 방법을 보여준다. 이 설치 프로세스를 수행한 후 kubeadm으로 클러스터를 만드는 방법에 대한 자세한 내용은 kubeadm으로 클러스터 생성하기 페이지를 참고한다.
kubernetes.io
# 쿠버네티스 apt 리포지터리를 사용하는데 필요한 패키지 설치
root@master:~# apt-get update
root@master:~# apt-get install -y apt-transport-https ca-certificates curl
# 쿠버네티스 GPG key 다운로드
root@master:~# curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg
root@master:~# echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
# 확인
root@master:~# cat /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
# apt 업데이트
root@master:~# apt-get update
Hit:1 https://download.docker.com/linux/ubuntu jammy InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [119 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main arm64 Packages [68.5 kB]
Hit:6 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease
Get:7 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [110 kB]
Fetched 298 kB in 3s (110 kB/s)
Reading package lists... Done
설치할 수 있는 kubeadm, kubelet, kubectl version 확인 후 특정 버전 설치(1.28.0-00)
root@master:~# apt-cache policy kubeadm
kubeadm:
Installed: (none)
Candidate: 1.28.2-00
Version table:
1.28.2-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.28.1-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.28.0-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.27.6-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
(중략)
root@master:~# apt-cache policy kubelet
kubelet:
Installed: (none)
Candidate: 1.28.2-00
Version table:
1.28.2-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.28.1-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.28.0-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.27.6-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.27.5-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
(중략)
root@master:~# apt-cache policy kubectl
kubectl:
Installed: (none)
Candidate: 1.28.2-00
Version table:
1.28.2-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.28.1-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.28.0-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.27.6-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.27.5-00 500
500 https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
1.27.4-00 500
(중략)
# 1.28.0-00 버전으로 설치
root@master:~# apt -y install kubelet=1.28.0-00 kubeadm=1.28.0-00 kubectl=1.28.0-00
# 해당 버전 고정
root@master:~# apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
root@master:~# systemctl daemon-reload
root@master:~# systemctl enable --now kubelet
kubelet이 active 상태가 아닌 loaded 되는 현상
root@master:~# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Wed 2024-02-28 07:58:57 UTC; 2s ago
Docs: https://kubernetes.io/docs/home/
Process: 24450 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/>
Main PID: 24450 (code=exited, status=1/FAILURE)
CPU: 60ms
Feb 28 07:58:57 master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Feb 28 07:58:57 master systemd[1]: kubelet.service: Failed with result 'exit-code'.
journalctl로 로그 보기
- /var/lib/kubelet/config.yaml 이 존재하지 않는다는 error 발생
- /var/lib/kubelet/config.yaml은 kubeadm init 이후에 생성되는 거기 때문에 당연함!
root@master:~# journalctl -exu kubelet -n 10
Feb 28 08:01:51 master kubelet[24573]: E0228 08:01:51.289090 24573 run.go:74]
"command failed" err="failed to load kubelet config file,
path: /var/lib/kubelet/config.yaml,
error: failed to load Kubelet config file /var/lib/kubelet/config.yaml,
error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\",
error: open /var/lib/kubelet/config.yaml: no such file or directory"
Feb 28 08:01:51 master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
여기까지 master, worker 동일하게 설정해 주기, snapshot 생성해도 되지만, Vmware Fusion Player 버전은 별도의 snapshot 기능을 제공하지 않음
Kubernetes Bootstraping(=kubeadm) 설정
https://v1-28.docs.kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
kubeadm init
This command initializes a Kubernetes control-plane node. Run this command in order to set up the Kubernetes control plane Synopsis Run this command in order to set up the Kubernetes control plane The "init" command executes the following phases: preflight
kubernetes.io
kubeadm init
# pod-network-cidr은 pod를 띄울 때 사용하고 싶은 IP, apiserver-advertise-address는 master IP
root@master:~# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.16.133.4
I0228 08:38:39.354789 25974 version.go:256] remote version is much newer: v1.29.2; falling back to: stable-1.28
[init] Using Kubernetes version: v1.28.7
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0228 08:39:18.671099 25974 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 172.16.133.4]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [172.16.133.4 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [172.16.133.4 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 7.502358 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 41p2t2.ae5patchsi4y7oo3
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.133.4:6443 --token 41p2t2.ae5patchsi4y7oo3 \
--discovery-token-ca-cert-hash sha256:97d1755c549fa30c867cbfb178fdc515f2e727a5804e503b8b5ca70a289c712d
kubernetes cluster를 사용하기 위한 명령어 실행
root@master:~# mkdir -p $HOME/.kube
root@master:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
root@master:~# chown $(id -u):$(id -g) $HOME/.kube/config
root@master:~# export KUBECONFIG=/etc/kubernetes/admin.conf
kubeadm join (클러스터 조인)
# 조인하고 싶은 worker1 node에서 실행
root@worker1:~# kubeadm join 172.16.133.4:6443 --token 41p2t2.ae5patchsi4y7oo3 \
--discovery-token-ca-cert-hash sha256:97d1755c549fa30c867cbfb178fdc515f2e727a5804e503b8b5ca70a289c712d
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
master에서 cluster join 됐는지 확인하기
아직 CNI(Container Network Interface가 설치되어 있지 않아 Status가 NotReady 상태
root@master:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master NotReady control-plane 8m5s v1.28.0 172.16.133.4 <none> Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.6.28
worker1 NotReady <none> 2m40s v1.28.0 172.16.133.5 <none> Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.6.28
CNI(Container Network Interface) 구성
Cilium설치
https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/
Cilium Quick Installation — Cilium 1.15.1 documentation
Install Cilium into the EKS cluster. Note If you have to uninstall Cilium and later install it again, that could cause connectivity issues due to aws-node DaemonSet flushing Linux routing tables. The issues can be fixed by restarting all pods, alternativel
docs.cilium.io
root@master:~# CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
root@master:~# CLI_ARCH=amd64
# amd64가 아닌 경우 arm64로 입력
root@master:~# if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
root@master:~# curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 36.2M 100 36.2M 0 0 11.3M 0 0:00:03 0:00:03 --:--:-- 19.1M
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 92 100 92 0 0 140 0 --:--:-- --:--:-- --:--:-- 140
root@master:~# sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
cilium-linux-arm64.tar.gz: OK
root@master:~# tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
cilium
root@master:~# rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
# cilium 설치
root@master:~# cilium install --version 1.15.1
ℹ️ Using Cilium version 1.15.1
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected kube-proxy has been installed
# cilium pod Running 확인하기, pending에서 바뀌는걸 볼 수 있음! 조금 걸리니까 5분 정도 기다리기
# Coredns도 Pending에서 Running으로 바뀜을 확인할 수 있음
root@master:~# watch kubectl get pods -A
Every 2.0s: kubectl get pods -A master: Wed Feb 28 08:59:26 2024
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-56hx2 1/1 Running 0 7m9s
kube-system cilium-7fzbc 1/1 Running 0 7m9s
kube-system cilium-operator-dd95cc587-v5j4s 1/1 Running 0 7m9s
kube-system coredns-5dd5756b68-46p7j 1/1 Running 0 19m
kube-system coredns-5dd5756b68-6sqp4 1/1 Running 0 19m
kube-system etcd-master 1/1 Running 0 19m
kube-system kube-apiserver-master 1/1 Running 0 19m
kube-system kube-controller-manager-master 1/1 Running 0 19m
kube-system kube-proxy-68qn6 1/1 Running 0 14m
kube-system kube-proxy-nv5dq 1/1 Running 0 19m
kube-system kube-scheduler-master 1/1 Running 0 19m
CNI 설치 완료 후 노드 Status NotReady에서 Ready로 바뀜을 확인하기
root@master:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 21m v1.28.0
worker1 Ready <none> 16m v1.28.0
root@master:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane 22m v1.28.0 172.16.133.4 <none> Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.6.28
worker1 Ready <none> 16m v1.28.0 172.16.133.5 <none> Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.6.28
Linux에서 Kubectl 명령어 자동 완성 사용하기
https://kubernetes.io/ko/docs/tasks/tools/included/optional-kubectl-configs-bash-linux/
리눅스에서 bash 자동 완성 사용하기
리눅스에서 bash 자동 완성을 위한 몇 가지 선택적 구성에 대해 설명한다.
kubernetes.io
root@master:~# echo 'source <(kubectl completion bash)' >>~/.bashrc
root@master:~# echo 'alias k=kubectl' >>~/.bashrc
root@master:~# echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
root@master:~# exec bash
'Container > Kubernetes' 카테고리의 다른 글
[k8s] minikube 클러스터 삭제 명령어 (0) | 2024.05.23 |
---|---|
[k8s] Minikube Kubernetes 용 ArgoCD설치 (0) | 2024.03.05 |
[k8s] Minikube에 node (worker) 추가하기 (0) | 2024.03.05 |
[k8s] Minikube란? 명령어, (Addon)애드온 (2) | 2024.03.05 |