Kubernetes(k8s) aws ec2 configuration
posted on 12 Aug 2020 under category note
1. 인스턴스 생성 후 터미널 접근 후 아래 sh파일 실행
cat << EoF > ~/kube-preparation.sh
#!/bin/bash
# check id
ID=\`whoami\`
if [ \$ID != "root" ]
then
echo "you must run as root"
exit 1
fi
# swapoff
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
swapoff -a
# disable selinux
setenforce 0
sed -i s/SELINUX=enforcing/SELINUX=disabled/ /etc/selinux/config
# install utils
yum install -y epel-release
yum install -y vim wget
# install docker
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io
systemctl enable docker && systemctl start docker
# change docker daemon driver from cgroupfs to systemd
cat <<EOF > /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload && systemctl restart docker
# edit kernel parameter
lsmod | grep br_netfilter
modprobe br_netfilter
cat << EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
# install kube binary
cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubeadm kubelet kubectl
systemctl enable kubelet
echo "success"
exit 0
EoF
2. Shell 수행
chmod +x ~/kube-preparation.sh
shell ~/kube-preparation.sh
1. 마스터 노드 구성
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address={ex)10.0.0.253}
1) 결과값
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.0.0.253:6443 --token sdr4tf.rx09vg869oe2mw4t \
--discovery-token-ca-cert-hash sha256:c2aa7a835dfe08870f2ead16051d287afcd1a4d5c3b1cdcd17dbbbe58ca9ef00
2. 정상노드 확인
[root@ip-10-0-0-253 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
ip-10-0-0-253.ap-northeast-2.compute.internal Ready master 7m51s v1.18.6
1. 노드 노출불가
[root@ip-10-0-0-253 ~]# kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@ip-10-0-0-253 ~]# mkdir -p $HOME/.kube
[root@ip-10-0-0-253 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@ip-10-0-0-253 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@ip-10-0-0-253 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
ip-10-0-0-253.ap-northeast-2.compute.internal NotReady master 27s v1.18.6
2. CNI 네트워크 에드온 Pending
[root@ip-10-0-0-253 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-5chfb 0/1 Pending 0 2m5s
kube-system coredns-66bff467f8-62lgx 0/1 Pending 0 2m5s
kube-system etcd-ip-10-0-0-253.ap-northeast-2.compute.internal 1/1 Running 0 2m22s
kube-system kube-apiserver-ip-10-0-0-253.ap-northeast-2.compute.internal 1/1 Running 0 2m22s
kube-system kube-controller-manager-ip-10-0-0-253.ap-northeast-2.compute.internal 1/1 Running 0 2m22s
kube-system kube-proxy-mvmfx 1/1 Running 0 2m6s
kube-system kube-scheduler-ip-10-0-0-253.ap-northeast-2.compute.internal 1/1 Running 0 2m21s
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
해결되지 않을경우 기존 flannel 삭제 후 calico 배포.
kubectl delete -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f https://docs.projectcalico.org/v3.11/manifests/calico.yaml
3. 정상 포드 확인
[root@ip-10-0-0-253 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-59877c7fb4-wzdtq 1/1 Running 0 53s
kube-system calico-node-77jw7 1/1 Running 0 53s
kube-system coredns-66bff467f8-5chfb 1/1 Running 0 7m26s
kube-system coredns-66bff467f8-62lgx 1/1 Running 0 7m26s
kube-system etcd-ip-10-0-0-253.ap-northeast-2.compute.internal 1/1 Running 0 7m43s
kube-system kube-apiserver-ip-10-0-0-253.ap-northeast-2.compute.internal 1/1 Running 0 7m43s
kube-system kube-controller-manager-ip-10-0-0-253.ap-northeast-2.compute.internal 1/1 Running 0 7m43s
kube-system kube-proxy-mvmfx 1/1 Running 0 7m27s
kube-system kube-scheduler-ip-10-0-0-253.ap-northeast-2.compute.internal 1/1 Running 0 7m42s
1. 워커 노드 구성(마스터노드에 연결)
kubeadm join 마스터노드아이피:6443 --token 마스터노드토큰 --discovery-token-ca-cert-hash "토큰검증을 위한 Root 인증서 해쉬값(마스터노드에서
확인 가능)"
예시)
kubeadm join 10.0.0.253:6443 --token sdr4tf.rx09vg869oe2mw4t \
--discovery-token-ca-cert-hash sha256:c2aa7a835dfe08870f2ead16051d287afcd1a4d5c3b1cdcd17dbbbe58ca9ef00
2. 추가 이후 [Master Node]에서 get node로 추가 노드 확인
[root@ip-10-0-0-253 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
ip-10-0-0-253.ap-northeast-2.compute.internal Ready master 24h v1.18.6
ip-10-0-0-84.ap-northeast-2.compute.internal Ready <none> 23h v1.18.6
1. 웹 대시보드 배포
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
1) 배포 확인
[root@ip-10-0-0-253 ~]# kubectl -n kubernetes-dashboard get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.100.86.114 <none> 8000/TCP 23h
kubernetes-dashboard ClusterIP 10.105.129.70 <none> 443/TCP 23h
2. ClusterIP 변경
kubectl -n kubernetes-dashboard edit service kubernetes-dashboard
spec:
clusterIP: 10.105.129.70
externalTrafficPolicy: Cluster
ports:
- nodePort: 31582
port: 443
protocol: TCP
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
[root@ip-10-0-0-253 ~]# kubectl -n kubernetes-dashboard get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.100.86.114 <none> 8000/TCP 24h
kubernetes-dashboard NodePort 10.105.129.70 <none> 443:31582/TCP 24h
3. 대시보드 관리자 계정 배포
cat << EOF > dashboard-admin.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
EOF
kubectl apply -f dashboard-admin.yaml
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
[root@ip-10-0-0-253 ~]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Data
====
ca.crt: 1025 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImRnTVl2Ri1Db3VQSEwtcURVdmNaX0k3eTk4bUNCeExUcmZwOWhtOXB0Y1EifQ.eyJpc3MiOiJrdWJlcm5ld
4. 대시보드 접근 확인
1. 2개의 nginx-web 포드 배포
cat << EOF > run-my-nginx.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
EOF
kubectl apply -f ./run-my-nginx.yaml
[root@ip-10-0-0-253 ~]# kubectl get pods -l run=my-nginx
NAME READY STATUS RESTARTS AGE
my-nginx-5dc4865748-c87j5 1/1 Running 0 3h30m
my-nginx-5dc4865748-vgqf5 1/1 Running 0 3h30m
2. Nginx 동작확인
kubectl expose deployment/my-nginx --type=NodePort
[root@ip-10-0-0-253 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 25h
my-nginx NodePort 10.97.235.46 <none> 80:30436/TCP 3h31m
3. Nginx ingress 배포
yum install git -y
로 인스톨 및 ingress사용에 필요한 배포 생성git clone [https://github.com/nginxinc/kubernetes-ingress/](https://github.com/nginxinc/kubernetes-ingress/)cd kubernetes-ingress/deployments
git checkout v1.7.
kubectl apply -f common/ns-and-sa.yaml
kubectl apply -f rbac/rbac.yaml
kubectl apply -f common/default-server-secret.yaml
kubectl apply -f common/nginx-config.yaml
kubectl apply -f common/vs-definition.yaml
kubectl apply -f common/vsr-definition.yaml
kubectl apply -f common/ts-definition.yaml
kubectl apply -f common/gc-definition.yaml
kubectl apply -f common/global-configuration.yaml
kubectl apply -f deployment/nginx-ingress.yaml
kubectl apply -f daemon-set/nginx-ingress.yaml
kubectl create -f service/nodeport.yaml
[root@ip-10-0-0-253 ~]# kubectl get pods --namespace=nginx-ingress
NAME READY STATUS RESTARTS AGE
nginx-ingress-7f659f75-rqfnq 1/1 Running 0 6h29m
nginx-ingress-hbm2z 1/1 Running 0 6h29m
cat << EOF > ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: hello-world.info
http:
paths:
- path: /
backend:
serviceName: my-nginx
servicePort: 80
EOF
kubectl apply -f ./ingress.yaml
[root@ip-10-0-0-253 ~]# kubectl describe ingress example-ingress
Name: example-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
hello-world.info
/ my-nginx:80 (192.168.200.16:80,192.168.200.17:80)
Annotations: ingress.kubernetes.io/rewrite-target: /
Events: <none>
[root@ip-10-0-0-253 ~]# kubectl get svc -n nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress NodePort 10.111.227.54 <none> 80:30069/TCP,443:32135/TCP 6h37m