[Intranet] Yasuhiro ABE, The University of Aizu.

1. PV and PVC

Persistent Volume(PV) is a feature of K8s that provides persistent storage like HDD/SSD.

In a container-based virtualization environment, when a process terminates, the contents previously stored inside are destroyed and initialized.

This is excellent in that if a process is contaminated from the outside, such as by a virus, it can be reset to its pre-contaminated state by restarting it, thus ensuring its integrity. However, this feature can be a problem for applications such as database servers, where the purpose is to store data.

In such virtualized environments, persistent (i.e., not disappearing when power is turned back on) storage is prepared in some form, which in Kubernetes is called PV.

In Kubernetes, PVs are prepared by the system. The user uses the prepared PV, requests a persistence area from the system (k8s) by defining a PVC (PV Claim), and stores the necessary data in the allocated area.

The k8s cluster used in this SCCP prepares PV in two major ways (Rook/Ceph and NFS). Since NFS is out of date at this time, we will present an example of using Rook/Ceph BlockStorage and FileStorage.

References: kubernetes.io::Persistent Volumes

1.1. PVC definition using BlockStorage (proprietary type)

PVCs of any capacity can be defined by using storageClassName: rook-ceph-block.

A sample definition file is shown below.

volumes.nginx-block-pvc.yaml.txt

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-ceph-block

name: is an arbitrary, easy-to-understand name.
storage: can be any required size.
When using Rook, accessModes: can only be "ReadWriteOnce".
If you use Rook, it is suitable for Database Server, because it can be accessed only from one Pod.

1.2. PVC definition using FileStorage (shared type)

BlockStorage is a private area that can be accessed by only one Pod, while FileStorage is a shared area that can be accessed by multiple Pods simultaneously. Previously, it was not possible to create FileStorage per namespace, but the version of Rook/Ceph has been improved and is now stable and usable.

volumes.nginx-file-pvc.yaml.txt

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-file-pvc
spec:
  accessModes: [ "ReadWriteMany" ]
  storageClassName: rook-cephfs
  resources:
    requests:
      storage: 1Gi

name: is an arbitrary, easy-to-understand name, as in BlockStorage.
storage: can be any required size, as in BlockStorage. The accessModes: can also be "ReadWriteOnce", but since the FileStorage is shared filesystem so please specify "ReadWriteMany" as is.
This is suitable for use on a web server that wants to share a single content, since it can be accessed by multiple pods (processes, servers).

2. Web server using BlockStorage

Prepare a web server (nginx-ceph-block) that uses BlockStorage.

First, prepare the following two YAML files by downloading or copying and pasting them.

volumes.deploy-nginx-blockstorage.yaml.txt.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-block-storage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-block-storage
  template:
    metadata:
      labels:
       app: nginx-block-storage
    spec:
      containers:
      - name: nginx-block-storage
        image: nginx:latest
        imagePullPolicy: "Always"
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-data-storage
          mountPath: /usr/share/nginx/html
      volumes:
      - name: nginx-data-storage
        persistentVolumeClaim:
          claimName: nginx-block-pvc

A separate Service definition is required to access this Pod.

volumes.svc-nginx-blockstorage.yaml.txt.

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-block-storage
spec:
  type: LoadBalancer
  ports:
     -  port: 80
        protocol: TCP
        targetPort: 80
  selector:
    app: nginx-block-storage

The first BlockStorage definition file and the instructions for downloading and running these two files are as follows

## If you have downloaded the files in advance, skip these three lines
$ wget https://web-int.u-aizu.ac.jp/~yasu-abe/en/sccp/manual/volumes.nginx-block-pvc.yaml.txt
$ wget https://web-int.u-aizu.ac.jp/~yasu-abe/en/sccp/manual/volumes.deploy-nginx-blockstorage.yaml.txt
$ wget https://web-int.u-aizu.ac.jp/~yasu-abe/en/sccp/manual/volumes.svc-nginx-blockstorage.yaml.txt

## Run the downloaded file with the kubectl command
$ kubectl -n $(id -un) apply -f volumes.nginx-block-pvc.yaml.txt
$ kubectl -n $(id -un) apply -f volumes.deploy-nginx-blockstorage.yaml.txt
$ kubectl -n $(id -un) apply -f volumes.svc-nginx-blockstorage.yaml.txt

After executing this step, connect to the configured web server with a web browser.

## Open a web browser and connect to the web server you created
$ browse http://$(kubectl -n $(id -un) get svc nginx-block-storage -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

Since the content (index.html) file has not yet been placed, use the following command to place the file.

$ kubectl -n $(id -un) exec -it "$(kubectl -n $(id -un) get pod -l app=nginx-block-storage -o jsonpath={.items[0].metadata.name})" -- bash -c "echo Hello World, $(id -un) at $(date) > /usr/share/nginx/html/index.html"

Go back to your web browser again and verify that the page is displayed.

3. Web server using FileStorage

Next, prepare the web server that will use FileStorage.

volumes.deploy-nginx-filestorage.yaml.txt

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-file-storage
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-file-storage
  template:
    metadata:
      labels:
       app: nginx-file-storage
    spec:
      containers:
      - name: nginx-file-storage
        image: nginx:latest
        imagePullPolicy: "Always"
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-data-storage
          mountPath: /usr/share/nginx/html
      volumes:
      - name: nginx-data-storage
        persistentVolumeClaim:
          claimName: nginx-file-pvc

Next, define a Service to access the Pod.

volumes.svc-nginx-filestorage.yaml.txt

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-file-storage
spec:
  type: LoadBalancer
  ports:
     -  port: 80
        protocol: TCP
        targetPort: 80
  selector:
    app: nginx-file-storage

Similar to the BlockStorage example, here are the steps to download and run the YAML file

## If you have downloaded the file in advance, skip these three lines
$ wget https://web-int.u-aizu.ac.jp/~yasu-abe/en/sccp/manual/volumes.nginx-file-pvc.yaml.txt
$ wget https://web-int.u-aizu.ac.jp/~yasu-abe/en/sccp/manual/volumes.deploy-nginx-filestorage.yaml.txt
$ wget https://web-int.u-aizu.ac.jp/~yasu-abe/en/sccp/manual/volumes.svc-nginx-filestorage.yaml.txt

## Run the downloaded file with the kubectl command
$ kubectl -n $(id -un) apply -f volumes.nginx-file-pvc.yaml.txt
$ kubectl -n $(id -un) apply -f volumes.deploy-nginx-filestorage.yaml.txt
$ kubectl -n $(id -un) apply -f volumes.svc-nginx-filestorage.yaml.txt

Once you have done this, connect to your configured web server with a web browser.

$ browse http://$(kubectl -n $(id -un) get svc nginx-file-storage -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

Since the content (index.html) file has not yet been placed, use the following command to place the file.

$ kubectl -n $(id -un) exec -it "$(kubectl -n $(id -un) get pod -l app=nginx-file-storage -o jsonpath={.items[0].metadata.name})" -- bash -c "echo Hello World, $(id -un) at $(date) > /usr/share/nginx/html/index.html"

Go back to your web browser again and verify that the page is displayed.

4. Characteristics of each web server

We have configured two types of web servers (nginx) here.

The configuration files for each are almost the same, although there are some differences between "block" and "file" configuration files. Here we explain the subtle differences.

4.1. [Review] How to open a web page

## Connect to a web server that uses Block Storage
$ browse http://$(kubectl -n $(id -un) get svc nginx-block-storage -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

## Connect to a web server that uses File Storage
$ browse http://$(kubectl -n $(id -un) get svc nginx-file-storage -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

4.2. replicas: difference in configuration

If you look closely, in the two Deployment YAML files (deploy-nginx-blockstorage.yaml.txt, deploy-nginx-filestorage.yaml.txt), there is a line that says replicas:. (.spec.replicas)

This specifies the number of nginx Pods (processes) that can run concurrently. It is preferable to have more than one running, as it can withstand more accesses, but BlockStorage’s 1 does not work well with more than one.

To check this behavior, try changing the number of replicas from 1 to 2.

The following command will launch emacs and change the number of replicas:.

$ env EDITOR=emacs kubectl -n $(id -un) edit deploy nginx-block-storage
## Exit emacs when you are done editing.

There is no change in behavior after making this change.

Let’s check the current status from the get command.

$ kubectl -n $(id -un) get pod

You will then see something like this

NAME READY STATUS RESTARTS AGE
nginx-block-storage-5f47458575-g7wvm 1/1 Running 0 84s
nginx-block-storage-5f47458575-nkbg4 0/1 ContainerCreating 0 7s
nginx-file-storage-7976586c4b-4p4xj 1/1 Running 0 26m
nginx-file-storage-7976586c4b-lbnw9 1/1 Running 0 26m

Pods with STATUS of ContainerCreating on the third line from the top will not work forever.

You can find this position with the describe command.

$ kubectl -n $(id -un) describe pod -l app=nginx-block-storage

You should then see a message similar to the following, ending with

 Warning  FailedMount         15s                   kubelet                  Unable to attach or mount volumes:
     unmounted volumes=[nginx-data-storage], unattached volumes=[kube-api-access-kq479 nginx-data-storage]:
     timed out waiting for the condition

This shows that nginx-data-storage is trying to mount the requested storage area (file system, PVC) and failing.

This is because BlockStorage is a proprietary type that can only be used from a single Pod.

FileStorage is unique because it can be shared with multiple Pods. After changing index.html once, the same content can be seen no matter how many times it is accessed from a web browser, indicating that the same file is referenced from multiple web servers.

Issue: If you can afford it, try increasing the number of replica: in deploy/nginx-file-storage from 2.

5. Check currently defined PVCs.

You can also check the status of your current PVCs by applying kubectl’s get command.

$ kubectl -n $(id -un) get pvc

The contents displayed will vary depending on your state. The following is an example of one output.

NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
nginx-block-pvc   Bound    pvc-b37bf308-310b-4679-90c6-174bcc0dcf6e   1Gi        RWO            rook-ceph-block   77m
nginx-file-pvc    Bound    pvc-c2ad02e3-ca2d-4c88-98b5-3999371f804d   1Gi        RWX            rook-cephfs       44m

6. Applied problems

6.1. Q1. [Create Block Storage with any name]

Download the Block Storage YAML file (nginx-block-pvc.yaml.txt, download) and change name: using an editor or other tool to create a PVC with a different name with a different name.

$ kubectl -n $(id -un) get pvc to see the before and after changes.

A1. Solution example

## Check current status

$ kubectl -n $(id -un) get pvc

## Create a file with a nonexistent filename (in this case, new-block-pvc.yaml)

$ curl https://web-int.u-aizu.ac.jp/%7eyasu-abe/ja/sccp/manual/volumes.nginx-block-pvc.yaml.txt > new-block-pvc.yaml

## Change name to the new name with an editor or similar. In this case, use the sed command.

$ sed -i -e 's/name: . *$/name: new-block-pvc/g' new-block-pvc.yaml

## Reflect the contents of new-block-pvc.yaml

$ kubectl -n $(id -un) apply -f new-block-pvc.yaml

## Check the status after the change

$ kubectl -n $(id -un) get pvc

6.2. Q2. [Delete Block Storage]

Please delete the Block Storage created in Q1.

Confirm the changes before and after with $ kubectl -n $(id -un) get pvc.

A2. Example solution

## Method 1. delete using prepared YAML file

$ kubectl -n $(id -un) delete -f new-block-pvc.yaml

## Method 2. use command::get TYPE::pvc to delete a target NAME.

$ kubectl -n $(id -un) get pvc

$ kubectl -n $(id -un) delete pvc new-block-pvc

6.3. Q3. [Extending BlockStorage

Storage capacity can be extended by specifying a larger value for storage:. Please change the size of nginx-block-pvc created by Block Storage to 2Gi.

A3. Example solution

## Method 1. edit the YAML file used to create nginx-block-pvc

$ curl https://web-int.u-aizu.ac.jp/~yasu-abe/en/sccp/manual/volumes.nginx-block-pvc.yaml.txt > nginx-block-pvc.yaml

$ sed -i -e 's/storage: . *$/storage: 2Gi/' nginx-block-pvc.yaml

$ kubectl -n $(id -un) apply -f nginx-block-pvc.yaml

## Method 2. Use command::edit to resize storage:.

$ env EDITOR=emacs kubectl -n $(id -un) edit pvc nginx-block-pvc

Notes. Changing only the size (storage:) of a PVC that has already been created is not immediately reflected. Only when the PVC is actually used will it be available in the changed size. For this reason, we use PVCs that are already connected to the pod, not PVCs created in Q1 and Q2.

6.4. Q4.[Application to MySQL server]

It is assumed that the official MySQL server image (DockerHub official MySQL container image) is used.

Please run the database server in the following manner.

create BlockStorage (rewrite volumes.nginx-block-pvc.yaml.txt)
define Deployment to create MySQL server (rewrite volumes.deploy-nginx-blockstorage.yaml.txt)
define a Service to connect to the MySQL server (volumes.svc-nginx-blockstorage.yaml.txt)
confirm that you can connect from mysql command

There is more than one way to do this. The hints and example solutions assume the configuration that seems most common with few changes.

Hints

The line to rewrite volumes.nginx-block-pvc.yaml.txt is the following item

name: line

The line to rewrite volumes.deploy-nginx-blockstorage.yaml.txt is the following item

・app: line(s)

・image: row(s)

・containerPort: row(s)

・ mountPath: row

The official MySQL image (mysql:8.0) has the following configuration for each of these values. Reference information - Dockerfile mysql:8

・ Port number for connection (containerPort): 3306

・ Data destination (mountPath): /var/lib/mysql

You also need to add a new env: setting (name: MYSQL_ROOT_PASSWORD, value: "secret"). (The secret part can be rewritten as desired.)

For more information on how to set this up, please refer to k8s official guide - Deploying MySQL.

The line to rewrite volumes.svc-nginx-blockstorage.yaml.txt is as follows

・ name: line

・ port:/targetPort: line (same port number as in Deployment)

・ app: line (same as matchLabels: specified in Deployment)

Please note that the service changes, with the exception of name:, must be consistent with the contents described in Deployment, or network connection will not be established.

To check the operation, make sure that the mysql command is installed on each ThinkPad terminal and that the following commands can be executed.

## The command line is constructed assuming that the Service name (name:) is "mysql-svc".
$ mysql -u root -p -h $(kubectl -n $(id -un) get svc mysql-svc -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

## Password is the value you set in the MYSQL_ROOT_USER environment variable in Deployment (e.g. secret)
Enter password: '{status.loadBalancer.ingress[0].ip}'.

## If User information is displayed in the following SQL, it is success.
mysql> select host,user from mysql.user;

+-----------+------------------+
| host | user |
+-----------+------------------+
| % | root | localhost | mysql.infoschema | mysql.infoschema
| localhost | mysql.infoschema |
| localhost | mysql.session | mysql.sys
| localhost | mysql.sys | mysql.session
| localhost | mysql.infoschema | mysql.session | localhost | mysql.sys
+-----------+------------------+
5 rows in set (0.00 sec)

A4. Example solution (link to each YAML file)

Details

volumes.deploy-mysql-blockstorage.yaml.txt

volumes.svc-mysql-blockstorage.yaml.txt]

volumes.mysql-block-pvc.yaml.txt]