📚 第7章 持久化存储

7.1 存储基础概念

7.1.1 Kubernetes存储概述

Kubernetes提供了多种存储解决方案来满足不同的应用需求：

• 临时存储：Pod重启后数据丢失
• 持久化存储：数据在Pod重启后仍然保留
• 共享存储：多个Pod可以同时访问的存储
• 动态存储：根据需求自动创建存储资源

7.1.2 存储类型

# 存储类型分类
apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-types
data:
  ephemeral: |
    - emptyDir
    - hostPath (临时使用)
  persistent: |
    - PersistentVolume (PV)
    - PersistentVolumeClaim (PVC)
  network: |
    - NFS
    - Ceph
    - GlusterFS
    - iSCSI
  cloud: |
    - AWS EBS
    - Azure Disk
    - GCE Persistent Disk

7.2 Volume详解

7.2.1 emptyDir

apiVersion: v1
kind: Pod
metadata:
  name: emptydir-pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: cache-volume
      mountPath: /cache
  - name: sidecar
    image: busybox
    command: ['sh', '-c', 'while true; do echo $(date) >> /cache/log.txt; sleep 10; done']
    volumeMounts:
    - name: cache-volume
      mountPath: /cache
  volumes:
  - name: cache-volume
    emptyDir:
      sizeLimit: 1Gi
      medium: Memory  # 使用内存作为存储介质

7.2.2 hostPath

apiVersion: v1
kind: Pod
metadata:
  name: hostpath-pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: host-volume
      mountPath: /usr/share/nginx/html
  volumes:
  - name: host-volume
    hostPath:
      path: /data/nginx
      type: DirectoryOrCreate

7.2.3 configMap和secret卷

apiVersion: v1
kind: Pod
metadata:
  name: config-pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config
    - name: secret-volume
      mountPath: /etc/secret
      readOnly: true
  volumes:
  - name: config-volume
    configMap:
      name: app-config
      items:
      - key: app.properties
        path: app.properties
        mode: 0644
  - name: secret-volume
    secret:
      secretName: app-secret
      defaultMode: 0400

7.3 PersistentVolume (PV)

7.3.1 PV基本概念

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-example
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /data/pv-example

7.3.2 访问模式

# 访问模式说明
apiVersion: v1
kind: ConfigMap
metadata:
  name: access-modes
data:
  ReadWriteOnce: |
    - 简写：RWO
    - 只能被单个节点以读写模式挂载
    - 最常用的模式
  ReadOnlyMany: |
    - 简写：ROX
    - 可以被多个节点以只读模式挂载
    - 适用于共享配置文件
  ReadWriteMany: |
    - 简写：RWX
    - 可以被多个节点以读写模式挂载
    - 需要支持的存储系统
  ReadWriteOncePod: |
    - 简写：RWOP
    - 只能被单个Pod以读写模式挂载
    - Kubernetes 1.22+

7.3.3 回收策略

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-reclaim-policy
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete  # Retain, Delete, Recycle
  storageClassName: fast
  nfs:
    server: nfs-server.example.com
    path: /path/to/data

7.4 PersistentVolumeClaim (PVC)

7.4.1 PVC基本使用

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-example
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: manual
  selector:
    matchLabels:
      environment: production
---
apiVersion: v1
kind: Pod
metadata:
  name: pvc-pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: storage
      mountPath: /usr/share/nginx/html
  volumes:
  - name: storage
    persistentVolumeClaim:
      claimName: pvc-example

7.4.2 动态PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-storage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: storage-app
  template:
    metadata:
      labels:
        app: storage-app
    spec:
      containers:
      - name: app
        image: postgres:13
        env:
        - name: POSTGRES_DB
          value: mydb
        - name: POSTGRES_USER
          value: user
        - name: POSTGRES_PASSWORD
          value: password
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: postgres-storage
        persistentVolumeClaim:
          claimName: dynamic-pvc

7.5 StorageClass

7.5.1 StorageClass基础

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

7.5.2 不同存储提供商的StorageClass

# AWS EBS StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-ebs-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
# Azure Disk StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-disk-premium
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_LRS
  cachingmode: ReadOnly
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
# GCE Persistent Disk StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gce-pd-ssd
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  replication-type: regional-pd
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

7.5.3 本地存储StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node-1

7.6 CSI (Container Storage Interface)

7.6.1 CSI驱动安装

# CSI驱动示例 - NFS CSI
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: nfs.csi.k8s.io
spec:
  attachRequired: false
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent
  - Ephemeral
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: csi-nfs-node
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-nfs-node
  template:
    metadata:
      labels:
        app: csi-nfs-node
    spec:
      hostNetwork: true
      containers:
      - name: node-driver-registrar
        image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0
        args:
        - --v=2
        - --csi-address=/csi/csi.sock
        - --kubelet-registration-path=/var/lib/kubelet/plugins/nfs.csi.k8s.io/csi.sock
        volumeMounts:
        - name: plugin-dir
          mountPath: /csi
        - name: registration-dir
          mountPath: /registration
      - name: nfs
        image: k8s.gcr.io/sig-storage/nfsplugin:v4.1.0
        args:
        - --nodeid=$(NODE_ID)
        - --endpoint=$(CSI_ENDPOINT)
        env:
        - name: NODE_ID
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: CSI_ENDPOINT
          value: unix:///csi/csi.sock
        volumeMounts:
        - name: plugin-dir
          mountPath: /csi
        - name: pods-mount-dir
          mountPath: /var/lib/kubelet/pods
          mountPropagation: Bidirectional
      volumes:
      - name: plugin-dir
        hostPath:
          path: /var/lib/kubelet/plugins/nfs.csi.k8s.io
          type: DirectoryOrCreate
      - name: registration-dir
        hostPath:
          path: /var/lib/kubelet/plugins_registry
          type: Directory
      - name: pods-mount-dir
        hostPath:
          path: /var/lib/kubelet/pods
          type: Directory

7.6.2 使用CSI存储

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs-server.example.com
  share: /path/to/nfs/share
volumeBindingMode: Immediate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: nfs-csi

7.7 存储快照

7.7.1 VolumeSnapshotClass

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-snapclass
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  encrypted: "true"

7.7.2 创建存储快照

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pvc-snapshot
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: my-pvc
---
# 从快照恢复PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restored-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: pvc-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  storageClassName: fast-ssd

7.8 存储管理脚本

7.8.1 存储监控脚本

#!/bin/bash
# storage-monitor.sh

echo "=== 存储资源监控 ==="

echo "\n1. PersistentVolumes状态:"
kubectl get pv -o custom-columns=NAME:.metadata.name,CAPACITY:.spec.capacity.storage,ACCESS:.spec.accessModes,RECLAIM:.spec.persistentVolumeReclaimPolicy,STATUS:.status.phase,CLAIM:.spec.claimRef.name

echo "\n2. PersistentVolumeClaims状态:"
kubectl get pvc --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,VOLUME:.spec.volumeName,CAPACITY:.status.capacity.storage,ACCESS:.status.accessModes

echo "\n3. StorageClasses:"
kubectl get storageclass -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,RECLAIM:.reclaimPolicy,BINDING:.volumeBindingMode

echo "\n4. 存储使用情况:"
for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
    echo "\n节点: $node"
    kubectl describe node $node | grep -A 5 "Allocated resources"
done

echo "\n5. 存储相关事件:"
kubectl get events --all-namespaces --field-selector reason=FailedMount,reason=FailedAttachVolume,reason=VolumeResizeFailed --sort-by='.lastTimestamp'

7.8.2 存储清理脚本

#!/bin/bash
# storage-cleanup.sh

echo "=== 存储资源清理 ==="

# 清理未绑定的PV
echo "\n1. 清理Available状态的PV:"
kubectl get pv --no-headers | grep Available | awk '{print $1}' | while read pv; do
    echo "删除PV: $pv"
    kubectl delete pv $pv
done

# 清理失败的PVC
echo "\n2. 清理Lost状态的PVC:"
kubectl get pvc --all-namespaces --no-headers | grep Lost | while read namespace pvc rest; do
    echo "删除PVC: $namespace/$pvc"
    kubectl delete pvc $pvc -n $namespace
done

# 清理孤儿VolumeAttachment
echo "\n3. 清理孤儿VolumeAttachment:"
kubectl get volumeattachment --no-headers | while read va node rest; do
    if ! kubectl get node $node &>/dev/null; then
        echo "删除VolumeAttachment: $va (节点 $node 不存在)"
        kubectl delete volumeattachment $va
    fi
done

echo "\n清理完成!"

7.8.3 存储备份脚本

#!/bin/bash
# storage-backup.sh

NAMESPACE=${1:-default}
BACKUP_DIR="/backup/$(date +%Y%m%d-%H%M%S)"

echo "=== 存储备份 ==="
echo "命名空间: $NAMESPACE"
echo "备份目录: $BACKUP_DIR"

mkdir -p $BACKUP_DIR

# 备份PVC定义
echo "\n1. 备份PVC定义:"
kubectl get pvc -n $NAMESPACE -o yaml > $BACKUP_DIR/pvcs.yaml

# 备份StorageClass定义
echo "2. 备份StorageClass定义:"
kubectl get storageclass -o yaml > $BACKUP_DIR/storageclasses.yaml

# 创建存储快照
echo "3. 创建存储快照:"
kubectl get pvc -n $NAMESPACE --no-headers | while read pvc rest; do
    cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: backup-$pvc-$(date +%Y%m%d-%H%M%S)
  namespace: $NAMESPACE
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: $pvc
EOF
    echo "创建快照: backup-$pvc-$(date +%Y%m%d-%H%M%S)"
done

echo "\n备份完成: $BACKUP_DIR"

7.9 存储故障排查

7.9.1 常见存储问题

#!/bin/bash
# storage-troubleshoot.sh

echo "=== 存储故障排查 ==="

# 检查PVC绑定问题
echo "\n1. 检查未绑定的PVC:"
kubectl get pvc --all-namespaces | grep Pending

# 检查存储驱动
echo "\n2. 检查CSI驱动状态:"
kubectl get csidriver
kubectl get csistoragecapacity --all-namespaces

# 检查存储节点
echo "\n3. 检查存储节点状态:"
kubectl get csinode

# 检查存储相关Pod
echo "\n4. 检查存储相关Pod:"
kubectl get pods -n kube-system | grep -E "csi|storage|provisioner"

# 检查存储事件
echo "\n5. 检查存储相关事件:"
kubectl get events --all-namespaces --field-selector reason=FailedMount
kubectl get events --all-namespaces --field-selector reason=FailedAttachVolume

# 检查节点存储容量
echo "\n6. 检查节点存储容量:"
kubectl top nodes
kubectl describe nodes | grep -A 5 "Allocated resources"

7.9.2 存储性能测试

# 存储性能测试Pod
apiVersion: v1
kind: Pod
metadata:
  name: storage-benchmark
spec:
  containers:
  - name: fio
    image: ljishen/fio
    command:
    - fio
    - --name=test
    - --ioengine=libaio
    - --iodepth=64
    - --rw=randwrite
    - --bs=4k
    - --direct=1
    - --size=1G
    - --numjobs=4
    - --runtime=60
    - --group_reporting
    - --filename=/data/testfile
    volumeMounts:
    - name: test-volume
      mountPath: /data
  volumes:
  - name: test-volume
    persistentVolumeClaim:
      claimName: benchmark-pvc
  restartPolicy: Never
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: benchmark-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd

7.10 存储最佳实践

7.10.1 存储规划

# 存储规划示例
apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-planning
data:
  guidelines: |
    1. 存储类型选择:
       - 数据库: 高IOPS SSD存储
       - 日志文件: 标准存储
       - 临时文件: emptyDir
       - 配置文件: ConfigMap/Secret
    
    2. 容量规划:
       - 预留20-30%的增长空间
       - 考虑备份和快照需求
       - 监控存储使用趋势
    
    3. 性能考虑:
       - 选择合适的存储类型
       - 配置适当的IOPS和吞吐量
       - 考虑网络延迟影响
    
    4. 可用性设计:
       - 使用副本和备份
       - 跨可用区部署
       - 定期测试恢复流程

7.10.2 存储安全

# 存储加密配置
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: encrypted-storage
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
  kmsKeyId: "arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
# RBAC for storage
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: storage-admin
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
  resources: ["volumesnapshots", "volumesnapshotclasses"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

7.11 本章总结

本章详细介绍了Kubernetes中的持久化存储，主要内容包括：

核心概念

• Volume类型：emptyDir、hostPath、configMap、secret等
• PV/PVC：持久化卷和持久化卷声明的概念和使用
• StorageClass：动态存储供应和存储类配置
• CSI：容器存储接口和第三方存储驱动

存储管理

• 存储监控：监控存储资源使用情况和状态
• 存储备份：创建存储快照和备份策略
• 故障排查：诊断和解决存储相关问题
• 性能优化：存储性能测试和调优

最佳实践

• 存储规划：合理选择存储类型和容量规划
• 安全配置：存储加密和访问控制
• 高可用设计：跨可用区存储和备份策略
• 成本优化：存储资源的合理使用和清理

注意事项

• 选择合适的存储类型和访问模式
• 注意存储的生命周期管理
• 定期备份重要数据
• 监控存储性能和容量使用

下一章我们将学习Kubernetes的网络管理，包括网络策略、服务网格等高级网络功能。