Kubernetes 高可用与配置实践

面试指北 · 通用

在K8s环境下部署确实需要很多调整，特别是可以利用云原生的优势来实现更强大的熔断降级机制。让我为你详细说明：

K8s环境下的高可用架构

graph TB
    A[用户请求] --> B[Ingress Controller<br/>NGINX/Traefik]
    B --> C[Service Mesh<br/>Istio/Linkerd]

    C --> D[GIS App Pod 1]
    C --> E[GIS App Pod 2] 
    C --> F[GIS App Pod N]

    D --> G[ConfigMap<br/>配置管理]
    E --> G
    F --> G

    D --> H[Secret<br/>敏感信息]
    E --> H
    F --> H

    I[HPA<br/>水平扩缩容] --> D
    I --> E
    I --> F

    J[Redis Cluster<br/>StatefulSet] --> D
    J --> E
    J --> F

    K[PostgreSQL<br/>Operator] --> D
    K --> E
    K --> F

    L[Prometheus<br/>监控] --> D
    L --> E
    L --> F
    L --> J
    L --> K

1. Deployment和Service配置

应用部署配置

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gis-app
  labels:
    app: gis-app
    version: v1
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0  # 保证滚动更新时服务不中断
  selector:
    matchLabels:
      app: gis-app
  template:
    metadata:
      labels:
        app: gis-app
        version: v1
    spec:
      containers:
      - name: gis-app
        image: gis-app:latest
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: "k8s"
        - name: REDIS_HOST
          value: "redis-cluster"
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: host

        # 资源限制
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

        # 健康检查探针
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 120
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3

        # 启动探针（避免慢启动被杀死）
        startupProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 30

        # 优雅关闭
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]

      # Pod反亲和性（确保Pod分布在不同节点）
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - gis-app
              topologyKey: kubernetes.io/hostname

---
apiVersion: v1
kind: Service
metadata:
  name: gis-app-service
spec:
  selector:
    app: gis-app
  ports:
  - port: 80
    targetPort: 8080
    name: http
  type: ClusterIP

2. HPA自动扩缩容

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gis-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gis-app
  minReplicas: 3
  maxReplicas: 20
  metrics:
  # CPU利用率
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  # 内存利用率
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  # 自定义指标：QPS
  - type: Pods
    pods:
      metric:
        name: requests_per_second
      target:
        type: AverageValue
        averageValue: "100"

  # 扩缩容行为配置
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # 5分钟稳定期
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 60   # 1分钟稳定期
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60

3. Istio服务网格熔断降级

VirtualService路由配置

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: gis-app-vs
spec:
  hosts:
  - gis-app-service
  http:
  - match:
    - headers:
        priority:
          exact: "high"
    route:
    - destination:
        host: gis-app-service
        subset: v1
      weight: 100
    timeout: 10s
    retries:
      attempts: 3
      perTryTimeout: 3s
      retryOn: 5xx,gateway-error,connect-failure,refused-stream

  # 默认路由
  - route:
    - destination:
        host: gis-app-service
        subset: v1
      weight: 100
    timeout: 30s
    fault:
      # 故障注入测试
      delay:
        percentage:
          value: 0.1
        fixedDelay: 5s

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: gis-app-dr
spec:
  host: gis-app-service
  trafficPolicy:
    # 连接池配置
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
        maxRetries: 3
        consecutiveGatewayErrors: 5
        interval: 30s
        baseEjectionTime: 30s
        maxEjectionPercent: 50

    # 熔断器配置
    outlierDetection:
      consecutive5xxErrors: 5
      consecutiveGatewayErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 30

  subsets:
  - name: v1
    labels:
      version: v1

4. 应用级熔断降级代码调整

/**
 * K8s环境下的熔断降级配置
 */
@Configuration
public class K8sResilienceConfig {

    @Bean
    public CircuitBreaker geoDataCircuitBreaker() {
        return CircuitBreaker.ofDefaults("geoDataService");
    }

    @Bean
    public Retry geoDataRetry() {
        return Retry.ofDefaults("geoDataService");
    }

    @Bean
    public Bulkhead geoDataBulkhead() {
        return Bulkhead.of("geoDataService", BulkheadConfig.custom()
            .maxConcurrentCalls(25)
            .maxWaitDuration(Duration.ofMillis(500))
            .build());
    }

    @Bean
    public TimeLimiter geoDataTimeLimiter() {
        return TimeLimiter.of("geoDataService", TimeLimiterConfig.custom()
            .timeoutDuration(Duration.ofSeconds(5))
            .build());
    }
}

/**
 * K8s环境下的GIS服务
 */
@Service
public class K8sGeoDataService {

    @Autowired
    private CircuitBreaker circuitBreaker;

    @Autowired
    private Retry retry;

    @Autowired
    private Bulkhead bulkhead;

    @Autowired
    private TimeLimiter timeLimiter;

    @Autowired
    private GeoDataRepository repository;

    @Autowired
    private RedisTemplate<String, Object> redisTemplate;

    /**
     * 带有完整保护机制的地理数据查询
     */
    public CompletableFuture<GeoDataResponse> getGeoDataAsync(String layerId, String bounds) {

        Supplier<CompletableFuture<GeoDataResponse>> decoratedSupplier = 
            CircuitBreaker.decorateSupplier(circuitBreaker, () -> {
                return Retry.decorateSupplier(retry, () -> {
                    return Bulkhead.decorateSupplier(bulkhead, () -> {
                        return CompletableFuture.supplyAsync(() -> {
                            try {
                                return queryGeoData(layerId, bounds);
                            } catch (Exception e) {
                                // 降级到缓存数据
                                return getCachedDataOrDefault(layerId, bounds);
                            }
                        });
                    }).get();
                }).get();
            });

        // 应用超时限制
        return timeLimiter.executeCompletionStage(decoratedSupplier)
            .toCompletableFuture()
            .exceptionally(throwable -> {
                // 最终降级方案
                log.error("所有重试都失败，返回默认数据", throwable);
                return getDefaultGeoData(layerId);
            });
    }

    private GeoDataResponse queryGeoData(String layerId, String bounds) {
        // 实际数据库查询
        return repository.findByLayerIdAndBounds(layerId, bounds);
    }

    private GeoDataResponse getCachedDataOrDefault(String layerId, String bounds) {
        // 首先尝试从Redis获取缓存
        String cacheKey = "geo:" + layerId + ":" + bounds;
        GeoDataResponse cached = (GeoDataResponse) redisTemplate.opsForValue().get(cacheKey);

        if (cached != null) {
            return cached;
        }

        // 返回默认数据
        return getDefaultGeoData(layerId);
    }

    private GeoDataResponse getDefaultGeoData(String layerId) {
        return GeoDataResponse.builder()
            .layerId(layerId)
            .features(Collections.emptyList())
            .message("服务暂时不可用，返回默认数据")
            .build();
    }
}

5. 健康检查增强

/**
 * K8s环境下的健康检查
 */
@Component
public class K8sHealthIndicator implements HealthIndicator {

    @Autowired
    private DataSource dataSource;

    @Autowired
    private RedisTemplate redisTemplate;

    @Value("${spring.application.name}")
    private String applicationName;

    @Override
    public Health health() {
        Health.Builder builder = new Health.Builder();

        try {
            // 检查数据库连接
            checkDatabase(builder);

            // 检查Redis连接
            checkRedis(builder);

            // 检查磁盘空间
            checkDiskSpace(builder);

            // 检查内存使用
            checkMemoryUsage(builder);

            return builder.up()
                .withDetail("application", applicationName)
                .withDetail("timestamp", Instant.now())
                .build();

        } catch (Exception e) {
            return builder.down()
                .withDetail("error", e.getMessage())
                .withDetail("timestamp", Instant.now())
                .build();
        }
    }

    private void checkDatabase(Health.Builder builder) throws SQLException {
        try (Connection connection = dataSource.getConnection()) {
            long startTime = System.currentTimeMillis();
            connection.isValid(5);
            long responseTime = System.currentTimeMillis() - startTime;

            builder.withDetail("database", Map.of(
                "status", "UP",
                "responseTime", responseTime + "ms"
            ));
        }
    }

    private void checkRedis(Health.Builder builder) {
        try {
            long startTime = System.currentTimeMillis();
            redisTemplate.opsForValue().get("health:check");
            long responseTime = System.currentTimeMillis() - startTime;

            builder.withDetail("redis", Map.of(
                "status", "UP",
                "responseTime", responseTime + "ms"
            ));
        } catch (Exception e) {
            builder.withDetail("redis", Map.of(
                "status", "DOWN",
                "error", e.getMessage()
            ));
            throw new RuntimeException("Redis health check failed", e);
        }
    }

    private void checkDiskSpace(Health.Builder builder) {
        File disk = new File("/");
        long totalSpace = disk.getTotalSpace();
        long freeSpace = disk.getFreeSpace();
        double usagePercent = ((double)(totalSpace - freeSpace) / totalSpace) * 100;

        if (usagePercent > 90) {
            throw new RuntimeException("磁盘使用率过高: " + String.format("%.2f%%", usagePercent));
        }

        builder.withDetail("diskSpace", Map.of(
            "total", totalSpace,
            "free", freeSpace,
            "usagePercent", String.format("%.2f%%", usagePercent)
        ));
    }

    private void checkMemoryUsage(Health.Builder builder) {
        Runtime runtime = Runtime.getRuntime();
        long totalMemory = runtime.totalMemory();
        long freeMemory = runtime.freeMemory();
        double usagePercent = ((double)(totalMemory - freeMemory) / totalMemory) * 100;

        if (usagePercent > 85) {
            throw new RuntimeException("内存使用率过高: " + String.format("%.2f%%", usagePercent));
        }

        builder.withDetail("memory", Map.of(
            "total", totalMemory,
            "free", freeMemory,
            "usagePercent", String.format("%.2f%%", usagePercent)
        ));
    }
}

/**
 * 就绪性探针
 */
@RestController
public class K8sHealthController {

    @Autowired
    private K8sHealthIndicator healthIndicator;

    @GetMapping("/actuator/health/liveness")
    public ResponseEntity<Map<String, String>> liveness() {
        // 存活探针：只检查应用是否能响应
        return ResponseEntity.ok(Map.of("status", "UP"));
    }

    @GetMapping("/actuator/health/readiness")
    public ResponseEntity<Map<String, Object>> readiness() {
        try {
            Health health = healthIndicator.health();
            return ResponseEntity.ok(Map.of(
                "status", health.getStatus().getCode(),
                "details", health.getDetails()
            ));
        } catch (Exception e) {
            return ResponseEntity.status(503).body(Map.of(
                "status", "DOWN",
                "error", e.getMessage()
            ));
        }
    }
}

6. ConfigMap和Secret管理

# ConfigMap配置
apiVersion: v1
kind: ConfigMap
metadata:
  name: gis-app-config
data:
  application-k8s.yml: |
    spring:
      datasource:
        hikari:
          maximum-pool-size: 20
          minimum-idle: 5
      redis:
        cluster:
          nodes: redis-cluster:6379
          max-redirects: 3
        timeout: 2000ms
        lettuce:
          pool:
            max-active: 20
            max-idle: 10

    management:
      endpoints:
        web:
          exposure:
            include: health,info,metrics,prometheus
      endpoint:
        health:
          show-details: always
          probes:
            enabled: true

    resilience4j:
      circuitbreaker:
        instances:
          geoDataService:
            sliding-window-size: 10
            failure-rate-threshold: 50
            wait-duration-in-open-state: 30s
            permitted-number-of-calls-in-half-open-state: 5
      retry:
        instances:
          geoDataService:
            max-attempts: 3
            wait-duration: 1s
      bulkhead:
        instances:
          geoDataService:
            max-concurrent-calls: 25
            max-wait-duration: 500ms

---
apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
type: Opaque
data:
  host: cG9zdGdyZXNxbC1zZXJ2aWNl  # base64 encoded
  username: Z2lzdXNlcg==
  password: <base64-encoded-password>

7. Ingress配置

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gis-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/rate-limit-window: "1m"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/upstream-fail-timeout: "5s"
    nginx.ingress.kubernetes.io/upstream-max-fails: "3"
    # 熔断配置
    nginx.ingress.kubernetes.io/server-snippet: |
      location /api/geo/data {
        limit_req zone=api burst=20 nodelay;
        proxy_next_upstream error timeout http_500 http_502 http_503;
      }
spec:
  tls:
  - hosts:
    - gis.example.com
    secretName: gis-tls-secret
  rules:
  - host: gis.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: gis-app-service
            port:
              number: 80

8. 监控和告警

# ServiceMonitor for Prometheus
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gis-app-monitor
spec:
  selector:
    matchLabels:
      app: gis-app
  endpoints:
  - port: http
    path: /actuator/prometheus
    interval: 30s

---
# PrometheusRule for alerting
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gis-app-alerts
spec:
  groups:
  - name: gis-app
    rules:
    - alert: GisAppHighErrorRate
      expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "GIS应用错误率过高"
        description: "错误率超过10%，持续2分钟"

    - alert: GisAppHighLatency
      expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "GIS应用响应时间过长"
        description: "95%请求响应时间超过1秒"

K8s vs 传统部署对比

核心优势总结

1. 自动化运维: HPA、健康检查、自愈能力
2. 服务网格: Istio提供的高级熔断降级能力
3. 云原生监控: Prometheus + Grafana完整监控栈
4. 配置分离: ConfigMap实现配置与代码分离
5. 多层次保护: Ingress + Service Mesh + Application三层保护

K8s环境下的部署能够提供更强大的自动化能力和更精细的控制，特别是在熔断降级方面有服务网格和应用双重保护机制。