How to Deploy Production Microservices on Kubernetes

## Introduction

Deploying microservices to a production Kubernetes cluster requires far more than running `kubectl apply -f deployment.yaml`. A true enterprise-ready setup demands **high availability**, **automated rolling upgrades**, **zero-downtime health probing**, **declarative Helm packaging**, and **GitOps automation** with ArgoCD.

In this tutorial, you will build and deploy a fault-tolerant microservice stack from scratch. You will configure production-grade resource quotas, implement Horizontal Pod Autoscaling (HPA), provision TLS certificates with cert-manager, and set up continuous synchronization with ArgoCD.

:::info
**Architecture Overview**:
The architecture you are deploying consists of an Ingress-NGINX controller terminating TLS, routing traffic to an API service with PodDisruptionBudgets, backed by an automated ArgoCD GitOps reconciliation engine.
:::

---

## Step 1 — Configuring Production-Ready Pod Manifests

Before packaging with Helm, let's examine the essential anatomy of a production-hardened Pod specification. Notice how we specify non-root security contexts, CPU/Memory requests, and strict liveness and readiness probes.

Create a file named `deployment.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: production
  labels:
    app.kubernetes.io/name: payments-api
    app.kubernetes.io/instance: prod-v1
    app.kubernetes.io/managed-by: helm
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: payments-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: payments-api
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
      containers:
        - name: payments-api
          image: ghcr.io/bitcodematrix/payments-service:v2.4.1
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env:
            - name: NODE_ENV
              value: "production"
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            failureThreshold: 2
```

:::tip
**Key Architectural Insight**:
Setting `maxUnavailable: 0` ensures that during rolling updates, old pods are never terminated before new healthy pods pass their `readinessProbe`. This is the cornerstone of zero-downtime Kubernetes deployments.
:::

---

## Step 2 — Packaging Services with Helm v3

Instead of maintaining hardcoded YAML manifests across development, staging, and production environments, we package our microservice into a flexible Helm chart.

Initialize your Helm chart structure:

```bash
helm create payments-chart
cd payments-chart
```

Define the parameterized `values.yaml` for production overrides:

```yaml
# values.yaml
replicaCount: 3

image:
  repository: ghcr.io/bitcodematrix/payments-service
  pullPolicy: IfNotPresent
  tag: "v2.4.1"

service:
  type: ClusterIP
  port: 8080

ingress:
  enabled: true
  className: "nginx"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rate-limit: "100"
  hosts:
    - host: api.bitcodematrix.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: payments-api-tls
      hosts:
        - api.bitcodematrix.com

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 12
  targetCPUUtilizationPercentage: 75
  targetMemoryUtilizationPercentage: 80
```

Render and validate your Helm templates locally before deployment:

```bash
helm template payments-api . -f values.yaml --namespace production | kubectl apply --dry-run=client -f -
```

---

## Step 3 — Configuring Horizontal Pod Autoscaling (HPA)

To dynamically respond to traffic spikes without manual intervention, configure a Horizontal Pod Autoscaler backed by metrics-server.

Create `hpa.yaml`:

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  minReplicas: 3
  maxReplicas: 15
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
```

:::warning
**Avoid Flapping (Thrashing)**:
Notice the `stabilizationWindowSeconds: 300` on `scaleDown`. This cooldown period prevents rapid scale-up and scale-down oscillation caused by bursty network traffic.
:::

---

## Step 4 — Implementing GitOps Delivery with ArgoCD

GitOps shifts the responsibility of cluster modification from human operators to an automated controller that continuously reconciles cluster state against Git.

Apply the declarative ArgoCD `Application` manifest:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-production
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'https://github.com/bitcodematrix/k8s-gitops-manifests.git'
    targetRevision: main
    path: environments/production/payments
    helm:
      valueFiles:
        - values.yaml
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
```

Apply the application to your ArgoCD control plane:

```bash
kubectl apply -f argocd-application.yaml -n argocd
```

Check sync status using the ArgoCD CLI:

```bash
argocd app get payments-production
```

Output:
```text
Name:               argocd/payments-production
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          production
URL:                https://argocd.bitcodematrix.com/applications/payments-production
Repo:               https://github.com/bitcodematrix/k8s-gitops-manifests.git
Target:             main
Path:               environments/production/payments
Sync Window:        Sync Allowed
Sync Status:        Synced to main (a8f93e1)
Health Status:      Healthy
```

---

## Step 5 — Verification and Load Testing

Let's verify our autoscaling response by generating synthetic HTTP load using `hey` or `k6`:

```bash
# Run 10,000 requests with 50 concurrent workers
hey -n 10000 -c 50 -m GET https://api.bitcodematrix.com/healthz
```

Monitor the pods and HPA scaling in real time:

```bash
kubectl get hpa payments-api-hpa -n production --watch
```

You will observe the replica count smoothly expand from 3 to 9 pods, distributing request volume seamlessly across cluster worker nodes.

---

## Conclusion

You have now established a complete, enterprise-grade deployment workflow for Kubernetes microservices. By combining **Helm modular templates**, **non-root security policies**, **proactive HPA autoscaling**, and **ArgoCD declarative GitOps**, your services are self-healing, auditable, and resilient to sudden traffic surges.

In the next tutorial of this series, we will dive into **eBPF-powered network observability using Cilium and Hubble**.