— Kubernetes Probes

A service suddenly starts returning errors. The first check looks reassuring:

Pod A    Running
Pod B Running
Pod C Running

No crash loops. CPU is normal. Memory is normal.

But the application depends on a database and Kafka, and one of those dependencies is unavailable.

                         Service
/ | \
▼ ▼ ▼
Pod A Pod B Pod C
│ │ │
└──┬──┴──┬──┘
│ │
DB ❌ Kafka

The Pods are running, but they may no longer be able to serve requests.

Where Probes Fit

A probe is a health signal exposed by the application and consumed by Kubernetes.

                    Service
/ | \
▼ ▼ ▼
Pod A Pod B Pod C
│
/health/ready
│
Application
/ \
DB Kafka

The application determines its operational state.

Kubernetes acts on that state.

There are three probe types:

Startup — Application has finished starting

Readiness — Pod can receive traffic

Liveness — Container should be restarted

Readiness controls traffic. Liveness controls restart.

Readiness : Remove a Pod From Traffic

Suppose the application exposes:

GET /health/ready

The endpoint checks conditions required to serve requests:

Database → UP
Kafka → UP

Kubernetes can configure:

readinessProbe:
httpGet:
path: /health/ready
port: 8080
periodSeconds: 5
failureThreshold: 3

Now the database goes down.

/health/ready
↓
Database ❌
↓
HTTP 503

Kubernetes marks the Pod NotReady.

                 Service
/ | \
▼ X ▼
Pod A Pod B Pod C
✅ NotReady ✅

Pod B is still running. It simply stops receiving traffic.

When the database recovers:

503 → 200

Pod B becomes ready again.

No restart was required.

The same idea can apply to Kafka.

If the service cannot successfully process an HTTP request without Kafka, Kafka availability can be part of readiness.

But this is an application decision.

For example, if the service uses an outbox pattern and can accept requests while Kafka is temporarily unavailable, Kafka doesn’t necessarily need to make the HTTP Pod NotReady.

The probe should represent:

Can this application safely handle its workload right now?

Not simply:

Are all my dependencies reachable?

Liveness Is Different

Now consider a different failure.

The database is healthy. Kafka is healthy. But the application itself is stuck.

JVM          → Running
Database → UP
Kafka → UP
Application → Stuck

A liveness endpoint can detect this condition.

Liveness fails
↓
Kubernetes
↓
Container restart

This is fundamentally different from readiness:

Readiness fails → stop sending traffic
Liveness fails  → restart container

If Kafka temporarily goes down, blindly using that failure for liveness could create:

Kafka DOWN
↓
Restart
↓
Kafka still DOWN
↓
Restart
↓
Restart...

The restart doesn’t fix Kafka.

Implementing a Custom Health Endpoint

In a Spring Boot service, the application can expose its own health logic:

@GetMapping("/health/ready")
public ResponseEntity<String> readiness() {

if (!databaseAvailable()) {
return ResponseEntity
.status(HttpStatus.SERVICE_UNAVAILABLE)
.body("Database unavailable");
}
if (!kafkaAvailable()) {
return ResponseEntity
.status(HttpStatus.SERVICE_UNAVAILABLE)
.body("Kafka unavailable");
}
return ResponseEntity.ok("READY");
}

Kubernetes doesn’t need to understand the database or Kafka.

It only needs the application’s answer:

200 → Ready
503 → Not Ready

Start with three replicas:

Pod A    Ready
Pod B Ready
Pod C Ready

Now simulate a database failure so /health/ready returns 503.

kubectl get pods

You should see something like:

NAME     READY
pod-a 1/1
pod-b 0/1
pod-c 1/1

Pod B hasn’t crashed.

It has simply been removed from the traffic path.

Restore the database:

503 → 200

Pod B returns to:

1/1

Now make the liveness endpoint fail.

After the configured failure threshold, Kubernetes restarts the container.

kubectl get pods

and the restart count increases.

The Architecture

The complete relationship is:

                   Load Balancer
│
Service
│
┌─────────┼─────────┐
▼ ▼ ▼
Pod A Pod B Pod C
│
Health Endpoint
│
┌────┴────┐
▼ ▼
DB Kafka

The application exposes its health.

Kubernetes evaluates it.

The Service determines which Pods receive traffic.

The result is simple

Running ≠ Healthy
Ready    → Can receive traffic
NotReady → Removed from traffic
Liveness failure → Restart

A Kubernetes Pod being Running only tells us that the container process is alive.

Probes tell Kubernetes whether that Pod is actually usable.


Running Pod Doesn’t Mean a Healthy Application was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.