What We Learned Running Docker in Production for Five Years

We started using Docker in production around 2018 — initially for one client's project, then gradually for most of our new deployments. Five years later, we use containers for nearly everything we build. It has been worth it, but not without some hard lessons along the way.

This is not a Docker tutorial. It is a collection of things we learned the hard way that we wish someone had told us before we started.

Your images will get huge if you are not careful

The first time we built a Docker image for a Node.js application, it was 1.2 GB. The application itself was maybe 50 MB of code and dependencies. The rest was the full Ubuntu base image plus every build tool we had installed during the process.

Use multi-stage builds. Your build stage can have all the compilers and dev dependencies it needs, but the final production image should only contain the runtime, your built application, and the minimal dependencies needed to run it. Switching to Alpine-based images and multi-stage builds brought that 1.2 GB image down to about 180 MB.

Image size matters because it directly affects deployment speed. When you push an update and the orchestrator pulls the new image on each server, a 180 MB image deploys in seconds while a 1.2 GB image takes minutes. In an emergency, those minutes matter.

Log management requires planning

Containers write logs to stdout and stderr. That is fine for development where you can just run docker logs. In production, with multiple containers running across multiple hosts, you need a centralized logging solution from day one.

We use a combination of the Docker logging driver to ship logs to a centralized service. The specific tool matters less than having one at all. What you absolutely cannot do is rely on docker logs in production — when a container is restarted or replaced, the logs from the old container are gone.

Also set a log rotation policy on the Docker daemon itself. By default, Docker keeps all container logs indefinitely. We had a server run out of disk space because a verbose application had written 40 GB of logs to the default JSON file driver.

Health checks are not optional

A container can be "running" (the process has not crashed) while the application inside it is completely broken — stuck in an infinite loop, waiting on a dead database connection, or serving 500 errors to every request. Without a health check, your orchestrator thinks everything is fine.

Define a HEALTHCHECK in every Dockerfile that actually tests the application's functionality. For a web service, this should be an HTTP request to an endpoint that exercises a meaningful code path — not just a "ping" that returns 200 if the process is alive. Hit an endpoint that queries the database and returns data. If that fails, the container is not healthy and should be replaced.

Persistent data needs real thought

Containers are ephemeral by design — they can be destroyed and recreated at any time. This is great for stateless application code but terrible for anything that stores data on the filesystem. We learned this lesson the painful way when a container restart wiped uploaded files that a client's application was storing inside the container.

Use Docker volumes for any data that needs to persist — uploaded files, database data, log files that need to outlive the container. Better yet, use an external service for persistent data: S3 for file uploads, a managed database service for database data, and a centralized logging service for logs.

The rule is simple: everything inside a container is temporary. If losing it would cause a problem, it should not be inside the container.

Keep your base images updated

It is easy to write a Dockerfile, get the application working, and then never touch the base image again. A year later, your container is running on a Linux distribution with known security vulnerabilities because the base image has not been updated.

Set up a process — even if it is just a monthly calendar reminder — to rebuild your images with the latest base image. Pin your base image to a specific minor version so you get security patches but not breaking changes. And run a vulnerability scanner against your images periodically.

Start simple with orchestration

When we started with Docker, Kubernetes was already the default recommendation for running containers in production. We tried it for a client project with three containers and spent more time managing the Kubernetes cluster than building the actual application.

For most of our clients — small to mid-size applications with moderate traffic — Docker Compose on a single server or a small cluster is perfectly sufficient. It is simple to understand, easy to debug, and does not require a dedicated infrastructure team to manage. We moved to Docker Compose for everything under a certain scale threshold and have not regretted it.

Kubernetes is the right tool when you need it. But for an application with three services and a database, it is almost always overkill.

Was it worth it?

Without question. Despite the learning curve and the operational lessons, Docker has made our deployment process dramatically more reliable. The ability to build once and deploy anywhere, to reproduce production issues locally, and to roll back a deployment by pointing to the previous image tag — these benefits are real and significant.

The key is to go in with realistic expectations. Docker is not magic. It solves some problems elegantly but introduces others that need to be addressed deliberately. The lessons above are not reasons to avoid Docker — they are the things you need to know to use it well.

If you are containerizing an application and want a second opinion on your setup, we are happy to take a look.