Debugging Docker Production Builds

By Adam Tuttle on Jun 29, 2022

Photo by Barrett Ward on Unsplash

Filing this one under things that I'm writing for my future self to reference.

Who among us hasn't had the experience where you're getting a bug report for a bug that you know you fixed, and it seems like your fix is just being ignored in production? (If you haven't had that experience, I envy you!)

This is especially frustrating when your production environment is a Docker container running in the cloud, built by a CI/CD server. You can't exactly SSH into the server and look at the code to make sure your deploy actually worked. I guess you could unblock that, but it seems like the wrong reason to open up SSH ports on your production containers and probably inadvisable.

Anyway, if you find yourself in this situation, here's how you can double-check what's actually inside that production container. My notes are from Amazon's AWS Elastic Container Repository (ECR) but the steps should be the same even if you're using a different service and the syntax varies.

1. Log in to ECR:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com

2. Pull the image from ECR to your local machine:

docker pull <account-id>.dkr.ecr.us-east-1.amazonaws.com/my/repo/name:tagname

3. Start the container locally:

docker run -it --entrypoint /bin/bash <account-id>.dkr.ecr.us-east-1.amazonaws.com/my/repo/name:tagname

This starts the container in interactive mode and rather than launching its usual startup command, runs, in this case, /bin/bash to give you a shell to browse around the container.

Now you can dig around and make sure your code changes are in fact represented in the production build, where and how you expected.

What should you do if the code is what you wanted it to be and the app still doesn't work?

Well, friend... That remains one of life's great mysteries.