Intro
Debugging Docker containers during development can be a real pain. The good news is that there are few great tools and techniques out there to help you get the info you need to get your Dockerized app properly running.
TLDR;
There are some really great tools and techniques out there:
- Sysdig
- Log aggregators
- Docker logs
- Docker Exec
Methods
One of the first ways you can start debugging a docker app is by simply running a container. Take for example, the redis container, you can simply run:
docker run redis
and Docker will start running the container and you will see the stdout for that container. This would be applicable for containers that have a running process defined by the ENTRYPOINT directive in the Dockerfile.
But let's say you are running the same container as a daemon (-d):
docker run --name redis_test -d redis
Now you cannot see the stdout, so now what? One quick way to see the stdout is to run:
docker logs redis_test
but this will only print out the container output up until you run that command. So you will not see what happens in the future unless you rerun the command.
If you want real time logging, Docker offers a great flag for logs, which will "attach" to the container's output (not to the container directly as described later) and display in real time. The -f flag, or follow ,lets you follow the logs:
docker logs -f redis_test
It should be noted that this technique is for containers that have a running process, as defined by the ENTRYPOINT for example. You cannot follow the logs for a container that was spun up, then ran a process then exited.
Another problem is that docker logs can be super lengthy for human reading, so one way to get around this is to output to a file then you can search that file with a tool like grep:
docker logs redis_test &> logs.txt
Should you want to debug you container by looking up a config files or maybe checking out things like /var/log or /etc/hosts, you will need a way to be able explore the container. My go to method is:
docker exec -it redis_test /bin/bash
This will actually attach a process to the container and run bash. So now you can proceed as if actually inside the container on a real terminal.
Another cool thing you can do is run a command inside the container. Let say you have a postgres container mounted to a Docker volume and you want to run a quick query on the database, but don't have any connectors, like say Datagrip or maybe a Python library. You can actually run a query using some clever hackery:
docker exec -it postgres-test /bin/bash -c "su postgres -c \"psql -d somedbname -c 'select * from users'\""
In the above example, the container name is test-postgres, we attach to the container with exec,then with the -c we log in as user postgres on the container, then use the -c bash option again to execute the psql command, and then use psql's -c option to run a query. It's like command-ception!!! The result however is a nice print out to stdout, of the query you just ran.
This example of exec and -c can be used in may other places too, where you would want to "automate" exec-ing into a container, running a command and then logging out.
Tools
There are some really great tools out there, and I won't really get into them too much since there are great tutorials out there and I don't want to repeat any work already done. So I will reference one of them for you, that is my favorite.
Sysdig
One word: amazing!
This app lets you literally dig into your host machine AND your containers with some keyboard shortcuts or mouse clicks! WOW!! Here is a link to get as taste of the power that is Sysdig, and the best part is they have a community edition: Sysdig video.
Conclusion
"With great power comes great debug-ability", said a wise uncle. Docker containers are powerful and if we don't get a hold of them during development it will make life a pain. One of the great traits of a good software engineering is to be able to understand how to debug their application, whether it's a tool, or some code.
When I started with Docker I was very frustrated with it, as I did not understand how Docker really worked. Once I discovered some great ways to exploit Docker's inner workings, like with Docker Exec, Docker became just another tool to help me build great software.
These are just a few a ways one can explore their container for debugging. Let me know if you have any cool tips or trick you'd like to share!