Memory Leak in Go — A Story

Photo by Daan Mooij on Unsplash

I had an experience that I’d like to share with you, I were in a situation where my Golang project was continuously allocating more memory after some time it had to be restarted, if you are in the same situation these points may help.

Finding out the line of the root cause

Your project may be huge and reading every line of code to find a suspicious line can be so much time-consuming you may find it easier to deploy an instance of you project and monitor the memory allocation.

if the size of heap increases without any request it means you most probably have a goroutine running in the background and there is a problem in there but if everything is ok. it’s a good idea to load test your project. if you have several HTTP routes then for example you can load test each route, monitor and then go to the next one. each route that causes memory leak should be investigated.

How to load test

You can simply run a bash script or use tools like Bombardier. As an example the following script do this:

How to monitor

You can use Prometeus and Grafana for better visualization which needs development but makes it easy to monitor, or you can use pprof.

Case Study

First of all let me mention a point, one open connection or something can’t cause a leak, if you are experiencing a leak it means you are doing something harmful in a long time for-loop or in each request.

This one is one of the hardest points to notice, let me go through an examples.

Reference

look at the above code, I have made a db instance passed it to a function then inside the function a struct has been made using the db instance as a field after the function returns the struct should be removed by garbage collector but Garbage Collector can’t do it, cause this struct has a reference to an instance that shouldn’t be collected.
Note: This case happens when we are working with pointers.
Note: This was just an example and can be generalized.
Note: That’s why global variable is a bad practice

Tickers

Don’t forget to stop tickers:

experience: It’s not common, but I have seen it in a code which caused resource leak:

the above code makes me confused that this would create a leak or not so I wrote the following code to make sure about the leak:

Context

Don’t forget to cancel contexts

Connections (DB, Redis and …)

Every socket which is opened should be closed. Usually, we use libraries for opening connections, and most of the times the library handles different cases, but it’s essential to pay attention, I explain more for the specific case of database connection.

CE @ Amirkabir University of Tech.