Memory Leak in Go — A Story
I had an experience that I’d like to share with you, I were in a situation where my Golang project was continuously allocating more memory after some time it had to be restarted, if you are in the same situation these points may help.
Finding out the line of the root cause
Your project may be huge and reading every line of code to find a suspicious line can be so much time-consuming you may find it easier to deploy an instance of you project and monitor the memory allocation.
if the size of heap increases without any request it means you most probably have a goroutine running in the background and there is a problem in there but if everything is ok. it’s a good idea to load test your project. if you have several HTTP routes then for example you can load test each route, monitor and then go to the next one. each route that causes memory leak should be investigated.
How to load test
You can simply run a bash script or use tools like Bombardier. As an example the following script do this:
How to monitor
You can use Prometeus and Grafana for better visualization which needs development but makes it easy to monitor, or you can use pprof.
Case Study
First of all let me mention a point, one open connection or something can’t cause a leak, if you are experiencing a leak it means you are doing something harmful in a long time for-loop or in each request.
This one is one of the hardest points to notice, let me go through an examples.
Reference
look at the above code, I have made a db instance passed it to a function then inside the function a struct has been made using the db instance as a field after the function returns the struct should be removed by garbage collector but Garbage Collector can’t do it, cause this struct has a reference to an instance that shouldn’t be collected.
Note: This case happens when we are working with pointers.
Note: This was just an example and can be generalized.
Note: That’s why global variable is a bad practice
Tickers
Don’t forget to stop tickers:
experience: It’s not common, but I have seen it in a code which caused resource leak:
the above code makes me confused that this would create a leak or not so I wrote the following code to make sure about the leak:
Context
Don’t forget to cancel contexts
Connections (DB, Redis and …)
Every socket which is opened should be closed. Usually, we use libraries for opening connections, and most of the times the library handles different cases, but it’s essential to pay attention, I explain more for the specific case of database connection.
