Memory Profiler
Command-line reference information for the memory-profiler program.
Last updated
Command-line reference information for the memory-profiler program.
Last updated
Memory profiling enables you to understand the memory allocation and behavior of your blockchain applications over time in Substrate-based clients. It identifies method calls in the context of how memory was allocated, combining this information with the number of allocated objects. In addition, profiling can be used to analyze memory leaks, identify where memory consumption is happening, define temporary allocations, and investigate excessive memory fragmentation within applications.
The profiler we recommend is .
You can download a precompiled binary release of the profiler from . The last version we've tested is 0.6.1, but any newer one will also most likely work.
Here's how you can download and unpack it from the command-line:
This will result in three files being unpacked. We're only interested in two of them:
libmemory_profiler.so - this is the memory profiler itself that we will hook into Substrate
memory-profiler-cli - this is the program we will later use to analyze the profiling data
You can also yourself.
First you need to make sure to have the following installed:
GCC toolchain
Rust nightly (we've tested version nightly-2021-06-08)
package manager (for building the GUI)
Then you should be able to build the profiler like this:
You'll find the binaries we need in target/release/libmemory_profiler.so and target/release/memory-profiler-cli.
If you're manually launching Substrate from the command-line, then hooking the profiler up to it boils down to just setting up a few extra environment variables and then launching it normally as you'd usually do.
First, we want to enable logging, tell the profiler where it's supposed to output its logs, where it should gather its profiling data, and optionally tell it to cull temporary allocations:
Then we can launch Substrate with the profiler attached:
Setting the LD_PRELOAD environment variable will instruct Linux's dynamic linker to inject the memory profiler into Substrate just before it's launched, which allows the profiler to hook into the system's memory allocation routines and track every memory allocation that Substrate's doing.
If you're running a Substrate-based node remotely you're probably using systemd to manage it. Here's how you could go about setting up profiling in such a situation.
We assume you've already either downloaded a precompiled binary of the profiler or compiled it from source, and you have it in your current directory.
First, we want to copy the memory profiler to a globally accessible location and set up a place where it can write its logs and gather the profiling data.
Then we want to set up a file with all of the environment variables to configure the profiler itself:
Now you want to open your systemd unit file for your node and add the following in the [Service] section:
Do not add another [Service] section if one already exists; just add the EnvironmentFile key to it. If you already have one EnvironmentFile key do not replace it; just add a second one, systemd will apply both.
Now you can reload your systemd daemon:
And then restart your service to start the profiling:
The profiling data will be gathered at /opt/memory-profiler/logs. If you want to disable the memory profiler just delete the EnvironmentFile key you've added to your unit file, and restart the service again.
One configuration knob that warrants extra consideration is MEMORY_PROFILER_CULL_TEMPORARY_ALLOCATIONS, which controls whenever the profiler will gather short lived allocations.
By default the profiler will gather every allocation that's made by the profiled application. That is a lot of data, and can be on the order of megabytes per second. This is great if you want to only profile for a short period of time, or if you specifically care about diagnosing temporary allocations, but it becomes problematic when you want to leave the profiler running for longer.
This is where the MEMORY_PROFILER_CULL_TEMPORARY_ALLOCATIONS option comes in. When you turn it on by setting it to 1 the profiler will omit all of the really short lived allocations and not write them out to disk. This significantly cuts down the amount of data that's generated, usually to the range of kilobytes per second, which makes it possible to leave the profiling running for days at a time.
Now that you've gathered the profiling data you can now analyze it.
Assuming you have both the memory-profiler-cli and the .dat file you've gathered in the same directory you can load the GUI for it:
This might take a while, depending or your exact hardware and on the amount of data you're trying to load. Eventually you should see something like this being printed out:
Now you can open your web browser and access the GUI at http://localhost:8080/.
It's a good idea to always check the logs generated by the profiler and see whenever there are any WRN or ERR logs present.
You might see the the following error or warning in the profiler's logs depending on which Linux distribution you're running:
This is generally harmless; at most this should only result in higher CPU usage when profiling. You can avoid it by doing something like this:
Although please note that this might have some security implications. Take a look at man perf_event_open for more details.
During analysis the whole data file has to be loaded into memory. If you don't have enough RAM and you'll try to load up a big file the analyzer might run out of memory and crash.
There are also , although besides the ones we've already shown changing them shouldn't be necessary in normal circumstances.
There's also that you can access you'd like to export the data into another format or inspect it programmatically.
