How to check for high memory in the recursor process

[chunyuliu106667777]

Description：

I am conducting an experiment where I built a Docker image of Ubuntu 22.04 and ran the container of the image. I successfully started a recursive service inside the container. At first, the process only used 9% of the server's memory, but after running for a while, I found through the top command that the process used more than 70% of the server's memory and continued to rise. Eventually, the process was killed by the system due to OOM. How can I investigate where the problem lies. Can you give me some suggestions for checking? Thanks a lot.

Information：

1. Running instructions：/usr/sbin/pdns_recursor --daemon=no --write-pid=no --disable-syslog --log-timestamp=no
2. Configuration (recursor.yml)：

dnssec:
  validation: off
recursor:
  hint_file: no-refresh
  include_dir: /etc/powerdns/recursor.d
  any_to_tcp: true
  lua_config_file: /etc/powerdns/config.lua
  lua_dns_script: /etc/powerdns/script.lua
  lua_maintenance_interval: 30
  threads: 4
  forward_zones_recurse:
    - zone: .
      forwarders:
        - 8.8.8.8
incoming:
  allow_from:
    - 0.0.0.0/0
    - "::/0"
  gettag_needs_edns_options: true
  port: 53
  listen:
    - 127.0.0.1

[rgacogne]

Would you mind telling us the amount of memory the server has?
The recursor caches a fair amount of data to improve performance, you might want to look at these, for example:

• https://docs.powerdns.com/recursor/yamlsettings.html#setting-yaml-recordcache-max-entries
• https://docs.powerdns.com/recursor/yamlsettings.html#setting-yaml-packetcache-max-entries
• https://docs.powerdns.com/recursor/yamlsettings.html#dnssec-aggressive-nsec-cache-size

[chunyuliu106667777]

The server memory is 4G，I haven't set the following three configurations, they should be the default values

[omoerbeek]

The first two numbers are the configures maximum number of entries.
The name of the third metric is aggressive-nsec-cache-size, it shows the current number of entries in the agressive NSEC cache, but as you have dnsec validation disabled this should be zero.

To get the current number of entries for the two main caches:

rec_control get cache-entries
rec_control get packetcache-entries

See https://docs.powerdns.com/recursor/metrics.html#gathered-information for other metrics.

Looking at https://docs.powerdns.com/recursor/performance.html#memory-usage, it feels your memory usage is pretty high,
so make sure your Lua code does not have a memory leak: try running without the Lua hooks to see if that makes a difference.

[chunyuliu106667777]

Okay, I will try the problem you mentioned.

[chunyuliu106667777]

I have found some issues. In my Lua script, I used large global variables that are updated irregularly. When changes occur, I call the rec_comtrol reload-lua-script command. Each time I call this command, the process's memory increases by 0.2% -0.5%. Here is the pseudocode of my Lua script.
local conf_status, CONF = pcall(load_conf)

function preresolve(dq)
.......
end

function postresolve(dq)
.......
end

function maintenance()
conf_status, CONF = pcall(load_conf)
end

When my configuration changes, the increase in size of the configuration itself can be basically ignored. However, when the reload-lua-script command is called, the memory will permanently increase by 0.2% -0.5% and will not decrease. Why does this happen, and how can I avoid it? My service version is 5.2.0.

[omoerbeek]

rec_control reload-lua-script should not cause a memory leak itself (I will double check that soon). I do wonder why you are both reloading the config in the maintenance call and doing rec_control reload-lua-script.. Without further details I cannot tell.

[chunyuliu106667777]

emm... This is indeed my problem. I have removed the maintenance function, but the result is the same.
I made some further attempts afterwards, and when I used the maintenance function instead of the reload-lua-script command and set lua_maintenance_interval, the memory usage also increased.

[omoerbeek]

It strongly suggests the leak is in your Lua code if the memory increases aslo happen when there is a maintainance function and you are not calling rec_control reload-lua-script. Check all you global data and other global resources in your Lua to see if something is growing. Sometimes it is as simple as a missing local...

[omoerbeek]

I did a test with a tight shell loop doing rec_control reload-lua-script and saw no leak.

[chunyuliu106667777]

Okay, My understanding is that if there is a memory leak in the logic of a part of the maintenance function in the Lua script, I should be able to reload the entire Lua script by closing the maintenance function and using the rec_comtrol reload-lua-script command to avoid calling suspicious functions multiple times. But what I have observed now is that the result is the same regardless of which method is used. Can you help explain the logic of the reload-lua-script command, or is there any relevant documentation that can help me understand more.

[omoerbeek]

The reload code replaces the whole Lua context for each worker thread, so I'm not sure what causes your memory growing. It must be some resource that is not cleaned up, but without further details it is impossible to tell. So far my my local testing showed that the reload mechanism itself is working fine and not causing growth.