Could not stop DNS renewal for target removed from upstream_id: target object does not contain name and port, context: ngx.timer

[sami-ul-islam]

Description:

I am encountering an issue in Kong when managing targets for an upstream. On every deployment restart the following error message we receives in kong logs:

2025/01/28 09:14:39 [error] 1411#0: *1338 stream [lua] targets.lua:186: on_target_event(): could not stop DNS renewal for target removed from cdf8e897-f305-49bd-bfe2-d2649edcec63: target object does not contain name and port, context: ngx.timer

I have the following target data for the upstream cdf8e897-f305-49bd-bfe2-d2649edcec63, retrieved using
http://<kong-admin-url>/upstreams/cdf8e897-f305-49bd-bfe2-d2649edcec63/targets:

{
    "data": [
        {
            "id": "bd0e11f3-a538-4362-b155-dc121ccfac7e",
            "upstream": {
                "id": "cdf8e897-f305-49bd-bfe2-d2649edcec63"
            },
            "target": "10.4.76.252:3000",
            "weight": 100,
            "created_at": 1738055675.551,
            "tags": [
                "managed-by-ingress-controller"
            ],
            "updated_at": 1738055675.551
        },
        {
            "id": "36f891b0-6004-44c3-b608-d715cb069bbd",
            "upstream": {
                "id": "cdf8e897-f305-49bd-bfe2-d2649edcec63"
            },
            "target": "10.4.80.113:3000",
            "weight": 100,
            "created_at": 1738055663.67,
            "tags": [
                "managed-by-ingress-controller"
            ],
            "updated_at": 1738055663.67
        }
    ],
    "next": null
}

Issue:

This is the function https://github.com/Kong/kong/blob/master/kong/runloop/balancer/targets.lua#L109 that is preforming these checks and it is called by https://github.com/Kong/kong/blob/master/kong/runloop/balancer/targets.lua#L186
From the above output i got it should return a string in the format id:ip:port for each target (e.g., 1d51682d-50ec-4768-ae21-0ec6215a9e66:10.4.76.247:3000).

Copied the function from the above link which is as follows:

local function get_dns_renewal_key(target)
  if target and (target.balancer or target.upstream) then
    local id = (target.balancer and target.balancer.upstream_id) or (target.upstream and target.upstream.id)
    if target.target then
      return id .. ":" .. target.target
    elseif target.name and target.port then
      return id .. ":" .. target.name .. ":" .. target.port
    end
  end

  return nil, "target object does not contain name and port"
end

The target object contains the necessary fields (upstream.id and target), so I expect the function to return the correct key (e.g., 1d51682d-50ec-4768-ae21-0ec6215a9e66:10.4.76.247:3000).

However, every time a target is created, I get the following error in the Kong logs:

2025/01/28 09:14:39 [error] 1411#0: *1338 stream [lua] targets.lua:186: on_target_event(): could not stop DNS renewal for target removed from cdf8e897-f305-49bd-bfe2-d2649edcec63: target object does not contain name and port, context: ngx.timer

Expected Behavior:

The get_dns_renewal_key() function should successfully return the DNS renewal key in the format id:ip:port for each target. The error log should not occur when a target is created.

Request:

• Please help identify why this error occurs even when the necessary fields (target and upstream.id) are present in the target data.
• Is there a potential issue with how Kong is handling the DNS renewal for targets in this case?

Additional Information:

• I have checked that the target data is correctly formed with the target field as ip:port (e.g., 10.4.76.247:3000), but the error still persists.
• The issue happens whenever new pods are created and old ones removed from targets
• We are running kong as helm chart (v2.46.0) from https://github.com/Kong/charts/tree/main/charts/kong
• These stop DNS renewal on target event were introduced in fix(balancer) stop DNS renewal on target event #9265 , fix(balancer) stop DNS renewal on target event #8497

Any help or guidance in resolving this issue would be greatly appreciated.

[ProBrian]

@chobits Could you help to have a look at this DNS issue?

[sami-ul-islam]

@chobits can you help me to resolve the above issue? let me know if you need any other info