sui_multigettransactionBlocks long response time

[EvanTedesco]

Steps to Reproduce Issue

Calling sui_multigettransactionBlocks for each block of the chain sporadically results in an unusually long response time. Sometimes it will run for days without issue but then response time spikes and continues to remain very high.

Expected Result

Blocks should be returned with a predictable response time such that a block scanning process can keep up to date with the chain.

Actual Result

Response time spikes and remains high preventing successive calls from completing and prohibiting the ability of a process to scan each block of the chain.

I see several references to this issue on the SUI discord with no apparent resolution. Any help is greatly appreciated.

[github-actions]

Thank you for opening this issue, a team member will review it shortly. Until then, please do not interact with any users that claim to be from Sui support and do not click on any links!

[stefan-mysten]

Thanks @EvanTedesco for flagging this. We'd need a bit more details to understand and help to root cause this.
What RPC are you using?
What SDK are you using?
Do you know if some of these tx blocks are recent or very old?

[EvanTedesco]

Thanks @stefan-mysten for the prompt response.

I am using https://fullnode.mainnet.sui.io/ rpc.
I am using the javascript SDK.
It is happening when scanning new blocks/transactions.

[stefan-mysten]

Thanks @EvanTedesco! That query is potentially quite heavy for the server. Could you share an example of what data you request from that query?

[EvanTedesco]

Basically I am trying to monitor the chain for transactions to a list of managed addresses. I am calling into the function with the options:

            {
              showBalanceChanges: true,
              showEffects: true,
              showInput: true
            }

[stefan-mysten]

Passed this internally. I don't think we have a good solution yet.

[phoenix-o]

Can you provide a specific full example of a JSON RPC query that is slow on your end (including transaction hashes)? Also, where are you querying it from, i.e. which region?

[EvanTedesco]

@stefan-mysten Thank you for looking into this and for keeping me up to date.

@phoenix-o Sure, here you go:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "sui_multiGetTransactionBlocks",
  "params": [
    [
      "2x6mLe4Z9pUEmsiS751mUpt3PMLGyfJSb8YnVZcqehYW",
      "375K9tevXFCj42zzwFff2Cx25iQihXM2ARcbj776xJJ3",
      "BeksanxiTnnizo2C4XPdoja3U5e9b4dU9LkQaJJhLHeL",
      "5sWdqcrHrZSbDW4mrszqmaU9v2Y3iUH5kqtRu6p3WLMK",
      "2FwMZ1DhhKWSgGxA14bVtm6cYM9Kgfjot9NN1YTtFWsy",
      "Bp8BnjXkcE96hMSWdYWVN3Eza7gnC9LsKchUiJpSHfHe",
      "CKqxJhxhC39okPrUik1Y6rCGdRnqzUSCiSN8JiQQwMup",
      "H1bT6WVKaSz3UKTSCSCw3NPvtsJFpE5gThxxQnT4eN9L",
      "HFaXYirwabwwNrPH1Ai1BfRk96HHQcZVGwCrFH2UXVb4"
    ],
    {
      "showInput": true,
      "showEffects": true,
      "showBalanceChanges": true
    }
  ]
}

[phoenix-o]

this query returned in 0.02s for me.
A couple of suggestions.
The current retention policy for the public FN tier is 2 days. We plan to increase it to 7 days soon, but for now, it remains at 2 days.
This means that if you query recent data within that time window, it will generally have better latency because the data will be served directly from a full node.
For older data, full nodes fall back to the Key Value store, which is only available in the US West region.
To improve latency, you can either deploy your process closer to the KV instance or ensure you're querying the most recent data

[EvanTedesco]

@phoenix-o Thank you for the prompt response and for the suggestions. Just to clarify, the data I am requesting is less than 2 days old when I see the delay. I will look into the geolocation suggestion as well but I am skeptical that is the solution given that the data I am requesting is new, and given the stark difference in performance when the delay presents itself. I will look further into it and see what I find. Thanks again for the assistance.

[EvanTedesco]

@phoenix-o @stefan-mysten After looking further into this it seems the delay is not random. It corresponds to a spike in CPS on the network that seems to happen daily around the same time(beginning at 10:00 AM Eastern standard time)(seen here on the weekly or daily timeframes).