Support for multi-resource queries

[jkroepke]

Hi,

Azure Monitor is offers multi-resource queries (preview).

https://azure.microsoft.com/en-us/updates/select-multiple-resources-in-azure-monitor-log-analytics/

My hope is to reduce the amount of calls to prevent rate-limit issues.

On Browser leve following request was used to gain multiple metrics:

POST /subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/providers/microsoft.Insights/metrics?timespan=2023-02-19T15:30:00.000Z/2023-02-20T15:35:00.000Z&interval=PT5M&metricnames=CPU Credits Consumed&aggregation=average&metricNamespace=microsoft.compute%2Fvirtualmachines&top=10&orderby=average desc&autoadjusttimegrain=true&validatedimensions=false&api-version=2021-05-01&region=westus3

filter: "Microsoft.ResourceId eq '/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/resourceGroups/default/providers/Microsoft.Compute/virtualMachines/bastion-win' or Microsoft.ResourceId eq '/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/resourceGroups/default/providers/Microsoft.Compute/virtualMachines/bastion-linux'"

I was trying to replicate it, but I had no success

scrape_configs:
  - job_name: azure-metrics-example
    scrape_interval: 1m
    metrics_path: /probe/metrics/resource
    params:
      name:
        - azure_metric
      template:
        - '{name}_{metric}_{aggregation}_{unit}'
      help:
        - Azure metric {metric} for {aggregation}
      subscription:
        - 6fb867d5-d552-4dbe-9f6a-7a151b6292aa
      target:
        - /subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a
      metricNamespace:
        - microsoft.compute/virtualmachines
      metric:
        - Available Memory Bytes
      interval:
        - PT1H
      timespan:
        - PT1H
      aggregation:
        - average
        - total
        - count
      metricFilter:
        - Microsoft.ResourceId eq '/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/resourceGroups/default/providers/Microsoft.Compute/virtualMachines/bastion-win' or Microsoft.ResourceId eq '/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/resourceGroups/default/providers/Microsoft.Compute/virtualMachines/bastion-linux'
      metricTop:
        - 10
    static_configs:
      - targets:
          - url-to-your-azure-metrics-exporter-instance

WARN[0089]/home/runner/work/azure-metrics-exporter/azure-metrics-exporter/metrics/prober.go:120 metrics.(*MetricProber).AddTarget unable to parse resource id: parsing failed for /subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a. Invalid resource Id format paramAggregation="[average,total,count]" paramHelp="[Azure metric {metric} for {aggregation}]" paramInterval="[PT1H]" paramMetric="[Available Memory Bytes]" paramMetricFilter="[Microsoft.ResourceId eq '/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/resourceGroups/default/providers/Microsoft.Compute/virtualMachines/bastion-win' or Microsoft.ResourceId eq '/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/resourceGroups/default/providers/Microsoft.Compute/virtualMachines/bastion-linux']" paramMetricNamespace="[microsoft.compute/virtualmachines]" paramMetricTop="[10]" paramName="[azure_metric]" paramSubscription="[6fb867d5-d552-4dbe-9f6a-7a151b6292aa]" paramTarget="[/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a]" paramTemplate="[{name}{metric}{aggregation}_{unit}]" paramTimespan="[PT1H]" requestPath=/probe/metrics/resource

Aside from that, there where no support for service discovery

See. also: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-dynamic-scope

Edit: It seems like its not availible in SDk, yet Azure/azure-sdk-for-go#20039

[mblaschke]

Do you have any information when this will be GA? Would be good if they include it in azure-sdk-for-go 👍

[jkroepke]

@mblaschke it is GA. The API is availible as stable in 2021-05-01.

https://github.com/Azure/azure-rest-api-specs/blob/main/specification/monitor/resource-manager/Microsoft.Insights/stable/2021-05-01/metrics_API.json#L40

Azure/azure-sdk-for-go#20039 is open already, because I would like to know why a stable API is not present in the SDK.

The Browser Portal is already using this API...

[mblaschke]

That would also speed up metric collection, so it would make sense to implement it and not sure if I want to wait until the sdk implements it.

[jkroepke]

I found it here: https://github.com/Azure/azure-sdk-for-go/blob/v68.0.0/services/preview/monitor/mgmt/2021-05-01-preview/metrics/metrics.go#L166-L195

but it seems like thats the "older" SDK.

---

I also tried this by using existing APIs, but I'm not able to set the region parameter:

package main

import (
	"context"
	"github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
	"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
	"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor"
	"log"
)

func main() {
	cred, err := azidentity.NewDefaultAzureCredential(nil)
	if err != nil {
		log.Fatalf("failed to obtain a credential: %v", err)
	}
	ctx := context.Background()
	client, err := armmonitor.NewMetricsClient(cred, &arm.ClientOptions{ClientOptions: policy.ClientOptions{
		APIVersion: "2021-05-01",
	}})

	res, err := client.List(ctx, "subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a", &armmonitor.MetricsClientListOptions{
		Timespan:        to.Ptr("PT1M"),
		Interval:        to.Ptr("PT1M"),
		Metricnames:     to.Ptr("Available Memory Bytes"),
		Aggregation:     to.Ptr("average"),
		Top:             to.Ptr(int32(10)),
		Orderby:         nil,
		Filter:          to.Ptr("Microsoft.ResourceId eq '/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/resourceGroups/default/providers/Microsoft.Compute/virtualMachines/bastion-win' or Microsoft.ResourceId eq '/subscriptions/e1608e24-0728-4efd-ba5b-a05693b53c5a/resourceGroups/default/providers/Microsoft.Compute/virtualMachines/bastion-linux'"),
		ResultType:      nil,
		Metricnamespace: to.Ptr("microsoft.compute/virtualmachines"),
	})
	if err != nil {
		log.Fatalf("failed to finish the request: %v", err)
	}
	// TODO: use response item
	_ = res
}

[mblaschke]

I'm currently waiting for the azidentity to go final for workload identity, so maybe wait until we get feedback from the azure sdk team.

[jkroepke]

@mblaschke is upcoming azure-sdk/azure-sdk-for-go@8894250

[jkroepke]

The armmonitor v0.9.0 package will include the calls for the APIs

Azure/azure-sdk-for-go#20039 (comment)
Azure/azure-sdk-for-go#20304

The armmonitor also depends against azcore 1.4.0

[mblaschke]

seems that something in azure-sdk-for-go is broken:
Azure/azure-sdk-for-go#20501

I'm unable to pass multiple metricnames in armmonitor v0.9.0

[kgeckhart]

👋 @mblaschke I see this issue was closed by implementing a means to list all metrics for a subscription. I would like to leverage multi-resource queries in other code paths as well. Happy to open a new issue to track it if you want to separate it and I'm happy to help with implementation.

[mblaschke]

@kgeckhart
what do you actually need/expect?

[kgeckhart]

@mblaschke thanks for the quick response!

If I am reading the code for the new endpoint correctly it's going to pull the data for all resources in the subscription based on the query params. The thought I had was to re-factor the existing code paths which currently execute per target to instead batch targets and query for their metrics using an explicit ResourceId filter instead of the current Microsoft.ResourceId eq '*' filter used in the new endpoint. I think this would allow someone to keep the current targeting behavior while reducing the total amount of azure calls.

[mblaschke]

Because of the different api i was choosing another endpoint because it works in a different style.
If you query the subscription based metrics you have to use Microsoft.ResourceId eq '*' as filter or you will get one metric per subscription and not per resource.

So for example you will get the sum of requests of AppGateways in the subscription and not per AppGateway.

That also means you can only split metrics by two additional filters where resource based metrics can have three filters because the query is sent to the resource instead of the subscription.

[kgeckhart]

Thanks for the explanation @mblaschke! I always forget about the awkward behavior of the aws metrics APIs where the filters completely change how the data is grouped. I was hoping to reduce the number of API calls required when targeting specific resources and I'll need to think a bit more about how/if that could be done in the available APIs.