Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Blob trigger function AKS trigger issue #1613

Open
anime-shed opened this issue Nov 21, 2024 · 1 comment
Open

Azure Blob trigger function AKS trigger issue #1613

anime-shed opened this issue Nov 21, 2024 · 1 comment

Comments

@anime-shed
Copy link

I have a blob-triggered Azure function, which is deployed on AKS with keda scaling based on the blob entries.
I used Azure/azure-functions-host#10624 to make each function accept only one blob item. The problem I have is that all created pods read the same file, but if I use queues-based triggers and scaling, different queue elements are read by different functions. According to my understanding, the blob trigger internally uses queues to do its tasks, so why is the behaviour different from having a blob trigger?

P.S.: I am moving the files to a different folder after the process is completed.

host.json

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  },
  "extensions": {
    "blobs": {
      "maxDegreeOfParallelism": 1
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  }
}
host: 2.0
"FUNCTIONS_WORKER_RUNTIME": "python",
"AzureWebJobsFeatureFlags": "EnableWorkerIndex",
"PYTHON_ISOLATE_WORKER_DEPENDENCIES": "1",
from blob_helper import initialize_blob_service_client,upload_dataframe_to_blob
import logging
app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)
@app.function_name(name="PythonFunction")
@app.blob_trigger(
    arg_name="myblob", 
    path="sheets/input/{name}",  # Blob path for trigger
    connection="DataLakeConnectionString"
)

keda

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: python-fuction-scaler
  namespace: prod
spec:
  scaleTargetRef:
    name: python-fuction 
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
    - type: azure-blob
      metadata:
        blobContainerName: "sheets"
        blobPrefix: "input"
        connectionFromEnv: "DataLakeConnectionString"
        targetBlobCount: "1"
      authenticationRef:
        name: secrets
@anime-shed
Copy link
Author

What I was able to find out was that each instance of the Azure Blob triggered function was creating a new queue and setting a lock in that, so is it possible to have a common queue for all? I think with that my issue should be solved.

@JAdluri JAdluri self-assigned this Mar 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants