SDK does not calculate signature correctly for files with commas

[Jose-Matsuda]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

I am trying to read and consume a file whose path contains a special character (in this case the comma ,). I am aware of this page that indicates that it may require special handling, but it seems that when reading in the key it does not uriencode it before generating the signature.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The call to return successfully and the file contents to be outputted. This call works fine when using the minio client as well as the go code when it is generated by postman.

Current Behavior

It fails and I get

error operation error S3: GetObject, https response error StatusCode: 403, RequestID: 183640373A3A05C5, 
HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8,
api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.
 Check your key and signing method.err not nilpanic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7fe9d5]

Reproduction Steps

I self-hosted a minio server using docker run -p 9000:9000 -p 9001:9001 quay.io/minio/minio server /data --console-address ":9001" created a bucket named test and then underneath created two files, qwerty.txt and bad,file.txt.

I then ran the following code

package main

import (
	"context"
	"fmt"
	"io"
	"log"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	// Load the Shared AWS Configuration (~/.aws/config)
	//[default]
	//aws_access_key_id = placeholder
	//aws_secret_access_key = placeholder
	//region = us-east-1 
	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithRegion("us-east-1"),
		config.WithBaseEndpoint("http://localhost:9000"),
	)
	if err != nil {
		log.Fatal(err)
		fmt.Print("Error above")
	}
	client := s3.NewFromConfig(cfg)

	bucketName := "test/"
        k := "qwerty.txt"
	k = "bad,file.txt"
	out2, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
		Bucket: &bucketName,
		Key:    &k,
	})
	if err != nil {
		fmt.Printf("error %v", err)
	}
	stringBlah, _ := io.ReadAll(out2.Body)
	fmt.Println(string(stringBlah))
}

Where it fails on bad,file.txt but not for qwerty.txt

Possible Solution

Improve key with uri-encoding

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

module test

go 1.22.2

require (
	github.com/aws/aws-sdk-go v1.55.6
	github.com/aws/aws-sdk-go-v2/config v1.29.14
	github.com/aws/aws-sdk-go-v2/service/s3 v1.79.2
)

require (
	github.com/aws/aws-sdk-go-v2 v1.36.3 // indirect
	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.10 // indirect
	github.com/aws/aws-sdk-go-v2/credentials v1.17.67 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.34 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.7.0 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.15 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.25.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.1 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.33.19 // indirect
	github.com/aws/smithy-go v1.22.3 // indirect
)

Compiler and Version used

go version go1.22.2 linux/amd64

Operating System and version

Ubuntu 20.04.2

[Madrigal]

I can't reproduce this.

Downloading an object with a comma with a comma on its name works as expected. The object key gets correctly URL-encoded and the object is retrieved as expected.

Using config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody) as described in the developer guide I get the following request and response

SDK 2025/04/14 16:35:32 DEBUG Request
GET /bad%2Cfile.txt?x-id=GetObject HTTP/1.1
Host: lmadrig-test-bucket.s3.us-west-2.amazonaws.com
User-Agent: aws-sdk-go-v2/1.36.3 ua/2.1 os/macos lang/go#1.24.1 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.79.2 m/E,b,n
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 8cf6ff3e-1a4e-4d3f-a98f-4cfc403cff54
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=ASIA47CR3K457BGV6G5L/20250414/us-west-2/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-checksum-mode;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=672f1864d5f2df529754a230202da778b42d6f13a2bd8fda998ecf954a042dfb
X-Amz-Checksum-Mode: ENABLED
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250414T203532Z
X-Amz-Security-Token: <xxx>

SDK 2025/04/14 16:35:32 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 12
Accept-Ranges: bytes
Content-Type: text/plain
Date: Mon, 14 Apr 2025 20:35:33 GMT
Etag: "6f5902ac237024bdd0c176cb93063dc4"
Last-Modified: Mon, 14 Apr 2025 20:35:04 GMT
Server: AmazonS3
X-Amz-Checksum-Crc64nvme: D3gOhIHSoh8=
X-Amz-Checksum-Type: FULL_OBJECT
X-Amz-Id-2: kKeQtbFxTCwrPO9K2iR5fW6zUFHUOw4NnLIcS8635GslI5Y09py6n3kURp4RF7kMMiyq5P4tYF6F1iaR5Zci5w==
X-Amz-Request-Id: Q4XCE5EPV5NY7DS2
X-Amz-Server-Side-Encryption: AES256

hello world

Notice how the GET request is mapped to GET /bad%2Cfile.txt?x-id=GetObject HTTP/1.1, showing a correct object encoding

[Jose-Matsuda]

Appreciate the quick reply as well as your detailed logs as it helped us understand what was going on! I'll close the ticket as there's nothing to report.

Just for anyone else if they run into this;

When running with minio on localhost the following configuration "worked" after following your example

	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithRegion("us-east-1"),
		config.WithBaseEndpoint("http://:9000"),
		config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody),
	)
	if err != nil {
		log.Fatal(err)
	}
	// Create an Amazon S3 service client
	client := s3.NewFromConfig(cfg)

	bucketName  := "localhost"
	k := "test/bad,file.txt"
        out2, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
		Bucket: &bucketName,
		Key:    &k,
	})

I was thinking that bucketName here was test, but when I run with that I get my initially reported error. I guess with the sdk (and also tested with an aws bucket) it prepends it to the baseEndpoint. Where we correctly get

GET /test/bad%2Cfile.txt?x-id=GetObject HTTP/1.1
Host: localhost.:9000

With our initial config of

bucketName := "test/"
k := "qwerty.txt"

working we were thinking that was the correct way since we could read the file and produces the non-encoded

GET /test/bad,file.txt?x-id=GetObject HTTP/1.1
Host: localhost:9000

On second glance this is obvious but when it half works we assumed it as truth.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.