Skip to content

MaxResponseBodySize and the correct way to retrieve a sample preview of the HTTP response #1994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
slicingmelon opened this issue Apr 9, 2025 · 1 comment
Open

MaxResponseBodySize and the correct way to retrieve a sample preview of the HTTP response #1994

slicingmelon opened this issue Apr 9, 2025 · 1 comment

Comments

@slicingmelon
Copy link

slicingmelon commented Apr 9, 2025 •
edited
Loading

Hello,

For the past few months, I've been working on a fasthttp based HTTP client. I struggled to find a way to read only a maximum number of bytes from the HTTP response. All testing was performed against live servers.

Obviously, MaxResponseBodysize is the important factor here, so I began testing various implementations.

fasthttp/client.go

Line 278 in 4c71125

MaxResponseBodySize int

Other notable configurations:

fasthttp/client.go

Line 255 in 4c71125

ReadBufferSize int

fasthttp/client.go

Line 260 in 4c71125

WriteBufferSize int

Below, I am providing sample code snippets that I tried until I got a working PoC.
In all examples, I am setting MaxResponseBodySize to 12288 bytes as I figured out that this includes response headers too, and for example, sites like github.com use a long list of headers.

Approach 1

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

StreamResponseBody:  false // This is the default setting in fasthttp

Response read via resp.Body()

fasthttp/http.go

Line 411 in 4c71125

func (resp *Response) Body() []byte {

Sample code snippet:

package main

import (
	"crypto/tls"
	"fmt"
	"log"

	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
	"github.com/valyala/fasthttp"
)

var (
	strSpace      = []byte(" ")
	strCRLF       = []byte("\r\n")
	strColonSpace = []byte(": ")
)

var headerBufPool bytesutil.ByteBufferPool

type RawHTTPResponseDetails struct {
	StatusCode      int
	ResponsePreview []byte // I want a sample preview bytes from the response body
	ResponseHeaders []byte
	ContentType     []byte
	ContentLength   int64
	ServerInfo      []byte
	ResponseBytes   int // Total bytes of the response received (Headers + Body up to maxBodySize)
}

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

func main() {
	url := "https://www.sample-videos.com/video321/mp4/360/big_buck_bunny_360p_30mb.mp4" // 30 mb video

	tlsConfig := &tls.Config{
		InsecureSkipVerify: true,
	}

	client := &fasthttp.Client{
		MaxResponseBodySize: maxBodySize,  // Set the maximum response body size
		ReadBufferSize:      rwBufferSize, // Set read buffer size
		WriteBufferSize:     rwBufferSize, // Set write buffer size
		StreamResponseBody:  false,        // By default, fasthttp uses StreamResponseBody = false
		TLSConfig:           tlsConfig,
	}

	req := fasthttp.AcquireRequest()
	defer fasthttp.ReleaseRequest(req)
	req.SetRequestURI(url)
	req.Header.SetMethod(fasthttp.MethodGet)

	resp := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(resp)

	err := client.Do(req, resp)

	// Check for errors but ALLOW ErrBodyTooLarge!
	if err != nil && err != fasthttp.ErrBodyTooLarge {
		log.Fatalf("Error fetching URL %s: %v", url, err)
	}

	respDetails := RawHTTPResponseDetails{}

	respDetails.StatusCode = resp.StatusCode()

	respDetails.ResponseHeaders = GetResponseHeaders(&resp.Header, respDetails.StatusCode, respDetails.ResponseHeaders)

	body := resp.Body()
	bodyLen := len(body)

	// Determine the preview size
	previewSize := RespBodyPreviewSize
	if bodyLen < previewSize {
		previewSize = bodyLen
	}

	// Create and populate the ResponsePreview
	preview := make([]byte, previewSize)
	copy(preview, body[:previewSize])
	respDetails.ResponsePreview = preview

	// Populate other details
	respDetails.ContentType = resp.Header.ContentType()
	respDetails.ContentLength = int64(resp.Header.ContentLength())
	respDetails.ServerInfo = resp.Header.Server()

	// Calculate total received bytes (header size + received body size)
	respDetails.ResponseBytes = resp.Header.Len() + bodyLen

	fmt.Println("--- Collected Response Details ---")
	fmt.Printf("Status Code: %d\n", respDetails.StatusCode)

	fmt.Printf("\nResponse Headers (%d bytes):\n", len(respDetails.ResponseHeaders))
	fmt.Println(string(respDetails.ResponseHeaders)) // Print headers including status line

	fmt.Printf("Content-Type: %s\n", string(respDetails.ContentType))
	fmt.Printf("Content-Length Header: %d\n", respDetails.ContentLength)
	fmt.Printf("Server: %s\n", string(respDetails.ServerInfo))
	fmt.Printf("Total Bytes Received (Headers + Body Preview/Truncated Body): %d\n", respDetails.ResponseBytes)

	fmt.Printf("\nResponse Body Preview (%d bytes):\n", len(respDetails.ResponsePreview))
	fmt.Println(string(respDetails.ResponsePreview))

	if err == fasthttp.ErrBodyTooLarge {
		fmt.Println("\nNote: Response body exceeded MaxResponseBodySize.")
		fmt.Printf("Full response body is likely larger than the received %d bytes (maxBodySize = %d).\n", bodyLen, maxBodySize)
	} else if int64(bodyLen) > RespBodyPreviewSize && respDetails.ContentLength > int64(RespBodyPreviewSize) {
		fmt.Printf("\nNote: Full response body (%d bytes reported by Content-Length) is larger than the preview size (%d bytes).\n", respDetails.ContentLength, RespBodyPreviewSize)
	} else if int64(bodyLen) > RespBodyPreviewSize {
		fmt.Printf("\nNote: Response body received (%d bytes) is larger than the preview size (%d bytes).\n", bodyLen, RespBodyPreviewSize)
	}

}

// GetResponseHeaders helper
func GetResponseHeaders(h *fasthttp.ResponseHeader, statusCode int, dest []byte) []byte {
	headerBuf := headerBufPool.Get()
	defer headerBufPool.Put(headerBuf)

	headerBuf.Write(h.Protocol())
	headerBuf.Write(strSpace)
	headerBuf.B = fasthttp.AppendUint(headerBuf.B, statusCode)
	headerBuf.Write(strSpace)
	headerBuf.Write(h.StatusMessage())
	headerBuf.Write(strCRLF)

	h.VisitAll(func(key, value []byte) {
		headerBuf.Write(key)
		headerBuf.Write(strColonSpace)
		headerBuf.Write(value)
		headerBuf.Write(strCRLF)
	})

	headerBuf.Write(strCRLF)

	return append(dest[:0], headerBuf.B...)
}

Output:

go run .\main.go
--- Collected Response Details ---
Status Code: 200

Response Headers (268 bytes):
HTTP/1.1 200 OK
Content-Length: 31466742
Content-Type: video/mp4
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Date: Wed, 09 Apr 2025 09:00:29 GMT
Last-Modified: Tue, 20 Oct 2020 10:45:05 GMT
Etag: "1e024f6-5b217ec8840b6"
Accept-Ranges: bytes


Content-Type: video/mp4
Content-Length Header: 31466742
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Total Bytes Received (Headers + Body Preview/Truncated Body): 7

Response Body Preview (0 bytes):


Note: Response body exceeded MaxResponseBodySize.
Full response body is likely larger than the received 0 bytes (maxBodySize = 12288).

This approach failed, and it also failed on many other servers that didn't include a Content-Lenght header in the response.

Approach 2

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

StreamResponseBody:  true // Enabled StreamResponseBody

Response is read via resp.BodyStream()
https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http.go#L333C1-L335C2

Based on these testcases, I tried to use resp.ReadLimitBody() and ensured resp.CloseBodyStream()

fasthttp/http_test.go

Line 2962 in 4c71125

err := resp.ReadLimitBody(bufio.NewReader(bytes.NewBufferString(simpleResp)), 8)

fasthttp/http.go

Line 337 in 4c71125

func (resp *Response) CloseBodyStream() error {

Looking back through my git history, other sample helper I tried for this approach:

// ReadLimitedResponseBodyStream reads limited bytes from a response body stream
// Appends the result to dest slice
func ReadLimitedResponseBodyStream(stream io.Reader, previewSize int, dest []byte) []byte {
	// Create a buffer for preview
	previewBuf := bytes.NewBuffer(make([]byte, 0, previewSize))

	// Read limited amount of data
	if _, err := io.CopyN(previewBuf, stream, int64(previewSize)); err != nil && err != io.EOF {
		return append(dest[:0], strErrorReadingPreview...)
	}

	return append(dest[:0], previewBuf.Bytes()...)
}

While this approach worked better and allowed me to successfully obtain a response preview, it still failed in most test cases, though not all. It was noted that it failed in concurrent requests. Sample errors:

Error: error when reading response headers: cannot find whitespace in the first line of response ";\xbb\xaa\x9a\x15\xb2\b\xb8\xa8\x00\xe7\xa1\xda=Ӻ\xac=\x03\xa6\x87\xf5Xz\xad:\xbb\xc2\xd8g\xc28\x8d&F`\xa6\x12\xd0\xeb\xa9\r\xa7\x97\x98Ϋ\xddxg\xd9HȢж\x9a\x8b\x89\b\xb4+\x90\x14\f\xf7h\xc3\xd0\xdcF\\\xef[\x0e\xf8C\x0f\xf9\x1c\xd6\x17\xf9Y\x14\xf2\xfb\x17X\x04\xc8\xe2\xb6J\xb8>\\\x85\x0e2\xa4f\xa1\xc6GE\xb5\xd9mvT\xb9\xd6[]\x96ړ\xa04\x9b\r\x1a\

Approach 3 - Almost working!

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

StreamResponseBody:  true // Enabled StreamResponseBody

Response read via resp.Body()

fasthttp/http.go

Line 411 in 4c71125

func (resp *Response) Body() []byte {

Code snippet:

package main

import (
	"crypto/tls"
	"fmt"
	"log"

	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
	"github.com/valyala/fasthttp"
)

var (
	strSpace      = []byte(" ")
	strCRLF       = []byte("\r\n")
	strColonSpace = []byte(": ")
)

var headerBufPool bytesutil.ByteBufferPool

type RawHTTPResponseDetails struct {
	StatusCode      int
	ResponsePreview []byte // I want a sample previw bytes from the response body
	ResponseHeaders []byte
	ContentType     []byte
	ContentLength   int64
	ServerInfo      []byte
	ResponseBytes   int // Total bytes of the response received (Headers + Body up to maxBodySize)
}

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample previw of 1024 bytes from the response body
)

func main() {
	url := "https://www.sample-videos.com/video321/mp4/360/big_buck_bunny_360p_30mb.mp4" // 30 mb video

	tlsConfig := &tls.Config{
		InsecureSkipVerify: true,
	}

	client := &fasthttp.Client{
		MaxResponseBodySize: maxBodySize,  // Set the maximum response body size
		ReadBufferSize:      rwBufferSize, // Set read buffer size
		WriteBufferSize:     rwBufferSize, // Set write buffer size
		StreamResponseBody:  true,         // By default, fasthttp uses StreamResponseBody = false
		TLSConfig:           tlsConfig,
	}

	req := fasthttp.AcquireRequest()
	defer fasthttp.ReleaseRequest(req)
	req.SetRequestURI(url)
	req.Header.SetMethod(fasthttp.MethodGet)

	resp := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(resp)

	err := client.Do(req, resp)

	// Check for errors but ALLOW ErrBodyTooLarge!
	if err != nil && err != fasthttp.ErrBodyTooLarge {
		log.Fatalf("Error fetching URL %s: %v", url, err)
	}

	respDetails := RawHTTPResponseDetails{}

	respDetails.StatusCode = resp.StatusCode()

	respDetails.ResponseHeaders = GetResponseHeaders(&resp.Header, respDetails.StatusCode, respDetails.ResponseHeaders)

	body := resp.Body()
	bodyLen := len(body)

	// Determine the preview size
	previewSize := RespBodyPreviewSize
	if bodyLen < previewSize {
		previewSize = bodyLen
	}

	// Create and populate the ResponsePreview
	preview := make([]byte, previewSize)
	copy(preview, body[:previewSize])
	respDetails.ResponsePreview = preview

	// Populate other details
	respDetails.ContentType = resp.Header.ContentType()
	respDetails.ContentLength = int64(resp.Header.ContentLength())
	respDetails.ServerInfo = resp.Header.Server()

	// Calculate total received bytes (header size + received body size)
	respDetails.ResponseBytes = resp.Header.Len() + bodyLen

	fmt.Println("--- Collected Response Details ---")
	fmt.Printf("Status Code: %d\n", respDetails.StatusCode)

	fmt.Printf("\nResponse Headers (%d bytes):\n", len(respDetails.ResponseHeaders))
	fmt.Println(string(respDetails.ResponseHeaders)) // Print headers including status line

	fmt.Printf("Content-Type: %s\n", string(respDetails.ContentType))
	fmt.Printf("Content-Length Header: %d\n", respDetails.ContentLength)
	fmt.Printf("Server: %s\n", string(respDetails.ServerInfo))
	fmt.Printf("Total Bytes Received (Headers + Body Preview/Truncated Body): %d\n", respDetails.ResponseBytes)

	fmt.Printf("\nResponse Body Preview (%d bytes):\n", len(respDetails.ResponsePreview))
	fmt.Println(string(respDetails.ResponsePreview))

	if err == fasthttp.ErrBodyTooLarge {
		fmt.Println("\nNote: Response body exceeded MaxResponseBodySize.")
		fmt.Printf("Full response body is likely larger than the received %d bytes (maxBodySize = %d).\n", bodyLen, maxBodySize)
	} else if int64(bodyLen) > RespBodyPreviewSize && respDetails.ContentLength > int64(RespBodyPreviewSize) {
		fmt.Printf("\nNote: Full response body (%d bytes reported by Content-Length) is larger than the preview size (%d bytes).\n", respDetails.ContentLength, RespBodyPreviewSize)
	} else if int64(bodyLen) > RespBodyPreviewSize {
		fmt.Printf("\nNote: Response body received (%d bytes) is larger than the preview size (%d bytes).\n", bodyLen, RespBodyPreviewSize)
	}

}

// GetResponseHeaders helper
func GetResponseHeaders(h *fasthttp.ResponseHeader, statusCode int, dest []byte) []byte {
	headerBuf := headerBufPool.Get()
	defer headerBufPool.Put(headerBuf)

	headerBuf.Write(h.Protocol())
	headerBuf.Write(strSpace)
	headerBuf.B = fasthttp.AppendUint(headerBuf.B, statusCode)
	headerBuf.Write(strSpace)
	headerBuf.Write(h.StatusMessage())
	headerBuf.Write(strCRLF)

	// Write all headers
	h.VisitAll(func(key, value []byte) {
		headerBuf.Write(key)
		headerBuf.Write(strColonSpace)
		headerBuf.Write(value)
		headerBuf.Write(strCRLF)
	})

	headerBuf.Write(strCRLF)

	return append(dest[:0], headerBuf.B...)
}

Output:

go run .\main.go
--- Collected Response Details ---
Status Code: 200

Response Headers (268 bytes):
HTTP/1.1 200 OK
Content-Length: 31466742
Content-Type: video/mp4
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Date: Wed, 09 Apr 2025 09:06:19 GMT
Last-Modified: Tue, 20 Oct 2020 10:45:05 GMT
Etag: "1e024f6-5b217ec8840b6"
Accept-Ranges: bytes


Content-Type: video/mp4
Content-Length Header: 31466742
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Total Bytes Received (Headers + Body Preview/Truncated Body): 31466749

Response Body Preview (1024 bytes):
ftypisomisomiso2mp4freeދ_mdat�&
�!���0
��@�dJa=w�����k��
�}y�jT�H��hr��}���G�+�EY��e�����&��.�T�Ӻ�O�^��Kʩ�Yz�:��Og���;��K�o��6�`�v��w�̈́v�׌��h&�F�����-�t"����aUY�b��[����
�o��$����ظ1ޑ���(/�w����v
���`�XN ��p�D��H�>�������g����~zխy�JUX��v��u��ґ��=�)��n]�4�`_��/L)]���j��{��C���r'*�"��w��u�f�l�m-��U�?�9�+�-�@=�8�4��������@�PP&

B�aXPN�Ap�DF    �B��D/�x�_3�U�}7Ϸ���Ǚ��RU�\`սW�����?ֈ��0u2#�ۯD��u�%���J4]ɛ��g�X���k+r�t�P���HSW���>�0�  ZS��b���u8�=$
�`)�0S��^����^ ������݂�`�YP␦
.�u��:J�]�S,�O��␦Ah]o*�M�ӓJTPM�w._~␦q���#���O�h$���FXo�crǖ�Eu���#H˞����[}��w@� Lh�@�`,$)�&��g����n�s[�VY.M(���ya�������y������ɑ}?��v��o�r�9��k:�����C��������s��Z��[�[�KNq ������ �tM�C�Ike�7�����ݡ]+��?���wn��nS�/4a_U`�"��EGA����Mh��J����zĤ ��b騭��+2������mf�0␦��"��w����q

Note: Full response body (31466742 bytes reported by Content-Length) is larger than the preview size (1024 bytes).

Using StreamResponsebody = true and reading the response using resp.Body() looked like the working approach; it worked for most of the requests. However, it also failed under high concurrency, and I'm still unsure of the exact cause.

I suspect that CloseBodyStream closes the stream too early, failing to retrieve the response:

fasthttp/http.go

Line 415 in 4c71125

_, err := copyZeroAlloc(bodyBuf, resp.bodyStream)

fasthttp/http.go

Line 416 in 4c71125

resp.closeBodyStream(err) //nolint:errcheck

Approach 4 - Finally working

I managed to get a final working PoC, this version worked in all test cases. I've successfully sent and retrieved more than 1 million requests/responses (concurrent) on many different servers.

const (
	maxBodySize         = 12288 // Limit the maximum body size (headers + body)
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

StreamResponseBody:  true

I found that resp.BodyWriteTo() calls Write() internally, so I ended up using a LimitedWriter implementation:

fasthttp/http.go

Line 637 in 4c71125

func (resp *Response) BodyWriteTo(w io.Writer) error { 

package main

import (
	"crypto/tls"
	"fmt"
	"io"
	"log"

	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
	"github.com/valyala/fasthttp"
)

var (
	strSpace      = []byte(" ")
	strCRLF       = []byte("\r\n")
	strColonSpace = []byte(": ")
)

var headerBufPool bytesutil.ByteBufferPool
var respPreviewBufPool bytesutil.ByteBufferPool

type RawHTTPResponseDetails struct {
	StatusCode      int
	ResponsePreview []byte // I want a sample preview bytes from the response body
	ResponseHeaders []byte
	ContentType     []byte
	ContentLength   int64
	ServerInfo      []byte
	ResponseBytes   int // Total bytes of the response received (Headers + Body up to maxBodySize)
}

type LimitedWriter struct {
	W io.Writer // Underlying writer
	N int64     // Max bytes remaining
}

func (l *LimitedWriter) Write(p []byte) (n int, err error) {
	if l.N <= 0 {
		return 0, io.EOF
	}
	if int64(len(p)) > l.N {
		p = p[0:l.N]
	}
	n, err = l.W.Write(p)
	l.N -= int64(n)
	return
}

const (
	maxBodySize         = 12288 // Limit the maximum body size (headers + body)
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

func main() {
	url := "https://www.sample-videos.com/video321/mp4/360/big_buck_bunny_360p_30mb.mp4" // 30 mb video

	tlsConfig := &tls.Config{
		InsecureSkipVerify: true,
	}

	client := &fasthttp.Client{
		MaxResponseBodySize: maxBodySize,
		ReadBufferSize:      rwBufferSize,
		WriteBufferSize:     rwBufferSize,
		StreamResponseBody:  true, // Enabling streaming mode is crucial!
		TLSConfig:           tlsConfig,
	}

	req := fasthttp.AcquireRequest()
	defer fasthttp.ReleaseRequest(req)
	req.SetRequestURI(url)
	req.Header.SetMethod(fasthttp.MethodGet)

	resp := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(resp)

	err := client.Do(req, resp)

	if err != nil && err != fasthttp.ErrBodyTooLarge {
		log.Fatalf("Error fetching URL %s: %v", url, err)
	}

	respDetails := RawHTTPResponseDetails{}

	respDetails.StatusCode = resp.StatusCode()
	respDetails.ContentType = resp.Header.ContentType()
	respDetails.ContentLength = int64(resp.Header.ContentLength())
	respDetails.ServerInfo = resp.Header.Server()
	respDetails.ResponseHeaders = GetResponseHeaders(&resp.Header, respDetails.StatusCode, respDetails.ResponseHeaders)

	previewBuf := respPreviewBufPool.Get()
	defer respPreviewBufPool.Put(previewBuf)

	// Create the LimitedWriter to control how much is read from the stream
	limitedWriter := &LimitedWriter{
		W: previewBuf,                 // Write into the pooled buffer
		N: int64(RespBodyPreviewSize), // Limit preview size
	}

	// Read stream into buffer via limitedWriter
	if err := resp.BodyWriteTo(limitedWriter); err != nil && err != io.EOF {
		log.Printf("Warning: Error reading response body stream: %v\n", err)
	}

	if len(previewBuf.B) > 0 {
		respDetails.ResponsePreview = append(respDetails.ResponsePreview[:0], previewBuf.B...)
		respDetails.ResponseBytes = resp.Header.Len() + len(previewBuf.B)
	}

	fmt.Println("--- Collected Response Details (Streaming Mode) ---")
	fmt.Printf("Status Code: %d\n", respDetails.StatusCode)
	fmt.Printf("\nResponse Headers (%d bytes):\n%s", len(respDetails.ResponseHeaders), string(respDetails.ResponseHeaders))
	fmt.Printf("Content-Type: %s\n", string(respDetails.ContentType))
	fmt.Printf("Content-Length Header: %d\n", respDetails.ContentLength)
	fmt.Printf("Server: %s\n", string(respDetails.ServerInfo))
	fmt.Printf("Total Bytes Received (Headers + Body Preview): %d\n", respDetails.ResponseBytes)

	fmt.Printf("\nResponse Body Preview (%d bytes):\n", len(respDetails.ResponsePreview))

	if len(respDetails.ResponsePreview) > 0 {
		fmt.Println(string(respDetails.ResponsePreview))
	} else {
		fmt.Println("<empty preview>")
	}
}

// GetResponseHeaders helper
func GetResponseHeaders(h *fasthttp.ResponseHeader, statusCode int, dest []byte) []byte {
	headerBuf := headerBufPool.Get()
	defer headerBufPool.Put(headerBuf)

	headerBuf.Write(h.Protocol())
	headerBuf.Write(strSpace)
	headerBuf.B = fasthttp.AppendUint(headerBuf.B, statusCode)
	headerBuf.Write(strSpace)
	headerBuf.Write(h.StatusMessage())
	headerBuf.Write(strCRLF)

	h.VisitAll(func(key, value []byte) {
		headerBuf.Write(key)
		headerBuf.Write(strColonSpace)
		headerBuf.Write(value)
		headerBuf.Write(strCRLF)
	})

	headerBuf.Write(strCRLF)

	return append(dest[:0], headerBuf.B...)
}

Output:

go run .\main.go
2025/04/09 13:33:21 Warning: Error reading response body stream: short write
--- Collected Response Details (Streaming Mode) ---
Status Code: 200

Response Headers (268 bytes):
HTTP/1.1 200 OK
Content-Length: 31466742
Content-Type: video/mp4
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Date: Wed, 09 Apr 2025 10:22:50 GMT
Last-Modified: Tue, 20 Oct 2020 10:45:05 GMT
Etag: "1e024f6-5b217ec8840b6"
Accept-Ranges: bytes

Content-Type: video/mp4
Content-Length Header: 31466742
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Total Bytes Received (Headers + Body Preview): 1031

Response Body Preview (1024 bytes):
ftypisomisomiso2mp4freeދ_mdat�&
�!���0
��@�dJa=w�����k��
�}y�jT�H��hr��}���G�+�EY��e�����&��.�T�Ӻ�O�^��Kʩ�Yz�:��Og���;��K�o��6�`�v��w�̈́v�׌��h&�F�����-�t"����aUY�b��[����
�o��$����ظ1ޑ���(/�w����v
���`�XN ��p�D��H�>�������g����~zխy�JUX��v��u��ґ��=�)��n]�4�`_��/L)]���j��{��C���r'*�"��w��u�f�l�m-��U�?�9�+�-�@=�8�4��������@�PP&

B�aXPN�Ap�DF    �B��D/�x�_3�U�}7Ϸ���Ǚ��RU�\`սW�����?ֈ��0u2#�ۯD��u�%���J4]ɛ��g�X���k+r�t�P���HSW���>�0�  ZS��b���u8�=$
�`)�0S��^����^ ������݂�`�YP␦
.�u��:J�]�S,�O��␦Ah]o*�M�ӓJTPM�w._~␦q���#���O�h$���FXo�crǖ�Eu���#H˞����[}��w@� Lh�@�`,$)�&��g����n�s[�VY.M(���ya�������y������ɑ}?��v��o�r�9��k:�����C��������s��Z��[�[�KNq ������ �tM�C�Ike�7�����ݡ]+��?���wn��nS�/4a_U`�"��EGA����Mh��J����zĤ ��b騭��+2������mf�0␦��"��w����q

Approach 4 seems to be working fine for me. I noticed that sometimes it returns io.ErrShortWrite, but I think this is expected based on my implementation.

My question now is, what would be the correct way to read up to MaxResponseBodySize and also capture a preview up to N bytes from the response? Or if I should stick to my approach.
This was referenced Apr 9, 2025
Open
Open
Open
@erikdubbelboer
Copy link
Collaborator

I have never done this myself but your approach 4 seems to be a good approach. I'll keep this issue open for others who might want to do the same. Thanks for exploring this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@erikdubbelboer @slicingmelon