net/http: add switches to accept/send bogus HTTP?

[crhino]

We are writing a reverse proxy that we want to be completely transparent in terms of the HTTP request URI. If a client sends an invalid Request URI, we want to forward it on as is and not modify it in any way.

We have been able to almost achieve this goal using the url.Opaque field, expect for http requests starting with //. In the URL.RequestURI() method here, we see that an explicit check is being made for a prefix of // and the Scheme is appended. This was added for issue #4860.

The issue #10433 mentions a workaround for the specific usecase of //, but using RawPath makes URL.RequestURI() return a url-encoded RequestURI through the use of EscapedPath(). This is also a violation of our goal of transparency.

If there is another workaround that we are not aware, help finding that out would be appreciated, otherwise we would ask whether or not this is a feature the Go developers would support.

Reference issue: cloudfoundry/gorouter#60

@crhino && @shashwathi

[bradfitz]

It's true that in general we don't let you send illegal stuff.

But why can't you send a request with two leading slashes? It would just look like the absolute-path form of http://tools.ietf.org/html/rfc7230#section-5.3 e.g. http://play.golang.org/p/9fU1tChOrU

What are the bytes of the HTTP request you're trying to send?

[crhino]

Here is a playground example of what we are trying to do: http://play.golang.org/p/r5-UXAoBsu

My understanding is that a path beginning with "//" is not allowed, according to http://tools.ietf.org/html/rfc3986#section-3.3. However, many clients allow this to be sent, and as such we want to pass it on unmodified.

[bradfitz]

My understanding is that a path beginning with "//" is not allowed, according to http://tools.ietf.org/html/rfc3986#section-3.3. However, many clients allow this to be sent, and as such we want to pass it on unmodified.

Go has no problem with them. See the snippet I wrote above (http://play.golang.org/p/9fU1tChOrU) for generating one.

[bradfitz]

I guess //£^sdf would be a better example of a path which can't be generated with Go.

[bradfitz]

What I can't quickly find reading cloudfoundry/gorouter#60 is why this matters.

What client is generating GET //foo paths?
What server is requiring //foo requests?

[youngm]

@bradfitz I believe the root problem is that when a client requests GET //foo, as it passes through the go written transparent proxy it is transformed to GET http://foo which as it passes onto the final destination server becomes a 404 since the destination server was expecting //foo or /foo but not http://foo. You can see it @crhino playground example that what the go request handler thinks it got a request for http://foo instead of //foo that was originally sent.

I think you're simple example only deals with generating the request. I think the problem is on the go request handler side not the go client request side.

Does that help?

Mike

[bradfitz]

@youngm, can you answer my two questions, though?

[youngm]

@bradfitz In the case of cloudfoundry/gorouter#60 the client is a Commercial off the shelf web based product performing an xhr request. Because it is a commercial product it is not easily modified unlike a custom build application. The Server is a Java Tomcat Server.

The javacript in this commercial application is obviously not supposed to send //foo requests. But, the desire here is that regardless of this client invalid request we'd like to write a transparent proxy using go http that doesn't munge a request like this.

[bradfitz]

But, the issue here is that regardless of this client invalid request it should be possible to write a transparent proxy using go http that doesn't munge a request like this.

It should certainly be possible to write a proxy in Go, but the answer might be using the net package and not the net/http package. The net/http package has only been getting stricter over time.

Even if we found a nice way for you to send bogus outgoing HTTP requests, I would want to make it opt-in, so we never speak invalid HTTP by default. But then we'd want to do the same thing for the HTTP server.

For instance, I bet there there already a dozen HTTP requests you won't be able to handle with the net/http server because they're rejected before your Handler gets it. I might even add rejecting bogus Request-Line targets now that I'm aware of it.

So really this bug is about deciding whether we add an opt-in mechanism for both client & server to allow bogus stuff.

[youngm]

@bradfitz I think that sums it up correctly. Do you want to weigh in @crhino ? Since has implications for gorouter?

[oxtoacart]

Where this gets interesting is with custom error pages. We have a project that was already doing some validation of its own and then returning not just a 400 status, but an error page with details (similar to what's suggested in #10123). Now with Go 1.6, the request doesn't even reach the handler infrastructure, so we don't have the ability to return a custom error page. It seems like there are two solutions to this:

1. Allow user of http.Server to disable certain validation as this ticket suggests
2. Allow user of http.Server to register custom error handlers for any validations that are performed

1 seems easier to implement initially and more broadly applicable. 2 seems more convenient for the specific case of displaying custom error pages.

@bradfitz What do you think?

cc: @uaalto

[oxtoacart]

@bradfitz This is the bug we were just talking about. Basically our main use for this is that we have an HTTP proxy server that mimics an unconfigured Apache server under certain circumstances. In the case of a 400 bad request, Apache has specific HTTP headers that it sends when responding, as well as specific HTML body content. We'd like to be able to continue responding with these.

This may be a slightly different use case than for this original bug, namely we're not interested in being able to handle and process bad requests, we just want to have control over the specifics of the response. However, if it was possible to let these bad requests just fall through to a regular http.Handler, that would allow us to do what we need.

[glasser]

What client is generating GET //foo paths?

For what it's worth, plenty of browsers (I just verified this in Chrome, Safari, and curl) will send GET //foo if a human user chooses to type http://host//foo into them. Popular websites seem to vary as to whether or not they ignore extraneous slashes (eg, www.google.com appears to, www.facebook.com appears to not).

[seankhliao]

I believe our later discussions have leaned towards not supporting this use case for net/http.