Enable PROPFIND of Director health test API

[jhiemstrawisc]

The cache's xrdcl-pelican plugin switched from doing a HEAD request against requested resources to a PROPFIND request, the answer to which is used by the cache to determine whether something is an object or a collection. That's hunky dory, except that it broke the pseudo Director origin responsible for generating health test files, because the Director's health test API wasn't handling the propfinds.

This commit adds PROPFIND handling and generates the XML needed by the cache to understand the response. Note that I constructed the XML by pinging a cache with a real PROPFIND and then templating the response with file name, size, and time stamp.

One item remaining here is an actual E2E test of the setup -- while the unit tests are passing just fine, they passed just fine when everything broke... a better setup is forthcoming.

[bbockelm]

The reason why we don't have unit test coverage here is that scitokens.cfg circular dependencies required circular logic (the audience URL in the scitokens.cfg requires the port number but the port number is only known after scitokens.cfg is parsed). The director uses audience URLs when generating tokens for the cache tests.

So, any future work for an end-to-end test requires:

1. Having a test-specific mode where the director doesn't add an audience. Don't love that because we are testing different code paths than we run in production.
2. (New!) Take advantage of [XrdSciTokens] Automatically add WLCG-audiences upon request xrootd/xrootd#2417 - now in the test container since this is in our patch build. That PR has XRootD put in the correct port, not the writer of the scitokens.cfg, breaking the circular dependency mentioned above.

I think (2) is the right way to go -- but it'll require a separate follow-up to leverage the xrootd functionality first. So, I'm OK with splitting E2E out into a separate PR.

[jhiemstrawisc]

I don't follow -- /pelican/monitoring is a public namespace, appearing in the cache's authfile as:

u * /pelican/monitoring lr

Have I completely missed something related to tokens and health test files?

[jhiemstrawisc]

Adding a second reviewer to help load balance some of these reviews while we're in crunch time for 7.14

[turetske]

Minor changes. I also don't know what Brian is asking for either.

[turetske]

I think there are benefits to both. Bash tests allow us to run integration tests using the commands and potential environment that are in the real word and are important to do. I would are that instead of "it's easier", instead to say that this "allows us to leverage the benefits of Golang's testing package".

[jhiemstrawisc]

I actually do think "easier" is an accurate description for some tests. For example, in the e2e cache test from this PR, I'm using the fact that I can directly call functions from our go packages to simplify some aspects of E2E testing while still spinning up an entire federation. This lets me tap into specific functionality while still testing that the whole communication pipeline for token fetching works as needed. This would be very difficult to do with raw bash.

[jhiemstrawisc]

That I can tell, all these failing tests are related to #1942. Until the underlying issue is resolved, let's plan to ignore test failures when they're related only to the memstat stuff. We could chase green boxes by re-running everything until we get lucky, but I don't think that's a good use of our time when the failures are isolated to a known bad test.