SP read request to Keyvault secrets fail with access denied

[TechnicallyWilliams]

Background:

Deployment service principal with "owner" and/or "reader" role of the keyvault resource is not able to read stored secrets during some ISO template integration tests.

The deployments that have run into this error have done so on subsequent deployments. Initial deployments have not surfaced this permissions error.

Description

Investigate a permanent fix or work around for keyvault read permissions.

Acceptance Criteria

Reference: [Done-Done Checklist] (https://github.com/Microsoft/code-with-engineering-playbook/blob/master/Engineering/BestPractices/DoneDone.md)

• Evidence that a deployment service principal can read secrets from keyvault during ISO template integration tests.

Also, here are a few points that need to be addressed:

1. One theory is that the additional app service service principal created during the initial deployment is the source of the problem.
2. It might be a product bug: https://blogs.technet.microsoft.com/kv/2018/08/31/announcing-virtual-network-service-endpoints-for-key-vault-preview/
3. Turns out this was not a product bug issue or naming collision issue. Instead, multiple deployments were sharing a keyvault resource that was only configurable for a single deployment service principal.

Resources

Deployment Error:
//
//...

Tasks

Assignee should break down work into tasks here

[KeithJRome]

We believe that this is solved with #281