I spent way too long on this, but the post from @Randall Okwei as well as a lot of trial and error allowed me to get this working without changing the service account from Network Service to Local System. This has to be some kind of bug in WAC. Anyway, hopefully the below helps some folks that are stuck, and maybe someone else facing this issue can pinpoint which step below actually fixed things:
- When the ajax 500 error appeared, I noticed the following Schannel error in the System event log:
If you are importing a PEM cert, you may want to try this article to convert your certificate to PFX via OpenSSL. I already had a PFX cert and this article ultimately didn't fix things for me, but may be useful for some.A fatal error occurred when attempting to access the TLS server credential private key. The error code returned from the cryptographic module is 0x8009030D. The internal error state is 10001. The SSPI client process is WindowsAdminCenter (PID: 9948).
- I noticed the self-signed cert (which works properly under the Network Service account) has Basic Constraints enabled. So, within my Windows CA, I duplicated the Web Server template, changed the compatibility to Server 2008 R2 and enabled the extension "Basic Constraints" and made it critical.
- Probably the most important step: With a cert from the new template imported into my Personal computer store, I right-clicked the cert --> All Tasks --> Managed Private Keys... I added the following permissions to match self-signed cert:
- Network Service account for the local machine: Full Control
- Users group for local machine: Read
- Probably the second most important step: Reinstall WAC and point to the new certificate. I was unable to get the Set-WACCertificateSubjectName command to actually apply the new cert. The Set-WACCertificateAcl gave an error in the WAC configuration log that was unfortunately overwritten during the reinstall, but it implied it couldn't read the private key properly.
I would recommend starting with steps 3 and 4 to see if that fixes it, otherwise it appears the Basic Constraints extension is required. I was NOT able to get this to work just by changing the private key permissions on an existing setup that was broken, which makes me think the private key permissions change plus the reinstall is what will fix it.