Still working on puppetizing cert installation on Windows, which is proving to be a slightly larger headache than hoped-for (although I should have expected it - as a co-worker said, “Puppet for Windows: Why Did You Make Us Build This? ™”). First thing I ran into is that Pupet logs its activity to the Application Event Log. So far so good. Then I discover that it logs the output of PowerShell command failures one line per event. So a response like

CertUtil: Unexpected "-csp" option

Usage:
  CertUtil [Options] [-dump]
  CertUtil [Options] [-dump] File
  Dump configuration information or files

Options:
  -f                -- Force overwrite
  -user             -- Use HKEY_CURRENT_USER keys or certificate store
  -Unicode          -- Write redirected output in Unicode
  -gmt              -- Display times as GMT
  -seconds          -- Display times with seconds and milliseconds
  -silent           -- Use silent flag to acquire crypt context
  -split            -- Split embedded ASN.1 elements, and save to files
  -v                -- Verbose operation
  -privatekey       -- Display password and private key data
  -pin PIN                  -- Smart Card PIN
  -p Password               -- Password
  -t Timeout                -- URL fetch timeout in milliseconds
  -sid WELL_KNOWN_SID_TYPE  -- Numeric SID
            22 -- Local System
            23 -- Network Service
            24 -- Local Service

CertUtil -?              -- Display a verb list (command list)
CertUtil -dump -?        -- Display help text for the "dump" verb
CertUtil -v -?           -- Display all help text for all verbs

is logged as 28 separate events. Ugh. #

I remain to be a little baffled here, though, even after having pieced together the error messages. I couldn’t figure out why the PowerShell command was rejecting the -csp option, which is demonstrably correct when run in PowerShell directly. Then I realized as evidenced by the output above and by the fact that subsequent events reject -importpfx as an invalid PowerShell command that the PowerShell interpreter is breaking the command up for some reason, and is reading certutil.exe -csp 'Microsoft Enhanced RSA and AES Cryptographic Provider' -p $password$ -importpfx $path$ as two (or more) separate commands, the first of which being certutil.exe -csp 'Microsoft Enhanced RSA and AES Cryptographic Provider'. Certutil then interprets that as a call without a verb, defaulting to the “dump” verb, which doesn’t take the -csp option. #

I hypothesized that perhaps something in the password, which contains symbols, was fracking things up, but changing the password to abcdefgh didn’t resolve the issue; so much for that. #

Instead, it appears to have something to do with the secrets expansion via string interpolation, because placing the password in plain text seems to work but interpolating it via "${secrets::shop_cert_pass}" fails. Plain text password is fine, ${secrets::shop_cert_pass} fails with PowerShell treating the command string as distinct commands, and $secrets::shop_cert_pass (i.e. without the curly braces) also is interpreted as distinct commands by powershell #

Finally sorted the root problem; the issue was with the way I was leveraging the puppet secrets module. We load secrets from local files on the puppet master, assign them to class variables in a secrets module, and then leverage the variables in our own classes. The variables get assigned the contents of the secrets files using file() or chomp(file()) variously. When the variable’s value is assigned using file() it includes a terminal newline, which, when interpolated in a PowerShell command definition, is interpreted as starting a new command, causing the behavior I described above. The final fix was to switch to using chomp(file()) instead. #

Other interesting things I ran into and / or learned:

1. using the Windows Puppet Command Prompt to run puppet agent --verbose will echo the return values from the catalog’s command to the console (more or less obvious, I guess). The exec resource’s logoutput may also come in handy here.
2. running puppet agent keeps the agent runnning “in the console” so it continues to request the catalog from the master and apply it on whatever cycle it’s configured for.
3. in our environment, at least, tail /var/log/daemon.log -f on the puppet master outputs the master’s activity, which helped solve some master-side problems (like permissions errors). #