1--- 2layout: "docs" 3page_title: "Signed SSH Certificates - SSH - Secrets Engines" 4sidebar_title: "Signed Certificates" 5sidebar_current: "docs-secrets-ssh-signed-ssh-certificates" 6description: |- 7 The signed SSH certificates is the simplest and most powerful in terms of 8 setup complexity and in terms of being platform agnostic. When using this 9 type, an SSH CA signing key is generated or configured at the secrets engine's mount. 10 This key will be used to sign other SSH keys. 11--- 12 13# Signed SSH Certificates 14 15The signed SSH certificates is the simplest and most powerful in terms of setup 16complexity and in terms of being platform agnostic. By leveraging Vault's 17powerful CA capabilities and functionality built into OpenSSH, clients can SSH 18into target hosts using their own local SSH keys. 19 20In this section, the term "**client**" refers to the person or machine 21performing the SSH operation. The "**host**" refers to the target machine. If 22this is confusing, substitute "client" with "user". 23 24This page will show a quick start for this secrets engine. For detailed documentation 25on every path, use `vault path-help` after mounting the secrets engine. 26 27## Client Key Signing 28 29Before a client can request their SSH key be signed, the Vault SSH secrets engine must 30be configured. Usually a Vault administrator or security team performs these 31steps. It is also possible to automate these actions using a configuration 32management tool like Chef, Puppet, Ansible, or Salt. 33 34### Signing Key & Role Configuration 35 36The following steps are performed in advance by a Vault administrator, security 37team, or configuration management tooling. 38 391. Mount the secrets engine. Like all secrets engines in Vault, the SSH secrets engine 40must be mounted before use. 41 42 ```text 43 $ vault secrets enable -path=ssh-client-signer ssh 44 Successfully mounted 'ssh' at 'ssh-client-signer'! 45 ``` 46 47 This enables the SSH secrets engine at the path "ssh-client-signer". It is 48 possible to mount the same secrets engine multiple times using different 49 `-path` arguments. The name "ssh-client-signer" is not special - it can be 50 any name, but this documentation will assume "ssh-client-signer". 51 521. Configure Vault with a CA for signing client keys using the `/config/ca` 53endpoint. If you do not have an internal CA, Vault can generate a keypair for 54you. 55 56 ```text 57 $ vault write ssh-client-signer/config/ca generate_signing_key=true 58 Key Value 59 --- ----- 60 public_key ssh-rsa AAAAB3NzaC1yc2EA... 61 ``` 62 63 If you already have a keypair, specify the public and private key parts as 64 part of the payload: 65 66 ```text 67 $ vault write ssh-client-signer/config/ca \ 68 private_key="..." \ 69 public_key="..." 70 ``` 71 72 Regardless of whether it is generated or uploaded, the client signer public 73 key is accessible via the API at the `/public_key` endpoint. 74 751. Add the public key to all target host's SSH configuration. This process can 76be manual or automated using a configuration management tool. The public key is 77accessible via the API and does not require authentication. 78 79 ```text 80 $ curl -o /etc/ssh/trusted-user-ca-keys.pem http://127.0.0.1:8200/v1/ssh-client-signer/public_key 81 ``` 82 83 ```text 84 $ vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem 85 ``` 86 87 Add the path where the public key contents are stored to the SSH 88 configuration file as the `TrustedUserCAKeys` option. 89 90 ```text 91 # /etc/ssh/sshd_config 92 # ... 93 TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem 94 ``` 95 96 Restart the SSH service to pick up the changes. 97 981. Create a named Vault role for signing client keys. 99 100 Because of the way some SSH certificate features are implemented, options 101 are passed as a map. The following example adds the `permit-pty` extension 102 to the certificate. 103 104 ```text 105 $ vault write ssh-client-signer/roles/my-role -<<"EOH" 106 { 107 "allow_user_certificates": true, 108 "allowed_users": "*", 109 "default_extensions": [ 110 { 111 "permit-pty": "" 112 } 113 ], 114 "key_type": "ca", 115 "default_user": "ubuntu", 116 "ttl": "30m0s" 117 } 118 EOH 119 ``` 120 121### Client SSH Authentication 122 123The following steps are performed by the client (user) that wants to 124authenticate to machines managed by Vault. These commands are usually run from 125the client's local workstation. 126 1271. Locate or generate the SSH public key. Usually this is `~/.ssh/id_rsa.pub`. 128If you do not have an SSH keypair, generate one: 129 130 ```text 131 $ ssh-keygen -t rsa -C "user@example.com" 132 ``` 133 1341. Ask Vault to sign your **public key**. This file usually ends in `.pub` and 135the contents begin with `ssh-rsa ...`. 136 137 ```text 138 $ vault write ssh-client-signer/sign/my-role \ 139 public_key=@$HOME/.ssh/id_rsa.pub 140 141 Key Value 142 --- ----- 143 serial_number c73f26d2340276aa 144 signed_key ssh-rsa-cert-v01@openssh.com AAAAHHNzaC1... 145 ``` 146 147 The result will include the serial and the signed key. This signed key is 148 another public key. 149 150 To customize the signing options, use a JSON payload: 151 152 ```text 153 $ vault write ssh-client-signer/sign/my-role -<<"EOH" 154 { 155 "public_key": "ssh-rsa AAA...", 156 "valid_principals": "my-user", 157 "key_id": "custom-prefix", 158 "extension": { 159 "permit-pty": "" 160 } 161 } 162 EOH 163 ``` 164 1651. Save the resulting signed, public key to disk. Limit permissions as needed. 166 167 ```text 168 $ vault write -field=signed_key ssh-client-signer/sign/my-role \ 169 public_key=@$HOME/.ssh/id_rsa.pub > signed-cert.pub 170 ``` 171 172 If you are saving the certificate directly beside your SSH keypair, suffix 173 the name with `-cert.pub` (`~/.ssh/id_rsa-cert.pub`). With this naming 174 scheme, OpenSSH will automatically use it during authentication. 175 1761. (Optional) View enabled extensions, principals, and metadata of the signed 177key. 178 179 ```text 180 $ ssh-keygen -Lf ~/.ssh/signed-cert.pub 181 ``` 182 1831. SSH into the host machine using the signed key. You must supply both the 184signed public key from Vault **and** the corresponding private key as 185authentication to the SSH call. 186 187 ```text 188 $ ssh -i signed-cert.pub -i ~/.ssh/id_rsa username@10.0.23.5 189 ``` 190 191## Host Key Signing 192 193For an added layers of security, we recommend enabling host key signing. This is 194used in conjunction with client key signing to provide an additional integrity 195layer. When enabled, the SSH agent will verify the target host is valid and 196trusted before attempting to SSH. This will reduce the probability of a user 197accidentally SSHing into an unmanaged or malicious machine. 198 199### Signing Key Configuration 200 2011. Mount the secrets engine. For the most security, mount at a different path from the 202client signer. 203 204 ```text 205 $ vault secrets enable -path=ssh-host-signer ssh 206 Successfully mounted 'ssh' at 'ssh-host-signer'! 207 ``` 208 2091. Configure Vault with a CA for signing host keys using the `/config/ca` 210endpoint. If you do not have an internal CA, Vault can generate a keypair for 211you. 212 213 ```text 214 $ vault write ssh-host-signer/config/ca generate_signing_key=true 215 Key Value 216 --- ----- 217 public_key ssh-rsa AAAAB3NzaC1yc2EA... 218 ``` 219 220 If you already have a keypair, specify the public and private key parts as 221 part of the payload: 222 223 ```text 224 $ vault write ssh-host-signer/config/ca \ 225 private_key="..." \ 226 public_key="..." 227 ``` 228 229 Regardless of whether it is generated or uploaded, the host signer public 230 key is accessible via the API at the `/public_key` endpoint. 231 2321. Extend host key certificate TTLs. 233 234 ```text 235 $ vault secrets tune -max-lease-ttl=87600h ssh-host-signer 236 ``` 237 2381. Create a role for signing host keys. Be sure to fill in the list of allowed 239domains, set `allow_bare_domains`, or both. 240 241 ```text 242 $ vault write ssh-host-signer/roles/hostrole \ 243 key_type=ca \ 244 ttl=87600h \ 245 allow_host_certificates=true \ 246 allowed_domains="localdomain,example.com" \ 247 allow_subdomains=true 248 ``` 249 2501. Sign the host's SSH public key. 251 252 ```text 253 $ vault write ssh-host-signer/sign/hostrole \ 254 cert_type=host \ 255 public_key=@/etc/ssh/ssh_host_rsa_key.pub 256 Key Value 257 --- ----- 258 serial_number 3746eb17371540d9 259 signed_key ssh-rsa-cert-v01@openssh.com AAAAHHNzaC1y... 260 ``` 261 2621. Set the resulting signed certificate as `HostCertificate` in the SSH 263configuration on the host machine. 264 265 ```text 266 $ vault write -field=signed_key ssh-host-signer/sign/hostrole \ 267 cert_type=host \ 268 public_key=@/etc/ssh/ssh_host_rsa_key.pub > /etc/ssh/ssh_host_rsa_key-cert.pub 269 ``` 270 271 Set permissions on the certificate to be `0640`: 272 273 ```text 274 $ chmod 0640 /etc/ssh/ssh_host_rsa_key-cert.pub 275 ``` 276 277 Add host key and host certificate to the SSH configuration file. 278 279 ```text 280 # /etc/ssh/sshd_config 281 # ... 282 283 # For client keys 284 TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem 285 286 # For host keys 287 HostKey /etc/ssh/ssh_host_rsa_key 288 HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub 289 ``` 290 291 Restart the SSH service to pick up the changes. 292 293### Client-Side Host Verification 294 2951. Retrieve the host signing CA public key to validate the host signature of 296target machines. 297 298 ```text 299 $ curl http://127.0.0.1:8200/v1/ssh-host-signer/public_key 300 ``` 301 302 ```text 303 $ vault read -field=public_key ssh-host-signer/config/ca 304 ``` 305 3061. Add the resulting public key to the `known_hosts` file with authority. 307 308 ```text 309 # ~/.ssh/known_hosts 310 @cert-authority *.example.com ssh-rsa AAAAB3NzaC1yc2EAAA... 311 ``` 312 3131. SSH into target machines as usual. 314 315## Troubleshooting 316 317When initially configuring this type of key signing, enable `VERBOSE` SSH 318logging to help annotate any errors in the log. 319 320```text 321# /etc/ssh/sshd_config 322# ... 323LogLevel VERBOSE 324``` 325 326Restart SSH after making these changes. 327 328By default, SSH logs to `/var/log/auth.log`, but so do many other things. To 329extract just the SSH logs, use the following: 330 331```sh 332$ tail -f /var/log/auth.log | grep --line-buffered "sshd" 333``` 334 335If you are unable to make a connection to the host, the SSH server logs may 336provide guidance and insights. 337 338### Name is not a listed principal 339 340If the `auth.log` displays the following messages: 341 342```text 343# /var/log/auth.log 344key_cert_check_authority: invalid certificate 345Certificate invalid: name is not a listed principal 346``` 347 348The certificate does not permit the username as a listed principal for 349authenticating to the system. This is most likely due to an OpenSSH bug (see 350[known issues](#known-issues) for more information). This bug does not respect 351the `allowed_users` option value of "\*". Here are ways to work around this 352issue: 353 3541. Set `default_user` in the role. If you are always authenticating as the same 355user, set the `default_user` in the role to the username you are SSHing into the 356target machine: 357 358 ```text 359 $ vault write ssh/roles/my-role -<<"EOH" 360 { 361 "default_user": "YOUR_USER", 362 // ... 363 } 364 EOH 365 ``` 366 3671. Set `valid_principals` during signing. In situations where multiple users may 368be authenticating to SSH via Vault, set the list of valid principles during key 369signing to include the current username: 370 371 ```text 372 $ vault write ssh-client-signer/sign/my-role -<<"EOH" 373 { 374 "valid_principals": "my-user" 375 // ... 376 } 377 EOH 378 ``` 379 380 381### No Prompt After Login 382 383If you do not see a prompt after authenticating to the host machine, the signed 384certificate may not have the `permit-pty` extension. There are two ways to add 385this extension to the signed certificate. 386 387- As part of the role creation 388 389 ```text 390 $ vault write ssh-client-signer/roles/my-role -<<"EOH" 391 { 392 "default_extensions": [ 393 { 394 "permit-pty": "" 395 } 396 ] 397 // ... 398 } 399 EOH 400 ``` 401 402- As part of the signing operation itself: 403 404 ```text 405 $ vault write ssh-client-signer/sign/my-role -<<"EOH" 406 { 407 "extension": { 408 "permit-pty": "" 409 } 410 // ... 411 } 412 EOH 413 ``` 414 415### No Port Forwarding 416 417If port forwarding from the guest to the host is not working, the signed 418certificate may not have the `permit-port-forwarding` extension. Add the 419extension as part of the role creation or signing process to enable port 420forwarding. See [no prompt after login](#no-prompt-after-login) for examples. 421 422```json 423{ 424 "default_extensions": [ 425 { 426 "permit-port-forwarding": "" 427 } 428 ] 429} 430``` 431 432### No X11 Forwarding 433 434If X11 forwarding from the guest to the host is not working, the signed 435certificate may not have the `permit-X11-forwarding` extension. Add the 436extension as part of the role creation or signing process to enable X11 437forwarding. See [no prompt after login](#no-prompt-after-login) for examples. 438 439```json 440{ 441 "default_extensions": [ 442 { 443 "permit-X11-forwarding": "" 444 } 445 ] 446} 447``` 448 449### No Agent Forwarding 450 451If agent forwarding from the guest to the host is not working, the signed 452certificate may not have the `permit-agent-forwarding` extension. Add the 453extension as part of the role creation or signing process to enable agent 454forwarding. See [no prompt after login](#no-prompt-after-login) for examples. 455 456```json 457{ 458 "default_extensions": [ 459 { 460 "permit-agent-forwarding": "" 461 } 462 ] 463} 464``` 465 466### Known Issues 467 468- On SELinux-enforcing systems, you may need to adjust related types so that the 469 SSH daemon is able to read it. For example, adjust the signed host certificate 470 to be an `sshd_key_t` type. 471 472- On some versions of SSH, you may get the following error: 473 474 ```text 475 no separate private key for certificate 476 ``` 477 478 This is a bug introduced in OpenSSH version 7.2 and fixed in 7.5. See 479 [OpenSSH bug 2617](https://bugzilla.mindrot.org/show_bug.cgi?id=2617) for 480 details. 481 482## API 483 484The SSH secrets engine has a full HTTP API. Please see the 485[SSH secrets engine API](/api/secret/ssh/index.html) for more 486details. 487