Chef Infra Client Security

[edit on GitHub]

All communication with the Chef Infra Server must be authenticated using the Chef Infra Server API, which is a REST API that allows requests to be made to the Chef Infra Server. Only authenticated requests will be authorized. Most of the time, and especially when using knife, Chef Infra Client, or the Chef Infra Server web interface, the use of the Chef Infra Server API is transparent. In some cases, the use of the Chef Infra Server API requires more detail, such as when making the request in Ruby code, with a knife plugin, or when using cURL.

Authentication

The authentication process ensures the Chef Infra Server responds only to requests made by trusted users. Public key encryption is used by the Chef Infra Server. When a node and/or a workstation is configured to run Chef Infra Client, both public and private keys are created. The public key is stored on the Chef Infra Server, while the private key is returned to the user for safe keeping. (The private key is a .pem file located in the .chef directory or in /etc/chef.)

Both Chef Infra Client and knife use the Chef Infra Server API when communicating with the Chef Infra Server. The chef-validator uses the Chef Infra Server API, but only during the first Chef Infra Client run on a node.

Each request to the Chef Infra Server from those executables sign a special group of HTTP headers with the private key. The Chef Infra Server then uses the public key to verify the headers and verify the contents.

chef-validator

Every request made by Chef Infra Client to the Chef Infra Server must be an authenticated request using the Chef Infra Server API and a private key. When Chef Infra Client makes a request to the Chef Infra Server, Chef Infra Client authenticates each request using a private key located in /etc/chef/client.pem.

However, during the first Chef Infra Client run, this private key does not exist. Instead, Chef Infra Client attempts to use the private key assigned to the chef-validator, located in /etc/chef/validation.pem. (If, for any reason, the chef-validator is unable to make an authenticated request to the Chef Infra Server, the initial Chef Infra Client run will fail.)

During the initial Chef Infra Client run, Chef Infra Client registers itself with the Chef Infra Server using the private key assigned to the chef-validator, after which Chef Infra Client will obtain a client.pem private key for all future authentication requests to the Chef Infra Server.

After the initial Chef Infra Client run has completed successfully, the chef-validator is no longer required and may be deleted from the node. Use the delete_validation recipe found in the chef-client cookbook (https://github.com/chef-cookbooks/chef-client) to remove the chef-validator.

During a Chef Infra Client Run

As part of every Chef Infra Client run, Chef Infra Client authenticates to the Chef Infra Server using an RSA private key and the Chef Infra Server API.

authentication_protocol_version

The authentication_protocol_version option in the client.rb file is used to determine the authentication protocol that communicates with Chef Infra Server. For example, specify protocol version 1.3 to enable support for SHA-256 algorithms:

knife[:authentication_protocol_version] = '1.3'

Note that authentication protocol 1.3 is only supported on Chef Server versions 12.4.0 and above.

SSL Certificates

Warning

The following information does not apply to hosted Chef Server 12, only to on-premises Chef Server 12.

Chef Server 12 and later enables SSL verification by default for all requests made to the server, such as those made by knife and Chef Infra Client. The certificate that is generated during the installation of the Chef Infra Server is self-signed, which means the certificate is not signed by a trusted certificate authority (CA) that ships with Chef Infra Client. The certificate generated by the Chef Infra Server must be downloaded to any machine from which knife and/or Chef Infra Client will make requests to the Chef Infra Server.

For example, without downloading the SSL certificate, the following knife command:

knife client list

responds with an error similar to:

ERROR: SSL Validation failure connecting to host: chef-server.example.com ...
ERROR: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 ...

This is by design and will occur until a verifiable certificate is added to the machine from which the request is sent.

`/.chef/trusted_certs`

The /.chef/trusted_certs directory stores trusted SSL certificates used to access the Chef Infra Server:

On each workstation, this directory is the location into which SSL certificates are placed after they are downloaded from the Chef Infra Server using the knife ssl fetch subcommand
On every node, this directory is the location into which SSL certificates are placed when a node has been bootstrapped with Chef Infra Client from a workstation

SSL_CERT_FILE

Use the SSL_CERT_FILE environment variable to specify the location for the SSL certificate authority (CA) bundle that is used by Chef Infra Client.

A value for SSL_CERT_FILE is not set by default. Unless updated, the locations in which Chef Infra will look for SSL certificates are:

Chef Infra Client: /opt/chef/embedded/ssl/certs/cacert.pem
ChefDK: /opt/chefdk/embedded/ssl/certs/cacert.pem
Chef Workstation: /opt/chef-workstation/embedded/ssl/certs/cacert.pem

Keeping the default behavior is recommended. To use a custom CA bundle, update the environment variable to specify the path to the custom CA bundle. If (for some reason) SSL certificate verification stops working, ensure the correct value is specified for SSL_CERT_FILE.

client.rb Settings

Use following client.rb settings to manage SSL certificate preferences:

Setting	Description
`local_key_generation`	Whether the Chef Infra Server or Chef Infra Client generates the private/public key pair. When `true`, Chef Infra Client generates the key pair, and then sends the public key to the Chef Infra Server. Default value: `true`.
`ssl_ca_file`	The file in which the OpenSSL key is saved. Chef Infra Client generates this setting automatically and most users do not need to modify it.
`ssl_ca_path`	The path to where the OpenSSL key is located. Chef Infra Client generates this setting automatically and most users do not need to modify it.
`ssl_client_cert`	The OpenSSL X.509 certificate used for mutual certificate validation. This setting is only necessary when mutual certificate validation is configured on the Chef Infra Server. Default value: `nil`.
`ssl_client_key`	The OpenSSL X.509 key used for mutual certificate validation. This setting is only necessary when mutual certificate validation is configured on the Chef Infra Server. Default value: `nil`.
`ssl_verify_mode`	Set the verify mode for HTTPS requests. Use `:verify_none` to do no validation of SSL certificates. Use `:verify_peer` to do validation of all SSL certificates, including the Chef Infra Server connections, S3 connections, and any HTTPS remote_file resource URLs used in a Chef Infra Client run. This is the recommended setting. Depending on how OpenSSL is configured, the `ssl_ca_path` may need to be specified. Default value: `:verify_peer`.
`verify_api_cert`	Verify the SSL certificate on the Chef Infra Server. When `true`, Chef Infra Client always verifies the SSL certificate. When `false`, Chef Infra Client uses the value of `ssl_verify_mode` to determine if the SSL certificate requires verification. Default value: `false`.

Knife Subcommands

The Chef Infra Client includes two knife commands for managing SSL certificates:

Use knife ssl check to troubleshoot SSL certificate issues
Use knife ssl fetch to pull down a certificate from the Chef Infra Server to the /.chef/trusted_certs directory on the workstation.

After the workstation has the correct SSL certificate, bootstrap operations from that workstation will use the certificate in the /.chef/trusted_certs directory during the bootstrap operation.

knife ssl check

Run the knife ssl check subcommand to verify the state of the SSL certificate, and then use the response to help troubleshoot issues that may be present.

Verified

If the SSL certificate can be verified, the response to

knife ssl check

is similar to:

Connecting to host chef-server.example.com:443
Successfully verified certificates from 'chef-server.example.com'

Unverified

If the SSL certificate cannot be verified, the response to

knife ssl check

is similar to:

Connecting to host chef-server.example.com:443
ERROR: The SSL certificate of chef-server.example.com could not be verified
Certificate issuer data:
  /C=US/ST=WA/L=S/O=Corp/OU=Ops/CN=chef-server.example.com/emailAddress=you@example.com

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.2u  20 Dec 2019
* Certificate file: /opt/chef-workstation/embedded/ssl/cert.pem
* Certificate directory: /opt/chef-workstation/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/grantmc/Downloads/chef-repo/.chef/trusted_certs"

TO FIX THIS ERROR:

If the server you are connecting to uses a self-signed certificate,
you must configure chef to trust that certificate.

By default, the certificate is stored in the following location on the
host where your Chef Infra Server runs:

  /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently:

  /Users/grantmc/Downloads/chef-repo/.chef/trusted_certs)

using SSH/SCP or some other secure method, then re-run this command to
confirm that the certificate is now trusted.

knife ssl fetch

Run the knife ssl fetch to download the self-signed certificate from the Chef Infra Server to the /.chef/trusted_certs directory on a workstation.

Verify Checksums

The SSL certificate that is downloaded to the /.chef/trusted_certs directory should be verified to ensure that it is, in fact, the same certificate as the one located on the Chef Infra Server. This can be done by comparing the SHA-256 checksums.

View the checksum on the Chef Infra Server:

ssh ubuntu@chef-server.example.com sudo sha256sum /var/opt/opscode/nginx/ca/chef-server.example.com.crt

The response is similar to:

<ABC123checksum>  /var/opt/opscode/nginx/ca/chef-server.example.com.crt

View the checksum on the workstation:

gsha256sum .chef/trusted_certs/chef-server.example.com.crt

The response is similar to:

<ABC123checksum>  .chef/trusted_certs/chef-server.example.com.crt

Verify that the checksum values are identical.