Configuring Kyverno

Customizing Kyverno

Kyverno has many different functions and supports a wide range of possible customizations. This section provides more information on Kyverno’s supporting resources and how they can be customized to tune certain behaviors.

Certificate Management

The Kyverno policy engine runs as an admission webhook and requires a CA-signed certificate and key to setup secure TLS communication with the Kubernetes API server. There are two ways to configure secure communications between Kyverno and the API server.

Default certificates

By default, Kyverno will automatically generate self-signed Certificate Authority (CA) and a leaf certificates for use in its webhook registrations. The CA certificate expires after one year. When Kyverno manage its own certificates, it will gracefully handle regeneration upon expiry.

After installing Kyverno, use the step CLI to check and verify certificate details.

Get all Secrets in Kyverno’s Namespace. The Secret names are configurable, see the container flags section for more details.

1$ kubectl -n kyverno get secret
2NAME                                                      TYPE                 DATA   AGE
3kyverno-cleanup-controller.kyverno.svc.kyverno-tls-ca     kubernetes.io/tls    2      21d
4kyverno-cleanup-controller.kyverno.svc.kyverno-tls-pair   kubernetes.io/tls    2      21d
5kyverno-svc.kyverno.svc.kyverno-tls-ca                    kubernetes.io/tls    2      21d
6kyverno-svc.kyverno.svc.kyverno-tls-pair                  kubernetes.io/tls    2      21d

Get and decode the CA certificate used by the admission controller.

1$ kubectl -n kyverno get secret kyverno-svc.kyverno.svc.kyverno-tls-ca -o jsonpath='{.data.tls\.crt}' | \
2  step base64 -d | step certificate inspect --short
3X.509v3 Root CA Certificate (RSA 2048) [Serial: 0]
4  Subject:     *.kyverno.svc
5  Issuer:      *.kyverno.svc
6  Valid from:  2023-04-14T18:33:37Z
7          to:  2024-04-13T19:33:37Z

Get and decode the certificate used to register the webhooks with the Kubernetes API server (assumes at least one validate policy is installed) and see the same CA root certificate is in use.

1$ kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | \
2  step base64 -d | step certificate inspect --short
3X.509v3 Root CA Certificate (RSA 2048) [Serial: 0]
4  Subject:     *.kyverno.svc
5  Issuer:      *.kyverno.svc
6  Valid from:  2023-04-14T18:33:37Z
7          to:  2024-04-13T19:33:37Z

Certificates rotation

By default, Kyverno will generate and manage certificates. CA certificates validity is one year and TLS certificates validity is 150 days.

At a minimum, managed certificates are checked for validity every 12 hours. Additionally, validity checks are performed when events occur on secrets containing the managed certificates.

The renewal process runs as follows:

1. Remove expired certificates contained in the secret
2. Check if remaining certificates will become invalid in less than 60 hours
3. If needed, generate a new certificate with the validity documented above
4. The new certificates is added to the underlying secret along with current certificatess that are still valid
5. Reconfigure webhooks with the new certificates bundle
6. Update the Kyverno server to use the new certificate

Basically, certificates will be renewed approximately 60 hours before expiry.

Custom certificates

You can install your own CA-signed certificate, or generate a self-signed CA and use it to sign certificates for admission and cleanup controllers. Once you have a CA and X.509 certificate-key pairs, you can install these as Kubernetes Secrets in your cluster. If Kyverno finds these Secrets, it uses them, otherwise it will fall back on the default certificate management method. When you bring your own certificates, it is your responsibility to manage the regeneration/rotation process. Only RSA is supported for the CA and leaf certificates.

Generate a self-signed CA and signed certificate-key pairs

If you already have a CA and a signed certificate, you can directly proceed to Step 2.

Below is a process which shows how to create a self-signed root CA, and generate a signed certificates and keys using step CLI:

1. Create a self-signed CA

1step certificate create kyverno-ca rootCA.crt rootCA.key --profile root-ca --insecure --no-password --kty=RSA

2. Generate leaf certificates with a five-year expiration

 1step certificate create kyverno-svc tls.crt tls.key --profile leaf \
 2            --ca rootCA.crt --ca-key rootCA.key \
 3            --san kyverno-svc --san kyverno-svc.kyverno --san kyverno-svc.kyverno.svc --not-after 43200h \
 4            --insecure --no-password --kty=RSA
 5
 6
 7step certificate create kyverno-cleanup-controller cleanup-tls.crt cleanup-tls.key --profile leaf \
 8            --ca rootCA.crt --ca-key rootCA.key \
 9            --san kyverno-cleanup-controller --san kyverno-cleanup-controller.kyverno --san kyverno-cleanup-controller.kyverno.svc --not-after 43200h \
10            --insecure --no-password --kty=RSA

3. Verify the contents of the certificate

1step certificate inspect tls.crt --short

The certificate must contain the SAN information in the X509v3 extensions section:

1X509v3 extensions:
2    X509v3 Subject Alternative Name:
3        DNS:kyverno-svc, DNS:kyverno-svc.kyverno, DNS:kyverno-svc.kyverno.svc

Configure Secrets for the CA and TLS certificate-key pair

You can now use the following files to create Secrets:

• rootCA.crt
• tls.crt
• tls.key
• cleanup-tls.crt
• cleanup-tls.key

To create the required Secrets, use the following commands (do not change the Secret names):

1kubectl create ns <namespace>
2
3kubectl create secret tls kyverno-svc.kyverno.svc.kyverno-tls-pair --cert=tls.crt --key=tls.key -n <namespace>
4kubectl create secret generic kyverno-svc.kyverno.svc.kyverno-tls-ca --from-file=rootCA.crt -n <namespace>
5
6kubectl create secret tls kyverno-cleanup-controller.kyverno.svc.kyverno-tls-pair --cert=cleanup-tls.crt --key=cleanup-tls.key -n <namespace>
7kubectl create secret generic kyverno-cleanup-controller.kyverno.svc.kyverno-tls-ca --from-file=rootCA.crt -n <namespace>

Kyverno uses Secrets created above to setup TLS communication with the Kubernetes API server and specify the CA bundle to be used to validate the webhook server’s certificate in the admission and cleanup webhook configurations.

Install Kyverno

You can now install Kyverno by selecting one of the available methods from the installation section.

Roles and Permissions

Kyverno creates several Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings some of which may need to be customized depending on additional functionality required. To view all ClusterRoles and Roles associated with Kyverno, use the command kubectl get clusterroles,roles -A | grep kyverno.

Roles

Kyverno creates the following Roles in its Namespace, one per controller type:

• kyverno:admission-controller
  • create, delete, get, patch, and update Leases to handle high availability configurations.
  • get, list, and watch Deployments so it can manage the Kyverno Deployment itself.
  • get, list, watch, create, and update Secrets to manage certificates used for webhook management.
  • get, list, and watch ConfigMaps for configuration changes.
• kyverno:reports-controller
  • get, list, and watch ConfigMaps for configuration changes.
  • create, delete, get, patch, and update Leases to handle high availability configurations.
• kyverno:background-controller
  • get, list, and watch ConfigMaps for configuration changes.
  • create, delete, get, patch, and update Leases to handle high availability configurations.
• kyverno:cleanup-controller
  • get, list, watch, create, and update Secrets to manage certificates used for webhook management.
  • get, list, and watch ConfigMaps for configuration changes.
  • create, delete, get, patch, and update Leases to handle high availability configurations.

ClusterRoles

Kyverno uses aggregated ClusterRoles to search for and combine ClusterRoles which apply to Kyverno. Each controller has its own set of ClusterRoles. Those ending in core are the aggregate ClusterRoles which are then aggregated by the top-level role without the core suffix.

The following ClusterRoles provide Kyverno with permissions to policies and other Kubernetes resources across all Namespaces.

• kyverno:admission-controller:core: aggregate ClusterRole for the admission controller
• kyverno:admission-controller: aggregated (top-level) ClusterRole for the admission controller
• kyverno:reports-controller:core: aggregate ClusterRole for the reports controller
• kyverno:reports-controller: aggregated (top-level) ClusterRole for the reports controller
• kyverno:background-controller:core: aggregate ClusterRole for the background controller
• kyverno:background-controller: aggregated (top-level) ClusterRole for the background controller
• kyverno:cleanup-controller:core: aggregate ClusterRole for the cleanup controller
• kyverno:cleanup-controller: aggregated (top-level) ClusterRole for the cleanup controller
• kyverno-cleanup-jobs: used by the helper CronJob to periodically remove excessive/stale admission reports if found
• kyverno:rbac:admin:policies: aggregates to admin the ability to fully manage Kyverno policies
• kyverno:rbac:admin:policyreports: aggregates to admin the ability to fully manage Policy Reports
• kyverno:rbac:admin:reports: aggregates to admin the ability to fully manage intermediary admission and background reports
• kyverno:rbac:admin:updaterequests: aggregates to admin the ability to fully manage UpdateRequests, intermediary resource for generate rules
• kyverno:rbac:view:policies: aggregates to view the ability to view Kyverno policies
• kyverno:rbac:view:policyreports: aggregates to view the ability to view Policy Reports
• kyverno:rbac:view:reports: aggregates to view the ability to view intermediary admission and background reports
• kyverno:rbac:view:updaterequests: aggregates to view the ability to view UpdateRequests, intermediary resource for generate rules

Customizing Permissions

Because the ClusterRoles used by Kyverno use the aggregation feature, extending the permission for Kyverno’s use in cases like mutate existing or generate rules or generating ValidatingAdmissionPolicies is a simple matter of creating one or more new ClusterRoles which use the appropriate labels. It is not necessary to modify any existing ClusterRoles created as part of the Kyverno installation. Doing so is not recommended as changes may be lost during an upgrade. Since there are multiple controllers each with their own ServiceAccount, granting Kyverno additional permissions involves identifying the correct controller and using the labels needed to aggregate to that ClusterRole.

For example, if a new Kyverno generate policy requires that Kyverno be able to create or modify Deployments, this is not a permission Kyverno has by default. Generate rules are handled by the background controller and so it will be necessary to create a new ClusterRole and assign it the aggregation labels specific to the background controller in order for those permissions to take effect.

This sample ClusterRole provides the Kyverno background controller additional permissions to create Deployments:

 1apiVersion: rbac.authorization.k8s.io/v1
 2kind: ClusterRole
 3metadata:
 4  labels:
 5    app.kubernetes.io/component: background-controller
 6    app.kubernetes.io/instance: kyverno
 7    app.kubernetes.io/part-of: kyverno
 8  name: kyverno:create-deployments
 9rules:
10- apiGroups:
11  - apps
12  resources:
13  - deployments
14  verbs:
15  - create

Once a supplemental ClusterRole has been created, get the top-level ClusterRole for that controller to ensure aggregation has occurred.

1kubectl get clusterrole kyverno:background-controller -o yaml

Generating Kubernetes ValidatingAdmissionPolicies and their bindings are handled by the admission controller and it will be necessary to grant the controller the required permissions to generate these types. In this scenario, a ClusterRole should be created and assigned the aggregation labels for the admission controller in order for those permissions to take effect.

This sample ClusterRole provides the Kyverno admission controller additional permissions to create ValidatingAdmissionPolicies and ValidatingAdmissionPolicyBindings:

 1apiVersion: rbac.authorization.k8s.io/v1
 2kind: ClusterRole
 3metadata:
 4  labels:
 5    app.kubernetes.io/component: admission-controller
 6    app.kubernetes.io/instance: kyverno
 7    app.kubernetes.io/part-of: kyverno
 8  name: kyverno:generate-validatingadmissionpolicy
 9rules:
10- apiGroups:
11  - admissionregistration.k8s.io
12  resources:
13  - validatingadmissionpolicies
14  - validatingadmissionpolicybindings
15  verbs:
16  - create
17  - update
18  - delete
19  - list

ConfigMap Keys

The following keys are used to control the behavior of Kyverno and must be set in the Kyverno ConfigMap. Kyverno watches for changes to this ConfigMap and will load any updates which occur.

1. defaultRegistry: sets the default image registry to use if one is not specified. Defaults to docker.io.
2. enableDefaultRegistryMutation: tells Kyverno whether it should update its internal context with the value of the defaultRegistry key when the condition is met. Defaults to "true".
3. excludeGroups: excludes the provided groups from any processing. Supports a comma-separated list of groups. Defaults to system:serviceaccounts:kube-system,system:nodes.
4. excludeUsernames: excludes user names from any processing. Supports the ! operator to negate an entry (ex., !john will include the username john if it was excluded via another parameter). Default is '!system:kube-scheduler'.
5. excludeRoles: list of Roles to exclude from processing. Default is undefined.
6. excludeClusterRoles: list of ClusterRoles to exclude from processing. Default is undefined.
7. generateSuccessEvents: specifies whether (true/false) to generate success events. Default is set to “false”.
8. matchConditions: uses CEL-based expressions in the webhook configuration to narrow which admission requests are forwarded to Kyverno. Requires Kubernetes 1.27+ with the AdmissionWebhookMatchConditions feature gate to be enabled.
9. resourceFilters: Kubernetes resources in the format “[kind,namespace,name]” where the policy is not evaluated by the admission webhook. For example –filterKind “[Deployment, kyverno, kyverno]” –filterKind “[Deployment, kyverno, kyverno],[Events, *, *]”. Note that resource filters do not apply to background scanning mode. See the Resource Filters section for more complete information.
10. webhooks: specifies the Namespace or object exclusion to configure in the webhooks managed by Kyverno. Default is '[{"namespaceSelector": {"matchExpressions": [{"key":"kubernetes.io/metadata.name","operator":"NotIn","values":["kyverno"]}]}}]'.
11. webhookAnnotations: instructs Kyverno to add annotations to its webhooks for AKS support. Default is undefined. See the AKS notes section for details.
12. webhookLabels: instructs Kyverno to add labels to its webhooks. Default is undefined.

Container Flags

The following flags can be used to control the advanced behavior of the various Kyverno controllers and should be set on the main container in the form of arguments. All container flags can be prefaced with one or two dashes (ex., -admissionReports or --admissionReports). The controller(s) to which a given flag is applicable is shown in parenthesis.

• A: Admission controller
• B: Background controller
• C: Cleanup controller
• R: Reports controller

Policy Report access

During a Kyverno installation, a ClusterRole named kyverno:rbac:admin:policyreports is created which has permissions to perform all operations the two main Policy Report custom resources, policyreport and clusterpolicyreport. To grant access to a Namespace admin, configure the following YAML manifest according to your needs then apply to the cluster.

• Replace metadata.namespace with the Namespace where this should be bound.
• Configure the appropriate subjects fields to bind the ClusterRole to the admin’s ID.

 1apiVersion: rbac.authorization.k8s.io/v1
 2kind: RoleBinding
 3metadata:
 4  name: admin-policyreports
 5  namespace: default
 6roleRef:
 7  apiGroup: rbac.authorization.k8s.io
 8  kind: ClusterRole
 9  name: kyverno:rbac:admin:policyreports
10subjects:
11- apiGroup: rbac.authorization.k8s.io
12  kind: User
13  name: mynsuser
14- apiGroup: rbac.authorization.k8s.io
15  kind: Group
16  name: mynsgroup
17# - kind: ServiceAccount
18#   name: default
19#   namespace: default

Webhooks

Kyverno requires a few different webhooks to function. The main resource validating and mutating webhooks, those responsible for directing the Kubernetes API server to send the resources which are the subject of policies, are registered and managed dynamically by Kyverno based on the configured policies. With Kyverno-managed webhooks, Kyverno only receives admission requests for the matching resources defined in the policies, thereby preventing unnecessary admission requests being forwarded to Kyverno. For example, after an installation of Kyverno and before installing any policies, the ValidatingWebhookConfiguration named kyverno-resource-validating-webhook-cfg will be effectively empty. Creation of the first policy which matches on Pod will result in Kyverno configuring its webhook with one rule which instructs the API server to send all Pods to Kyverno. As resources have different scopes (Namespaced vs Cluster) the webhook rule scope is defined by the resource type itself. For Cluster scoped resources the default webhook rule scope * will be used. For Namespaced resources, the rule scope will be set to Namespaced. In case of wildcard * resources are used in a policy, the scope is derived from the policy type (ClusterPolicy will result in * and Policy will result in Namespaced rule scope). The matching of webhooks in general can be further restricted on compatible versions of Kubernetes (v1.27+) by using the matchConditions ConfigMap setting.

The dynamic management of resource webhooks is enabled by default but can be turned off by the flag --autoUpdateWebhooks=false. If disabled, Kyverno creates the default webhook configurations that forward admission requests for all resources and with FailurePolicy set to Ignore. In the majority of cases, these dynamically-managed webhooks should not be disabled. Understand that using statically-configured webhooks instead means that you, the operator, are now responsible for their configuration and, if left at their default, will result in the Kubernetes API server sending every type of resource (including subresources like /status) creates, updates, deletes, and connect operations. This will dramatically increase the processing required by Kyverno, even if few policies exist.

The failure policy of a webhook controls what happens when a request either takes too long to complete or encounters a failure. The default of Fail indicates the request will fail. A value of Ignore means the request will be allowed regardless of its status. Kyverno by default configures its webhooks in a mode of Fail to be more secure by default. This can be changed with a global flag --forceFailurePolicyIgnore which forces Ignore mode. Additionally, the failurePolicy and webhookTimeoutSeconds policy configuration options allow granular control of webhook settings on a per-policy basis.

Namespace Selectors

It is possible to instruct the API server to not send AdmissionReview requests at all for certain Namespaces based on labels. Kyverno can filter on these Namespaces using a namespaceSelector object in the webhooks key of the ConfigMap. For example, in the below snippet, a new object has been added with a namespaceSelector object which will filter on Namespaces with the label kubernetes.io/metadata.name=kyverno. The effect this will produce is the Kubernetes API server will only send AdmissionReview requests for resources in Namespaces except those labeled with kubernetes.io/metadata.name equals kyverno. The webhooks key only accepts as its value a JSON-formatted namespaceSelector object. Note that when installing Kyverno via the Helm chart and setting Namespace exclusions, it will cause the webhooks object to be automatically created in the Kyverno ConfigMap. The Kyverno and kube-system Namespaces are excluded by default.

1apiVersion: v1
2data:
3  webhooks: '[{"namespaceSelector":{"matchExpressions":[{"key":"kubernetes.io/metadata.name","operator":"NotIn","values":["kube-system"]},{"key":"kubernetes.io/metadata.name","operator":"NotIn","values":["kyverno"]}]}}]'
4kind: ConfigMap
5metadata:
6  name: kyverno
7  namespace: kyverno

Match Conditions

In addition to filtering out AdmissionReview requests by namespace selectors and Kyverno Configmap based match conditions, you can define match conditions in Kyverno policies if you need fine-grained request filtering. These match conditions will be used for registering main resource validating and mutating webhooks that are managed by Kyverno. Match conditions are CEL expressions. All match conditions must evaluate to true for a policy to be applied. Requires Kubernetes 1.27+ with the AdmissionWebhookMatchConditions feature gate to be enabled.

1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3spec:
4  webhookConfiguration:
5    matchConditions:
6    # Match requests made by non-node users.
7    - name: 'exclude-kubelet-requests'
8      expression: '!("system:nodes" in request.userInfo.groups)' 

Resource Filters

Resource filters are a way to instruct the Kyverno engine which AdmissionReview requests sent by the API server to disregard. It is a much more precise way of filtering through unnecessary requests so Kyverno processes only what it needs and can be thought of as a “last line of defense”. Resource filters are not the same as webhooks. A logical flow of a request from the API server to the Kyverno engine is shown below. In order for a given request to be fully processed by the Kyverno engine, it must pass webhooks and then all the resource filters.

graph LR
  id1["API Server"] --> id2["Webhook"]
  id2 --> id3["Resource Filters"]
  id3 --> id4["Kyverno engine"]

Resource filters, unlike webhooks, may effect more than just admission requests–they have the ability to effect policy reports in background scans as well. By default, however, the configured resource filters are not taken into account during background reporting scans. This behavior may be configured with the --skipResourceFilters container flag as described in the Container Flags section. Also note that resource filters do not effect mutations for existing resources.

The Kubernetes kinds that should be ignored by policies can be filtered out by modifying the value of data.resourceFilters in Kyverno’s ConfigMap stored in its Namespace. The default name of this ConfigMap is kyverno but can be changed by modifying the value of the environment variable INIT_CONFIG in the Kyverno deployment spec. data.resourceFilters must be a sequence of one or more [<Kind>,<Namespace>,<Name>] entries with * as a wildcard (*/* indicates all subresources within all kinds). Thus, an item [Secret,*,*] means that admissions of kind Secret in any Namespace and with any name will be ignored. Wildcards are also supported in each of these sequences. For example, the sequence [Pod,foo-system,redis*] filters out kind Pod in Namespace foo-system having names beginning with redis.

By default, a number of kinds are skipped in the default configuration including Nodes, Events, APIService, SubjectAccessReview, and more. Filters may be added and removed according to your needs.

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: kyverno
5  namespace: kyverno
6data:
7  # resource types to be skipped by Kyverno
8  resourceFilters: '[*/*,kyverno,*] [Event,*,*] [*/*,kube-system,*] [*/*,kube-public,*]
9    [*/*,kube-node-lease,*] [Node,*,*] [Node/*,*,*] <snip>'

Changes to the ConfigMap are read dynamically during runtime. Resource filters may also be configured at installation time via a Helm value.

Proxy

Kyverno supports using of a proxy by setting the standard environment variables, HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. These variables must be defined for the main container of a given Kyverno Deployment.