Mutate Rules

A mutate rule can be used to modify matching resources and is written as either a RFC 6902 JSON Patch or a strategic merge patch.

By using a patch in the JSONPatch - RFC 6902 format, you can make precise changes to the resource being created. A strategic merge patch is useful for controlling merge behaviors on elements with lists. Regardless of the method, a mutate rule is used when an object needs to be modified in a given way.

Resource mutation occurs before validation, so the validation rules should not contradict the changes performed by the mutation section. To mutate existing resources in addition to those subject to AdmissionReview requests, use mutateExisting policies.

This policy sets the imagePullPolicy to IfNotPresent if the image tag is latest:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: set-image-pull-policy
 5spec:
 6  rules:
 7    - name: set-image-pull-policy
 8      match:
 9        any:
10        - resources:
11            kinds:
12            - Pod
13      mutate:
14        patchStrategicMerge:
15          spec:
16            containers:
17              # match images which end with :latest
18              - (image): "*:latest"
19                # set the imagePullPolicy to "IfNotPresent"
20                imagePullPolicy: "IfNotPresent"

RFC 6902 JSONPatch

A JSON Patch, implemented as a mutation method called patchesJson6902, provides a precise way to mutate resources and supports the following operations (in the op field):

• add
• replace
• remove

The patchesJson6902 method can be useful when a specific mutation is needed which cannot be performed by patchesStrategicMerge. For example, when needing to mutate a specific object within an array, the index can be specified as part of a patchesJson6902 mutation rule.

One distinction between this and other mutation methods is that patchesJson6902 does not support the use of conditional anchors. Use preconditions instead. Also, mutations using patchesJson6902 to Pods directly are not converted to higher-level controllers such as Deployments and StatefulSets through the use of the auto-gen feature. Therefore, when writing such mutation rules for Pods, it may be necessary to create multiple rules to cover all relevant Pod controllers.

This patch policy adds, or replaces, entries in a ConfigMap with the name config-game in any Namespace.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: policy-patch-cm
 5spec:
 6  rules:
 7    - name: pCM1
 8      match:
 9        any:
10        - resources:
11            names:
12              - config-game
13            kinds:
14              - ConfigMap
15      mutate:
16        patchesJson6902: |-
17          - path: "/data/ship.properties"
18            op: add
19            value: |
20              type=starship
21              owner=utany.corp
22          - path: "/data/newKey1"
23            op: add
24            value: newValue1          

If your ConfigMap has empty data, the following policy adds an entry to config-game.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: policy-add-cm
 5spec:
 6  rules:
 7    - name: pCM1
 8      match:
 9        any:
10        - resources:
11            names:
12              - config-game
13            kinds:
14            - ConfigMap
15      mutate:
16        patchesJson6902: |-
17          - path: "/data"
18            op: add
19            value:
20              ship.properties: '{"type": "starship", "owner": "utany.corp"}'          

This is an example of a patch that removes a label from a Secret:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: policy-remove-label
 5spec:
 6  rules:
 7    - name: remove-unwanted-label
 8      match:
 9        any:
10        - resources:
11            kinds:
12            - Secret
13      mutate:
14        patchesJson6902: |-
15          - path: "/metadata/labels/purpose"
16            op: remove          

This policy rule adds elements to a list. In this case, it adds a new busybox container and a command. Note that because the path statement is a precise schema element, this will only work on a direct Pod and not higher-level objects such as Deployments. Testing the below policy requires setting spec.automountServiceAccountToken: false in a Pod.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: insert-container
 5spec:
 6  rules:
 7  - name: insert-container
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Pod
13    mutate:
14      patchesJson6902: |-
15        - op: add
16          path: "/spec/containers/1"
17          value:
18            name: busybox
19            image: busybox:latest
20        - op: add
21          path: "/spec/containers/1/command"
22          value:
23          - ls        

When needing to append an object to an array of objects, for example in pod.spec.tolerations, use a dash (-) at the end of the path.

1mutate:
2  patchesJson6902: |-
3    - op: add
4      path: "/spec/tolerations/-"
5      value:
6        key: networkzone
7        operator: Equal
8        value: dmz
9        effect: NoSchedule    

JSON Patch uses JSON Pointer to reference keys, and keys with tilde (~) and forward slash (/) characters need to be escaped with ~0 and ~1, respectively. For example, the following adds an annotation with the key of config.linkerd.io/skip-outbound-ports with the value of "8200".

1- op: add
2  path: /spec/template/metadata/annotations/config.linkerd.io~1skip-outbound-ports
3  value: "8200"

Some other capabilities of the patchesJson6902 method include:

• Adding non-existent paths
• Adding non-existent arrays
• Adding an element to the end of an array (use negative index -1)

Strategic Merge Patch

The kubectl command uses a strategic merge patch with special directives to control element merge behaviors. The patchStrategicMerge overlay resolves to a partial resource definition.

This policy adds a new container to the Pod, sets the imagePullPolicy, adds a command, and sets a label with the key of name and value set to the name of the Pod from AdmissionReview data. Once again, the overlay in this case names a specific schema path which is relevant only to a Pod and not higher-level resources like a Deployment.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: strategic-merge-patch
 5spec:
 6  rules:
 7  - name: set-image-pull-policy-add-command
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Pod
13    mutate:
14      patchStrategicMerge:
15        metadata:
16          labels:
17            name: "{{request.object.metadata.name}}"
18        spec:
19          containers:
20            - name: nginx
21              image: nginx:latest
22              imagePullPolicy: Never
23              command:
24              - ls

Note that when using patchStrategicMerge to mutate the pod.spec.containers[] array, the name key must be specified as a conditional anchor (i.e., (name): "*") in order for the merge to occur on other fields.

Mutate rules written with this style, if they match exclusively on a Pod, are subject to auto-generation rules for Pod controllers.

Conditional logic using anchors

Like with validate rules, conditional anchors are supported on mutate rules. Refer to the anchors section for more general information on conditionals.

An anchor field, marked by parentheses and an optional preceding character, allows conditional processing for mutations.

The mutate overlay rules support three types of anchors:

The anchors values support wildcards:

1. * - matches zero or more alphanumeric characters
2. ? - matches a single alphanumeric character

Conditional anchors are only supported with the patchStrategicMerge mutation method.

Conditional anchor

Conditional anchors are used to control when a mutation should occur by using an “if” logical evaluation, for example if a field is not already present or if a value is not the desired value. Contrast this with the example from the strategic merge patch section above where the mutation is applied regardless of any conditions.

A conditional anchor evaluates to true if the anchor tag exists and if the value matches the specified value. Processing stops if a tag does not exist or when the value does not match. Once processing stops, any child elements or any remaining siblings in a list will not be processed.

For example, this will add or replace the value 6443 for the port field, for all ports with a name value that starts with “secure”.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: policy-set-port
 5spec:
 6  rules:
 7  - name: set-port
 8    match:
 9      any:
10      - resources:
11          kinds :
12            - Endpoints
13    mutate:
14      patchStrategicMerge:
15        subsets:
16        - ports:
17          - (name): "secure*"
18            port: 6443

If the anchor tag value is an object or array, the entire object or array must match. In other words, the entire object or array becomes part of the “if” clause. Nested conditional anchor tags are not supported.

Add if not present anchor

One of the most common scenarios when mutating resources is to add/change a field to a specified value only if it is not already present. This is done by using the add anchor (short for “add if not present” anchor) with the notation +(...) for the tag.

Typically, every non-anchor tag-value is applied as part of the mutation. If the add anchor is set on a tag, the tag and value are only applied if they do not exist in the resource. If the tag is already present then the mutation is not applied. This anchor should only be used on lists/arrays if inside a foreach loop as it is not intended to be an iterator. See the foreach section below for an example.

In this example, the label lfx-mentorship will be assigned to ConfigMaps with a value of kyverno but only if the label is not already present.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: lfx-add-labels
 5spec:
 6  rules:
 7  - name: lfx-mentorship
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - ConfigMap
13    mutate:
14      patchStrategicMerge:
15        metadata:
16          labels:
17            +(lfx-mentorship): kyverno

Multiple of these add anchors may also be combined in the same rule definition when they are used as siblings. Take this example which adds fields to the securityContext of a Pod if they are not found.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: add-default-securitycontext
 5spec:
 6  rules:
 7  - name: add-default-securitycontext
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Pod
13    mutate:
14      patchStrategicMerge:
15        spec:
16          securityContext:
17            +(runAsNonRoot): true
18            +(runAsUser): 1000
19            +(runAsGroup): 3000
20            +(fsGroup): 2000

Global Anchor

Similar to validate rules, mutate rules can use the global anchor. When a global anchor is used, the condition inside the anchor, when true, means the rest of the pattern will be applied regardless of how it may relate to the global anchor.

For example, the below policy will add an imagePullSecret called my-secret to any Pod if it has a container image beginning with corp.reg.com.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: add-imagepullsecrets
 5spec:
 6  rules:
 7  - name: add-imagepullsecret
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Pod
13    mutate:
14      patchStrategicMerge:
15        spec:
16          containers:
17          - <(image): "corp.reg.com/*"
18          imagePullSecrets:
19          - name: my-secret

The below Pod meets this criteria and so the imagePullSecret called my-secret is added.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: static-web
 5  labels:
 6    role: myrole
 7spec:
 8  containers:
 9    - name: web
10      image: corp.reg.com/nginx
11      imagePullSecrets:
12      - name: my-secret
13      ports:
14        - name: web
15          containerPort: 80
16          protocol: TCP

Processing flow and combinations

The anchor processing behavior for mutate conditions is as follows:

1. First, all conditional anchors are processed. Processing stops when the first conditional anchor returns a false. Mutation proceeds only if all conditional anchors return a true. Note that for conditional anchor tags with complex (object or array) values, the entire value (child) object is treated as part of the condition as explained above.

2. Next, all tag-values without anchors and all add anchor tags are processed to apply the mutation.

Anchors may also be combined in mutate rules for more control and flexibility.

For example, this policy combines the global and add anchors to mutate pods containing an emptyDir volume to add the cluster-autoscaler.kubernetes.io/safe-to-evict annotation if it is not specified.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: add-safe-to-evict
 5  annotations:
 6    pod-policies.kyverno.io/autogen-controllers: none
 7spec:
 8  rules:
 9  - name: annotate-empty-dir
10    match:
11      any:
12      - resources:
13          kinds:
14          - Pod
15    mutate:
16      patchStrategicMerge:
17        metadata:
18          annotations:
19            +(cluster-autoscaler.kubernetes.io/safe-to-evict): true
20        spec:
21          volumes:
22          - <(emptyDir): {}

Mutate Existing resources

In addition to standard mutations, Kyverno also supports mutation on existing resources with patchStrategicMerge and patchesJson6902. Unlike regular mutate policies that are applied through the AdmissionReview process, mutate existing policies are applied in the background (via the background controller) which update existing resources in the cluster. These “mutate existing” policies, like traditional mutate policies, are still triggered via the AdmissionReview process but apply to existing resources. This decoupling also allows triggering on one resource and mutating a totally different one. They may also optionally be configured to apply upon updates to the policy itself. This has two important implications:

1. Mutation for existing resources is an asynchronous process. This means there will be a variable amount of delay between the period where the trigger was observed and the existing resource was mutated.
2. Custom permissions are almost always required. Because these mutations occur on existing resources and not an AdmissionReview (which does not yet exist), Kyverno may need additional permissions which it does not have by default. See the section on customizing permissions on how to grant additional permission to the Kyverno background controller’s ServiceAccount to determine, prior to installing mutate existing rules, if additional permissions are required. Kyverno will perform these permissions checks at the time a mutate existing policy is installed. Missing or incorrect permissions will result in failure to create the policy.

To define a “mutate existing” policy, trigger resources need to be specified in the match block. The target resources–resources targeted for mutation–are specified in each mutate rule under mutate.targets. Mutate existing rules differ from standard mutate rules when these targets are defined. Note that all target resources within a single rule must share the same definition schema. For example, a mutate existing rule fails if this rule mutates both Pod and Deployment as they do not share the same OpenAPI V3 schema (except metadata).

Because the match and mutate.targets[] stanzas have two separate scopes, when wishing to match on and mutate the same kind of resources any exclusionary conditions must be placed in the correct scope. Match filter criteria do not implicitly function as the input filter for target selection. For example, wishing to match on and mutate existing Ingress resources which have the annotation corp.org/facing=internal should at least have the annotation as a selection criteria in the targets section and may use either anchors or preconditions as described further below. Placing this annotation in the match clause will only result in Kyverno triggering upon those resources and not necessarily mutating them.

This policy, which matches when the trigger resource named dictionary-1 in the staging Namespace changes, writes a label foo=bar to the target resource named secret-1 also in the staging Namespace.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: mutate-existing-secret
 5spec:
 6  rules:
 7    - name: mutate-secret-on-configmap-event
 8      match:
 9        any:
10        - resources:
11            kinds:
12            - ConfigMap
13            names:
14            - dictionary-1
15            namespaces:
16            - staging
17      mutate:
18        targets:
19        - apiVersion: v1
20          kind: Secret
21          name: secret-1
22          namespace: "{{ request.object.metadata.namespace }}"
23        patchStrategicMerge:
24          metadata:
25            labels:
26              foo: bar

By default, the above policy will not be applied when it is installed. This behavior can be configured via mutateExistingOnPolicyUpdate attribute. If you set mutateExistingOnPolicyUpdate to true, Kyverno will mutate the existing secret on policy CREATE and UPDATE AdmissionReview events. When mutateExistingOnPolicyUpdate is specified as true, mutate.targets must be specified.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: mutate-existing-secret
 5spec:
 6  rules:
 7    - name: mutate-secret-on-configmap-event
 8      match:
 9        any:
10          - resources:
11              kinds:
12                - ConfigMap
13              names:
14                - dictionary-1
15              namespaces:
16                - staging
17      mutate:
18        mutateExistingOnPolicyUpdate: true
19        # ...
20        targets:
21          - apiVersion: v1
22            kind: Secret
23            name: secret-1
24            namespace: "{{ request.object.metadata.namespace }}"
25        # ...

When defining a list of targets[], the fields name and namespace are not strictly required but encouraged. If omitted, it implies a wildcard ("*") for the omitted field which can have unintended impact on other resources.

Target resources can also be selected using label selectors. The below policy example shows a policy that matches ConfigMaps with a label should-match=yes on Secret events.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: mutate-existing-configmap
 5spec:
 6  rules:
 7    - name: mutate-configmap-on-secret-event
 8      match:
 9        any:
10        - resources:
11            kinds:
12            - Secret
13      mutate:
14        targets:
15        - apiVersion: v1
16          kind: ConfigMap
17          selector:
18            matchLabels:
19              should-match: 'yes'
20        patchStrategicMerge:
21          metadata:
22            labels:
23              foo: bar

In order to more precisely control the target resources, mutate existing rules support both context variables and preconditions. Preconditions which occur inside the targets[] array must use the target prefix as described below.

This sample below illustrates how to combine preconditions and conditional anchors within targets[] to precisely select the desired existing resources for mutation. This policy restarts existing Deployments if they are consuming a Secret that has been updated assigned label kyverno.io/watch: "true" AND have a name beginning with testing-.

 1apiVersion: kyverno.io/v2beta1
 2kind: ClusterPolicy
 3metadata:
 4  name: refresh-env-var-in-pods
 5spec:
 6  rules:
 7  - name: refresh-from-secret-env
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Secret
13          selector:
14            matchLabels:
15              kyverno.io/watch: "true"
16          operations:
17          - UPDATE
18    mutate:
19      mutateExistingOnPolicyUpdate: false
20      targets:
21        - apiVersion: apps/v1
22          kind: Deployment
23          namespace: "{{request.namespace}}"
24          preconditions:
25            all:
26            - key: "{{target.metadata.name}}"
27              operator: Equals
28              value: testing-*
29      patchStrategicMerge:
30        spec:
31          template:
32            metadata:
33              annotations:
34                corp.org/random: "{{ random('[0-9a-z]{8}') }}"
35            spec:
36              containers:
37              - env:
38                - valueFrom:
39                    secretKeyRef:
40                      <(name): "{{ request.object.metadata.name }}"

Mutate existing rules are force reconciled every hour by default regardless of the mutateExistingOnPolicyUpdate value. The reconciliation interval can be customized through use of the environment variable BACKGROUND_SCAN_INTERVAL set on the background controller.

Starting from kyverno v1.11.2, mutate existing rules that trigger on deletion of a resource will be skipped unless explicitly specified that the DELETE operation should match

For example,the following policy should add a label to a configmap when a deployment is created or updated

 1apiVersion: kyverno.io/v1
 2kind: Policy
 3metadata:
 4  name: mutate-configmap-on-undefined-deployment-operation
 5spec:
 6  background: false
 7  rules:
 8    - name: mutate-configmap-on-undefined-deployment-operation
 9      match:
10        all:
11          - resources:
12              kinds:
13                - Deployment
14      mutate:
15        targets:
16          - apiVersion: v1
17            kind: ConfigMap
18            name: example
19            namespace: example
20        patchesJson6902: |-
21          - path: "/metadata/labels/modified-by-kyverno"
22            op: add
23            value: "true"          

To have it also run the mutation when the deployment is deleted, the policy should be modified as such

 1apiVersion: kyverno.io/v1
 2kind: Policy
 3metadata:
 4  name: mutate-configmap-on-undefined-deployment-operation
 5spec:
 6  background: false
 7  rules:
 8    - name: mutate-configmap-on-undefined-deployment-operation
 9      match:
10        all:
11          - resources:
12              kinds:
13                - Deployment
14              operations:
15              # add other operations if needed
16              - DELETE
17      mutate:
18        targets:
19          - apiVersion: v1
20            kind: ConfigMap
21            name: example
22            namespace: example
23        patchesJson6902: |-
24          - path: "/metadata/labels/modified-by-kyverno"
25            op: add
26            value: "true"          

Variables Referencing Target Resources

To reference data in target resources, you can define the variable target followed by the path to the desired attribute. For example, using target.metadata.labels.env references the label env in the target resource.

This policy copies the ConfigMaps’ value target.data.key to their label with the key env.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: sync-cms
 5spec:
 6  rules:
 7  - name: concat-cm
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - ConfigMap
13          names:
14          - cmone
15          namespaces:
16          - foo
17    mutate:
18      mutateExistingOnPolicyUpdate: false
19      targets:
20        - apiVersion: v1
21          kind: ConfigMap
22          name: cmtwo
23          namespace: bar
24        - apiVersion: v1
25          kind: ConfigMap
26          name: cmthree
27          namespace: bar
28      patchesJson6902: |-
29        - op: add
30          path: "/metadata/labels/env"
31          value: "{{ target.data.key }}"          

The {{ @ }} special variable is added to reference the in-line value of the target resource.

This policy adds the value of keyone from the trigger ConfigMap named cmone in the foo Namespace as the prefix to target ConfigMaps in their data with keynew.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: sync-cms
 5spec:
 6  rules:
 7  - name: concat-cm
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - ConfigMap
13          names:
14          - cmone
15          namespaces:
16          - foo
17    mutate:
18      mutateExistingOnPolicyUpdate: false
19      targets:
20        - apiVersion: v1
21          kind: ConfigMap
22          name: cmtwo
23          namespace: bar
24        - apiVersion: v1
25          kind: ConfigMap
26          name: cmthree
27          namespace: bar
28      patchStrategicMerge:
29        data:
30          keynew: "{{request.object.data.keyone}}-{{@}}"

Once a mutate existing policy is applied successfully, there will be an event and an annotation added to the target resource:

 1$ kubectl describe deploy foobar
 2...
 3Events:
 4  Type     Reason             Age                From                   Message
 5  ----     ------             ----               ----                   -------
 6  Normal   PolicyApplied      29s (x2 over 31s)  kyverno-mutate         policy add-sec/add-sec-rule applied
 7
 8$ kubectl get deploy foobar -o yaml
 9apiVersion: apps/v1
10kind: Deployment
11metadata:
12  annotations:
13    ...
14    policies.kyverno.io/last-applied-patches: |
15      add-sec-rule.add-sec.kyverno.io: added /spec/template/spec/containers/0/securityContext

To troubleshoot policy application failure, inspect the UpdateRequest Custom Resource to get details. Successful UpdateRequests may be automatically cleaned up by Kyverno.

For example, if the corresponding permission is not granted to Kyverno, you should see a value of Failed in the updaterequest.status field, however a permission check is performed when a policy is installed.

$ kubectl get ur -n kyverno
NAME       POLICY    RULETYPE   RESOURCEKIND   RESOURCENAME   RESOURCENAMESPACE   STATUS   AGE
ur-swsdg   add-sec   mutate     Deployment     foobar         default             Failed   84s


$ kubectl describe ur ur-swsdg -n kyverno
Name:         ur-swsdg
Namespace:    kyverno
...
Status:
  Message:  deployments.apps "foobar" is forbidden: User "system:serviceaccount:kyverno:kyverno-service-account" cannot update resource "deployments" in API group "apps" in the namespace "default"
  State:    Failed

Mutate Existing Tips

Combine multiple generate existing rules into a single rule when they mutate the same resource. When these rules are combined, only a single mutation happens which is quicker and more efficient. Otherwise, by using separate rules, multiple UpdateRequests may be created as Kyverno attempts to reconcile changes from both rules simultaneously. This can take longer, require more processing resources, and in some extreme instances can fail.

Mutate Rule Ordering (Cascading)

In some cases, it might be desired to have multiple levels of mutation rules apply to incoming resources. The match statement in rule A would apply a mutation to the resource, and the result of that mutation would trigger a match statement in rule B that would apply a second mutation. In such cases, Kyverno can accommodate more complex mutation rules, however rule ordering matters to guarantee consistent results.

For example, assume you wished to assign a label to each incoming Pod describing the type of application it contained. For those with an image having the string either cassandra or mongo you wished to apply the label type=database. This could be done with the following sample policy.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: database-type-labeling
 5spec:
 6  rules:
 7    - name: assign-type-database
 8      match:
 9        any:
10        - resources:
11            kinds:
12            - Pod
13      mutate:
14        patchStrategicMerge:
15          metadata:
16            labels:
17              type: database
18          spec:
19            (containers):
20            - (image): "*cassandra* | *mongo*"

Also, assume that for certain application types a backup strategy needs to be defined. For those applications where type=database, this would be designated with an additional label with the key name of backup-needed and value of either yes or no. The label would only be added if not already specified since operators can choose if they want protection or not. This policy would be defined like the following.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: database-backup-labeling
 5spec:
 6  rules:
 7    - name: assign-backup-database
 8      match:
 9        any:
10        - resources:
11            kinds:
12              - Pod
13            selector:
14              matchLabels:
15                type: database
16      mutate:
17        patchStrategicMerge:
18          metadata:
19            labels:
20              +(backup-needed): "yes"

In such a case, Kyverno is able to perform cascading mutations whereby an incoming Pod that matched in the first rule and was mutated would potentially be further mutated by the second rule. In these cases, the rules must be ordered from top to bottom in the order of their dependencies and stored within the same policy. The resulting policy definition would look like the following:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: database-protection
 5spec:
 6  rules:
 7  - name: assign-type-database
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Pod
13    mutate:
14      patchStrategicMerge:
15        metadata:
16          labels:
17            type: database
18        spec:
19          (containers):
20          - (image): "*cassandra* | *mongo*"
21  - name: assign-backup-database
22    match:
23      any:
24      - resources:
25          kinds:
26          - Pod
27          selector:
28            matchLabels:
29              type: database
30    mutate:
31      patchStrategicMerge:
32        metadata:
33          labels:
34            +(backup-needed): "yes"

Test the cascading mutation policy by creating a Pod using the Cassandra image.

1$ kubectl run cassandra --image=cassandra:latest
2pod/cassandra created

Perform a get or describe on the Pod to see the result of the metadata.

1$ kubectl describe po cassandra
2Name:         cassandra
3Namespace:    default
4<snip>
5Labels:       backup-needed=yes
6              run=cassandra
7              type=database
8<snip>

As can be seen, both type=database and backup-needed=yes were applied according to the mutation rules.

Verify that applying your own backup-needed label with the value of no triggers the first mutation rule but not the second.

1$ kubectl run cassandra --image=cassandra:latest --labels backup-needed=no

Perform another get or describe to verify the backup-needed label was not altered by the mutation rule.

1$ kubectl describe po cassandra
2Name:         cassandra
3Namespace:    default
4<snip>
5Labels:       backup-needed=no
6              type=database
7<snip>

Combining Mutate and Generate

In some use cases, it may be necessary to perform a mutation as a follow-on action from a Kyverno generate rule. In these cases, you can combine generation with mutation to achieve the desired effect. Some use cases for this include mutating a cloned resource to add fields not present in the source and performing a mutation of an existing resource in response to a generate rule.

When combining “standard” mutation and generation behaviors, order the rules in a single policy such that the generate occurs first followed by the mutate.

In the below policy, a Secret named regcred is generated into a newly-created Namespace followed by the addition of the foo: bar label on the generated Secret.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: sync-secrets
 5spec:
 6  rules:
 7  - name: sync-image-pull-secret
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Namespace
13    generate:
14      apiVersion: v1
15      kind: Secret
16      name: regcred
17      namespace: "{{request.object.metadata.name}}"
18      synchronize: true
19      clone:
20        namespace: default
21        name: regcred
22  - name: mutate-secret
23    match:
24      any:
25      - resources:
26          kinds:
27          - Secret
28          names:
29          - regcred
30    mutate:
31      patchStrategicMerge:
32        metadata:
33          labels:
34            foo: bar

When combining a generation with a mutation on an existing resource, the Kyverno background controller must be instructed to not filter out the request from the previous rule. By default, these internal requests are disregarded as a form of loop protection. Set the rule-level field skipBackgroundRequests: false in such cases so the background controller processes the request from the generate rule before it. You will be responsible for ensuring your policy and rule combinations do not result in a loop. This step was not necessary in the previous use case because the admission controller was responsible for processing the “standard” mutate rule during admission review.

In the below policy, a ConfigMap is generated into a newly-created Namespace. A follow-on Kyverno mutate existing rule is used to mark the Namespace as “ready” via a label only once the resource has been successfully created. Note that the skipBackgroundRequests: false field is present on the mutate existing rule and not the generate rule.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: demo-cluster-policy
 5spec:
 6  rules:
 7  - name: demo-generate
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Namespace
13          operations:
14          - CREATE
15    generate:
16      apiVersion: v1
17      kind: ConfigMap
18      name: somecustomcm
19      namespace: "{{request.object.metadata.name}}"
20      synchronize: false
21      data:
22        metadata:
23          labels:
24            custom/related-namespace: "{{request.object.metadata.name}}"
25        data:
26          key: value
27  - name: demo-mutate-existing
28    skipBackgroundRequests: false
29    match:
30      all:
31        - resources:
32            kinds:
33              - ConfigMap
34            selector:
35              matchLabels:
36                custom/related-namespace: "?*"
37    mutate:
38      mutateExistingOnPolicyUpdate: false
39      targets:
40        - apiVersion: v1
41          kind: Namespace
42          name: '{{ request.object.metadata.labels."custom/related-namespace" }}'
43      patchStrategicMerge:
44        metadata:
45          labels:
46            custom/namespace-ready: "true"

foreach

A foreach declaration can contain multiple entries to process different sub-elements e.g. one to process a list of containers and another to process the list of initContainers in a Pod.

A foreach must contain a list attribute that defines the list of elements it processes and either a patchStrategicMerge or patchesJson6902 declaration. For example, iterating over the list of containers in a Pod is performed using this list declaration:

1list: request.object.spec.containers
2patchStrategicMerge:
3  spec:
4    containers:
5      ...

When a foreach is processed, the Kyverno engine will evaluate list as a JMESPath expression to retrieve zero or more sub-elements for further processing. The value of the list field may also resolve to a simple array of strings, for example as defined in a context variable. The value of the list field should not be enclosed in braces even though it is a JMESPath expression.

foreach statements may also declare an optional order field for controlling whether to iterate over the list results in ascending or descending order.

A variable element is added to the processing context on each iteration. This allows referencing data in the element using element.<name> where name is the attribute name. For example, using the list request.object.spec.containers when the request.object is a Pod allows referencing the container image as element.image within a foreach.

Each foreach declaration can optionally contain the following declarations:

• Context: to add additional external data only available per loop iteration.
• Preconditions: to control when a loop iteration is skipped.
• foreach: a nested foreach declaration described below.

For a patchesJson6902 type of foreach declaration, an additional variable called elementIndex is made available which allows the current index number to be referenced in a loop.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: foreach-json-patch
 5spec:
 6  rules:
 7    - name: add-security-context
 8      match:
 9        any:
10        - resources:
11            kinds:
12              - Pod
13            operations:
14              - CREATE
15      mutate:
16        foreach: 
17        - list: "request.object.spec.containers"
18          patchesJson6902: |-
19            - path: /spec/containers/{{elementIndex}}/securityContext
20              op: add
21              value:
22                runAsNonRoot: true            

For a complete example of the patchStrategicMerge method that mutates the image to prepend the address of a trusted registry, see below.

 1apiVersion : kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: prepend-registry
 5spec:
 6  background: false
 7  rules:
 8  - name: prepend-registry-containers
 9    match:
10      any:
11      - resources:
12          kinds:
13          - Pod
14          operations:
15          - CREATE
16          - UPDATE
17    mutate:
18      foreach:
19      - list: "request.object.spec.containers"
20        patchStrategicMerge:
21          spec:
22            containers:
23            - name: "{{ element.name }}"           
24              image: registry.io/{{ images.containers."{{element.name}}".name}}:{{images.containers."{{element.name}}".tag}}

Note that the patchStrategicMerge is applied to the request.object. Hence, the patch needs to begin with spec. Since container names may have dashes in them (which must be escaped), the {{element.name}} variable is specified in double quotes.

Loops which use the patchStrategicMerge method may also use conditional anchors as mentioned earlier. This policy uses the add anchor inside a loop to add the allowPrivilegeEscalation field with a value set to false but only if it isn’t present first. The conditional around the name field is required in this style of mutation.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: pss-migrate
 5spec:
 6  rules:
 7  - name: set-allowprivescalation
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Pod
13    mutate:
14      foreach:
15      - list: "request.object.spec.containers[]"
16        patchStrategicMerge:
17          spec:
18            containers:
19            - (name): "{{ element.name }}"
20              securityContext:
21                +(allowPrivilegeEscalation): false

As with conditional mutations outside the context of a loop, so too may multiple anchors be used inside, for example with this policy to add default resource requests to containers within a Pod if they are not specified.

 1apiVersion : kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: add-default-resources
 5spec:
 6  rules:
 7  - name: add-default-requests
 8    match:
 9      any:
10      - resources:
11          kinds:
12          - Pod
13    preconditions:
14      any:
15      - key: "{{request.operation || 'BACKGROUND'}}"
16        operator: AnyIn
17        value:
18        - CREATE
19        - UPDATE
20    mutate:
21      foreach:
22      - list: "request.object.spec.containers[]"
23        patchStrategicMerge:
24          spec:
25            containers:
26            - (name): "{{element.name}}"
27              resources:
28                requests:
29                  +(memory): "100Mi"
30                  +(cpu): "100m"

It is important to understand internally how Kyverno treats foreach rules. Some general statements to keep in mind:

• No internal patches are produced until all foreach iterations are complete.
• foreach supports multiple loops but will be processed in declaration order and not independently or in parallel. The results of the first loop will be available internally to subsequent loops.
• Cascading mutations as separate rules will be necessary for a mutation to be further mutated by other logic.
• Use of the order field is required in some situations, for example when removing elements from an array use Descending.

Nested foreach

The foreach object also supports nesting multiple foreach declarations to form loops within loops. This is especially useful when the mutations you need to perform are either replacements or removals as these require the use of JSON patches (patchesJson6902). When using nested loops, the special variable {{elementIndex}} requires a loop number to identify which element to process. Preconditions are supported only at the top-level loop and not per inner loop.

For example, consider a scenario in which you must replace all host names which end in old.com with new.com in an Ingress resource under the spec.tls[].hosts[] list. Because spec.tls[] is an array of objects, and hosts[] is an array of strings within each object, a foreach declaration must iterate over each object in the tls[] array and then internally loop over each host in the hosts[] array.

 1apiVersion: networking.k8s.io/v1
 2kind: Ingress
 3metadata:
 4  name: myingress
 5  labels:
 6    app: myingress
 7spec:
 8  rules:
 9  - host: myhost.corp.com
10    http:
11      paths:
12      - backend:
13          service: 
14            name: myservice
15            port: 
16              number: 8080
17        path: /
18        pathType: ImplementationSpecific
19  tls:
20  - hosts:
21    - foo.old.com
22    - bar.old.com
23    secretName: mytlscertsecret

This type of advanced mutation can be performed with nested foreach loops as shown below. Notice that in the JSON patch, the path value references the current index of tls[] as {{elementIndex0}} and the current index of hosts[] as {{elementIndex1}}. In the value field, the {{element}} variable still references the current value of the hosts[] array being processed.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: replace-image-registry
 5spec:
 6  background: false
 7  rules:
 8    - name: replace-dns-suffix
 9      match:
10        any:
11          - resources:
12              kinds:
13                - Ingress
14      mutate:
15        foreach:
16          - list: request.object.spec.tls[]
17            foreach:
18              - list: "element.hosts"
19                patchesJson6902: |-
20                  - path: /spec/tls/{{elementIndex0}}/hosts/{{elementIndex1}}
21                    op: replace
22                    value: "{{ replace_all('{{element}}', '.old.com', '.new.com') }}"                  

GitOps Considerations

It is very common to use Kyverno in a GitOps approach where policies and/or other resources, which may be affected by those policies, are deployed via a state held in git by a separate tool such as Argo CD or Flux. In the case of mutations, the classic problem which arises is Kyverno (or some other tool) mutates a resource which was created by a GitOps tool. The mutation changes the resource in a way which causes divergence from the state held in git. When this divergence is detected by the GitOps controller, said controller attempts to reconcile the resource to bring it back into alignment with the desired state held in git. This process continues in an endless loop and creates “fighting” between the mutating admission controller and the GitOps controller. This type of fighting is problematic as it increases churn in the cluster, increases resource consumption by the respective tools, and can lead to disruption if it occurs on a large scale.

While Kyverno can interoperate with GitOps solutions when mutate rules are used, there are a few considerations of which to be aware and some recommended configurations.

Flux

Flux uses server-side apply along with dry-run mode when calculating a diff to determine if the actual state has diverged from desired state. Because Kyverno supports mutations in dry-run mode, the resource returned to Flux in this mode already includes the result of any would-be mutations. As a result, Flux natively accommodates mutating admission controllers such as Kyverno usually without any modifications needed. It is not necessary to inform Flux of the nature and number of mutations to expect for a given resource.

However, as dry-run mode causes mutation webhooks to be invoked just as if not in dry-run mode, this produces additional load on Kyverno as it must perform the same mutations every time a diff is calculated. The Flux synchronization interval should be checked and balanced to ensure only the minimal amount of processing overhead is introduced.

ArgoCD

See the platform notes page for general recommendations with Argo CD first.

ArgoCD v2.10+

ArgoCD version 2.10 added support for sever side diff which leverage the Kubernetes Server Side Apply feature.

Enabling SSA based diffs, requires an annotation to be specified on the application or globally via the “argocd-cmd-params-cm” config map.

Here is a YAML fragment that shows the annotation in an ArgoCD Application resource:

1apiVersion: argoproj.io/v1alpha1
2kind: Application
3metadata:
4  annotations:
5    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true 
6
7    ...

This CNCF blog post provides a complete example.

ArgoCD v2.9 and below

In order to use older versions of Argo CD with Kyverno Mutate policy rules, it will require configuring the Application custom resource with one or more ignoreDifferences entries to instruct Argo CD to ignore the mutations created by Kyverno. Some of these options include jqPathExpressions, jsonPointers, and managedFieldsManagers. For example, if a Kyverno mutate rule is expected to add a label foo to all Deployments, the Argo CD Application may need a section as follows.

1ignoreDifferences:
2  - group: apps
3    kind: Deployment
4    jqPathExpressions:
5    - .metadata.labels.foo

It may also be helpful to configure a managedFieldsManagers with a value of kyverno in the list to instruct Argo CD to allow Kyverno to own the fields it is mutating.