Skip to content

ClusterRole missing permissions to access 'openshift-kube-apiserver' namespace on EKS, causes 403 error events in EKS audit logs. #2672

@Parkalla

Description

@Parkalla

What steps did you take and what happened:

Description

We run an EKS Cluster version 1.31.
Trivy Operator is deployed via the Helm chart aquasecurity/trivy-operator version 0.29.1.

Our deployment uses the default RBAC settings which creates Roles, ClusterRoles, Bindings, etc. with the default rules.
Via targetNamespaces we have configured to scan all namespaces.

Issue

We observe many Kubernetes API client errors originating from the trivy-operator job.

  • Trivy Operator logs show no errors.
  • EKS audit logs show many events with 403 Forbidden errors like this:
    { "metadata": {}, "status": "Failure", "message": "namespaces \"openshift-kube-apiserver\" is forbidden: User \"system:serviceaccount:security:trivy-operator\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"openshift-kube-apiserver\"", "reason": "Forbidden", "details": { "name": "openshift-kube-apiserver", "kind": "namespaces" }, "code": 403 }
    What is interesting:
  • The namespace "openshift-kube-apiserver" appears in the logs.
  • This seems to be an OpenShift system namespace, but we run EKS clusters, not OpenShift.
  • The setting excludeNamespaces: "openshift-kube-apiserver" does not prevent the operator from attempting a GET on this namespace, it still tries to access it.

Workaround

We manually updated the default ClusterRole to add the following permissions:
- verbs: - get - list - watch apiGroups: - "" resources: - namespaces
After this change, the 403 errors stopped.

Questions

  • Is it a known issue that the default ClusterRole lacks permissions to list and watch namespaces?
  • Is there an intended way to customize or extend the default RBAC rules (besides disabling RBAC creation and managing it manually)?

What did you expect to happen:

  • The default ClusterRole created by the Helm chart should include sufficient permissions to access all namespaces when targetNamespaces: "" is configured for cluster-wide scanning.
  • The operator should fully respect the excludeNamespaces setting by not attempting any API calls against excluded namespaces.
  • We should not get numerous 403.

Anything else you would like to add:
I’m really wondering why we see the namespace openshift-kube-apiserver in the logs. This seems to be an OpenShift system namespace, but we run EKS clusters.
Besides that, we don’t see that namespace on the clusters at all.
Any insights on why the operator tries to check for it would be awesome, especially since a 403 points to the fact that it must exist to deny access.

Environment:

  • Trivy-Operator version (use trivy-operator version): Helm chart aquasecurity/trivy-operator version 0.29.1
  • Kubernetes version (use kubectl version): EKS 1.31
  • OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc):

Thank you for your help!

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions