-
Notifications
You must be signed in to change notification settings - Fork 456
container enrichment fixes and improvements #4276
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
container enrichment fixes and improvements #4276
Conversation
In case enrichment is requested on a non container cgroup, return the metadata struct with an empty container id instead. User is responsible for handling this "not-found" case by themselves. Apply this in locations where enrichment is called (pipeline and control plane).
Make the image info query in containerd enrichment more robust. Procedure now begins by first querying the containerd image service, and only then using the cri directly as a fallback. Additionally, fix a typo in the CRI query which appended the image name as its digest, even when found.
fc57be7
to
49079a0
Compare
@rscampos as we discussed, I assigned you for review. Tks. |
@rscampos v0.22.1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
I've tested at 3 diff env: GKE, EKS and Kind (dev env.)
On GKE is working by default with your PR:
{"timestamp":82281540079,"threadStartTime":-492716535350,"processorId":0,"processId":1,"cgroupId":20,"threadId":1,"parentProcessId":0,"hostProcessId":1,"hostThreadId":1,"hostParentProcessId":0,"userId":0,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"systemd","executable":{"path":""},"hostName":"gke-cluster-1-d","containerId":"","container":{},"kubernetes":{},"eventId":"2018","eventName":"container_create","matchedPolicies":["default-policy"],"argsNum":10,"returnValue":0,"syscall":"mkdir","stackAddresses":[0],"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":731681712,"processEntityId":731681712,"parentEntityId":4140133194,"args":[{"name":"runtime","type":"const char*","value":"containerd"},{"name":"container_id","type":"const char*","value":"803f0425bc3be4db695f4edfde8031154fe1c554ec3f7a42f9a1f5f9adf269bc"},{"name":"ctime","type":"unsigned long","value":1725997383092771811},{"name":"container_image","type":"const char*","value":"docker.io/library/ubuntu:latest"},{"name":"container_image_digest","type":"const char*","value":"sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab63ee"},{"name":"container_name","type":"const char*","value":"ubuntu"},{"name":"pod_name","type":"const char*","value":"ubuntu"},{"name":"pod_namespace","type":"const char*","value":"default"},{"name":"pod_uid","type":"const char*","value":"12852ab6-a6d0-4b1d-8eae-177c28cd79be"},{"name":"pod_sandbox","type":"bool","value":false}]}
I've tested on EKS but don't work by default. We need to mount cgroup as suggested by you. Otherwise on EKS isn't going to work. If cgroup is mounted, it works as the following logs:
{"timestamp":58798502401,"threadStartTime":58777251627,"processorId":1,"processId":78016,"cgroupId":1,"threadId":78018,"parentProcessId":77967,"hostProcessId":78016,"hostThreadId":78018,"hostParentProcessId":77967,"userId":0,"mountNamespace":4026531840,"pidNamespace":4026531836,"processName":"runc","executable":{"path":""},"hostName":"ip-172-31-28-51","containerId":"","container":{},"kubernetes":{},"eventId":"2018","eventName":"container_create","matchedPolicies":["default-policy"],"argsNum":10,"returnValue":0,"syscall":"mkdirat","stackAddresses":[0],"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":4225719277,"processEntityId":1290136627,"parentEntityId":1097452082,"args":[{"name":"runtime","type":"const char*","value":"containerd"},{"name":"container_id","type":"const char*","value":"bf731d343a12204147bcb0a20b72cb5f0baffa4701bb2672639925bba2add691"},{"name":"ctime","type":"unsigned long","value":1725992607203534471},{"name":"container_image","type":"const char*","value":"docker.io/library/ubuntu@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab63ee"},{"name":"container_image_digest","type":"const char*","value":"sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab63ee"},{"name":"container_name","type":"const char*","value":"ubuntu-container"},{"name":"pod_name","type":"const char*","value":"ubuntu-deployment-89664856f-xrfnj"},{"name":"pod_namespace","type":"const char*","value":"default"},{"name":"pod_uid","type":"const char*","value":"51aa0cc9-ca4d-4cc8-916e-158e11d4471f"},{"name":"pod_sandbox","type":"bool","value":false}]
On deploy/helm/tracee/templates/daemonset.yaml
need to mount:
volumeMounts:
- mountPath: /sys/fs/cgroup
name: sys-fs-cgroup
readOnly: true
volumes:
- name: sys-fs-cgroup
hostPath:
path: /sys/fs/cgroup
On dev env. using Kind, even with cgroup mount doesn't work.
Maybe we need another fix in order to work by default on EKS as you suggested.
1. Explain what the PR does
49079a0 feat(enrich): improve containerd image info enrich
cb72aed fix(enrich): silence noncontainer cgroup errors
2. Explain how to test it
Deploy tracee on a containerd emv
Enrichment should work as usual
Digests should appear correctly (
sha256:abcdef121324
)No error messages of no containerid should follow
3. Other comments
Fix #4257
Fix #3870