Skip to content

Conversation

kolyshkin
Copy link
Contributor

@kolyshkin kolyshkin commented Aug 30, 2025

Error messages between runc and crun are not synchronized, and in some case exit codes can be different, too.

Commit dd1bcab ("CI: use local registry, part 2 of 3: fix tests") removed the special case handling for runc from the "podman run --seccomp-policy image (block all syscalls)" test case, and so it fails, for example, like this:

  Error: failed to connect to container's attach socket: /tmp/podman-e2e-2877753109/subtest-1698249469/p/root/overlay-containers/62585e98da7dc3fdb32d3b6de0980c762a8a6cde008ed35c68727fb97f5369c7/userdata/attach: no such file or directory
  [FAILED] Command exited with status 127 (expected 126)

or this:

  time="2025-08-29T17:16:52-07:00" level=error msg="cannot start a container that has stopped"
  Error: `/usr/bin/runc start 63ce789f7037d9545cde832d29343704cab842e7288046407d0efa347d5ecb77` failed: exit status 1
  [FAILED] Command exited 126 as expected, but did not emit 'OCI runtime error: runc: read from the init process'

(depending on runc version, phase of the moon etc.)

We can not reasonably expect a specific error message and exit code in such an unusual scenario, but let's try.

With this commit, the above test passes successfully on my machine.

Fixes: dd1bcab ("CI: use local registry, part 2 of 3: fix tests")
Reported-by: Yiqiao Pu [email protected]

Does this PR introduce a user-facing change?

None

@openshift-ci openshift-ci bot added the do-not-merge/release-note-label-needed Enforce release-note requirement, even if just None label Aug 30, 2025
Copy link
Member

@Luap99 Luap99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@openshift-ci openshift-ci bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. release-note-none and removed do-not-merge/release-note-label-needed Enforce release-note requirement, even if just None labels Aug 30, 2025
@ricardobranco777
Copy link
Contributor

ricardobranco777 commented Aug 30, 2025

It's missing the IsRemote() condition for runc. It works with make localintegration but with make remoteintegration I'm getting this:

https://openqa.opensuse.org/tests/5277949/file/podman_e2e-remoteintegration.txt

• [FAILED] [1.100 seconds]
Podman run
/root/podman/test/e2e/run_seccomp_test.go:29
  [It] podman run --seccomp-policy image (block all syscalls)
  /root/podman/test/e2e/run_seccomp_test.go:50

  Timeline >>
  > Enter [BeforeEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:117 @ 08/30/25 15:56:20.86
  Running: /usr/bin/podman --storage-opt overlay.imagestore=/tmp/podman-e2e-2726467530/imagecachedir --root /tmp/podman-e2e-2726467530/subtest-3296714577/p/root --runroot /tmp/podman-e2e-2726467530/subtest-3296714577/p/runroot --runtime runc --conmon /usr/bin/conmon --network-config-dir /etc/containers/networks --network-backend netavark --cgroup-manager systemd --tmpdir /tmp/podman-e2e-2726467530/subtest-3296714577/p --events-backend file --db-backend sqlite --storage-driver overlay system service --time 0 unix:///run/podman/podman-c92695a34aca65752ac88713b0a02271a4fd16ba94e31a009ed1f14b7bc040d5.sock
  < Exit [BeforeEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:117 @ 08/30/25 15:56:20.961 (101ms)
  time="2025-08-30T15:56:20-04:00" level=warning msg="IdleTracker: StateClosed transition by connection marked un-managed" X-Reference-Id=0xc000168008
  > Enter [It] podman run --seccomp-policy image (block all syscalls) - /root/podman/test/e2e/run_seccomp_test.go:50 @ 08/30/25 15:56:20.962
  Running: /usr/bin/podman-remote --remote --url unix:///run/podman/podman-c92695a34aca65752ac88713b0a02271a4fd16ba94e31a009ed1f14b7bc040d5.sock run --name temp-working-container quay.io/libpod/testimage:20241011 true
  Running: /usr/bin/podman-remote --remote --url unix:///run/podman/podman-c92695a34aca65752ac88713b0a02271a4fd16ba94e31a009ed1f14b7bc040d5.sock commit -q --change LABEL io.containers.seccomp.profile='{"defaultAction":"SCMP_ACT_ERRNO"}' temp-working-container workingimage
  Getting image source signatures
  Copying blob sha256:b66a884aaf08f1c410c61682a7072d68a0d837ba8dc16ff13b9509bdceb32fd2
  Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
  Copying blob sha256:bfb770d23d016014fce593a358be31cc87463611db5f51a5d43bb79000066541
  Copying config sha256:6c1a5a8caf7fdcc13f38af020606f7380e298e35ecc2144c30c7c31fc97587e0
  Writing manifest to image destination
  6c1a5a8caf7fdcc13f38af020606f7380e298e35ecc2144c30c7c31fc97587e0
  Running: /usr/bin/podman-remote --remote --url unix:///run/podman/podman-c92695a34aca65752ac88713b0a02271a4fd16ba94e31a009ed1f14b7bc040d5.sock run --seccomp-policy image workingimage ls
  time="2025-08-30T15:56:21-04:00" level=error msg="cannot start a container that has stopped"
  Error: `/usr/bin/runc start ad819e1dc1165b5dbfec51414a9ddd65252d20306e64dc980486e62afbd0637e` failed: exit status 1
  [FAILED] Expected
      <*integration.PodmanSessionIntegration | 0xc0000c84e0>: {
          PodmanSession: {
              Session: {
                  Command: {
                      Path: "/usr/bin/podman-remote",
                      Args: [
                          "/usr/bin/podman-remote",
                          "--remote",
                          "--url",
                          "unix:///run/podman/podman-c92695a34aca65752ac88713b0a02271a4fd16ba94e31a009ed1f14b7bc040d5.sock",
                          "run",
                          "--seccomp-policy",
                          "image",
                          "workingimage",
                          "ls",
                      ],
                      Env: nil,
                      Dir: "",
                      Stdin: nil,
                      Stdout: <*io.multiWriter | 0xc00045db18>{
                          writers: [
                              <*gbytes.Buffer | 0xc000b0b480>{
                                  contents: nil,
                                  readCursor: 0,
                                  lock: {
                                      _: {},
                                      mu: {state: 0, sema: 0},
                                  },
                                  detectCloser: nil,
                                  closed: true,
                              },
                              <*internal.Writer | 0xc000538000>{
                                  buffer: {
                                      buf: [82, 117, 110, 110, 105, 110, 103, 58, 32, 47, 117, 115, 114, 47, 98, 105, 110, 47, 112, 111, 100, 109, 97, 110, 32, 45, 45, 115, 116, 111, 114, 97, 103, 101, 45, 111, 112, 116, 32, 111, 118, 101, 114, 108, 97, 121, 46, 105, 109, 97, 103, 101, 115, 116, 111, 114, 101, 61, 47, 116, 109, 112, 47, 112, 111, 100, 109, 97, 110, 45, 101, 50, 101, 45, 50, 55, 50, 54, 52, 54, 55, 53, 51, 48, 47, 105, 109, 97, 103, 101, 99, 97, 99, 104, 101, 100, 105, 114, 32, 45, 45, 114, 111, 111, 116, 32, 47, 116, 109, 112, 47, 112, 111, 100, 109, 97, 110, 45, 101, 50, 101, 45, 50, 55, 50, 54, 52, 54, 55, 53, 51, 48, 47, 115, 117, 98, 116, 101, 115, 116, 45, 51, 50, 57, 54, 55, 49, 52, 53, 55, 55, 47, 112, 47, 114, 111, 111, 116, 32, 45, 45, 114, 117, 110, 114, 111, 111, 116, 32, 47, 116, 109, 112, 47, 112, 111, 100, 109, 97, 110, 45, 101, 50, 101, 45, 50, 55, 50, 54, 52, 54, 55, 53, 51, 48, 47, 115, 117, 98, 116, 101, 115, 116, 45, 51, 50, 57, 54, 55, 49, 52, 53, 55, 55, 47, 112, 47, 114, 117, 110, 114, 111, 111, 116, 32, 45, 45, 114, 117, 110, 116, 105, 109, 101, 32, 114, 117, 110, 99, 32, 45, 45, 99, 111, 110, 109, 111, 110, 32, 47, 117, 115, 114, 47, 98, 105, 110, 47, 99, 111, 110, 109, 111, 110, 32, 45, 45, 110, 101, 116, 119, 111, 114, 107, 45, 99, 111, 110, 102, 105, 103, 45, 100, 105, 114, 32, 47, 101, 116, 99, 47, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 47, 110, 101, 116, 119, 111, 114, 107, 115, 32, 45, 45, 110, 101, 116, 119, 111, 114, 107, 45, 98, 97, 99, 107, 101, 110, 100, 32, 110, 101, 116, 97, 118, 97, 114, 107, 32, 45, 45, 99, 103, 114, 111, 117, 112, 45, 109, 97, 110, 97, 103, 101, 114, 32, 115, 121, 115, 116, 101, 109, 100, 32, 45, 45, 116, 109, 112, 100, 105, 114, 32, 47, 116, 109, 112, 47, 112, 111, 100, 109, 97, 110, 45, 101, 50, 101, 45, 50, 55, 50, 54, 52, 54, 55, 53, 51, 48, 47, 115, 117, 98, 116, 101, 115, 116, 45, 51, 50, 57, 54, 55, 49, 52, 53, 55, 55, 47, 112, 32, 45, 45, 101, 118, 101, 110, 116, 115, 45, 98, 97, 99, 107, 101, 110, 100, 32, 102, 105, 108, 101, 32, 45, 45, 100, 98, 45, 98, 97, 99, 107, 101, 110, 100, 32, 115, 113, 108, 105, 116, 101, 32, 45, 45, 115, 116, 111, 114, 97, 103, 101, 45, 100, 114, 105, 118, 101, 114, 32, 111, 118, 101, 114, 108, 97, 121, 32, 115, 121, 115, 116, 101, 109, 32, 115, 101, 114, 118, 105, 99, 101, 32, 45, 45, 116, 105, 109, 101, 32, 48, 32, 117, 110, 105, 120, 58, 47, 47, 47, 114, 117, 110, 47, 112, 111, 100, 109, 97, 110, 47, 112, 111, 100, 109, 97, 110, 45, 99, 57, 50, 54, 57, 53, 97, 51, 52, 97, 99, 97, 54, 53, 55, 53, 50, 97, 99, 56, 56, 55, 49, 51, 98, 48, 97, 48, 50, 50, 55, 49, 97, 52, 102, 100, 49,...

  Gomega truncated this representation as it exceeds 'format.MaxLength'.
  Consider having the object provide a custom 'GomegaStringer' representation
  or adjust the parameters in Gomega's 'format' package.

@kolyshkin
Copy link
Contributor Author

@ricardobranco777 thanks! Can you please test the updated patch?

Error messages between runc and crun are not synchronized, and
in some case exit codes can be different, too.

Commit dd1bcab ("CI: use local registry, part 2 of 3: fix tests")
removed the special case handling for runc from the
"podman run --seccomp-policy image (block all syscalls)"
test case, and so it fails, for example, like this:

	  Error: failed to connect to container's attach socket: /tmp/podman-e2e-2877753109/subtest-1698249469/p/root/overlay-containers/62585e98da7dc3fdb32d3b6de0980c762a8a6cde008ed35c68727fb97f5369c7/userdata/attach: no such file or directory
	  [FAILED] Command exited with status 127 (expected 126)

or this:

	  time="2025-08-29T17:16:52-07:00" level=error msg="cannot start a container that has stopped"
	  Error: `/usr/bin/runc start 63ce789f7037d9545cde832d29343704cab842e7288046407d0efa347d5ecb77` failed: exit status 1
	  [FAILED] Command exited 126 as expected, but did not emit 'OCI runtime error: runc: read from the init process'

(depending on runc version, phase of the moon etc.)

We can not reasonably expect a specific error message and exit code in
such an unusual scenario, but let's try.

With this commit, the above test passes successfully on my machine.

Tested with:

	make localintegration FOCUS="block all syscalls" OCI_RUNTIME=/usr/local/bin/runc
	make remoteintegration FOCUS="block all syscalls" OCI_RUNTIME=/usr/local/bin/runc

While at it, simplify the error message check for crun (we don't have to
check for the entire message, [the same] substring is fine for both
local and remote cases).

Fixes: dd1bcab ("CI: use local registry, part 2 of 3: fix tests")
Reported-by: Yiqiao Pu <[email protected]>
Signed-off-by: Kir Kolyshkin <[email protected]>
Copy link

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

@ricardobranco777
Copy link
Contributor

@ricardobranco777 thanks! Can you please test the updated patch?

Now it works. Thanks!

Copy link
Member

@Luap99 Luap99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM again, @containers/podman-maintainers PTAL

Copy link
Contributor

openshift-ci bot commented Sep 3, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kolyshkin, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ricardobranco777
Copy link
Contributor

It still may somestimes fail because 'cannot start a container that has stopped' was not emitted. I restarted the job and got an error:

https://openqa.opensuse.org/tests/5284084/file/podman_e2e-remoteintegration.txt

# Test messages # [It] Podman run podman run --seccomp-policy image (block all syscalls)
# failure: 

Expected
    <*integration.PodmanSessionIntegration | 0xc0004d84b0>: {
        PodmanSession: {
            Session: {
                Command: {
                    Path: "/usr/bin/podman-remote",
                    Args: [
                        "/usr/bin/podman-remote",
                        "--remote",
                        "--url",
                        "unix:///run/podman/podman-fad10ebdfd492a28edda2432ce0b92efe2178071681858791065fe6eb23e4864.sock",
                        "run",
                        "--seccomp-policy",
                        "image",
                        "workingimage",
                        "ls",
                    ],
                    Env: nil,
                    Dir: "",
                    Stdin: nil,
                    Stdout: <*io.multiWriter | 0xc0017b79f8>{
                        writers: [
                            <*gbytes.Buffer | 0xc001704ec0>{
                                contents: nil,
                                readCursor: 0,
                                lock: {
                                    _: {},
                                    mu: {state: 0, sema: 0},
                                },
                                detectCloser: nil,
                                closed: true,
                            },
                            <*internal.Writer | 0xc0001a2d80>{
                                buffer: {
                                    buf: [82, 117, 110, 110, 105, 110, 103, 58, 32, 47, 117, 115, 114, 47, 98, 105, 110, 47, 112, 111, 100, 109, 97, 110, 32, 45, 45, 115, 116, 111, 114, 97, 103, 101, 45, 111, 112, 116, 32, 111, 118, 101, 114, 108, 97, 121, 46, 105, 109, 97, 103, 101, 115, 116, 111, 114, 101, 61, 47, 116, 109, 112, 47, 112, 111, 100, 109, 97, 110, 45, 101, 50, 101, 45, 49, 49, 54, 56, 49, 57, 48, 52, 55, 48, 47, 105, 109, 97, 103, 101, 99, 97, 99, 104, 101, 100, 105, 114, 32, 45, 45, 114, 111, 111, 116, 32, 47, 116, 109, 112, 47, 112, 111, 100, 109, 97, 110, 45, 101, 50, 101, 45, 49, 49, 54, 56, 49, 57, 48, 52, 55, 48, 47, 115, 117, 98, 116, 101, 115, 116, 45, 49, 51, 50, 49, 49, 56, 52, 54, 55, 54, 47, 112, 47, 114, 111, 111, 116, 32, 45, 45, 114, 117, 110, 114, 111, 111, 116, 32, 47, 116, 109, 112, 47, 112, 111, 100, 109, 97, 110, 45, 101, 50, 101, 45, 49, 49, 54, 56, 49, 57, 48, 52, 55, 48, 47, 115, 117, 98, 116, 101, 115, 116, 45, 49, 51, 50, 49, 49, 56, 52, 54, 55, 54, 47, 112, 47, 114, 117, 110, 114, 111, 111, 116, 32, 45, 45, 114, 117, 110, 116, 105, 109, 101, 32, 114, 117, 110, 99, 32, 45, 45, 99, 111, 110, 109, 111, 110, 32, 47, 117, 115, 114, 47, 98, 105, 110, 47, 99, 111, 110, 109, 111, 110, 32, 45, 45, 110, 101, 116, 119, 111, 114, 107, 45, 99, 111, 110, 102, 105, 103, 45, 100, 105, 114, 32, 47, 101, 116, 99, 47, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 47, 110, 101, 116, 119, 111, 114, 107, 115, 32, 45, 45, 110, 101, 116, 119, 111, 114, 107, 45, 98, 97, 99, 107, 101, 110, 100, 32, 110, 101, 116, 97, 118, 97, 114, 107, 32, 45, 45, 99, 103, 114, 111, 117, 112, 45, 109, 97, 110, 97, 103, 101, 114, 32, 115, 121, 115, 116, 101, 109, 100, 32, 45, 45, 116, 109, 112, 100, 105, 114, 32, 47, 116, 109, 112, 47, 112, 111, 100, 109, 97, 110, 45, 101, 50, 101, 45, 49, 49, 54, 56, 49, 57, 48, 52, 55, 48, 47, 115, 117, 98, 116, 101, 115, 116, 45, 49, 51, 50, 49, 49, 56, 52, 54, 55, 54, 47, 112, 32, 45, 45, 101, 118, 101, 110, 116, 115, 45, 98, 97, 99, 107, 101, 110, 100, 32, 102, 105, 108, 101, 32, 45, 45, 100, 98, 45, 98, 97, 99, 107, 101, 110, 100, 32, 115, 113, 108, 105, 116, 101, 32, 45, 45, 115, 116, 111, 114, 97, 103, 101, 45, 100, 114, 105, 118, 101, 114, 32, 111, 118, 101, 114, 108, 97, 121, 32, 115, 121, 115, 116, 101, 109, 32, 115, 101, 114, 118, 105, 99, 101, 32, 45, 45, 116, 105, 109, 101, 32, 48, 32, 117, 110, 105, 120, 58, 47, 47, 47, 114, 117, 110, 47, 112, 111, 100, 109, 97, 110, 47, 112, 111, 100, 109, 97, 110, 45, 102, 97, 100, 49, 48, 101, 98, 100, 102, 100, 52, 57, 50, 97, 50, 56, 101, 100, 100, 97, 50, 52, 51, 50, 99, 101, 48, 98, 57, 50, 101, 102, 101, 50, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

To satisfy at least one of these matchers: [%!s(*utils.ExitMatcher=&{<nil> 126 126 cannot start a container that has stopped  Command exited 126 as expected, but did not emit 'cannot start a container that has stopped'}) %!s(*utils.ExitMatcher=&{<nil> 127 126 failed to connect to container's attach socket  Command exited with status 126 (expected 127)})]
  In [It] at: /root/podman/test/e2e/run_seccomp_test.go:64 @ 09/03/25 06:00:29.306

  Full Stack Trace
    github.com/containers/podman/v5/test/e2e.init.func95.4()
        /root/podman/test/e2e/run_seccomp_test.go:64 +0x2c3
  < Exit [It] podman run --seccomp-policy image (block all syscalls) - /root/podman/test/e2e/run_seccomp_test.go:49 @ 09/03/25 06:00:29.306 (703ms)
  > Enter [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:130 @ 09/03/25 06:00:29.307
  Running: /usr/bin/podman-remote --remote --url unix:///run/podman/podman-fad10ebdfd492a28edda2432ce0b92efe2178071681858791065fe6eb23e4864.sock stop --all -t 0
  33d36d1c6feaf5797994e3bdc71be95e9cd61c89028c80f687120e07a1d356d5
  9c44caa088773c56028aa0396de46350685806c926af7f10b1872661b070a8df
  Running: /usr/bin/podman-remote --remote --url unix:///run/podman/podman-fad10ebdfd492a28edda2432ce0b92efe2178071681858791065fe6eb23e4864.sock pod rm -fa -t 0
  Running: /usr/bin/podman-remote --remote --url unix:///run/podman/podman-fad10ebdfd492a28edda2432ce0b92efe2178071681858791065fe6eb23e4864.sock rm -fa -t 0
  33d36d1c6feaf5797994e3bdc71be95e9cd61c89028c80f687120e07a1d356d5
  9c44caa088773c56028aa0396de46350685806c926af7f10b1872661b070a8df
  < Exit [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:130 @ 09/03/25 06:00:29.528 (221ms)

@baude
Copy link
Member

baude commented Sep 4, 2025

@ricardobranco777 do you want us to hold merging this then ?

@ricardobranco777
Copy link
Contributor

@ricardobranco777 do you want us to hold merging this then ?

Yes because it fails sometimes.

@kolyshkin
Copy link
Contributor Author

It still may somestimes fail because 'cannot start a container that has stopped' was not emitted. I restarted the job and got an error:

https://openqa.opensuse.org/tests/5284084/file/podman_e2e-remoteintegration.txt

Frankly I don't understand. The log clearly shows the message:

Running: /usr/bin/podman-remote --remote --url unix:///run/podman/podman-fad10ebdfd492a28edda2432ce0b92efe2178071681858791065fe6eb23e4864.sock run --seccomp-policy image workingimage ls
  time="2025-09-03T06:00:29-04:00" level=error msg="cannot start a container that has stopped"
  Error: `/usr/bin/runc start 9c44caa088773c56028aa0396de46350685806c926af7f10b1872661b070a8df` failed: exit status 1
  [FAILED] Expected

The issue might be that the message is printed on stdout not stderr. I have checked it is not the case:

   runc create --console-socket /tmp/bats-run-W3wxO2/runc.WSV49c/tty/sock test_busybox (status=0)
   STDOUT:
   STDERR: time="2025-09-04T17:54:15-07:00" level=warning msg="could not find any syscalls for arch amd64"
   runc start test_busybox (status=1)
   STDOUT:
   STDERR: time="2025-09-04T17:54:15-07:00" level=error msg="cannot start a container that has stopped"

Will keep digging.

@Luap99
Copy link
Member

Luap99 commented Sep 5, 2025

@kolyshkin I think the problem is that this is in the remote test, the stdout/err of the runtime is attach to the server process not the podman-remote run process. As such this is not getting captured on the right process.

I think this is why we need the IsRemote() there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. release-note-none
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants