Failing to push large image layers

[viktor-f]

Not sure if this should have been an issue instead, let me know if you want me to move it.

I have some issues pushing images with large layers to harbor. Using the docker cli I can see the large layers uploading until the "meter" gets full, then it stops for a while until it fails and then retries the upload. Looking at the different logs in Harbor I can see two relevant log lines in core and in registry.

Log for core: 2023/06/08 09:01:25 http: proxy error: context canceled
Log for registry:

time="2023-06-08T09:01:26.497450171Z" level=error msg="client disconnected during blob PATCH" auth.user.name="harbor_registry_user" contentLength=583054592 copied=276194948 error="unexpected EOF" go.version=go1.18.5 http.request.host=harbor.<redacted> http.request.id=4499b24d-cac3-46cb-b34c-416c63b381b2 http.request.method=PATCH http.request.remoteaddr=185.189.28.150 http.request.uri="/v2/viktor-test/cocalc/blobs/uploads/658c9b1f-19fd-4de7-81f1-b5a54f377f7b?_state=WsmyqlXhWkdR_WM9a-l_xZSRC7sfgVYpVDklXcSrZGJ7Ik5hbWUiOiJ2aWt0b3ItdGVzdC9jb2NhbGMiLCJVVUlEIjoiNjU4YzliMWYtMTlmZC00ZGU3LTgxZjEtYjVhNTRmMzc3ZjdiIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTA2LTA4VDA4OjU3OjE0Ljc2MTQ3NjA0OVoifQ%3D%3D" http.request.useragent="docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~22.04.3 kernel/5.19.0-43-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \(linux\))" vars.name="viktor-test/cocalc" vars.uuid=658c9b1f-19fd-4de7-81f1-b5a54f377f7b 
2023/06/08 09:01:27 http: superfluous response.WriteHeader call from github.com/docker/distribution/vendor/github.com/gorilla/handlers.(*responseLogger).WriteHeader (handlers.go:125)
time="2023-06-08T09:01:27.677228976Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="client disconnected" err.message="unknown error" go.version=go1.18.5 http.request.host=harbor.<redacted> http.request.id=4499b24d-cac3-46cb-b34c-416c63b381b2 http.request.method=PATCH http.request.remoteaddr=185.189.28.150 http.request.uri="/v2/viktor-test/cocalc/blobs/uploads/658c9b1f-19fd-4de7-81f1-b5a54f377f7b?_state=WsmyqlXhWkdR_WM9a-l_xZSRC7sfgVYpVDklXcSrZGJ7Ik5hbWUiOiJ2aWt0b3ItdGVzdC9jb2NhbGMiLCJVVUlEIjoiNjU4YzliMWYtMTlmZC00ZGU3LTgxZjEtYjVhNTRmMzc3ZjdiIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTA2LTA4VDA4OjU3OjE0Ljc2MTQ3NjA0OVoifQ%3D%3D" http.request.useragent="docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~22.04.3 kernel/5.19.0-43-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=52.789488779s http.response.status=500 http.response.written=89 vars.name="viktor-test/cocalc" vars.uuid=658c9b1f-19fd-4de7-81f1-b5a54f377f7b 

As you can see especially the logs from core are not very specific. Adding debug logging did not really reveal anything more relevant to this. I tried going through the network traffic and it seems like the error starts with core returning a 502 http error to the client, then you get some reset messages going to core from the client (or possibly from nginx in between).

I'm running harbor in several kubernetes clusters, installed via the helm chart running version 2.6. I can see this problem in several clusters, but it seems to depend a bit on the cloud provider. For one cloud provider I can see errors starting with layers that are ~1 GB, while on another cloud provider I can see errors for layers that are ~7GB. The installations are also not always consistent in what size fails, e.g. sometimes the first cloud fails at 800MB and sometimes it can manage 1.5GB. In one cloud I'm storing images in object storage via s3 (though it's not aws) and in the other it's object storage via swift api.

It's very possible that there is some issue with the object storage on these clouds. But I would like to know exactly what happens here, is there some timeout in harbor, is there some actual error talking to the object storage, or something else? Ultimately I want to be able to support larger layers (at least up to ~2GB), but if that is not possible then it would be good to just understand what is happening.

[wy65701436]

hi, what's the harbor version? Based on the failure error, it appears that the request was canceled due to a timeout, which could be caused by either the proxy or Harbor components.

To troubleshoot this issue, you can start by tuning the request/response timeout by setting a larger value and retrying the operation.

[viktor-f]

Hi, I'm currently running 2.6.0.
Could you point me to the timeout config you are talking about?

[yunghollow91]

Hey @viktor-f , I'm encountering the same problem as you do, did you manage to have a workaround for this issue?

[yunghollow91]

For anyone also encountering this issue, we've discovered that the problem lies on ingress-nginx just randomly closing long HTTP connections after some time. We mitigated this by switching to a different ingress.

[crazyelectron-io]

@yunghollow91 doe you have an example/snippet of where you changed that? I cannot find any timeout related parameter in the Helm values.yaml (running Harbor on Kubernetes). Or is this your own ingress controller outside of the Harbor installed ngnix (I use Traefik there)?

[nelsonrogers]

Did you find the solution with traefik?

[crazyelectron-io]

Not yet. Need to find some time to experiment...

[rhaarhoff]

@crazyelectron-io found a solution for the Traefik yet? Running on K3s Treafik v2. And anything over 700MB times out.

[bockha]

Setting the transport.respondingTimeouts.readTimeout (default 60s) to 0 fixed the issue for me.

[acmeiva]

Setting the transport.respondingTimeouts.readTimeout (default 60s) to 0 fixed the issue for me.

Yes, that works in my case as well. I set up traefik with Harbor in docker-compose, my config file is like this

entryPoints:
  web:
    address: ":80"
    transport:
      respondingTimeouts:
        readTimeout: 0
        writeTimeout: 0
  websecure:
    address: ":443"
    transport:
      respondingTimeouts:
        readTimeout: 0
        writeTimeout: 0

api:
  dashboard: true
  insecure: true

certificatesResolvers:
  cloudflare:
    acme:
      email: ---
      storage: /letsencrypt/acme.json
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 0
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

providers:
  docker:
    exposedByDefault: false
  file:
    filename: /etc/traefik/dynamic.yml