Could not read served logs: [Errno -2] Name or service not known

[AsoTora]

Apache Airflow version

helm chart version 1.14.0

What happened

I have setup dags with git sync, disabled persistance and enabled the remote logging, I'm Using external postgres and S3.

values.yaml:

defaultAirflowRepository: registry.digitalocean.com/****
    defaultAirflowTag: main
    config:
      core:
        remote_logging: "True"
      logging:
        remote_logging: "True"
        # logging_level: "DEBUG"
        remote_base_log_folder: "s3://airflow-*********/logs"
        remote_log_conn_id: "log_s3_conn"
        encrypt_s3_logs: "False"
    registry:
      secretName: do-registry-secrets
    useStandardNaming: true
    executor: "CeleryExecutor"
    allowPodLaunching: true
    ingress:
      ....

    pgbouncer:
      enabled: true

    workers:
      persistence:
        enabled: false
      terminationGracePeriodSeconds: 600 # celery worker warm shutdown
      resources:
        requests:
          cpu: 500m
          memory: "2Gi"
        limits:
          cpu: 1
          memory: "3Gi"
      hpa:
        enabled: true
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 80
          - type: Resource
            resource:
              name: memory
              target:
                type: Utilization
                averageUtilization: 90
        minReplicaCount: 1
        maxReplicaCount: 2

    dags:
      persistence:
        enabled: false
      gitSync:
        enabled: true
        repo: [email protected]:*********
        branch: stage
        subPath: "dags"
        sshKeySecret:********
        knownHosts: |
          github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
          github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
          github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
        resources:
          requests:
            cpu: 10m
            memory: 64Mi
          limits:
            cpu: 40m
            memory: 128Mi
    #################################################
    ###                 extra                   #####
    #################################################
    rbac:
      createSCCRoleBinding: true
    # https://airflow.apache.org/docs/helm-chart/stable/manage-logs.html
    # Stdout logs are saved to Grafana Loki
    logs:
      persistence:
        enabled: false
    createUserJob:
      useHelmHooks: false
      applyCustomEnv: false
    migrateDatabaseJob:
      useHelmHooks: false
      applyCustomEnv: false
    postgresql:
      enabled: false
    minio:
      enabled: false

In the UI s3 connection:

Now i randomly get:

*** No logs found on s3 for ti=<TaskInstance: hello_world.hello_task manual__2024-07-11T17:45:40.089474+00:00 [success]>
*** Could not read served logs: [Errno -2] Name or service not known

In worker pods in k8s i see that the user is used correctly:

[2024-07-18 13:56:54,658: INFO/ForkPoolWorker-15] Using connection ID 'log_s3_conn' for task execution.
[2024-07-18 13:56:54,659: INFO/ForkPoolWorker-15] AWS Connection (conn_id='log_s3_conn', conn_type='aws') credentials retrieved from login and password

there are also no permissions issues, as far as i can tell, because simple hello_world dag does work and creates the files on s3..

Worker logs for dag with s3 issues

I see the issue when running 1 specific dag from our older Airflow instance, this also correlates with the following logs in worker pod:

            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle '_thread.lock' object
[2024-07-18 14:04:03,864: ERROR/ForkPoolWorker-15] Task airflow.providers.celery.executors.celery_executor_utils.execute_command[3d69b14b-c354-472e-9649-2de323e11069] raised unexpected: AirflowException('Celery command failed on host: airflow-stage-worker-7d6d776848-gjhcv with celery_task_id 3d69b14b-c354-472e-9649-2de323e11069')
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 135, in execute_command
    _execute_in_fork(command_to_exec, celery_task_id)
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 150, in _execute_in_fork
    raise AirflowException(msg)
airflow.exceptions.AirflowException: Celery command failed on host: airflow-stage-worker-7d6d776848-gjhcv with celery_task_id 3d69b14b-c354-472e-9649-2de323e11069

Any idea why this may be happening?

[OlafenwaMoses]

I am getting the same error as well with the worker on Airflow 3.0.0, celery and redis. Was working fine before 2.11.0 > 3.0.0 upgrade.