Fix gcp remote log module import in airflow local settings

[gopidesupavan]

Gcp log module path pointed to logs. causing ModuleNotFoundError.

User reported in slack. https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1745522030825469

│ Traceback (most recent call last):                                                                                                                                                                                                                
│   File "/home/airflow/.local/lib/python3.10/site-packages/airflow/logging_config.py", line 57, in load_logging_config                                                                                                                             
│     logging_config = import_string(logging_class_path)                                                                                                                                                                                            
│   File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 39, in import_string                                                                                                                             
│     module = import_module(module_path)                                                                                                                                                                                                           
│   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module                                                                                                                                                              
│     return _bootstrap._gcd_import(name[level:], package, level)                                                                                                                                                                                   
│   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import                                                                                                                                                                                 
│   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load                                                                                                                                                                              
│   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked                                                                                                                                                                     
│   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked                                                                                                                                                                               
│   File "<frozen importlib._bootstrap_external>", line 883, in exec_module                                                                                                                                                                         
│   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed                                                                                                                                                                    
│   File "/log_config/k8_log_config.py", line 8, in <module>                                                                                                                                                                                        
│     from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG                                                                                                                                                            
│   File "/home/airflow/.local/lib/python3.10/site-packages/airflow/config_templates/airflow_local_settings.py", line 183, in <module>                                                                                                              
│     from airflow.providers.google.cloud.logs.gcs_task_handler import GCSRemoteLogIO                                                                                                                                                               
│ ModuleNotFoundError: No module named 'airflow.providers.google.cloud.logs'      

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[gopidesupavan]

This needs to go v3-0-test ryt?

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

[sgrzemski]

Any chance this gets released soon? We had high hopes for Airflow 3.0, but the upgrade is blocked because of this issue :(

[kaxil]

Any chance this gets released soon? We had high hopes for Airflow 3.0, but the upgrade is blocked because of this issue :(

3.0.1 will be released on 7th May, so not too far now. Btw you can also apply patch since it is just 1-liner

[ldacey]

I tried to update the local settings to fix the import error but then I ran into:

GCSRemoteLogIO.__init__() missing 3 required positional arguments: 'gcp_keyfile_dict', 'scopes', and 'project_id'

Same conn_id I always used in Airflow 2. I don't have any of those things defined since I just use Workload Identity and add the service account annotations.

[gopidesupavan]

I tried to update the local settings to fix the import error but then I ran into:

GCSRemoteLogIO.__init__() missing 3 required positional arguments: 'gcp_keyfile_dict', 'scopes', and 'project_id'

Same conn_id I always used in Airflow 2. I don't have any of those things defined since I just use Workload Identity and add the service account annotations.

How your configuring this one?

You need to set them in the logging section in conf with remote_task_handler_kwargs as key

[sgrzemski]

Well, 3.0.1 has the very same error that @ldacey posted...
It's bugged. The docs says the remote_task_handler_kwargs should not be required. gcp_keyfile_dict does not exist when using Workload Identity.

[ldacey]

I tried to update the local settings to fix the import error but then I ran into:

GCSRemoteLogIO.__init__() missing 3 required positional arguments: 'gcp_keyfile_dict', 'scopes', and 'project_id'

Same conn_id I always used in Airflow 2. I don't have any of those things defined since I just use Workload Identity and add the service account annotations.

How your configuring this one?

You need to set them in the logging section in conf with remote_task_handler_kwargs as key

There is no keyfile dict or keyfile path though, so hopefully we can pass null or an empty string. Even if we were not using workload identity, I would assume this information would be retrieved from the gcp_conn_id? We wouldn't add sensitive information like the keyfile dict to the values.yaml.

[gopidesupavan]

I tried to update the local settings to fix the import error but then I ran into:

GCSRemoteLogIO.__init__() missing 3 required positional arguments: 'gcp_keyfile_dict', 'scopes', and 'project_id'

Same conn_id I always used in Airflow 2. I don't have any of those things defined since I just use Workload Identity and add the service account annotations.

How your configuring this one?
You need to set them in the logging section in conf with remote_task_handler_kwargs as key

There is no keyfile dict or keyfile path though, so hopefully we can pass null or an empty string. Even if we were not using workload identity, I would assume this information would be retrieved from the gcp_conn_id? We wouldn't add sensitive information like the keyfile dict to the values.yaml.

This should solve the above case #50590

If you want to workout now before that change released. you can do like this.

google_key_path = "<somelocation>"
remote_task_handler_kwargs = {"project_id": "", "gcp_keyfile_dict": {},"scopes": ["https://www.googleapis.com/auth/devstorage.read_write"]}

[sgrzemski]

that's what I did, but workers are still struggling to get the data shipped to GCS:
AttributeError: 'GCSRemoteLogIO' object has no attribute 'processors'

[gopidesupavan]

that's what I did, but workers are still struggling to get the data shipped to GCS: AttributeError: 'GCSRemoteLogIO' object has no attribute 'processors'

yeah thats because of the recent provider changes not released yet.