Conditionally allow request redirection with environment variable

[raka-gunarto]

Sometimes you're behind a corporate firewall like Cisco Umbrella which does funny business like adding redirections. Add redirect domain allowlist with environment variable.

Example use:

• set env var HF_DOWNLOAD_REDIRECT_ALLOWLIST=opendns.com if there's a corp firewall redirecting your web requests to opendns.com

[raka-gunarto]

This has been tested successfully behind a Cisco Umbrella firewall inserting redirects to opendns.com, can someone please review?

[hanouticelina]

Hi @raka-gunarto,
rather than modifying the client to follow redirects to third party domains (I would prefer not to add external redirect logic to the client), would it be possible on your side to either:

• whitelist direct access to huggingface.co in Cisco Umbrella
• or route traffic through your corporate HTTPS proxy? more details in the documentation here: https://huggingface.co/docs/huggingface_hub/v0.34.4/en/package_reference/utilities#huggingface_hub.configure_http_backend

[raka-gunarto]

Yeah we did that in the end, but can it not silently fail as well? I was scratching my head for hours, because it just thinks it reached a CDN and then silently dies because it doesn't see what it expects to see.

[hanouticelina]

@raka-gunarto could you open an issue with a minimal script that reproduces the issue with the full log output with HF_DEBUG environment variable set to 1 and your environment info (you can run the hf env command and copy paste the output)? thank you!

[raka-gunarto]

@hanouticelina I've just tried to reproduce and now it just fails outright, I think previously it grabbed the metadata but cisco umbrella wasn't consistent in redirecting and allowing some things through resulting partial downloads of models. I suppose I'll just close this PR since this really would affect a small subset of people. Maybe include the returned response on a failed request for metadata (so at least the user can see if it was something intercepting their requests, etc. I've included logs and hf env output anyway below.

Send: curl -X HEAD -H 'Accept: */*' -H 'Accept-Encoding: identity' -H 'Connection: keep-alive' -H 'user-agent: sentence-transformers/None; hf_hub/0.34.4; python/3.10.12; torch/2.8.0' https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/modules.json
Request ae69c655-50c7-45de-8833-b880cebe1c46: HEAD https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/modules.json (authenticated: False)
No sentence-transformers model found with name sentence-transformers/all-MiniLM-L6-v2. Creating a new one with mean pooling.
Send: curl -X HEAD -H 'Accept: */*' -H 'Accept-Encoding: identity' -H 'Connection: keep-alive' -H 'Cookie: X-OpenDNS-Session=3ad4ff230e17204031085de076ce1e0184b79270fc43_QZBFvS89' -H 'user-agent: unknown/None; hf_hub/0.34.4; python/3.10.12; torch/2.8.0; transformers/4.56.0; session_id/915d8375c45e45d7a98b46a6d4bd0821' https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/adapter_config.json
Request 5adc4991-2e5c-4308-959b-74133d4cd1c2: HEAD https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/adapter_config.json (authenticated: False)
Send: curl -X HEAD -H 'Accept: */*' -H 'Accept-Encoding: identity' -H 'Connection: keep-alive' -H 'Cookie: X-OpenDNS-Session=82aa49210cb680493a09e2c0d4e057321ab79270fc48_6XsWuN8s' -H 'user-agent: unknown/None; hf_hub/0.34.4; python/3.10.12; torch/2.8.0; transformers/4.56.0; session_id/915d8375c45e45d7a98b46a6d4bd0821; file_type/config; from_auto_class/True' https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/config.json
Request bb8c3b79-6070-462d-8dd4-3b9c792d0c03: HEAD https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/config.json (authenticated: False)
Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1568, in _get_metadata_or_catch_error
    raise FileMetadataError(
huggingface_hub.errors.FileMetadataError: Distant resource does not seem to be on huggingface.co. It is possible that a configuration issue prevents you from downloading resources from https://huggingface.co. Please check your firewall and proxy settings and make sure your SSL certificates are updated.

- huggingface_hub version: 0.34.4
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /home/vscode/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.8.0
- Jinja2: 3.1.6
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: 11.3.0
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 2.2.6
- pydantic: 2.11.7
- aiohttp: N/A
- hf_xet: 1.1.9
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/vscode/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/vscode/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/vscode/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /home/vscode/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_DISABLE_XET: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10