How does homebrew store bottles?

[talwat]

Output of brew config

HOMEBREW_VERSION: 4.0.6-147-gb3684e5
ORIGIN: https://github.com/Homebrew/brew
HEAD: b3684e55a7a0090e50d975c347961cb57a5ffcb5
Last commit: 19 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 54e2b0782d64b19a7647adfbc4099df97ab60f67
Core tap last commit: 6 weeks ago
Core tap branch: master
Core tap JSON: 18 Mar 14:08 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: vim
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit arm_firestorm_icestorm
Clang: 14.0.0 build 1400
Git: 2.37.1 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.6.3-arm64
CLT: 14.2.0.0.1.1668646533
Xcode: N/A
Rosetta 2: false

Output of brew doctor

Your system is ready to brew.

Description of issue

I was wondering how exactly the homebrew team stores binary bottles. Whenever I install something, sometimes I see it downloading from a link (https://ghcr.io/v2/homebrew/core) which turns out to be the github package registry. But that's for dockerfiles, and packages? An explanation would be greatly appreciated, because I want to know whether I can do something similar myself for my own projects.

PS: I apologize for using discussions like this, because I wasn't sure where else to ask this, and I figured this was the most appropriate place. Sorry!

[carlocab]

As you've observed, we store them in the GitHub Packages. The GitHub Package registry can be used for arbitrary binary blobs, not just Docker containers.

[talwat]

How do you use them like that though? Like can I have some kind of example or a tutorial of some sort?

[alebcay]

I don't think this is particularly documented anywhere, but you can see what's happening under the hood with brew fetch --debug <some formula>. Here's what the process looks like with brew fetch --debug hello:

1. GET the OCI artifact manifest:

   $ curl --disable --cookie /dev/null --globoff --show-error \
   --user-agent Homebrew/4.0.10-7-g91251d5\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 12.6.3\)\ curl/7.79.1 --header Accept-Language:\ en \
   --fail --progress-bar --retry 3 \
   --header Accept:\ application/vnd.oci.image.index.v1\+json \
   --header Authorization:\ Bearer\ QQ== \
   --location --remote-time \
   --output manifest.json \
   https://ghcr.io/v2/homebrew/core/hello/manifests/2.12.1

   Not all of the headers may be needed, but note that the Accept and Authorization are necessary (even though the API allows for unauthenticated access, an empty Bearer token must be provided).

2. Read the manifest. My machine is Intel-based and is running macOS Monterey, so it chooses this manifest from the list:

   {
     "mediaType": "application/vnd.oci.image.manifest.v1+json",
     "digest": "sha256:205f7a66495737af32db3125a63fc229622d8917b65eaf2436e4093f18948dc7",
     "size": 1911,
     "platform": {
       "architecture": "amd64",
       "os": "darwin",
       "os.version": "macOS 12"
     },
     "annotations": {
       "org.opencontainers.image.ref.name": "2.12.1.monterey",
       "sh.brew.bottle.digest": "62534bceb8f7074827fa2146dd13603018aaf07c82e22cfef96571c8133ce8a1",
       "sh.brew.tab": "{\"homebrew_version\":\"3.4.11-152-ga3fab02\",\"changed_files\":[],\"source_modified_time\":1653865426,\"compiler\":\"clang\",\"runtime_dependencies\":[],\"arch\":\"x86_64\",\"built_on\":{\"os\":\"Macintosh\",\"os_version\":\"macOS 12\",\"cpu_family\":\"penryn\",\"xcode\":\"13.4\",\"clt\":\"13.4.0.0.1.1651278267\",\"preferred_perl\":\"5.30\"}}"
     }
   }

   From here, we obtain the value of sh.brew.bottle.digest.

3. Submit a HEAD request to find the actual storage location of the desired bottle.

   $ curl --disable --cookie /dev/null --globoff --show-error \
   --user-agent Homebrew/4.0.10-7-g91251d5\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 12.6.3\)\ curl/7.79.1 \
   --header Accept-Language:\ en \
   --retry 3 \
   --header Authorization:\ Bearer\ QQ== \
   --location --silent --head --request GET \
   https://ghcr.io/v2/homebrew/core/hello/blobs/sha256:62534bceb8f7074827fa2146dd13603018aaf07c82e22cfef96571c8133ce8a1

   The initial 307 response points to a pkg-containers.githubusercontent.com domain in the location response header, which is the actual data of the bottle.

   content-length: 0
   content-type: application/vnd.oci.image.layer.v1.tar+gzip
   docker-distribution-api-version: registry/2.0
   location: https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:62534bceb8f7074827fa2146dd13603018aaf07c82e22cfef96571c8133ce8a1?se=2023-03-28T12%3A30%3A00Z&sig=79p7U17wtTW%2Fe1ti7aSnsFZeKor5GFcW5FrX84obBeI%3D&sp=r&spr=https&sr=b&sv=2019-12-12
   date: Tue, 28 Mar 2023 12:23:00 GMT
   x-github-request-id: F047:27AC:2519A33:38F1378:6422DC24
   

   Note that the URL in this location response header is not a permalink. When starting a new download transaction, start with the HEAD request first to obtain a current, valid URL.

4. GET the location.

   $ curl --disable --cookie /dev/null --globoff --show-error \
   --user-agent Homebrew/4.0.10-7-g91251d5\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 12.6.3\)\ curl/7.79.1 \
   --header Accept-Language:\ en \
   --fail --progress-bar --retry 3 --location --remote-time --output bottle.tar.gz \
   'https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:62534bceb8f7074827fa2146dd13603018aaf07c82e22cfef96571c8133ce8a1?se=2023-03-28T12%3A30%3A00Z&sig=79p7U17wtTW%2Fe1ti7aSnsFZeKor5GFcW5FrX84obBeI%3D&sp=r&spr=https&sr=b&sv=2019-12-12'

   Now the actual bottle tarball is downloaded.

   $ tar tzf bottle.tar.gz
   hello/2.12.1/
   hello/2.12.1/.brew/
   hello/2.12.1/.brew/hello.rb
   hello/2.12.1/AUTHORS
   hello/2.12.1/COPYING
   hello/2.12.1/ChangeLog
   hello/2.12.1/NEWS
   hello/2.12.1/README
   hello/2.12.1/TODO
   hello/2.12.1/bin/
   hello/2.12.1/bin/hello
   hello/2.12.1/share/
   hello/2.12.1/share/info/
   hello/2.12.1/share/info/hello.info
   hello/2.12.1/share/man/
   hello/2.12.1/share/man/man1/
   hello/2.12.1/share/man/man1/hello.1

[alebcay]

Regarding uploading bottles, I don't have as much insight but the code that handles that is at https://github.com/Homebrew/brew/blob/b753315b0b1e78b361612bf4985502bf9dca5582/Library/Homebrew/github_packages.rb#L196-L428.

[ReenigneArcher]

Thanks for sharing your insight! I also noticed they were stored in ghcr and didn't understand how that was possible for non docker images.