This article will describe how to download an image from a (docker) container registry.
Intro
Remember the good ‘ol days when you could just download software by visiting a website and click “download”?
Even apt
and yum
repositories were just simple HTTP servers that you could just curl
(or wget
) from. Using the package manager was, of course, more secure and convenient — but you could always just download packages manually, if you wanted.
But have you ever tried to curl
an image from a container registry, such as docker? Well friends, I have tried. And I have the scars to prove it.
It was a remarkably complex process that took me weeks to figure-out. Lucky you, this article will break it down.
Examples
Specifically, we’ll look at how to download files from two OCI registries.
Terms
First, here’s some terminology used by OCI
- OCI – Open Container Initiative
- blob – A “blob” in the OCI spec just means a file
- manifest – A “manifest” in the OCI spec means a list of files
Prerequisites
This guide was written in 2024, and it uses the following software and versions:
- debian 12 (bookworm)
- curl 7.88.1
- OCI Distribution Spec v1.1.0 (which, unintuitively, uses the ‘/v2/‘ endpoint)
Of course, you’ll need ‘curl
‘ installed. And, to parse json, ‘jq
‘ too.
sudo apt-get install curl jq
What is OCI?
OCI stands for Open Container Initiative.
OCI was originally formed in June 2015 for Docker and CoreOS. Today it’s a wider, general-purpose (and annoyingly complex) way that many projects host files (that are extremely non-trivial to download).
One does not simply download a file from an OCI-complianet container registry. You must:
- Generate an authentication token for the API
- Make an API call to the registry, requesting to download a JSON “Manifest”
- Parse the JSON Manifest to figure out the hash of the file that you want
- Determine the download URL from the hash
- Download the file (which might actually be many distinct file “layers”)
In order to figure out how to make an API call to the registry, you must first read (and understand) the OCI specs here.
OCI APIs
OCI maintains three distinct specifications:
- image spec
- runtime spec
- distribution spec
OCI “Distribution Spec” API
To figure out how to download a file from a container registry, we’re interested in the “distribution spec”. At the time of writing, the latest “distribution spec” can be downloaded here:
- https://github.com/opencontainers/distribution-spec/releases/tag/v1.1.0
- https://github.com/opencontainers/distribution-spec/releases/download/v1.1.0/oci-distribution-spec-v1.1.0.pdf
The above PDF file defines a set of API endpoints that we can use to query, parse, and then figure out how to download a file from a container registry. The table from the above PDF is copied below:
ID | Method | API Endpoint | Success | Failure |
---|---|---|---|---|
end-1 | GET |
/v2/ |
200 |
404 /401 |
end-2 | GET / HEAD |
/v2/<name>/blobs/<digest> |
200 |
404 |
end-3 | GET / HEAD |
/v2/<name>/manifests/<reference> |
200 |
404 |
end-4a | POST |
/v2/<name>/blobs/uploads/ |
202 |
404 |
end-4b | POST |
/v2/<name>/blobs/uploads/?digest=<digest> |
201 /202 |
404 /400 |
end-5 | PATCH |
/v2/<name>/blobs/uploads/<reference> |
202 |
404 /416 |
end-6 | PUT |
/v2/<name>/blobs/uploads/<reference>?digest=<digest> |
201 |
404 /400 |
end-7 | PUT |
/v2/<name>/manifests/<reference> |
201 |
404 |
end-8a | GET |
/v2/<name>/tags/list |
200 |
404 |
end-8b | GET |
/v2/<name>/tags/list?n=<integer>&last=<integer> |
200 |
404 |
end-9 | DELETE |
/v2/<name>/manifests/<reference> |
202 |
404 /400 /405 |
end-10 | DELETE |
/v2/<name>/blobs/<digest> |
202 |
404 /405 |
end-11 | POST |
/v2/<name>/blobs/uploads/?mount=<digest>&from=<other_name> |
201 |
404 |
end-12a | GET |
/v2/<name>/referrers/<digest> |
200 |
404 /400 |
end-12b | GET |
/v2/<name>/referrers/<digest>?artifactType=<artifactType> |
200 |
404 /400 |
end-13 | GET |
/v2/<name>/blobs/uploads/<reference> |
204 |
404 |
In OCI, files are (cryptically) called “blobs
“. In order to figure out the file that we want to download, we must first reference the list of files (called a “manifest
“).
The above table shows us how we can download a list of files (manifest) and then download the actual file (blob).
Examples
Let’s look at how to download files from a couple different OCI registries:
Docker Hub
Probably the most well-known OCI registry is Docker Hub. Indeed, as stated above, Docker was one of the original founding members of OCI, and they contributed the original specifications and implementations of the OCI runtime and image specifications.
In this example, we’ll manually download the official docker image for hitch, and then we’ll manually import it into docker.
Determine Package Name
The easiest way to figure out a given docker image’s namespace is to use the WUI: hub.docker.com.
After loading the above page, just type the package name (in our case ‘hitch
‘) into the search bar.
It might be difficult to determine which is the official author of the docker image that you want to download, as many third party providers may provide an image of similar or identical name.
If the official project documentation doesn’t explicitly link to the official listing on Docker Hub, it may be a good idea to contact the developer to confirm which account is publishing their official docker releases (and then open a ticket or PR to update the docs with this info). In any case, you should never expect the files downloaded from Docker Hub to be authentic, even if you confirmed the namespace to be correct — because Docker infamously lacks authentication of everything that it downloads by default.
The namespace of the repository for a given package in Docker Hub is typically ‘library/<package_name>
‘. So, for ‘hitch
‘, our namespace is going to be ‘library/hitch
‘.
Determine Package Tag
Click on the “Tags” tab to view all of the tags available for the image.
In our case, we’ll use the latest version at the time of writing, which is ‘1.8.0-1
‘
Get an Auth Token
Execute the following command to get an authentication token from Docker Hub within the scope of the ‘hitch
‘ package’s namespace.
# get a JSON with an anonymous token curl -so "token.json" "https://auth.docker.io/token?service=registry.docker.io&scope=<resourcetype>:<component>/<component>:<action>"; # extract token from JSON token=$(cat token.json | jq -jr ".token")
ⓘ INFO: If you’re like me and wondering where the heck is the OCI spec that defines this `
/token
` endpoint for authentication, know that it doesn’t exist 🤦To learn more about the syntax of this URL and ‘
scope
‘ GET variable, see docker’s Token Scope Documentation.
The above commands will get a free/temporary token that you can use in subsequent API calls. If all went well, there will be no output from these commands. Here’s an example execution
user@disp7456:~$ curl -so "token.json" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/hitch:pull"; user@disp7456:~$ user@disp7456:~$ token=$(cat token.json | jq -jr ".token") user@disp7456:~$
List the Tags
We can list all of the available tags for the ‘hitch
‘ package with the ‘GET /v2/<name>/tags/list
‘ API endpoint, as shown in the table above
curl -i -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/<package_name>/tags/list
Here’s an example execution. Note that it affirms the existence of the ‘1.8.0-1
‘ tag that we found above.
user@disp7456:~$ curl -i -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/hitch/tags/list HTTP/1.1 200 OK content-type: application/json docker-distribution-api-version: registry/2.0 date: Tue, 04 Jun 2024 21:11:33 GMT content-length: 131 strict-transport-security: max-age=31536000 {"name":"library/hitch","tags":["1","1.7","1.7.0","1.7.0-1","1.7.2","1.7.2-1","1.7.3","1.7.3-1","1.8","1.8.0","1.8.0-1","latest"]} user@disp7456:~$
Download the Manifest
We can download the manifest for the ‘1.8.0-1
‘ tag of the ‘hitch
‘ package with the ‘GET /v2/<name>/manifests/<reference>
‘ API endpoint, as shown in the table above
curl -o manifest.json -s -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/<package_name>/manifests/<tag>
And here’s an example execution that downloads the manifest for the ‘1.8.0-1
‘ tag of the ‘hitch
‘ package
user@disp7456:~$ curl -o manifest.json -s -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/hitch/manifests/1.8.0-1 user@disp7456:~$ user@disp7456:~$ ls manifest.json user@disp7456:~$
Parse the Manifest
The previous step downloaded a file named ‘manifest.json
‘. This ‘manifest.json
‘ file lists all of the “layers” that make up the image of the ‘hitch
‘ package.
Each “layer” consists of a tarball and some metadata about the layer in json format. The information that we need to download the layer’s tar file is located in the ‘manifest.json
‘ file. And the metadata about each layer is also in the ‘manifest.json
‘ file.
The format of the ‘manifest.json
‘ file is ‘vnd.docker.distribution.manifest.v1+json
‘, which is defined in Image Manifest Version 2, Schema 1.
Most importantly, the ‘manifest.json
‘ file contains two parallel arrays of the same length:
- fsLayers[]
- history[]
Consider this truncated snippet of the manifest for the ‘hitch
‘ package’s ‘1.8.0-1
‘ tag:
{ "schemaVersion": 1, "name": "library/hitch", "tag": "1.8.0-1", "architecture": "amd64", "fsLayers": [ { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a0e9543db8c1238572466cf00b55436bc7b7e849f7cb305128f391a94b75c2fc" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:728328ac3bde9b85225b1f0d60f5c149f5635a191f5d8eaeeb00e095d36ef9fd" } ], "history": [ { "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"443/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[],\"Image\":\"sha256:996009c7d7eb032c9ea750e5decc1f8aedbf4530b892cf4ebc7716e1458f36d9\",\"Volumes\":null,\"WorkingDir\":\"/etc/hitch\",\"Entrypoint\":[\"docker-hitch-entrypoint\"],\"OnBuild\":null,\"Labels\":null},\"container\":\"0ff54ee96c4bbfe77da3b2124720ef95c6154d3bc1d3e40a168920dd818367c4\",\"container_config\":{\"Hostname\":\"0ff54ee96c4b\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"443/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD []\"],\"Image\":\"sha256:996009c7d7eb032c9ea750e5decc1f8aedbf4530b892cf4ebc7716e1458f36d9\",\"Volumes\":null,\"WorkingDir\":\"/etc/hitch\",\"Entrypoint\":[\"docker-hitch-entrypoint\"],\"OnBuild\":null,\"Labels\":{}},\"created\":\"2024-05-14T05:23:11.666992342Z\",\"docker_version\":\"20.10.23\",\"id\":\"6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb\",\"os\":\"linux\",\"parent\":\"c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188\",\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188\",\"parent\":\"1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240\",\"created\":\"2024-05-14T05:23:11.581588417Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) EXPOSE 443\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240\",\"parent\":\"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad\",\"created\":\"2024-05-14T05:23:11.489285564Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENTRYPOINT [\\\"docker-hitch-entrypoint\\\"]\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad\",\"parent\":\"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5\",\"created\":\"2024-05-14T05:23:11.403178706Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY file:1abf3c94dce5dc9f6617dc8d36a6fe6f4f7236189d4819f16cefb54288e80e0d in /usr/local/bin/ \"]}}" }, { "v1Compatibility": "{\"id\":\"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5\",\"parent\":\"5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f\",\"created\":\"2024-05-14T05:23:11.304477182Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) WORKDIR /etc/hitch\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f\",\"parent\":\"5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8\",\"created\":\"2024-05-14T05:23:11.160227264Z\",\"container_config\":{\"Cmd\":[\"|5 DISTVER=bullseye PKGCOMMIT=f12ab7958bc4885f3f00311cbca5103d9e6ba794 PKGVER=1 SHASUM=62b3554d668c9d17382415db10898bf661ee76343e4ee364f904457efda6cb1eeee7cb81d7a3897734024812b64b1c0e2dc305605706d81a0c1f6030508bf7e2 SRCVER=1.8.0 /bin/sh -c set -ex; BASE_PKGS=\\\"apt-utils curl dirmngr dpkg-dev debhelper devscripts equivs fakeroot git gnupg pkg-config\\\"; export DEBIAN_FRONTEND=noninteractive; export DEBCONF_NONINTERACTIVE_SEEN=true; tmpdir=\\\"$(mktemp -d)\\\"; cd \\\"$tmpdir\\\"; apt-get update; apt-get install -y --no-install-recommends $BASE_PKGS; git clone https://github.com/varnish/pkg-hitch.git; cd pkg-hitch; git checkout ${PKGCOMMIT}; rm -rf .git; curl -Lf https://hitch-tls.org/source/hitch-${SRCVER}.tar.gz -o $tmpdir/orig.tgz; echo \\\"${SHASUM} $tmpdir/orig.tgz\\\" | sha512sum -c -; tar xavf $tmpdir/orig.tgz --strip 1; sed -i -e \\\"s/@SRCVER@/${SRCVER}/g\\\" -e \\\"s/@PKGVER@/${PKGVER:-1}/g\\\" -e \\\"s/@DISTVER@/$DISTVER/g\\\" debian/changelog; mk-build-deps --install --tool=\\\"apt-get -o Debug::pkgProblemResolver=yes --yes\\\" debian/control; sed -i '' debian/hitch*; dpkg-buildpackage -us -uc -j\\\"$(nproc)\\\"; apt-get -y purge --auto-remove hitch-build-deps $BASE_PKGS; apt-get -y --no-install-recommends install ../*.deb; sed -i 's/daemon = on/daemon = off/' /etc/hitch/hitch.conf; rm -rf /var/lib/apt/lists/* \\\"$tmpdir\\\"\"]}}" }, { "v1Compatibility": "{\"id\":\"5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8\",\"parent\":\"c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be\",\"created\":\"2024-05-14T05:21:33.061082853Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG SHASUM=62b3554d668c9d17382415db10898bf661ee76343e4ee364f904457efda6cb1eeee7cb81d7a3897734024812b64b1c0e2dc305605706d81a0c1f6030508bf7e2\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be\",\"parent\":\"24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d\",\"created\":\"2024-05-14T05:21:32.967727298Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG PKGCOMMIT=f12ab7958bc4885f3f00311cbca5103d9e6ba794\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d\",\"parent\":\"f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34\",\"created\":\"2024-05-14T05:21:32.875807605Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG DISTVER=bullseye\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34\",\"parent\":\"65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8\",\"created\":\"2024-05-14T05:21:32.781941821Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG PKGVER=1\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8\",\"parent\":\"863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270\",\"created\":\"2024-05-14T05:21:32.682503634Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ARG SRCVER=1.8.0\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270\",\"parent\":\"e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd\",\"created\":\"2024-05-14T01:28:27.043980081Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) CMD [\\\"bash\\\"]\"]},\"throwaway\":true}" }, { "v1Compatibility": "{\"id\":\"e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd\",\"created\":\"2024-05-14T01:28:26.699066026Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:9b38b383dd93169a663eed88edf3f2285b837257ead69dc40ab5ed1fb3f52c35 in / \"]}}" } ], ... ] }
The sha256sum used to download the blob of the first layer is found at the first element of the fsLayers[]
array (fsLayers[0]
). The metadata about this first layer is found at the first element of the history[]
array (history[0]
).
The sha256sum used to download the blob of the second layer is found at second element of the fsLayers[]
array (fsLayers[1]
). The metadata about this second layer is found at the second element of the history[]
array (history[1]
).
Et cetera…
Download the Layers
So how do we download each of these layers separately, yet organize them such that we can later import them as a single image into docker? The answer to that lies in the Docker Image Specification v1.0.0.
The above spec provides an example tree of the files:
For example, here’s what the full archive of
library/busybox
is (displayed intree
format):
.
├── 5785b62b697b99a5af6cd5d0aabc804d5748abbb6d3d07da5d1d3795f2dcc83e
│ ├── VERSION
│ ├── json
│ └── layer.tar
├── a7b8b41220991bfc754d7ad445ad27b7f272ab8b4a2c175b9512b97471d02a8a
│ ├── VERSION
│ ├── json
│ └── layer.tar
├── a936027c5ca8bf8f517923169a233e391cbb38469a75de8383b5228dc2d26ceb
│ ├── VERSION
│ ├── json
│ └── layer.tar
├── f60c56784b832dd990022afc120b8136ab3da9528094752ae13fe63a2d28dc8c
│ ├── VERSION
│ ├── json
│ └── layer.tar
└── repositories
There are one or more directories named with the ID for each layer in a full image. Each of these directories contains 3 files:
- `VERSION` – The schema version of the `json` file
- `json` – The JSON metadata for an image layer
- `layer.tar` – The Tar archive of the filesystem changeset for an image
layer.The content of the
VERSION
files is simply the semantic version of the JSON metadata schema:
1.0
And the
repositories
file is another JSON file which describes names/tags:{ "busybox":{ "latest":"5785b62b697b99a5af6cd5d0aabc804d5748abbb6d3d07da5d1d3795f2dcc83e" } }Every key in this object is the name of a repository, and maps to a collection of tag suffixes. Each tag maps to the ID of the image represented by that tag.
So, as shown in the quote from the spec above, what we need to do is to create a set of directories — one for each layer — named after the layer’s ID. Each of these layer-specific directories must contain 3 files:
- The actual tarball of the layer (named ‘
layer.tar
‘), - The metadata of the layer (in a file literally named ‘
json
‘ — with no file extension) - A file named ‘
VERSION
‘ whose contents is literally just ‘1.0
‘
All of the information that we need is in the ‘manifest.json
‘ file that we already downloaded. Let’s just loop through each layer that it defines, download the layer.tar
tarball, create the ‘json
‘ metadata file, and hard-code the ‘VERSION
‘ file with the following BASH snippet
num_layers=$(cat manifest.json | jq -r ".history | length") for ((i = 0 ; i < $num_layers ; i++)); do layer_blobSum=$(cat manifest.json | jq -r ".fsLayers[$i].blobSum") layer_metadata=$(cat manifest.json | jq -r ".history[$i].v1Compatibility") layer_id=$(echo $layer_metadata | jq -r ".id") echo $layer_id echo $layer_blobSum mkdir -p "layers/$layer_id" echo "1.0" > "layers/$layer_id/VERSION" echo $layer_metadata > "layers/$layer_id/json" curl -o "layers/$layer_id/layer.tar" -#LH "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/library/<package_name>/blobs/${layer_blobSum}" done
And here’s an example execution that executes the above snippet to download all of the layers onto disk in a set of directories as defined by the Docker Image Specification v1.0.0.
user@disp7456:~$ num_layers=$(cat manifest.json | jq -r ".history | length") user@disp7456:~$ user@disp7456:~$ for ((i = 0 ; i < $num_layers ; i++)); do layer_blobSum=$(cat manifest.json | jq -r ".fsLayers[$i].blobSum") layer_metadata=$(cat manifest.json | jq -r ".history[$i].v1Compatibility") layer_id=$(echo $layer_metadata | jq -r ".id") echo $layer_id echo $layer_blobSum mkdir -p "layers/$layer_id" echo "1.0" > "layers/$layer_id/VERSION" echo $layer_metadata > "layers/$layer_id/json" curl -o "layers/$layer_id/layer.tar" -#LH "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/library/hitch/blobs/${layer_blobSum}" done 6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% 1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% 8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad sha256:3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a ################################################################ 100.0% a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% 5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f sha256:a0e9543db8c1238572466cf00b55436bc7b7e849f7cb305128f391a94b75c2fc ################################################################ 100.0% 5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% 24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% 65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% 863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ################################################################ 100.0% e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd sha256:728328ac3bde9b85225b1f0d60f5c149f5635a191f5d8eaeeb00e095d36ef9fd ################################################################ 100.0% user@disp7456:~$ user@disp7456:~$ tree layers/ layers/ ├── 1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240 │ ├── json │ ├── layer.tar │ └── VERSION ├── 24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d │ ├── json │ ├── layer.tar │ └── VERSION ├── 5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8 │ ├── json │ ├── layer.tar │ └── VERSION ├── 5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f │ ├── json │ ├── layer.tar │ └── VERSION ├── 65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8 │ ├── json │ ├── layer.tar │ └── VERSION ├── 6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb │ ├── json │ ├── layer.tar │ └── VERSION ├── 863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270 │ ├── json │ ├── layer.tar │ └── VERSION ├── 8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad │ ├── json │ ├── layer.tar │ └── VERSION ├── a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5 │ ├── json │ ├── layer.tar │ └── VERSION ├── c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be │ ├── json │ ├── layer.tar │ └── VERSION ├── c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188 │ ├── json │ ├── layer.tar │ └── VERSION ├── e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd │ ├── json │ ├── layer.tar │ └── VERSION └── f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34 ├── json ├── layer.tar └── VERSION 14 directories, 39 files user@disp7456:~$
Finally, besides these layer-specific dirs, we need one additional file (named simply ‘repository
‘, with no file extension) at the same height as these dirs. As defined by the Docker Image Specification v1.0.0, this file should state the name & tag of the image, and it points to the first layer of the image.
Note the 0th
item in the history[]
array is the first layer of the image, so we can create this file with the following command
start_image=$(cat manifest.json | jq -r ".history[0].v1Compatibility") start_image_id=$(echo $start_image | jq -r ".id") cat > layers/repositories <<EOF { "<image_name>": { "stable": "$start_image_id" } } EOF
And here’s an example execution to create our ‘repository
‘ file for the ‘hitch
‘ package.
user@disp7456:~$ start_image=$(cat manifest.json | jq -r ".history[0].v1Compatibility") user@disp7456:~$ user@disp7456:~$ start_image_id=$(echo $start_image | jq -r ".id") user@disp7456:~$ user@disp7456:~$ cat > layers/repositories <<EOF { "hitch": { "stable": "$start_image_id" } } EOF user@disp7456:~$ user@disp7456:~$ ls layers 1d2af5a156bbc461d98824c2f6bfe295327d4419105c0b7f88f14cb28d0bb240 24e7aee556d6a38bfa2e13430db8a998c023a2920017eabc0b3bf0dd7661bf7d 5a12a2c67ff9b5bfad288a4ede18d08c259c301efb85403d08a40ea2ad0eb1f8 5a78b0e89bbae2390b83e60174ae1efc583f766eff7dffaffa747ccb67472d0f 65c7b6d17437bf7a3216e2fea283071e9b5c0d71c6b97472baa8807a30b5d9d8 6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb 863a608d086b1bcf7f9b30ccf57260e6cb5d3d793b4e1131aa8f6041b07a7270 8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5 c03ad9230005f64133de4501e14a882ef25f03443da4da55ca002d5619f998be c48ca3d95161bbcdfcaa2e016a675965d55f4f06147ef4445c69347c5965f188 e00e363f3a25341591a5a5e724e20ae3e70f0396be8483a07c0b39d25d33fecd f0d07a99d7d1f0b849a4cbe8fc4552d374f4448c2e7f8bfd908aa43132c4ec34 repositories user@disp7456:~$ user@disp7456:~$ cat layers/repositories { "hitch": { "stable": "6703605aae83084affcafb4abcc7c556f0e436c4992ae224f1f58e88242328cb" } } user@disp7456:~$
Your ‘layers/
‘ directory should now be prepared-to-spec for importing the entire image into docker.
For reference, here’s the contents of just one of the layers:
user@disp7456:~$ ls layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/ json layer.tar VERSION user@disp7456:~$ user@disp7456:~$ cat layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/VERSION 1.0 user@disp7456:~$ user@disp7456:~$ cat layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/json {"id":"8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad","parent":"a8d8314458142ee2a4ebccb19f48b6f9c696100103c3d49cbbe7ecd2575120e5","created":"2024-05-14T05:23:11.403178706Z","container_config":{"Cmd":["/bin/sh -c #(nop) COPY file:1abf3c94dce5dc9f6617dc8d36a6fe6f4f7236189d4819f16cefb54288e80e0d in /usr/local/bin/ "]}} user@disp7456:~$ user@disp7456:~$ sha256sum layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar 3148f4af0a813bcff0a3ed2562aabfb1b596b52ef36eb5eb4d82ce836350b73a layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar user@disp7456:~$ user@disp7456:~$ tar -tvf layers/8f914c821cbe154cc6677bceb043669e4295ad5cfb6409efa9c2ec1beba75fad/layer.tar drwxr-xr-x 0/0 0 2024-05-12 19:00 usr/ drwxr-xr-x 0/0 0 2024-05-12 19:00 usr/local/ drwxr-xr-x 0/0 0 2024-05-14 00:23 usr/local/bin/ -rwxrwxr-x 0/0 319 2024-05-14 00:21 usr/local/bin/docker-hitch-entrypoint user@disp7456:~$
Load the Image
Finally, you can load the layers as one image into docker with docker image load
tar -cC layers . | docker image load
Here’s an example execution
user@disp7456:~$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE user@disp7456:~$ user@disp7456:~$ tar -cC layers . | docker load e00e363f3a25: Loading layer [==================================================>] 31.43MB/31.43MB 863a608d086b: Loading layer [==================================================>] 32B/32B 65c7b6d17437: Loading layer [==================================================>] 32B/32B f0d07a99d7d1: Loading layer [==================================================>] 32B/32B 24e7aee556d6: Loading layer [==================================================>] 32B/32B c03ad9230005: Loading layer [==================================================>] 32B/32B 5a12a2c67ff9: Loading layer [==================================================>] 32B/32B 5a78b0e89bba: Loading layer [==================================================>] 1.573MB/1.573MB a8d831445814: Loading layer [==================================================>] 32B/32B 8f914c821cbe: Loading layer [==================================================>] 415B/415B 1d2af5a156bb: Loading layer [==================================================>] 32B/32B c48ca3d95161: Loading layer [==================================================>] 32B/32B 6703605aae83: Loading layer [==================================================>] 32B/32B user@disp7456:~$ user@disp7456:~$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE hitch stable f07eadb841be 3 weeks ago 85.1MB user@disp7456:~$
The image is now available in docker
.
GitHub Packages
GitHub Packages was launched as a Beta in May 2019. It allowed users to publish packages in many formats, including images uploaded to a GitHub Docker Registry. In September 2020, GitHub added a generic Container Registry as a Beta to GitHub Packages. In June 2021, GitHub migrated all images uploaded to GitHub Packages’ Docker Registry (at the domain ‘docker.pkg.github.com
‘) to their Container Registry (at the domain ‘ghcr.io
‘).
This GitHub Packages Container Registry lets users (like Brew) upload (and download) images in accordance with the Open Container Initiative (OCI) Specifications (described above).
Determine Package Name
In this example, let’s say that we want to download the ‘vim
‘ package from the Homebrew project, which is hosted on GitHub Packages.
To start, go the org’s page on GitHub
Click on “Packages”
Type the name of the package that you want to download
This will tell you the name of the package, which you may want to cross-check with formulae.brew.sh
Determine Package Version
To figure out which version (tag) we want to download from GitHub Packages, we need to check the Brew “recipe”. All the brew recipes are found in the Homebrew/homebrew-core repo.
Click the “Formulas” directory, and then click the “v” directory (which holds all the brew formula files for packages that start with the letter ‘v’).
Finally, click the ‘vim.rb
‘ file.
You may just want to download the latest version of vim, but let’s say that we’re using an old machine that can’t be updated beyond MacOS 11.7.10 (Big Sur). If you check the latest recipe, you see entries for sonoma (macOS 12), ventura (macOS 13), and monterey (macOS 14). But big_sur is absent because it isn’t supported.
The easiest way that I’m aware-of to determine what is the latest brew package’s version that supports your OS (without having to fight with the API) is to view the history of the recipe file.
After clicking-through all the previous versions, we can see that big_sur was removed (and not replaced) in a commit on Sep 28, 2023 in commit a153795, which has a commit message “vim: update 9.0.1900_1 bottle”. Therefore, it looks like “big_sur” was no longer supported the vim package version ‘9.0.1900_1’. The version immediately before that is ‘9.0.1900’, and it looks like it does have a package for “big_sur”
Now that we know that we want to download ‘9.0.1900’ version of the ‘vim’ package, we can go back to the GitHub Packages page and find the tag for this version.
Click on “View all tagged versions” and then scroll-down to the tag that corresponds to the version we want (9.0.1900).
Get an Auth Token
Execute the following command to get an authentication token from GitHub Packages.
# get a JSON with an anonymous token curl -so "token.json" "https://ghcr.io/token?service=ghcr.io&scope=<resourcetype>:<component>/<component>/<component>:<action>"; # extract token from JSON token=$(cat token.json | jq -jr ".token")
The above commands will get a free/temporary token that you can use in subsequent API calls. If all went well, there will be no output from these commands. Here’s an example execution
user@disp7456:~$ curl -so "token.json" "https://ghcr.io/token?service=ghcr.io&scope=repository:homebrew/core/go:pull"; user@disp7456:~$ user@disp7456:~$ token=$(cat token.json | jq -jr ".token") user@disp7456:~$
List the Tags
We can list all of the available tags for the ‘vim
‘ package with the ‘GET /v2/<name>/tags/list
‘ API endpoint, as shown in the table above
curl -i -H "Authorization: Bearer ${token}" https://ghcr.io/v2/homebrew/core/<package_name>/tags/list
Here’s an example execution. Note that it affirms the existence of the ‘9.0.1900
‘ tag that we found above.
user@disp7456:~$ curl -i -H "Authorization: Bearer ${token}" https://ghcr.io/v2/homebrew/core/vim/tags/list HTTP/2 200 content-type: application/json docker-distribution-api-version: registry/2.0 link: </v2/homebrew/core/vim/tags/list?last=9.0.1500&n=0>; rel="next" date: Mon, 06 May 2024 01:06:25 GMT content-length: 1164 x-github-request-id: CD4E:29D249:57F4C1F:5A244F1:66382D11 {"name":"homebrew/core/vim","tags":["8.2.2725_1","8.2.2750","8.2.2775","8.2.2800","8.2.2825","8.2.2850","8.2.2875","8.2.2875_1","8.2.2900","8.2.2925","8.2.2950","8.2.2975","8.2.3000","8.2.3025","8.2.3050","8.2.3075","8.2.3100","8.2.3125","8.2.3150","8.2.3175","8.2.3200","8.2.3225","8.2.3250","8.2.3275","8.2.3300","8.2.3325","8.2.3350","8.2.3400","8.2.3450","8.2.3500","8.2.3550","8.2.3600","8.2.3650","8.2.3700","8.2.3750","8.2.3750_1","8.2.3800","8.2.3850","8.2.3900","8.2.3950","8.2.4000","8.2.4050","8.2.4100","8.2.4150","8.2.4200","8.2.4250","8.2.4300","8.2.4300_1","8.2.4350","8.2.4400","8.2.4450","8.2.4500","8.2.4550","8.2.4600","8.2.4650","8.2.4700","8.2.4750","8.2.4800","8.2.4800_1","8.2.4850","8.2.4850-1","8.2.4900","8.2.4950","8.2.5000","8.2.5050","8.2.5100","8.2.5150","8.2.5150_1","9.0.0000","9.0.0050","9.0.0100","9.0.0150","9.0.0150_1","9.0.0200","9.0.0250","9.0.0300","9.0.0350","9.0.0350_1","9.0.0350_2","9.0.0650","9.0.0700","9.0.0750","9.0.0800","9.0.0850","9.0.0900","9.0.0950","9.0.1000","9.0.1050","9.0.1050-1","9.0.1100","9.0.1100_1","9.0.1150","9.0.1200","9.0.1250","9.0.1300","9.0.1350","9.0.1400","9.0.1400-1","9.0.1450","9.0.1500"]} user@disp7456:~$
Download the Manifest
We can download the manifest for the ‘9.0.1900
‘ tag of the ‘vim
‘ package with the ‘GET /v2/<name>/manifests/<reference>
‘ API endpoint, as shown in the table above
curl -o manifest.json -s -H "Authorization: Bearer ${token}" -H 'Accept: application/vnd.oci.image.index.v1+json' https://ghcr.io/v2/homebrew/core/<package_name>/manifests/<tag>
And here’s an example execution that downloads the manifest for the ‘9.0.1900
‘ tag of the ‘vim
‘ package
user@disp7456:~$ curl -o manifest.json -s -H "Authorization: Bearer ${token}" -H 'Accept: application/vnd.oci.image.index.v1+json' https://ghcr.io/v2/homebrew/core/vim/manifests/9.0.1900 user@disp7456:~$ user@disp7456:~$ ls manifest.json user@disp7456:~$
⚠ Note that we MUST specify the Accept header here. If we don’t, then the registry will respond with the following error message
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"OCI index found, but Accept header does not support OCI indexes"}]}
For more information on the OCI media types and their MIME types, see the list of OCI Image Media Types.
Parse the Manifest
The previous step downloaded a file named ‘manifest.json
‘. This ‘manifest.json
‘ file lists all of the many files available for the ‘9.0.1900
‘ version of the ‘vim
‘ package — spanning many OS versions and processor architectures.
In order to proceed and download our actual brew bottle file, we need to extract the ‘sh.brew.bottle.digest
‘ for our target system, which is a sha256sum hash that we’ll use in the ‘GET /v2/<name>/blobs/
‘ API call to actually download the file from the container registry.
There’s only one dictionary in the ‘manifests
‘ array that has a ‘platform
‘ dict with the following keys:
"architecture": "amd64"
, and"os.version": "macOS 11.7"
…And that’s the one we want:
{ "schemaVersion": 2, "manifests": [ ... { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:5b538ff92ab00c3658b152dee240e30f9ffa65d817540b6a461460b02b93ceda", "size": 5357, "platform": { "architecture": "amd64", "os": "darwin", "os.version": "macOS 11.7" }, "annotations": { "org.opencontainers.image.ref.name": "9.0.1900.big_sur", "sh.brew.bottle.digest": "6cbad503034158806227128743d2acc08773c90890cea12efee25c4a53399d02", "sh.brew.bottle.size": "13598246", "sh.brew.tab": "{\"homebrew_version\":\"4.1.11-36-g184efd9\",\"changed_files\":[\"share/vim/vim90/doc/vim2html.pl\",\"share/vim/vim90/tools/efm_filter.pl\",\"share/vim/vim90/tools/efm_perl.pl\",\"share/vim/vim90/tools/pltags.pl\",\"share/vim/vim90/tools/shtags.pl\",\"share/man/da/man1/vim.1\",\"share/man/da/man1/vimtutor.1\",\"share/man/da.ISO8859-1/man1/vim.1\",\"share/man/da.ISO8859-1/man1/vimtutor.1\",\"share/man/da.UTF-8/man1/vim.1\",\"share/man/da.UTF-8/man1/vimtutor.1\",\"share/man/de/man1/vim.1\",\"share/man/de.ISO8859-1/man1/vim.1\",\"share/man/de.UTF-8/man1/vim.1\",\"share/man/fr/man1/evim.1\",\"share/man/fr/man1/vim.1\",\"share/man/fr/man1/vimtutor.1\",\"share/man/fr.ISO8859-1/man1/evim.1\",\"share/man/fr.ISO8859-1/man1/vim.1\",\"share/man/fr.ISO8859-1/man1/vimtutor.1\",\"share/man/fr.UTF-8/man1/evim.1\",\"share/man/fr.UTF-8/man1/vim.1\",\"share/man/fr.UTF-8/man1/vimtutor.1\",\"share/man/it/man1/evim.1\",\"share/man/it/man1/vim.1\",\"share/man/it/man1/vimtutor.1\",\"share/man/it.ISO8859-1/man1/evim.1\",\"share/man/it.ISO8859-1/man1/vim.1\",\"share/man/it.ISO8859-1/man1/vimtutor.1\",\"share/man/it.UTF-8/man1/evim.1\",\"share/man/it.UTF-8/man1/vim.1\",\"share/man/it.UTF-8/man1/vimtutor.1\",\"share/man/ja/man1/evim.1\",\"share/man/ja/man1/vim.1\",\"share/man/ja/man1/vimtutor.1\",\"share/man/man1/evim.1\",\"share/man/man1/vim.1\",\"share/man/man1/vimtutor.1\",\"share/man/pl/man1/evim.1\",\"share/man/pl/man1/vim.1\",\"share/man/pl/man1/vimtutor.1\",\"share/man/pl.ISO8859-2/man1/evim.1\",\"share/man/pl.ISO8859-2/man1/vim.1\",\"share/man/pl.ISO8859-2/man1/vimtutor.1\",\"share/man/pl.UTF-8/man1/evim.1\",\"share/man/pl.UTF-8/man1/vim.1\",\"share/man/pl.UTF-8/man1/vimtutor.1\",\"share/man/ru.KOI8-R/man1/evim.1\",\"share/man/ru.KOI8-R/man1/vim.1\",\"share/man/ru.KOI8-R/man1/vimtutor.1\",\"share/man/ru.UTF-8/man1/evim.1\",\"share/man/ru.UTF-8/man1/vim.1\",\"share/man/ru.UTF-8/man1/vimtutor.1\",\"share/man/tr/man1/evim.1\",\"share/man/tr/man1/vim.1\",\"share/man/tr/man1/vimtutor.1\",\"share/man/tr.ISO8859-9/man1/evim.1\",\"share/man/tr.ISO8859-9/man1/vim.1\",\"share/man/tr.ISO8859-9/man1/vimtutor.1\",\"share/man/tr.UTF-8/man1/evim.1\",\"share/man/tr.UTF-8/man1/vim.1\",\"share/man/tr.UTF-8/man1/vimtutor.1\",\"share/vim/vim90/filetype.vim\"],\"source_modified_time\":1694864306,\"compiler\":\"clang\",\"runtime_dependencies\":[{\"full_name\":\"gettext\",\"version\":\"0.21.1\",\"declared_directly\":true},{\"full_name\":\"libsodium\",\"version\":\"1.0.18\",\"declared_directly\":true},{\"full_name\":\"lua\",\"version\":\"5.4.6\",\"declared_directly\":true},{\"full_name\":\"ncurses\",\"version\":\"6.4\",\"declared_directly\":true},{\"full_name\":\"ca-certificates\",\"version\":\"2023-08-22\",\"declared_directly\":false},{\"full_name\":\"openssl@3\",\"version\":\"3.1.2\",\"declared_directly\":false},{\"full_name\":\"berkeley-db\",\"version\":\"18.1.40\",\"declared_directly\":false},{\"full_name\":\"gdbm\",\"version\":\"1.23\",\"declared_directly\":false},{\"full_name\":\"perl\",\"version\":\"5.36.1\",\"declared_directly\":true},{\"full_name\":\"mpdecimal\",\"version\":\"2.5.1\",\"declared_directly\":false},{\"full_name\":\"readline\",\"version\":\"8.2.1\",\"declared_directly\":false},{\"full_name\":\"sqlite\",\"version\":\"3.43.1\",\"declared_directly\":false},{\"full_name\":\"xz\",\"version\":\"5.4.4\",\"declared_directly\":false},{\"full_name\":\"python@3.11\",\"version\":\"3.11.5\",\"declared_directly\":true},{\"full_name\":\"libyaml\",\"version\":\"0.2.5\",\"declared_directly\":false},{\"full_name\":\"ruby\",\"version\":\"3.2.2\",\"declared_directly\":true}],\"arch\":\"x86_64\",\"built_on\":{\"os\":\"Macintosh\",\"os_version\":\"macOS 11.7\",\"cpu_family\":\"penryn\",\"xcode\":\"13.2.1\",\"clt\":\"13.2.0.0.1.1638488800\",\"preferred_perl\":\"5.30\"}}" } }, ... ], "annotations": { "com.github.package.type": "homebrew_bottle", "org.opencontainers.image.created": "2023-09-16", "org.opencontainers.image.description": "Vi 'workalike' with many additional features", "org.opencontainers.image.documentation": "https://formulae.brew.sh/formula/vim", "org.opencontainers.image.license": "Vim", "org.opencontainers.image.ref.name": "9.0.1900", "org.opencontainers.image.revision": "a78712b62dd7fba03243de22c20e4858d0fd1802", "org.opencontainers.image.source": "https://github.com/homebrew/homebrew-core/blob/a78712b62dd7fba03243de22c20e4858d0fd1802/Formula/v/vim.rb", "org.opencontainers.image.title": "vim", "org.opencontainers.image.url": "https://www.vim.org/", "org.opencontainers.image.vendor": "homebrew", "org.opencontainers.image.version": "9.0.1900" } }
And, voilà, the block above shows us the ‘sh.brew.bottle.digest
‘ that we need in the next step
You can either pick through the file in your fav editor of-choice, but I found (when downloading many files from GitHub Packages) it was much easier to use ‘jq
‘
cat manifest.json | jq '.manifests[] | select(.platform."os.version" | startswith("macOS <version_num>")) | select(.platform.architecture=="<arch>")' | jq -r '.annotations."sh.brew.bottle.digest"'
For example, here’s an execution that uses ‘jq
‘ to extract the bottle digest hash for the blob for macOS 11 on a system with an amd64 processor.
user@disp7456:~$ cat manifest.json | jq '.manifests[] | select(.platform."os.version" | startswith("macOS 11")) | select(.platform.architecture=="amd64")' | jq -r '.annotations."sh.brew.bottle.digest"' 6cbad503034158806227128743d2acc08773c90890cea12efee25c4a53399d02 user@disp7456:~$
Download the file
Now, we can finally download our file if we just pass the ‘sh.brew.bottle.digest
‘ hash found above into the GET /v2/<name>/blobs/
‘ API endpoint, as shown in the table above
curl -Lo <package_name>-<package_version>.bottle.tar.gz -H "Authorization: Bearer ${token}" -H 'Accept: application/vnd.oci.image.layer.v1.tar+gzip' https://ghcr.io/v2/homebrew/core/<package_name>/blobs/sha256:<sh.brew.bottle.digest>
For example, the following command downloads ‘vim-9.0.1900.bottle.tar.gz
‘
user@disp7456:~$ curl -Lo vim-9.0.1900.bottle.tar.gz -H "Authorization: Bearer ${token}" -H 'Accept: application/vnd.oci.image.layer.v1.tar+gzip' https://ghcr.io/v2/homebrew/core/vim/blobs/sha256:6cbad503034158806227128743d2acc08773c90890cea12efee25c4a53399d02 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 12.9M 100 12.9M 0 0 1450k 0 0:00:09 0:00:09 --:--:-- 2798k user@disp7456:~$ user@disp7456:~$ ls vim-9.0.1900.bottle.tar.gz user@disp7456:~$
Install the file
Finally, you can now transfer the file ‘vim-9.0.1900.bottle.tar.gz
‘ to your macOS system and install it with brew.
brew reinstall --verbose --debug path/to/<package_name>-<package_version>.tar.gz
Here’s an example execution
user@host ~ % /usr/local/bin/brew reinstall --debug --verbose build/deps/vim-9.0.1900.bottle.tar.gz /usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FromNameLoader): loading vim /usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FromBottleLoader): loading build/deps/vim-9.0.1900.bottle.tar.gz ... ␛[34m==>␛[0m ␛[1mSummary␛[0m 🍺 /usr/local/Cellar/vim/9.0.1900: 2,220 files, 40.3MB ... user@host ~ %
The package ‘vim
‘ is now installed.
Why?
I wrote this article because many, many folks have inquired about how to manually download files from OCI registries on the Internet, but their simple queries are usually returned with a barrage of useless counter-questions: why the heck would you want to do that!?!
The answer is varied.
Some people need to get files onto a restricted environment. Either their org doesn’t grant them permission to install software on the machine, or the system has firewall-restricted internet access — or doesn’t have internet access at all.
3TOFU
Personally, the reason that I wanted to be able to download files from an OCI registry was for 3TOFU.
Unfortunately, most apps using OCI registries are extremely insecure. Docker, for example, will happily download malicious images. By default, it doesn’t do any authenticity verifications on the payloads it downloaded. Even if you manually enable DCT, there’s loads of pending issues with it.
Likewise, the macOS package manager brew has this same problem: it will happily download and install malicious code, because it doesn’t use cryptography to verify the authenticity of anything that it downloads. This introduces watering hole vulnerabilities when developers use brew to install dependencies in their CI pipelines.
My solution to this? 3TOFU. And that requires me to be able to download the file (for verification) on three distinct linux VMs using curl or wget.
⚠ NOTE: 3TOFU is an approach to harm reduction.
It is not wise to download and run binaries or code whose authenticity you cannot verify using a cryptographic signature from a key stored offline. However, sometimes we cannot avoid it. If you’re going to proceed with running untrusted code, then following a 3TOFU procedure may reduce your risk, but it’s better to avoid running unauthenticated code if at all possible.
Registry (ab)use
Container registries were created in 2013 to provide a clever & complex solution to a problem: how to package and serve multiple versions of simplified sources to various consumers spanning multiple operating systems and architectures — while also packaging them into small, discrete “layers”.
However, if your project is just serving simple files, then the only thing gained by uploading them to a complex system like a container registry is headaches. Why do developers do this?
In the case of brew, their free hosing provider (JFrog’s Bintray) shutdown in 2021. Brew was already hosting their code on GitHub, so I guess someone looked at “GitHub Packages” and figured it was a good (read: free) replacement.
Many developers using Container Registries don’t need the complexity, but — well — they’re just using it as a free place for their FOSS project to store some files, man.
Further Reading
- OCI Distribution Spec v1.1.0
- Docker Registry HTTP API V2 Protocol Specification
- Docker Image Specification v1.0.0
- List of OCI Media Types
- Docker Registry (Documentation)
- GitHub Packages (Documentation)
- CNCF Distribution Specification (GitHub)
- OCI Distribution Specification (GitHub)
- Request to add the not-yet-existing OCI Auth Specification (GitHub)
- Moby Project (GitHub)
- My GitHub Ticket learning how to manually download brew package dependencies from GitHub Packages for 3TOFU with BusKill
- More info about the Docker Image Spec (V1 and V2) – https://matrix.ai/blog/docker-image-specification-v1-vs-v2/
- Cool tool that serves as a proxy for downloading docker images for you – DockerImageSave
- Script for downloading “frozen” images from docker – download-frozen-image-v2.sh
- Python tool to download docker images – docker-drag
- My article on 3TOFU
Related Posts
Hi, I’m Michael Altfield. I write articles about opsec, privacy, and devops ➡
I am using ubuntu machine with non direct internet and trying to download docker image with curl.
I’ve tried this post but i got the manifest with media_type ‘application/vnd.oci.image.manifest.v1+json’.I don’t know how to parse it and get the right image.
The script I wroted:
# get auth token
curl -so "token.json" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/hitch:pull"
# download Minifest
token=$(cat token.json | jq -jr ".token")
curl -o manifest.json -s -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/hitch/manifests/1.8.0-1
The manifest file:
{
"manifests": [
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "amd64",
"org.opencontainers.image.base.digest": "sha256:f94d0d6faaec313eed26c957e7aef75225f0bf8b8376ef83082ed0d8792255f7",
"org.opencontainers.image.base.name": "debian:bullseye-slim",
"org.opencontainers.image.created": "2024-09-05T00:06:53Z",
"org.opencontainers.image.revision": "3fe55c4296018b04c7e1d2efdfcadce9c4017e0a",
"org.opencontainers.image.source": "https://github.com/varnish/docker-hitch.git#3fe55c4296018b04c7e1d2efdfcadce9c4017e0a:.",
"org.opencontainers.image.url": "https://hub.docker.com/_/hitch",
"org.opencontainers.image.version": "1"
},
"digest": "sha256:a640f78641befbe8a4696dde47c51aeadfa1e5e81087a7534982b6bb4d76f54a",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "amd64",
"os": "linux"
},
"size": 1707
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "amd64",
"vnd.docker.reference.digest": "sha256:a640f78641befbe8a4696dde47c51aeadfa1e5e81087a7534982b6bb4d76f54a",
"vnd.docker.reference.type": "attestation-manifest"
},
"digest": "sha256:018537a7b555d7a2b5498a24b7d974cbef39a648b8e5c724548efb802d19239c",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "unknown",
"os": "unknown"
},
"size": 841
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "arm32v7",
"org.opencontainers.image.base.digest": "sha256:ac122a7511a9d58028712a4c28707d15053f3c011c431056c8858a93af0cd906",
"org.opencontainers.image.base.name": "debian:bullseye-slim",
"org.opencontainers.image.created": "2024-09-05T04:20:03Z",
"org.opencontainers.image.revision": "3fe55c4296018b04c7e1d2efdfcadce9c4017e0a",
"org.opencontainers.image.source": "https://github.com/varnish/docker-hitch.git#3fe55c4296018b04c7e1d2efdfcadce9c4017e0a:.",
"org.opencontainers.image.url": "https://hub.docker.com/_/hitch",
"org.opencontainers.image.version": "1"
},
"digest": "sha256:1fc0c07464828a8461a4f1713179de2262c251e1701f9faea7caefbc31d28507",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v7"
},
"size": 1709
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "arm32v7",
"vnd.docker.reference.digest": "sha256:1fc0c07464828a8461a4f1713179de2262c251e1701f9faea7caefbc31d28507",
"vnd.docker.reference.type": "attestation-manifest"
},
"digest": "sha256:40183653cdcced07c48d0dd598615ab7f298ea2b41a105787176834c0077a3f7",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "unknown",
"os": "unknown"
},
"size": 841
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "arm64v8",
"org.opencontainers.image.base.digest": "sha256:fd83fda9b857c7a0dac79eb937029e8554e2553bce0c7590af73813e86f71a5a",
"org.opencontainers.image.base.name": "debian:bullseye-slim",
"org.opencontainers.image.created": "2024-09-05T12:13:27Z",
"org.opencontainers.image.revision": "3fe55c4296018b04c7e1d2efdfcadce9c4017e0a",
"org.opencontainers.image.source": "https://github.com/varnish/docker-hitch.git#3fe55c4296018b04c7e1d2efdfcadce9c4017e0a:.",
"org.opencontainers.image.url": "https://hub.docker.com/_/hitch",
"org.opencontainers.image.version": "1"
},
"digest": "sha256:7a3867123ef8fd60326d01e910c401816a8d8a0bad120982744be24a19e6c5e1",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "arm64",
"os": "linux",
"variant": "v8"
},
"size": 1709
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "arm64v8",
"vnd.docker.reference.digest": "sha256:7a3867123ef8fd60326d01e910c401816a8d8a0bad120982744be24a19e6c5e1",
"vnd.docker.reference.type": "attestation-manifest"
},
"digest": "sha256:6af2bc5c199033596a7bcc472d215babe95f289b3b90151f2c41967352304827",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "unknown",
"os": "unknown"
},
"size": 841
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "i386",
"org.opencontainers.image.base.digest": "sha256:0f810568366d5e9730afa52e94e0ad3855265e44f9ff13b7545fbd16c9b4e029",
"org.opencontainers.image.base.name": "debian:bullseye-slim",
"org.opencontainers.image.created": "2024-09-05T00:06:55Z",
"org.opencontainers.image.revision": "3fe55c4296018b04c7e1d2efdfcadce9c4017e0a",
"org.opencontainers.image.source": "https://github.com/varnish/docker-hitch.git#3fe55c4296018b04c7e1d2efdfcadce9c4017e0a:.",
"org.opencontainers.image.url": "https://hub.docker.com/_/hitch",
"org.opencontainers.image.version": "1"
},
"digest": "sha256:fef8e6fafab48181903336af38823a04b89bff350074a4502947e54df6a54c87",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "386",
"os": "linux"
},
"size": 1706
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "i386",
"vnd.docker.reference.digest": "sha256:fef8e6fafab48181903336af38823a04b89bff350074a4502947e54df6a54c87",
"vnd.docker.reference.type": "attestation-manifest"
},
"digest": "sha256:5d39da2bc3cb1f979ae655c82bab4d10238efa8e7f1f5f34669887b1d29939b0",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "unknown",
"os": "unknown"
},
"size": 841
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "ppc64le",
"org.opencontainers.image.base.digest": "sha256:a47b5afb1c8d91095b132d0cb0cac44fe04cb5ef00dfb324114eb0a7fa90c6ab",
"org.opencontainers.image.base.name": "debian:bullseye-slim",
"org.opencontainers.image.created": "2024-09-05T00:23:24Z",
"org.opencontainers.image.revision": "3fe55c4296018b04c7e1d2efdfcadce9c4017e0a",
"org.opencontainers.image.source": "https://github.com/varnish/docker-hitch.git#3fe55c4296018b04c7e1d2efdfcadce9c4017e0a:.",
"org.opencontainers.image.url": "https://hub.docker.com/_/hitch",
"org.opencontainers.image.version": "1"
},
"digest": "sha256:1bdcd7bcd169b8de699dae282ed9132427b99bb8f3bfbac1cacbc2ce1d5b104f",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "ppc64le",
"os": "linux"
},
"size": 1709
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "ppc64le",
"vnd.docker.reference.digest": "sha256:1bdcd7bcd169b8de699dae282ed9132427b99bb8f3bfbac1cacbc2ce1d5b104f",
"vnd.docker.reference.type": "attestation-manifest"
},
"digest": "sha256:930b2a56783983d94990c5d8eb69841e1848a966f490e074c6f503c302919ed6",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "unknown",
"os": "unknown"
},
"size": 841
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "s390x",
"org.opencontainers.image.base.digest": "sha256:f36860cbae1d277d81cfee14f4fbb21b86c2e413b19bfc118225225133cc4ebf",
"org.opencontainers.image.base.name": "debian:bullseye-slim",
"org.opencontainers.image.created": "2024-09-05T02:39:47Z",
"org.opencontainers.image.revision": "3fe55c4296018b04c7e1d2efdfcadce9c4017e0a",
"org.opencontainers.image.source": "https://github.com/varnish/docker-hitch.git#3fe55c4296018b04c7e1d2efdfcadce9c4017e0a:.",
"org.opencontainers.image.url": "https://hub.docker.com/_/hitch",
"org.opencontainers.image.version": "1"
},
"digest": "sha256:d91204d37425e40a174c327556e101e756d3baae8af570d95d555175f76557b2",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "s390x",
"os": "linux"
},
"size": 1707
},
{
"annotations": {
"com.docker.official-images.bashbrew.arch": "s390x",
"vnd.docker.reference.digest": "sha256:d91204d37425e40a174c327556e101e756d3baae8af570d95d555175f76557b2",
"vnd.docker.reference.type": "attestation-manifest"
},
"digest": "sha256:03077a449c74e88e0f899654f6a39799feba8a345a61c14977d87473d71d3cc8",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"platform": {
"architecture": "unknown",
"os": "unknown"
},
"size": 841
}
],
"mediaType": "application/vnd.oci.image.index.v1+json",
"schemaVersion": 2
}
Tried on each digests and show me errors.
curl -o "test.tar" -H "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/library/hitch/blobs/sha256:f94d0d6faaec313eed26c957e7aef75225f0bf8b8376ef83082ed0d8792255f7"
cat test.tar
{"errors":[{"code":"BLOB_UNKNOWN","message":"blob unknown to registry","detail":"sha256:f94d0d6faaec313eed26c957e7aef75225f0bf8b8376ef83082ed0d8792255f7"}]}
Thank you for watching.
Hi 🙂 It appears that the docker server is giving you the wrong manifest type! Check the end of the manifest:
"mediaType": "application/vnd.oci.image.index.v1+json",
"schemaVersion": 2
}
As the guide states, we’re expecting an “image manifest” (
vnd.docker.distribution.manifest.v1+json
), but this gave us an “image index” (vnd.oci.image.index.v1+json
).The documentation for an “image index” can be found here: https://github.com/opencontainers/image-spec/blob/2d95dde8615f9ed8a911873f8eb72a8582af577a/image-index.md
> The image index is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. While the use of an image index is OPTIONAL for image providers, image consumers SHOULD be prepared to process them.
>
> This section defines the application/vnd.oci.image.index.v1+json media type.
My understanding is that an “image index” is returned when there’s many distinct “image manifests” for different platforms (eg linux vs windows or arm vs amd64 processors).
So this just adds a step where you parse the image index and then download the image manifest. For example, to get the “linux” + “arm64” image manifest, we need to download the image manifest blob (
sha256:a640f78641befbe8a4696dde47c51aeadfa1e5e81087a7534982b6bb4d76f54a
)curl -o manifest.json -s -H 'Accept: application/vnd.docker.distribution.manifest.v1+json' -H "Authorization: Bearer ${token}" https://registry-1.docker.io/v2/library/hitch/manifests/sha256:a640f78641befbe8a4696dde47c51aeadfa1e5e81087a7534982b6bb4d76f54a
What’s strange, though, is that this query returns a ‘
application/vnd.oci.image.index.v2+json
‘ media type.I opened a bug report about this for Docker Hub:
* https://github.com/docker/hub-feedback/issues/2410
Thank you for this detailed guide on using curl and wget to download container images. It’s always refreshing to come across clear and concise tutorials that make complex tasks easier to understand. Your step-by-step approach truly helps streamline the process for users, and I appreciate the effort you put into making this accessible. Looking forward to more helpful content like this!
Best regards,
Jaseph