Skip to content

Commit f3f718b

Browse files
WwwsylviaWwwsylvia
authored
docs: add concepts of targets (#917)
Resolves: #814 Signed-off-by: Lixia (Sylvia) Lei <[email protected]>
1 parent 2ef884f commit f3f718b

File tree

1 file changed

+375
-0
lines changed

1 file changed

+375
-0
lines changed

‎docs/Targets.md

Lines changed: 375 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,375 @@
1+
# Targets and Content Stores in `oras-go` v2
2+
3+
> [!IMPORTANT]
4+
> Prerequisite reading: [Modeling Artifact](./Modeling-Artifacts.md)
5+
6+
In `oras-go` v2, artifacts are modeled as [Directed Acyclic Graphs (DAGs)](https://en.wikipedia.org/wiki/Directed_acyclic_graph) stored in [Content-Addressable Storages (CASs)](https://en.wikipedia.org/wiki/Content-addressable_storage). Each node in the graph represents a [descriptor](https://github.com/opencontainers/image-spec/blob/v1.1.1/descriptor.md) of the content, which must include the following three properties:
7+
8+
- `mediaType`: The media type of the referenced content.
9+
- `digest`: The digest of the targeted content.
10+
- `size`: The size, in bytes, of the raw content.
11+
12+
An example of a descriptor for an [OCI Image Manifest](https://github.com/opencontainers/image-spec/blob/v1.1.1/manifest.md) is shown below:
13+
14+
```json
15+
{
16+
"mediaType": "application/vnd.oci.image.manifest.v1+json",
17+
"size": 7682,
18+
"digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270"
19+
}
20+
```
21+
22+
## Interfaces
23+
24+
`oras-go` v2 defines four major interfaces based on the graph-modeling concepts and descriptors: [`Storage`](#storage), [`GraphStorage`](#graphstorage), [`Target`](#target), and [`GraphTarget`](#graphtarget).
25+
26+
### Storage
27+
28+
The [`Storage`](https://pkg.go.dev/oras.land/oras-go/v2/content#Storage) interface represents a content-addressable storage (CAS) where content is accessed via descriptors. It provides the following functions:
29+
30+
- `Fetch`: Retrieves the content identified by the descriptor from the CAS.
31+
- `Exists`: Checks whether the described content is present in the CAS.
32+
- `Push`: Adds content matching the expected descriptor to the CAS.
33+
34+
For example, consider the following graph stored in a `Storage`, where node names are aliases for descriptors:
35+
36+
```mermaid
37+
graph TD;
38+
39+
M0["Manifest m0"]--config-->Blob0["Blob b0"]
40+
M0--layers-->Blob1["Blob b1"]
41+
M0--layers-->Blob2["Blob b2"]
42+
```
43+
44+
The effects of the `Fetch` and `Exists` functions would be:
45+
46+
```
47+
Fetch(m0) == content_m0
48+
49+
Exists(b0) == true
50+
Exists(b3) == false
51+
```
52+
53+
If a new blob `b3` is pushed to the storage, the graph would update as follows:
54+
55+
```mermaid
56+
graph TD;
57+
58+
M0["Manifest m0"]--config-->Blob0["Blob b0"]
59+
M0--layers-->Blob1["Blob b1"]
60+
M0--layers-->Blob2["Blob b2"]
61+
Blob3["Blob b3"]
62+
```
63+
64+
### GraphStorage
65+
66+
The [`GraphStorage`](https://pkg.go.dev/oras.land/oras-go/v2/content#GraphStorage) interface extends [`Storage`](#storage) by adding support for predecessor finding. It provides the following functions:
67+
68+
- `Fetch`
69+
- `Exists`
70+
- `Push`
71+
- **`Predecessors`**: Finds the nodes directly pointing to a given node in the graph.
72+
73+
For the [same graph](#storage), the `Predecessors` function would act as follows:
74+
75+
```
76+
Predecessors(b0) == [m0]
77+
Predecessors(m0) == []
78+
```
79+
80+
### Target
81+
82+
The [`Target`](https://pkg.go.dev/oras.land/oras-go/v2#Target) interface represents a CAS with tagging capability. It provides the following functions:
83+
84+
- `Fetch`
85+
- `Exists`
86+
- `Push`
87+
- **`Resolve`**: Resolves a tag string to a descriptor.
88+
- **`Tag`**: Associates a descriptor with a tag string.
89+
90+
For example, consider a graph stored in a Target where `m0` is associated with two tags, `"foo"` and `"bar"`:
91+
92+
```mermaid
93+
graph TD;
94+
95+
M0["Manifest m0"]--config-->Blob0["Blob b0"]
96+
M0--layers-->Blob1["Blob b1"]
97+
M0--layers-->Blob2["Blob b2"]
98+
99+
TagFoo>"Tag: foo"]-.->M0
100+
TagBar>"Tag: bar"]-.->M0
101+
```
102+
103+
The effects of the `Resolve` function would be:
104+
105+
```
106+
Resolve("foo") == m0
107+
Resolve("bar") == m0
108+
Resolve("hello") == nil
109+
```
110+
111+
If a new tag `"v1"` is added to `m0`, the graph would update as follows:
112+
113+
```mermaid
114+
graph TD;
115+
116+
M0["Manifest m0"]--config-->Blob0["Blob b0"]
117+
M0--layers-->Blob1["Blob b1"]
118+
M0--layers-->Blob2["Blob b2"]
119+
120+
TagFoo>"Tag: foo"]-.->M0
121+
TagBar>"Tag: bar"]-.->M0
122+
TagV1>"Tag: v1"]-.->M0
123+
```
124+
125+
### GraphTarget
126+
127+
The [`GraphTarget`](https://pkg.go.dev/oras.land/oras-go/v2#GraphTarget) interface combines the capabilities of [`GraphStorage`](#graphstorage) and [`Target`](#target). It provides the following functions:
128+
129+
- `Fetch`
130+
- `Exists`
131+
- `Push`
132+
- `Resolve`
133+
- `Tag`
134+
- `Predecessors`
135+
136+
## Content Stores
137+
138+
In `oras-go` v2, a content store is an implementation of the [`Target`](#target) interface—specifically, the [`GraphTarget`](#graphtarget) interface.
139+
140+
The library provides four built-in content stores:
141+
142+
- [Memory Store](#memory-store): Stores everything in memory.
143+
- [OCI Store](#oci-store): Stores content in the [OCI-Image layout](https://github.com/opencontainers/image-spec/blob/v1.1.1/image-layout.md) on the file system.
144+
- [File Store](#file-store): Stores location-addressable content on the file system.
145+
- [Repository Store](#repository-store): Communicates with remote artifact repositories (e.g. `ghcr.io`, `docker.io`).
146+
147+
### Memory Store
148+
149+
The memory store, available in the [`content/memory`](https://pkg.go.dev/oras.land/oras-go/v2/content/memory) package, stores all content in memory, where each blob's content is mapped to its corresponding descriptor.
150+
151+
> [!TIP]
152+
> The memory store is often used for building and storing artifacts in memory before copying them to other stores, such as remote repositories.
153+
154+
### OCI Store
155+
156+
The OCI store, available in the [`content/oci`](https://pkg.go.dev/oras.land/oras-go/v2/content/oci) package, follows the OCI [`image-spec v1.1.1`](https://github.com/opencontainers/image-spec/blob/v1.1.1/image-layout.md) to store blob content on the file system.
157+
158+
For example, consider an artifact and its signature represented by the following graph:
159+
160+
```mermaid
161+
graph TD;
162+
163+
SignatureManifest["Signature Manifest<br>(sha256:e5727b...)"]--subject-->Manifest
164+
SignatureManifest--config-->Config
165+
SignatureManifest--layers-->SignatureBlob["Signature blob<br>(sha256:37f884)"]
166+
167+
Manifest["Manifest<br>(sha256:314c7f...)"]--config-->Config["Config blob<br>(sha256:44136f...)"]
168+
Manifest--layers-->Layer0["Layer blob 0<br>(sha256:b5bb9d...)"]
169+
Manifest--layers-->Layer1["Layer blob 1<br>(sha256:7d865e...)"]
170+
```
171+
172+
The corresponding directory structure on the file system would look like this:
173+
174+
```bash
175+
$ tree repo
176+
repo/
177+
├── blobs
178+
│   └── sha256
179+
│   ├── 314c7f20dd44ee1cca06af399a67f7c463a9f586830d630802d9e365933da9fb
180+
│   ├── 37f88486592fd90ace303ee38f8d1ff698193e76c76d3c1fef8627a39e677696
181+
│   ├── 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
182+
│   ├── 7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
183+
│   ├── b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
184+
│   └── e5727bebbcbbd9996446c34622ca96af67a54219edd58d261112f1af06e2537c
185+
├── index.json
186+
├── ingest
187+
└── oci-layout
188+
```
189+
190+
In the layout:
191+
192+
- All content, whether manifests or layer blobs, are all placed under the `blobs` directory.
193+
- The path to each piece of content is determined by its digest.
194+
- The `index.json` file is an [Image Index](https://github.com/opencontainers/image-spec/blob/v1.1.1/image-layout.md#index-example) JSON object. It serves as the entry point for the graph and provides tagging functionality.
195+
- The `ingest` directory is used temporarily during blob processing. It can be safely removed after the push operation and should be cleaned up before creating a tar archive of the OCI layout. This directory is not defined in the OCI specification.
196+
- The `oci-layout` file is a marker of the base of the OCI Layout.
197+
198+
The OCI Layout offers several advantages:
199+
200+
- It is fully compliant with OCI `image-spec v1.1.1`, ensuring compatibility with tools beyond ORAS.
201+
- Its clean and straightforward structure makes it easy to manage and replicate.
202+
203+
> [!TIP]
204+
> The OCI store is a practical option for maintaining a local copy of a remote repository.
205+
206+
### File Store
207+
208+
The file store, available in the [`content/file`](https://pkg.go.dev/oras.land/oras-go/v2/content/file) package, supports both content-addressable and location-addressable storage. It is designed for packaging arbitrary files or directories and allows adding them directly from the local file system.
209+
210+
When a file or directory is added, the file store creates a descriptor containing annotations with essential metadata. The process differs depending on whether a file or a directory is added:
211+
212+
- **File Addition**: When a file is added, its contents are stored as a blob. A descriptor is generated from the blob with an `"org.opencontainers.image.title"` annotation indicating the original file name.
213+
- **Directory Addition**: When a directory is added, it is first tar-archived and compressed into a blob. The descriptor generated for a directory includes multiple annotations:
214+
- `"org.opencontainers.image.title"`: Indicates the original directory name.
215+
- `"io.deis.oras.content.digest"`**(ORAS-specific)**: Represents the digest of the tar'ed content before compression.
216+
- `"io.deis.oras.content.unpack"`**(ORAS-specific)**: A flag indicating that the blob represents a directory that needs to be decompressed and un-tar'ed.
217+
218+
For example, consider the following directory structure on disk:
219+
220+
```bash
221+
$ tree
222+
.
223+
├── hello.txt
224+
└── mydir
225+
├── bar.txt
226+
└── foo.txt
227+
228+
$ cat hello.txt
229+
hello
230+
231+
$ cat mydir/foo.txt
232+
foo
233+
234+
$ cat mydir/bar.txt
235+
bar
236+
```
237+
238+
Adding the file `hello.txt` results in a blob with a descriptor similar to this:
239+
240+
```json
241+
{
242+
"mediaType": "application/vnd.custom",
243+
"digest": "sha256:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03",
244+
"size": 6,
245+
"annotations": {
246+
"org.opencontainers.image.title": "hello.txt"
247+
}
248+
}
249+
```
250+
251+
Adding the directory `mydir` results in a tar-archived and compressed blob with a descriptor similar to this:
252+
253+
```json
254+
{
255+
"mediaType": "application/vnd.custom.tar+gzip",
256+
"digest": "sha256:b14fa80f5afd3822dce52a711566586c8f89e5dc211e4b0d2819f219b102fe7a",
257+
"size": 164,
258+
"annotations": {
259+
"io.deis.oras.content.digest": "sha256:9d418a0549e5bc45d195aacec889d30e520b9f2bd9feab57d57de6d9cdd66172",
260+
"io.deis.oras.content.unpack": "true",
261+
"org.opencontainers.image.title": "mydir"
262+
}
263+
}
264+
```
265+
266+
To create an artifact, a manifest needs to be [packed](https://pkg.go.dev/oras.land/oras-go/v2#PackManifest) to reference the two blobs and will also be stored in the file store. The manifest content might look like this:
267+
268+
```json
269+
{
270+
"schemaVersion": 2,
271+
"mediaType": "application/vnd.oci.image.manifest.v1+json",
272+
"artifactType": "application/vnd.example",
273+
"config": {
274+
"mediaType": "application/vnd.oci.empty.v1+json",
275+
"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
276+
"size": 2,
277+
"data": "e30="
278+
},
279+
"layers": [
280+
{
281+
"mediaType": "application/vnd.custom",
282+
"digest": "sha256:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03",
283+
"size": 6,
284+
"annotations": {
285+
"org.opencontainers.image.title": "hello.txt"
286+
}
287+
},
288+
{
289+
"mediaType": "application/vnd.custom.tar+gzip",
290+
"digest": "sha256:b14fa80f5afd3822dce52a711566586c8f89e5dc211e4b0d2819f219b102fe7a",
291+
"size": 164,
292+
"annotations": {
293+
"io.deis.oras.content.digest": "sha256:9d418a0549e5bc45d195aacec889d30e520b9f2bd9feab57d57de6d9cdd66172",
294+
"io.deis.oras.content.unpack": "true",
295+
"org.opencontainers.image.title": "mydir"
296+
}
297+
}
298+
],
299+
"annotations": {
300+
"org.opencontainers.image.created": "2025-03-07T08:34:23Z"
301+
}
302+
}
303+
```
304+
305+
In the file store, blobs with names are location-addressed by file paths, while other content (e.g., manifests and config blobs) is maintained in fallback storage. By default, the fallback storage is a limited in-memory CAS store.
306+
307+
For the above example, the graph stored in the file store would be like this:
308+
309+
```mermaid
310+
graph TD;
311+
312+
Manifest["Manifest<br>(in memory)"]--config-->Config["Config blob<br>(in memory)"]
313+
Manifest--layers-->Layer0["hello.txt<br>(on disk)"]
314+
Manifest--layers-->Layer1["mydir<br>(on disk)"]
315+
```
316+
317+
Unlike the OCI store, the file store only persists named contents (e.g., `hello.txt` and `mydir`) on disk, while all metadata is stored in memory.
318+
319+
> [!IMPORTANT]
320+
> Once the file store is terminated, it cannot be restored to its original state from the file system.
321+
322+
### Repository Store
323+
324+
The repository store, available in the [`registry/remote`](https://pkg.go.dev/oras.land/oras-go/v2/registry/remote) package, communicates with remote artifact repositories using APIs defined in the [OCI distribution-spec v1.1.1](https://github.com/opencontainers/distribution-spec/blob/v1.1.1/spec.md).
325+
326+
Unlike other content stores, the repository store handles manifests and non-manifest blobs separately. This distinction exists because the URI paths for manifests and blobs differ, with manifests accessed via `/v2/<name>/manifests/` and blobs accessed via `/v2/<name>/blobs/`.
327+
328+
The repository store manages manifests through the `ManifestStore` sub-store and handles blobs through the `BlobStore` sub-store. It automatically determines which sub-store to use based on the media type specified in the descriptor.
329+
330+
It is important to note that, the `ManifestStore` implements the `Predecessors` function based on the [Referrers API](https://github.com/opencontainers/distribution-spec/blob/v1.1.1/spec.md#listing-referrers) or, when the Referrers API is unavailable, [`Referrers Tag Schema`](https://github.com/opencontainers/distribution-spec/blob/v1.1.1/spec.md#unavailable-referrers-api). However, both approaches only support referrer discovery and do not provide generic predecessor finding.
331+
332+
Below is a mapping of major repository functions to their corresponding registry API endpoints:
333+
334+
#### Manifest Store Mappings
335+
336+
| Function Name | API endpoint |
337+
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
338+
| `Fetch` | GET `/v2/<name>/manifests/<reference>` |
339+
| `Exists` | HEAD `/v2/<name>/manifests/<reference>` |
340+
| `Push` | PUT `/v2/<name>/manifests/<reference>` |
341+
| `Resolve` | HEAD `/v2/<name>/manifests/<reference>` |
342+
| `Tag` | PUT `/v2/<name>/manifests/<reference>` |
343+
| `Predecessors` | GET `/v2/<name>/referrers/<digest>?artifactType=<artifactType>`<br>Fallback to `Referrers Tag Schema` if unavailable |
344+
345+
#### Blob Store Mappings
346+
347+
| Function Name | API endpoint |
348+
| ------------- | ---------------------------------------------------------------------------------------------- |
349+
| `Fetch` | GET `/v2/<name>/blobs/<reference>` |
350+
| `Exists` | HEAD `/v2/<name>/blobs/<reference>` |
351+
| `Push` | POST `/v2/<name>/blobs/uploads/`<br>PUT `/v2/<name>/blobs/uploads/<reference>?digest=<digest>` |
352+
| `Resolve` | HEAD `/v2/<name>/blobs/<reference>` |
353+
354+
### Summary of Content Stores
355+
356+
| Name | Description | Persistent Storage | Predecessors Support | Scenarios |
357+
| ---------------- | ---------------------------------------------------------------------------------- | ------------------------------ | --------------------------- | ------------------------------------------ |
358+
| Memory Store | Stores everything in memory | No | Yes | Memory caching, testing |
359+
| OCI Store | Stores content in OCI-Image layout on the file system | Yes | Yes | Local cache or copy of remote repositories |
360+
| File Store | Stores location-addressable content on file system | Partial (For named blobs only) | Yes | Packaging arbitrary files |
361+
| Repository Store | Communicates with remote artifact repositories (e.g. `ghcr.io`, `docker.io`, etc.) | Yes | Partial (via Referrers API) | Accessing remote repositories |
362+
363+
### How to Choose the Appropriate Content Store
364+
365+
```mermaid
366+
flowchart TD;
367+
368+
Q1{"Access remote repository?"}
369+
Q1--Y-->Repository["Repository Store"]
370+
Q1--N-->Q2{"Reading/writing arbitrary files?"}
371+
Q2--Y-->File["File Store"]
372+
Q2--N-->Q3{"Need persistent storage?"}
373+
Q3--Y-->OCI["OCI Store"]
374+
Q3--N-->Memory["Memory Store"]
375+
```

0 commit comments

Comments
 (0)