1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
|
---
stage: Create
group: IDE
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments"
type: reference
---
# External storage for static objects **(FREE)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31025) in GitLab 12.3.
Configure GitLab to serve repository static objects (such as archives or raw blobs) from external
storage such as a content delivery network (CDN).
## Configure external storage
To configure external storage for static objects:
1. On the left sidebar, expand the top-most chevron (**{chevron-down}**).
1. Select **Admin Area**.
1. On the left sidebar, select **Settings > Repository**.
1. Expand the **External storage for repository static objects** section.
1. Enter the base URL and an arbitrary token. When you [set up external storage](#set-up-external-storage),
use a script that sets these values as `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`.
1. Select **Save changes**.
The token is required to distinguish requests coming from the external storage, so users don't
circumvent the external storage and access the application directly. GitLab expects
this token to be set in the `X-Gitlab-External-Storage-Token` header in requests
originating from the external storage.
## Serving private static objects
GitLab appends a user-specific token for static object URLs belonging to private projects so
external storage can be authenticated on the user's behalf.
When processing requests originating
from the external storage, GitLab checks the following to confirm the user may access the requested
object:
- The `token` query parameter.
- The `X-Gitlab-Static-Object-Token` header.
## Requests flow example
The following example shows a sequence of requests and responses between:
- The user.
- GitLab.
- The content delivery network.
```mermaid
sequenceDiagram
User->>GitLab: GET /project/-/archive/master.zip
GitLab->>User: 302 Found
Note over User,GitLab: Location: https://cdn.com/project/-/archive/master.zip?token=secure-user-token
User->>CDN: GET /project/-/archive/master.zip?token=secure-user-token
alt object not in cache
CDN->>GitLab: GET /project/-/archive/master.zip
Note over CDN,GitLab: X-Gitlab-External-Storage-Token: secure-cdn-token<br/>X-Gitlab-Static-Object-Token: secure-user-token
GitLab->>CDN: 200 OK
CDN->>User: master.zip
else object in cache
CDN->>GitLab: GET /project/-/archive/master.zip
Note over CDN,GitLab: X-Gitlab-External-Storage-Token: secure-cdn-token<br/>X-Gitlab-Static-Object-Token: secure-user-token<br/>If-None-Match: etag-value
GitLab->>CDN: 304 Not Modified
CDN->>User: master.zip
end
```
## Set up external storage
While this procedure uses [Cloudflare Workers](https://workers.cloudflare.com) for external storage,
other CDNs or Function as a Service (FaaS) systems should work using the same principles.
1. Choose a Cloudflare Worker domain if you haven't done so already.
1. In the following script, set the following values for the first two constants:
- `ORIGIN_HOSTNAME`: the hostname of your GitLab installation.
- `STORAGE_TOKEN`: any arbitrary secure token. You can get a token by running
`pwgen -cn1 64` on a UNIX machine. Save this token for the Admin Area, as
described in the [configuring](#configure-external-storage) section.
```javascript
const ORIGIN_HOSTNAME = 'gitlab.installation.com' // FIXME: SET CORRECT VALUE
const STORAGE_TOKEN = 'very-secure-token' // FIXME: SET CORRECT VALUE
const CACHE_PRIVATE_OBJECTS = false
const CORS_HEADERS = {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'GET, HEAD, OPTIONS',
'Access-Control-Allow-Headers': 'X-Csrf-Token, X-Requested-With',
}
self.addEventListener('fetch', event => event.respondWith(handle(event)))
async function handle(event) {
try {
let response = await verifyAndHandle(event);
// responses returned from cache are immutable, so we recreate them
// to set CORS headers
response = new Response(response.body, response)
response.headers.set('Access-Control-Allow-Origin', '*')
return response
} catch (e) {
return new Response('An error occurred!', {status: e.statusCode || 500})
}
}
async function verifyAndHandle(event) {
if (!validRequest(event.request)) {
return new Response(null, {status: 400})
}
if (event.request.method === 'OPTIONS') {
return handleOptions(event.request)
}
return handleRequest(event)
}
function handleOptions(request) {
// Make sure the necessary headers are present
// for this to be a valid pre-flight request
if (
request.headers.get('Origin') !== null &&
request.headers.get('Access-Control-Request-Method') !== null &&
request.headers.get('Access-Control-Request-Headers') !== null
) {
// Handle CORS pre-flight request
return new Response(null, {
headers: CORS_HEADERS,
})
} else {
// Handle standard OPTIONS request
return new Response(null, {
headers: {
Allow: 'GET, HEAD, OPTIONS',
},
})
}
}
async function handleRequest(event) {
let cache = caches.default
let url = new URL(event.request.url)
let static_object_token = url.searchParams.get('token')
let headers = new Headers(event.request.headers)
url.host = ORIGIN_HOSTNAME
url = normalizeQuery(url)
headers.set('X-Gitlab-External-Storage-Token', STORAGE_TOKEN)
if (static_object_token !== null) {
headers.set('X-Gitlab-Static-Object-Token', static_object_token)
}
let request = new Request(url, { headers: headers })
let cached_response = await cache.match(request)
let is_conditional_header_set = headers.has('If-None-Match')
if (cached_response) {
return cached_response
}
// We don't want to override If-None-Match that is set on the original request
if (cached_response && !is_conditional_header_set) {
headers.set('If-None-Match', cached_response.headers.get('ETag'))
}
let response = await fetch(request, {
headers: headers,
redirect: 'manual'
})
if (response.status == 304) {
if (is_conditional_header_set) {
return response
} else {
return cached_response
}
} else if (response.ok) {
response = new Response(response.body, response)
// cache.put will never cache any response with a Set-Cookie header
response.headers.delete('Set-Cookie')
if (CACHE_PRIVATE_OBJECTS) {
response.headers.delete('Cache-Control')
}
event.waitUntil(cache.put(request, response.clone()))
}
return response
}
function normalizeQuery(url) {
let searchParams = url.searchParams
url = new URL(url.toString().split('?')[0])
if (url.pathname.includes('/raw/')) {
let inline = searchParams.get('inline')
if (inline == 'false' || inline == 'true') {
url.searchParams.set('inline', inline)
}
} else if (url.pathname.includes('/-/archive/')) {
let append_sha = searchParams.get('append_sha')
let path = searchParams.get('path')
if (append_sha == 'false' || append_sha == 'true') {
url.searchParams.set('append_sha', append_sha)
}
if (path) {
url.searchParams.set('path', path)
}
}
return url
}
function validRequest(request) {
let url = new URL(request.url)
let path = url.pathname
if (/^(.+)(\/raw\/|\/-\/archive\/)/.test(path)) {
return true
}
return false
}
```
1. Create a new worker with this script.
1. Copy your values for `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`.
Use those values [to configure external storage for static objects](#configure-external-storage).
|