| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
This reverts merge request
https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/803
because it caused an incident when serving large compressed files
Changelog: fixed
|
|
Changelog: performance
|
|
|
|
|
|
|
|
|
|
In https://gitlab.com/gitlab-org/gitlab-pages/-/issues/702 we
discovered that by caching many `*zip.File` instances, we end up
storing many Go timezone objects on the heap: one per cached
`*zip.File`. This adds up to about 25% of the heap size.
In this commit we set the timestamp of `*zip.File` to UTC. Because Go
re-uses a single timezone object for UTC, this causes the original
timezone objects of the `*zip.File` instances to be garbage collected.
This reduces the heap size.
Here is an example program that demonstrates the effect:
```golang
package main
import (
"archive/zip"
"log"
"os"
"runtime/pprof"
)
func main() {
if err := load(os.Args[1]); err != nil {
log.Fatal(err)
}
log.Printf("load finished: %d archives", len(readers))
pprof.WriteHeapProfile(os.Stdout)
}
var readers []*zip.ReadCloser
func load(filename string) error {
for i := 0; i < 1000; i++ {
zr, err := zip.OpenReader(filename)
if err != nil {
return err
}
if os.Getenv("FORCE_UTC") == "1" {
for _, zf := range zr.File {
zf.Modified = zf.Modified.UTC()
}
}
readers = append(readers, zr)
}
return nil
}
```
|
|
|
|
|
|
|
|
Using `gitlab.com/gitlab-org/golang-archive-zip` instead of the `archive/zip`
package from the stdlib.
Changelog: changed
|
|
so it can be reused by different components like the
zip vfs and the rate limiter.
related to https://gitlab.com/gitlab-org/gitlab-pages/-/issues/626.
|
|
Changelog: fixed
|
|
|
|
Moves the http.Client initialization inside the `httprange` package to
the zip VFS. This makes the type `Resource` depend on an http.Client
that needs to be passed on initialization.
It also makes the zip VFS initialize the client. It's possible to
reconfigure it to register a file protocol by calling vfs.Reconfigure
explicitly.
|
|
|
|
If archive is broken (which should in fact never happen)
we fail the first request and mark the cache entry
as invalid. It will be refreshed on a next try.
|
|
Remove to-do and update test
|
|
|
|
|
|
In case of archives that do not store directories
we would fail to automatically serve `index.html`
for a `/` type of request.
This makes us create directories when traversing
the file list.
```
Archive: public-without-dirs.zip
Length Date Time Name
--------- ---------- ----- ----
40 2020-09-15 02:47 public/subdir/hello.html
14 2020-09-15 03:35 public/subdir/2bp3Qzs9CCW7cGnxhghdavZ2bJDTzvu2mrj6O8Yqjm3YMRozRZULxBBKzJXCK16GlsvO1GlbCyONf2LTCndJU9cIr5T3PLDN7XnfG00lEmf9DWHPXiAbbi0v8ioSjnoTqdyjELVKuhsGRGxeV9RptLMyGnbpJx1w2uECiUQSHrRVQNuq2xoHLlk30UAmis1EhGXP5kKprzHxuavsKMdT4XRP0d79tie4tjqtfRsP4y60hmNS1vSujrxzhDa
33 2020-09-15 02:47 public/subdir/linked.html
31 2020-09-15 02:47 public/404.html
33 2020-09-15 02:47 public/index.html
258 2020-10-13 12:40 public/bad_symlink.html
18 2020-10-13 12:40 public/symlink.html
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Renames all metrics to make them easier to understand.
Adds a missing metric for number of open requests made by httprange.
|
|
|
|
|
|
|
|
|
|
Adds a bunch of new metrics related to
https://gitlab.com/gitlab-org/gitlab-pages/-/issues/423.
It uses [httptrace.ClienTrace](https://golang.org/src/net/http/httptrace/trace.go)
to add a bunch of very granular metrics that happen when an http connection is established.
|
|
|
|
Assert that if a context is canceled the read request will fail.
Add missing ctx to zip
|
|
|
|
|
|
|
|
|
|
|
|
Adds a zip package capable of reading zip files from an
httprange.Resource. It reads archive file contents into memory per
archive that is requested from the vfs.Root.
WIP: add simple test for archive reader
WIP: fix build
WIP: update archive test
WIP: adding tests for archive
WIP: print more info
WIP: update zip file correct symlink
WIP: use correct file
Add bad symlink file to zip
Update handling long symlinks
update documentation and reorg code
fix up stuff
|
