1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
|
---
stage: none
group: unassigned
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Test Import Project
For testing, we can import our own [GitLab CE](https://gitlab.com/gitlab-org/gitlab-foss/) project (named `gitlabhq` in this case) under a group named `qa-perf-testing`. Project tarballs that can be used for testing can be found over on the [performance-data](https://gitlab.com/gitlab-org/quality/performance-data) project. A different project could be used if required.
There are several options for importing the project into your GitLab environment. They are detailed as follows with the assumption that the recommended group `qa-perf-testing` and project `gitlabhq` are being set up.
## Importing the project
There are several ways to import a project.
### Importing via UI
The first option is to simply [import the Project tarball file via the GitLab UI](../user/project/settings/import_export.md#import-a-project-and-its-data):
1. Create the group `qa-perf-testing`
1. Import the [GitLab FOSS project tarball](https://gitlab.com/gitlab-org/quality/performance-data/-/blob/master/projects_export/gitlabhq_export.tar.gz) into the Group.
It should take up to 15 minutes for the project to fully import. You can head to the project's main page for the current status.
This method ignores all the errors silently (including the ones related to `GITALY_DISABLE_REQUEST_LIMITS`) and is used by GitLab users. For development and testing, check the other methods below.
### Importing via the `import-project` script
A convenient script, [`bin/import-project`](https://gitlab.com/gitlab-org/quality/performance/blob/master/bin/import-project), is provided with [performance](https://gitlab.com/gitlab-org/quality/performance) project to import the Project tarball into a GitLab environment via API from the terminal.
Note that to use the script, it requires some preparation if you haven't done so already:
1. First, set up [`Ruby`](https://www.ruby-lang.org/en/documentation/installation/) and [`Ruby Bundler`](https://bundler.io) if they aren't already available on the machine.
1. Next, install the required Ruby Gems via Bundler with `bundle install`.
For details how to use `bin/import-project`, run:
```shell
bin/import-project --help
```
The process should take up to 15 minutes for the project to import fully. The script checks the status periodically and exits after the import has completed.
### Importing via GitHub
There is also an option to [import the project via GitHub](../user/project/import/github.md):
1. Create the group `qa-perf-testing`
1. Import the GitLab FOSS repository that's [mirrored on GitHub](https://github.com/gitlabhq/gitlabhq) into the group via the UI.
This method takes longer to import than the other methods and depends on several factors. It's recommended to use the other methods.
### Importing via a Rake task
> The [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/import_export/import.rake) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20724) in GitLab 12.6, replacing a GitLab.com Ruby script.
This script was introduced in GitLab 12.6 for importing large GitLab project exports.
As part of this script we also disable direct and background upload to avoid situations where a huge archive is being uploaded to GCS (while being inside a transaction, which can cause idle transaction timeouts).
We can simply run this script from the terminal:
Parameters:
| Attribute | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `username` | string | yes | User name |
| `namespace_path` | string | yes | Namespace path |
| `project_path` | string | yes | Project name |
| `archive_path` | string | yes | Path to the exported project tarball you want to import |
```shell
bundle exec rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]"
```
If you're running Omnibus, run the following Rake task:
```shell
gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]"
```
#### Troubleshooting
Check the common errors listed below, what they mean, and how to fix them.
##### `Exception: undefined method 'name' for nil:NilClass`
The `username` is not valid.
##### `Exception: undefined method 'full_path' for nil:NilClass`
The `namespace_path` does not exist.
For example, one of the groups or subgroups is mistyped or missing
or you've specified the project name in the path.
The task only creates the project.
If you want to import it to a new group or subgroup then create it first.
##### `Exception: No such file or directory @ rb_sysopen - (filename)`
The specified project export file in `archive_path` is missing.
##### `Exception: Permission denied @ rb_sysopen - (filename)`
The specified project export file can not be accessed by the `git` user.
Setting the file owner to `git:git`, changing the file permissions to `0400`, and moving it to a
public folder (for example `/tmp/`) fixes the issue.
##### `Name can contain only letters, digits, emojis ...`
```plaintext
Name can contain only letters, digits, emojis, '_', '.', '+', dashes, or spaces. It must start with a letter,
digit, emoji, or '_', and Path can contain only letters, digits, '_', '-', or '.'. It cannot start
with '-', end in '.git', or end in '.atom'.
```
The project name specified in `project_path` is not valid for one of the specified reasons.
Only put the project name in `project_path`. For example, if you provide a path of subgroups
it fails with this error as `/` is not a valid character in a project name.
##### `Name has already been taken and Path has already been taken`
A project with that name already exists.
##### `Exception: Error importing repository into (namespace) - No space left on device`
The disk has insufficient space to complete the import.
During import, the tarball is cached in your configured `shared_path` directory. Verify the
disk has enough free space to accommodate both the cached tarball and the unpacked
project files on disk.
### Importing via the Rails console
The last option is to import a project using a Rails console:
1. Start a Ruby on Rails console:
```shell
# Omnibus GitLab
gitlab-rails console
# For installations from source
sudo -u git -H bundle exec rails console -e production
```
1. Create a project and run `Project::TreeRestorer`:
```ruby
shared_class = Struct.new(:export_path) do
def error(message)
raise message
end
end
user = User.first
shared = shared_class.new(path)
project = Projects::CreateService.new(user, { name: name, namespace: user.namespace }).execute
begin
#Enable Request store
RequestStore.begin!
Gitlab::ImportExport::Project::TreeRestorer.new(user: user, shared: shared, project: project).restore
ensure
RequestStore.end!
RequestStore.clear!
end
```
1. In case you need the repository as well, you can restore it using:
```ruby
repo_path = File.join(shared.export_path, Gitlab::ImportExport.project_bundle_filename)
Gitlab::ImportExport::RepoRestorer.new(path_to_bundle: repo_path,
shared: shared,
importable: project).restore
```
We are storing all import failures in the `import_failures` data table.
To make sure that the project import finished without any issues, check:
```ruby
project.import_failures.all
```
## Performance testing
For Performance testing, we should:
- Import a quite large project, [`gitlabhq`](https://gitlab.com/gitlab-org/quality/performance-data#gitlab-performance-test-framework-data) should be a good example.
- Measure the execution time of `Project::TreeRestorer`.
- Count the number of executed SQL queries during the restore.
- Observe the number of GC cycles happening.
You can use this snippet: `https://gitlab.com/gitlab-org/gitlab/snippets/1924954` (must be logged in), which restores the project, and measures the execution time of `Project::TreeRestorer`, number of SQL queries and number of GC cycles happening.
You can execute the script from the `gdk/gitlab` directory like this:
```shell
bundle exec rails r /path_to_script/script.rb project_name /path_to_extracted_project request_store_enabled
```
## Troubleshooting
This section details known issues we've seen when trying to import a project and how to manage them.
### Gitaly calls error when importing
If you're attempting to import a large project into a development environment, you may see Gitaly throw an error about too many calls or invocations, for example:
```plaintext
Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1?
```
This is due to a [n+1 calls limit being set for development setups](gitaly.md#toomanyinvocationserror-errors). You can work around this by setting `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable, restarting your development environment and importing again.
## Access token setup
Many of the tests also require a GitLab Personal Access Token. This is due to numerous endpoints themselves requiring authentication.
[The official GitLab docs detail how to create this token](../user/profile/personal_access_tokens.md#create-a-personal-access-token). The tests require that the token is generated by an administrator and that it has the `API` and `read_repository` permissions.
Details on how to use the Access Token with each type of test are found in their respective documentation.
|
