diff options
20 files changed, 307 insertions, 24 deletions
diff --git a/CHANGELOG-EE.md b/CHANGELOG-EE.md index fe44d526d51..e461983526c 100644 --- a/CHANGELOG-EE.md +++ b/CHANGELOG-EE.md @@ -1,5 +1,136 @@ Please view this file on the master branch, on stable branches it's out of date. +## 12.9.0 (2020-03-22) + +### Removed (1 change) + +- Remove Security Dashboard button from project page. !24781 + +### Fixed (20 changes) + +- Ensure User IDs are Removed When Moving to All Users Strategy. !25190 +- Fix dismissed vulnerability comment updates. !25328 +- Generate illustrations URLs in the backend. !25375 +- Check user role for each of namespaces while deciding if it is using a paid seat in it. !25437 +- Change MR approval popover to use hover trigger. !25494 +- Change copy of Welcome screen. !25526 +- Fix weight not showing in linked issues. !25718 +- Scope name uniqueness of approval rules to type. !25875 +- Pass correct Time object to Value Stream Management DataCollector. !25885 +- API for Namespace members has correct boolean if user using paid seat. !25988 +- Set the default SSL cert path environment variables when invoking indexer. !26036 +- Add pagination to Instance Level Security Dashboard project selector. !26138 +- Exclude creation event from audit changes. !26140 +- Resolve Usage Ping timeouts on security_products_usage for large hosts. !26296 +- Fix SCIM API not creating new users. !26514 +- BulkIndexer change _routing -> routing to support ES 7. !26639 +- Ensure design events are correctly visible. !26708 +- Add support for async search within Epics dropdown. !26980 +- Fix configuration issue for License Compliance. !27016 +- Geo - Fix ambiguous reference column while loading migrated local files to clean up. !27252 + +### Changed (22 changes, 1 of them is from the community) + +- Add auto_renew attribute to gitlab_subscriptions and expose via api. !24896 (jejacks0n) +- Remove admin user count from Users statistics. !25337 +- Improve error messages in the Dependency List. !25369 +- Do not automatically add approvers as participants of each merge request. !25546 +- Improve loading UX in the License Management list. !25620 +- Improve audit log & events header layout. !25821 +- Adjust the MR approval rule to use a smaller modal. !25955 +- Resolve Code review start time doesnt account for approvals. !25997 +- Prevent 'Invite group' for groups outside a group-managed account group. !26081 +- Improve language when MR Security Report is outdated. !26150 +- Prevent projects from being shared outside a group with managed accounts. !26163 +- Prevent projects from being shared outside a group with managed accounts for forked projects. !26186 +- Extract time range information to be reused. !26431 +- Restrict inviting outside group to a project forked from a group with enforced SSO. !26456 +- Add severity badge to security reports. !26715 +- Consider MR reassign as code review start. !26891 +- Add application limits for Ci::PipelineSchedule. !27004 +- Update license compliance policy classifications text in the UI. !27058 +- Remove approval metric recalculation when approval was removed. !27071 +- Move Value Stream Analytics to the group level. !27304 +- Remove release tag filter from Issues Analytics. !208903 +- Change dependency & license scan pipeline text message. + +### Performance (4 changes) + +- Optimize service desk issue counters query performance in usage data. !26445 +- Preload database records in bulk indexing. !26754 +- Geo: Make the LFS registry the SSOT to optimize query performance. !27154 +- Remove feature flag elastic_bulk_incremental_updates. !27293 + +### Added (58 changes, 9 of them are from the community) + +- Audit number of required approvals changes. !7531 +- Add weight and progress information in Roadmap Epic bars. !18957 +- Adds the ability to dismiss multiple vulnerabilities. !21480 +- Add code quality tab to pipeline view. !21527 +- Add milestones to roadmap. !22748 +- Add Threat Monitoring page. !22911 +- Add epic in filtered search. !22958 +- Add deep links for cycle analytics. !23493 +- Allow existing users to transfer their account to SAML Group managed accounts. !24329 +- Use snowplow for user error reporting in trial. !24430 +- Time period for usage activity by stage in usage data. !24431 +- Adds sorting to group packages api. !24432 +- Export Board Milestones in Group Export. !24606 +- Add possibility to search for epics by partial iids using GraphQL. !24673 +- Add restrict outer forks functionality for group SAML. !24698 +- Release Jira connect feature. !24818 +- Adds project_path sorting to group package API. !24833 +- Add webhook limits for groups. !25129 +- Default design images to 100% of viewport width. !25327 +- Add npm_package_requests_forwarding to application settings. !25346 +- Add bot count to Users statistics in Admin Area. !25355 +- Adding CODEOWNER approval metrics for merge requests. !25386 +- Separate 4 ee entities into own class file. !25400 (Rajendra Kadam) +- Improve zoom reset after resizing window in Design Management. !25406 +- Introduce Project level issues analytics. !25417 +- Add tooltip to guests count in Users Statistics. !25433 +- Separate epic, epic_issue, linked epic ee entities into own class file. !25592 (Rajendra Kadam) +- Add API endpoints for Group Activity Analytics. !25606 +- Add GraphQL mutation to set the epic of an issue. !25628 +- Add merge request metrics first approved at timestamp. !25681 +- Separate ee entities into own class file. !25709 (Rajendra Kadam) +- Embed cluster health metrics in GitLab-flavored Markdown. !25739 +- Always show the Design Tab on Issues but indicate user if feature is not available. !25796 +- Status page settings. !25820 +- Audit user blocked via API. !25872 +- Create a vulnerability-list component. !25927 +- Audit project MR approval permission changes. !25959 +- Add health status data to the epic GraphQL endpoint. !25989 +- Display correct approval rules based on target branch in Edit MR form. !26053 +- Activate triggering webhooks from a subgroup feature. !26070 +- Separate vulnerabilities, feature flags, dependency entities into own class files. !26088 (Rajendra Kadam) +- Support drag-and-drop design uploads in Design Management. !26139 +- Add distinct user count of secure stage items to usage ping. !26194 +- Support drag-and-drop on existing designs in Design Management. !26446 +- Add worker to sync paid seats info daily in the background. !26467 +- Separate project alias, build info, package entities into own class files. !26493 (Rajendra Kadam) +- Support async loading & search of projects within Epics Tree. !26661 +- Add code review reassign metric structure. !26664 +- Separate npm, subs and geo entities into own class file. !26674 (Rajendra Kadam) +- Separate geo_node and license entities into own class file. !26685 (Rajendra Kadam) +- Separate merge request approval entities into own class files. !26860 (Rajendra Kadam) +- Include snippet description as part of snippet title search (advanced search, Elasticsearch reindexing required). !26870 +- Add Roadmap GraphQL and unfiltered epic aggregates. !26919 +- Add NPM package requests forwarding to npmjs.org. !26932 +- Separate approval entities into own class files. !27336 (Rajendra Kadam) +- View description changes history of issues, epics and merge requests. !27445 +- Enable license compliance policies tab for on prem installs. +- Add a bulk processor for ES incremental updates. + +### Other (5 changes, 1 of them is from the community) + +- Refactor WeightSelect to use gitlab-ui dropdown. !20770 +- Update to new severity badge design. !25489 +- Rename close_{at,by} to dismissed_{at,by} in Vulnerabilities. !25609 (Michał Zając) +- Style MR approval popover link to be consistent text size. !25657 +- Allow users to be marked as service users. !202680 + + ## 12.8.7 (2020-03-16) ### Fixed (1 change) diff --git a/app/assets/javascripts/repository/components/preview/index.vue b/app/assets/javascripts/repository/components/preview/index.vue index 99f100eaa60..f96523bb497 100644 --- a/app/assets/javascripts/repository/components/preview/index.vue +++ b/app/assets/javascripts/repository/components/preview/index.vue @@ -56,7 +56,7 @@ export default { </gl-link> </div> </div> - <div class="blob-viewer"> + <div class="blob-viewer" data-qa-selector="blob_viewer_content"> <gl-loading-icon v-if="loading > 0" size="md" color="dark" class="my-4 mx-auto" /> <div v-else-if="readme" ref="readme" v-html="readme.html"></div> </div> diff --git a/app/models/merge_request_diff.rb b/app/models/merge_request_diff.rb index fe769573e29..9136c6cc5d4 100644 --- a/app/models/merge_request_diff.rb +++ b/app/models/merge_request_diff.rb @@ -434,8 +434,50 @@ class MergeRequestDiff < ApplicationRecord merge_request_diff_files.reset end + # Transactionally migrate the current merge_request_diff_files entries from + # external storage, back to the database. This is the rollback operation for + # +migrate_files_to_external_storage!+ + # + # If this diff isn't in external storage, the method is a no-op. + def migrate_files_to_database! + return unless stored_externally? + return if merge_request_diff_files.count == 0 + + rows = convert_external_diffs_to_database + + transaction do + MergeRequestDiffFile.where(merge_request_diff_id: id).delete_all + Gitlab::Database.bulk_insert('merge_request_diff_files', rows) + update!(stored_externally: false) + end + + # Only delete the external diff file after the contents have been saved to + # the database + remove_external_diff! + merge_request_diff_files.reset + end + private + def convert_external_diffs_to_database + opening_external_diff do |external_file| + merge_request_diff_files.map do |diff_file| + row = diff_file.attributes.except('diff') + + raise "Diff file lacks external diff offset or size: #{row.inspect}" unless + row['external_diff_offset'] && row['external_diff_size'] + + # The diff in the external file is already base64-encoded if necessary, + # matching the 'binary' attribute of the row. Reading it directly allows + # a cycle of decode-encode to be skipped + external_file.seek(row.delete('external_diff_offset')) + row['diff'] = external_file.read(row.delete('external_diff_size')) + + row + end + end + end + def diffs_in_batch_collection(batch_page, batch_size, diff_options:) Gitlab::Diff::FileCollection::MergeRequestDiffBatch.new(self, batch_page, diff --git a/app/views/layouts/nav/_breadcrumbs.html.haml b/app/views/layouts/nav/_breadcrumbs.html.haml index 1b799477093..c344d3d484f 100644 --- a/app/views/layouts/nav/_breadcrumbs.html.haml +++ b/app/views/layouts/nav/_breadcrumbs.html.haml @@ -7,7 +7,7 @@ = button_tag class: 'toggle-mobile-nav', type: 'button' do %span.sr-only= _("Open sidebar") = icon ('bars') - .breadcrumbs-links.js-title-container + .breadcrumbs-links.js-title-container{ data: { qa_selector: 'breadcrumb_links_content' } } %ul.list-unstyled.breadcrumbs-list.js-breadcrumbs-list - unless hide_top_links = header_title diff --git a/changelogs/unreleased/37995-allow-external-diffs-to-be-removed.yml b/changelogs/unreleased/37995-allow-external-diffs-to-be-removed.yml new file mode 100644 index 00000000000..ae0431caa1c --- /dev/null +++ b/changelogs/unreleased/37995-allow-external-diffs-to-be-removed.yml @@ -0,0 +1,5 @@ +--- +title: Allow external diff files to be removed +merge_request: 27602 +author: +type: changed diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 001e2883de0..3597fe6df1e 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -67,6 +67,8 @@ in your SAML IdP: auto_link_saml_user: true ``` +1. Ensure that the SAML [`NameID`](../user/group/saml_sso/index.md#nameid) and email address are fixed for each user, as described in the section on [Security](#security). Otherwise, your users will be able to sign in as other authorized users. + 1. Add the provider configuration: For Omnibus package: @@ -430,6 +432,8 @@ args: { } ``` +Make sure you read the [Security](#security) section before changing this value. + ## Response signature validation (required) We require Identity Providers to sign SAML responses to ensure that the assertions are @@ -544,6 +548,17 @@ args: { GitLab will sign the request with the provided private key. GitLab will include the configured public x500 certificate in the metadata for your Identity Provider to validate the signature of the received request with. For more information on this option, see the [Ruby SAML gem documentation](https://github.com/onelogin/ruby-saml/tree/v1.7.0). The Ruby SAML gem is used by the [OmniAuth SAML gem](https://github.com/omniauth/omniauth-saml) to implement the client side of the SAML authentication. +## Security + +Avoid user control of the following attributes: + +- [`*NameID*`](../user/group/saml_sso/index.md#nameid) +- *Email* when used with `omniauth_auto_link_saml_user` + +These attributes define the SAML user. If users can change these attributes, they can impersonate others. + +Refer to the documentation for your [SAML Identity Provider](../user/group/saml_sso/index.md#providers) for information on how to fix these attributes. + ## Troubleshooting ### GitLab+SAML Testing Environments diff --git a/lib/gitlab/git_access_snippet.rb b/lib/gitlab/git_access_snippet.rb index e11c1ea527c..3ba2fcecd6f 100644 --- a/lib/gitlab/git_access_snippet.rb +++ b/lib/gitlab/git_access_snippet.rb @@ -98,7 +98,10 @@ module Gitlab def check_single_change_access(change) Checks::SnippetCheck.new(change, logger: logger).validate! - Checks::PushFileCountCheck.new(change, repository: repository, limit: Snippet::MAX_FILE_COUNT, logger: logger).validate! + + if Feature.enabled?(:snippet_count_check) + Checks::PushFileCountCheck.new(change, repository: repository, limit: Snippet::MAX_FILE_COUNT, logger: logger).validate! + end rescue Checks::TimedLogger::TimeoutError raise TimeoutError, logger.full_message end diff --git a/lib/gitlab/sidekiq_cluster.rb b/lib/gitlab/sidekiq_cluster.rb index c19bef1389a..70df40fc35d 100644 --- a/lib/gitlab/sidekiq_cluster.rb +++ b/lib/gitlab/sidekiq_cluster.rb @@ -1,5 +1,7 @@ # frozen_string_literal: true +require 'shellwords' + module Gitlab module SidekiqCluster # The signals that should terminate both the master and workers. @@ -73,9 +75,9 @@ module Gitlab counts = count_by_queue(queues) cmd = %w[bundle exec sidekiq] - cmd << "-c #{self.concurrency(queues, min_concurrency, max_concurrency)}" + cmd << "-c#{self.concurrency(queues, min_concurrency, max_concurrency)}" cmd << "-e#{env}" - cmd << "-gqueues: #{proc_details(counts)}" + cmd << "-gqueues:#{proc_details(counts)}" cmd << "-r#{directory}" counts.each do |queue, count| @@ -83,7 +85,7 @@ module Gitlab end if dryrun - puts "Sidekiq command: #{cmd}" # rubocop:disable Rails/Output + puts Shellwords.join(cmd) # rubocop:disable Rails/Output return end @@ -112,7 +114,7 @@ module Gitlab else "#{queue} (#{count})" end - end.join(', ') + end.join(',') end def self.concurrency(queues, min_concurrency, max_concurrency) diff --git a/lib/gitlab/sidekiq_cluster/cli.rb b/lib/gitlab/sidekiq_cluster/cli.rb index 0a9624950c2..245d918e382 100644 --- a/lib/gitlab/sidekiq_cluster/cli.rb +++ b/lib/gitlab/sidekiq_cluster/cli.rb @@ -64,7 +64,9 @@ module Gitlab 'No queues found, you must select at least one queue' end - @logger.info("Starting cluster with #{queue_groups.length} processes") + unless @dryrun + @logger.info("Starting cluster with #{queue_groups.length} processes") + end @processes = SidekiqCluster.start( queue_groups, @@ -385,6 +385,7 @@ module QA # Classes describing components that are used by several pages. # module Component + autoload :Breadcrumbs, 'qa/page/component/breadcrumbs' autoload :CiBadgeLink, 'qa/page/component/ci_badge_link' autoload :ClonePanel, 'qa/page/component/clone_panel' autoload :LazyLoader, 'qa/page/component/lazy_loader' diff --git a/qa/qa/page/component/breadcrumbs.rb b/qa/qa/page/component/breadcrumbs.rb new file mode 100644 index 00000000000..656aa380bbd --- /dev/null +++ b/qa/qa/page/component/breadcrumbs.rb @@ -0,0 +1,19 @@ +# frozen_string_literal: true + +module QA + module Page + module Component + module Breadcrumbs + def self.included(base) + base.view 'app/views/layouts/nav/_breadcrumbs.html.haml' do + element :breadcrumb_links_content + end + end + + def has_breadcrumb?(text) + has_element?(:breadcrumb_links_content, text: text) + end + end + end + end +end diff --git a/qa/qa/page/project/show.rb b/qa/qa/page/project/show.rb index 61047c42fcc..00298ff9fb5 100644 --- a/qa/qa/page/project/show.rb +++ b/qa/qa/page/project/show.rb @@ -5,8 +5,13 @@ module QA module Project class Show < Page::Base include Page::Component::ClonePanel + include Page::Component::Breadcrumbs include Page::Project::SubMenus::Settings + view 'app/assets/javascripts/repository/components/preview/index.vue' do + element :blob_viewer_content + end + view 'app/assets/javascripts/repository/components/table/row.vue' do element :file_name_link end @@ -90,7 +95,7 @@ module QA def click_file(filename) within_element(:file_tree_table) do - click_on filename + click_element(:file_name_link, text: filename) end end @@ -115,6 +120,10 @@ module QA has_element?(:project_name_content, text: name) end + def has_readme_content?(text) + has_element?(:blob_viewer_content, text: text) + end + def last_commit_content find_element(:commit_content).text end diff --git a/qa/qa/specs/features/browser_ui/1_manage/group/transfer_project_spec.rb b/qa/qa/specs/features/browser_ui/1_manage/group/transfer_project_spec.rb index 6f75940e1f0..9eab03323a8 100644 --- a/qa/qa/specs/features/browser_ui/1_manage/group/transfer_project_spec.rb +++ b/qa/qa/specs/features/browser_ui/1_manage/group/transfer_project_spec.rb @@ -47,8 +47,8 @@ module QA Page::Project::Settings::Main.perform(&:click_project) Page::Project::Show.perform do |project| - expect(project).to have_text(target_group.path) - expect(project).to have_text(edited_readme_content) + expect(project).to have_breadcrumb(target_group.path) + expect(project).to have_readme_content(edited_readme_content) end end end diff --git a/qa/qa/specs/features/browser_ui/3_create/repository/protocol_v2_push_ssh_spec.rb b/qa/qa/specs/features/browser_ui/3_create/repository/protocol_v2_push_ssh_spec.rb index e845c3ca8ea..ec00091b3ce 100644 --- a/qa/qa/specs/features/browser_ui/3_create/repository/protocol_v2_push_ssh_spec.rb +++ b/qa/qa/specs/features/browser_ui/3_create/repository/protocol_v2_push_ssh_spec.rb @@ -60,12 +60,15 @@ module QA end end - project.visit! project.wait_for_push_new_branch + project.visit! - expect(page).to have_content(file_name) - expect(page).to have_content(file_content) expect(git_protocol_reported).to eq(git_protocol) + + Page::Project::Show.perform do |show| + expect(show).to have_file(file_name) + expect(show).to have_readme_content(file_content) + end end end end diff --git a/qa/qa/specs/features/browser_ui/3_create/repository/push_http_private_token_spec.rb b/qa/qa/specs/features/browser_ui/3_create/repository/push_http_private_token_spec.rb index c713f11af7d..d8aaffc3713 100644 --- a/qa/qa/specs/features/browser_ui/3_create/repository/push_http_private_token_spec.rb +++ b/qa/qa/specs/features/browser_ui/3_create/repository/push_http_private_token_spec.rb @@ -24,8 +24,10 @@ module QA Page::Project::Show.perform(&:wait_for_viewers_to_load) - expect(page).to have_content('README.md') - expect(page).to have_content('This is a test project') + Page::Project::Show.perform do |project| + expect(project).to have_file('README.md') + expect(project).to have_readme_content('This is a test project') + end end end end diff --git a/qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_spec.rb b/qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_spec.rb index 9db5fe5292f..c2da5104930 100644 --- a/qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_spec.rb +++ b/qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_spec.rb @@ -12,8 +12,10 @@ module QA push.commit_message = 'Add README.md' end.project.visit! - expect(page).to have_content('README.md') - expect(page).to have_content('This is a test project') + Page::Project::Show.perform do |project| + expect(project).to have_file('README.md') + expect(project).to have_readme_content('This is a test project') + end end end end diff --git a/spec/bin/sidekiq_cluster_spec.rb b/spec/bin/sidekiq_cluster_spec.rb index 67de55ad9f5..c0240214a6b 100644 --- a/spec/bin/sidekiq_cluster_spec.rb +++ b/spec/bin/sidekiq_cluster_spec.rb @@ -1,6 +1,7 @@ # frozen_string_literal: true require 'spec_helper' +require 'shellwords' describe 'bin/sidekiq-cluster' do using RSpec::Parameterized::TableSyntax @@ -18,9 +19,9 @@ describe 'bin/sidekiq-cluster' do output, status = Gitlab::Popen.popen(cmd, Rails.root.to_s) expect(status).to be(0) - expect(output).to include('"bundle", "exec", "sidekiq"') - expect(output).to include(included) - expect(output).not_to include(excluded) + expect(output).to include('bundle exec sidekiq') + expect(Shellwords.split(output)).to include(included) + expect(Shellwords.split(output)).not_to include(excluded) end end end @@ -36,9 +37,9 @@ describe 'bin/sidekiq-cluster' do output, status = Gitlab::Popen.popen(cmd, Rails.root.to_s) expect(status).to be(0) - expect(output).to include('"bundle", "exec", "sidekiq"') - expect(output).to include('-qdefault,1') - expect(output).to include('-qcronjob:ci_archive_traces_cron,1') + expect(output).to include('bundle exec sidekiq') + expect(Shellwords.split(output)).to include('-qdefault,1') + expect(Shellwords.split(output)).to include('-qcronjob:ci_archive_traces_cron,1') end end end diff --git a/spec/frontend/repository/components/preview/__snapshots__/index_spec.js.snap b/spec/frontend/repository/components/preview/__snapshots__/index_spec.js.snap index 8eeae9b8455..69b7a3931f8 100644 --- a/spec/frontend/repository/components/preview/__snapshots__/index_spec.js.snap +++ b/spec/frontend/repository/components/preview/__snapshots__/index_spec.js.snap @@ -27,6 +27,7 @@ exports[`Repository file preview component renders file HTML 1`] = ` <div class="blob-viewer" + data-qa-selector="blob_viewer_content" > <div> <div diff --git a/spec/lib/gitlab/git_access_snippet_spec.rb b/spec/lib/gitlab/git_access_snippet_spec.rb index f52fe8ef612..877a760152d 100644 --- a/spec/lib/gitlab/git_access_snippet_spec.rb +++ b/spec/lib/gitlab/git_access_snippet_spec.rb @@ -209,6 +209,16 @@ describe Gitlab::GitAccessSnippet do expect { push_access_check }.to raise_forbidden('foo') end + + context 'when feature flag :snippet_count_check is disabled' do + it 'does not check push file count' do + stub_feature_flags(snippet_count_check: false) + + expect(Gitlab::Checks::PushFileCountCheck).not_to receive(:new) + + expect { push_access_check }.not_to raise_error + end + end end private diff --git a/spec/models/merge_request_diff_spec.rb b/spec/models/merge_request_diff_spec.rb index 8167241faa8..6d2ad3f0475 100644 --- a/spec/models/merge_request_diff_spec.rb +++ b/spec/models/merge_request_diff_spec.rb @@ -156,6 +156,41 @@ describe MergeRequestDiff do end end + describe '#migrate_files_to_database!' do + let(:diff) { create(:merge_request).merge_request_diff } + + it 'converts from external to in-database storage' do + stub_external_diffs_setting(enabled: true) + + expect(diff).to be_stored_externally + expect(diff).to receive(:update!).and_call_original + + file = diff.external_diff + file_data = file.read + diff.migrate_files_to_database! + + expect(diff).not_to be_stored_externally + expect(file).not_to exist + expect(diff.merge_request_diff_files.map(&:diff).join('')).to eq(file_data) + end + + it 'does nothing with an in-database diff' do + expect(diff).not_to be_stored_externally + expect(diff).not_to receive(:update!) + + diff.migrate_files_to_database! + end + + it 'does nothing with an empty diff' do + stub_external_diffs_setting(enabled: true) + MergeRequestDiffFile.where(merge_request_diff_id: diff.id).delete_all + + expect(diff).not_to receive(:update!) + + diff.migrate_files_to_database! + end + end + describe '#latest?' do let!(:mr) { create(:merge_request, :with_diffs) } let!(:first_diff) { mr.merge_request_diff } |