Skip to content

Commit 8780575

Browse files
TemikusTemikus
authored
Merge pull request #397 from Temikus/parallelize_more
More parallelization
2 parents d1d8058 + 306f2af commit 8780575

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+258
-75
lines changed

‎.rubocop.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ Style/RegexpLiteral:
1616
Style/FrozenStringLiteralComment:
1717
Enabled: false
1818

19+
Style/MutableConstant:
20+
Enabled: false
21+
1922
Style/HashSyntax:
2023
EnforcedStyle: no_mixed_keys
2124

@@ -30,7 +33,7 @@ AllCops:
3033
- "vendor/**/*"
3134
- "db/schema.rb"
3235
UseCache: false
33-
TargetRubyVersion: 2.1
36+
TargetRubyVersion: 2.3
3437
Style/CollectionMethods:
3538
Description: Preferred collection methods.
3639
StyleGuide: https://github.com/bbatsov/ruby-style-guide#map-find-select-reduce-size

‎README.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,13 @@
44

55
The main maintainers for the Google sections are @icco, @Temikus and @plribeiro3000. Please send pull requests to them.
66

7-
**As of v0.1.1, Google no longer supports Ruby versions less than 2.0.0.**
7+
## Important notices
88

9-
**As of v1.0.0, fog-google includes google-api-client as a dependency**
9+
- As of **v0.1.1**, Google no longer supports Ruby versions less than 2.0.0.
1010

11-
**See [MIGRATING.md](MIGRATING.md) for migration between major versions**
11+
- As of **v1.0.0**, fog-google includes google-api-client as a dependency, there is no need to include it separately anymore.
12+
13+
See **[MIGRATING.md](MIGRATING.md)** for migration between major versions.
1214

1315
## Storage
1416

‎ci/docker-image/Dockerfile‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,6 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get -y -qq update && apt-get -y -qq insta
1313
libxslt-dev
1414

1515
# Ubuntu 16.04 will fetch us 2.3.x without any issues.
16-
RUN DEBIAN_FRONTEND=noninteractive apt-get -y -qq install ruby ruby-dev
17-
18-
RUN apt-get clean
16+
RUN DEBIAN_FRONTEND=noninteractive apt-get -y -qq install ruby ruby-dev && apt-get clean
1917

2018
RUN gem install bundler

‎ci/pipeline.yml‎

Lines changed: 112 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
jobs:
2-
- name: integration-tests-compute
2+
- name: integration-compute-core_compute
33
serial: true
44
plan:
55
- get: fog-google-src
@@ -9,10 +9,10 @@ jobs:
99
- put: pull-request
1010
params: {path: fog-google-src, status: pending}
1111

12-
- task: full-integration-tests
12+
- task: run-integration-tests
1313
file: fog-google-src/ci/tasks/run-int.yml
1414
params:
15-
rake_task: test:compute
15+
rake_task: test:compute-core_compute
1616
codecov_token: {{codecov_token}}
1717
google_project: {{google_project}}
1818
google_json_key_data: {{google_json_key_data}}
@@ -22,7 +22,79 @@ jobs:
2222
params:
2323
path: fog-google-src
2424
status: failure
25-
25+
26+
- name: integration-compute-core_networking
27+
serial: true
28+
plan:
29+
- get: fog-google-src
30+
resource: pull-request
31+
version: every
32+
trigger: true
33+
- put: pull-request
34+
params: {path: fog-google-src, status: pending}
35+
36+
- task: run-integration-tests
37+
file: fog-google-src/ci/tasks/run-int.yml
38+
params:
39+
rake_task: test:compute-core_networking
40+
codecov_token: {{codecov_token}}
41+
google_project: {{google_project}}
42+
google_json_key_data: {{google_json_key_data}}
43+
google_client_email: {{google_client_email}}
44+
on_failure:
45+
put: pull-request
46+
params:
47+
path: fog-google-src
48+
status: failure
49+
50+
- name: integration-compute-instance_groups
51+
serial: true
52+
plan:
53+
- get: fog-google-src
54+
resource: pull-request
55+
version: every
56+
trigger: true
57+
- put: pull-request
58+
params: {path: fog-google-src, status: pending}
59+
60+
- task: run-integration-tests
61+
file: fog-google-src/ci/tasks/run-int.yml
62+
params:
63+
rake_task: test:compute-instance_groups
64+
codecov_token: {{codecov_token}}
65+
google_project: {{google_project}}
66+
google_json_key_data: {{google_json_key_data}}
67+
google_client_email: {{google_client_email}}
68+
on_failure:
69+
put: pull-request
70+
params:
71+
path: fog-google-src
72+
status: failure
73+
74+
- name: integration-compute-loadbalancing
75+
serial: true
76+
plan:
77+
- get: fog-google-src
78+
resource: pull-request
79+
version: every
80+
trigger: true
81+
- put: pull-request
82+
params: {path: fog-google-src, status: pending}
83+
84+
- task: run-integration-tests
85+
file: fog-google-src/ci/tasks/run-int.yml
86+
params:
87+
rake_task: test:compute-loadbalancing
88+
codecov_token: {{codecov_token}}
89+
google_project: {{google_project}}
90+
google_json_key_data: {{google_json_key_data}}
91+
google_client_email: {{google_client_email}}
92+
on_failure:
93+
put: pull-request
94+
params:
95+
path: fog-google-src
96+
status: failure
97+
2698
- name: integration-tests-monitoring
2799
serial: true
28100
plan:
@@ -33,7 +105,7 @@ jobs:
33105
- put: pull-request
34106
params: {path: fog-google-src, status: pending}
35107

36-
- task: full-integration-tests
108+
- task: run-integration-tests
37109
file: fog-google-src/ci/tasks/run-int.yml
38110
params:
39111
rake_task: test:monitoring
@@ -57,7 +129,7 @@ jobs:
57129
- put: pull-request
58130
params: {path: fog-google-src, status: pending}
59131

60-
- task: full-integration-tests
132+
- task: run-integration-tests
61133
file: fog-google-src/ci/tasks/run-int.yml
62134
params:
63135
rake_task: test:pubsub
@@ -71,7 +143,31 @@ jobs:
71143
path: fog-google-src
72144
status: failure
73145

74-
- name: integration-tests-sql
146+
- name: integration-sqlv1
147+
serial: true
148+
plan:
149+
- get: fog-google-src
150+
resource: pull-request
151+
version: every
152+
trigger: true
153+
- put: pull-request
154+
params: {path: fog-google-src, status: pending}
155+
156+
- task: run-integration-tests
157+
file: fog-google-src/ci/tasks/run-int.yml
158+
params:
159+
rake_task: test:sql-sqlv1
160+
codecov_token: {{codecov_token}}
161+
google_project: {{google_project}}
162+
google_json_key_data: {{google_json_key_data}}
163+
google_client_email: {{google_client_email}}
164+
on_failure:
165+
put: pull-request
166+
params:
167+
path: fog-google-src
168+
status: failure
169+
170+
- name: integration-sqlv2
75171
serial: true
76172
plan:
77173
- get: fog-google-src
@@ -81,10 +177,10 @@ jobs:
81177
- put: pull-request
82178
params: {path: fog-google-src, status: pending}
83179

84-
- task: full-integration-tests
180+
- task: run-integration-tests
85181
file: fog-google-src/ci/tasks/run-int.yml
86182
params:
87-
rake_task: test:sql
183+
rake_task: test:sql-sqlv2
88184
codecov_token: {{codecov_token}}
89185
google_project: {{google_project}}
90186
google_json_key_data: {{google_json_key_data}}
@@ -105,7 +201,7 @@ jobs:
105201
- put: pull-request
106202
params: {path: fog-google-src, status: pending}
107203

108-
- task: full-integration-tests
204+
- task: run-integration-tests
109205
file: fog-google-src/ci/tasks/run-int.yml
110206
params:
111207
rake_task: test:storage
@@ -124,10 +220,14 @@ jobs:
124220
- get: fog-google-src
125221
resource: pull-request
126222
passed: [integration-tests-storage,
127-
integration-tests-sql,
128223
integration-tests-pubsub,
129224
integration-tests-monitoring,
130-
integration-tests-compute]
225+
integration-sqlv1,
226+
integration-sqlv2,
227+
integration-compute-core_compute,
228+
integration-compute-core_networking,
229+
integration-compute-instance_groups,
230+
integration-compute-loadbalancing]
131231
trigger: true
132232
on_success:
133233
put: pull-request

‎ci/tasks/run-int.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ EOL
2929

3030
pushd ${release_dir} > /dev/null
3131

32-
bundle install
32+
bundle install --jobs=3 --retry=3
3333

3434
FOG_MOCK=false COVERAGE=true CODECOV_TOKEN=${codecov_token} rake ${rake_task}
3535

‎ci/tasks/run-int.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
platform: linux
33
image_resource:
44
type: docker-image
5-
source: {repository: everlag/fog-google}
5+
source: {repository: rubyfog/fog-google}
66
inputs:
77
- name: fog-google-src
88
path: src/fog-google

‎tasks/test.rake‎

Lines changed: 43 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,20 +24,33 @@ namespace :test do
2424
t.name = "unit"
2525
t.description = "Run Unit tests"
2626
t.libs << "test"
27-
t.pattern = FileList['test/unit/**/test_*.rb']
27+
t.pattern = FileList["test/unit/**/test_*.rb"]
2828
t.warning = false
2929
t.verbose = true
3030
end
3131

32-
Rake::TestTask.new do |t|
33-
t.name = "compute"
34-
t.description = "Run Compute API tests"
35-
t.libs << "test"
36-
t.pattern = FileList["test/integration/compute/test_*.rb"]
37-
t.warning = false
38-
t.verbose = true
32+
# This autogenerates rake tasks based on test folder structures
33+
# This is done to simplify running many test suites in parallel
34+
COMPUTE_TEST_TASKS = []
35+
Dir.glob("test/integration/compute/**").each do |task|
36+
suite_collection = task.gsub(/test\/integration\/compute\//, "")
37+
component_name = task.gsub(/test\/integration\//, "").split("/").first
38+
Rake::TestTask.new(:"#{component_name}-#{suite_collection}") do |t|
39+
t.libs << "test"
40+
t.description = "Autotask - run #{component_name} integration tests - #{suite_collection}"
41+
t.pattern = FileList["test/integration/#{component_name}/#{suite_collection}/test_*.rb"]
42+
t.warning = false
43+
t.verbose = true
44+
end
45+
COMPUTE_TEST_TASKS << "#{component_name}-#{suite_collection}"
3946
end
4047

48+
desc "Run Compute API tests"
49+
task :compute => COMPUTE_TEST_TASKS
50+
51+
desc "Run Compute API tests in parallel"
52+
multitask :compute_parallel => COMPUTE_TEST_TASKS
53+
4154
Rake::TestTask.new do |t|
4255
t.name = "monitoring"
4356
t.description = "Run Monitoring API tests"
@@ -56,6 +69,28 @@ namespace :test do
5669
t.verbose = true
5770
end
5871

72+
# This autogenerates rake tasks based on test folder structures
73+
# This is done to simplify running many test suites in parallel
74+
SQL_TEST_TASKS = []
75+
Dir.glob("test/integration/sql/**").each do |task|
76+
suite_collection = task.gsub(/test\/integration\/sql\//, "")
77+
component_name = task.gsub(/test\/integration\//, "").split("/").first
78+
Rake::TestTask.new(:"#{component_name}-#{suite_collection}") do |t|
79+
t.libs << "test"
80+
t.description = "Autotask - run #{component_name} integration tests - #{suite_collection}"
81+
t.pattern = FileList["test/integration/#{component_name}/#{suite_collection}/test_*.rb"]
82+
t.warning = false
83+
t.verbose = true
84+
end
85+
SQL_TEST_TASKS << "#{component_name}-#{suite_collection}"
86+
end
87+
88+
desc "Run SQL API tests"
89+
task :compute => SQL_TEST_TASKS
90+
91+
desc "Run SQL API tests in parallel"
92+
multitask :compute_parallel => SQL_TEST_TASKS
93+
5994
Rake::TestTask.new do |t|
6095
t.name = "sql"
6196
t.description = "Run SQL API tests"

‎test/integration/compute/core_compute/test_coverage.rb‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
require "helpers/integration_test_helper"
2+
3+
# This is a simple coverage helper that helps differentiate
4+
# the tests when run in parallel so the final coverage report
5+
# can be properly combined together from multiple runners
6+
SimpleCov.command_name "test:compute-core_compute" if ENV["COVERAGE"]

‎test/integration/compute/test_disk_types.rb‎ renamed to ‎test/integration/compute/core_compute/test_disk_types.rb‎

File renamed without changes.

‎test/integration/compute/test_disks.rb‎ renamed to ‎test/integration/compute/core_compute/test_disks.rb‎

File renamed without changes.

0 commit comments

Comments
 (0)