Skip to content

Conversation

brendandahl
Copy link
Collaborator

Testing to see how well the browser tests parallelize in CI.

@brendandahl
Copy link
Collaborator Author

test-browser-chrome-2gb test-browser-chrome-wasm64-4gb test-browser-firefox
No Parallel (medium - 1 runner)
0:46:16 0:42:50 0:36:09
0:46:46 0:45:12 0:40:50
0:45:36 0:43:26 0:34:26
0:46:44 0:33:16 0:35:59
0:48:29 0:44:50 0:34:02
average 0:46:46 0:41:55 0:36:17
Parallel (medium 4 - 2 runners)
0:29:26 0:28:15 0:22:45
0:28:00 0:27:00 0:21:42
0:28:58 0:31:12 0:21:41
0:26:58 0:26:29 0:21:57
0:27:18 0:26:16 0:20:33
average 0:28:08 0:27:50 0:21:44
speedup from baseline 1.66 1.51 1.67
Parallel (xlarge 16 - 8 runners)
0:09:49 0:09:29 0:07:30
speedup from baseline 4.76 4.42 4.84

@sbc100
Copy link
Collaborator

sbc100 commented Sep 5, 2025

Given that several other non-browser test suites take ~30min it seems like spending more money to make the browser tests quicker would not be wroth it unless we also were to speed up those other test suites too.

I uploaded #25194 so see if I could speed up the new slowest runner: test-wasm64-4gb

@juj
Copy link
Collaborator

juj commented Sep 5, 2025

For curiosity, I tested different parallelism values on my Linux box:

16: Total core time: 1778.549s. Wallclock time: 114.699s. Parallelization: 15.51x.

32: Total core time: 2059.722s. Wallclock time: 70.208s. Parallelization: 29.34x.
32: Total core time: 2042.970s. Wallclock time: 70.036s. Parallelization: 29.17x.

64: Total core time: 2532.726s. Wallclock time: 49.703s. Parallelization: 50.96x.
64: Total core time: 2506.703s. Wallclock time: 49.115s. Parallelization: 51.04x.
64: Total core time: 2568.892s. Wallclock time: 50.292s. Parallelization: 51.08x.
64: Total core time: 2489.469s. Wallclock time: 49.365s. Parallelization: 50.43x
64: Total core time: 2471.243s. Wallclock time: 48.887s. Parallelization: 50.55x.
64: Total core time: 2550.984s. Wallclock time: 49.412s. Parallelization: 51.63x.

80: Total core time: 2817.407s. Wallclock time: 47.546s. Parallelization: 59.26x.
80: Total core time: 2848.213s. Wallclock time: 47.651s. Parallelization: 59.77x.
80: Total core time: 2805.875s. Wallclock time: 47.016s. Parallelization: 59.68x.

96: Total core time: 3016.244s. Wallclock time: 46.486s. Parallelization: 64.89x.
96: Total core time: 2976.783s. Wallclock time: 45.791s. Parallelization: 65.01x.
96: Total core time: 2931.232s. Wallclock time: 44.712s. Parallelization: 65.56x.
96: Total core time: 3005.507s. Wallclock time: 46.153s. Parallelization: 65.12x.
96: Total core time: 2939.468s. Wallclock time: 47.395s. Parallelization: 62.02x.
96: Total core time: 3307.454s. Wallclock time: 52.337s. Parallelization: 63.20x.

128: Total core time: 4522.871s. Wallclock time: 56.776s. Parallelization: 79.66x.
128: Total core time: 3951.026s. Wallclock time: 47.712s. Parallelization: 82.81x.
128: Total core time: 3866.550s. Wallclock time: 52.671s. Parallelization: 73.41x.
128: Total core time: 3629.438s. Wallclock time: 50.599s. Parallelization: 71.73x.
128: Total core time: 3704.257s. Wallclock time: 54.433s. Parallelization: 68.05x.
128: Total core time: 3727.838s. Wallclock time: 53.200s. Parallelization: 70.07x.

160: Total core time: 4832.477s. Wallclock time: 59.705s. Parallelization: 80.94x.
160: <failing, many tests fail on webgl context not being obtained>
160: <failing, many tests fail on webgl context not being obtained>

The system has 256 logical hardware threads, so this experiment showed that the sweet spot is at maybe 64 parallel runners, as adding more is wasting electricity for diminishing returns.

@brendandahl
Copy link
Collaborator Author

That was on browser tests?

@juj
Copy link
Collaborator

juj commented Sep 5, 2025

That was on browser tests?

Err, yeah, exactly. Running browser test suite with your awesome parallelised patch. Also, using Firefox as the test browser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants