Adding build scripts for building the workspace packages using pnpm #21326

bhaveshbhati · 2025-09-16T11:13:14Z

Context

This pull request introduces parallel build support for tasks, improves build performance, and adds robust concurrent download utilities. The main changes include a new parallel build script, enhancements to the build process, and updates to dependencies and workspace configuration for better task filtering and concurrency.

Work item: https://mseng.visualstudio.com/AzureDevOps/_workitems/edit/2318396

Task Name

NotationV0 - Only change is updating the build command. So not updating the Task version.

Description

Parallel build support and task filtering:

Added new build-parallel.js script that enables parallel building of tasks using pnpm workspace filters, supporting both single and multiple task patterns for efficient concurrent builds.
Updated package.json with new scripts (build:parallel, serverBuild:parallel) and added pnpm as a devDependency to support parallel builds. [1] [2]
Added pnpm-workspace.yaml to define workspace packages and exclude common tasks from parallel builds.

Concurrent download improvements:

Introduced build-scripts/download-utils.js with downloadArchiveConcurrentAsync, providing safe, atomic concurrent archive extraction to prevent race conditions during parallel builds.
Integrated new concurrent download utility into make-util.js so downloads are handled atomically when parallel build is enabled. [1] [2]

Build process enhancements:

Modified make.js to allow skipping main build steps and only running pre-build steps when requested, supporting parallel build orchestration.
Changed the NotationV0 task build script to use the new build system for consistency with parallel builds..

Risk Assessment (Low)

Low risk. The change is limited to argument formatting and does not affect core build or task execution logic. Backward compatibility is maintained.

Change Behind Feature Flag (No)

This change cannot be behind a feature flag, as it affects command-line argument formatting required for correct build execution.

Tech Design / Approach

The approach was to update the function responsible for generating the filter argument, ensuring double quotes are always applied.
No architectural changes; the update is localized to argument construction.

Documentation Changes Required (No)

No documentation changes required, as this is an internal implementation detail.

Unit Tests Added or Updated (No)

No new unit tests were added. Existing build/test flows cover this logic.

Additional Testing Performed

Manual testing: Verified that builds using the updated filter logic work as expected.
Regression: Ensured no impact on other build scenarios.

Logging Added/Updated (No)

No logging changes were necessary for this update.

Telemetry Added/Updated (No)

No telemetry changes required.

Rollback Scenario and Process (Yes)

Rollback is straightforward: revert the changes in build-parallel.js to the previous logic.

Dependency Impact Assessed and Regression Tested (Yes)

No new dependencies introduced. Regression testing confirmed no breakage in existing build flows.

Checklist

Related issue linked (if applicable)
Task version was bumped — see versioning guide
Verified the task behaves as expected

bhaveshbhati · 2025-09-16T11:15:31Z

/azp run

azure-pipelines · 2025-09-16T11:15:51Z

Azure Pipelines successfully started running 3 pipeline(s).

bhaveshbhati · 2025-09-17T09:23:49Z

/azp run

azure-pipelines · 2025-09-17T09:24:10Z

Azure Pipelines successfully started running 3 pipeline(s).

bhaveshbhati · 2025-09-17T09:31:03Z

/azp run

azure-pipelines · 2025-09-17T09:31:23Z

Azure Pipelines successfully started running 3 pipeline(s).

bhaveshbhati · 2025-09-17T16:30:33Z

/azp run

azure-pipelines · 2025-09-17T16:30:54Z

Azure Pipelines successfully started running 3 pipeline(s).

…om/microsoft/azure-pipelines-tasks into users/bhatibhavesh/concurrentbuild

bhaveshbhati · 2025-09-19T08:45:53Z

/azp run

azure-pipelines · 2025-09-19T08:46:16Z

Azure Pipelines successfully started running 3 pipeline(s).

build-scripts/download-utils.js

build-parallel.js

bhaveshbhati · 2025-09-24T06:51:58Z

/azp run

azure-pipelines · 2025-09-24T06:52:05Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

bhaveshbhati · 2025-09-24T06:52:29Z

/azp run

azure-pipelines · 2025-09-24T06:52:50Z

Azure Pipelines successfully started running 3 pipeline(s).

Copilot

Pull Request Overview

This PR introduces parallel build support for Azure Pipelines tasks using pnpm workspace filtering, enhancing build performance through concurrent execution. The changes add a new parallel build script, implement concurrent download utilities to prevent race conditions, and update the workspace configuration to support filtered parallel builds.

Adds parallel build infrastructure with pnpm workspace support and task filtering capabilities
Implements concurrent-safe download utilities to handle race conditions during parallel builds
Updates build configuration and task-specific build commands for consistency

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pnpm-workspace.yaml	Defines workspace packages for pnpm, excluding Common tasks from parallel builds
package.json	Adds pnpm dependency and new parallel build scripts
make.js	Adds conditional logic to skip main build steps when only pre-build steps are needed
make-util.js	Integrates concurrent download utility for parallel build scenarios
ci/build-all-steps.yml	Updates branch filter condition for CI pipeline
build-scripts/download-utils.js	New concurrent download utility with atomic extraction and race condition prevention
build-parallel.js	New parallel build orchestration script with pnpm workspace filtering
Tasks/NotationV0/package.json	Updates build command to use make.js instead of direct tsc

Copilot · 2025-09-26T08:37:08Z

build-scripts/download-utils.js

+        if(throwOnError)
+        {
+            throw new Error('Failed to run: ' + cl + ' exit code: ' + err.status);
+        }else{


Missing space after 'if' keyword and inconsistent brace placement. The condition should be if (throwOnError) and the else should follow standard formatting conventions with proper spacing.

Suggested change

if(throwOnError)

{

throw new Error('Failed to run: ' + cl + ' exit code: ' + err.status);

}else{

if (throwOnError)

{

throw new Error('Failed to run: ' + cl + ' exit code: ' + err.status);

}

else

{

Copilot · 2025-09-26T08:37:08Z

build-parallel.js

+        const quotedArgs = args.map(arg => {
+            // Quote arguments that contain pipes, parentheses, or spaces
+            if (arg.includes('|') || arg.includes('(') || arg.includes(')') || arg.includes(' ')) {
+                return `"${arg}"`;
+            }
+            return arg;
+        });


The argument quoting logic is flawed because arguments that are already quoted (like "./Tasks/{Task1,Task2,Task3}") will be double-quoted, potentially breaking shell command execution. The function should check if arguments are already properly quoted before adding additional quotes.

Copilot · 2025-09-26T08:37:08Z

build-parallel.js

+            return ['--filter', `"./Tasks/${tasks[0]}"`];
+        } else {
+            // Create a single filter pattern that matches multiple task paths
+            const pathPattern = `./Tasks/{${tasks.join(',')}}`;
+            return ['--filter', `"${pathPattern}"`];
+        }
+    }
+
+    // Handle single task or other patterns - use path-based filter
+    return ['--filter', `"./Tasks/${taskPattern}"`];


The filter argument generation is inconsistent and could lead to confusion. All three return statements manually wrap values in double quotes, but this should be handled consistently in one place, preferably in the executeCommand function's argument processing.

Suggested change

return ['--filter', `"./Tasks/${tasks[0]}"`];

} else {

// Create a single filter pattern that matches multiple task paths

const pathPattern = `./Tasks/{${tasks.join(',')}}`;

return ['--filter', `"${pathPattern}"`];

}

}

// Handle single task or other patterns - use path-based filter

return ['--filter', `"./Tasks/${taskPattern}"`];

return ['--filter', `./Tasks/${tasks[0]}`];

} else {

// Create a single filter pattern that matches multiple task paths

const pathPattern = `./Tasks/{${tasks.join(',')}}`;

return ['--filter', `${pathPattern}`];

}

}

// Handle single task or other patterns - use path-based filter

return ['--filter', `./Tasks/${taskPattern}`];

Copilot · 2025-09-26T08:37:09Z

build-parallel.js

+            return ['--filter', `"./Tasks/${tasks[0]}"`];
+        } else {
+            // Create a single filter pattern that matches multiple task paths
+            const pathPattern = `./Tasks/{${tasks.join(',')}}`;
+            return ['--filter', `"${pathPattern}"`];
+        }
+    }
+
+    // Handle single task or other patterns - use path-based filter
+    return ['--filter', `"./Tasks/${taskPattern}"`];


The filter argument generation is inconsistent and could lead to confusion. All three return statements manually wrap values in double quotes, but this should be handled consistently in one place, preferably in the executeCommand function's argument processing.

Suggested change

return ['--filter', `"./Tasks/${tasks[0]}"`];

} else {

// Create a single filter pattern that matches multiple task paths

const pathPattern = `./Tasks/{${tasks.join(',')}}`;

return ['--filter', `"${pathPattern}"`];

}

}

// Handle single task or other patterns - use path-based filter

return ['--filter', `"./Tasks/${taskPattern}"`];

return ['--filter', `./Tasks/${tasks[0]}`];

} else {

// Create a single filter pattern that matches multiple task paths

const pathPattern = `./Tasks/{${tasks.join(',')}}`;

return ['--filter', pathPattern];

}

}

// Handle single task or other patterns - use path-based filter

return ['--filter', `./Tasks/${taskPattern}`];

Copilot · 2025-09-26T08:37:09Z

build-parallel.js

+            return ['--filter', `"./Tasks/${tasks[0]}"`];
+        } else {
+            // Create a single filter pattern that matches multiple task paths
+            const pathPattern = `./Tasks/{${tasks.join(',')}}`;
+            return ['--filter', `"${pathPattern}"`];
+        }
+    }
+
+    // Handle single task or other patterns - use path-based filter
+    return ['--filter', `"./Tasks/${taskPattern}"`];


The filter argument generation is inconsistent and could lead to confusion. All three return statements manually wrap values in double quotes, but this should be handled consistently in one place, preferably in the executeCommand function's argument processing.

Suggested change

return ['--filter', `"./Tasks/${tasks[0]}"`];

} else {

// Create a single filter pattern that matches multiple task paths

const pathPattern = `./Tasks/{${tasks.join(',')}}`;

return ['--filter', `"${pathPattern}"`];

}

}

// Handle single task or other patterns - use path-based filter

return ['--filter', `"./Tasks/${taskPattern}"`];

return ['--filter', `./Tasks/${tasks[0]}`];

} else {

// Create a single filter pattern that matches multiple task paths

const pathPattern = `./Tasks/{${tasks.join(',')}}`;

return ['--filter', `${pathPattern}`];

}

}

// Handle single task or other patterns - use path-based filter

return ['--filter', `./Tasks/${taskPattern}`];

tarunramsinghani · 2025-09-26T08:49:57Z

build-scripts/download-utils.js

+        }
+
+        // Atomically move to final location (only one process will succeed)
+        try {


if this is remained from prev run it will assume it was downloaded fresh and skip ?

tarunramsinghani · 2025-09-26T10:47:40Z

build-parallel.js

+    console.log('Running parallel build...');
+    console.log('pnpm command:', 'pnpm', args.join(' '));
+    try {


Currently the make.js prints output like this

===============================================================================
📊 BUILD SUMMARY

Total tasks: 5
✅ Successful: 0
❌ Failed: 0
⏭️ Skipped: 0

That does not work with this script anymore, we will need to update that as well.

tarunramsinghani · 2025-09-26T10:49:46Z

build-parallel.js

+        console.log('');
+        console.log('Examples:');
+        console.log('  node build-parallel.js build --task "MyTask"');
+        console.log('  node build-parallel.js serverBuild --task "@(TaskA|TaskB|TaskC)"');


This command silently ignores the wrong taskname instead it should throw an error to avoid typos and inform users of that.

node build-parallel.js build --task "@(BashV3|NonExistentTask|CmdLineV2)"

Bhavesh Bhati and others added 6 commits September 15, 2025 23:22

Scripts for building the workspaces

98091d9

Update build to run parallel

027937e

Temp changes to validate on Mac and Linux

71bfcf9

Put the tasks in quotes

7af443b

Revert the changes for build

0a9e761

Merge branch 'master' into users/bhatibhavesh/concurrentbuild

76bcb5a

Merge branch 'master' into users/bhatibhavesh/concurrentbuild

8242c8b

bhaveshbhati marked this pull request as ready for review September 17, 2025 09:23

bhaveshbhati requested review from JeyJeyGao, tarunramsinghani and a team as code owners September 17, 2025 09:23

Minor, fix the package.json

e6f9cf7

Merge branch 'master' into users/bhatibhavesh/concurrentbuild

24eec8d

Bhavesh Bhati added 2 commits September 19, 2025 14:15

Correct package and package-lock

4f77963

Merge branch 'users/bhatibhavesh/concurrentbuild' of https://github.c…

feed74d

…om/microsoft/azure-pipelines-tasks into users/bhatibhavesh/concurrentbuild

sanjuyadav24 reviewed Sep 22, 2025

View reviewed changes

build-scripts/download-utils.js Outdated Show resolved Hide resolved

build-parallel.js Show resolved Hide resolved

build-parallel.js Show resolved Hide resolved

Code review comments

3c61592

sanjuyadav24 approved these changes Sep 24, 2025

View reviewed changes

Merge branch 'master' into users/bhatibhavesh/concurrentbuild

c37e6a7

tarunramsinghani requested a review from Copilot September 26, 2025 08:35

Copilot AI reviewed Sep 26, 2025

View reviewed changes

tarunramsinghani reviewed Sep 26, 2025

View reviewed changes

Adding build scripts for building the workspace packages using pnpm #21326

Are you sure you want to change the base?

Adding build scripts for building the workspace packages using pnpm #21326

Uh oh!

Conversation

bhaveshbhati commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Task Name

Description

Risk Assessment (Low)

Change Behind Feature Flag (No)

Tech Design / Approach

Documentation Changes Required (No)

Unit Tests Added or Updated (No)

Additional Testing Performed

Logging Added/Updated (No)

Telemetry Added/Updated (No)

Rollback Scenario and Process (Yes)

Dependency Impact Assessed and Regression Tested (Yes)

Checklist

Uh oh!

bhaveshbhati commented Sep 16, 2025

Uh oh!

azure-pipelines bot commented Sep 16, 2025

Uh oh!

bhaveshbhati commented Sep 17, 2025

Uh oh!

azure-pipelines bot commented Sep 17, 2025

Uh oh!

bhaveshbhati commented Sep 17, 2025

Uh oh!

azure-pipelines bot commented Sep 17, 2025

Uh oh!

bhaveshbhati commented Sep 17, 2025

Uh oh!

azure-pipelines bot commented Sep 17, 2025

Uh oh!

bhaveshbhati commented Sep 19, 2025

Uh oh!

azure-pipelines bot commented Sep 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bhaveshbhati commented Sep 24, 2025

Uh oh!

azure-pipelines bot commented Sep 24, 2025

Uh oh!

bhaveshbhati commented Sep 24, 2025

Uh oh!

azure-pipelines bot commented Sep 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

tarunramsinghani Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

tarunramsinghani Sep 26, 2025

Choose a reason for hiding this comment

=============================================================================== 📊 BUILD SUMMARY

Total tasks: 5 ✅ Successful: 0 ❌ Failed: 0 ⏭️ Skipped: 0

bhaveshbhati commented Sep 16, 2025 •

edited

Loading

===============================================================================
📊 BUILD SUMMARY

Total tasks: 5
✅ Successful: 0
❌ Failed: 0
⏭️ Skipped: 0