Add support for goproxy_server and go.env files #12747

jurre · 2025-07-30T14:31:52Z

What are you trying to accomplish?

In order for customers to leverage go proxy servers and/or to configure environment variables used by go to determine how to resolve dependencies, we'll now do two things:

If a go.env file is present at the root of the repository, we'll pull it in and tell the Go toolchain to use it.
If a credential for a goproxy_server is passed in, and no go.env exists, we export the URLs from the credentials as GOPROXY, allowing the toolchain to use it. By default we also append direct so that direct access to dependencies is still possible.

This should give customers all the flexibility they need to configure how go mod accesses dependencies for their projects, with nice defaults.

Anything you want to highlight for special attention from reviewers?

The approach I've taken here is to export these values at the file fetching step, from here on the toolchain should use them.

Since every job runs in its own container, once the job finished the environment is wiped when done.

How will you know you've accomplished your goal?

Tests and manual verification

Checklist

I have run the complete test suite to ensure all tests and linters pass.
I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
I have written clear and descriptive commit messages.
I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
I have ensured that the code is well-documented and easy to understand.

jakecoffman

👍 LGTM, just optional nitpicks

go_modules/lib/dependabot/go_modules/file_fetcher.rb

jakecoffman · 2025-07-30T15:30:02Z

go_modules/lib/dependabot/go_modules/file_parser.rb

+        if go_env
+          T.must(go_env)
+          File.write(T.must(go_env).name, T.must(go_env).content)
+          go_env_path = Pathname.new(T.must(go_env).name).realpath.to_s
+          ENV["GOENV"] = go_env_path
+        end


In theory you could always set GOENV to the repo without checking the file exists. The Go CLI will not error if it doesn't find a go.env there. Tried it locally seems to be ok with it.

Hmm I think I prefer safeguarding since it's cheap, just in case the behavior ever changes

kbukum1

LGTM

jurre · 2025-07-31T13:28:40Z

This actually needs a bit more work, there are some places where we're explicitly passing in a GOPRIVATE environment variable based on what's passed in from job payload, and I think we need to unset these if they are set via the go.env file

go_modules/spec/dependabot/go_modules/file_updater/go_mod_updater_spec.rb

Nishnha · 2025-08-20T15:32:52Z

go_modules/lib/dependabot/go_modules/file_parser.rb

+      def set_goproxy_variable
+        return if go_env&.content&.include?("GOPROXY")
+
+        goproxy_credentials = credentials.select { |cred| cred["type"] == "goproxy_server" }


where do these goproxy_server credentials come from? In the go tests I see git_source credentials being used.

They would come in just like other credentials

go_modules/lib/dependabot/go_modules/file_parser.rb

alhss · 2025-08-22T20:25:39Z

goprivate testing

I tested your branch against a test repo containing a vitess dependency using two different go.env configurations to validate the environment variable handling works as expected. I built a Docker image from this branch and ran the tests against this custom image to ensure the go.env functionality was properly integrated.

When I set GOPRIVATE="*" in the go.env file, Dependabot behaved exactly as intended - it bypassed the Go package manager proxy and fetched directly from the GitHub source repository. This direct fetching naturally excluded the problematic incompatible versions that were causing issues in the original vitess problem.
When I set GOPRIVATE="" to force proxy usage, Dependabot encountered the same error we've been seeing in the opaque repo when attempting to fetch vitess versions from the Go package manager. This consistent error behavior across both test environments confirms that your go.env implementation is correctly parsing the environment variables and routing dependency requests according to the specified GOPRIVATE settings.

alhss · 2025-08-25T19:14:20Z

GOPROXY testing

I successfully created a GO private registry through Jfrog. And used it as proxy here
When I set GOPRIVATE="" , all Go modules were successfully routed through the private registry as expected. However, when is set to GOPRIVATE="*" or specific patterns(github.com) or not present at all, Go bypassed the configured proxy entirely and fetched directly from upstream sources like GitHub.
I'm not sure if that's the intended behavior, When a goprivate is not present, the goproxy is completely bypassed. What do you think? @jurre

jurre · 2025-08-26T11:04:04Z

Go bypassed the configured proxy entirely and fetched directly from upstream sources like GitHub.

Hmm, yeah I think that is problematic because we inject GOPRIVATE=* outside of users' control, I think we might need to avoid setting automatically it if a registry is configured, and only set it if explicitly configured via go.env?

alhss · 2025-08-26T14:08:43Z

Go bypassed the configured proxy entirely and fetched directly from upstream sources like GitHub.

Hmm, yeah I think that is problematic because we inject GOPRIVATE=* outside of users' control, I think we might need to avoid setting automatically it if a registry is configured, and only set it if explicitly configured via go.env?

@jurre i think the best solution here is probably Only inject GOPRIVATE when GOPROXY is not present, and do nothing when goproxy is present. that reduces complexity on customers about always setting goprivate

jurre · 2025-08-26T19:06:37Z

@jurre i think the best solution here is probably Only inject GOPRIVATE when GOPROXY is not present, and do nothing when goproxy is present. that reduces complexity on customers about always setting goprivate

I think that's right, let me run through the logic and think about potential edge cases for a bit tomorrow though

go_modules/lib/dependabot/go_modules/file_parser.rb

In order for customers to leverage go proxy servers and/or to configure environment variables used by go to determine how to resolve dependencies, we'll now do two things: - If a `go.env` file is present at the root of the repository, we'll pull it in and tell the Go toolchain to use it. - If a credential for a goproxy_server is passed in, and no `go.env` exists, we export the URLs from the credentials as `GOPROXY`, allowing the toolchain to use it. By default we also append `direct` so that direct access to dependencies is still possible. This should give customers all the flexibility they need to configure how go mod accesses dependencies for their projects, with nice defaults. The approach I've taken here is to export these values at the file fetching step, from here on the toolchain should use them. Since every job runs in its own container, once the job finished the environment is wiped when done.

Co-authored-by: Jake Coffman <[email protected]>

This makes it easier to configure things correctly once, since the local variables would otherwise override what is configured via go.env

By default dependabot will set `GOPRIVATE=*`, which will cause none of the dependencies to go through the proxy that was configured. When `go.env` configures this value it is still respected. Co-authored-by: alhss <[email protected]>

jurre · 2025-08-28T12:34:16Z

Seeing a failure in CI now that seems not related to the changes here and I can reproduce on main, but taking a look at it nonetheless

alhss · 2025-08-28T19:34:24Z

Seeing a failure in CI now that seems not related to the changes here and I can reproduce on main, but taking a look at it nonetheless

It was a flaky test

jurre requested a review from a team as a code owner July 30, 2025 14:31

github-actions bot added the L: go:modules Golang modules label Jul 30, 2025

jurre force-pushed the jurre/go-proxy-support branch from fffd129 to 5b8ebd6 Compare July 30, 2025 15:15

jakecoffman previously approved these changes Jul 30, 2025

View reviewed changes

jurre dismissed jakecoffman’s stale review via 2898822 July 30, 2025 15:41

kbukum1 previously approved these changes Jul 30, 2025

View reviewed changes

jakecoffman mentioned this pull request Jul 31, 2025

Set goprivate experiment for Go modules with git_source credentials github/dependabot-action#1526

Open

jurre dismissed kbukum1’s stale review via 89a5b00 August 4, 2025 09:20

jurre force-pushed the jurre/go-proxy-support branch 3 times, most recently from 0cd2a3e to cd07532 Compare August 5, 2025 09:52

jurre commented Aug 5, 2025

View reviewed changes

go_modules/spec/dependabot/go_modules/file_updater/go_mod_updater_spec.rb Show resolved Hide resolved

jurre force-pushed the jurre/go-proxy-support branch 4 times, most recently from adb6f9c to a308639 Compare August 6, 2025 08:00

jurre force-pushed the jurre/go-proxy-support branch from a308639 to 4e5520e Compare August 20, 2025 09:55

Nishnha reviewed Aug 20, 2025

View reviewed changes

go_modules/lib/dependabot/go_modules/file_parser.rb Show resolved Hide resolved

jurre force-pushed the jurre/go-proxy-support branch 2 times, most recently from 16a37c9 to c19a2da Compare August 27, 2025 12:29

alhss reviewed Aug 27, 2025

View reviewed changes

go_modules/lib/dependabot/go_modules/file_parser.rb Show resolved Hide resolved

jurre force-pushed the jurre/go-proxy-support branch from c19a2da to fdd8331 Compare August 28, 2025 11:22

jurre and others added 2 commits August 28, 2025 13:22

fetch support file explicitly

506c67a

Co-authored-by: Jake Coffman <[email protected]>

Rely on global Go environment variables for all commands

445f7ff

This makes it easier to configure things correctly once, since the local variables would otherwise override what is configured via go.env

jurre force-pushed the jurre/go-proxy-support branch from fdd8331 to d6101c6 Compare August 28, 2025 11:22

jurre force-pushed the jurre/go-proxy-support branch from d6101c6 to ae9c361 Compare August 28, 2025 11:24

Nishnha approved these changes Aug 28, 2025

View reviewed changes

alhss added 2 commits August 29, 2025 09:42

Merge branch 'main' into jurre/go-proxy-support

d4c52eb

Merge branch 'main' into jurre/go-proxy-support

641565c

alhss merged commit 637b597 into main Aug 29, 2025
81 checks passed

alhss deleted the jurre/go-proxy-support branch August 29, 2025 16:21

Add support for goproxy_server and go.env files #12747

Add support for goproxy_server and go.env files #12747

Uh oh!

Conversation

jurre commented Jul 30, 2025

What are you trying to accomplish?

Anything you want to highlight for special attention from reviewers?

How will you know you've accomplished your goal?

Checklist

Uh oh!

jakecoffman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakecoffman Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

jurre Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

kbukum1 left a comment

Choose a reason for hiding this comment

Uh oh!

jurre commented Jul 31, 2025

Uh oh!

Uh oh!

Nishnha Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

jurre Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alhss commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

goprivate testing

Uh oh!

alhss commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GOPROXY testing

Uh oh!

jurre commented Aug 26, 2025

Uh oh!

alhss commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jurre commented Aug 26, 2025

Uh oh!

Uh oh!

jurre commented Aug 28, 2025

Uh oh!

alhss commented Aug 28, 2025

Uh oh!

Uh oh!

Uh oh!

alhss commented Aug 22, 2025 •

edited

Loading

alhss commented Aug 25, 2025 •

edited

Loading

alhss commented Aug 26, 2025 •

edited

Loading