Handle unmodified sensitived fields when updating connections #53943
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pierrejeambrun
requested review from
ashb,
kaxil,
amoghrajesh,
bbovenzi,
ryanahamilton,
jscheffl,
shubhamraj-git,
ephraimbuddy,
rawwar,
jason810496 and
bugraoz93
as code owners
boring-cyborg
bot
added
area:API
pierrejeambrun
removed
area:dev-tools
area:API
You just found a bug. I was able to reproduce on main, seems to happen when 'extra_json' is a str. I created the issue: #53963, more info there. |
@shubhamraj-git I was able to reproduce on main, I just opened an issue there: |
pierrejeambrun
force-pushed
the
fix-52301
branch
from
5542e33 to
1499d29
Compare
This was referenced Jul 31, 2025
ashb
approved these changes
Jul 31, 2025
Lee-W
reviewed
Jul 31, 2025
amoghrajesh
approved these changes
Jul 31, 2025
pierrejeambrun
added
the
backport-to-v3-0-test
Backport failed to create: v3-0-test. View the failure log Run details
You can attempt to backport this manually by running: cherry_picker 0abcfdf v3-0-testThis should apply the commit to the v3-0-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continue |
|
Manual backport #53973 |
pierrejeambrun
added a commit
that referenced
this pull request
Jul 31, 2025
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 31, 2025
For logs, using `***` is fine, but as part of the changes introduced in apache#53943 we decided it might be nice to use an even-less-frequently-appearing thing than `***` so we can detect modified secrets. This gives us the ability to do that at the redaction layer
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 31, 2025
For logs, using `***` is fine, but as part of the changes introduced in apache#53943 we decided it might be nice to use an even-less-frequently-appearing thing than `***` so we can detect modified secrets. This gives us the ability to do that at the redaction layer
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 31, 2025
For logs, using `***` is fine, but as part of the changes introduced in apache#53943 we decided it might be nice to use an even-less-frequently-appearing thing than `***` so we can detect modified secrets. This gives us the ability to do that at the redaction layer
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 31, 2025
For logs, using `***` is fine, but as part of the changes introduced in apache#53943 we decided it might be nice to use an even-less-frequently-appearing thing than `***` so we can detect modified secrets. This gives us the ability to do that at the redaction layer
RoyLee1224
pushed a commit
to RoyLee1224/airflow
that referenced
this pull request
Jul 31, 2025
…#53943) * Fix redacted values editing * Small improvements * Small adjustments * Update UI and fix some errors * Address PR comments
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 1, 2025
For logs, using `***` is fine, but as part of the changes introduced in apache#53943 we decided it might be nice to use an even-less-frequently-appearing thing than `***` so we can detect modified secrets. This gives us the ability to do that at the redaction layer
ashb
added a commit
that referenced
this pull request
Aug 4, 2025
…53977) * Allow secrets redact function to have different redaction than `***` For logs, using `***` is fine, but as part of the changes introduced in #53943 we decided it might be nice to use an even-less-frequently-appearing thing than `***` so we can detect modified secrets. This gives us the ability to do that at the redaction layer * Deal with OpenLineage subclassing SecretsMasker class
40 tasks
ferruzzi
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Aug 7, 2025
…#53943) * Fix redacted values editing * Small improvements * Small adjustments * Update UI and fix some errors * Address PR comments
ferruzzi
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Aug 7, 2025
…pache#53977) * Allow secrets redact function to have different redaction than `***` For logs, using `***` is fine, but as part of the changes introduced in apache#53943 we decided it might be nice to use an even-less-frequently-appearing thing than `***` so we can detect modified secrets. This gives us the ability to do that at the redaction layer * Deal with OpenLineage subclassing SecretsMasker class
fweilun
pushed a commit
to fweilun/airflow
that referenced
this pull request
Aug 11, 2025
…#53943) * Fix redacted values editing * Small improvements * Small adjustments * Update UI and fix some errors * Address PR comments
fweilun
pushed a commit
to fweilun/airflow
that referenced
this pull request
Aug 11, 2025
…pache#53977) * Allow secrets redact function to have different redaction than `***` For logs, using `***` is fine, but as part of the changes introduced in apache#53943 we decided it might be nice to use an even-less-frequently-appearing thing than `***` so we can detect modified secrets. This gives us the ability to do that at the redaction layer * Deal with OpenLineage subclassing SecretsMasker class
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #52301
closes: #53753
How it works:
For connection, 'password' and 'extras' are merged with their original value when doing an update. The function works similarly to the
redactfunction, it will recursively handle all sort of data types and detect sensitive values that were not modified in the 'new_value' and then restore the value from the unredacted previous value.See the warning note:
After adding a key 'new_key' to the extra and saving this is what we get in the UI:
And from the CLI we can see that both password, and extra redacted field were preserved:
The only downside is that we cannot 'insert' a real '***' for redacted value because this is how we detect that the value didn't change. I think it's a fair limitation, '***' shouldn't never be a valid value for a sensitive field anyway. @ashb is working on a follow up PR to instead use unicode characters that looks like '***' but are not, to make it even less likely that it will be blocking for users. (They would have to chose a very weird value for their secret).
Another example, it also handle well arrays: