feat(core): Implement UCS kill switch for emergency fallback

[maverox]

feat(core): Implement UCS enable/disable configuration for operational control

Type of Change

• Bugfix
• New feature
• Enhancement
• Refactoring
• Dependency updates
• Documentation
• CI/CD

Fixes #8644

Description

This PR implements a database-driven enable/disable configuration for the Unified Connector Service (UCS) integration, providing operators with immediate control to enable or disable UCS without requiring code changes or deployments.

Key Features Implemented:

1. Database Configuration: UCS enabled state stored as UCS_ENABLED config
2. Priority Check: UCS enabled check happens before rollout percentage evaluation
3. Immediate Effect: Changes take effect without service restart
4. Clear Error Handling: Returns appropriate error when UCS is disabled

Architecture:

Payment Request → should_call_unified_connector_service() →
├── Check if UCS client exists
├── Check if UCS is enabled (NEW)
├── Check rollout percentage
└── Return decision (true/false)

Key Technical Improvements:

• Added Enable Check: is_ucs_enabled() function queries database config
• Early Return Logic: UCS enabled state evaluated before any other UCS conditions
• Logging Support: Errors logged when UCS enabled config fetch fails
• Consistent Decision Flow: UCS enabled config affects both regular and internal payment operations

Files Modified:

• crates/router/src/consts.rs - Added UCS_ENABLED constant
• crates/router/src/core/payments/helpers.rs - Added is_ucs_enabled() helper function
• crates/router/src/core/unified_connector_service.rs - Integrated UCS enabled check in decision flow
• crates/router/src/core/payments.rs - Updated fallback behavior for internal operations

Important Caveat:

• Regular Payment Operations (payments_operation_core): When UCS is disabled, should_call_unified_connector_service() returns false and traditional connector flow is used
• Internal Payment Operations (internal_payments_operation_core): Fallback to traditional connector flow has been removed - system now returns error when UCS is unavailable
• This ensures that internal operations have explicit failure handling rather than silent fallback

Additional Changes

• This PR modifies the API contract
• This PR modifies the database schema
• This PR modifies application configuration/environment variables

Motivation and Context

Operational Safety Requirements:

This change addresses critical operational needs for managing the UCS integration in production environments:

Incident Response:

• Immediate Control: During UCS outages, operators can disable it instantly
• No Deployment Required: Enable/disable via database update
• Reduced MTTR: Faster incident resolution without code changes

Risk Management:

• Additional Safety Layer: Complements existing percentage-based rollout
• Granular Control: Can disable UCS while keeping rollout config intact
• Debugging Support: Easily isolate UCS-related issues

Implementation Benefits:

• Zero Downtime: Configuration change without service restart
• Audit Trail: Database config changes are trackable
• Simple Toggle: Enable/disable UCS by changing config value
• Testing Flexibility: Easy A/B testing in production

Use Cases:

1. Emergency Response: Disable UCS during critical incidents
2. Maintenance Windows: Temporarily disable during UCS updates
3. Performance Issues: Quick disable if UCS causes latency
4. Debugging: Isolate issues to UCS vs traditional flow

How did you test it?

Manual Testing

Test Scenario: UCS Enable/Disable Toggle Behavior

1. Default Behavior (UCS Not Configured/Disabled)

• Make a payment request
• Verify traditional connector flow is used when config is not set
• Check logs confirm UCS integration is inactive

2. Enable UCS

curl --location 'http://localhost:8080/configs/UCS_ENABLED' \
--header 'Content-Type: application/json' \
--header 'api-key: test_admin' \
--header 'x-tenant-id: public' \
--data '{
    "key": "UCS_ENABLED",
    "value": "true"
}'

3. Test with UCS Enabled

• Make payment request and verify UCS is called based on rollout percentage
• Check logs for UCS activation based on rollout logic

Sample Payment Request:

curl --location 'http://localhost:8080/payments' \
--header 'Content-Type: application/json' \
--header 'api-key: dev_test_key' \
--data '{
    "amount": 2000,
    "currency": "USD",
    "confirm": true,
    "capture_method": "automatic",
    "payment_method": "card",
    "payment_method_data": {
        "card": {
            "card_number": "4242424242424242",
            "card_exp_month": "10",
            "card_exp_year": "25",
            "card_holder_name": "John Doe",
            "card_cvc": "737"
        }
    },
    "billing": {
        "address": {
            "line1": "1467",
            "line2": "Harrison Street",
            "line3": "Harrison Street",
            "city": "San Francisco",
            "state": "California",
            "zip": "94122",
            "country": "US",
            "first_name": "John",
            "last_name": "Doe"
        }
    }
}'

4. Disable UCS

curl --location 'http://localhost:8080/configs/UCS_ENABLED' \
--header 'Content-Type: application/json' \
--header 'api-key: test_admin' \
--header 'x-tenant-id: public' \
--data '{
    "key": "UCS_ENABLED",
    "value": "false"
}'

5. Verify UCS is Disabled

• Make another payment request (use same curl command from step 3)
• Regular Payment Operations: Verify traditional connector flow is used
• Internal Payment Operations: Verify operations return error (no fallback)
• Confirm system behavior with UCS disabled

Additional Test Cases:

Test Invalid Config Value:

curl --location 'http://localhost:8080/configs/UCS_ENABLED' \
--header 'Content-Type: application/json' \
--header 'api-key: test_admin' \
--header 'x-tenant-id: public' \
--data '{
    "key": "UCS_ENABLED",
    "value": "invalid_value"
}'

• Verify system defaults to UCS disabled
• Check error logs for parsing failure

Retrieve Current UCS Status:

curl --location 'http://localhost:8080/configs/UCS_ENABLED' \
--header 'api-key: test_admin' \
--header 'x-tenant-id: public'

• Verify response shows current UCS enabled state

Expected Results:

• UCS enable/disable takes immediate effect without service restart
• When disabled (false), should_call_unified_connector_service() returns false
• When enabled (true), normal UCS rollout logic applies
• Regular payment operations use traditional connector flow when UCS disabled
• Internal payment operations return error when UCS disabled (no fallback)
• Invalid/missing config defaults to UCS disabled for safety

Checklist

• I formatted the code cargo +nightly fmt --all
• I addressed lints thrown by cargo clippy
• I reviewed the submitted code
• I added unit tests for my changes where possible

[semanticdiff-com]

Review changes with  

Changed Files

File	Status
crates/router/src/consts.rs	38% smaller
crates/router/src/core/unified_connector_service.rs	33% smaller
crates/router/src/core/payments.rs	0% smaller
crates/router/src/core/payments/helpers.rs	0% smaller

[Copilot]

Pull Request Overview

This PR introduces a database-driven kill switch for the Unified Connector Service (UCS) to allow operators to disable UCS without deployments.

• Adds UCS_KILL_SWITCH_ACTIVE constant for the kill switch key
• Implements is_kill_switch_active helper and integrates early kill-switch check in should_call_unified_connector_service
• Updates internal payment flow to return an explicit error when UCS is disabled

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
crates/router/src/consts.rs	Added UCS_KILL_SWITCH_ACTIVE constant for kill-switch configuration
crates/router/src/core/payments/helpers.rs	Implemented is_kill_switch_active helper to fetch and parse kill-switch status
crates/router/src/core/unified_connector_service.rs	Integrated kill-switch check and removed redundant client existence check at the end of rollout logic
crates/router/src/core/payments.rs	Changed internal payments path to return error when UCS kill-switch is active, removing fallback

Comments suppressed due to low confidence (2)

crates/router/src/core/payments/helpers.rs:2040

• [nitpick] Consider adding a doc comment above is_kill_switch_active to explain its purpose, default behavior on errors, and how it fits into the UCS decision flow.

pub async fn is_kill_switch_active(state: &SessionState, config_key: &str) -> RouterResult<bool> {

crates/router/src/core/payments/helpers.rs:2040

• There are no unit tests for is_kill_switch_active covering success, parse failures, and DB errors. Adding tests would help ensure correct behavior and prevent regressions.

pub async fn is_kill_switch_active(state: &SessionState, config_key: &str) -> RouterResult<bool> {

[Copilot]

The log message uses a literal {config_key:?} placeholder instead of interpolating the variable. Consider using something like "Failed to parse kill switch config for key {}" and passing config_key as an argument for clarity.

Suggested change

	logger::error!(error = ?err, "Failed to parse {config_key:?} kill switch config");
	Ok(false)
	}
	},
	Err(err) => {
	logger::error!(error = ?err, "Failed to fetch {config_key:?} kill switch config from DB");
	logger::error!(error = ?err, config_key = config_key, "Failed to parse kill switch config for key {}", config_key);
	Ok(false)
	}
	},
	Err(err) => {
	logger::error!(error = ?err, config_key = config_key, "Failed to fetch kill switch config for key {} from DB", config_key);

[Copilot]

This log also uses {config_key:?} literally. Change to a formatted message, e.g., "Failed to fetch kill switch config for key {} from DB", passing config_key as an argument.

Suggested change

	logger::error!(error = ?err, "Failed to parse {config_key:?} kill switch config");
	Ok(false)
	}
	},
	Err(err) => {
	logger::error!(error = ?err, "Failed to fetch {config_key:?} kill switch config from DB");
	logger::error!(error = ?err, "Failed to parse kill switch config for key {}", config_key);
	Ok(false)
	}
	},
	Err(err) => {
	logger::error!(error = ?err, "Failed to fetch kill switch config for key {} from DB", config_key);

[Copilot]

Chaining attach_printable directly on return Err(...) may not wrap the error as intended. Consider applying attach_printable before converting or returning the error, for example:

let err = ApiErrorResponse::InternalServerError.attach_printable("...");
return Err(err);