lfx-v2-query-service

HTTP service for LFX API consumers to perform access-controlled queries for LFX resources, including typeahead and full-text search.

Architecture Overview

The implementation follows the clean architecture principles where:

• Domain Layer: Contains business logic and interfaces
• Service Layer: Orchestrates business operations
• Infrastructure Layer: Implements external dependencies
• Presentation Layer: Handles HTTP/API concerns (generated by Goa)

Directory Structure

├── .github/                        # Github files
│   └── workflows/                  # Github Action workflow files
├── charts/                         # Helm charts
├── design/                         # GOA design specification files
├── gen/                            # GOA generated code (HTTP server, client, OpenAPI)
├── cmd/                            # Services (main packages)
│   └── service/                    # Service implementation
├── internal/                       # Internal service packages
│   ├── domain/                     # Domain logic layer
│   │   ├── model/                  # Domain models and entities
│   │   └── port/                   # Domain interfaces/ports
│   ├── service/                    # Business logic/use cases layer
│   ├── infrastructure/             # Infrastructure layer
│   └── middleware/                 # HTTP middleware components
└── pkg/                            # Shared packages for internal and external services

Key Components

Domain Layer (internal/domain/)

• ResourceService: Contains business logic and validation

Model (internal/domain/model/)

• Domain Models: Core business entities and data structures
• Value Objects: Immutable objects that represent domain concepts

Ports (internal/domain/port/)

• ResourceSearcher Interface: Defines the contract for resource search operations
• OrganizationSearcher Interface: Defines the contract for organization search operations
• AccessControlChecker Interface: Defines the contract for access control operations
• Authenticator Interface: Defines the contract for authentication operations

Service Layer (internal/service/)

• Business Logic: Application-specific business rules and operations
• Service Orchestration: Coordinates between domain models and infrastructure

Infrastructure Layer (internal/infrastructure/)

Authentication Implementation

The authentication system provides JWT-based authentication with support for Heimdall tokens:

• JWT Authentication (internal/infrastructure/auth/jwt.go):

  • Validates JWT tokens using JWKS (JSON Web Key Set)
  • Extracts custom claims including principal and email
  • Supports PS256 signature algorithm (default for Heimdall)
  • Configurable JWKS URL and audience

• Mock Authentication (internal/infrastructure/mock/auth.go):

  • Uses environment variable for mock principal
  • Bypasses JWT validation for local development

Authentication Configuration:

• AUTH_SOURCE: Choose between "mock" or "jwt" (default: "jwt")
• JWKS_URL: JSON Web Key Set endpoint URL
• JWT_AUDIENCE: Intended audience for JWT tokens
• JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL: Mock principal for development

OpenSearch Implementation

The OpenSearch implementation includes query templates, a searcher, and a client for interacting with the OpenSearch cluster.

NATS Implementation

The NATS implementation consists of a client, access control logic, and request/response models for messaging and access control.

Clearbit Implementation

The Clearbit implementation provides organization search capabilities using the Clearbit Company API. It includes a client for API communication, searcher for organization queries, and configuration management for API credentials and settings.

Dependency Injection

Dependency injection is performed in cmd/main.go, where the concrete implementations for resource search and access control are selected based on configuration and then injected into the service constructor.

Benefits of This Architecture

1. Testability: Easy to swap implementations for testing
2. Flexibility: Can easily switch between different search backends and access control systems
3. Maintainability: Clear separation of concerns
4. Scalability: Easy to add new search and access control implementations
5. Independence: Layers don't depend on external frameworks

Docker

Building the Docker Image

Example: Build the Docker image with Make

make docker-build

Running with Docker

Basic Docker Run

make docker-run

Usage

Running Locally

With Mock Implementation (Default for Development)

# Using mock implementations
SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock go run cmd/main.go

# With custom port
SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock go run cmd/main.go -p 3000

With Production Services

# production-like setup
SEARCH_SOURCE=opensearch \
ORG_SEARCH_SOURCE=clearbit \
ACCESS_CONTROL_SOURCE=nats \
OPENSEARCH_URL={{placeholder}} \
OPENSEARCH_INDEX=resources \
NATS_URL{{placeholder}} \
CLEARBIT_CREDENTIAL=your_clearbit_api_key \
CLEARBIT_BASE_URL=https://company.clearbit.com \
CLEARBIT_TIMEOUT=30s \
CLEARBIT_MAX_RETRIES=5 \
CLEARBIT_RETRY_DELAY=2s \
go run cmd/main.go

Available Environment Variables

Search Implementation:

• SEARCH_SOURCE: Choose between "mock" or "opensearch" (default: "opensearch")

Organization Search Implementation:

• ORG_SEARCH_SOURCE: Choose between "mock" or "clearbit" (default: "clearbit")

OpenSearch Configuration:

• OPENSEARCH_URL: OpenSearch URL (default: http://localhost:9200)
• OPENSEARCH_INDEX: OpenSearch index name (default: "resources")

Access Control Implementation:

• ACCESS_CONTROL_SOURCE: Choose between "mock" or "nats" (default: "nats")

NATS Configuration:

• NATS_URL: NATS server URL (default: nats://localhost:4222)
• NATS_TIMEOUT: Request timeout duration (default: "10s")
• NATS_MAX_RECONNECT: Maximum reconnection attempts (default: "3")
• NATS_RECONNECT_WAIT: Time between reconnection attempts (default: "2s")

Clearbit Configuration:

• CLEARBIT_CREDENTIAL: Clearbit API key (required for organization search)
• CLEARBIT_BASE_URL: Clearbit API base URL (default: https://company.clearbit.com)
• CLEARBIT_TIMEOUT: HTTP client timeout for API requests (default: "10s")
• CLEARBIT_MAX_RETRIES: Maximum number of retry attempts for failed requests (default: "3")
• CLEARBIT_RETRY_DELAY: Delay between retry attempts (default: "1s")

Authentication Configuration:

• AUTH_SOURCE: Choose between "mock" or "jwt"
• JWKS_URL: JSON Web Key Set endpoint URL
• JWT_AUDIENCE: Intended audience for JWT tokens
• JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL: Mock principal for development (required when AUTH_SOURCE=mock)

Server Configuration:

• -p: HTTP port (default: "8080")
• -bind: Interface to bind on (default: "*")
• -d: Enable debug logging

API Usage

The service exposes a RESTful API through the Goa framework with JWT authentication:

Resource Search API

GET /query/resources?name=committee&type=committee&v=1
Authorization: Bearer <jwt_token>

Parameters:

• name: Resource name or alias (supports typeahead search)
• type: Resource type to filter by
• parent: Parent resource for hierarchical queries
• tags: Array of tags to filter by
• sort: Sort order (name_asc, name_desc, updated_asc, updated_desc)
• page_token: Pagination token
• v: API version (required)

Response:

{
  "resources": [
    {
      "type": "committee",
      "id": "123",
      "data": {
        "name": "Technical Advisory Committee",
        "description": "Main technical governance body",
        "status": "active"
      }
    }
  ],
  "page_token": "offset_50",
  "cache_control": "public, max-age=300"
}

Organization Search API

Query Organizations:

GET /query/orgs?name=Linux Foundation&domain=linuxfoundation.org&v=1
Authorization: Bearer <jwt_token>

Parameters:

• name: Organization name (optional)
• domain: Organization domain or website URL (optional)
• v: API version (required)

Response:

{
  "name": "Linux Foundation",
  "domain": "linuxfoundation.org",
  "industry": "Non-Profit",
  "sector": "Technology",
  "employees": "100-499"
}

Organization Suggestions API:

GET /query/orgs/suggest?query=linux&v=1
Authorization: Bearer <jwt_token>

Parameters:

• query: Search query for organization suggestions (required, minimum 1 character)
• v: API version (required)

Response:

{
  "suggestions": [
    {
      "name": "Linux Foundation",
      "domain": "linuxfoundation.org",
      "logo": "https://example.com/logo.png"
    }
  ]
}

Clearbit API Integration

The service integrates with Clearbit's Company API to provide enriched organization data for search operations. This integration allows the service to fetch detailed company information including industry classification, employee count, and domain information.

Clearbit API Setup

1. Obtain API Credentials

To use Clearbit integration, you need to obtain an API key from Clearbit:

1. Sign up for a Clearbit account at https://clearbit.com
2. Navigate to your API settings to generate an API key
3. Copy the API key for use in your environment configuration

2. Configure Environment Variables

Set the required environment variables for Clearbit integration:

# Required: Clearbit API key
export CLEARBIT_CREDENTIAL=your_clearbit_api_key_here

# Optional: Custom configuration (defaults shown)
export CLEARBIT_BASE_URL=https://company.clearbit.com
export CLEARBIT_TIMEOUT=30s
export CLEARBIT_MAX_RETRIES=3
export CLEARBIT_RETRY_DELAY=1s

# Set organization search source to use Clearbit
export ORG_SEARCH_SOURCE=clearbit

3. API Usage and Features

The Clearbit integration supports the following search operations:

Search by Company Name:

• Searches for companies using their registered business name
• Falls back to domain-based search for additional data enrichment

Search by Domain:

• More accurate search method using company domain names
• Provides comprehensive company information

4. Error Handling

The Clearbit integration includes robust error handling:

• 404 Not Found: Returns appropriate "organization not found" errors
• Rate Limiting: Automatic retry with exponential backoff
• Network Issues: Configurable retry attempts with delays
• API Errors: Proper error propagation with context

Testing

The clean architecture makes testing straightforward:

// Use mock implementations for unit tests
searcher := mock.NewMockResourceSearcher()
searcher.AddResource(testResource)

accessChecker := mock.NewMockAccessControlChecker()
accessChecker.SetAccessResult(testResult)

service := service.NewResourceService(searcher, accessChecker)
result, err := service.QueryResources(ctx, criteria)

Extending the Architecture

To add a new search implementation:

1. Create a new package in internal/infrastructure/
2. Implement the domain.ResourceSearcher interface
3. Add configuration options to main.go
4. Update the dependency injection switch statement

To add a new access control implementation:

1. Create a new package in internal/infrastructure/
2. Implement the domain.AccessControlChecker interface
3. Add configuration options to main.go
4. Update the dependency injection switch statement

Development

Prerequisites

This project uses the GOA Framework for API generation. You'll need to install GOA before building the project.

Installing GOA Framework

Follow the GOA installation guide to install GOA:

go install goa.design/goa/v3/cmd/goa@latest

Verify the installation:

goa version

Building and Development

1. Generate Code

The project uses GOA to generate API code from the design specification. Run the following command to generate all necessary code:

goa gen github.com/linuxfoundation/lfx-v2-query-service/design

This command generates:

• HTTP server and client code
• OpenAPI specification
• Service interfaces and types
• Transport layer implementations

2. Initial Project Structure

Note: The initial cmd structure was generated using GOA's example generator:

goa example github.com/linuxfoundation/lfx-v2-query-service/design

This command generated the basic server structure, which was then customized and adjusted to fit our project's clean architecture principles.

3. Development Workflow

1. Make design changes: Edit files in the design/ directory

2. Regenerate code: Run goa gen github.com/linuxfoundation/lfx-v2-query-service/design after design changes

3. Build the project:

   go build cmd

4. Run with mock data (for development):

   SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock go run ./cmd

5. Run tests:

   make test
   
   # or run go test to set custom flags
   go test ./... -v

6. Lint the code

   # From the root of the directory, run megalinter (https://megalinter.io/latest/mega-linter-runner/) to ensure the code passes the linter checks. The CI/CD has a check that uses megalinter.
   npx mega-linter-runner .

7. Docker build + K8

   # Build the dockerfile (from the root of the repo)
   docker build -t lfx-v2-query-service:<release_number> .
   
   # Install the helm chart for the service into the lfx namespace (from the root of the repo)
   helm install lfx-v2-query-service ./charts/lfx-v2-query-service/ -n lfx
   
   # Once you have already installed the helm chart and need to just update it, use the following command (from the root of the repo):
   helm upgrade lfx-v2-query-service ./charts/lfx-v2-query-service/ -n lfx

Contributing

To contribute to this repository:

1. Fork the repository
2. Make your changes
3. Submit a pull request

License

Copyright The Linux Foundation and each contributor to LFX.

This project’s source code is licensed under the MIT License. A copy of the license is available in LICENSE.

This project’s documentation is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0). A copy of the license is available in LICENSE-docs.