Firefly Azure Integration ARM Templates

🚀 Overview

Firefly Azure Integration provides enterprise-grade ARM templates for seamlessly connecting your Azure subscriptions with Firefly's cloud asset management and FinOps platform. These templates automate the complete setup process, from service principal creation to advanced monitoring infrastructure deployment.

🎯 What is Firefly?

Firefly is a comprehensive cloud asset management platform that helps organizations:

Discover & Inventory all cloud resources across multiple providers
Optimize Costs with intelligent recommendations and automated policies
Ensure Compliance with security and governance standards
Manage Infrastructure as Code with drift detection and remediation
Monitor Changes in real-time with event-driven architecture

✨ Key Features & Capabilities

🔧 Flexible Deployment Options

📊 Single Subscription Deployment

Perfect for small to medium organizations
Quick setup with minimal configuration
Ideal for testing and proof-of-concept scenarios

🏢 Multi-Subscription Deployment

Enterprise-scale deployment across multiple subscriptions
Centralized management with distributed monitoring
Bulk onboarding with parallel processing

🌐 Management Group Deployment (Available separately)

Organization-wide deployment across entire management group hierarchies
Automatic discovery of new subscriptions
Enterprise governance and compliance at scale

🛡️ Advanced Security & Permissions

Built-in Azure RBAC Roles

Reader: Read-only access to Azure resources
Billing Reader: Access to cost and billing information
App Configuration Data Reader: Configuration data access
Security Reader: Security recommendations and alerts
Storage Blob Data Reader: Conditional access to storage blobs

Custom Role Definitions

Firefly Custom Role: Specialized permissions for:
- Storage account key access
- Database connection strings
- Kubernetes cluster credentials
- Web app configurations
- Redis cache keys
- Search service keys
- Log Analytics workspace keys

Conditional Access Policies

Terraform state file access (*state, *.tfstateenv:*)
Restricted blob access with intelligent filtering
Network-level security controls

🔄 Event-Driven Monitoring (Optional)

Real-Time Infrastructure Tracking

Azure Event Grid Integration: Instant notifications for resource changes
Storage Account Monitoring: Centralized log collection and analysis
Diagnostic Settings: Automatic configuration across all subscriptions
Webhook Integration: Direct integration with Firefly's event processing pipeline

Monitoring Infrastructure

Dedicated storage accounts per subscription
Event Grid system topics with custom filtering
Automated diagnostic settings deployment
Configurable retention and delivery policies

🏷️ Resource Management & Tagging

Intelligent Tagging System

Custom tag support through editable grid interface
Automatic firefly: true tag application
Tag inheritance across all created resources
Compliance and cost allocation support

Resource Organization

Dedicated resource groups per subscription (firefly-monitoring-{subscriptionId})
Unique naming conventions to prevent conflicts
Centralized resource lifecycle management

🌐 Network Security & Access Control

Storage Network Rules (Optional)

Restrict access to predefined Firefly IP addresses
Enhanced security for sensitive environments
Configurable IP allowlists
TLS 1.2+ enforcement

Firefly IP Addresses (Pre-configured)

3.224.145.192
54.83.245.177
3.213.167.195
54.146.252.237
34.226.97.113

🔧 Advanced Configuration Options

Environment-Specific Settings

Production/Non-Production environment flags
Auto-Discovery capabilities for new resources
Infrastructure as Code detection and monitoring
Custom directory domain configuration

Integration Naming

Automatic Subscription Name Detection: Uses actual Azure subscription display names
Intelligent Fallback: Uses subscription ID if name unavailable
Conflict Resolution: Automatic handling of duplicate names

📋 Prerequisites

Azure Requirements

Permissions: Contributor or Owner role on target subscription(s)
Azure AD Rights: Ability to create service principals and assign roles
Subscription Access: Valid Azure subscription(s) to monitor

Firefly Requirements

Active Firefly Account: Sign up here
API Credentials: Access Key and Secret Key from Firefly dashboard
Webhook Access: Firefly webhook endpoint accessibility

🚀 Quick Start

Option 1: Single/Multi-Subscription Deployment

Option 2: Management Group Deployment

⚠️ Offboarding Template: Completely removes all Azure resources created by Firefly onboarding.

📝 Step-by-Step Deployment Guide

Step 1: Service Principal Setup

The deployment wizard will guide you through creating a service principal:

Click "Create new" in the Service Principal section
Name your application (e.g., "Firefly-Integration")
Select account type (single or multi-tenant based on your needs)
Click "Register" to create the service principal
Create a client secret:
- Click "+ New Client Secret"
- Set expiration (recommend 24 months)
- Copy the secret value immediately (it won't be shown again)
Return to deployment and paste the client secret

Step 2: Configuration Parameters

Integration Configuration

✅ Enable Event-Driven Monitoring: Real-time resource change tracking
✅ Production Environment: Mark as production for enhanced monitoring
✅ Multi-Subscription Deployment: Monitor multiple subscriptions simultaneously
✅ Enforce Storage Network Rules: Restrict access to Firefly IPs only

Firefly Credentials

Access Key: Your Firefly API access key
Secret Key: Your Firefly API secret key

Advanced Options

Custom Tags: Add organizational tags to all resources
Target Subscriptions: Select additional subscriptions to monitor
Directory Domain: Your organization's domain (defaults to firefly.ai)

Step 3: Review & Deploy

Review all settings in the final step
Accept terms and conditions
Click "Create" to start deployment
Monitor progress in the Azure portal (typically 5-10 minutes)

Step 4: Post-Deployment Verification

Check Firefly Dashboard: Verify new integration appears
Validate Resource Discovery: Confirm Azure resources are being discovered
Test Event Monitoring: Make a test change to verify real-time monitoring
Review Permissions: Ensure service principal has correct role assignments

🏗️ Architecture Overview

Single Subscription Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Azure Subscription                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────┐    ┌─────────────────────────────────────┐ │
│  │ Service         │    │ RBAC Roles                          │ │
│  │ Principal       │───▶│ • Reader                            │ │
│  │                 │    │ • Billing Reader                    │ │
│  └─────────────────┘    │ • Security Reader                   │ │
│                         │ • App Configuration Data Reader     │ │
│                         │ • Custom Firefly Role              │ │
│                         │ • Storage Blob Data Reader         │ │
│                         └─────────────────────────────────────┘ │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │ Monitoring Infrastructure (Optional)                        │ │
│  │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │
│  │ │ Storage     │ │ Event Grid  │ │ Diagnostic Settings     │ │ │
│  │ │ Account     │ │ Topic       │ │ (Activity Logs)         │ │ │
│  │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │
│  └─────────────────────────────────────────────────────────────┘ │
│                                │                                │
└────────────────────────────────┼────────────────────────────────┘
                                 │
                                 ▼
                    ┌─────────────────────────┐
                    │ Firefly Platform        │
                    │ • Resource Discovery    │
                    │ • Cost Optimization     │
                    │ • Security Monitoring   │
                    │ • Compliance Tracking   │
                    └─────────────────────────┘

Multi-Subscription Architecture

┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│ Subscription 1  │  │ Subscription 2  │  │ Subscription N  │
│                 │  │                 │  │                 │
│ ┌─────────────┐ │  │ ┌─────────────┐ │  │ ┌─────────────┐ │
│ │Monitoring   │ │  │ │Monitoring   │ │  │ │Monitoring   │ │
│ │Infrastructure│ │  │ │Infrastructure│ │  │ │Infrastructure│ │
│ └─────────────┘ │  │ └─────────────┘ │  │ └─────────────┘ │
└─────────┬───────┘  └─────────┬───────┘  └─────────┬───────┘
          │                    │                    │
          └────────────────────┼────────────────────┘
                               │
                               ▼
                  ┌─────────────────────────┐
                  │ Shared Service Principal│
                  │ • Cross-subscription    │
                  │   RBAC assignments      │
                  │ • Centralized auth      │
                  └─────────────────────────┘
                               │
                               ▼
                  ┌─────────────────────────┐
                  │ Firefly Platform        │
                  │ ┌─────────────────────┐ │
                  │ │ Unified Dashboard   │ │
                  │ │ • All subscriptions │ │
                  │ │ • Cost analytics    │ │
                  │ │ • Security posture  │ │
                  │ │ • Compliance view   │ │
                  │ └─────────────────────┘ │
                  └─────────────────────────┘

🔧 Advanced Configuration

Custom Role Permissions

The template creates custom roles with specific permissions:

{
  "actions": [
    "Microsoft.Storage/storageAccounts/listkeys/action",
    "Microsoft.DocumentDB/databaseAccounts/listConnectionStrings/action",
    "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
    "Microsoft.Web/sites/config/list/Action",
    "Microsoft.Cache/redis/listKeys/action",
    "Microsoft.AppConfiguration/configurationStores/ListKeys/action",
    "Microsoft.Search/searchServices/listQueryKeys/action",
    "Microsoft.OperationalInsights/workspaces/sharedkeys/action"
  ]
}

Event Grid Configuration

Event subscriptions are configured with:

Event Types: Microsoft.Storage.BlobCreated
Delivery: Webhook to Firefly endpoint
Retry Policy: 30 attempts over 24 hours
Batch Size: Single events for real-time processing

Storage Account Security

When network rules are enforced:

Default Action: Deny all traffic
Allowed IPs: Only Firefly IP addresses
TLS Version: Minimum TLS 1.2
Public Access: Controlled blob access only

🐛 Troubleshooting

Common Issues & Solutions

Service Principal Creation Failed

# Check Azure AD permissions
az ad sp list --display-name "Firefly-Integration"

# Verify you have Application Administrator role
az role assignment list --assignee <your-user-id> --all

Role Assignment Failed

# Check subscription permissions
az role assignment list --scope "/subscriptions/<subscription-id>"

# Verify Owner or User Access Administrator role
az role definition list --name "Owner"

Integration Creation Failed

✅ Verify Firefly credentials are correct
✅ Check network connectivity to https://prodapi.firefly.ai
✅ Ensure service principal has required permissions
✅ Validate subscription ID format

Event Monitoring Not Working

# Check diagnostic settings
az monitor diagnostic-settings subscription list

# Verify storage account access
az storage account show --name <storage-account-name>

# Test event grid subscription
az eventgrid system-topic event-subscription list --system-topic-name <topic-name>

Validation Commands

# PowerShell validation script
$subscriptionId = "<your-subscription-id>"
$spObjectId = "<service-principal-object-id>"

# Check service principal
Get-AzADServicePrincipal -ObjectId $spObjectId

# List role assignments
Get-AzRoleAssignment -ObjectId $spObjectId -Scope "/subscriptions/$subscriptionId"

# Check storage account (if event-driven enabled)
Get-AzStorageAccount -ResourceGroupName "firefly-monitoring-$subscriptionId"

# Verify diagnostic settings
Get-AzDiagnosticSetting -ResourceId "/subscriptions/$subscriptionId"

📊 Monitoring & Observability

Deployment Monitoring

Azure Portal: Real-time deployment progress
Activity Log: Detailed operation logs
Resource Health: Post-deployment validation

Integration Health

Firefly Dashboard: Integration status and metrics
Resource Discovery: Automated inventory updates
Cost Analytics: Billing data synchronization
Security Posture: Compliance and security insights

Event Monitoring (When Enabled)

Real-time Events: Resource creation, modification, deletion
Storage Metrics: Log ingestion and processing rates
Webhook Delivery: Success/failure rates and retry statistics
Diagnostic Logs: Administrative activity tracking

🔐 Security Best Practices

Service Principal Management

🔄 Rotate secrets regularly (every 12-24 months)
🔒 Use certificate authentication when possible
📝 Document service principal usage and ownership
🚫 Avoid sharing credentials across environments

Access Control

👥 Limit service principal editors to security team
📋 Regular access reviews of role assignments
🔍 Monitor sign-in logs for unusual activity
🚨 Set up alerts for permission changes

Network Security

🌐 Enable storage network rules for sensitive environments
🔐 Use private endpoints where applicable
📊 Monitor network access patterns
🛡️ Implement conditional access policies

Compliance & Governance

📋 Tag all resources for cost allocation
📝 Document integration purpose and data flows
🔍 Regular compliance audits of permissions
📊 Monitor resource usage and costs

🤝 Support & Community

Getting Help

📧 Email Support: [email protected]
🐛 GitHub Issues: Report bugs and feature requests
📚 Documentation: Firefly Knowledge Base
💬 Community: Firefly Community Forum

Contributing

We welcome contributions! Please see our Contributing Guidelines for details on:

🐛 Bug reports and feature requests
🔧 Code contributions and improvements
📝 Documentation updates
🧪 Testing and validation

📚 Additional Resources

Firefly Platform

Azure Resources

Integration Examples

Template Components

📋 azurefireflydeploy.json - Main onboarding template
📋 azurefireflydeploy-managementgroups.json - Management group deployment
🗑️ azurefirelfyoffboard.json - Offboarding template
🎨 CreateUIDefinition.json - UI for single/multi-subscription onboarding
🎨 CreateUIDefinition-managementgroups.json - UI for management group deployment
🎨 CreateUIDefinition-offboard.json - UI for offboarding

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🏆 Acknowledgments

💙 Microsoft Azure team for ARM template platform
🚀 Firefly Engineering team for platform development
🌟 Community Contributors for feedback and improvements
🔧 DevOps Community for best practices and patterns

Made with ❤️ by the Firefly Team
Empowering cloud excellence through intelligent automation

Name		Name	Last commit message	Last commit date
Latest commit History 256 Commits
.github		.github
.gitignore		.gitignore
CreateUIDefinition-managementgroups.json		CreateUIDefinition-managementgroups.json
CreateUIDefinition.json		CreateUIDefinition.json
README.md		README.md
azurefireflydeploy-managementgroups.json		azurefireflydeploy-managementgroups.json
azurefireflydeploy.json		azurefireflydeploy.json
firefly.gif		firefly.gif

gofireflyio/arm-firefly-azure-onboarding

Folders and files

Latest commit

History

Repository files navigation