Feature Request: Splat for all resources of a type

[philomory]

Current Terraform Version

Terraform v0.12.0-alpha4 (2c36829d3265661d8edbd5014de8090ea7e2a076)
+ provider.random v2.0.0-5-g612dff2-dev

Use-cases

I'd like to be able to use resource_type.*.id to get a list of the id of every resource of a type; currently, you can only do this for resources that have a count attribute (or eventually for_each, I guess). Sometimes, you have a bunch of resources you want to declare explicitly because they don't necessarily have very much in common, but you still want to refer to them all from another resource.

My explicit use-case is the third-party auth0 provider. I've got a bunch of auth0_client resources that all have their own custom configuration. Then, I've got an auth0_connection resource which represents our actual identity provider (an LDAP server). In creating the auth0_connection resource, I need to pass a list of auth0_client ids that that connection will be used for. And I'd love to get that list of ids by simply writing auth0_client.*.client_id. But that's not possible.

Attempted Solutions

This didn't work:

resource "random_id" "foo" {       
  byte_length = 1                  
}                                  
                                   
resource "random_id" "bar" {       
  byte_length = 1                  
}                                  
                                   
output "example" {                 
  value = "${random_id.*.hex}"     
}                                  

I got this error:

$terraform validate

Error: Invalid reference

  on test.tf line 12, in output "example":
  12:   value = "${random_id.*.hex}"

A reference to a resource type must be followed by at least one attribute
access, specifying the resource name.

A similar attempt with random_id[*].hex also failed.

[Justin-W]

I have a need for the same use case as described above.

In addition, I'd like this 'resource type splat' feature to be usable in conjunction with the feature described in issue #9067 (and/or the upcoming for/for_each with optional filtering via an if qualifier) to enable me to create a single output variable whose value is a list of some or all of the resources (entire resources, not just IDs, or names) of a given type, without specifying or knowing the names or IDs of the included resources.

The need for this is based on my use of code generation tools to generate HCL dynamically. I want separate parts of my code generation to be able to generate HCL that will be combined with each other and applied as a single set of TF configuration, but also to have my separate code generation components to remain encapsulated and unaware of each other. But I'd like to have 1 component that generates output variables that collect all or some of the resources (and/or specific computed/exported attributes of them) into named output variables.

For example:

output "all_eips_list" {
  value = "${aws_eip.*}"
}

output "all_eip_public_ips_list" {
  value = "${aws_eip.*.public_ip}"
}

output "all_eip_public_to_private_ips_map" {
  # Result is a map from eip public IP to private IP address, such as:
  #  {"192.168.1.2" = "10.1.2.3", "192.168.1.5" = "10.2.3.4"}
  value = {
    for eip in aws_eip.*:
    eip.public_ip => eip.private_ip
  }
}

output "some_eip_public_to_private_ips_map" {
  # Result is a map from eip public IP to private IP address, such as:
  #  {"192.168.1.2" = "10.1.2.3", "192.168.1.5" = "10.2.3.4"}
  value = {
    for eip in aws_eip.*:
    eip.public_ip => eip.private_ip
    if eip.associate_with_private_ip_address
  }
}

Similar use cases exist for aws_lb.dns_name, aws_s3_bucket, and many other resources where the HCL configuration may contain a dynamic number of a specific type of resource (but not count-based repetition; each resource of the same type is explicitly declared separately, distinctly, and independently, as each resource of the same type has unique properties and configuration) and computed or exported attributes that aren't known until terraform apply or terraform refresh is executed.

[Justin-W]

A further (related but distinct) use case is that output variables similar to the examples above would (in many cases) be useful as 'reusable' HCL code, and could be included (without requiring any changes to the output variable HCL nor any configuration via input variables, etc.) in almost any arbitrary TF configuration that includes the referenced providers and resource types.

This would be useful to people that manage multiple TF configurations as a way of providing standardized (i.e. consistent across TF configurations, but with custom content and format determined by the output variables' reusable HCL definition) outputs.

This 'reusable outputs' use case applies equally well regardless of whether the separately managed TF configurations are manually generated or generated by code or by some other means.

[Justin-W]

Also, given that this change probably requires a change to HCL itself, it'd be great if this could be included in the initial v0.12 release. Is there still a chance of that?

[zygimantas]

One more use case:

I am creating N identical dynamodb tables in a module.
Module is called for each region (regional module).
In a root module, I need to create global dynamodb tables and I need regional table names.
It would be good, to return this output from a regional module:

output "table_names" {
  value = aws_dynamodb_table.*.name
}

[uutest74]

yes need this for output vars - different customized vms (windows and linux) created by module as different resources the same type!

[acdha]

I was recently working on getting some of the less common resources tagged and some kind of ability like this would have been really useful to avoid needing to maintain a list of resources:

locals {
  resources_tags = setproduct(
    concat(
      values(aws_efs_mount_target.a)[*].network_interface_id,
      values(aws_efs_mount_target.b)[*].network_interface_id,
      values(aws_efs_mount_target.c)[*].network_interface_id,
      values(aws_efs_mount_target.d)[*].network_interface_id,
      values(aws_efs_mount_target.e)[*].network_interface_id,
      module.vpc.vpc_endpoint_efs_network_interface_ids,
      module.vpc.private_route_table_ids,
      module.vpc.public_route_table_ids,
    ),
    [for key, value in local.tags :
      [key, value]
    ]
  )
}

resource "aws_ec2_tag" "all" {
  count       = length(local.resources_tags)
  resource_id = local.resources_tags[count.index][0]
  key         = local.resources_tags[count.index][1][0]
  value       = local.resources_tags[count.index][1][1]
}

[farfromunique]

Here we are, approaching 3 years after this was opened. Is this at all likely to happen?

[HariSekhon]

I need this to for_each for github_repositories too.

For now, I've quickly worked around it using an external program to generate the list from the terraform state for any given resource type. The terraform_resources.sh script can be found here:

https://github.com/HariSekhon/DevOps-Bash-tools

and working Terraform code using it can be found here:

https://github.com/HariSekhon/Terraform

in the github_team_repository.tf and github_branch.tf files - each of which are getting different attributes using that script.

[Charles-Stuart]

Jumping on this bandwagon, for called modules within a workspace. I need to output certain metadata values for each system built by a module, and I can't use foreach as each system needs to be able to have different providers specified for its module call.

Example: This is currently the only way I got it working. This is NOT manageable at scale at all. Each new server built requires the sys admins to also add to the output value... which people have already forgotten, causing failed builds.

module "srv-1" {
  providers = {...}
}
module "srv-2" {
  providers = {... something different ...}
}

output "all-systems" {
  value = merge(module.srv-1, module.srv-2)
}

All we really need is the ability for "outputs" to iterate over the built resources and modules in state, instead of needing to know the exact resource id ahead of time.

[markpendlebury]

Adding my use case:

Currently creating a bunch of aws_apigatewayv2_domain_name's to be later used in aws_apigatewayv2_api_mappings

Would be nice to have something similar to the following:

resource "aws_apigatewayv2_domain_name" "www"{
  domain_name = var.env == "prod" ? "www.${var.domain}.co.uk"  : "www.${var.env}.${var.domain}.co.uk"
}

resource "aws_apigatewayv2_domain_name" "top"{
  domain_name = var.env == "prod" ? "${var.domain}.co.uk"  : "${var.env}.${var.domain}.co.uk"
}

then a local similar to this:

locals {
  domains = [aws_apigatewayv2_domain_name.*.domain_name]
}

to then be referenced with:

resource "aws_apigatewayv2_api_mapping" "domain" {
    for_each = for_each(local.domains, "domain")
    ....
}

[skgsergio]

A use case in Azure: creating diagnostics settings for a bunch of resources of a type

resource "azurerm_monitor_diagnostic_setting" "keyvaults" {
  for_each = toset(azurerm_key_vault.*.id)  # <-- or something like that

  name               = "storage-export"
  target_resource_id = each.key
  storage_account_id = azurerm_storage_account.diag_logs.id

  log {
    category = "AuditEvent"
    enabled  = true

    retention_policy {
      enabled = true
      days    = 0
    }
  }

  log {
    category = "AzurePolicyEvaluationDetails"
    enabled  = true

    retention_policy {
      enabled = true
      days    = 0
    }
  }
}