Terraform has become the lingua franca of DevOps. As Terraform’s popularity grew, the surrounding ecosystem matured to help solve common challenges. Two great examples are Terratest which provides a modular testing framework for your IaC, and Terraformer which helps you quickly get existing infrastructure managed by Terraform. In the project’s own words, Terraformer is reverse terraform. It provides a CLI tool to generate terraform files from existing infrastructure. Let’s see it in action…
While you can install from source, installation is usually as easy as reaching for a cold brew…
❯ brew install terraformer
Since Homebrew works on Linux these days, this likely works for most readers… If you do prefer a manual install, one advantage is being able to select providers in the process:
export PROVIDER=aws
curl -LO https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-linux-amd64
chmod +x terraformer-${PROVIDER}-linux-amd64
sudo mv terraformer-${PROVIDER}-linux-amd64 /usr/local/bin/terraformer
Whatever the approach, you should see the help message once successful:
❯ terraformer
Usage:
[command]
Available Commands:
help Help about any command
import Import current state to Terraform configuration
plan Plan to import current state to Terraform configuration
version Print the version number of Terraformer
Flags:
-h, --help help for this command
-v, --version version for this command
Use " [command] --help" for more information about a command.
Terraformer supports a number of Terraform’s providers – but not all. Out of the box, you get support for every major IaaS and common things you may have heard of like GitHub and Kubernetes. The nice thing is this is all open source, so if your favorite provider isn’t supported you have the power to contribute!
For a simple example, we’ll use the Datadog provider to import a dashboard and synthetic test. In my professional life, we like to view monitoring as IaC just like the infrastructure itself. However, monitoring often evolves organically. Some times it’s just easier to prove something out in a UI. Either way, it’s nice to be able to import your work of art rather than starting with a clean slate. Terraformer to the rescue…
Let’s create a work directory to experiment, and import a resource… Similar to
Terraform, Terraformer has a plan step. That will spit out a plan.json you can
review before doing the actual import. Files are created in
generated/${providerName} directories:
❯ mkdir ~/datadog; cd ~/datadog
❯ terraformer plan datadog --resources=synthetics --filter=datadog_synthetics_test=abc-123-xyz --api-key=${DD_API_KEY} --app-key=${DD_APP_KEY}
2020/08/12 11:09:44 datadog importing... synthetics
2020/08/12 11:09:44 open /Users/mhoskins/.terraform.d/plugins/darwin_amd64: no such file or directory
Wait a minute, an error message? It’s not very helpful either, but at least it mentions plugins… Let’s see if configuring the Datadog provider stops harshing our buzz:
❯ echo 'provider "datadog" {}' > main.tf
❯ terraform init
...
* provider.datadog: version = "~> 2.12"
Terraform has been successfully initialized!
Now that we’ve wired up the provider, try our plan again:
❯ terraformer plan datadog --resources=synthetics --filter=datadog_synthetics_test=abc-123-xyz --api-key=${DD_API_KEY} --app-key=${DD_APP_KEY}
2020/08/13 12:13:15 datadog importing... synthetics
2020/08/13 12:13:17 Refreshing state... datadog_synthetics_test.tfer--synthetics_abc-002D-123-002D-xyz
2020/08/13 12:13:18 Saving planfile to generated/datadog/terraformer/plan.json
❯ terraformer import plan generated/datadog/terraformer/plan.json 2020/08/13 12:14:34 datadog Connecting....
2020/08/13 12:14:34 datadog save synthetics
2020/08/13 12:14:34 datadog save tfstate for synthetics
That went a lot better! We’re able to plan and import a resource. We could also import several resources at once… It won’t make a mess, but simply pull in any changes or the same resources idemotently:
❯ terraformer plan datadog --resources=dashboard,synthetics --filter=datadog_dashboard=xyz-efg-cb9 --filter=datadog_synthetics_test=abc-123-xyz --api-key=${DD_API_KEY} --app-key=${DD_APP_KEY}
2020/08/13 15:05:49 datadog importing... dashboard
2020/08/13 15:05:51 Refreshing state... datadog_dashboard.tfer--dashboard_xyz-002D-efg-002D-cb9
2020/08/13 15:05:51 datadog importing... synthetics
2020/08/13 15:05:52 Refreshing state... datadog_synthetics_test.tfer--synthetics_abc-002D-123-002D-xyz
2020/08/13 15:05:52 Saving planfile to generated/datadog/terraformer/plan.json
Now we’ve got our dashboard and synthetic resources in generated/datadog. This is a great starting point, but before committing you’ll usually want to organize a bit and do some cleanup to integrate with your project:
❯ cp generated/datadog/dashboard/dashboard.tf .
❯ cp generated/datadog/synthetics/synthetics_test.tf .
❯ chmod 644 *.tf
❯ cat main.tf
terraform {
required_version = "~> 0.12.0"
required_providers {
datadog = "~> 2.12"
}
}
# Reads DD_API_KEY and DD_APP_KEY from environment
provider "datadog" {}
❯ git add *.tf
❯ git commit -am 'initial commit'
Does it work as expected?
❯ terraform plan
...
Plan: 2 to add, 0 to change, 0 to destroy.
❯ terraform apply
...
datadog_synthetics_test.tfer--abc-002D-123-002D-xyz: Creating...
datadog_dashboard.tfer--dashboard_xyz-002D-efg-002D-cb9: Creating...
datadog_synthetics_test.tfer--synthetics_abc-002D-123-002D-xyz: Creation complete after 0s [id=aaa-bbb-ccc]
datadog_dashboard.tfer--dashboard_xyz-002D-efg-002D-cb9: Creation complete after 1s [id=ddd-eee-fff]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
“Legacy” services which don’t fit nicely into the latest toolchain du’jour are a common problem in larger infrastructures. Even in smaller, hipper, more agile shops you often have organic growth or PoCs-turned-production that you’d like to whip into shape. Terraformer can help wrangle existing infrastructure, quickly importing resources from supported providers. While the provider list is not as extensive as Terraform’s out of the box, common IaaS and DevOps tools are covered.
Since it’s an open source project, you have complete visiblity into what you are running and can help the community by contributing code. Terraformer can easily import individual resources, or an entire service at once – perhaps you want to write automation to generate the import commands? Importing is low-stress, with plans that can be reviewed and idempotency which allows building up plans slowly without stomping on existing resources.