By Justin Wilcoxen, REI Systems’ Solution Architect and Dustin Weber, REI Systems’ Associate Technical Architect

Federal IT systems support some of the nation’s most visible and mission critical operations, which means even a small and undocumented infrastructure change can create a security exposure, disrupt services, or trigger audit findings. Delivery teams must move quickly, support multiple environments, and control cloud costs, while ensuring that every change is transparent, authorized, and fully traceable. 

Infrastructure as Code with Terraform strengthens consistency and operational discipline, but it cannot prevent drift when updates are made directly in the cloud console. Once that happens, the integrity of the environment begins to break down, reducing reliability, repeatability, and audit readiness. 

To address this challenge, we implemented a GitOps driven workflow using Atlantis, combined with weekly drift detection and Cloud GitOps-driven workflow using Atlantis, combined with weekly drift detection and CloudTrail-based ensure that every change whether intentional or out of band is captured, reviewed, and auditable. 

The Real Challenge: Configuration Drift 

We’ve lived through what configuration drift really means. Even with Terraform in place, real-world pressures often force teams to make changes outside of code. When an environment goes down, waiting for a pull request review isn’t always an option. Engineers jump into the AWS console to fix a route table, tweak an IAM policy, or open a firewall port “just for a minute.” 

• An engineer “just this once” tweaks an AWS security group in the console. 
• An emergency hotfix happens directly in production. 
• An autoscaling feature modifies infrastructure outside of Terraform’s awareness. 

From that moment, Terraform’s state no longer perfectly reflects reality. That’s configuration drift. The risks stack up quickly, and we’ve seen every one of these play out in real environments such as at USCIS: 

• Unexpected outages: A future terraform apply may silently overwrite or delete those manual changes, causing the classic Friday 5 p.m. incident. 
• Security gaps: A manually relaxed firewall rule or public S3 bucket stays open because Terraform doesn’t know it exists. 
• Cost surprises: Orphaned resources or oversized autoscaling groups run indefinitely, inflating cloud bills. 
• Audit headaches: No clear record of who changed what or why, breaking change-management expectations and compliance requirements. 

To eliminate that threat, we needed two things: 

1. A disciplined workflow so all intended changes go through code. 
2. A safety net to detect and trace any changes that still happen out-of-band. 

GitOps for Terraform: Why We Chose Atlantis 

To enforce “code or nothing,” we adopted a GitOps workflow using Atlantis integrated with GitHub. 

Atlantis is an open-source tool that automates Terraform operations through pull requests (PRs). As a CNCF Sandbox project, it’s fully community-driven rather than a commercial product, which aligns well with our preference for transparent, extensible tooling. Instead of engineers running terraform plan and terraform apply from their laptops, Atlantis runs them centrally in response to PRs. 

How the PR Workflow Works

Why This Matters for Government IT 

For government programs, this model delivers several critical advantages: 

• Separation of duties: Engineers submit changes; designated approvers must sign off before deployment. This directly supports compliance frameworks that require change control and multi-party review. 
• Credential security: Developers no longer need cloud credentials on their laptops. Atlantis holds the necessary AWS credentials in a secure, central location, with tightly scoped permissions. 
• Consistent, repeatable execution: Terraform is always executed in the same controlled environment and workflow. No more “it worked on my machine” plans or accidental applies from the wrong directory or workspace. 
• Complete audit trail: Each change is documented: who opened the PR, what the plan showed, who approved, who triggered apply, and when. Auditors can follow the entire chain by browsing PR history without jumping between tools. 

In short, Atlantis enforces the discipline IaC promises and provides the transparency government stakeholders expect. 

Continuous Drift Detection: A Monthly “Infra Health Check” 

Even with strict policies, manual or automated out-of-band changes can still happen. Someone may tweak a resource in an emergency, or an older account might still contain manually created infrastructure. That’s why we added a recurring drift detection routine—run monthly for systems like USCIS, but adjustable based on system criticality and risk tolerance. ructure. That’s why we added a weekly drift detection routine on top of Atlantis.

Our Drift Detection Bot

We built a simple but powerful mechanism to ensure Drift Detection it’s part of our ongoing operational hygiene:

Closing the Loop with AWS CloudTrail

Detecting that something changed is only half the job. Programs also need clarity on who made the change and how it occurred. This is where integrating AWS CloudTrail could complete the picture.

CloudTrail logs every API call to an AWS account whether triggered via the console, CLI, SDK, or a service. As part of a recommended workflow, when a weekly drift report flags an unexpected change, teams could: 

• Identify the affected resource and the type of change from the plan output. 
• Query CloudTrail logs for relevant events, for example, ModifyInstanceAttribute, security group changes, S3 configuration updates around the timeframe the drift likely occurred. 
• Pinpoint the IAM user or role responsible, along with timestamp and request context. 

This approach would help answer: 

• Who made the change? 
• When did it happen? 
• Was it a human action, script, or service? 
• Was it authorized or an exception to the process? 

For government leadership, this would provide accountability in practice. Every change whether made via code (Atlantis + GitHub) or manually through the console could become fully traceable 

From Slogan to Practice: Infrastructure as Code That Sticks 

Using Terraform alone doesn’t guarantee that your infrastructure actually matches your code. The combination of Atlantis-enforced PR workflows, Automated weekly drift detection and CloudTrail-backed forensics and alerting makes IaC an everyday operational reality 

For government IT programs, this approach offers something invaluable: confidence. Confidence that what’s running in AWS matches what’s in the Terraform repository. Confidence that changes follow a documented process. And confidence that if something does slip through, you’ll know who did it, when, and how. 

That’s the foundation not just for passing audits, but for building resilient, scalable systems that agencies and citizens can rely on day after day.