Engineering Logs

A client wanted private-subnet EC2s to reach github.com/their-org but nothing else on github.com. I built a Terraform sandbox to test whether AWS Network Firewall could actually enforce it. It can, but only after three non-obvious fixes: dropping the blanket pass-tls rule that leaks every path, using flowbits to allow the response direction for the tagged org-path request, and pinning clients to HTTP/1.1 because the firewall's http.uri keyword ignores HTTP/2 (which github speaks by default). Then I tested the real git workflow: clone, pull, and push to the approved org all work (including a 184 MB clone, which stays entirely on github.com and never redirects to a CDN), while what actually breaks is narrower than expected: Git LFS, release-asset, and raw-file downloads from githubusercontent.com, plus SSH. This post walks through the architecture, each fix with the error that exposed it, the full test matrix, and why the working demo still isn't the right control for 'only our org' (the HTTP/1.1 pin is client-side, githubusercontent.com can't be org-scoped, SSH must be blocked outright). The robust answer is GitHub Enterprise Managed Users with SAML SSO.

Ornith is an open-source agentic coding model that serves an OpenAI-compatible API, and Claude Code only speaks the Anthropic API, so wiring one to the other means putting a LiteLLM translation proxy in the middle. I built the whole stack on a single EC2 instance with Terraform, first on CPU with llama.cpp and then on GPU with vLLM on an L40S. This post covers the architecture, the gotchas (Claude Code's 43K-token prompt, trust-remote-code, the FlashInfer ninja/nvcc PATH trap, Python 3.11), and a measured CPU-vs-GPU comparison where the same model's prompt processing goes from about 7 minutes to about 4 seconds. It ends with a third run on a spare laptop with an integrated GPU and no cloud at all, where the prefill gets slow enough that Claude Code's own timeout cancels the request before it ever returns a token.

A Step Functions pipeline that patches a hardened Amazon Linux 2023 golden AMI on the 1st of every month with no manual work. It launches a temp instance from the current AMI, runs the OS update via an SSM Automation document, snapshots a new AMI, re-encrypts it under a dedicated org-wide KMS key, shares it across the AWS Organization, and publishes the new ID to an SSM parameter shared via RAM so consumers auto-resolve the patched image on their next deploy. Covers why you use a separate KMS key instead of the landing zone's EBS key, the kms:CreateGrant gotcha on consumers, and full-ARN SSM resolution across accounts.

I run a small EventBridge-first SIEM for my AWS org, built around one idea: before it pages me, a second model pass on Bedrock tries to refute the first one's alert. Source accounts forward a curated slice of high-signal events to a central custom bus, one analyzer Lambda enriches and triages everything, and SES sends the mail. Five feeds hang off it: suspicious CloudTrail management calls (key creation, policy attaches, security-tool tampering, trail deletion, making a bucket public), root console sign-ins, Security Hub HIGH/CRITICAL findings, security-group-open-to-the-world findings, and Route 53 DNS Firewall threat-list hits from my domain controllers. Most feeds run analyze-then-refute; the DNS Firewall hits are deterministically bad and skip the model. Covers the two-pass design and why it beats a single verdict, a real CreateUser false positive from Identity Center sync and the one-line fix, the cheap guards (allowlist, circuit breaker, dedup) that keep it from flooding the inbox, the CloudWatch-Logs-to-EventBridge bridge that makes a new log source a one-day feed, and testing end to end with synthetic events.

Two RHEL 8 servers migrated with AWS Application Migration Service kernel-panicked on their test launches with 'Unable to mount root fs'. The boot disk's rootvg replicated as an empty LVM physical volume and the boot kernel's initramfs was 0 bytes, even though MGN reported the disk fully replicated. A sudoers error in the conversion log was a red herring (a healthy server had the same hardened config). The real cause was an incomplete initial replication of the partitioned boot disk, fixed with a full Stop/Start re-replication. The takeaway: instead of waiting 15-20 minutes for an MGN Test launch to learn whether the volumes replicated, mount the MGN staging snapshot of the boot disk on a throwaway Amazon Linux 2023 helper and check rootvg and the initramfs in two minutes.

A hands-on walkthrough of a complete AWS Elastic Disaster Recovery (DRS) loop in a lab: protect two EC2 instances in one Availability Zone, simulate the AZ failing, fail over to a second AZ, and fail back. Each instance ran a heartbeat app printing its live instance ID, AZ, and a counter so state survival was visible at a glance. Covers the gotchas that cost real time: the launch template subnet controls which AZ you recover into, the reverse replication button stays greyed until the failback agent checks in, Windows scheduled-task apps serve a stale page for a few minutes, conversion is the slow phase, failback launches a brand-new instance instead of reviving the original, and DRS never redirects traffic for you.

How I replaced a two-EC2 Flask app that coordinated AWS MGN cutover waves across 23 accounts with a fully serverless Lambda + API Gateway + S3 stack. Annual run cost dropped from ~$500-700 to ~$15-65, the laptop-driven collector script went away, and the IAM surface shrank to two purpose-built cross-account roles.

A weekly multi-account AWS security posture pipeline built with Prowler, ECS Fargate, S3, Lambda, Claude, Confluence, and SES. Every Sunday at 02:00 UTC, an EventBridge cron fires an ECS task that scans every member account in the org, drops OCSF findings to S3, and triggers a Lambda that summarizes via Claude and posts to a per-week Confluence child page plus a SES email to leadership. Covers the cross-account assume-role design, SES sandbox workaround, OCSF parsing memory cliff, and CloudWatch alarm tuning.

A client engaged me to solve cross-project visibility for their leadership team. I built and delivered a starter morning brief system: a Python script that reads STATUS.md files from each project folder, sends them to an LLM with a prompt that forces per-project H2 headings (so quieter workstreams don't get dropped), renders to HTML, and ships via the client's existing email pipeline at 6 AM. Covers the architecture, the structural prompt fix that prevents the model from editorializing, and the discipline problem of getting team members to update STATUS.md at end of session (solved by promoting the wrap-up from a passive instruction into a registered action).

Cross-account EFS mounting from a spoke VPC via Transit Gateway has three friction points: the EFS DNS name doesn't resolve outside the owner account, the resource policy enforces TLS + org-scoped IAM (so plain NFS mounts fail), and a fresh EFS root is root-squashed until you do a one-time chown. This post walks through the full workflow including the v1.36.0 vs v2.0.0 amazon-efs-utils trade-off on RHEL 9 (Rust toolchain dependency), TLS+IAM mount semantics, fstab persistence, and the temporary ClientRootAccess elevation dance for new file systems.

A complete centralized firewall inspection architecture on AWS using Transit Gateway, AWS Network Firewall, and a hub-and-spoke VPC design. All spoke traffic (egress, ingress, east-west) routes through a central inspection VPC with stateful rule evaluation. Includes a tag-based firewall bypass system using EventBridge and Lambda, a shared Endpoints VPC with Route 53 PHZs for cross-VPC DNS resolution, TGW route table isolation with blackhole routes for environment separation, and detailed cost estimates. Covers the design decisions and tradeoffs so you can adapt the architecture to your own environment.

Domain-joined Windows EC2 instances silently fail to register PTR records in Active Directory DNS reverse lookup zones. The root cause is a Windows registry path mismatch where GPO writes RegisterAdapterName to the Policies hive but the DNS client only reads the adapter-specific GUID path. The fix is a 3-line PowerShell WMI script deployed as a GPO startup script, with SSM Run Command for immediate push to existing machines. Covers reverse zone setup, zone permissions, Route 53 Resolver integration, and verification procedures.

How I cut page load times by 60% for users behind China's Great Firewall using AWS Global Accelerator, cross-account endpoints, and geo-based DNS routing. Covers the full setup including cross-account endpoint attachments (create in the resource owner's account, not the accelerator owner's), SCP region restriction fixes for the us-west-2 control plane, geo-based DNS routing to target only Asia-Pacific users, CloudWatch monitoring gotchas (use the accelerator GUID, not DNS name), pricing breakdown (~$18-25/month), and the GA vs CloudFront decision matrix.

AWS Shield Network Security Director is a new capability (public preview) that extends Shield beyond DDoS into full network security posture management. It discovers VPCs, EC2 instances, ALBs, Transit Gateways, and security services across multiple accounts via AWS Organizations, builds an interactive topology map, evaluates configurations for issues like overly permissive security groups and missing WAF protections, and provides prioritized findings with remediation guidance. Includes Amazon Q Developer integration for natural-language security queries. This post covers the setup, architecture, what it can and can't see (notably: no third-party firewall analysis), and how it complements a Palo Alto + GWLB centralized inspection architecture.

Build a hub-and-spoke security architecture using Palo Alto VM-Series firewalls, AWS Gateway Load Balancer (GWLB), and Transit Gateway to inspect all North-South and East-West traffic across multiple AWS accounts. Includes Terraform modules, centralized CIDR management, and a deep dive on cross-zone load balancing behavior with GWLB and TGW appliance mode.

Fully automate a Windows two-tier PKI deployment on AWS using Terraform for infrastructure provisioning and Ansible for the certificate workflow. Deploy a Domain Controller, Standalone Root CA, and Standalone Subordinate CA in ~22 minutes with Infrastructure as Code.

SQL Server 2022 introduced native S3 connectivity via PolyBase, letting you query CSV and Parquet files in S3 buckets using standard T-SQL. This guide walks through the full setup: installing PolyBase, configuring credentials and external data sources, creating external tables, and running federated queries that combine local SQL Server data with S3 data in a single SELECT.

AWS TEAM (Temporary Elevated Access Management) provides just-in-time privileged access with automatic revocation. This open-source solution integrates with IAM Identity Center for request-based, approval-gated, time-bound access to AWS accounts.

Replicate S3 data between AWS accounts with this step-by-step guide. Covers IAM roles, bucket policies, replication rules, KMS encryption support, and troubleshooting common issues like AccessDenied errors and failed replications.

Build a comprehensive AWS backup security strategy with Vault Lock for WORM protection, Logically Air-Gapped Vaults for cross-account isolation, GuardDuty Malware Scanning to prevent restoring infected data, and Audit Manager for continuous compliance monitoring. Learn how to implement all four layers for maximum ransomware protection.

Master EC2 bootstrapping with this comprehensive guide covering User Data scripts, Golden AMIs with Packer, S3-hosted scripts, SSM State Manager, and configuration management tools. Learn the recommended hybrid approach for production environments that combines fast boot times with flexibility and ongoing compliance.

Recover unresponsive Windows EC2 instances using AWS Serial Console and Windows SAC (Special Administration Console). This guide walks through enabling Serial Console access, connecting via SAC, booting into Safe Mode with Networking for troubleshooting, and returning to normal operation - all without detaching volumes.

Eliminate root credentials from AWS member accounts while maintaining secure break-glass access. This guide covers enabling centralized root access management from your management account, delegating administration, deleting root credentials, and recovering access for emergencies - all while following AWS security best practices.

Learn how to upgrade AWS Managed Microsoft AD from Standard to Enterprise edition using the new self-service API. This guide covers the upgrade process, key limitations, edition differences, and how to enable multi-region replication after upgrading.

Learn how to automate EBS volume lifecycle management with a Lambda-based solution that converts GP2 to GP3 volumes (20% savings), cleans up orphaned volumes, and manages snapshot retention - all deployed with Terraform and featuring a safe two-phase rollout approach.

When a Windows EC2 instance won't boot or accept RDP, AWS Systems Manager automation runbooks can fix it without manual volume swaps. This guide covers AWSSupport-ExecuteEC2Rescue for automated fixes, AWSSupport-StartEC2RescueWorkflow for custom offline registry edits, and the User Data method as an alternative - plus a hands-on test you can run yourself.

Learn how to implement multi-region disaster recovery for Amazon WorkSpaces using Multi-Region Resilience (MRR). This guide covers architecture, Route 53 DNS failover, cross-region data replication, and step-by-step implementation for achieving sub-30-minute RTO.

AWS Resource Explorer multi-account search lets you find resources across your entire AWS Organization for free. This post covers the full setup with Quick Setup and manual approaches, the index architecture, delegated administrator gotchas (removing it deletes all views), quotas (5 queries/sec, 10 views max, 1,000 result cap), eventual consistency delays up to 36 hours, and practical tips for view strategy, sharing via RAM, and when to use Resource Explorer vs. AWS Config.

Learn how to securely integrate an on-premises CMDB application with multiple AWS accounts using cross-account IAM role assumption. This guide covers the architecture, CloudFormation StackSets deployment, Python implementation, and security best practices.

Should you centralize VPC endpoints in a hub VPC or deploy them in each spoke VPC? This cost analysis breaks down the pricing for Transit Gateway, VPC endpoints, and Route 53 to help you make the right architectural decision for your AWS environment.

Should you use AWS Secrets Manager or SSM Parameter Store? This comparison covers the key differences in cost, features, rotation capabilities, and IaC support to help you choose the right service for your secrets and configuration management needs.

Use IAM Roles Anywhere to grant AWS access to on-premises workloads using your existing Windows PKI certificates. This guide covers exporting your subordinate CA cert, creating IAM roles and trust anchors, issuing certificates, and configuring the credential helper for automatic temporary credentials.

Deploy SCPs and RCPs safely using a battle-tested 6-step process. This guide covers CloudTrail auditing with Athena, EventBridge monitoring for dry-run simulation, staged rollout strategies, and troubleshooting common issues.

Integrate AppStream 2.0 with IAM Identity Center using SAML for smooth SSO access. This guide covers creating the SAML application, IAM identity provider, role configuration, and attribute mappings for a complete federation setup.

AWS Systems Manager Session Manager provides secure EC2 access without SSH keys, bastion hosts, or open inbound ports. This guide covers complete Terraform setup including VPC endpoints, IAM roles, security groups, and CLI connection methods.

AWS EC2Rescue runbooks don't support encrypted EBS volumes - not even with AWS-managed keys. This post walks through building a custom rescue workflow with AWS Step Functions that handles encrypted volumes natively, including the complete state machine definition, IAM roles with KMS permissions, CloudFormation template, and PowerShell fix scripts for common Windows issues.

Route 53 health checks monitor endpoint availability and enable automatic DNS failover. This guide covers endpoint, calculated, and CloudWatch-based health checks with a complete active-passive failover demonstration.

Deploy AWS EKS with Terraform using modules for VPC, EKS cluster, ALB controller, and demo applications. This guide covers managed node groups, AWS Load Balancer Controller, and complete Kubernetes resource definitions.

FinOps brings together business, finance, and IT to manage cloud costs effectively. This guide covers best practices including cross-functional teams, cost monitoring, optimization strategies, and building a cost-aware culture.

CloudFormation StackSets deploy resources across multiple AWS accounts and regions from a central management account. This guide shows how to deploy IAM roles organization-wide with automatic deployment to future accounts.

SSH multi-hopping lets you connect through intermediate hosts to reach a final destination. This guide covers creating SSH keys on Windows/Mac/Linux, tunnel syntax, practical examples, and troubleshooting common issues.

Implement branch protection in AWS CodeCommit using IAM policies to restrict junior developers from pushing or merging to the main/master branch. Learn to set up a secure developer workflow with pull request approvals by senior team members.

Use AWS Systems Manager Automation to upgrade Windows EC2 instances without manual intervention. This guide walks you through the complete process including prerequisites, IAM setup, and troubleshooting tips.

Migrate DynamoDB tables across AWS accounts using S3 as an intermediary. This guide covers export with export-table-to-point-in-time, cross-account S3 bucket setup, and import with table creation parameters.

Automate IAM access key hygiene by identifying and deactivating unused or stale credentials using this bash script. Improve your security posture by eliminating access keys that haven't been used in 6 months or have never been used.

Automate the release of unused AWS Elastic IP addresses across all regions using this bash script for CloudShell. Reduce unnecessary charges and maintain efficient resource utilization by identifying and releasing unattached EIPs that are costing you money.

Automate the cleanup of unused AWS security groups across all regions using this bash script for CloudShell. Reduce your security blast radius and administrative overhead by identifying and removing orphaned security groups that are no longer attached to any resources.

Master SSH key-based authentication from start to finish. Learn how to generate RSA, DSA, or ECDSA key pairs, securely transfer public keys to remote servers, and establish password-less connections for enhanced security and convenience.

Automate the cleanup of unused AWS EBS snapshots across all regions using this bash script for CloudShell. Reduce storage costs and maintain a lean cloud environment by identifying and removing snapshots older than your retention threshold.

Learn how to use EC2 Instance Connect Endpoint to securely connect to private EC2 instances without bastion hosts, public IPs, or agents. This guide covers setup, configuration, and connecting to both Linux and Windows instances through encrypted WebSocket tunnels.

Learn how to build an event-driven automation pipeline using AWS EventBridge, SNS, and Lambda with Terraform to capture EC2 lifecycle events and store them as JSON files in S3 for auditing and compliance purposes.

AWS Client VPN enables secure remote access to AWS resources using OpenVPN. This guide covers certificate creation with easy-rsa, ACM import, endpoint configuration, authorization rules, and client setup with troubleshooting tips.

S3 Object Lock prevents object deletion/modification using Governance mode (overridable) or Compliance mode (immutable). This guide covers implementation, legal holds, monitoring with CloudTrail and S3 Inventory, and common pitfalls to avoid.

Mount S3 buckets as local drive letters on Windows using S3 Drive. This enables familiar file operations directly in Windows Explorer, PowerShell scripting, and easier backups without constant CLI uploads/downloads.

Migrate SQL Server to Amazon RDS with near-zero downtime using native backup/restore plus AWS DMS CDC replication. This approach preserves stored procedures, views, and functions that a DMS full-load alone would miss, and keeps data synced through the cutover window.

Install and configure Apache web server on Windows with HTTPS/SSL support. Learn to download Apache, install it as a Windows service, extract SSL certificates from PFX files using OpenSSL, and configure secure HTTPS connections.

Securely share S3 buckets across AWS accounts using IAM roles and bucket policies. Learn the step-by-step CLI commands to set up cross-account access while following security best practices and the principle of least privilege.

Install AWS MGN replication agents on Windows and Linux source servers, configure global launch templates for consistent EC2 settings, and set up post-launch automation using Systems Manager or User Data scripts for smooth cloud migration.

Create a Site-to-Site VPN between AWS and your on-premise network using only Windows Server as your router. This guide covers Terraform code for AWS VPN resources and PowerShell commands to configure Windows Server RRAS for IKEv2 VPN connectivity.

Share AWS resources like Transit Gateways across accounts easily with AWS Resource Access Manager (RAM). Learn to enable RAM trusted access in Organizations, create resource shares, and simplify multi-account network architecture without complex permission policies.

Set up comprehensive AWS audit logging with CloudTrail, S3, and CloudWatch Logs integration using a single CloudShell script. Learn to create all required IAM roles, policies, and resources for operational auditing and compliance monitoring.

Install the AWS CLI version 2 on Linux and Windows with step-by-step commands. Learn the quick installation methods using curl and msiexec, configure credentials, and start automating AWS tasks from your command line immediately.

Discover all private and public IP addresses across your entire AWS environment using Python and Boto3. This script queries ENIs across all regions, handles permission errors gracefully, and exports results to CSV for easy IP address inventory management.

Configure Terraform remote state storage using AWS S3 with DynamoDB locking for safe team collaboration. Learn to create versioned S3 buckets, set up state locking with DynamoDB, and migrate your Terraform state for reliable infrastructure management.

Deploy an AWS Site-to-Site VPN connection using Terraform with complete Infrastructure as Code. This guide covers creating Customer Gateways, Virtual Private Gateways, VPN connections with static routing, and test EC2 instances for quick hybrid connectivity setup.

Connect AWS and Azure using Site-to-Site VPN with Terraform. This guide covers complete Terraform configurations for both clouds, including Virtual Private Gateway, Virtual Network Gateway, and test VMs with troubleshooting tips.

Enable VPC Flow Logs using Terraform to capture network traffic data for security analysis and troubleshooting. This guide covers CloudWatch Log Group creation, IAM role configuration, and Flow Log setup with timing considerations and troubleshooting tips.

AWS Application Migration Service (MGN) provides automated lift-and-shift migration to AWS. This guide covers network requirements, agent installation, staging area architecture, and the launch process with troubleshooting tips.

AWS Elastic Disaster Recovery (DRS) provides fast, reliable recovery of on-premises and cloud servers to AWS with RPO in seconds and RTO in minutes. This guide covers setup, configuration, drills, failover, and failback procedures with troubleshooting tips.

Configure Route 53 Resolver endpoints to enable DNS resolution between on-premises networks and AWS. This guide covers inbound and outbound endpoints, conditional forwarding rules, and Active Directory integration with troubleshooting tips.

Deploy AWS Network Firewall in a centralized inspection VPC with Transit Gateway to inspect all north-south and east-west traffic. This guide covers architecture design, VPC creation, routing configuration, and traffic flow patterns.

Connect your on-premises Palo Alto firewall to AWS using Site-to-Site VPN. This guide covers Customer Gateway creation, Virtual Private Gateway setup, and complete Palo Alto configuration including IKE/IPsec settings and troubleshooting tips.

Import on-premises VMs to AWS without using CloudEndure or AWS Migration Services. This guide walks you through exporting your VM, uploading to S3, creating IAM roles, and running the import - plus troubleshooting tips for common issues.

Import RHEL VMs to AWS using VM Import/Export. Key differences from Windows: install NVMe/Xen drivers with dracut before export, use BYOL licensing via Red Hat Cloud Access, and connect via SSH. Includes supported versions, troubleshooting, and step-by-step instructions.

Use Terraform to deploy a highly available NGINX website with an Application Load Balancer, two EC2 instances across different AZs, and an S3 bucket for website files and log storage. Complete with troubleshooting tips and full code.