How I Cut AWS CloudWatch Costs by 50%: Moving VPC Flow Logs to S3

Why CloudWatch Costs Are So High

Last month, a customer came to me frustrated about their AWS bill. After reviewing their billing dashboard, we discovered something alarming: over 50% of their costs were coming from CloudWatch vended logs, specifically VPC Flow Logs.

(Vended logs are logs automatically generated by AWS services like VPC Flow Logs, Route 53 query logs, or CloudFront access logs—you don't create them yourself, AWS creates them for you.)

This is a common problem I see with AWS users. VPC Flow Logs are incredibly useful for monitoring network traffic and troubleshooting security issues, but when you're storing everything in CloudWatch Logs, the costs can quickly spiral out of control.

Here's why CloudWatch gets expensive for VPC Flow Logs:

• Data ingestion charges per GB
• Storage costs that accumulate over time
• No automatic retention policies by default
• Vended logs from AWS services adding up quickly

The Fix: Move VPC Flow Logs to S3

After analyzing their use case, I realized they didn't need real-time querying for most of their flow logs. They mainly used them for weekly security reviews and occasional troubleshooting. This made them perfect candidates for S3 storage.

How I Did It (4 Simple Steps)

1Move VPC Flow Logs to S3 with Parquet Format

Instead of sending VPC Flow Logs to CloudWatch, I configured them to go directly to S3. The key was choosing Parquet format instead of plain text.

In the VPC console, I created a new flow log with these settings:

• Destination: S3 bucket
• Log file format: Parquet (this is crucial for compression)

That's it. Parquet format automatically compresses the logs by 80-90%, which means massive storage savings.

2Set Up S3 Lifecycle Policies for Archival

Storage costs can still add up if you keep everything in S3 Standard forever. The most cost-effective approach is to configure a Lifecycle Rule that moves logs to S3 Glacier classes automatically.

I configured a simple Lifecycle Rule for the log bucket:

• Day 0-30: S3 Standard (for immediate analysis with Athena)
• Day 30+: Transition to S3 Glacier Instant Retrieval (cheaper, but still queryable)
• Day 90+: Transition to S3 Glacier Deep Archive (lowest cost: $0.00099/GB, for compliance)
• Day 365: Expire (delete) the objects permanently

This "set it and forget it" policy ensures we never pay full price for logs we haven't looked at in months.

3Set Retention Policies on CloudWatch Log Groups

This was the biggest quick win! Many log groups had no retention policy, meaning logs were kept forever and costs kept growing.

I went through each log group and set appropriate retention periods based on actual needs:

• Application logs: 30 days
• Debug logs: 7 days
• Audit logs: 90 days or more (based on compliance)

You can do this in the CloudWatch console by selecting each log group and choosing "Edit retention setting". This immediately stops old logs from piling up and costing money.

4Query Logs When Needed with Athena

Since the logs are now in S3, I set up Amazon Athena to query them when needed. You only pay for what you query, which is much cheaper than storing everything in CloudWatch.

The Parquet format makes queries fast and inexpensive because it's already compressed and organized efficiently.

The Results: 50% Lower CloudWatch Bill

The results were immediate and impressive:

• CloudWatch costs dropped by 50% in the first billing cycle
• Parquet compression reduced storage needs by 85%
• Query performance actually improved with Athena + Parquet
• Retention policies prevented future cost creep

Frequently Asked Questions

What is Parquet format in AWS?

Parquet is a columnar storage format that compresses data by 80-90%. Instead of storing data row by row like a text file, it stores data column by column, which makes it much more efficient for analytics and dramatically reduces storage costs.

How much can I save by moving VPC Flow Logs to S3?

Most customers save 50% or more on their CloudWatch costs. The exact savings depend on your log volume, but the combination of lower S3 storage costs and Parquet compression typically delivers massive savings compared to CloudWatch Logs.

Will I lose real-time access to my logs?

VPC Flow Logs sent to S3 have a slight delay of 5-10 minutes, but you can query them anytime using Amazon Athena. For real-time application monitoring and alerting, continue using CloudWatch Logs. Use S3 for historical analysis and compliance.

What should my CloudWatch log retention be?

It depends on your needs. A good starting point is 7 days for debug logs, 30 days for application logs, and 90+ days for audit logs required by compliance. Never leave retention as "Never expire" unless absolutely required.

Can I migrate existing CloudWatch logs to S3?

Yes, you can export existing CloudWatch logs to S3, but it's a manual process and incurs export costs. It's usually better to start fresh with new logs going to S3 and let old CloudWatch logs expire based on their retention period.

My Takeaway

Moving VPC Flow Logs from CloudWatch to S3 with Parquet format was one of the easiest wins for cost optimization I've implemented. The combination of direct S3 delivery, Parquet compression, and proper retention policies delivered immediate results.

If you're seeing high CloudWatch costs on your AWS bill, start by running a billing analysis in AWS Cost Explorer. Look for vended logs, check which log groups have no retention policies, and identify candidates for S3 migration. The savings can be substantial.

For more information on VPC Flow Logs, check out the official AWS VPC Flow Logs documentation and the S3 pricing guide.