Unleashing the AWS Detailed Billing Record
by Joe Kinsella
With little fanfare last year, Amazon released programmatic access to a Detailed Billing Record (DBR), an hourly statement of all activity by resources and tags for an account or a series of linked accounts. With its release, Amazon successfully delivered the most comprehensive billing transparency of any public cloud provider, giving customers the specific time and description for incurred costs down to fractions of a penny ($0.000001 to be exact) and resource (e.g. instance i-29779204).
But in delivering this transparency, AWS also created a new challenge for their customers: navigating the voluminous and dense data provided with the DBR.
Amazon Billing Reports
Amazon provides customers four different billing reports, each of which can be turned on by enabling programmatic access to your billing information. These reports get delivered to an S3 bucket in CSV format, typically updating every few hours. The four reports include:
- Monthly report - A monthly statement that provides a summary of costs at a service level. This is effectively a CSV version of the monthly statement you receive from Amazon.
- Monthly cost allocation report - This is the same as the previous report but includes tag information to allow basic categorization of the charges.
- Detailed billing report - A statement of all billing charges on an hourly basis without tags and resource information. This CSV was the predecessor to the DBR.
- Detailed billing report with resources and tags (a.k.a. the DBR) - A statement of all billing charges on an hourly basis with both tag and resource information.
Why Use the DBR?
The DBR can be used for many purposes, including performing a cost analysis from a business perspective, and for investigating unexpected charges in a statement. Some common questions that might get answered from a DBR include:
- What specific charges make up my Elastic Load Balancer costs?
- What volumes are responsible for the increase in PIOPS charges in my HRIS application?
- Why did my network charges increase so much within my development environment?
- ...and so on...
In short, the DBR is a comprehensive and authoritative source of all charges for your AWS infrastructure.
Limitations of the DBR
It’s hard to complain about the DBR, since it has unleashed a horde of valuable information and insight into costs. But there are some limitations to be aware of:
- Not all costs reported by resource - While most costs are reported by resource, some are not (e.g. RedShift, DynamoDB, EC2 Snapshot). Some of these limitations are simply the result of some AWS teams not delivering the right level of metric data, and likely will be resolved over time. Others (e.g. snapshots) are more fundamental and likely will not be resolved in the near future.
- Reserved instance reporting - While you have complete transparency of the application of reserved instance pricing to an instance, the DBR does not identify the specific reservation applied, making it challenging to answer complex RI questions.
- Data overload - This report will produce several thousand rows of information for a small customer, but will generate millions and tens of millions of rows of information per month for medium and large customers. This volume of data is beyond the scope of Microsoft Excel, and so typically requires some scripts / software to analyze.
There is also one other edge case limitation to be aware of that is not likely to be encountered by the typical customer: changes to your statement that occur after invoicing (e.g. a billing error) will not get reflected back into the DBR. So while it is authoritative in almost all cases, there are occasional exceptions.
Unleashing the DBR
You can do some basic work with the DBR with Excel and a little scripting - but if you use CloudHealth, we do all the heavy lifting for you. We automatically monitor for the availability of new DBRs from AWS, and collect and process the data to provide you real-time access to highly customizable reports. We also will automatically aggregate multiple DBRs for you, so you can get an integrated view of cost or usage even if you have multiple consolidated bills and/or standalone accounts.
To illustrate, I’ll walk you through a few simple examples. Below is a cost history report for multiple accounts, showing costs for the production environment only for a customer:
All items in this graph are clickable, allowing you to navigate from a specific item (e.g. S3 API charges), to the specific line items in the DBR that support these charges. So if you are wondering why your S3 API charges went up this week, just click on the specific item in the chart to see the line item charges from AWS:
All of our reports are also available in tabular format as shown below. They can be exported and allow you to customize for viewing up to three dimensions at a time with filters available on any dimension (e.g. show ELB charges for a single customer).
This means that within 1-2 clicks and less than a second of time, you can navigate from a business perspective (e.g. detailed costs for a specific department), to the specific line items in the DBR that support these costs. This works for customers of all scale, even those that have billions of line item charges over their last year.
You can also analyze costs within specific types of services, such as EC2 instances. For example, the below chart shows the cost per day of EC2 instances categorized by functional role (e.g. web servers, search servers, databases):
Note: In this case, the categorization actually comes from the roles defined in Chef.
I could go on showing you dozens of uses for the DBR to support the analysis of cost and usage of your cloud infrastructure. If you are not a CloudHealth customer, we highly encourage you to take advantage of the power of the Detailed Billing Record. If you are a CloudHealth customer, you are just a few clicks of your mouse or taps of your finger to unleashing its power.
If you have any questions about the DBR, please don’t hesitate to contact us at firstname.lastname@example.org