Choosing the Right Cost Allocation Reporting Solution

by Joe Kinsella

Cost allocation is a well understood accounting practice in IT organizations. But finding the right solution to support your business needs presents some significant challenges.

In this article we will explain the key differences that make cloud cost allocation reporting complex. We will review the available options, and outline requirements you can use to find the right solution for your organization.

Why Is Cloud Cost Allocation So Hard?

While allocating costs for physical IT infrastructure is an activity that has been performed for multiple decades, cost allocation in the cloud is an emerging business activity that brings with it substantial new complexities. These complexities are driven by the unique attributes of cloud computing, including:

• Dynamic infrastructure - Instead of making a capital expenditure for servers and storage a few times a year, cloud infrastructure gets provisioned / de-provisioned on-demand by the hour and minute.
• Delegation of control - Instead of lines of business collaborating with IT on the infrastructure required to support their needs, they frequently now have direct control over the provisioning and de-provisioning of their infrastructure.
• Pay as you go - Over-provisioned infrastructure in physical data centers does not typically have a substantial impact on operating budgets, since the underutilized infrastructure was likely purchased via a capital expenditure in a previous budget. But in the cloud, all infrastructure is paid for based on consumption, and therefore has an impact on operating budgets.
• Scale - Due to the low-level components available in IaaS (e.g. compute, storage, application-level service), the cloud has substantially increased the number of assets / resources an organization must manage and report costs against.

Key Requirements

Organizations will vary substantially in the depth and breadth of requirements they need in a cost allocation reporting solution. Small organizations can often get by with basic solutions, including internally developed scripts, the Amazon Cost Explorer, open source, or freemium services targeted at small businesses. Enterprises and fast-growing technical companies with more sophisticated needs are more likely to need a commercial solution, or to invest in an internally developed system.

To better assess your organization’s need, let's walk through the key requirements for evaluating a cloud cost allocation reporting solution.

#1: Business Perspectives

Most organizations will have several different ways they want to report their costs. Some common perspectives might include by department, application, product line, environment, customer and application role (e.g. Cassandra). These perspectives will likely change over time and vary by stakeholder. For example, finance may want a monthly breakdown of costs by product line or shared environment; operations may need a cost breakdown by project or team; and engineering may want a cost breakdown by application role.

In choosing the right solution for your organization, it is important to determine the level of flexibility you require for defining and managing business perspectives. Basic cost allocation solutions will allow you to report by Amazon tags and a few other simple attributes (e.g. security groups). More sophisticated solutions provide much greater richness for associating resources and their usage with different business perspectives, including the ability to express complex rules, define exceptions by rule / resource, create perspectives across a broad range of resource types, and then generate business groups from critical systems of record outside of AWS (e.g. Chef, Puppet, proprietary systems).

Below is a sample screenshot from CloudHealth that shows the creation of a perspective based on Chef environments. The service provides enterprise-scale flexibility, with support for creating business perspectives on 50+ types of resources, hundreds of attributes, and a variety of non-AWS data sources.

#2: Detailed Cost Reporting

The level of detail at which you can report costs is often a critical requirement in deciding on the right cost allocation solution for your organization. There are typically three different levels of detail provided in most cost allocation solutions. These include:

• By service - Report on costs by AWS service (e.g. EC2, EBS, RDS). This is the highest and most simplistic level of reporting.
• By service Item - Report on costs by detailed components within AWS services (e.g. EBS PIOPS, EC2 data transfer, S3 API calls). This is essential for gaining additional insight into costs, and supporting answering the “why” questions (e.g. why did my EBS costs go up this month?).
• By resource - Resource level reporting provides the detailed costs associated with specific resources. This is the lowest level of reporting and is critical for identifying specific usage and reason for costs (e.g. how much did we spend for compute, storage and data transfer on marketing’s web server)?

Basic solutions will associate costs based on only tags; more sophisticated solutions will allocate costs based on resources and/or usage. The below screenshot from CloudHealth shows the tabular report of costs by an environment, including amortization costs from reserved instance purchases.

Each of the items in the chart and tabular report in CloudHealth are hyperlinked to allow you to seamlessly navigate between services, service items and resources. The image below shows a drill-down line item cost report directly from the Amazon Detailed Billing Record (DBR).

#3: Customizability

Being able to customize the cost allocation reports for different stakeholders is often a critical feature for many organizations. Some core features to look for include the ability to report in both charts and tables, export to different formats (e.g. CSV, image), customize the charts, and filter on specific criteria.

Below is a sample chart from CloudHealth showing hourly instance costs for web servers in production in the us-east region.

#4: Maintain History

Maintaining a record of costs to allow for historical reporting is another feature to look for in a cost allocation solution. The retention of data and its accessibility for ad-hoc reporting needs are typical differentiator for different solutions. More sophisticated solutions will retain data for a year or more, and will make it available at different levels of time granularity (e.g. hourly, daily, weekly, monthly).

Below is a monthly historical report from CloudHealth reported by service item.

#5: Automated Delivery

Most stakeholders want to receive their information and data proactively, either via email or a messaging application. Flexibility in scheduling content for your stakeholders is another key differentiator in cost allocation solutions. Basic solutions will require stakeholders to log into a web site or run a script to get their data; more sophisticated solutions facilitate highly relevant and customized scheduled report delivery directly to stakeholders through their preferred means of communication.

#6: More Than Just Tags

If your infrastructure is relatively small and changes infrequently, you might be able to get by with a basic solution driven primarily from tags. But for more sophisticated needs, you will likely want a solution that does more than just tags.

In deciding what solution is right for you, you should be aware of some of the limitations of Amazon tags:

• Not available on all resources - While the availability of tag support across Amazon resources has increased over the last couple years, there are still many resources that are not taggable (e.g. CloudFront distributions, elastic IPs, S3 object), and also more that are untaggable (e.g. S3 objects, support).
• Requires custom integration - While small / static environments can tag using the AWS Console, everyone else needs to build tagging into their process / automation for provisioning infrastructure.
• Immutable - When you tag an AWS resource (e.g. an instance), all usage related to this resource is permanently marked with your user defined tags. These tags are recorded in billing records, such as the Amazon Detailed Billing Record (DBR). While you can change the tags associated with a resource, you cannot change the historical reporting of usage from its previous tag. This limits flexibility in supporting the more sophisticated needs of larger organizations.
• Not cloud portable - If you support or plan to support more than one cloud provider, you will need a more general-purpose solution than simply AWS tags.

#7: Enterprise Scale

A moderate to large-scale user of AWS can produce 50+ million rows of hourly cost data per month. If you store a year or more of this data and allow reporting for different time intervals (e.g. hourly, daily), it’s easy to require the storage of billions of rows of statement detail. In addition, many larger organizations have more complexity in their AWS accounts, such as multiple consolidated billing accounts, multiple standalone accounts, and a large number of accounts (it is not uncommon for an enterprise to have hundreds of AWS accounts). Choose a solution that meets your scale and performance needs.

CloudHealth Platform

CloudHealth delivers next-generation IT Service Management (ITSM), with a focus on delivering executive insight and recommendations to enterprise and enterprise-scale organizations. Among the many features of the product is support for a sophisticated enterprise-scale cost allocation reporting. The solution is primarily targeted at organizations requiring the advanced features discussed throughout this article.

Final Considerations

Today there are a number of available options available for cost allocation including: basic tools provided by AWS (e.g. Cost Explorer), open source tools, commercial products / services, and internal custom development. The solution that is right for you will depend upon the scale and pace of change of your infrastructure, the internal needs of key stakeholders, and your budget. The information discussed throughout this article is intended to help you make an informed decision for your organization that meets your financial reporting requirements.