by Joe Kinsella
Cost allocation is a well understood accounting practice in IT organizations. But finding the right solution to support your business needs presents some significant challenges.
In this article we will explain the key differences that make cloud cost allocation reporting complex. We will review the available options, and outline requirements you can use to find the right solution for your organization.
While allocating costs for physical IT infrastructure is an activity that has been performed for multiple decades, cost allocation in the cloud is an emerging business activity that brings with it substantial new complexities. These complexities are driven by the unique attributes of cloud computing, including:
Organizations will vary substantially in the depth and breadth of requirements they need in a cost allocation reporting solution. Small organizations can often get by with basic solutions, including internally developed scripts, the Amazon Cost Explorer, open source, or freemium services targeted at small businesses. Enterprises and fast-growing technical companies with more sophisticated needs are more likely to need a commercial solution, or to invest in an internally developed system.
To better assess your organization’s need, let's walk through the key requirements for evaluating a cloud cost allocation reporting solution.
Most organizations will have several different ways they want to report their costs. Some common perspectives might include by department, application, product line, environment, customer and application role (e.g. Cassandra). These perspectives will likely change over time and vary by stakeholder. For example, finance may want a monthly breakdown of costs by product line or shared environment; operations may need a cost breakdown by project or team; and engineering may want a cost breakdown by application role.
In choosing the right solution for your organization, it is important to determine the level of flexibility you require for defining and managing business perspectives. Basic cost allocation solutions will allow you to report by Amazon tags and a few other simple attributes (e.g. security groups). More sophisticated solutions provide much greater richness for associating resources and their usage with different business perspectives, including the ability to express complex rules, define exceptions by rule / resource, create perspectives across a broad range of resource types, and then generate business groups from critical systems of record outside of AWS (e.g. Chef, Puppet, proprietary systems).
Below is a sample screenshot from CloudHealth that shows the creation of a perspective based on Chef environments. The service provides enterprise-scale flexibility, with support for creating business perspectives on 50+ types of resources, hundreds of attributes, and a variety of non-AWS data sources.
The level of detail at which you can report costs is often a critical requirement in deciding on the right cost allocation solution for your organization. There are typically three different levels of detail provided in most cost allocation solutions. These include:
Basic solutions will associate costs based on only tags; more sophisticated solutions will allocate costs based on resources and/or usage. The below screenshot from CloudHealth shows the tabular report of costs by an environment, including amortization costs from reserved instance purchases.
Each of the items in the chart and tabular report in CloudHealth are hyperlinked to allow you to seamlessly navigate between services, service items and resources. The image below shows a drill-down line item cost report directly from the Amazon Detailed Billing Record (DBR).
Being able to customize the cost allocation reports for different stakeholders is often a critical feature for many organizations. Some core features to look for include the ability to report in both charts and tables, export to different formats (e.g. CSV, image), customize the charts, and filter on specific criteria.
Below is a sample chart from CloudHealth showing hourly instance costs for web servers in production in the us-east region.
Maintaining a record of costs to allow for historical reporting is another feature to look for in a cost allocation solution. The retention of data and its accessibility for ad-hoc reporting needs are typical differentiator for different solutions. More sophisticated solutions will retain data for a year or more, and will make it available at different levels of time granularity (e.g. hourly, daily, weekly, monthly).
Below is a monthly historical report from CloudHealth reported by service item.
Most stakeholders want to receive their information and data proactively, either via email or a messaging application. Flexibility in scheduling content for your stakeholders is another key differentiator in cost allocation solutions. Basic solutions will require stakeholders to log into a web site or run a script to get their data; more sophisticated solutions facilitate highly relevant and customized scheduled report delivery directly to stakeholders through their preferred means of communication.
If your infrastructure is relatively small and changes infrequently, you might be able to get by with a basic solution driven primarily from tags. But for more sophisticated needs, you will likely want a solution that does more than just tags.
In deciding what solution is right for you, you should be aware of some of the limitations of Amazon tags:
A moderate to large-scale user of AWS can produce 50+ million rows of hourly cost data per month. If you store a year or more of this data and allow reporting for different time intervals (e.g. hourly, daily), it’s easy to require the storage of billions of rows of statement detail. In addition, many larger organizations have more complexity in their AWS accounts, such as multiple consolidated billing accounts, multiple standalone accounts, and a large number of accounts (it is not uncommon for an enterprise to have hundreds of AWS accounts). Choose a solution that meets your scale and performance needs.
CloudHealth delivers next-generation IT Service Management (ITSM), with a focus on delivering executive insight and recommendations to enterprise and enterprise-scale organizations. Among the many features of the product is support for a sophisticated enterprise-scale cost allocation reporting. The solution is primarily targeted at organizations requiring the advanced features discussed throughout this article.
Today there are a number of available options available for cost allocation including: basic tools provided by AWS (e.g. Cost Explorer), open source tools, commercial products / services, and internal custom development. The solution that is right for you will depend upon the scale and pace of change of your infrastructure, the internal needs of key stakeholders, and your budget. The information discussed throughout this article is intended to help you make an informed decision for your organization that meets your financial reporting requirements.