Case Study

How CloudHealth Helped Toast Quickly Scale Their Cloud Environment During COVID-19

Granular Reporting
The EBS Volumes Report shows unattached storage volumes, saving Toast more than $27k per month
Quick Optimization
The RDS Instances Report is “a quick win” that saves Toast about $12K annually
COVID-19 Savings
Toast saved $50,000 during the first month of COVID-19 without negatively impacting business performance

Toast empowers restaurants of all sizes to build great teams, increase revenue, improve operations, and delight guests. Toast pairs their deep understanding of the restaurant industry with powerful cloud-based software and restaurant-grade hardware to deliver an intuitive, all-in-one platform across point of sale, guest marketing, digital ordering and delivery, and payroll and HR.

Born in the Cloud

Toast has been an engineering-driven company since its founding in 2011. As a startup, Toast built their infrastructure quickly in the cloud. The competitive restaurant technology was running on-premises, and Toast was a leader of its kind: a cloud-based solution that could scale quickly on demand.

The technology operations team is responsible for Toast’s cloud strategy and for making sure infrastructure meets demand. Another group, the scale team, ensures databases are healthy and conducts research on cloud technology. Toast also regularly evaluates new AWS offerings to find the best solutions for their environment.

Cloud Challenges

Toast’s biggest hurdle as a cloud-native startup was decentralization. There was no unified vision at the outset and the company had to quickly standardize their processes in order to control costs. Ken Siskind, Engineer Programming Manager at Toast, used to work primarily with the TechOps management team. Now Ken works across teams and with other Program Managers to improve feature delivery processes and control AWS cloud costs.

Early on, Toast was using AWS native tools that weren’t sufficient for deep cost analysis. The Toast team needs granular visibility into the cost of running new features they develop, and AWS native tools are like a black box. From a leadership perspective, it’s crucial to understand costs to know if they’re getting good payback for what’s being built. “AWS native tools weren’t enough to figure out where our spend was. The team was moving so fast and focusing on features and we realized we needed another tool to help figure out where our spend was,” Ken said. The TechOps management team realized toast needed a third-party tool that could help them spend more wisely.

Partnering with CloudHealth

Toast uses many CloudHealth reporting features to optimize their cloud environment. Custom reports are delivered daily to the appropriate stakeholders.'

"By just glancing at a report, within seconds you can see if there’s a big spike, and in a matter of minutes you can turn around and turn off AWS services that shouldn’t be running."

Ken Siskind
Engineer Programming Manager, Toast

The team uses the RDS Instance Usage Report to see instances that haven’t been used in 30 days. Ken describes the report as “a quick win” that saves the team about $1K per month with one email. Ken sees the report and can quickly ping the Toast DevOps team to get to the root of the problem. “It’s very satisfying to see the next time that report comes out, that line item is gone. You realize that just with one email you quickly save money,” Ken highlighted.

With the Unattached EBS Volumes Report, Ken realized there were cost spikes by teams that had gone unnoticed. Once the team identified the wasted spend, they were able to quickly optimize.

Image
toast-ebs-volume-savings

 

“Having those email reports really help us know what to keep an eye on,” Ken highlighted.

Toast also reviews daily Cost History Reports within the CloudHealth platform to see a detailed breakdown of where their money is being spent. A monthly cost report keeps the team accountable for cloud spend, showing exactly how they’re doing against their budget. Due to the level of detail in Toast’s reports, CloudHealth helps break down and communicate the important information, improving general data hygiene and eliminating wasted time and effort.

A weekly analysis of costs used to take hours, but with CloudHealth it only takes 15 minutes a week. “We don’t have to dig around and spend a lot of time within the tool now that we have the infrastructure,” Ken said.

Adapting to COVID-19 with CloudHealth

Up until COVID-19 hit, Toast was investing heavily in the cloud and didn’t worry about optimizing cloud costs. The main focus was to release more features and stabilize the platform. But at the start of the pandemic, the team immediately shifted strategy and started thinking about how they could cut costs.

Toast dove deeper into the CloudHealth platform to figure out opportunities to optimize.

Image
toast covid-19 savings

 

The team looked at oversized and underused resources to save money fast. “It was a big effort to reach out to teams and pare back what we were doing,” Ken shared. When going to production with new services, “you really need to be diligent about spinning up and launch dates,” Ken highlighted.

CloudHealth Perspectives have been a huge help for Toast during COVID-19, allowing the team to see their costs by relevant business groupings, such as by team or environment. When Ken and team began building Perspectives last summer, they didn’t know where to begin. The CloudHealth team set up reports in advance so they could hit the ground running. Toast’s CloudHealth Technical Account Manager showed the team new features and how to use them so they felt up-to-speed with everything in the platform.

“When we have a problem, the CloudHealth team is really responsive. It’s been a great team to work with. I don’t know how we would’ve gotten to the point we are without the CloudHealth team.”

Ken Siskend
Engineer Programming Manager, Toast

Toast plans to continue using CloudHealth reports to discover new opportunities to optimize their cloud resources and save big on their AWS bill.