Pepperdata Blog Bonus Myth of Apache Spark Optimization

In this blog series we’ve examined Five Myths of Apache Spark Optimization. But one final, bonus myth remains unaddressed: Bonus Myth: I’ve done everything I can. The rest of the application waste is just the cost of running Apache Spark. Unfortunately, many companies running cloud environments have come to think of application waste as a cost of doing business, as inevitable as rent and taxes. This acquiescence to cloud waste has become pervasive, affecting even the most sophisticated IT teams. In fact, respondents in a recent survey indicated a 39 percent year-over-year increase in the amount of cloud spend over budget, and a third of companies say they will exceed their cloud budget by up to 40 percent. In this final blog installment in this series we’ll demonstrate that things don’t have to be this way—and we’ll propose a solution. Cost Optimization at the Infrastructure Level In this blog series we’ve examined five different options that can help remediate cost overruns in the cloud, along with the benefits and limitations of each: Observability and monitoring Cluster Autoscaling Instance Rightsizing Manual application tuning Spark Dynamic Allocation The fundamental gap with all of these options is that none of these solutions addresses […] The post Bonus Myth of Apache Spark Optimization appeared first on Pepperdata.