{"id":4399,"date":"2020-06-24T08:00:08","date_gmt":"2020-06-24T15:00:08","guid":{"rendered":"https:\/\/intuitblog.com\/?p=4399"},"modified":"2024-03-13T09:24:10","modified_gmt":"2024-03-13T16:24:10","slug":"intuits-messaging-platform-on-public-cloud-scaling-high-and-beyond","status":"publish","type":"post","link":"https:\/\/www.intuit.com\/blog\/innovative-thinking\/intuits-messaging-platform-on-public-cloud-scaling-high-and-beyond\/","title":{"rendered":"Intuit\u2019s Messaging Platform on Public Cloud: Scaling High and Beyond"},"content":{"rendered":"

Authors:<\/em> <\/strong>Madhavan Narayanan, Amit Jere, Jason Webb, Amit Kaushal, Sankara Rao, Sajith Sebastian, Gokul Sarangapani<\/span><\/p>\n

Abstract<\/b><\/p>\n

Asynchronous integration is a key enterprise pattern that enables loosely coupled services.\u00a0 With JMS, loose coupling is enabled by services exchanging messages via a messaging system. Intuit\u2019s <\/span>Messaging Platform<\/span><\/i>, built on top of RedHat\u2019s ActiveMQ JMS technology, is running in AWS and <\/span>supports queueing use cases for asynchronous integration between components in the same application and between different services that are deployed in the same or different AWS regions<\/span><\/i>. It is a highly transactional multi-tenant platform, currently used by 2000+ service integrations processing 60+ Billion messages per year at a concurrency of 36K+ messages per second with less than 20 ms TP99 publish response time with 99.999 availability and 100% self-service support for Day 0\/1\/2 operations. Several tier zero integrations for Intuit flagship products like Quickbooks Online and TurboTax Online are built on top of the platform. While the platform operated out of Intuit\u2019s private data centers (aka IHP) for initial few years since its inception in 2015 and successfully served customer needs, with the migration to AWS in Summer of 2018 and also due to increasing customer traffic a significant degradation in performance and scalability was seen. In this paper, we present the various issues that were observed, the activities\/experiments that were done to troubleshoot them, and the solutions implemented thereof to continue to meet the business needs of high availability, performance, and scale for the next 3 years. The \u201cOutcomes\u201d section describes the improvements to the performance and scalability of the Intuit Messaging Platform.<\/span><\/p>\n

1. Overview<\/b><\/h3>\n

This section lists the significant improvements that were done to the performance and scalability of the Intuit Messaging Platform. The details of how these improvements were achieved are explained from section 2 onwards.<\/span><\/p>\n

Outcomes
\n<\/b>The table given below summarizes the improvements seen in the key metrics of the platform.<\/span><\/p>\n

\"\"<\/p>\n

2. Intuit Messaging Platform<\/b><\/h3>\n

What is Intuit Messaging Platform?
\n<\/b>Intuit Messaging Platform is an enterprise messaging solution that consists of a highly distributed set of ActiveMQ broker instances. ActiveMQ is an open source messaging software developed by Apache Software foundation and supports a fully compliant JMS 1.1 client.<\/span><\/p>\n

The platform is running in 2 AWS regions (US-WEST-2 and US-EAST-2) that are connected over WAN and other than providing DR, it provides an ability for clients to produce in one region and consume in another region, if required. To ensure high availability within one AWs region, the brokers in a swimlane are distributed across 3 AWS availability zones in one AWS region. Further to avoid noisy neighbor situations and manage the blast radius, the broker instances are grouped into multiple swimlanes, with each swimlane running a set of brokers interconnected using ActiveMQ\u2019s network-of-brokers topology. There are ~10 swimlanes in Intuit Messaging Platform and customers are typically assigned a specific swimlane through an onboarding process and are allowed to produce and consume messages within the swimlane.<\/span><\/p>\n

While ActiveMQ\u2019s network-of-brokers setup makes it possible for messages to be consumed from any broker within a swimlane, Apache Camel is used to route messages across swimlanes wherever needed. Camel is also used to support custom routing within a broker using header-based filters.\u00a0<\/span><\/p>\n

The diagram in Appendix A shows what a swimlane in AWS looks like.<\/span>\u00a0<\/b><\/p>\n

Why Intuit Messaging Platform?<\/b><\/p>\n