The world is on the verge of technical breakthroughs according to IT news and ads. We have immediate access to a vast body of knowledge, AI influences our decisions, and devices are powerful and intelligent enough to make decisions on our behalf. I’ve been working with many brilliant IT and business folks who are at the forefront of innovation and adopting new services, tools and applications. Yet when it comes to connecting platforms and teams, typical proposed solutions are direct point-to-point and often batch integrations.
Since most of our team is working with the MuleSoft platform, the “Why MuleSoft” question comes up quite frequently. Of course, it’s not limited to MuleSoft, and applies to any integration stack. The real question is “why do we need to stick anything in the middle? We can quickly extract or import a file, or expose an API and other systems can just call it”.
I like these discussions even though at times it feels a bit like a Sisyphean task. So why MuleSoft? Let’s walk through simple examples, the value the API and event platform add, and talk about the edge cases where the point-to-point integrations would make more sense.
Integration Architecture Principles
There are several things to consider when designing system integrations. First and foremost, the integration should work, meaning the right data in System A must be available in System B (and in most cases to systems C, D, … as well) at the right moment. This is where most project teams focus first – how to get these systems connected in the fastest way possible, minimizing the work required, costs, and dependencies on other teams or vendors.
As the processes mature beyond the “startup” mode, companies quickly realize that all connections must follow fundamental integration architecture principles:
- Security – often an after-thought for the initial phase, this is one of the most overlooked yet critical features of every integration. Common threat protections, policies, encryption of data in-flight and at rest, authentication and authorization, role-based access, governance and compliance processes, must be at a core of each mature integration practice
- Customer Experience – often directly correlated with how fast the event or a data change in in one system can be visible in others. This realtime-first architecture approach allows teams to engineer instantaneous and responsive experiences for customers, partners and employees
- Quality of Service – built-in auto-recovery, monitoring, log aggregation and monitoring help companies operate a growing number of integrations with minimal effort and proactively get notified of the issues so they could be addressed before users or customers start complaining
- Speed of Delivery – the time it takes to add a new partner, customer or internal system, often directly related to the amount of integrations reused vs built from scratch for each new engagement
Simple workflow
Let’s suppose we are implementing a commerce channel for a brand new SaaS provider. The reason I picked this example is that SaaS businesses tend to operate on a relatively small number of applications and therefore could be a good simple example for this article. Minimally we need an e-commerce app where customers can sign up for a service and maintain their payment methods, a provisioning system that provides the actual service, an accounting / financial system for invoicing and booking the revenue, and a CRM for sales and support
So why MuleSoft (or any other integration platform) would be a good architectural choice even in this simple solution?
Security: The integration platform offers unified API management and compliance, protecting the underlying applications from various attacks. Of course, it can also be done at each individual application’s level, which would in most cases multiply the effort and result in more gaps and therefore higher risks as the number of teams and systems grow
Customer Experience: the platforms such as MuleSoft and Confluent / Kafka are a great choice to implement API and event streaming-based integrations (although the MuleSoft platform is equally comfortable handling large ETL use cases). This helps make key customer journeys more interactive, for example, enabling service provisioning in near real-time
Quality of service: what happens if the provisioning service is not available or CRM API is overloaded? How do alert the relevant operations team that the order is missing required data? Should we implement the retry and guaranteed delivery/queue up for later processing? The choice is – either each team implements the logic of the same exception (meaning tripling or quadrupling the effort) or cuts corners and falls back to a manual exception process where we learn about the issues from our business teams or worst yet from our customers. The integration platform often serves as a unified exception and an alerting hub that collects the events from other applications and implements common lookups, dashboards, and alerts via various channels
Reuse: The direct point-to-point connections are tied to a specific source and target systems, severely limiting any opportunity for reuse. Every time a new system is introduced, it requires developing the same integrations again. But we are talking about a simple case here, so how is it relevant for this example? Fair enough, let’s make our use case a bit more interesting.
Omni-channel commerce
The success of any SaaS service directly depends on how easily customers can discover, try, and sign up for the product. Moreover, customers expect to transact via any of the available channels. To make our example just a bit more realistic, let’s assume our SaaS startup matures and now decides to let customers sign up for our service via several marketplaces, for example, AWS and Azure. Now the subscription order can come through either a direct commerce portal or from marketplaces.
In the point-to-point world, we would need to build a custom direct link to each marketplace and maintain the same security, quality of service etc, resulting in tons of copy/paste and code duplication across many systems teams
The main benefit the integration platform adds in this scenario is the ability to reuse most of the CRM, provisioning, and financial systems integrations, just need to implement the new marketplace-facing components.
It might appear we just move all point-to-point logic to the integration platform and shifted the responsibility to deal with all this mess to the integration team. That’s where the right integration architecture helps maximize the reuse and reduce the direct application dependencies with approaches such as API layering and common data models.
As the number of channel applications grows, the benefit of leveraging a unified integration platform and architecture quickly becomes apparent. The time it takes to roll out a new channel is significantly shorter than with the P2P option.
As our company grows, it acquires or partners with other SaaS service providers and offers bundle subscriptions that need to be provisioned in multiple product systems. Similar to adding more commerce channels, the integration platform enables the reuse and accelerates the rollout of new SaaS provisioning systems compared to direct point-to-point options.
Over time we end up dealing with quite a few customer-facing channel apps, provisioning systems, CRMs, subscription billing platforms, data warehouses, marketing tools, and other supporting apps. The point-to-point paradigm approach, while it seemed to be quick and simple for the first 2-3 apps, becomes a huge liability for the company – most process changes must be done in multiple places, and the coordination between the teams becomes a pain, and troubleshooting is hard. The overall solution is calcified, slow to change and expensive to operate.
P2P still has its place
I think by now everybody gets the idea that the P2P approach is not as simple as it looked at the beginning of this journey. So is there a place for it? Well, there are several use cases where it would still make sense to keep direct integrations.
- Out-of-the-box connections – when there is a built-in integration between two applications, and there’s no opportunity for reusing these integrations, the direct route works. For example, if the billing system has built-in support for payment processing, and no other system would leverage the same payment service, then the out-of-the-box direct connection works just fine as long as it aligns with the company security, quality of service and governance requirements
- Very large data sets – when dealing with huge volumes of data, adding an extra stop in the middle may result in longer processing times.
- Time-sensitive real-time processing systems – the direct real-time connections are almost always a bit faster, and the middleware adds a slight lag time. For the majority of use cases, this extra processing time is tiny and doesn’t materially impact any customer experience, however, for some contexts such as high-frequency trading, this would be a good consideration for architecting a point-to-point solution.
Conclusion
You’ve made it to the end of this post! Hope you are now convinced that you can’t scale your business without a solid integration platform and architecture. Even for simple tasks, they add security, governance, quality of service, reuse and improved developer productivity. As companies mature and their application landscape evolves and becomes more complex these platforms start looking especially attractive. Next time you hear “Why MuleSoft” (or WSO2 or Kafka or any other API / Event / Data platform) hope this article helps you better evaluate options and ultimately deliver a secure responsive flexible self-healing solution.
Recent Comments