We were integration specialists working for large companies. We developed strategy, architecture, process and skilled people to mature the integration capability of our organisations.

We felt confident in spending significant resources as we believed strongly in our goal, providing long-term agility for our organisation and independence from large vendors.

We fell short.

What happened?

1. Falling Short

"Keep your eyes on the stars and your feet on the ground," Theodore Roosevelt

The place to develop an organisation's application integration capability is known as either an Integration Centre of Excellence (ICoE) or more commonly as an Integration Competency Centre (ICC). As an industry we use this term to describe an organisation's efforts, not necessarily a specific team.

An ICC can be described as being one of the following levels (Schmidt & Lyle, 2005):

• Level 0 - Project Silos
  • Pre-competency establishment, best-efforts, project-by-project
  • Benefit: Independent and Innovative
• Level 1 - Best Practices
  • First stage of competency, standards defined
  • Benefit: Shared knowledge
• Level 2 - Standard Services
  • Technology platform enforced
  • Benefit: Enforcement of standards
• Level 3 - Shared Services
  • Common resources centralised
  • Benefit: Resource optimisation
• Level 4 - Central Services
  • All resources centralised
  • Benefit: Complete control
• Level 5 - Self-Service
  • Automation of all processes
  • Benefit: Innovative and Efficient

Many organisations target a Level 5 as it makes integration invisible to the organisation and is a powerful enabler of change. Unfortunately it is our experience that very few companies ever reach such a level of competency. The select few that do make it, are often unable to sustain such an endeavour due to funding or other organisational factors.

One major concern that is rarely discussed is the impact of reaching too far. If an organisation attempts to reach higher levels, but is unable to achieve the target, it can be worse than staying at a lower level. A good example is a Level 4 ICC, where the whole organisation must integrate through a central team. If the team is unable to perform adequately the business may have been better off with running at Level 0 where each project is able to make decisions based on their specific need.

As ICCs are central to all change within an organisation, reaching too high can be risky.

2. Astronomical Cost

"30 cents in every dollar spent on software projects is spent on integration", Gartner 2014

In order for an integration competency to become an enabler, significant and consistent investment is required over time. Some very large enterprises do have the level of investment required to build a Level 5 competency. For everyone else, the investment required can be considered too high.

It is important however to consider not just the direct costs, but also the indirect costs that occur when the promise of integration doesn't meet the mark.

• Direct project costs like designers, developers, testers, etc.
• Indirect project costs, tools-down delays to other groups when integration deliverables are late.
• Direct support & maintenance costs in staff, licensing, infrastructure.
• Indirect support costs when "ticket tennis" is played within support organisation due to a lack of visibility.
• Recruiting & retaining key SMEs who hold critical enterprise IP.
• Indirect cost to enterprise & solution architecture in deep-diving into technology due to a lack of high-level visibility.
• Direct costs of complex licensing arrangement that can lead to major unexpected costs (e.g. physical to virtual).
• Indirect end-to-end regression testing costs related to upgrading, OS, DB, network, application stack, integration platform, integration pattern, etc.

Which do you believe would add up to more in your organisation? The direct or indirect costs? When the entire cost of integration is added together it can be overwhelming. If we were to include the result of lost business opportunity when we don't deliver projects, it would be astronomical.

3. Division over Time

The technology of an enterprise is generally developed by layering systems over each other. The superseding systems are the focus of all new change and continuous improvement. We can draw some parallels to archaeology as we rarely give any attention to the previous layers. The difference is that unlike archaeology which deals in centuries, IT can create many new layers in a single decade.

Some people argue that major transformation programmes can completely remove the previous layer. Our experience is these "mass extinction" events are rarely successful. Instead, the new norm is the concurrent management of many layers of technology. With the emergence of cloud and digital, we see the divergence of ideas accelerating.

This theory works down to application integration platforms. Most medium to large organisations host at least two major integration platforms. They may come under different names including Workload Automation, ETL, B2B, EAI, Message Queuing, ESB, A2A, RPC, and API. The goal however is the same, connecting applications. Each platform is designed to meet a particular need. In some cases this need has passed or the technology has been superseded, but in many other cases the technology continues to deliver for its particular purpose.

Don't get us wrong, we are not suggesting you keep platforms alive that serve no purpose, but after decades of shovelling dirt over the previous generation we might want to question our layering approach. One of the most important aspects of a self-service ICC is providing visibility to how everything connects. Most ICCs do not have accountability across all integration technologies, leaving organisations without an accurate picture for how its applications connect.

4. Technology Over People

It is a major struggle to keep an integration platform alive from day-to-day. Resources are deployed to support and manage complex integration platforms and infrastructure. Other resources are sent to complex programmes to manage the design and delivery of new integrations. This often leads to groups that have considerable technology skills, but little understanding of the broader ambition required for an ICC to meet its objectives.

We see significant evidence of a lack of people-focus in both the setup and ongoing management of ICCs:

1. Many ICCs are only built around a subset of the integration technology. Some members may be great people mediators but only for a limited scope of integration technology.
2. Rarely are ICCs involved in the early stages of complex integration programmes or vendor agreements. Many projects are fatally wounded  in the very early stages due to a lack of appropriate technology advice. 
3. Often developers are not equipped with the skills to manage the complex people relationships that are present in many integration programmes. Project and technical leadership can only assist to a point.
4. Developers focus on development tools to improve the efficiency of their day job. These tools could be hugely valuable to the organisation, but are not built in a way that allows anyone else to connect.
5. [Get in touch, we have a few more...]

Integration developers and support staff are not just another group of technologists. Their lack of training in people and collaboration skills has a significant negative impact on an organisations ability to deliver complex IT solutions.

5. Command & Control

We all aspire to empower our customers, whether it is a business unit integrating a core system or an external vendor working on a mobile application. We also work to simplify our internal processes with the goal of reducing internal cost. These two goals are perfectly valid, however when misaligned they can come into conflict. This is especially prevalent in a world where one customer differs significantly from another. Let's use an example:

• External Mobile Vendor: A mobile provider wishes to complete a iOS/Android application to assist with a marketing campaign. Some additional fields are required on the read customer API. The lifespan of the mobile application is between 3 and 6 months.
• Major Internal B2B Programme: A major initiative delivering a new B2B gateway that will interact with new vendor network. Over 50 integrations requiring development. The lifespan of the B2B network is probably 10 years plus.

These two customers have radically different value drivers. The mobile vendor probably views cost and speed to deliver as the most important. The B2B programme would be very conscious of rigour and reliable delivery timeframes, especially as any unexpected delay to integration components almost always has significant downstream cost implications.

Centralised ICCs who have very different customers, but focus on a too prescriptive universal process are at significant risk of failing. An inflexible set of control can be very dangerous in a competency that is key to making any business change. 

Why is it important?

1. We Didn't Solve the Issues of the Past

Application integration has always been an important part of many organisations.  Here is our list of some of the challenges that we believe can be assisted by a mature integration capability.

• Business Process Orchestration
• External Partner Integration (B2B)
• Legacy Modernisation
• Big-data and Insight
• Channel Development (Web, Mobile, etc.)
• Audit and Security
• Mergers, Aquisitions and Divestment

There are certainly other skills that are important for meeting these challenges, but poor integration can become a major blocker.

2. The Digital & Cloud Revolution

Enterprise cloud services (especially fully-managed Software-as-a-Service) is slowly starting to mature to the point that it is challenging our on-premise systems and capability. There are strong arguments to move non-core functions to an organisation that runs at a greater, more cost-effective scale. The challenge isn't in transition to such services, but in how we retain strategic control, governance and an end-to-end view on customer.

There is some debate on what "digital" means for an organisation. Is it a responsive interaction that a new generation of customers now expect?  Is it the digitisation of a businesses process from within? Is it just getting better at the web-first journey we started many years ago? Whatever the definition, we know it is pushing the need for more open and real-time access to our systems.

Please watch our video to see how we believe these revolutions are disrupting IT as we know it today.

What now?

1. Liberate The Business From Single-Method Integration

"By 2017, in large organisations at least 65% of new integration flows will be developed outside the control of IT departments", Gartner 2013

It will probably happen anyway, but strategies that restrict an entire enterprise to a single technology/process/quality are going to fail in the long-term. As ICCs we must remove that which implies too much about what our customers want. Let's use some specific examples.

We believe these statements are impacting to some customers (i.e. bad):

• "We only use a single platform to do all integration"
• "We only use files for B2B communication"
• "We always use web services for internal communication"

We believe these statements are non-impacting to any customer (i.e. NOT bad):

• "We prefer to use a single platform for most real-time integration at this time."
• "We prefer to use files for B2B communication when cost-effectiveness and simplicity are important at this time."
• "We implement most internal communication using web services at this time."

Non-impacting statements do not restrict customers to a single technology/process/quality. Impacting statements can have significant impacts on the customers of the ICC and therefore the entire business.

The question remains, how to cost-effectively manage governance and strategic control without restricting our customers to a single pace of change? We believe this is where metadata is critical.

2. DRIVE TO ZERO COUPLING

In IT we are very good at coupling. We attempt to couple not just technology, but people, timing and teams. Let's use some examples:

• "To be safe we must have an outage for every change we make to our HA infrastructure layer"
• "To be safe it is best if one project releases two weeks apart from the other"
• "We can only make changes to our integration platform in the order we prescribed two weeks before release"
• "Although our source control system cannot see any conflicts between these projects it is best if we force full enterprise regression testing"
• "If we change this line of code we will need to regression test the entire enterprise"

Each of these statements, if left to become a standard, has significant cost and agility implications.

How do we reduce coupling?

• Deploy smaller units of software that are cohesive and loosely coupled (some call this Microservices).
• If centralised run-time systems are required only allow for offline activities that do not require everything to be regression tested when they change.
• Use platforms that provide the minimum required functionality for the particular software unit being deployed. This will improve the corrosion resilience of the software unit (in other words less upgrades over time).
• [Get in touch for more...]

As in other areas of software development, these strategies present a significant opportunity to improve the field of application integration. Again, the question still remains, how do we manage a smaller set of software units that provide this major reduction in coupling.

3. A Meta-First Approach to Integration

"I wouldn't say you have an online life and a real life. I think technology is just mapping and organizing what already exists," Ashton Kutcher

We have always had considerable focus on metadata within integration competencies. For example, when a support call was made we used spreadsheets, integration IDs and other mechanisms to track where the issue resided. All of this information, everything but the code, we consider metadata. Unfortunately the methods we use to retain metadata (e.g. highly technical spreadsheets or complex project documentation), mean that in practice it is only assessed by a few individuals within the ICC.

As we ramp up the number of integrations and variation in technologies, we must shift to viewing metadata as the number one concern. Today's world, where only a few individuals have any reasonable access to metadata, will not work as we increase complexity. Imaging if only a handful of people had access to a map of a city and you had to line up and ask them for directions every day?

Coupled with this understanding of the "passive" metadata around how things connect, we must also have "active" metadata around each and every instance of an interaction (some call this big data). The idea of a single intrusive control-bus (Hope & Woolf, 2004) is no longer valid in a world where there is little hope of controlling each technology platform. Instead we must focus on machine data learning systems which can cost-effectively manage data from a variety of sources with minimal knowledge of how they operate.

When combined in a simple and cohesive manner, metadata and machine data learning systems can release something that has been unattainable until now. A loosely-coupled, fine-grained and diverse mix of technologies, that is simple to understand from a high-level.

4. Self-Service (level 5) Must be the Norm

We believe in a future where companies are differentiated based on their capability to interact with cloud-based service providers while retaining excellent customer insight. In this new world the ticket for entry will be a self-service integration capability that is able to provide significant control and rigour, while being highly responsive and agile.

Some organisations have lead the way in this endeavour with considerable sustained investment. The challenge over the coming years will be in achieving such capability for a much broader set of organisation at a significantly reduced cost. This isn't a simple problem solve and there will be hurdles along the way, but the shared experience we have built over the past 20 years will help set up a new path to bring transparent, best-of-breed integration to many more organisations.

How can Memento help?

Memento provides service to organisations that helps them achieve a self-service integration capability. Such a capability is so responsive, tailored and mature it becomes invisible to the business. Integration requests are submitted electronically, right-fit technology is selected, and delivery is responsive, all while ensuring a single empowering view across the enterprise. Our founding principles help us achieve this goal.

1. Transparent
2. Comprehensive
3. Independent
4. Best-of-breed
5. Sustainable

Synergies between a sustainable best-of-breed technology mix and our people focussed way of working provide very different outcomes for our customers.