Revolution That Started When Information

¶ … revolution that started when information technologies became aligned with business strategies has resulted in an emphasis on enterprise architectures that will bring greater flexibility and agility in responding to business strategies' demands. The emergence of Service Oriented Architectures (SOA) is a direct result of the alignment of information technologies to line-of-business goals and objectives. The strong focus on creating an architectural framework that allows for agility in sensing and responding to demands is also critical for any enterprise today. Essentially, SOA is a software architecture that starts with an interface definition and builds the entire application topology as a topology of interfaces, interface implementations and interface calls. SOA would be better-named "interface-oriented architecture." SOA is a relationship of services and service consumers, both software modules large enough to represent a complete business function. Services are software modules that are accessed by name via an interface, typically in a request-reply mode. Service consumers are software that embeds a service interface proxy (the client representation of the interface).

The intent of this thesis is to explore the current state of the SOA and its many implications for enterprises, across all major function areas, specifically focusing on the integrative data aspects of this it initiative. SOAs are essential to the strategic plans of many enterprises, and as a result are re-aligning the development strategies of thousands of software companies globally.

Service-Oriented Architecture Overview

Across the industry, interest in SOA is running high. These days, "service-oriented architecture" is consistently one of the top five search terms on many research company's websites, and in a January 2006 LWC Research survey, half of all companies -- and 77% of large enterprises -- reported that they are or will be using SOA by the end of 2006.

The factors behind this high level of adoption center squarely on the need for more efficiently driving business strategies using it as an agile and responsive resource. Evidence is further mounting that SOA provides real benefits for business integration and flexibility. It is important to keep in mind however that the main benefit of SOA is the opportunity for incremental development, deployment, maintenance and extension of business applications.

Many myths have developed in the industry around SOA. The reality is more modest, but also more immediately beneficial than the hype. SOA brings these benefits to enterprise it:

Incremental development and deployment of business software

Reuse of business components in multiple business experiences

Low-cost assembly of some new business processes

Clarity of application topology

SOA does not bring these mistakenly attributed benefits:

Simple software engineering

Free integration or interoperability

Technology independence

Vendor independence

The ultimate architecture for the modern enterprise

Through 2007, growing enterprise experience with SOA process and SOA-based applications will eliminate the myths and instill appreciation for the real benefits of SOA in most enterprises. Evolving tools, skills and best practices will make development of SOA-style applications easier than development of monolithic applications. This change will shift the massive software industry mainstream into the new software-engineering reality: By 2008, in all likelihood SOA will be a prevailing software engineering practice, ending the 40-year domination of monolithic software architecture. SOA is a good practice for software design. Although it is not an answer to all problems, SOA is useful and should be part of most modern software projects. Over time, lack of SOA will become a competitive disadvantage for most enterprises. Mainstream enterprises should invest today in understanding SOA and building SOA design and development skills.

This thesis begins with an overview of the concept of the Private Trading Exchange, and how this specific it strategy is instrument in the development of the Service Oriented Architecture.

How Private Trading Exchanges Lead to SOAs

With the high level of interest in SOA, vendors of all stripes are providing SOA products and features, presenting a confusing array of options for architects crafting their SOA platforms. Application server platforms, integration products, Web service specialist vendors, and management tool vendors all are aligning their product strategies to be a responsive resource to enterprises, and it architects and business strategists must determine which ones will provide the right combination of features. The confusing array of options was captured well by one customer services client, who asked about SOA, "What is the best selection of tools for our firm?" It only complicates matters that SOA platform investment priorities are different, depending on your initial and future plans for SOA.

Companies are asking themselves a multiple of questions including which products to launch through which channels, how to track sales performance throughout multiple channels, how to align and synchronize pricing across both indirect and direct channels. The advents of Service Oriented Architectures have been also driven by the need to synchronize the flow of information between buyers, suppliers, and customers. The interaction of these three critical constituencies first lead to the development of Public and Private Trading Exchanges (PTX), and the growth of collaborative commerce at the supply chain level. The structure of the PTX was first defined by AMR Research (2001) in the report, Building a Case for the Private Trading Exchange.

This landmark report was actually setting the stage for the creation of Service Oriented Architectures, as the many integration points between buyers, suppliers, and customers. Figure 1 shows an example of the diagram of what a PTX consists of. This is the beginning of the actual progression of it architectures to SOAs.

Figure 1: Structure of the Private Trading Exchange

The fundamental concepts behind PTXs in turn fueled the development of a series of exchanges that would later serve as the foundation for different SOA strategies as well. The Independent Vertical Exchange (IVX), Independent Horizontal Exchange (IHX), Consortium Trading Exchange (CTX) and the continual refinement of the PTX all lead to the development of SOA as a unifying concept of strategies, integration points and the role of the synchronization points in the architecture. Figure 2 provides an illustration fo the structural differences of these varying approaches to building trading exchanges, a precursor and the foundation of Service Oriented Architectures.

Figure 2: The structure of Private Trading Exchanges

As the foundation for the SOA architecture, the following PTXs is where many of the lessons learned in the initial definition of what would eventually become a more agile, more business centric approach to aligning it with the strategic direction of an enterprise. The various forms of PTXs that emerged are briefly described here, and each has had a specific influence on the current state of SOA planning, implementation, and use:

Independent Vertical Exchanges -- Exchanges that facilitate trade in order to make a market more efficient, and which have formed without significant investment from existing industry players. Examples include CheMatch, e-STEEL, and Sci-Quest. These exchanges struggle with liquidity, and will ultimately merge with physical intermediaries or become vertically-specific Application Service Providers (ASPs).

Independent Horizontal Exchanges -- Exchanges that facilitate economies for a specific business process that transcends vertical industries. Examples include National Transportation Exchange, e-Credit, and Branders.com. These exchanges struggle with domain expertise across verticals, and ultimately may become business process outsourcers to increase revenue.

Consortium Trading Exchanges -- Trade participants within an industry that join forces to make the vertical market more efficient. Examples include Transora, Covisint, and WorldWide Retail Exchange. These exchanges struggle with consensus among members and scope.

Private Trading Exchanges -- a single company that implements marketplace technology to enhance its own processes. These exchanges struggle with trust and multiple points of integration. When considering the corporate structures of Fortune 2000 companies, with multiple divisions and broad geographic scope, don't be fooled into thinking that the Private Trading Exchange is a one-to-many model. As the model matures, these companies will be as complex as those based on the many-to-many model of the Independent Trading Exchange (ITX). In addition, the PTX should benefit all members of the exchange, not just the deploying company.

2001-2004: Transition Period from PTX to SOA

As exchanges in general proved to be more structurally sound for order management and less agile than planned for, coupled with the fact that in many exchanges, the only factor everyone could agree on was how to complete a transaction, SOAs began to gain significant momentum.

Long-term business competitiveness and success require continual change and adaptation. Service-oriented architecture helps to solve immediate problems such as connecting to business partners, accessing legacy applications, and integrating across technology boundaries but, at a more strategic and fundamental level, SOA is about creating an it environment to support continuous business optimization.

With each new it project, applications increasingly embody your business processes which in turn make strategies only as flexible as applications. Too often, with your processes frozen in company's applications, it is hard to make the business connections necessary to change and improve processes, and it's speed of change is the limiting factor on business speed of change. By contrast, SOA structures it applications in line with your business processes and, as a result:

SOA speeds business change. In its core design strategy of packaging applications into business units of work, service orientation reflects and models business processes. As the business changes, developers can more easily map business process changes to applications and then implement the appropriate it changes.

SOA facilitates business connections. With business processes packaged as modular, accessible business services, enterprises can connect them where and when they are needed to optimize processes across customers, partners, suppliers, and their own internal applications

SOA enhances business control. Because services model business processes, the flow of data and transactions through service-oriented applications is valuable business data. SOA infrastructure actively manages service flows and can provide flexible and dynamic access to this data, which enterprises can use to analyze and optimize business results and process costs.

As flexible, service-based applications make business change easier and faster, business people will take advantage of their new found agility to drive competitive advantage through a faster cycle of introducing new capabilities and optimizing core processes. To guide this faster optimization cycle and provide data to justify ongoing change, SOA's business process feedback loops provide flexible insight into business processes.

While there is no "one size fits all" approach to SOAs today, there are "platform design centers," or domains for which to plan the integration and coordinated use of multiple products and technologies makes a more cohesive SOA architecture possible than would have been the case with cobbling together disparate and unrelated applications. This concept of the SOA as a design centers provide a framework for organizing and integrating SOA platform choices. The following are identified four major design centers for SOA platforms:

Service life-cycle environment (SLE). The SLE provides tools and development infrastructure for defining, building, and delivering business services and the business processes they serve. As a coordinated set of tools for turning business requirements into business services, the SLE facilitates rapid implementation of business change.

Service delivery network (SDN). The SDN controls application-to-application connections and quality of service across technology, trust, and enterprise boundaries. It is the core messaging infrastructure for an SOA platform, providing both new and existing protocols, such as message-oriented middleware; distributed object protocols, such as Internet Inter-ORB Protocol (IIOP); enterprise services buses; enterprise application integration (EAI); and Web services.

Service command platform (SCP). The SCP provides both it- and business-level management. System monitoring and management are the foundation, but the SCP also provides insight for business management by dynamically tapping into the business data inherent in the flow of messages through the SDN.

Core application platforms. Your core application, data, and integration platforms host the business logic of your business services. As such, it is appropriate and necessary to include them in planning for your SOA platforms.

SOA Case Study: (Flughafen Zurich AG): SOAP-Based Portal

Business drivers for SOA. The firm Unique (pronounced YOU-nick) is the firm that operates Zurich Airport. In its pursuit of more efficient airport operations, Unique built a portal to improve management insight into major operational activities. This required drawing data from multiple core applications, each of which was run by a different outsourced service provider.

Solutions delivered. Unique's challenge in building the portal was that baggage systems, ground radar, and airport operations applications were each outsourced to different providers running on different underlying platforms and operating systems. Web services provided the mechanism to connect across these barriers; SOA provided the design model for interactions between applications.

SOA platform strategy. Uniques' new portal application, called ZEUS, has three major layers: the portal itself, a messaging infrastructure, and the business services provided by the underlying applications. The portal also includes some of its own services, such as a cache of operational data as is shown in Figures 3 and 4. Figure 3 shows the SOA architecture and Figure 4 illustrates the sourcing concept for SOA components.

Figure 3: SOA Architecture for Unique

Figure 4: Sourcing Decisions for Uniques' SOA Efforts

Core application platform. Unique required each external provider to implement custom SOAP interfaces to its applications, making them easily accessible to ZEUS. Unique wrote ZEUS itself on the.NET platform using.NET WinForms in the UI layer and C#.NET Web services in the business services layer.

SLE. Unique developed the portal layer's user interface and business services using Visual Studio.NET, with Visual SourceSafe for software configuration management. The SLE tools that external providers use are transparent to ZEUS.

SDN. Within Unique's SDN, SOAP is the dominant protocol for accessing services. But SOAP by itself does not provide a high quality of service, so Unique controls and consolidates access to service requests through webMethods. In case of a SOAP request failure, webMethods' reliable delivery features ensure that the message is eventually delivered. The external SOAP interfaces are accessed over a private network, and no further security is used for the connection.

SCP. Unique uses webMethods' message-flow monitoring as the core of its SCP.

Future platform possibilities. For a while after ZEUS was first delivered, Unique's SOA platform was adequate for current needs. Now, Unique is planning to add business process management (BPM) and workflow capabilities to increase its process flexibility and control. Beyond these, possible future additions include Web services management to report on the operations of both external services (accessed through webMethods) and internal services running on the.NET platform.

SOA Case Study: Charles Schwab: Unified Service Delivery Network

The adoption of SOA in the financial services industry continues to outpace many other service sectors of the global economy. In applying the design center concept to a financial services firm Charles Schwab, the following insights were gained relative to SOA adoption.

Business drivers for SOA. As is typical of many large enterprises, a financial firm had a wide diversity of it projects in progress simultaneously. Some of the firm's specific initiatives included applications for customer information, account inquiry, check imaging, and insurance sales and servicing. Rather than developing separate integration solutions for each project, the firm sought to provide common access architecture to share the business logic of applications that were spread across disparate technology platforms. SOA provided the right design model for reusable business services, and Web services provided an open, standard protocol to connect across platforms.

Solutions delivered. The firm built an infrastructure for service access across all of its major application platforms. The infrastructure was designed for general use across a variety of business solutions and scenarios. So far, the firm's service infrastructure does not support externally accessible services.

SOA platform strategy. Because its starting point was to create a common strategy for cross-application connections, the firm's focus has been on the service delivery network component of its SOA platform see figures 5 and 6:

Figure 5: SOA Architecture for Charles Schwab

Figure 5: Sourcing decisions for Charles Schwab's SOA

Key decisions for its SOA platform were as follows:

Core application platform. The major platforms on which the firm's business logic resides are Java (WebLogic), HP Integrity NonStop, and IMS on an IBM mainframe. In addition, BEA WebLogic Integration provides integration functions for the Java environment. The firm has a mix of custom and packaged applications.

SLE. In addition to the native tools for each of the firm's core application platforms, the development tool suite includes XMLSpy for editing XML specifications. Although the firm has not yet put in place a repository or registry, it uses Contivo's metadata management software to manage its inventory of data and service interface specifications. Contivo documents the meaning and usage of data elements and interfaces and generates XML transformation specifications that are used in request processing.

SDN. The firm's SDN is founded on SOAP interfaces for cross-platform compatibility. To speed up XML parsing and transformation, most of its reusable services are routed through DataPower's XML acceleration product as a single point of control, yielding faster performance, greater scalability, and easier development. The firm uses Systinet's tools and servers to add reliable delivery and failover to the SOAP capabilities of its J2EE infrastructure. It uses both SOAP and WebSphere MQ to access legacy platforms, including IBM's MQ-to-IMS bridge. Behind the initial SOAP-based connection points, any application may connect to downstream applications using native protocols.

As yet, the firm has not found it appropriate to rely on Web services standards for transactions. Where it is necessary to ensure the integrity of its service calls, the firm has implemented compensating transaction logic within both the service requester and the service itself.

SCP. The firm has not yet found it necessary to implement Web services management. Instead, it uses HP OpenView to manage its services by controlling the underlying platforms where the services are running. To accomplish integrated deployment to its SOA platform, the firm uses scripting tools to write custom deployment scripts.

Future platform possibilities. The firm has plans to expand the flexibility and robustness of its SOA applications by adding BPM, event-based processing, and enhanced security and identity management for services to its platform. To get more direct data on the operation of its SOAP interfaces, the company may evaluate Web services management solutions in the future. It may also consider using Web services transaction specifications in place of its custom compensating transaction logic as the specs mature.

Lessons Learned from the Design Center Strategy of SOA Implementation

The strongest results and best practices are emerging for those companies that start with the service delivery network of their it platforms' overlay to business strategies, and then build out their business processes. From these case studies, the major themes and lessons learned are as follows:

Start with a build-out from existing it infrastructure. Charles' Schwab's case -- where the initial infrastructure investments were only in an XML authoring tool and a custom, lightweight XML framework -- demonstrates that you can make a strong and successful start on SOA by working from your existing infrastructure.

Start with your SDN. Because SOA is about making connections across boundaries (technology, security, organization, or application), the SDN was commonly the first SOA platform investment made by the case study firms. Unique's first problem to solve was how to connect to its outsourcers.

Rely on major platform vendors including IBM, Oracle, SAP and Microsoft for SOA cornerstones. Most major vendors are pursuing a wide variety of SOA capabilities, and the early adopters profiled in this thesis that it is more effective to focus analysis on fewer vendors with broader capabilities.

Have a service-level agreement (SLA) management strategy from the start. In research each of the specific SOA accomplishments of each early adopter profiled, the importance of a common business-it understanding of requirements for service availability, reliability, performance, and scalability to an appropriate management strategy. A management strategy does not require the purchase of a WSM product:

Companies attaining best practices are being careful when putting business logic in the delivery network. Unique found that when it put business rules into the SDN, it was hard to maintain the consistency and coherence of business rules. Unique found that it was better to consolidate business rules as much as possible into the implementation of the service, giving control over service business logic to one organization: the service provider.

Define governance of data and service semantics. SOA early adopters have emphasized the importance of ensuring that all stakeholders have a common understanding of data and service semantics. Even slight misunderstandings of data usage can cause major issues. Charles Schwab has gone the farthest in its SOA platform to capture, share, and manage semantics, using Contivo to manage data and interface definitions.

Carefully evaluate SOAP/XML performance. Companies achieving best practices clearly understand the response time and latency requirements of your applications, and build prototypes to understand precisely what performance they can expect from your SOA platform. Charles Schwab found it necessary and appropriate to invest in an XML acceleration appliance.

Most early adopters of SOA realize that they will not need a single repository of data until their processes warrant it. It is significant to note that at present, not one of the case study firms has a formal service repository. Charles Schwab comes closest, with its use of Contivo for data and interface definitions. They have found that when they are early in their SOA efforts, when the number of interfaces is manageable, or when SOA is used among close-knit teams, they can be reasonably successful by using tactical methods to communicate available services. They have also learned that as their body of services grows, a repository alone is not a complete solution. The investment in a repository will be worthwhile once the firms have matured their processes and disciplines for using one.

Those companies achieving best practices are actively looking at service orchestration now. Both Unique and Charles Schwab found orchestration (for example, WS-BPEL process flows) highly valuable. In addition, BPM and orchestration are on the future evaluation lists of Unique and Charles Schwab.

Companies attaining best practices develop your own internal standards for Web services usage. With so many Web services specifications circulating in the market (50-plus) and with so few of them actually set as standards (about 15), there is much room for confusion in using Web services, and confusion leads to a lack of interoperability. To avoid losing control, make explicit organizational decisions about which specifications and standards to use and how to use them. Most of the Web services specifications are implemented as additional items within the header of a SOAP message, and different choices on what is in the header can lead to problems. Companies getting the best results are those that base your standards on the usage profiles from the Web Services Interoperability Organization (WS-I).

Companies who are getting best practices in SOA also adopt your own Web services interoperability testing regime. In defining internal usage standards, these companies make sure to prototype and test usage across multiple vendor implementations. The bank found it particularly important to test the interoperability of security standards (e.g., WS-Security). The financial firm Charles Schwab highlighted immaturity in Web services specifications for reliable delivery.

The highest performing companies craft a vision to guide tactical evolution of your SOA platform. Even when these firms made tactical SOA platform investments, they were guided by a longer-term vision. For example, neither Unique nor Charles Schwab has implemented WSM. Instead, they have made a tactical choice to manage their services using existing management tools. However, they are planning for future WSM use by carefully planning and controlling their use of headers in SOAP messages.

Best Practices in creating an SOA Vision in Enterprises

To evolve to an SOA platform that supports the big impact potential of responsive, adaptive, continually optimized business, enterprises must strive to get the architecture right. To get the architecture right, these organizations must design it according to the value that it is to provide. The big strategic business value of SOA -- rapid, continual, adaptive business optimization -- requires that SOA deliver on three core value propositions, which are defined as follows:

Rapid, flexible business change. Via traditional programming languages and metadata-based specification, your platform must provide tools and infrastructure for flexible modeling, creation, modification, testing, and maintenance of your business services.

Rich, deep business connections. Across a diverse set of enterprise and technology boundaries, your SOA platform must support diverse combinations of specific requirements for security, reliability, responsiveness, style of interaction, and more.

Business-level and it-level feedback and control. Control is the business value provided by strong management. Besides traditional it management requirements, your SOA platform must allow you to dynamically tap into your service request-response flows and extract key data for managing your business.

Beyond these the three core values of SOA, enterprise platforms must also encompass the traditional technologies that now run global businesses, including application platforms (.NET, J2EE, legacy), integration solutions, packaged applications, and custom applications.

Evaluating SOA Adoption Today

Large and medium-size enterprises lead SOA adoption with 62% and 61%, respectively, using SOA by the end of 2006, versus 44% for small enterprises. Adoption is strongest among the very largest firms -- enterprises with 40,000 or more employees come in at 67% adoption. The correlation is stronger when measuring "large" as the ratio of it staff size to total number of employees. Enterprises where it is 4% or more of the employee base report 71% adoption of SOA by the end of 2006 versus 42% for those where it is less than 1.5% of the employee base. Considering that, in the LWC Research survey, different size enterprises have about the same mix of it staff ratios, this is a better predictor of interest in SOA than enterprise size. A similar correlation holds when segmenting SOA adoption by simply the number of it staff. LWC Research believes that a relatively larger it staff tends to mean that a firm is doing more with it, which leads to greater it complexity and a higher level of interest in better architectures, such as SOA. but, even the smallest firms are rapidly adopting SOA: 44% of SMBs reported that implementing an SOA was a high or critical priority.

Business Transformation Is the Strategic Impact of SOA

SOA can be used in a tactical way, wherein services are designed around targeted needs to more flexibly connect specific applications, or in a strategic way, wherein services are designed in business terms around business units of work and transaction requirements. The more strategic approach creates, in effect, a digital model of the business that is ready to connect to any business process, customer, partner, or supplier wherever and whenever it is needed. This results in a higher degree of strategic business flexibility, and LWC Research data shows that enterprises -- especially large enterprises -- realize the more strategic value of SOA.

It is not surprising that the data shows that most firms using SOA use it for internal integration. Many expected very little external integration via SOA, but because there are simple ways of establishing a basic level of security for external connections (primarily virtual private networks, most frequently two-way SSL), 42% of large enterprise SOA users are doing external integration, as are about 30% of medium-size and small enterprises. But it is surprising that, this early into the market-entry cycle of SOA, 46% of large enterprise SOA users (and about 27% of SOA users at medium-size and small enterprises) say they are using it to achieve strategic business transformation This is a radical difference from the introduction into the market of object-oriented (OO) development and component-based development (CBD), neither of which was credited with enabling business transformation. and, once again, all of these numbers are very similar across both Europe and North America.

Fewer Medium-Size Firms Have Enterprise Strategies for SOA firm can either: 1) use SOA sporadically across a variety of projects, or 2) work to achieve a coordinated enterprise wide strategy for SOA. Among those already using SOA, medium-size enterprises are less likely to have enterprise commitments to SOA. Conversely, enterprises with less than 100 it staff or 400 or more it staff are more likely to have enterprise commitments. This makes sense considering LWC Research's conversations with medium-size firms that often report that, unlike larger companies, their it departments are too small to have a formal enterprise architecture team and, unlike their smaller counterparts, too large to move the enterprise in a common direction without an architecture team. The trend for "medium-size" to mean less enterprise adoption of SOA is there, but is weaker, when slicing the data by it staff size ratio: Raw number of it staff is a stronger indicator here.

Web Services Adoption is higher than, and overlaps with, SOA Adoption

Consultancies including LWC Research, Gartner and others data shows that there is greater adoption of Web services than SOA. With very similar numbers across both Europe and North America, 67% report that they are already using Web services versus 39% that are now using SOA (with or without an enterprise strategy). But adoption of one encourages adoption of the other. Among Web services users, current use of SOA goes up to 47% (59% expected by the end of 2006). Among current SOA users, Web services usage goes up to 80%. However, while Web services users are more likely than average to be using SOA and vice versa, there remain significant numbers using one or the other but not both. Forty-one percent of Web services users report no plans for SOA, while only 20% of current SOA users are either unaware of Web services or aware of but not using Web services.

Many researchers in this field has long maintained that firms should separate their thinking on SOA and Web services, and a likely explanation of the greater adoption of Web services over SOA begins by understanding the difference between the two. Service orientation is a set of application design and architecture concepts that provides better packaging and structure for enterprise applications. Web services provide a standard set of protocols for connecting applications across technology and enterprise boundaries. Using the two together, Web services provide an additional level of value on top of SOA by making services available to a broader audience. In addition, as Web services mature, it will enable a more dynamic and configurable application environment. Web services adoption is likely higher because simple use of Web services is easy and direct. Developers can quickly connect two systems using Web services without thinking through the broader design issues associated with a full view of SOA -- and even without thinking much about the design of service interfaces at all. However, without SOA's broader design concepts, developers risk creating services that are not as flexible and reusable over the long haul.

Firms are doing SOA with or without direct funding common question among those considering SOA is, "How are enterprises paying for it?" The nirvana that many questioners seek is that they might find a large bucket of infrastructure money with which to build their SOA -- money that is justified based on the long-term value of SOA without the need to demonstrate near-term savings on specific projects. Such money would save them the trouble and difficulty of finding creative ways to evolve their SOA with each business solution project delivery. In the researchers' discussions with users, some seem to believe that only firms with large it R&D budgets can make progress on SOA. LWC Research survey data shows that, while it helps to have large budgets, many, if not most, firms have to get by without one as they move to SOA.

High Spending on Application Maintenance Doesn't Stop SOA Adoption