Tuesday, April 22, 2008
SOA and its Linkages with Service Design Lifecycle
Heartening to see the fact that the authors first and foremost mentioned "service architecture" as a first view within EA framework shown as well as spend time detailing their view of SOA in this book.
(b) Service Design Package
Obviously a lot of what is written in this book is rather simplistic in terms of what is widely applied to packages or apps in the past has been re-written here - and ignores the type of teams - consumer/producer - within a SOA project, but I would think that authors can be excused as this concepts are now maturing and may not have been widely established when the books were written
- the SOA practisioners can be advised to read this book to understand how to transform their deliverables into actionable service delivery artifacts as required or whether they can be merged in one.
(c) The book starts off by asking for a clear definition of word "service" - thanks - so that everyone is clear on how business service, technical service or consulting service is defined.
SOA and Linakges with Service Strategy
While reviewing the book on service strategy in its initial draft, I could not notice help noticing how there are inadvertently two notions of a strategy -esp for shared services IT departments like us. We have several business units each having its own IT strategy and then obviously our own IT strategy. A lot of relevance of Service Strategy book can be found focusing very well on second type of IT strategy carrying forward the themes from each BU.
Lets first look at how most IT Strategist define an IT Strategy. Typically in context of relevant aspects of what and how business wants to execute its own strategies we develop two main components: Application Startegy and Operations strategy. Underpinning these are three other areas: Enterprise architecture - a connected view of what current and future state IT looks like, Decision Making guidelines - Governance rules regarding investments underlying all the application, operations change plan decided and a resourcing plan agreed with the BU.
It seems obvious to anyone that you cannot have a standalone operations startegy call it service strategy if you like or not. And obviosuly you would have noticed from previous posts on this blog that even seemingly harmless decisions like SOA as core design philosophy for business can have radical impact on the operations side - be it tooling, process, resources, reporting etc.
So It needs to be argued that service strategy cannot be done on its own in isolation and requires coordination of overall IT strategy.
On Consolidated Service Portfolio:
we talked about service portfolio plan in its own isolation wrt SOA earlier in this blog. Service strategy book nicely also talks about service portfolio which is not just a catalogue of what is operational, but also what can be potentially offered. I would suggest that the SOA SPP and the Operations SPP have to be merged in near future. It is also not uncommon to include technical services (aka IT services) in SOA SPP in mature organization so that IT as a whole can present business, technical and consulting service portfolio together - there are many views to the same - some focusing on a BU and some are more holistic in nature.
Thursday, April 17, 2008
Impact of SOA on IT Ops and Infrastructure
Conceptual Impact Items:
(.) Recognize that the notion of application as business service is being replaced By SOA business services
(.) Move to Business Service Management rather than IT Service Management – rather than just manage IT as Business, be part of Business is the challenge
(.) Inherent Dynamicity in SOA environment – what is a called chain today can dynamically alter tomorrow for good reasons
(.) Feedback loops on QoS achievement of service back into development time is important for people reusing the service. Cannot be an afterthought
(.) Reworking of SLAs to change focus from managing SD own risk to Business risks and policies
Benefits
(+) Role of service manager responsible for a domain of services from transition, operations and improvement perspective will come into limelight
(+) Better and tighter integration expected between service operations/transition teams with solution development teams
(+) SOA is the basis for enabling BSM and allowing service delivery to make this leap in easier way than traditional model
Negatives
(-) Far more complicated environment to manage with lot of moving parts and complicated Infrastructure
(-) Chargeback for FTE deployed to manage applications today will be shaken up as the shared service model will now extend to software further complicated the resources deployed to manage the same. Eventually this will get simplified as the execution on services will determine the chargeback between business units and within IT as well.
(-) Expecting a single consolidated product set of policy management and monitoring at the same time from Mega Vendors is not going to happen in near future.
(-) Change management and release management for services is order of magnitudes more complicated than traditional applications
WHICH ITIL Version for SOA?
As enterprise architects, we have many times defined IT Strategy – we seem to have Infrastructure architects who are seen as dictating terms on Operations strategy to service delivery teams. At the same time, most Operations groups are happy sitting at the back waiting for stuff to be given to them, focusing on day to day work rather than on seeing the big picture or aligning with business goals they need to be serving. A lot of this attitude stems from the fact that most service delivery folks just read two books of ITIL v2 – service support and service delivery and forgot the rest of them. For them everything starts and stops with these two books. Even service delivery processes which involve more involved thinking are usually pushed back to architects to handle.

ITIL V3 takes a major revamp as part of its focus and puts Business IT alignment and enablement back into focus revamping the structure of books in a way that no service delivery guy can back off from the strategic part, design part and continuous improvement part and just focus on operations and at best transition. Note the power of focus and the fact that certification strategy seems to make a versatilist service delivery resource rather than too focused process oriented element monitoring resource.
I heard Sharon Taylor speak last year on Business IT alignment messages in ITIL V3 – all of these sound magical to me as this is exactly what we have been talking about in successful SOA deployments. Now the service delivery guys cannot hide behind lack of established processes or cheat sheats that they have been accustomed to in the past from ITIL V2 anymore. Further sections just explore this in more detail covering individual books and showcasing how SOA and ITIL V3 co-exist.
The power of focus should not be misjudged or under-estimated – ITIL V3 is what you would want to ensure Strategic SOA deployments will work longer term for you. However make no mistake that the process definitions and tooling will evolve in ITIL V3 to handle the complexities of SOA/BPM – there is no magic pill for all remedies in the books for all types of problems seen in the deployments.
Distinction between ITSM and BSM
The main hindrance in accepting ITSM as a framework for enabling SOA from a service delivery viewpoint is the fact that it is really too “IT focused” or “IT Service focused” – it never actually tries to make a leap into understanding business in business terms and ensuring that service delivery ensures participating in those business outcomes. It is forever stuck in checking QoS of either large monoliths apps or technical services like email, internet access, network accessibility etc. The classic IT service catalogue will focus on Service support and hosting only.
Business Service management takes a different route and maps back the “Business service” to the underlying IT components and tends to measure the QoS of the business service including throughout, availability, and performance as first level concept. The business service if properly aligned to business goals indirectly measures the overall business performance as well. The key to note here is that many early business service management projects seem to claim that they are managing business in business terms but unfortunately the service itself being monitored was not aligned to expected business outcomes or their QoS did not consider the “big picture” – typically these were done to manage or fill the IT Infrastructure parts of very focused BPR projects.
Now note why SOA is important to BSM and vice-a-versa – SOA ensures that Business services are defined at right granularity – the alignment to business goals – the QoS specifications etc – BSM them ensures the Infrastructure monitoring and deployment environment is mapped and measured as per the service levels drafted against the business service. Doing it this way, esp. collating the QoS or business needs against pure play IT services (e.g. email) ensures IT service delivery starts understanding the implication of IT service downtime in email server in terms of no of business services affected and potentially business outcomes affected.
So going forward it is imperative to set the stage for rolling out SOA on BSM rather than ITSM. Unless off course you consider BSM to be a specialization or part of ITSM.
Impact of SOA on Infrastructure and Operations
Conceptual Impact Items:
(.) Recognize that the notion of application as business service is being replaced By SOA business services
(.) Move to Business Service Management rather than IT Service Management – rather than just manage IT as Business, be part of Business is the challenge
(.) Inherent Dynamicity in SOA environment – what is a called chain today can dynamically alter tomorrow for good reasons
(.) Feedback loops on QoS achievement of service back into development time is important for people reusing the service. Cannot be an afterthought
(.) Reworking of SLAs to change focus from managing SD own risk to Business risks and policies
Benefits
(+) Role of service manager responsible for a domain of services from transition, operations and improvement perspective will come into limelight
(+) Better and tighter integration expected between service operations/transition teams with solution development teams
(+) SOA is the basis for enabling BSM and allowing service delivery to make this leap in easier way than traditional model
Negatives
(-) Far more complicated environment to manage with lot of moving parts and complicated Infrastructure
(-) Chargeback for FTE deployed to manage applications today will be shaken up as the shared service model will now extend to software further complicated the resources deployed to manage the same. Eventually this will get simplified as the execution on services will determine the chargeback between business units and within IT as well.
(-) Expecting a single consolidated product set of policy management and monitoring at the same time from Mega Vendors is not going to happen in near future.
(-) Change management and release management for services is order of magnitudes more complicated than traditional applications
what Infrastructure runs SOA solutions
Lets talk about the context of what Infrastructure is needed to design and run SOA in all its totality. Note that here we are talking about more than 20 services at least in a complex business organization.
First lets take a classic look at what drives the service lifecycle before commenting on the Infrastructure:
SPECIFICATION TREE STATUS – ENTERPRISE ARCHITECTURE OWNERSHIP - Starts with “start state” and takes it to “
PROVISIONING TREE STATUS – DEVELOPMENT & EA Team shared OWNERSHIP - Being provisioned composite state – start from Specified and takes it to Certifiable state
TRANSITION TREE STATUS – SERVICE TRANSITION TEAM OWNERSHIP - Takes from Certifiable state to
OPERATIONAL TREE STATUS – SERVICE DELIVERY OWNERSHIP - Takes from
Lets look at what pieces are now involved in these lifecycles:
(1) Specifications: We have come to the conclusion that an IDE is not the best place to start doing SOA service specifications. The best way to do this and keep appropriate linkages is to start off in an EA repository – which keeps the enterprise business and IT strategy, process models, application models but also service descriptions as per richer service specifications and maps both upward and downward components. Keeping this solely in an IDE is not possible
(2) Provisioning Tree: This is typically done mostly in an IDE if custom developed or proprietary development tools. But it is not uncommon to have externalized tooling to support functional testing and performance testing of service operations
(3) Transition Tree Status: This typically requires two service registries – one wherein the service specifications as well as service UDDI compliant model is pushed to which acts as a complete repository of searchable and browsable service artifacts, appropriate links are published out so that link back to service components, technology models, infrastructure models or upstream to business processes can be shown straight off the EA repository. The other registry just maintains a discoverable service list for runtime consumers and is updated back with Policy management tools on achieved QoS. CMDB deployments to cover element level linkages are down now to services. The repositories maintain status of all version of services and service specifications with their status. It is possible to do simple stuff with newer registries to automate the approval process for state movements manually or automated checks.
(4) Operational Tree Status: This is the really complicated piece and expect to have following components:
a. Business Process Orchestration or State machine engine and the related Business activity Monitoring engine
b.
c. Runtime Policy management engine which can centrally manage security, performance, availability monitoring of deployed services
d. It is not uncommon to have Business rule engine separately, a complex event processing engine which can take immediate feedback loops to understand patterns amongst all the stuff which goes around and can trigger intelligent pattern recognition and somewhat cognitive behavior.
e. Sometimes large complex SOA implementations will come with Master data management solutions which publish out certified golden record as a message or provide data services from their repositories.
f. Many organizations, but not all, will have WSRP or JSR 168 compliant Portal engines which provide end user mashed-up web 2.0 interfaces as well. It is but expected that service implementations will be residing and delivered from some application server and their own database servers.
g. Monitoring all this complex infrastructure is a set of Monitoring tools – typically starting with an end user response time monitor, element monitoring, Tracer monitor, event correlations etc.
h. Since SOA also focuses on external world so much, most organizations integrate with suppliers, partners, customers etc using electronic system to system interfaces leveraging B2B Transaction engines capable of supporting EDI as a basic minimum with digital identities and certificates, transformation back to canonical formats, or even domain specific B2B protocols like RossetaNet, HL7 etc.
Note that situation can get complicated when there are more than one process orchestration engines or ESBs within an enterprise – not uncommon – and has happened fast which may or may not talk to each other very well.
How many services in an average enterprise
Considering an average of 10 to 18 domains per organization, with an average of 10-12 services per domain – we are looking at between 100 to 200 services as first class business services in an average organization end to end. If I map this to my current IT services portfolio of applications that are being served for my XPZ port operations, we support a total count of so called applications = 40. This number is going to increase to 150 services as we gradually go on adopting SOA. A lot of these services are from corporate shared services ownership as well.
Approach to Strategic SOA
At this point, let us not get stuck with what is a symptom of Integration projects more akin to traditional EAI enabled through JBOWS (Just of Bunch of web services) but focus more on how we achieve strategic SOA visioning and realization. In more than one sense, this approach is what
Coming back to service portfolio plan – this is a combination of several artifacts – and to give a high level content overview of the same, we delve into classic definition as described by CBDI forum and some of my comments on how this has been practically implemented in many organization.
An SPP is a list of all software services that will be needed to assemble solutions as per SOA principles. Creating a plan upfront has several benefits because we can establish case for SOA – understand what will be shared going forward, how do we reduce duplications in data/logic/capabilities, how do we plan to aquire them and how will they be laid out in IT landscape by time/infrastructure/technology etc.
First and foremost using very well proven techniques of compartmentalizing processes, business types and organization roles into Business domains is done for an organization. In my earlier life starting EA, my ex-company used to refer to them as Business Components – Large coarse grained collections with very little dependency between them. In practice, it is possible to split an organization into 10-18 (typically 12-13) such chunks.
- At what level do we intend to specify services – all its crowning glory now or we sketch the services list at one go and leave for a project to come in before we really sketch it out in completeness per domain as a later stage (recommended practice as you can die doing complete service/ operations and their specification list)
- How do we want to aquire services – commercial decisions aligned to IT Gov/arch policies
- How to certify/ reuse the services
- Governance rules
- Architecture and Layering policy – example was shown above – you may want to hedge bets on something like Microsoft Motion, IBM SOA method, OASIS recommendations as well.
- Service lifecycle tooling support – which tools/standards are used to store requirements, design time, validation and runtime of services
The services within each business domain will then have a service view which depicts there specifications in far more details than is indicated by a WSDL i.e. transcending typical I/O and end point type of stuff – but be more focused on how business owner is, what is the service representing in portofolio layer, QoS spec etc. The similar spec is applied at operations of services. Upstream linkages (to business processes or higher layer of services) and downstream linkages are created.
The services are mapped to technology that will create implementations for the service operations.
Technologies are then shown deployed on infrastructure – servers, data centers, devices, network – by various environments in implementation views.
Tuesday, April 15, 2008
Exampes of Business Service in Brave new World
However lets look at a context of how SOA would generate business services which are in themselves - first class assets - I will try and leverage examples from my practical experience rather than focus on simple customer examples.
First and foremost, there is a notion of service layering which SOA governance board or an equivalent party has defined for you. In our case, we have been using what is classically referred to CBDI-SAE (www.cbdiforum.com) meta model for service layering.
| Layer/Main Role | Operations Are | Dependencies | More rules | Data Storage |
| Solution Logic: UI, channel and dialog-specific processing | Non-existent - this layer does not contain services | May call Process, Core Business & Utility services directly | Supplies user interface, validation, user messages, session management | Not normally, except for temporary session data |
| Process Services: orchestrate other services; apply business process-specific rules | UI-independent, but designed for a specific business process | May call Capability, Core Business & Utility services directly | May be called by solutions that support other business processes | Only where not stored by Core Business Services. Stored data likely more transient than for core services. |
| Capability Services: may be standalone, or utilize lower level services | UI- and process independent, and designed to support one particular ‘business capability’ | May call Core Business, Underlying & Utility services directly, unless it’s a standalone capability | Standalone capability services must not depend on shared services (but can be an assembly of non-shared services) | If standalone, stores own data; if integ-rated, may store any data not managed by core business & lower services. |
| Core Business Services: apply enterprise-wide business rules | UI- and business process- independent, so can be used in different contexts | May directly call other Core Business, Underlying and Utility Services | Cyclic dependencies not normally permitted, except for “call-back”. May not call Process Services | Normally maintains principal data stores, unless this delegated to Underlying Services |
| Underlying Services: apply the business rules, but not exposed as well-formed service | Highly generic or implementation-based, so its interface not ideal for exposing to solution developers | May call Utility Services, but normally would not | May not call Core Business or Process Services | Often maintains significant data stores, and core services call underlying services to access this data & its processing rules. |
| Utility Services: shared by Core Business Services | UI-, business process- and often domain- independent | May call other Utility Services directly. Some may use Underlying Services | Cyclic dependencies not normally permitted. | Often required — for directories, look up tables and logs for example. |
Now lets apply these sets of principles to a domain in area of Port Operations called "Marine". This is typically a department which is run by a port authoirty to service the container terminal or General cargo terminal and essentially guides the ships from high seas to the terminal berths. It uses very well qualified pilots (like airlines pilots), tugs (small boats which nudge and push big ships) as the main resources to get the ships coming in at a slotted time as per terminal's request. The planning algorithm is quite complex as it needs to take into account the distances between the high seas and various locations in the terminal. Also consider the impact of tide, weather (esp Visibility) etc on the same. Sometimes, there are exceptions to the rule - Military vessels just can barge in on high priority - the priority rules are a combination of which ships have been waiting for how long in high seas and where they are supposed to go etc.

With this brief background, lets start exploring how a service and event driven system is built around this domain.
The solution layer we can call "Marine app" is a combination of several UI channels - bunch of jspslenght of channel (dont expect network or even GPRS coverage to be there always) etc. Note that there are atleast 3-4 entry points to the app - some of the application logic is being serviced by non-reusable application logic specific to that channel.
The next layer - which we call process services - has three processes. Lets focus on most important ones - the "Request to Berth" - this can be submitted based on a terminal working schedule published and changed by terminals or through the external agent directly from portal.
This is an end to end process of accepting the request as tentative, communicating any changes during initial plan or during movements to the originator and finally closing the move by sending an invoicing event to billing system. Each intermediate callback can have a SLA esp when invoked externally
The second set of services called capability services will typically be what are the fundamental capabilities that exist in this domain - in most software service views, we dont show what is manual - but for clarity - the fundamental long lasting capabilities in this domain are about being able to plan the movement, being able to track the movement BUT NOT TO ACTUALLY MOVE THE VESSEL (THIS IS NON-AUTOMATED SERVICE). There is a tendency in SOA architects to map this layer to activities - our judgment has been on fundamental long lasting capabilities rather than just pure play activities. It is important to note the alignment of these services to business objectives and startegywrt to business rules but also to service differentiation within the rules - if 90% of my revenues come from CT - I give them preference in planning, I dont try and delay CT vessels, if CT's biggest customer is XYZ shipping line, my planning service recognizes the service context and tries to give it maximum attention.
On the core business service - a simplified view is presented - but these are major nouns with the marine boundaries where data stores are maintained and rules applied - Vessel, Voyage, Resources (pilots), and Agents making the requests as well as vessel movement plan itself.
Rather than confuse the picture with unpublished underlying services, we focus on utility services which are mostly around weather, tide, employee services shared across multiple domains.
Note the following with these services
1. We are talking to services (and not operations) - each service may have one or more operations
2. Each service has a business owner responsible for governance and evolution management
3. Each service is tied to business outcomes and goals that it needs to serve.
4. Each service incidentally has a service level on security, availability, response time specification, message delivery policies (once and only once, reliability of messaging), idempotency
5. Each service has a richer service specification than standard Input output protocol details of contacting the service and the end point reference, usually this will have a business name of service, technical name of service, pre-conditions, post-conditions, which business processes it serves, what is it dependent on (aka service assembly dependencies), commodization level (Core capability, specialty, innovation, treadmill)
6. Most of the services as show above are described and are discoverable either through UDDI spec or Reusable asset specification with in an enterprise registry/repository. The registry/repository is synced up to development asset library (EA tool or integrated service environment). Not unusual to have two sets of such registry/repository - one for all states of services before they get operational and one for all operational services in production
7. Each service has a lifecycle and governance associated with it. Each service transition involved either automated checks (like WSI interoperability check, performance tests, and automated functional tests) or manual tests and approvals.
Given all this, it should be evident that we are not just treating business service as a derived principle but as a first class concept and type which stands on its own.
The History of SOA & ITSM
It needs to be understood that history and origins of SOA are in pureplay software development world - it may have been driven by Gartner to begin with in 1996 under a slightly different pseudonym - but eventually caught the fancy with advent of web services (the earliest version). The rationale for emergence of SOA over and above object oriented and component based software and its distinctions is nicely summarized by Thomas Earl in his articles on SOA magazine.
Service management as a discipline is about making sure that SLAs are set up. Monitored, and controlled for IT services being provided. ITIL is one of primary and best known frameworks for part-ITSM implementation. ITIL version 2 was released sometime in 1999 and contributed to operational excellence by publishing best practices in several domains – most notably these were followed to the core in many organizations in areas of service support and service delivery. A major refresh to ITIL called ITIL v3 happened in 2007 and is in process of rolling out as I write this across the world. However note that while “service” is common between ITSM and SOA – they have carried very different semantics.
What is "service" in SOA world?
It is important to first staff off by a clear articulation of what SOA is and more importantly what service in SOA is. as you may notice in debates worldwide, this is a bone of contention which never has a real right answer. I do tend to think that after all the crap we have been on service granularity debate, we are now close to a clear reasonable definition and incidentally that definition is close to what Business thinks of in terms of service.
A classic definition of Business Service as businesses perceive "is a capability (standalone or choreographed in implementation) which delivers value to consumer (internal or external)". It is important to stress the word - "CAPABILITY" - as well as the fact that it delivers value to customers - internal and external. Note that at this time we dont make a distinction between whether it is external/internal - an internal capability to procure something is an internal service being provided by Procurement function to enterprise - may not have a direct bearing on an enterprise customer if the organization is say in Government business or in my case stevedoring business.
It is also understood that capabilities have to exist in hierarchy - either standalone - atomic (executed at one time by one function) or a choreographed capability - e.g. risk assessment undertaken by banks - it is an amalgamation of series of capabilities - by the person accepting the application, by credit risk functions assessing customer credit profile, market risks etc. Also note "Value" - this is sometimes causing of heartburn between EA groups and business. Every organization has a value chain and sometimes certain capabilities are core or in context of core and sometimes some capabilities are non-core. For example, Human Resource management in classic sense is a non-core capability to an organization running stevedoring business, but rostering of human resources for planning will still be "in context of core". The decisions on sourcing of these business services are taken later on, so at this time from business perspective, all of these are services.
Interestingly there are some connotations to business definition of service which are relevant to SOA. Say my
Most competitive businesses keep defining a service portfolio but also aggressively track the relevance and value of services.
The definition of software service is no different from the business service. There is a one to one correspondence to that definition in context of automation i.e. if you tell me that capabilities, business process (end to end value capability orchestrated/choreographed) is applicable.
The key characteristics of these services are : loosely coupled, networked, well defined contract, discoverable, standardized, Interface and implementation separation, platform neutral etc - all of which are fairly straightforward to understand
SOA then is a simple architecture style wherein software is delivered through services which mimic business capabilities. Some people may want to say that given the tectonic shift which SOA is brining to add policies, framework and process within the content centric aspect as highlighted by architecture.
Note that history and birth of this movement is entrenched in software development history.
SOA's core value is "ready to integrate", "reuse" and "agility to change". Incidentally an additional benefit of SOA is business process focus and its clarity to IT staff - making it the best possible tool in arsenal to bring IT solution and business teams together like never before. Given that most SOA projects programs enforce a burden on business to own up software services, the alignment keeps on increasing.
Introduction
My own thoughts on this are not driven by practical experience on implementing ITIL but more from the perspective of SOA practitioner and commonsense management of the same. I have been a system admin and DBA in previous life and have seen some ITIL v2 implementations. I do admit to lecturing service delivery on what it means to have processes etc. in large outsourcing assignments etc - but I do claim to be a SOA practitioner. In my current capacity as Chief Architect of a large Dubai based conglomerate, I do get involved in service delivery initiatives.
Before doing the talk, I had a chance to review what was published by vendors - IBM, BMC, HP as well on blogsphere - as well as a chance to recap on what Sharon Taylor the chief architect of ITIL v3 had to speak on Business IT alignment (heard her last year). Also had put in a few SOA based systems in production and felt the heat of missing linkages of ancient practices of ITIL/ITSM being applied to SOA products. It has been interesting - so I thought let me summarize my observations and practical experiences in a now nearly inactive blog and make sure that this is somehow heard enough.