Did you ever wonder to know the secret sauce when you come across a world-class solution? Does it make you desire to understand the thought-process that would’ve gone behind the scenes until the solution was implemented, especially when the organization is world’s second largest data center?
Well, the datacenter was considering getting an enterprise ITSM system along with Cloud Mgmt, CRM, ERP, Facility Mgmt etc. The leadership had spent decent amount of time in shortlisting the products in respective categories. One of the success criteria was to ensure that the tools communicated with each other in a seamless way. Considering the criticality of hosting infrastructure like VMs (virtual machines), VPCs (virtual private clouds), Colocation, databases, etc. for major customers, it was necessary to ensure that not a single transaction should be lost. Traceability was another parameter that had to be considered.
The systems that were shortlisted were tussom (ITSM, ITAM, Discovery & Project Mgmt), Apiculus (Cloud Mgmt), CRM (Salesforce), SAP (ERP), Facility Asset Mgmt and RFID. Although every tool had its web services/APIs, it was necessary to define individual as well as integrated processes.
Kafka, a distributed streaming platform, was decided to be implemented to consider a seamless integration between the other enterprise systems. The important aspect to consider was fail-safe transactions apart from integration. Kafka provides a unified, high-throughput, low-latency platform for handling real-time data feeds.
API Endpoint, Request Payload, Success Response Payload and Failure Response Payload were defined for every integration between source and target application/tool/system/solution. The downstream system fields, business rules, destination system & field names, data type, length and possible values were defined as a part of mapping source to destination. This was defined for every possible event of creating, updating etc. related to the respective systems. This ensured that owners of respective systems to review the details to every field mapped and corresponding event considered.
Post this, necessary topics and partitions were created for respective systems, which helped the corresponding system owners to read and write only in allocated topics and partitions. JSON was agreed as common data interchange format.
Sounds simple and straightforward right? Guess what? Some systems did not have a mechanism/connector to be integrated with Kafka. Those applications ended up using Kafka web services and letting their web services be leveraged by Kafka. Several integration tests were carried out post individual system testing and finally, the solution was deployed to Production that helped complete end to end automation –
- Track asset lifecycle from the time it is procured to making inward entry at gate (SEZ option was also considered) to getting connected to network and discovered to setting it on repair mode (if required) to movements between racks and floors or outward entry etc. Its warranty, PO, tickets raised for the asset, relationship with other assets etc. was tracked and maintained.
- Users could raise PR (Purchase Request) without having a need to log into ERP.
- Accounts and contracts were created into ITSM tool as soon as the leads were qualified in CRM as customers and contracts were signed.
- Necessary Implementation request, tasks, Deactivation requests etc. were automatically created based on the line items of products selected by customers.
- VMs were automatically provisioned into Apiculus based on the data gathered in contracts.
- ERP tool could generate automatic invoice based on the usage of assets/resources.
- RFID helped to track movements of asset inward, outward and between racks & floors, automatically.
- External customers could use Cloud Mgmt tool to place and update their infrastructure requirements.
- ITSM tool was used to support and service internal as well as external customers. It also offered additional service of MRQ (Material Request) and MIT (Material Issue Task) features. Additionally, a common Communication module was developed that was leveraged by other systems to send notifications based on customer, event, activity etc.
You may be wondering why web services of respective tools could be used and the above integration could be achieved. The difference is that without using system like Kafka, it would get difficult to identify and maintain failed transactions and to call web services again. For instance, if ITSM system would simply call web services of ERP when it is down, it would have to wait and try again until it succeeds. Also, if it happens for multiple transactions, it will get even more difficult to maintain and call web services only for those failed transactions. Kafka takes care of it by maintaining the transaction and helping identify if it was completed. Hence, in case the destination system is down, it can pick up the transaction whenever it is up.