Dynamically Integrating services

[Elhanan Maayan]

hi..

we have an application that should communicate with several different sources. while one source may be in rest and json, another maybe in soap and xml. the application may employ several calls to several web services, (while one call's parameters should be based on it's previous calls results) but the end result should always be the application's own xml format.

what i want to do is find framework that would be read the his process from a file, make the transformation from one xml to our own xml, or from json format, to it's xml counterpart, and then transform that xml to our own format.

so instead of adding more conversion code, i could create another such transformation file and and upload it to the application.

i understand that camel might be the one, but i should mention we don't use spring or osgi, nor any j2ee container (it's jse)

[Arun Kumarr]

Here's my attempt to how I would approach such a problem.
Let's assume you need 'X' to solve your problem.

"we have an application that should communicate with several different sources. while one source may be in rest and json, another maybe in soap and xml."
- I will take this as a requirement that 'X' should be capable of talking to different systems having different sources of data.
'X' should also be capable of understanding different data formats.
Other implicit requirements (hidden between your requirements) are 'X' should be able to use different application protocols with ease JDBC, SQL, HTTP, TCP/IP, JMS, AMQP etc., dependent on how many systems and what protocols you have. Also since you are having to build 'X' which currently talk to 'N' number of systems with 'K <=N' number of protocols, bear in mind that the systems and protocols can be changed tomorrow or new systems/protocols can be added tomorrow. So your 'X' should also be capable of accommodating that too with ease. So extensibility is a capability.

"the application may employ several calls to several web services, (while one call's parameters should be based on it's previous calls results) but the end result should always be the application's own xml format."
- I will take this as a requirement where your 'X' should be able to strike conversations with systems and act according to the "mood" of the system. Say, systems say you've asked for func aaa() and I'm not in a position to do it or I can't do it because you've passed in wrong data etc., Now given this, 'X' should also have the capability to talk across systems and that may involve different protocols, that may involve transaction and if a system is not responding that may involve trying to talk to the system multiple times (breaking the ice...) and in due course entirely stop doing the current process and saying it can't do it because 'X' tried to communicate with a particular system 20 times and cannot proceed further. I'd recommend to read about BPMN, workflows or light weight flow frameworks.

"what i want to do is find framework that would be read the his process from a file, make the transformation from one xml to our own xml, or from json format, to it's xml counterpart, and then transform that xml to our own format. "

- Here is where I will differ. What you need to do it define the process first and then get into the details of whether the process is defined in a file or database or any other store, if you really have a preference on how the process is stored.
What's more intriguing in this requirement is you've mentioned all formats should be 'transformed' to your own XML. By this definition you will have two problems.
Your system format is 'A' and you have 'N' systems. Now you need to have transformation of 2xAxN transformation logic written down. Secondly your system format 'A' should be accommodative enough to have all the attributes mentioned in the 'N' systems. I'd recommend to read around data/format normalization. JBI is open mechanism where you will have normalization built into your framework for 'X'. Again, not a big fan of this approach. But again that's an architectural decision which has to be taken and it will affect your design and development extensively.

"so instead of adding more conversion code, i could create another such transformation file and and upload it to the application."
- Again you will end up maintaining 2xAxN transformation logic.

"i understand that camel might be the one, but i should mention we don't use spring or osgi, nor any j2ee container (it's jse) "

- Given all these capabilities, Can you analyze if camel is the best tool for this. The questions which you may want to ask yourselves is,

1. Can we do a business process retry in camel?
2. How difficult will it be? Can camel span across multiple systems?
3. Is it easy to design a flows in camel? Does it have a UI based flow designer?
4. Is Camel a simple routing framework or a full fledged process engine? (what the heck is routing framework??)
5. How does camel support transactions across systems?
6. Is it easy to plug-in frameworks to support security in camel?
7. Can camel detect service end points?
8. How does camel support legacy systems? for e.g., can I call a C-function from Camel?
9. Can I administer Camel via an administration screen? Can I control access? Can I define policies.. Can I.. Can I...?

I cannot give you a straight answer, but I'd suggest you to go read about Integration Patters (Gregor Hohpe and Bobby Wolfe), Point to Point communications, Hub and Spoke architecture, Publish-Subscribe mechanisms, queues, Problems with communication between NxM systems.
Good luck coming up with 'X'. Do pose any questions you have.

[Elhanan Maayan]

well, i should narrow the parameters
first of all my "own" xml is actually an xml format retrieved from an external system, and currently there's a bunch of existing code that reads it, compares it to our own internal structure,so i was hoping to leverage that code.
second , there are no transactions involved, all services are read only. basically we are talking about something similar to an architecture zone. get a list of buildings, for each building get the list of floors, and for each floors get the list of rooms.
all those different systems i was talking about should contain the same architecture schema, they'll probably just serve outside it in different format. and protocol but the outcome should be the same.

[Arun Kumarr]

"first of all my "own" xml is actually an xml format retrieved from an external system, and currently there's a bunch of existing code that reads it, compares it to our own internal structure,so i was hoping to leverage that code. "
-- This statement is ambiguous. your "own" xml is xml format retrieved from external system.
-- internal structure you mean - database or any sort of domain model.

I assume that all you need is a simple read requests.and all the systems currently you are talking to will send the same attributes in different formats using different protocols.
Given this requirement, you can use a simple flow framework to connect, read and transform.

Now I have some queries for you?
1. How many systems do you have connect to?
2. What is the calling mechanism (Protocol and format) for the systems you are connecting?
3. Will you have new systems to be connected to in the future?
4. Why do you think the answers to questions 1,2 and 3 are important for making a solution?

[Elhanan Maayan]

i'm making it my own, because even though it was came formatted so from a different system, since we have code that uses, it would be easier to use that code then re-factor to a new format, which would basically be the same.

currently the only protocol i know is soap and xml, future systems maybe different protocols but as these services should be public i'm guess they would be standard protocols.

i'm pretty sure there will be new systems connecting in the future, i'm just not sure what they will be, (they would be the same type of system, just different vendors)