External Input Adapter Example/Howto

[Former Member]

Hi,

We are making good progress in our project. In order to reduce the number of layers in our architecture now, we would like to make use of an external input adapter to publish data from an external source into our ESP project.

First of all, there is the question of whtether to use internal or external custom built input adapter for this. I do not really have a concrete fundament to make this decission, as I do not know the exact difference between both.

As I understand, an external adapter uses the pub/sub methods to publish data to ESP. What we need now is an end to end example for an external input adapter, meaning:

     - cnxml file

     - sources of the adapter itself (in the install directories there are only compiled versions of this) best would be an example for C code as well as Java code

     - Is there any information abour how many events per second a single input stream can take using these kinds of adapters? The events are rather small and have only 4 fields. Target is an Input Stream, not a Window.

     - What is the best way to implement this as a high-speed connection? At the moment we use socket connections for this.

What we have right now is a c - tool that can take an event and publish it with pubsub api to ESP. However we struggled getting this integrated into ESP model as an adapter, so that we do not need to start this tool ourselves. Probably its just a mistake in the cnxml file. The question was which methods need to be implemented in order to get an external output adapter publishing data to ESP from ESP server start on.

I hope i didn't make it too confusing, I am kind of missing the right words today I am sure that the problem we have is a rather simple one in this case.

Best regards,

Dave

[vijaigeethach · Accepted solution]

Hi Dave,

There are 2 types of adapters

1.Internal adapters

2. External adapters

Internal adapters run as a sepearte thread in the esp_server process itself, whereas external adapter are a separate process by itself. It is only the external adpaters that use the pub/sub api interface.

From reading your email I guess you are trying to write a custom internal adapter.Creating a cnxml file is needed only for a custom internal adapter and is not needed for a external input adapter using the pub/sub api.  I am not sure whether you have ESP 5.0 or ESP 5.1 the custom internal adapter documentation is much better in ESP 5.1

Please see http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc01615.0500/doc/html/swa12...

Custom internal adapter can be written only in C/C++ and the sample available in the section "Sample Custom Internal adapter Implementation" should be helpful for you.

As for the speed  part of your question, I will leave it to the others who are using this forum.

Thanks,

Geetha.

[Former Member]

Ok we made it fly now. External adapter being started with esp project works now -great!

I only have a few questions left regarding the modelling and the configuration:

- in the cnxml file (i have used PLUGIN_TEMPLATE as template) it seems as if i have to (mandatorily) set a node for <Library file="simple_ext" type="binary"/>. If I leave out this node, i cannot compile my model, but i am not making any use of simple_ext lib at all as I have an external adapter. Is there any way to get rid of this node in the cnxml?

- In my model it seems as if I have to attach the input adapter (my self written external adapter) to one specific stream. This is however semantically not correct since the adapter servers more than one input stream. Is there a workaround to model this or do i have to live with it? At the moment i have randomly picked on of the streams and attached the adapter to it as input adapter.

Thanks a lot for the fast and competent responses! You really helped us out here.

Best regards,

Dave

[Former Member]

Hi Dave,

Since you mention the PLUGIN_TEMPLATE.cnxml file, this makes me think you are using ESP 5.0 (this file is no longer included with ESP 5.1 ).  If so, you should really consider using ESP 5.1 which was released this month.  I think there are litterally thousands of bug fixes in ESP 5.1 that are not available in ESP 5.0.

Also included with ESP 5.1 is a complete end-to-end external adapter example program located in $ESP_HOME/adapters/sample.  It is a Java example but it shows you everything you need to complete an external adapter including discovery and the cnxml file.

Two answer your questions:

1) I believe that you must leave the 'simple_ext' part in.  The ESP 5.1 docs do a better job of explaining it here:

http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc01615.0510/doc/html/swa1309871877907...

To summarize, it says that 'simple_ext' must always be used for custom external adapters and that simple_ext has a function that parses this cnxml file and looks up the internal fields to find particular commands and their default values.  So I think it makes the adapter use $ESP_HOME/lib/adapters/libesp_adapter_simple_ext_lib.so behind the scenes.

2) The documentation states that a "custom external adapter is essentially any application that publishes or subscribes to a stream or a set of streams in Event Stream Processor." so I understand that you should be able to connect it to multiple streams.

If you can provide more detail about problems you are having with this, we can look into it further.

Thanks,

  Neal

[Former Member]

Hi Neal,

Thanks for the explanation. We are currently upgrading to 5.1.

to 1)

Setting simple_ext in the cnxml makes esp use libesp_adapter_simple_ext_lib.so indeed. We compiled our adapter as libesp_adapter_<name>_lib.so and put it in there, but this caused error during server execution. If simple_ext has to be set there, thisi fine for me.

to 2)

Of course I can model the Adapter multiple times into my ccl and then have it connected to all the streams it is writing to. I was just wondering about the freenes i have there. It actually makes no difference whether i model it to one or all streams. As long as it is connected to one stream in the model, it will publish to all streams. I guess this playes a bigger role, when we start using schema discovery.

Thanks again for the help. I think this topic is pretty clear now to us

[Former Member]

Thanks a lot guys!

That really helps us making the decision between external / internal adapter. As I understand now we were using external custom adapters all the time. Our goal is to go for internal adapters to have them started/ended together with the ESP process as this cleans up our landscape.

Thanks for the documentation and links, I will go ahead and dive into internal adapters now.

Best regards,

Dave

[Former Member]

I have just found a sentense in the document Jeff provided that made me struggle:

"ESP does provide optional mechanisms where the lifecycle of an external Adapter may be managed by an ESP project, if the appropriate adapter configuration file (.cnxml file) settings are included within the project configuration."

as well as

"Can optionally be started and stopped by the project if the adapter is configured to be under project control (and the adapter is deployed on the same machine as the project)."

So it is possible to have external adapters started and stopped together with a project - thats exactly what we want!

[JWootton]

Yup. Here's where this is described in the product doc:

http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc01615.0510/doc/html/swa1309871837780...

and within that chapter, specifically:

http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc01615.0510/doc/html/swa1309872041538...

[JWootton]

Dave - I can add some things to Geetha's reply. First let me point out this new white paper on ESP integration options that might be helpful. (by the way, a plug: this lives in the new ESP Developer Center that was just launched today! on SCN - in fact it's the new home for technical discussions like this one).

The choice between building an internal adapter or external adapter depends on what's important to you. There's no one right or wrong answer (that's why we offer the choice). In terms of performance, while internal adapters have a very slight edge, it's not significant.

Some of the key differences:

- internal adapters have to be written in C/C++; with external adapters, Java and .NET are options

- internal adapters run as part of the ESP project; external adapters run separately, which gives you more deployment flexibility, e.g. they can run on a different machine, but also means that you have to be responsible for starting them, for recovery, etc

- a single external adapter can publish to multiple streams/windows and/or subscribe to multiple streams/windows - and in fact a single external adapter can both publish and subscribe; every internal adapter is bound to a single input stream/window or output stream/window

External adapters all connect to the ESP project using a socket. In terms of how the adapter should connect to the external system...well that depends.  A discussion might be best.