how to REALLY monitor a J2EE-engine?

We recently started implementing an EP 6.0 project as well as a NWDI infrastructure to develop some thing for Internet Sales on CRM 2005.

I installed a NW2004s SR1 and patched the instance up to SP7 - so latest and greatest.

However, doing a search for "exception" in the log directory I get a number of 201670 (yes, you've read right).

Using the logviewer I see on each start and during runtime ~ 200 - 300 exceptions that are getting thrown, some of them often, some of them only one - so I counted about 80 - 90 different exceptions.

How should I as an administrator find out, which ones of them are crucial for the application, what is a real error and what is just an exception, because something is installed but not (yet) configured?

Setting up NetWeaver Administrator will not help much since I would get 90 mails every day with different errors and I would see there also those 90 different errors.

How to distinguish between relevant/irrelevant exceptions/errors? There's no source code around and I doubt, that it's necessary for an administrator, to learn the full J2EE API just to get those informations. It can't also be that I need to create OSS calls to get an answer from the development.

Where is some documentation what part of the engine is used where, what errors can be ignored and which of them needs to be followed?

Markus

SAP Managed Tags:
SAP NetWeaver

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Accepted Solutions (0)

Answers (1)

Hello Markus,

don't know what you see there but please be aware that Java usually shows exceptions as stack traces. That means that the throsing object hands over the exception to the calling object and this one to the next calling object and so on. This produces quite long stacktraces that include the original error message somewhere in the middle.

Though, your message could be a single error.

Try to find a real error message inside this and that should help you going forward.

Regards,

Benny

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi Benny,

thanx for your answer.

I know what a stack trace is but if you have after a fresh installation and startup 100s of them - how to deal with them? I'm NOT a developer but I'm the administrator of that box - and I don't want to open OSS calls for each of those exceptions.

Additionally it's completely intransparent (for me), what service is calling what other service, how they depend on each other and which of those services is used e. g. by an EP or by XI.

There is no documentation and since there are no sources, I can't even take one of our developers to MAYBE have an idea what's wrong.

When I set up Solution Manager Diagnostics and enable email alerting I get about 70 - 90 mails when I start up the engine.

How should I from an administrators perspective know now, which of those exceptions is really crucial and which not?

Regards,

Markus

Hi Markus,

I need to ask you a question (sorry not giving any answer for your issue),we installed solution manager diagnostics on Solution Manager 4.0 server and transferred two of our instances (XI and Portal) onto SMD landscape. If i go to http://host:port/smd/admin it is showing the Agents as started for XI and Portal and also showing the details in landscape management, but if i to smd option, it is showing empty screens under monitoring as well as reporting and unexpected error as message in configuration tab. Normally after installaing the agent in satellitte systems and do a Monitored System Setup option is there anything in specific to be done to make the reporting or monitoring option works from SMD tool ?

Thanks

Hi Markus,

I guess most of your errors are follow ups to one significant problem you have. There is no way to go through them one by one. Best would be the first one and see if solving the problem doesn't solve most or all of the rest.

Regards,

Benny

Hi Benny,

the "first" one is related to some Adobe thing, some later I see a few of logconfigurator, some of workflow, some of the portal runtime and others.

So there are so many exceptions after a fresh installation - and I have found no documentation whatsoever that they can be "ignored" or that some configuration is missing.

How should I set up a monitoring if I can't determin, whether those exceptions are somehow related? Maybe the user will get errors in the portal and they are related to the workflow. Is the only way to "monitor" the system wait for users calling that something is not working?

Markus

Hi Markus,

ok, the Adobe "thing" is the Adobe Form server, that should be connected to the java server. If that gives you any error, something fundamentally is wrong.

Lets maybe start with this: Where did you find those messages? I had quite some installations but never found error messages without a reason.

Oh yes, and what exactly doid you install?

Regards,

Benny

Hi Benny,

it's easy to "reproduce":

Use a NW 2004s SR1 installation set, install an AP Java + EP + BI (so usage type AP Java, EP, BI Java). After the installation check the logfiles.

And try to do that on a non-Windows operating system. You fill find hundreds of exceptions logged in the default trace, even if (or because?) the system is naked and just installed.

The important thing here is to NOT use Windows.

Regards,

Markus

Hi Markus,

first of all I'd recommend to adjust the level of severity of your logs in the log configuration. As I found lately, severity is on "info" and in a productive system it should more be on "error". I'm not sure if this will significantly help in your case as I expect errors (aka exceptions) to still go through, but it's worth the try.

If you stills get lots of errors, please create an oss message for support.

Regards,

Benny

Hi Benny,

thank you for the advise - but this has been done already. I still have exceptions all over - even after a new fresh installation.

I created an OSS message and was urged to create separate OSS messages for each one of the exceptions. Some of them may be related to each other but nobody from the support and/or development is (yet) able to tell, which application/service depend on what and what might be related.

I am not disposed to create a dozen OSS messages because those error messages can be reproduced very easily by just installing an instance and look into the logs after the installation - so it's not a customer specific problem but a general one. If there is something like QA @ SAP it should be done there - and not on the customers side.

You get e. g. about 34 exceptions when you gracefully shutdown a J2EE instance - because services are calling other services which are already shut down. So you get another exception.

I really wonder how people deal with that or if it's just not of interest by developers. An exception is in my understanding what is is - a programmatically error that should either be catched defined or not thrown at all.

Stop your instance
delete all "defaultTrace.trc" and "application.log"
start your instance
wait for the startup finished
stop your instance
search for "exception" in defaultTrace*.trc

If that doesn't convince you, that there is certainly something wrong you can call me dumb.

Before we're not able at a customer side to distinguish between "startup and shutdown" exceptions (no matter how) and real errors it would be grossly negligent to take such system into production with some thousand users - because you are just unable to monitor the system.

Markus

Hello Markus,

I certainly will not call a customer something like "dumb". I would prefer less insulting terms ... but that's another story and does absolutely not apply to you. Meanwhile I follow this up through "somethin like QA @ SAP", as I heard this story not only from you.

This might take some time, as it's still not clear what happens why. However, you can help my efforts by posting some of your exceptions as OSS. That will put some pressure on this issue from the other side.

Regards,

Benny

Hi Benny,

> I certainly will not call a customer something like

> "dumb". I would prefer less insulting terms ... but

> that's another story and does absolutely not apply to

> you. Meanwhile I follow this up through "somethin

> like QA @ SAP", as I heard this story not only from

> you.

I really hope people don't get me wrong here, I don't wanna offend someone, I'm just not one of those "everything-is-(becoming)-good"-guys.

Certainly, that whole topic is quite complex and SAP was kind of urged to push out a (Java-) product to compete on the market. For the customers, this is mainly a Good Thing (TM), although one should have waited long enough to have an environment, where you can run business critical applications on a complete new platform. ABAP is really mature - grown over the years and I'm certainly aware of the fact, that Java needs that time too. But on the opposite side, one should really try to concentrate on stability/manageability instead of adding components to components and layers on top of layers on top of layers.

> This might take some time, as it's still not clear

> what happens why. However, you can help my efforts by

> posting some of your exceptions as OSS. That will put

> some pressure on this issue from the other side.

I already did that, in fact there is one call open. I have also heard, that there have been created "task forces" to specify messages, in sense of what is really an error and what not.

I don't have a general problem with creating OSS messages but:

We are a certified SAP Competence Center - and we MUST NOT create more than a specific amount of OSS messages through the year; that number of created messages is a performance figure for recertification procedure. In fact this means, that if we create too many messages (whether entitled or not), we risk our certification (and eventually means paying $$$) - thus I need to think several times, if I create an OSS message or not.

So you see it's not an "I don't want" - but more an "I cannot". That this approach is counterproductive is obvious, but those byelaws have been created by SAP themselves, so effectively this produces less feedback, less error messages, less real-life experiences - just by design, especially when companies are big enough to have certified competence centers (> 1000 users).

That's also a reason why I try to communicate through SDN - because feedback is apparently inappropriate in the support channel (as I was told so several times in the OSS) and there is - DSAG left aside - no other way to post stimuli and to ask about design/general decisions.

If it's necessary to be a J2EE developer/specialist to drive an EP - pretty ok for me - but one should communicate that clearly - and really educate people in the training at ESZ (Building 5?) on course "J2EE Administration I && II" to do so.

Finally I have two very basic questions:

- How can I find out who is logged in in the portal if there's more than one server process?

(analog SM04/AL08 in an ABAP system)

- How can I find out if the portal is running (not only the successful start of sap.com/irj)?

(analog SM50/SM66/ST22/SM21)

Respectfully,

Markus

Hello Markus,

I don't feel insulted, thank you

Over the last week I ran around and tried to find out what happens here. Result: problem is knwon and under work (as I expected, as usually such a thing should have found it's way through to development already).

The only addition I will take care is that this goes into the manual to solve the current situation, as the issue with error messages cannot be solved in this release. Actually we will define a whole new level of QM and new rules to solve this problem.

You see, we very well do know our customers problem even if you cannot see immediate reaction, like it is always with big ships: it takes some time from turning the wheel until the ship starts turning itself.

As I'm not a specialist for your questions I'll try ma very best:

> Finally I have two very basic questions:

> - How can I find out who is logged in in the portal

> if there's more than one server process?

> (analog SM04/AL08 in an ABAP system)

There is an "user activity report" you can access as a admin. But don't ask me how to get there...

> - How can I find out if the portal is running (not

> only the successful start of sap.com/irj)?

> (analog SM50/SM66/ST22/SM21)

Well, if the irj is not enough, then the question is what you define the portal to be. It consist of many programs that may or may not run currently.

Regards,

Benny

> Hello Markus,

> I don't feel insulted, thank you

> Over the last week I ran around and tried to find out

> what happens here. Result: problem is knwon and under

> work (as I expected, as usually such a thing should

> have found it's way through to development already).

In this case one should communicate so officially, that would keeping people from bothering with the support

> The only addition I will take care is that this goes

> into the manual to solve the current situation, as

> the issue with error messages cannot be solved in

> this release. Actually we will define a whole new

> level of QM and new rules to solve this problem.

Cannot be solved in this release? Wow.. that's an interesting statement. That means in fact, that customers will be unable to run productive environments in the "current" version (which was released just two months ago) and need to wait until 7.10 ff is coming out. Great news! 😕

> You see, we very well do know our customers problem

> even if you cannot see immediate reaction, like it is

> always with big ships: it takes some time from

> turning the wheel until the ship starts turning

> itself.

YOU know maybe, why does the support not?

> As I'm not a specialist for your questions I'll try

> ma very best:

> > Finally I have two very basic questions:

> >

> > - How can I find out who is logged in in the

> portal

> > if there's more than one server process?

> > (analog SM04/AL08 in an ABAP system)

> There is an "user activity report" you can access as

> a admin. But don't ask me how to get there...

Yes, this report is on a "per server" basis and logs everything, what a user has done; however, it does not tell me, which user is NOW logged on. I assume there is no such functionality (as of now).

> >

> > - How can I find out if the portal is running (not

> > only the successful start of sap.com/irj)?

> > (analog SM50/SM66/ST22/SM21)

> Well, if the irj is not enough, then the question is

> what you define the portal to be. It consist of many

> programs that may or may not run currently.

Yes - but due to the intransparency actually existing, I even don't have a clue what programs/applications are used, and for me there's no way to find out without hiring a SAP-Java developer for each of the sap.com/*-components - because the general support component for Java (BC-JAS) doesn't know either.

I don't know if it's a good idea to "extend" the Java 5 standard to e. g. enable monitoring in a convenient way (of course other things matter too). Let's see what will happen. Since the actual newest J2EE server installation CD downloadable here is (again) Windows only I can't do a test installation.

Thank you though for your efforts - I appreciate it.

Markus

Hello Markus

> > The only addition I will take care is that this

> goes

> > into the manual to solve the current situation, as

> > the issue with error messages cannot be solved in

> > this release. Actually we will define a whole new

> > level of QM and new rules to solve this problem.

> Cannot be solved in this release? Wow.. that's an

> interesting statement. That means in fact, that

> customers will be unable to run productive

> environments in the "current" version (which was

> released just two months ago) and need to wait until

> 7.10 ff is coming out. Great news! 😕

What cannot be solved is to guarantee that this kind of problems comes back after each SP. To assure it is solved will need some manual action that will be described in the Technical Operations Manual.

> Thank you though for your efforts - I appreciate it.

You're welcome.

Regards,

Benny

Hi Benny,

[...]

> > Cannot be solved in this release? Wow.. that's an

> > interesting statement. That means in fact, that

> > customers will be unable to run productive

> > environments in the "current" version (which was

> > released just two months ago) and need to wait

> until

> > 7.10 ff is coming out. Great news! 😕

> >

> What cannot be solved is to guarantee that this kind

> of problems comes back after each SP. To assure it is

> solved will need some manual action that will be

> described in the Technical Operations Manual.

In ABAP this is very well possible (to some extent), why not in Java?

Regards,

Markus

Markus,

because ABAP is about 15 years older then Java and processes for ABAP have been set for a long time. And just to answer the next obvious question: Not all processes can be just copied into the Java world....

Regards,

Benny

Hallo Markus,

don't know if this of any interest to you:

With regards to finding out who is logged on -

1. Start Visual Administrator

2. Navigate to Server (n) (n) > Services > Security Provider

3. In the resulting tray on the right select Runtime > Login Sessions

As for how to REALLY monitor a J2EE Engine ... If you have details, I would appreciate any comments.

Cheers,

Wolfgang

Just as a follow up:

We have meanwhile a Federated Portal Network, one consumer and four producer portals.

As for "monitoring" goes, we are on the way implementing SolManDiag, although that thing itself gives us as much trouble as we have with the portals - but we're slowly approaching OSS call by OSS call. Our major pains are the dependencies between the installed JDK, Wily, ST/A-PI, ST_PI and, which is a MAJOR thing, SAP_BASIS in the satellite systems.

We still have problems with portals working and suddenly just showing up blank pages in the browser without any errors in one of the logfiles (which portal?), we see quite a lot of "error 500" using XSS applications on the first time and then no more - and other non-deterministic behaviours I really can't explain logically.

In case of a real problem, due to real troubleshooting appearing to be too cumbersome and lengthy with OSS, we iteratively restart all affected instances (producer, consumer, SLD...) until the application appears "working" again. This is totally unprofessional but it's the only way we will get a working system back in time.

I recently gave a lecture at a workshop @ SAP in Walldorf about "Experiences and Management (Problems) of Java applications", I was told, that some engineers were there, I didn't know all the people so I can't state, if the mentioned problems will reach someone, who understands the customers (or the administrators) pain...

So - as far as real "monitoring" goes - I'd say more "keeping alive" than "monitoring".

I very much appreciate all the feedback I get here and I will keep updating that thread as soon as we get better

Markus

Hi Markus,

I can understand your problems. We are just currently going live with XI and the XI Rosettanet adapter.

After much problems, (for exemple it took 2 weeks to understand that an error was functional and not technical...), the pre-production tests were OK with our international partner.

Now, we've been trying for 5 days with no success to connect the 2 production systems together. Nobody understands anything in the Java error messages.

I am not (yet, I hope) an XI specialist but I am very afraid when I see a nationally known specialist scratching his heads and saying, he can't see the difference with the working pre-production environment and don't know how to interpret the error messages.

Well, I guess we wil have to wait for SAP Rosettanet team to connect in our system...

And next, we will have to give this production system to the exploitation contractor.

I'm supposed to write a detailed documentation with nearlly all possible error messages and how to react to them. The system should also be monitored by Patrol...

We have tried to implement XI monitoring mails but we don't know how to interpret the error messages. I already know that the contractor will not be able to react to any errors...

IMHO, the J2EE systems (not only SAP ones, but also weblogic or websphere) are completely immature and can't be monitored efficiently and proffessionnally.

SAP did add a completely unnecessary (IMHO) complexity with the dual stack systems which makes it a mess (I can't find an other word).

I hope that this situation will get much better in the coming years. I think that SAP as a company has the resources to do it.

The question is how long will we have to wait ?

Regards,

Olivier

I can understand your problems. We are just currently going live with XI and the XI Rosettanet adapter.
After much problems, (for exemple it took 2 weeks to understand that an error was functional and not technical...), the pre-production tests were OK with our international partner.
Now, we've been trying for 5 days with no success to connect the 2 production systems together. Nobody understands anything in the Java error messages.

What I´ve heard is, that an upgraded system (means, installation with SR1 or SR2 and upgraded to SP 12) is NOT the same as installing SR3 which includes SP12...

I am not (yet, I hope) an XI specialist but I am very afraid when I see a nationally known specialist scratching his heads and saying, he can't see the difference with the working pre-production environment and don't know how to interpret the error messages.
Well, I guess we wil have to wait for SAP Rosettanet team to connect in our system...

I wish you good luck! Once in production, there´s no way back, especially not for an XI/PI system...

<...>

We have tried to implement XI monitoring mails but we don't know how to interpret the error messages. I already know that the contractor will not be able to react to any errors...

Isn´t that negligent (to some extent)? I know, you can´t do anything against...

IMHO, the J2EE systems (not only SAP ones, but also weblogic or websphere) are completely immature and can't be monitored efficiently and proffessionnally.

"Webs Fear"?

Well - Java on the server is by (my) definition a GoodThing (TM) because there are much more Java developers in the wild than there are ABAP developers. It´s just a major pain to run those kind of applications and keep them running.

A developer @ Sun once said to me "Each administrator for a specific J2EE application needs to have a corresponding developer on-site to help the admin keeping the system alive..." - that speaks for itself.

SAP did add a completely unnecessary (IMHO) complexity with the dual stack systems which makes it a mess (I can't find an other word).

I just heard on the mentioned workshop that all those argumentations of "using shared memory together" and other pretended technical advantages of a dual stack don´t exculpate the additional amount of administrative tasks needed. Nowadays basis consultants tell to NOT install dual stacks if avoidable...

I hope that this situation will get much better in the coming years. I think that SAP as a company has the resources to do it.

I´m sure it will. The next generation Java engine is developed by SAP (based on the standard for J2EE 5.0) and what I saw as preview was really auspicious. I can´t say, however, how it will be handled or if problems will be easier to locate and solve.

The question is how long will we have to wait ?

Well... I wouldn´t hold my breath.

Markus

Hi Markus,

Apologies, this thread is old but worth a mention.

I was in ramp up for 630 J2EE and starting from the beginning it was clear that ERROR means there is some type of error that can even be of user cause and so ERROR was a rubbish thing to monitor (a bit like looking for red messages in SM21).

However, right from the start I have been alerting on CRITICAL errors as these mean trouble. Seems to work well..

What do you think?

Apologies, this thread is old but worth a mention.

No problem with that!

I was in ramp up for 630 J2EE and starting from the beginning it was clear that ERROR means there is some type of error that can even be of user cause and so ERROR was a rubbish thing to monitor (a bit like looking for red messages in SM21).

We don´t have many "red messages" in SM21 - and if, then there is really a problem (like a dump)

However, right from the start I have been alerting on CRITICAL errors as these mean trouble. Seems to work well..

There are less of them - true - but they are even less speaking:


#1.5^H#00118513360B002E000000250000399900044B606D5EC8B1#1208777430714#com.sap.engine.services.failover.st
orage#sap.com/irj#com.sap.engine.services.failover.storage#Guest#0##n/a##55b6fe700f9611ddbf4800118513360b
#SAPEngine_Application_Thread[impl:3]_31##0#0#Error##Java###null
[EXCEPTION]
 {0}#1#com.sap.engine.services.failover.exceptions.PersistentStorageException: Exception occurred during
I/O operations.
        at com.sap.engine.services.failover.storage.DatabasePersistentStorage.serializeSession(DatabasePe
rsistentStorage.java:123)
        at com.sap.engine.services.servlets_jsp.server.container.ApplicationThreadDestroyer.serializeSess
ions(ApplicationThreadDestroyer.java:104)
        at com.sap.engine.services.servlets_jsp.server.container.ApplicationThreadDestroyer.run(Applicati
onThreadDestroyer.java:72)


#1.5^H#00118513360B00070000000100006B1100044D3854843446#1210804238299#com.sap.engine.services.security.po
licyconfiguration##com.sap.engine.services.security.policyconfiguration####n/a##60ab1ab0220511dda5f700118
513360b#SAPEngine_System_Thread[impl:5]_67##0#0#Warning#1#/System/Security/PolicyConfiguration#Java###Exc
eption occurred during remove of policy root [{0}].
[EXCEPTION]
 {1}#2#service.iiop#com.sap.engine.frame.core.configuration.NameNotFoundException: A configuration entry
with the name "service.iiop" cannot be found in the configuration "security/configurations".
        at com.sap.engine.core.configuration.impl.ConfigurationDataCacheAllImpl.getValue(ConfigurationDat
aCacheAllImpl.java:98)
        at com.sap.engine.core.configuration.impl.ReadAccessConfiguration.getConfigEntry(ReadAccessConfig
uration.java:849)
        at com.sap.engine.services.security.server.PolicyRoots.removePolicyRoot(PolicyRo

I gave up on Java.

We run the instances and in case of a problem we restart all affected Java systems subsequently until the error vanishes because trying to "find" an error in a federated portal network (with some scenarios 6 or 7 machines affected) is not only cumbersome and very time consuming but in most cases meaningless because the support needs a reproducibility, a case, otherwise they can´t help - and recommend to "install the latest support packages". This is also true for production system. In case I open a prio-1 call I don´t only got hold off sleep because they are working on the call 24 h but we finally end in restarting the engines though.

There´s one call open since June 2006 - which is now two years old - about the dependencies in the packages and how to find out which service/applications needs which, to monitor only the really necessary ones. Since the development itself is not able to tell which dependencies they generate then, well, it´s better for everyones nerves and finally health to "just reboot" as you were (and maybe are) doing with Windows boxes if they don´t behave as they should. Maybe it´s by design/defnition like this, I don´t know.

We´re now in the process of putting all Java instances onto clusters to do an automatic switchover so the instances get restarted every few days and we avoid any problems that are caused by instances running too long.

I wish there was another solution or an idea of how to deal with that - I don´t know how others handle that but I have to take of about 50 instances (ABAP + Java) so I don´t just have the time of collecting dozens of logfiles of half a dozen machines and open an OSS call try to explain someone at the other side of the world in a foreign language (english is not my native language) what´s happening. The problem starts with "in which component do I open the call?"

Markus

lol, I looove your attitude :)... man, you have been damaged by J2EE

I used to run the OSS Basis team in the UK many years ago and if you want to go offline into E-Mails I can explain why you get such poor responses and perhaps some tips on how to get past them. Sorry my German is too poor to replicate.

I'm part of a big organisation and we have plenty of J2EE's out there. 1 customer has about 100 SAP systems and another 80. I wish we were back in the days of 3 system landscapes but I guess it would be dull

There is an art to reading a J2EE error and I'm afraid to say that critical ones ofte do need consideration but not always. However bouncing the systems is a practical solution to most issues when time is short.

We use Central CCMS to monitor J2EE's and tend to alert on threads and security sessions and don't even bother with the logs any more until there is an issue. However our engines are stable and if they do fal over I will tuck in there and find out why to avoid a recurrance so we do not need to restart.

As for your comments on applications requiring other applications. There are references in place called hard and soft dependencies and you could spend time yourself to investigate all of these to get a good map of not only what required what but what order things truly start in. The trouble is these dependencies can change between SP's so it is a long and pointless exercise. Java dependencies are a nasty but necessary evil. It is a different beast to ABAP.

I had a call open for 3/4 of a year where I was arguing that CCMS should be able to monitor J2EE backups that are using BACKINT. There final response after lots and lots of chasing was .... no we do not have the ability but perhaps we will think about it in the next release...lol

So I offer you this, some help where I can (I am not a guru but a simple Basis person like yourself but I have worked a lot with J2EE and NWDI and monitoring (I even braved Sherlok in the early days) perhaps I can help were you are getting stuck with bad OSS design.

I will answer you off list since my post contains "forbidden words" - although I can't find one. SDN team doesn't reply.

Markus

hey i stumbled over this thread after having on on this bulls... buggy ...j2ee.... motherf... damn software problems...

can you please explain what "forbidden words" are?

China rulez

Roman

Ask a Question

Top Q&A Solution Author

User

Count