Afaria Services don't start after server reboot

[Former Member]

Hello experts,

Since Afaria 7 SP7 two of our Afaria instance have a problem after restarting the Windows Server where Afaria components are installed. Most of the time they are not able to start - even when set to "Automatic". I can see that behavior on

• Afaria Server service
• Afaria Backend Portal Server service
• API Server service.

When I manually start the Afaria Server i get following error

The Afaria Backend Portal Server service and the Afaria API service can be started manually.

The Windows Event Viewer show following Erros:

XRS4941: Service: The scheduling COM component of the  Afaria service failed to start.  EXPLANATION: The scheduling component failed to start. The Afaria service will run, but any scheduled activities will not run. ACTIONS: Stop and restart the Afaria service. If the problem persists you may need to reinstall the Afaria software. If the problem still persists, contact your support representative.

XRS4940: Service: An unexpected error has occurred in the following Afaria component: XSOutboundServer.exe. The service is stopping.  EXPLANATION: A fatal error occurred that could not be handled. All communications are disabled.  ACTIONS: Close all other Afaria applications, then attempt to start the Afaria service again. If this error persists, try shutting down the machine before restarting the service. If this error still persists, contact your support representative.

And a lot of others ...

I did and update to SP8 today, but that did not solve the problem. The instance is online since about 20 months now, and did not have that problem till we installed SP7.

Could you please be so kind and help me out troubleshooting the problem?

Thank you a lot

Ernst

[Former Member · Accepted solution]

Hi Ernst Muellner,

There are two solution:

1. Kill all the background services:

XS services process from task manager and then try to start Afaria Server Service.

2. Do sequential restart as follow:

--> Database Server.

--> Afaria Server.

--> Afaria Admin Server.

Regards,

Subash Acharya

[Former Member]

For error XRS4941. Go to folder ..../AfariaServer/Bin, cmd here.

regsvr32 XSPath.exe - it reregister COM object.

If it not work, try launch XSOutboundServer.exe manually.
It report a error with COM CLSID. Search it in registry - take name of object that is marked for deletion (In my case it was XSPath.exe). And type in CMD (cd to dir with object from registry) "regsvr32 name" ('name' - name of object).

Than try start server - it's ok for me.

[former_member186998]

Hello Ernst,

> I also tried to set the service to a delayed start, but that did not halp at all.

Is this service only API service?

Please change [Startup Type] of the following two services.

• Afaria API (Automatic -> Automatic(Delayed Start))
• Afaria Backend Portal Package Server (Automatic -> Automatic(Delayed Start))

Thanks,

Atsushi

[Former Member]

Hello Atsusi,

Thank you for your answer.

The server that I am troublesooting at the moment holds only the Afaria API service. Its only role is the Administrative Console. I also  changed the service startup mode to "Automatic (delayed)", but it did not help.

br

Ernst

[jtaylor]

Ernst,

Since nothing else seems to be resulting in any useful information for troubleshooting with the API, I have one thing that we could try...

If you are able to install DebugDiag (Microsoft utility, current version is 2 update 2 I believe):

1. install on the API server
2. add a "crash" rule
   1. for "a specific NT service"
   2. for the Afaria API service
   3. Action type for unconfigured first chance exceptions: Log Stack Trace
   4. Action limit for unconfigured...: 20 or so
   5. Maximum number of userdumps: 1
   6. Specify a place to save the dumps on next page and save the rule.
3. Restart the server
4. open DebugDiag after logging in and noticing that API service did not start on it's on.
5. disable the crash rule that you created in step 2.
6. check/clean the text/log files for any information you would not want to post on the forum, and share the file here  (or create an incident in service marketplace with the file)

After doing that, I would be curious if you set the recovery options for the API service to "restart the service" for first/second/subsequent failures, if the service would eventually start without manual intervention. It would be best not to have these recovery options set during the log collection above because it might generate too much data.

[Former Member]

Hello John,

I went through the KBA1851170 and set the log level to "max". But the only line I am getting in the log file after a server reboot is:

<<<Service Start: 03.12.2015 13:38:05>>>

Not more ...

In the Windows Event Logs I can find the two following lines where API service fails to start.

A timeout was reached (30000 milliseconds) while waiting for the Afaria API service to connect.

The Afaria API service failed to start due to the following error:

The service did not respond to the start or control request in a timely fashion.

When I start the API server manually, then it starts without any problems and the log file gets full of information.

br

Ernst

[michael_loop]

It sounds like you are only checking the Application Event log. In cases like these, there is often a useful error in the System Event log at the same time (or in this case, within 30 seconds.) Note that it may be logged as an Informational event rather than an error, if for example the source is "Application Pop-up".

Please have a look and let us know.

Mike Loop - SAP Product Support

[jtaylor]

Out of curiosity, if you set the startup of the API service to delayed, does it start without problems? When you restart that server, do you also restart other servers, or just that server?

It sounds kind of  like there's some resource that the service relies on, which is not available initially, but is available when you manually start it... I'm wracking my brain to figure out what that might be, but I'm really not sure, unless the server is underpowered and the service somehow tries to start before network resources are available.

[Former Member]

Hello Michael,

Thank you for your answer. I just checked the Application Log at the same timestamp where I get above two errors in the System Log. Indeed there is a warning from the User Profile Service.

Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards. 

DETAIL -

2 user registry handles leaked from \Registry\User\S-1-5-21-1501905483-372659036-2377019522-1116:

Process 1280 (\Device\HarddiskVolume3\Afaria\AfariaApiService\Bin\AfariaServiceHost.exe) has opened key \REGISTRY\USER\S-1-5-21-1501905483-372659036-2377019522-1116

Process 1280 (\Device\HarddiskVolume3\Afaria\AfariaApiService\Bin\AfariaServiceHost.exe) has opened key \REGISTRY\USER\S-1-5-21-1501905483-372659036-2377019522-1116\Control Panel\International

It seems that Afaria API wants to do something in the registry during startup of the server and is not able to do it.

How can I solve this?

Remember - I have exactly the same problem on two Afaria Instances (Demo and Test). The productive system works well (but is still SP5).

Thank you for your help - i really appreciate.

Ernst

[Former Member]

Hello John,

I am only restarting the server that holds the API Server (Admin Console). The Server holding the other roles stays online with all Afaria Services running. Also the DB Server stays online during my tests.

The resources I am having for the Admin Console Server are two CPUs and four Gigs of memory.

That should be sufficient for a Admin Console.

br

Ernst

[michael_loop]

Leaked registry handles is a common message in the System Event log, and in my experience they have always been benign. According to Microsoft's KB 947238, this is a housecleaning step when a User Account is closed but registry handles remain open.

Are these the only System messages relating to Afaria within 30 seconds of the Application Event error?

Does the service account have the following permissions on the API server machine: "Log on as service", "Interactive Logon"?

Regards,

Mike Loop - SAP Product Support

[Former Member]

Thank you for your answer Michael,

The infrastructure guys told me the Service Account has sufficient rights to "Log on as a service" and "Interactive Logon". And as I wrote in this thread already the instance is online for nearly two years already - and it was working well till my problems started.

System and Application log looks like this ... the problem occurred exactly at 15:17:00

[Former Member]

Hello everybody,

First I want to thank all of you trying to help me solving the problem.

I am still troubleshooting the issue. On the Afaria Master Server (with the roles Afaria Server, Enrollment Server and Package Server) I was removing two lately applied .NET patches (KBA3097996 and KB3098781). Now all the three services are starting up as usual directly after a server reboot. At least they did the last couple of times i rebooted (i am doing this for testing purposes only of course).

But the API service installed on a different virtual machine still makes me sorrows. Also there I was removing the two .NET patches, but still the service wont come up after a server reboot. When I am starting the service manually after logging on, it starts without any problem. I also tried to set the service to a delayed start, but that did not help at all.

The following two error messages I am getting in the Windows event log

• A timeout was reached (30000 milliseconds) while waiting for the Afaria API service to connect.
• The Afaria API service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.

Not more - not less.

Any more ideas?

Thanks a lot

Ernst

[tracy_barkley]

Ernst,

I am glad to hear that removing the .NET patches as John suggested got the server started.  The API timeout sounds very much like a blocked port. Is port 135 open to the Afaria server?  This is the standard DCOM Port we use for communication betweent the two.   If that were blocked, the error you mention can occur.

Tracy

[jtaylor]

Ernst,

If Tracy's recommendation doesn't produce the desired results, you should enable API logging (to file) and look for errors there.

The following KBA describes how to enable the API logging:
http://service.sap.com/sap/support/notes/1851170

[Former Member]

Hello Tracy,

Thank you for your answer. I tried to telnet the Afaria Server from the Afaria Admin Server on the port 135. And Telnet was not able to connect somewhere. So Port 135 is either blocked (o don't think so, because the network guys are telling me that nothing is blocked by their firewalls) or nothing in listening on port 135 on the Afaria Server side. Windows Firewall is off on both servers.

But why is the API Server starting without any problems when I start it manually after logging on to the server? Does the API service not need the port 135 then?

It is really strange.

Thank you.

Ernst

[tracy_barkley]

Ernst,

The API must be able to reach the server then on port 135 when you log in.  The API service must have the Afaria server service running when it attempts to reach the server.  If you restart just the API server does it work successfully? 

Tracy

[Former Member]

Tracy,

My apologize - I made a mistake in my test.

Port 135 is available and listening on the Afaria Server.

But when I reboot the Admin Server with the API role, the API service still tries to start, but cant start for any unknown reason.

br

Ernst

[Former Member]

Hello Ernst,

Thank you for your update.

As per the server error log i see there are XRS4940 errors recorded.

There are few points listed here which may help:

1. May be windows server is applying .NET windows patches and the XSOutboundserver.exe and cannot start properly when it is half applied.

Please make sure that all Windows patches and Completely installed, and allow the Afaria Server to reboot.

Hope this helps.

Regards,

Vinayak

[Former Member]

It is not a Master/Slave environment.

[Former Member]

Hello Anis and Vinayak

Thank you for your answers.

The system is fully patched, so the KBA 2226842 don't really help me.

I did a reboot of the server in the sequence Anis mentioned, but unfortunately all the three services are down when I log onto the system after reboot.

Why is this happening? I need the services running after a reboot.

Some more ideas?

Thanks

Ernst

[jtaylor]

Ernst, in addition to the above, can you tell us what patch level you were at before applying SP7? The only other thing that I could imagine would be if you skipped a required update between the previous running version and SP7... and we soemhow didn't catch it during the upgrade. It may be worth going back and rechecking the upgrade path from the previous version to SP7, and making sure that you followed that path.

[Former Member]

Hi  Ernst,

Try to start services manually and let me know the error.

In addition check the Afaria logs in Afaria Administrator console if you can see any relevant logs for Service stop etc.

Let me know the same.

Regards,

Subash Acharya

[tracy_barkley]

One other suggestion.  Make sure that if you have device communication set to start HTTP it isn't on the same port as the IIS web service.  A port conflict can cause unreliable service behavior.  It doesn't sound like this, but it is one more thing to check.

Tracy

[Former Member]

Hello John,

We did the Update from Afaria 7 SP5 HF13 to Afaria 7 SP7. And today during troubleshooting from SP7 to SP8.

As far s I know, that update path is supported.

Ernst

[Former Member]

Please see the FATAL Errors I got in the Afaria Admin Console today during troubleshooting.

[jtaylor]

You are correct, that is a supported path. Have you checked the database connectivity using the account you specified in the installs? I don't think the upgrade would have gone through if this was a problem, but that, along with Tracy's comment, are the normal causes for the Afaria service not starting up.

[Former Member]

Hi Ernst,

The error states its a fatal error, follow below troubleshooting steps:

1. Check the Device Communication setting under "Afaria Admin Console--> Server Tab--> Device communication.

2. Re-Apply the Service pack again and check. Ensure the service pack are installed as per the recommended steps by SAP.

Regards,

Subash Acharya

[jtaylor]

Ernst, I was going based on the assumption that the service was stopping immediately after start... can you confirm if that is the case or not? Do you see high memory or CPU utilization out of XSDiagnosticTests if you look in Task Manager?

[jtaylor]

Ernst, if you're still experiencing this... given the combination of the services that won't start, I would: 1. Check database connectivity.

2. Look for patches to .NET that were applied recently and remove those temporarily to see if it resolved the problem. I'm not aware of any incompaitbilites, but there may have been a problem with the patch installation.

3. Uninstall .NET and Reinstall to the requirements level.

4. If the above don't resolve, you should open an incident in Service Marketplace and upload the event viewer logs. It may also be helpful if you had extracted info from databse table A_LOG_MESSAGE for the time period since shortly before applying the update.

[Former Member]

Hi Taylor,

I will have a look to the .NET patches that got applied recently to the servers.

Thanks for this hint.

br

Ernst

[Former Member]

Hello Ernst,

Please find the KBA # which explains about Afaria Server service to start with XRS4940 on XSOutboundserver - Afaria

http://service.sap.com/sap/support/notes/2226842

Hope this helps.

Regards,

Vinayak