Frequent Locked Out issue with <SID>ADM

[Girwar]

Hi Experts,

One of our BW production server is getting frequent event logs about the <SID>ADM logon failure and id gets locked out in time range of 2-10 minutes. It is getting locked during the call for SAPStartsrv.exe. What process/job could be triggerring these frequent calls to Sapstartsrv.exe.

No errors are there in the work directory WP logs. 

There is no OS level service using this id. Also no other third party tool running on this server. Could you please assist to identify the source of the error and why it says Security ID is SAPService<SID> but locking the Account Name: <SID>ADM.

An account failed to log on.

Subject:
Security ID:  <DOMAIN>\SAPService<SID>
Account Name:  SAPSERVICE<SID>
Account Domain:  <Domain>
Logon ID:  0x3ab038

Logon Type:   3

Account For Which Logon Failed:
Security ID:  NULL SID
Account Name:  <SID>adm
Account Domain:  <Domain>

Failure Information:
Failure Reason:  Unknown user name or bad password.
Status:   0xc000006d
Sub Status:  0xc000006a

Process Information:
Caller Process ID: 0x710
Caller Process Name: D:\usr\sap\<SID>\DVEBMGSXX\exe\sapstartsrv.exe

Network Information:
Workstation Name: <SERVER>
Source Network Address: -
Source Port:  -

[Girwar]

Hi All,

Just to update problem was resolved last week. The SAPCCM4X was corrupted for the affected system in our Solution Manager. As you must be knowing, lastest SAPCCM4X is included in SAPSTARTSRV.exe it was trying to call it from Solution Manager and terminating with logon errors. I redo the connection in RZ21- topology and issue got fixed.

Appreciate your help and time.

Regards,

Girwar

[former_member182657]

Hi Girwar,

Last i would like to suggest method to Disable loopback check on Windows Server by following link Disabling Loopback Check from Windows Server 2008 R2 | Azhar Hussain

After that please check again with Event Viewer logs for further logs.

Hope solutions from my end will help you for troubleshoot the issue.

Regards,

Gaurav

[Girwar]

Hi Gaurav,

How we can say this is a loopback issue. I prefer to identify the root cause before making any changes at registry level.
I will do some more research and update if any findings or more details.

Appreciate help from everyone.

Regards,

Girwar

[former_member182657]

Hi Girwar,

Just a little confirmation i need.Are you working on Windows Server 2008 R2 edition,and from when you are facing this issue (Means any recent update/changes at OS level).

Regards,

Gaurav

[Former Member]

Can you attach screenshot with events in Event Viewer? Also can you provide Event ID for this message? Also, if possible, can you attach full message (without modifications)?

[Girwar]

Event log details....

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          2/18/2014 12:54:50 PM
Event ID:      4625
Task Category: Logon
Level:        Information
Keywords:      Audit Failure
User:          N/A
Computer:      Server01.kcc.com
Description:
An account failed to log on.

Subject:
Security ID:  KCUS\SAPSERVICESID
Account Name:  SAPSERVICESID
Account Domain:  DOMAIN
Logon ID:  0x34d866

Logon Type:  3

Account For Which Logon Failed:
Security ID:  NULL SID
Account Name:  SIDadm
Account Domain:  DOMAIN

Failure Information:
Failure Reason:  Unknown user name or bad password.
Status:  0xc000006d
Sub Status:  0xc000006a

Process Information:
Caller Process ID: 0x4dc
Caller Process Name: D:\usr\sap\SID\D21\exe\sapstartsrv.exe

Network Information:
Workstation Name: Server01
Source Network Address: -
Source Port:  -

Detailed Authentication Information:
Logon Process:  Advapi 
Authentication Package: Negotiate
Transited Services: -
Package Name (NTLM only): -
Key Length:  0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
    <EventID>4625</EventID>
    <Version>0</Version>
    <Level>0</Level>
    <Task>12544</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8010000000000000</Keywords>
    <TimeCreated SystemTime="2014-02-18T12:54:50.224466500Z" />
    <EventRecordID>17816102</EventRecordID>
    <Correlation />
    <Execution ProcessID="532" ThreadID="616" />
    <Channel>Security</Channel>
    <Computer>Server01.kcc.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="SubjectUserSid">S-1-5-21-73153925-784800294-903097961-89459</Data>
    <Data Name="SubjectUserName">SAPSERVICESID</Data>
    <Data Name="SubjectDomainName">Domain</Data>
    <Data Name="SubjectLogonId">0x34d866</Data>
    <Data Name="TargetUserSid">S-1-0-0</Data>
    <Data Name="TargetUserName">SIDadm</Data>
    <Data Name="TargetDomainName">Domain</Data>
    <Data Name="Status">0xc000006d</Data>
    <Data Name="FailureReason">%%2313</Data>
    <Data Name="SubStatus">0xc000006a</Data>
    <Data Name="LogonType">3</Data>
    <Data Name="LogonProcessName">Advapi  </Data>
    <Data Name="AuthenticationPackageName">Negotiate</Data>
    <Data Name="WorkstationName">Server01</Data>
    <Data Name="TransmittedServices">-</Data>
    <Data Name="LmPackageName">-</Data>
    <Data Name="KeyLength">0</Data>
    <Data Name="ProcessId">0x4dc</Data>
    <Data Name="ProcessName">D:\usr\sap\SID\D21\exe\sapstartsrv.exe</Data>
    <Data Name="IpAddress">-</Data>
    <Data Name="IpPort">-</Data>
  </EventData>
</Event>

[former_member182657]

Hi Girwar,

Have you recently updated your OS means Windows update.Please confirm.

Regards,

Gaurav

[former_member182657]

Hi Girwar,

One thing which i want to point from the logs is

"The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network)."

For this i found

Logon Type	Description
2	Interactive (logon at keyboard and screen of system) Windows 2000 records Terminal Services logon as this type rather than Type 10.
3	Network (i.e. connection to shared folder on this computer from elsewhere on network or IIS logon - Never logged by 528 on W2k and forward. See event 540)

Make sure about the share folder access on this server from your network.It may be the culprit.

Also please go through link http://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=4625

Regards,

Gaurav

[Girwar]

Hello Gaurav,

Thank you for the response. Yes, this is a network call and as mentioned in my previous messages may be an API call. But question is what kind of sharefile will be trigering the SAPSTARTSRV.exe so frequently. I tried to look into share folder but nothing much relevant, this is a BW system and connected with Solution manager.

I will try to get some details.

Regards,

Girwar

[Former Member]

LogonType as 3 (network) does not always means network access to shared resource. From Microsoft documentation: "This logon type is intended for high performance servers to authenticate plaintext passwords". After successful logon the current thread (in sapstartsrv.exe process) impersonates that user and can perform actions with security context of that user. Please check status of sapstartsrv process and get pid of that if it is active. Try to call some sapstrartsrv functionality with sapcontrol program and provide results here.

[Former Member]

Hello Girwar,

Just check if you are able to logon to OS using <sid>adm user.

I suspect the password for user SAPService<SID> may be incorrect.

If you are on Windows, change it in the following manner:

●  To change the password at operating system level on each Windows application server, enter in a command prompt:

C:\>lusrmgr.msc
Right-click on <SID>ADM and choose Set Password.

●  Using the services control manager modify the logon properties of the SAP<SID>_<INST> services to match the newly chosen password:

C:\>services.msc
Right-click on SAP<SID>_<INST>and choose Properties. Choose the Log On tab, and change the password.

Refer to: http://help.sap.com/saphelp_nwpi71/helpdata/en/aa/1dc94af0fa11d3a6510000e835363f/content.htm

Best Regards,

Anita

[Girwar]

Hi Anita,

Thank you for the response. I can login with <SID>ADM in the server.Also, the SAP system is up and running. The service SAP<SID>_XX is started and the password for SAPSERVICE<SID> is correct one. It has been tested by doing a restart of Service and SAP system.

There is no error log in SM21 no ABAP dumps. One thing I noticed from the event log is this call is via some API.

"Detailed Authentication Information:

Logon Process: Advapi 

Authentication Package: Negotiate

Transited Services: -

Package Name (NTLM only): -

Key Length: 0"

What process could be triggerring the call for SID<AMD> to trigger sapstartsrv.exe every few minutes and that too via a API logon method. Where else could be the password stored for SID<ADM>?

Regards,
Girwar

[Former Member]

Hello Girwar,

Check your work process logs to get to the root cause.

Best Regards,

Anita

[Girwar]

Hello Anita,

Thank you once again for the response.

I mentioned in my post that there is no relevant error message in SM21 or Workprocess log.
First I need to find, which job/process could be there that keep calling sapstartsrv.exe with PL3ADM login. As per windows log this is API call, my guess was this is linked to CCMS or some other monitoring tool but no clue to confirm which one is the cause.

Regards,

Girwar

[former_member188883]

Hi Meena,

Could you check whether any windows service is running with user <sidadm>.

Set the password in that service and take a restart of this service.

Note: Change of password has to be done with SAP application down.

Regards,

Deepak Kori

[Former Member]

Hello Girwar,

My bad, missed the logs bit mentioned earlier by you.

Try out Deepak's suggestion. Also check the same for SAPService<SID> user.

The last resort would be to open an incident with SAP.

Best Regards,

Anita

[Girwar]

Hi Deepak,

Currently there is no service using the <SID>ADM. We have three instances and all are getting these errors in the log.

The service SAP<SID>_XX is started using the ID SAPSERVICE<SID> and that is correct. As per logs error is coming since long time and no password was changed in that time period.

Our system is up and running fine no impact to system or any service. But the flooding of these log failure causing the lock issue for user id and filling up the event logs.

What could be triggerring every few minutes call for SAPSTARTSRV.exe ??

Regards,

Girwar

[former_member188883]

Hi Girwar,

As you mentioned all development, QAS and PRD has similar error messages.

Can you perform a simple test on development

1) Stop sap application and monitor whether even viewer shows any error message for user locking.

2) Check under which user saposcol is running and other SAP services are running ?

Stop these services as well.

Let us know whether any error messages still appear in windows log

Regards,

Deepak Kori

[Girwar]

Hi Deepak,

By all instances, I mean dialog instance of the Production system. All three instances and CI are getting the <SID>ADM lock issue. We are not using SAPOSCOL, we use Diagnostic agent that use on <SID>ADM login to start and run service.

Regards,

Girwar