System.AccessViolationException in 64 bit ASE client on server

[Former Member]

we are receiving the following message on our servers.   bin  has only Sybase.AdoNet4.AseClient.dll  . Please advice what is the potential cause of the issue.

Application: w3wp.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.AccessViolationException

Stack:

   at Sybase.Data.AseClient.Unmanaged.CMD_PerformGetNextResultSet(IntPtr, IntPtr ByRef, Int32 ByRef, Int32 ByRef, Int32 ByRef, Byte ByRef)

   at Sybase.Data.AseClient.Unmanaged.CMD_PerformGetNextResultSet(IntPtr, IntPtr ByRef, Int32 ByRef, Int32 ByRef, Int32 ByRef, Byte ByRef)

   at Sybase.Data.AseClient.AseDataReader.RetrieveNextResult()

   at Sybase.Data.AseClient.AseDataReader.GetNextResult(Boolean)

   at Sybase.Data.AseClient.AseDataReader.NextResult(Boolean)

   at Sybase.Data.AseClient.AseParameter.get_Value()

   at Sybase.Data.AseClient1.AseCommand.Close(Boolean)

   at Sybase.Data.AseClient.AseCommand.Dispose(Boolean)

   at System.ComponentModel.Component.Finalize()

Faulting application name: w3wp.exe, version: 8.0.9200.16384, time stamp: 0x50108835

Faulting module name: sybdrvado20.dll, version: 2.157.1300.0, time stamp: 0x53c85084

Exception code: 0xc0000005

Fault offset: 0x0000000000007119

Faulting process id: 0x142c

Faulting application start time: 0x01d0d41e996f4948

Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

Faulting module path: C:\Windows\TEMP\Sybase.AdoNet4.AseClient.64bits.4.157.1300.0\sybdrvado20.dll

Report Id: e4a82238-4011-11e5-9417-005056935d0f

Faulting package full name:

Faulting package-relative application ID:

0 Views  (edit) Tags: .net, enterprise, sybase, sdk, ado.net, ase

[Former Member]

Is it possible to try SP133 build?

Where should i download this from please provide me instructions.

Here is the more error info

DetailID = 5

  Count:    1

  Type:     System.TypeInitializationException

  Message:  The type initializer for 'Sybase.Data.AseClient1.AseConnection' threw an exception.

  Type:     System.UnauthorizedAccessException

  Message:  Access to the path 'C:\Windows\TEMP\Sybase.AdoNet4.AseClient.64bits.4.157.1300.0' is denied.

  Stack:   

  [GCFrame]

  [GCFrame]

  [PrestubMethodFrame]

  Sybase.Data.AseClient.AseConnection..ctor(System.String)

  dbtemp.DBServer.GetNewConnection()

  dbtemp.DBServer.CreateTransaction()

  ERMS.LanguageLoader..ctor()

  ASP.global_asax.Session_Start(System.Object, System.EventArgs)

  System.Web.SessionState.SessionStateModule.CompleteAcquireState()

  System.Web.SessionState.SessionStateModule.BeginAcquireState(System.Object, System.EventArgs, System.AsyncCallback, System.Object)

  System.Web.HttpApplication+AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

  System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

  System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)

  System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)

  System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)

  System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

  System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

  DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)

  [InlinedCallFrame]

  [InlinedCallFrame]

  DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)

  System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

  System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

  DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)

  [ContextTransitionFrame]

DetailID = 6

  Count:    1

  Type:     System.TypeInitializationException

  Message:  The type initializer for 'Sybase.Data.AseClient1.AseConnection' threw an exception.

  Type:     System.UnauthorizedAccessException

  Message:  Access to the path 'C:\Windows\TEMP\Sybase.AdoNet4.AseClient.64bits.4.157.1300.0' is denied.

  Stack:   

  [GCFrame]

  [PrestubMethodFrame]

  Sybase.Data.AseClient.AseConnection..ctor(System.String)

  dbtemp.DBServer.GetNewConnection()

  dbtemp.DBServer.CreateTransaction()

  ERMS.SystemMonitor.checkLock()

  ERMS.SystemMonitor.monitorUILock()

  System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

  System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

  System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

  System.Threading.ThreadHelper.ThreadStart()

  [GCFrame]

  [DebuggerU2MCatchHandlerFrame]

  [ContextTransitionFrame]

  [DebuggerU2MCatchHandlerFrame]

DetailID = 7

  Count:    1

  Type:     dbtemp.DBException

  Message:  DBServer::GetConnection: Sybase.AdoNet4.AseClient: The type initializer for 'Sybase.Data.AseClient1.AseConnection' threw an exception.:    at Sybase.Data.AseClient1.AseConnection..ctor(AseConnection realConnection, String connectionString)

   at Sybase.Data.AseClient.AseConnection..ctor(String connectionString)

   at dbtemp.DBServer.GetNewConnection()

  Stack:   

  [HelperMethodFrame]

  dbtemp.DBServer.GetNewConnection()

  [GCFrame]

  [PrestubMethodFrame]

  Sybase.Data.AseClient.AseConnection..ctor(System.String)

  dbtemp.DBServer.GetNewConnection()

  dbtemp.DBServer.CreateTransaction()

  ERMS.SystemMonitor.checkLock()

  ERMS.SystemMonitor.monitorUILock()

  System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

  System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

  System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

  System.Threading.ThreadHelper.ThreadStart()

  [GCFrame]

  [DebuggerU2MCatchHandlerFrame]

  [ContextTransitionFrame]

  [DebuggerU2MCatchHandlerFrame]

DetailID = 8

  Count:    1

  Type:     dbtemp.DBException

  Message:  DBServer::GetConnection: Sybase.AdoNet4.AseClient: The type initializer for 'Sybase.Data.AseClient1.AseConnection' threw an exception.:    at Sybase.Data.AseClient1.AseConnection..ctor(AseConnection realConnection, String connectionString)

   at Sybase.Data.AseClient.AseConnection..ctor(String connectionString)

   at dbtemp.DBServer.GetNewConnection()

  Stack:   

  [HelperMethodFrame]

  dbtemp.DBServer.CreateTransaction()

  [HelperMethodFrame]

  dbtemp.DBServer.GetNewConnection()

  [GCFrame]

  [PrestubMethodFrame]

  Sybase.Data.AseClient.AseConnection..ctor(System.String)

  dbtemp.DBServer.GetNewConnection()

  dbtemp.DBServer.CreateTransaction()

  ERMS.SystemMonitor.checkLock()

  ERMS.SystemMonitor.monitorUILock()

  System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

  System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

  System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

  System.Threading.ThreadHelper.ThreadStart()

  [GCFrame]

  [DebuggerU2MCatchHandlerFrame]

  [ContextTransitionFrame]

  [DebuggerU2MCatchHandlerFrame]

DetailID = 9

  Count:    1

  Type:     dbtemp.DBException

  Message:  DBServer::GetConnection: Sybase.AdoNet4.AseClient: The type initializer for 'Sybase.Data.AseClient1.AseConnection' threw an exception.:    at Sybase.Data.AseClient1.AseConnection..ctor(AseConnection realConnection, String connectionString)

   at Sybase.Data.AseClient.AseConnection..ctor(String connectionString)

   at dbtemp.DBServer.GetNewConnection()

  Stack:   

  [HelperMethodFrame]

  ERMS.SystemMonitor.monitorUILock()

  [HelperMethodFrame]

  dbtemp.DBServer.CreateTransaction()

  [HelperMethodFrame]

  dbtemp.DBServer.GetNewConnection()

  [GCFrame]

  [PrestubMethodFrame]

  Sybase.Data.AseClient.AseConnection..ctor(System.String)

  dbtemp.DBServer.GetNewConnection()

  dbtemp.DBServer.CreateTransaction()

  ERMS.SystemMonitor.checkLock()

  ERMS.SystemMonitor.monitorUILock()

  System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

  System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

  System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

  System.Threading.ThreadHelper.ThreadStart()

  [GCFrame]

  [DebuggerU2MCatchHandlerFrame]

  [ContextTransitionFrame]

  [DebuggerU2MCatchHandlerFrame]

[Former Member]

Hi Ram,

This looks like a different issue.  Can you please let us know how, exactly, do you deploy the provider?

When AseConnection is invoked, the Provider contains the un-managed DLLs, embedded in it's own DLL. These are copied to location, like in this case maybe C:\windows\temp\ because it is IIS doing this.

It is unusual to be denied access to the Windows\TEMP directory and this is happening on new connections - this wasn't happening before in the access violation - in that situation you are connecting , etc. So what changed in this case?  Something did since the Provider doesn't spontaneously decide itself don't have rights to a specific directory - this is something happening externally.

This directory:

'C:\Windows\TEMP\Sybase.AdoNet4.AseClient.64bits.4.157.1300.0

is where the Provider copies the proper version of sybdrvado20.dll - this is the un-managed layer handling the TDS - the communication to the ASE.  When the constructor:

Sybase.Data.AseClient.AseConnection..ctor(System.String)

call is invoked, the Provider first checks for existence of that directory, then if not, creates directory and copies the platform specific version of the sybdrvado20.dll there.  If it exists, it loads that DLL. In your case it is clearly being denied access to the directory and this isn't making sense.

The only thing I can think of now is the directory was initially created by one app pool, with a specific ID and now another app pool is trying to access and has no rights to this directory.  Can you check this?  See if app pools are using different windows ids?

Also, please find the specific rights for the 'C:\Windows\TEMP\Sybase.AdoNet4.AseClient.64bits.4.157.1300.0 directory - you can look at the directory properties, and check Security I think - this should list the users and their access rights. The app pool users require full access to the directory.

Cheers,

-Paul

[dawn_kim]

Hi Ram,

Okay to capture the tds layer, put PROTOCOLCAPTURE=c:\somelocation\mytrace in the connection string of the application.
Or you can do a wire shark maybe this will catch the thread.

We had an old issue but this was fixed in SDK 15.7 SP120 and higher and is included with the SDK 15.7 SP130 and SP133.
The workaround was to make sure  group IIS_IUSERS needs full permissions on the C:\Windows\Temp folder. Can you double check and make sure this happening?

So like Paul said above, it is something about he initial connection, so make sure the users that are part of the app pool have permissions to that directory also.

Thanks,
Dawn Kim

[Former Member]

Hi Paul,

Can you please let us know how, exactly, do you deploy the provider?

Sybase.AdoNet4.AseClient.dll  is referenced in Visual studio Bin folder of Asp.net website  and published the web project.

When deployed, Bin folder will have website dll and Sybase.AdoNet4.AseClient.dll  along with web project folders.

Do you see any problem above?

I will check on the folder permissions and re post the info.

Regards,

Ram

[Former Member]

Update on folder.

IIS_IUSERS group has full control over the following folder.

C:\Windows\TEMP\Sybase.AdoNet4.AseClient.64bits.4.157.1300.0 directory -

1) If permission is the issue how  app_pool restart is helping...?

2) Can you please tell what  ASE version  this DLL belong  Sybase.AdoNet4.AseClient.dll  product version 4.157.1300.0.

I will keep you posted on sp133 and TDS.

-Ram

[dawn_kim]

Hi Ram,

Did SDK 15.7 SP133 work for you?
This is a link that talks about IIS and application pool:
ref http://stackoverflow.com/questions/3868612/what-is-an-iis-application-pool

So Paul wanted to make sure the pool user is added to the permissions of the .dll and the temp directory.

Thanks,
Dawn Kim

[Former Member]

No . with SP133 same issue. on app start few database calls break like mentioned before. When app pool shuts down due to inactivity Issue happens again. This errors crashing the app_pool. what else we can try ? how to read the TDS files.

[dawn_kim]

Hi Ram,

If you added protocolcapture=c:\mylocation\mytrace to the connection string you should get traces.
To translate these traces please read the links for ribo up above.
Go to the command prompt.
Basically create two environmental variables: set RIBO_HOME=C:\$sybase\jutils-3_0\ribo and set JAVA_HOME=C:\$sybas\jre64 or some other java location.

Then type: ribo mytrace.0.tds mytrace0.txt

You should be able to read the text file and see the last command the application sent.

You might want to try adding AseCommandDisposeOptional=true to the connection string this might help with some clean up with the .net provider.

Thanks,
Dawn Kim

[Former Member]

Hi Kim,

Can  you confirm if we need to install ASE Client on windows server to  run asp.net website(asp.net website bin folder has sybase ado.net dlls, Is that not enough) ?

-Ram

[dawn_kim]

Hi Ram,

It depends, please read this it describes the .ddls you need for the ado.net provider.

ADO.NET files to deploy - http:/wiki.scn.sap.com/wiki/x/oghiGQ

Thanks,
Dawn Kim

[Former Member]

Hi Ram,

We certify on the Operating System (win 2012) but we don't certify by IIS.  We support .NET framework based on the Provider being used.

In your case:

1) Looks like you're using 4.0 .NET with the ASE 4.0 Provider.

2) In the stack the closing of the command occurs - it is checking to make sure all result sets are processed and flushed out.  The lower layer where TDS protocol (communication between ASE and client) seems to be having trouble in the check for getting potential result set.

I don't know if this means there is a result in there or not.

Does this happen often (how many times per X?) Did it start happening with an upgrade of something (what?) or is this a new application.

How do you mitigate it now?

What sort of load is on this app at the time - about how many concurrent connections?

TDS tracing might be helpful to see if there is indeed a result set.

It could be this is a timing issue but seems some memory is corrupted somewhere, somehow.

Is it possible to try SP133 build?

Cheers,

-Paul

[Former Member]

Hi Paul,

1) In my case we are using 4.0 .NET with the ASE 4.0 Provider(Sybase.AdoNet4.AseClient.dll  product version 4.157.1300.0) this is the only sybase DLL in the bin on the server.

(No sybase component installed on server)

2) I am more into .net Role. What is TDS?.Should it be installed on windows 2012 server box?  or on Sybase server host box.

3)Does this happen often (how many times per X?)

Error is happening 5-6 times in day (based on traffic it  may be more ). 1 or 2 App pool  crash in  a day.

4)Did it start happening with an upgrade of something (what?) or is this a new application.

We migrated from  Windows 2003 (32 bit ) to  Windows  2012 64 bit. and it started happening (we dont have issues when we are on 2003 box (32 bit))

5)How do you mitigate it now?

We are restarting appool whenever it is happening.

6)What sort of load is on this app at the time - about how many concurrent connections?

500 approx

7)It could be this is a timing issue but seems some memory is corrupted somewhere, somehow.

Is it possible to try SP133 build?

[Former Member]

Hi Ram,

TDS is Tabular Data Stream - it's the communication protocol between client and server and doesn't require installation, it is built into the product - it is how communication occurs between the .NET clients and the ASE.

Seems to be an issue related to upgrading application to a more robust OS/machine.  During the AseCommand.Dispose() call the Provider checks to make sure all network and intermediary buffers are cleaned out before destroying the AseCommand object.

I just got your other reply and need to look into that.

To get SP133 you can go to the SAP download page:

1. Go to  https://support.sap.com/software.html

2. Select the Support Packages and Patches item

3. Select A-Z

4. Select "S"

5. Select SDK for SAP ASE and then select the version and platform (15.7 or
16.0)

Cheers,

-Paul

[Former Member]

Hi Paul,

This is asking us used id  password . I dont have to enter any when i downloaded SDK. Please help.

Regards,

Ram

[dawn_kim]

Hi Ram,

See if you can get the free development SDK.

There is a trial/free version of the SDK up on the site currently.

Go here: http://scn.sap.com/community/downloads

There is an article titled "The SCN Download Center is Now on the SAP Store: What's New?"

In this article there is a link to the SAP store click it

Than do a search for "SAP SDK for Adaptive Server Enterprise"

Click the free trial and it will ask for your information and send you an email.

From the link you get you should see a list of the current SDKs.

[dawn_kim]

Hi Ram,

If you do process explorer and watch w3wp.exe and see where it is loading the files. I have a feeling your application is pulling sybdrvado20.dll from some other location or a different version then the temp directory.

So basically when you use the newer driver everything is included in the one .dll. It expands in a temp directory. Make sure the user has permissions on this directory also. I remember seeing an issue with SSIS with this.

If the above doesn't help add protocolcapture=C:\somepath\mytrace to the connection string.
You can change this to a text file  by using ribo and read through this it might tell you what is the last call being made so maybe this might point to issue.
ref: Setting up oledb with Protocol Capture - SAP Connectivity - SCN Wiki
ref: TDS Tracing tool Ribo on Windows - ERP Financials - SCN Wiki

Thanks,
Dawn Kim

[Former Member]

We don't have any other sybase file except   Sybase.AdoNet4.AseClient.dll  4.157.1300.0 in windows 2012 server. Please advise

Analysis symbol:

Rechecking for solution: 0

Report Id: cb7c6f90-419d-11e5-9417-005056935d0b

Report Status: 4

Hashed bucket: "

Error 8/13/2015 5:29:29 AM Application Error 1000 (100) "Faulting application name: w3wp.exe, version: 8.0.9200.16384, time stamp: 0x50108835

Faulting module name: sybdrvado20.dll, version: 2.157.1300.0, time stamp: 0x53c85084

Exception code: 0x40000015

Fault offset: 0x0000000000007c32

Faulting process id: 0x1c68

Faulting application start time: 0x01d0d59b5268bfa1

Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

Faulting module path: C:\Windows\TEMP\Sybase.AdoNet4.AseClient.64bits.4.157.1300.0\sybdrvado20.dll

Report Id: cb7c6f90-419d-11e5-9417-005056935d0b

Faulting package full name:

Faulting package-relative application ID: "

[Former Member]

We need urgent help in figuring this out. Please confirm Sybase.AdoNet4.AseClient.dll  product version 4.157.1300.0 is compatiable with windows 2012 54 bit server IIS 8.0

Analysis symbol:

Rechecking for solution: 0

Report Id: fb4aa589-4401-11e5-9417-005056935d0b

Report Status: 8

Hashed bucket: "

Information 8/17/2015 6:16:08 AM Windows Error Reporting 1001 None "Fault bucket , type 0

Event Name: CLR20r3

Response: Not available

Cab Id: 0

Problem signature:

P1: w3wp.exe

P2: 8.0.9200.16384

P3: 50108835

P4: Sybase.AdoNet4.AseClient

P5: 4.157.1300.0

P6: 53c85590

P7: 405

P8: 67

P9: Sybase.Data.AseClient.Ase

P10: