Limit n° services Adaptive Anywhere on Windows 2012?

[Former Member]

Hello all,

first of all, a little background information on how we use  :

Our ASP provider uses a windows 2003 db server, where +- 300 sybase 9 (Adaptive Server Anywhere 9.2) engines were running (the process for sybase 9 is dbsrv9.exe).

Last year we start migrating towards sybase 16, but for us it's impossible to migrate evrything at once, so we are migrating one by one.

The current situation is 200 sybase 9 engines and +- 100 sybase 16 engine on the db server.

Our ASP provider informed us, that since windows 2003 was ending, they wanted to migrate to a windows 2012 db server.

We did this migration 2 weeks ago.

All the sybase 16 engines were migrated without any problem, and we could start them just fine.

With the sybase 9 engines we encountered a very weird problem :

as said we have 200 engines left, that we wanted to start, but for some reason, we could only start 175 engines.

The other 25 engines refused to start, the only error they gave was "Could not start DB server"

That was the only error we received, nothing more. in the event log, in the sybase log, everywhere we got "could not start db server"

At that time we started to change serverports, we started to use different admin db's the service uses, started to remove db's out of the service, but nothing helped.

We even tried to raising the number of sybase 9 licences with the dblic command, but same error.

What we then noticed was the following : when we stopped service nr 170, we could start a service (that refused to start with the "could not start db server" error) just fine, but service nr 170 refused to start now ...

That's how we started to play around with services, when we stopped a service, we could start another one just fine ... so it seems that there is a cap on simaltaneous engines.

The questions we have now :

Is the cap a problem of sybase 9? (allthough we had it running just fine on a windows 2003 server).

Is this a windows 2012 problem? (my idea is that this is a windows 2012 problem, but can't be sure at the moment)

Are we going to have the same problem with sybase 16?

Biggest problem right now, is the 175 cap on services, cause we can't start some of our services.

We are converting them as fast we can towards sybase 16, but as of now there are clients of us that can't work.

So if anyone can give us some tips or advice, we would be very obliged

Regards,

Stephen

[former_member188493]

As a temporary workaround, can you start two or more SQL Anywhere 9 databases on one service? This can be done by specifying two or more *.db files on the command line where you used to have one before. Connection strings would probably have to change slightly to name the common server and unique database.

Connections and user ids are local to each database; there is no concept of "server login".

[Former Member]

We are allready doing that.

In our cloud environment we use a service for a main client and his sub clients. Each main client has a seperate service (with his own serverport) to ensure privacy restrictions.

This way a "smart" main or sub client, can't "accidentally" connect to someone elses database, cause he doesn't know the service configuration.

We were thinking to move server main clients together, but then we have to solve this possible privacy problem somehow.

[jeff_albion]

Hi Stephen,

A key question to solving this puzzle: did you also switch architectures from x86 to x64 when migrating or are you comparing x64 to x64?

Also to confirm, this is Windows 2012, not Windows 2012 R2, correct?

What are your server start-up command lines?

Do you perhaps have any start-up logs from the old servers to check to see what their maximum cache sizes were and then compare them to recent logs to see what the servers are using now?

Regards,

Jeff Albion

SAP Active Global Support

[Former Member]

Hi Jeff,

I checked the version of the current server and it is windows 2012 R2 x64, so not windows 2012 base version.

The old server we used was a windows 2003 R2 enterprise edition x64.

So both servers were in 64 bit environment.

The command lines we use for our sybase 9 engines are on both servers the same.

This is for instance a startup line we use :

-c 30M -x tcpip(serverport=49277) -gd ALL -ti 0 -gp 8192 -n 9CYCLA \\CLOUD\APPDATA\CYCLA\ADMINSA9CYCLA.DB -n ADMINSA9CYCLA

In some cases we have more then adminsa9 db connected to a service, so then we repeat the adminsa9 line.

All our service are created with the same configuration, only the serverport changes, the name of the service and the name of the adminsa9 db.

If i start this service as one of the 175 services, it will start without any problem.

If i start it as number 176, then the service won't start.

the logs of the old servers are at the moment quite hard to get, cause the old servers are offline

they had the same name, so we had to remove them, otherwhise the services would not start.

[jeff_albion]

Hi Stephen,

We have changed the maximum cache memory (-ch) defaults between version 9 and 16, which is the maximum memory that will be reserved with the OS when the database server starts up.

As an example, the version 9 defaults on my system using -c 30M give the limits:

    Minimum cache size: 30720K, maximum cache size: 262144K

(From the version 9 documentation: "By default the upper limit is approximately the lower of 256 Mb and 90% of the physical memory of the machine")

The version 16 defaults on my system using -c 30M are significantly different:

   Minimum cache size: 34472K, maximum cache size: 7438452K

(From the version 16 documentation: "By default the upper limit is approximately the lower of the maximum cache size and 90% of the total physical memory of the computer." where the maximum cache size "depends on the operating system." )

You can use an explicit -ch switch (e.g. -ch 256M to return it to version 9 defaults) to limit the maximum amount of memory each database server will reserve.

Regards,

Jeff Albion

SAP Active Global Support

[Former Member]

Hi Jeff,

this is allready some great advice for our sybase 16 engines, so they dont drain the memory of the server to fast.

but the memory usage isn't causing the problem with they sybase 9 engines i'm afraid.

when we start the 175 sybase 9 engines and the 95 sybase 16 engines, our server uses 12gb ram memory. but the server has a total of 64 GB ram, so that not even close to his limits.

at the end of the day, the server uses between 30gb and 35gb memory (but now i know why sybase 16 engines uses up that much, and how to stop them )

but even with 30gb of free memory we can only start 175 engines.

the remaining ones refuses to start, with only "database could not be started" error.

if need, i can provide some log files of the error

we added the -o option in the service to get the log file, but this also gives "database could be not be started"

i don't know if there are options for more extended logs?

thanks for the help allready!

Regards,

Stephen

[former_member188493]

This Q&A may or may not apply to your situation: Windows Server 2012 R2 runs out of ephemeral ports, though it shouldn't

[jeff_albion]

Hi Stephen,

What are the bitnesses of the database server binaries in each case - is SQL Anywhere 9 also x64 or are you moving from x86 to x64 server binaries between version 9 and 16?

---

What's the exact error message you're seeing when you are trying to start the database?  I can't find the message "database could be not be started" in our code.

Have you tried limiting the cache on all of the started server processes now?

What happens if you don't start the last database server as a service and try to launch it just on the command line?

Regards,

Jeff Albion

SAP Active Global Support

[Former Member]

this is something i will forward to our tech guys, so they can look it up

thx!!

[Former Member]

Hi Jeff,

the sybase 9 engines are running in 32 bit, the sybase 16 engines are running in 64bit.

this is the message in the event viewer :

The description for Event ID 1 from source ASA 9.0 cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event

The following information was included with the event:

ASANYs_9VDPT

Could not start server

the xml view gives me this :

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

- <System>

  <Provider Name="ASA 9.0" />

  <EventID Qualifiers="0">1</EventID>

  <Level>2</Level>

  <Task>0</Task>

  <Keywords>0x80000000000000</Keywords>

  <TimeCreated SystemTime="2015-03-06T09:33:54.000000000Z" />

  <EventRecordID>86432</EventRecordID>

  <Channel>Application</Channel>

  <Computer>P-ODB-001</Computer>

  <Security />

  </System>

- <EventData>

  <Data>ASANYs_9VDPT</Data>

  <Data>Could not start server</Data>

  </EventData>

  </Event>

if i stop a service that is running, i can start the 9VDPT service without a problem.

if i try to start the service that was running, i get the same error as above.

i can add the -o option in a service, but is -o the most extended log?

then i can post that log file here too

[jeff_albion]

Hi Stephen,

i can add the -o option in a service, but is -o the most extended log?

No, -o is just the console log which contains basic information about the server start-up. Can you post it here, if it is being generated?

<Data>Could not start server</Data>

"Could not start server" means that we failed to get a new process ID when we tried to make an operating system call ( _beginthread ) against the database server's main thread. It returned -1:

"On an error, _beginthread returns -1L, and errno is set to EAGAIN if there are too many threads, to EINVAL if the argument is invalid or the stack size is incorrect, or to EACCES if there are insufficient resources (such as memory)."

Based on your description of starting and stopping services, I would suspect we're seeing EACCES. You can try using Process Monitor to see what happens when the Process Create / Process Start / Thread Create events happen during a service start:

Note how I received a parent PID and thread ID when it was successful. When this fails, the reason should be printed in the "Result" column.

Have you tried setting -ch on all of the services now? Have you tried the command-line test?

Regards,

Jeff Albion

SAP Active Global Support

[Former Member]

Hi Jeff,

we are currently adding the -ch option to our engines. also we are moving the sybae engines to a new and seperate server.

Once this is done we will have a server for our sybase 9 engines and a server for our sybase 16 engines.

This way we make sure that the sybase 9 engines have all the memory they need.

When this move is done, i'll try to start all of our Sybase 9 engines again, and i will report back with the errorlog of Process Monitor and the command line.

This will probably be on monday, cause the move takes some time.

Thanks for the help allready!

Stephen.

[Former Member]

Hi Jeff,

here we are again.

This weekend we moved all the sybase 16 engines towards a new server. So now we have 2 windows 2012 R2 server, one with only sybase 9 engines and one with only sybase 16 engines.

The one with sybase 9 engines has at the moment 12 GB RAM (we noticed that this is enough for sybase 9). With all the enines started, the server uses 10GB RAM in total, so we have 2GB ram left.

So we are sure this isn't a ram issue.

This is a current view of our Sybase Central :

at the moment 175 engines are started, the service TOP9TOP13 is number 176 i want to start.

when i try to start the engine, i instantly get this :

when i stop engine TOP9TOP12, i can start TOP9TOP13 with no problem :

when i try to start top9top12 i get the same error as before :

so i started procmon, stopped engine top9top12 and first started top9top12 and then top9top13

the top9top13 engine gave me the error. in procmon i got this :

this is the detailed info of the service that does not start :

process create :

process start:

process exit:

thread exit:

if i look between the last thread create and the first time i see thread exit, there are only a couple of events happening :

so i really have no idea why the engine refuses to start.

ps : when i try to start the service from a cmd box, i get the same problem in procmon :

Regards,

Stephen

[jeff_albion]

Hi Stephen,

Hmmm - I would have expected that diagnostic to be more useful. Let's keep going...

ps : when i try to start the service from a cmd box, i get the same problem in procmon :

1. Can you start just a regular "dbsrv9 -n demo demo.db -n demo" command on the desktop? (Don't use the service command line at all).
2. Can you start any other third-party (non-system) services, not related to SQL Anywhere?
3. Are you using "Interact with Desktop" (dbsvc -i) at all with these SQL Anywhere services? I would expect the answer is "no" but if you are, you should disable the feature.

---

One reason you might not be able to start a service is because you have too many processes using (desktop) heap memory in session 0 (See: https://support.microsoft.com/en-ca/kb/947246 ). This is not related to how much RAM is available on the system.

To determine this, this is best diagnosed with dheapmon from Microsoft, although I would say from a quick local test with the utility, it is not the easiest to install (see here, and here). I would try the approach from the first thread to use the Windows 2003 compatibility mode on the dheapmon utility/DLL and see if you can install it that way.

Once you get the utility going, you can check for the amount of heap space is left in the Service (s0) heaps:

================================

>dheapmon

Desktop Heap Information Monitor Tool (Version 8.1.2925.0)

Copyright (c) Microsoft Corporation.  All rights reserved.

-------------------------------------------------------------

  Session ID:    1 Total Desktop: ( 65088 KB -   11 desktops)

  WinStation\Desktop            Heap Size(KB)    Used Rate(%)

-------------------------------------------------------------

  WinSta0\Winlogon (s1)               192              7.4

  WinSta0\Disconnect (s1)              96              6.5

  WinSta0\Default (s1)              20480              5.4

  WinSta0\DummyDesktopForInstantResume (s1)  20480              0.0

  msswindowstation\mssrestricteddesk (s0)    768              0.8

  Service-0x0-3e5$\Default (s0)       768              2.2

  Service-0x0-3e4$\Default (s0)       768              1.9

  Service-0x0-3e7$\Default (s0)       768             12.8

  WinSta0\Winlogon (s0)               192             10.5

  WinSta0\Disconnect (s0)              96              6.5

  WinSta0\Default (s0)              20480              0.1

-------------------------------------------------------------

================================

If you are running out of heap memory in the service area, you can adjust this a number of ways - the overall heap for the system is controlled by:

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\SessionViewSize

which is 48M by default:

You can also configure the size of the individual desktop session via the instructions on the Microsoft KB: https://support.microsoft.com/en-ca/kb/947246 by editing HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems\Windows

Mine is already at the prescribed maximum (20480KB) for Windows Vista and up, but if you are running out of heap in just your session 0, maybe this value needs to be bumped up:

Do you remember if you had to do something similar for your previous Windows 2003 server installation also?

Regards,

Jeff Albion

SAP Active Global Support

[Former Member]

Hey Jeff,

thanks again for the answers, i'm gonna ask our tech guys to look into the heap memory.

we arent using the interact to desktop option, we had problems with that in the past.

other services are starting fine, for instance we have a backup program that uses a service, and the we can stop and start the service without a problem.

when i got the answers from the tech guys, i will report here again.

[Former Member]

Hi Jeff, it's been a while, apologies for that.

our tech guys tried to open the heapmem program on the windows 2012r2 server, but they were not able to open or install it. the program doesn't respond when opened.

so we shifted all our efforts to converting from sybase 9 to sybase 16. with the conversions we dropped below the 175 active services, so our problem with sybase 9 was solved.

on the new server we kept converting services and databases to sybase 16.

but i'm afraid we have the same issues now, with sybase 16 ...

this time we can only start 85 services ... when we want to start nr 86 we get the same errors like we got with sybase 9 ...

so now we're stuck again, but this time on sybase 16.

are there any tools for sybase 16 we can use to see what is going on?

thx in advance