on 11-16-2006 11:08 AM
Recently we've installed ten application servers to help our AS/400. We've moved all the users to the logon group and all uses only this servers.
As I said the central instance is AS/500 v5r3, the aplication server are Windows 2003 64 with sp1.The kernel is also 64 bit ( 640 unicode ). The sap release is 4.7 enterprise. The configuration of the server is:
- 2 Opteron processor with two kernel each.
- 16 Gb of RAM.
- 20 Gb of paging file ( following the SAP recommendations ).
The sap parameters have also been changed. Using perfmon and the st06 transactions i can see that the page out is too high. Is any reason for that ? Any recommendation ?
this is a known problem on Windows Servers which is related to Windows Memory Managment. Microsoft is currently working to improve it.
regards
Peter
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
no, not yet. But we will do, when we have a fix for it. Until then the situation is as it has developed in the past 12 Years:
The page out story is as old as SAP R/3 on Windows and related to the mechanism called extended memory within R/3 and Windows Memory Management. It is not a special issue to the 64 Bit Windows platforms.
In order to support bigger memory configurations on 32 Bit Windows and Linux extended memory (ABAP Memory) is implemented in a way that only the active user context of an ABAP user is part of a work process virtual address space. Before a work process processes a user request the memory pages of the user context (abap memory of the modus etc) are mapped into the work process virtual address space. At the end of the processing in the work process the context is rolled out - the related memory pages are detached from the work process virtual address space.
In Windows Memory Management this action (unmapping) is handeld in a similar way as the Working Set Trimmer works (which gets active when system comes in a short of memory state): unmapped unmodified pages where added to a standby list, modified pages are added to the modified page list. In all Windows versions the pagewriter will start cleaning up the modified page list (writing modified pages back to their backing store -pagefile- the page changes the status to unmodified and is added to the standby list) if the modified pages list contains more than a certain number of pages. The different Windows versions have different algorythms how many pages are written back to pagefile, but the threshold for starting the action is identically from Windows 2000 Professional to Windows 2003 Server Datacenter Edition for IA64: 800 Pages - or 3.2 MB (a very,very small number)
In the past Microsoft explains that the reason for this behaviour is related to prepare memory shortage situations: as long as the system has sufficient resources it will preventively page out modified memory pages in a moderately manner. In case of a memory shortage they can drop the pages off the standby list and can immediately use the physical memory for new memory requests, without first writing them to pagefile. When Windows 2003 was introduced we obtained a significantly higher pagout rate compared to Windows 2000. Microsoft explains this with the necessity of some changes in memory management caused of kernelmode locks we had in some configurations. As long as sufficient memory is provided these pagesouts should not be a bootleneck for the overall performance of the system, claimed Microsoft.
Comparing the situation today and 7 years (4.6C days) ago you can see, thattwo factors have changed dramatically:
1. processor performance has increased rapidly.
2. the average size of a user context increased in a similar way and with them the number of modified pages in a transaction.
but one thing did not change that fast: disk spead. capacity has increased a lot but the number of spindles, surfaces and RPMs are stil the same. So a random accesses to a 4k page needs nearly the same time as 3 years ago.
we have shown Microsoft where the problem sticks in their operating system by modifying the Windows Management variable which defines the modified page list maximum in the Windows kernel using the Windows Kernel Debugger.
It immediately brings down pageouts to 0 the overall pageout rate is reasonable.
It's like Doping: it helps a lot, but is completly unsupported and you can't oversee the side effects.
But you know: changing this part of Windows code needs extended testing - there is a lot of other software running on them - not only SAP and fixing this issue must not harm the performance of any other software (like Database, Exchange, Internet Information Server......).
Maybe it helps to open a message at Microsoft, in order to show them they have customers having some problems with the operating system performance.
regards
Peter
regards
Peter
Many many thanks.
I'm going to open the message in Microsoft. Do you think I have to open a message in SAP first ? Probably Microsoft will say that the problem is in the SAP side and the aplication server configuration. If they won't find any problem in the system ( related to microsoft ) i'm sure they are going to invoice me the consultant services. If I've a message oppened in SAP i could refer to it when I open the Microsoft problem.
I need to know if your response is the same that SAP will aswer to me. What do you think about it ?
Best regards.
Hi!
We applied WindowsServer2003-KB931308-v2-ia64-ENU.exe
on two 4way Itanium productive application servers.
And the memory paged out dropped from (peak) 55 GB per hour to some 200 MB per hour. This is a reduction by 99.6 percent, if my calculation is right...
Before the patch the paging out depended heavily on the workload: We saw a broad distribution from 300 MB in the deep night to 55 GB in the hot morning hours. Now we see an even distribution the whole day over.
Kind regards, Rudi
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I think this was an ABAP instance problem in the first line.
But as it is a Windows system issue, IMHO the fix should be helpful for every application having allocated plenty of memory and using this heavily.
We looked at other servers running things like Exchange or SQLServer only and we saw no high paging rates like those we see on servers running SAP ABAP instances.
And we have no machine running Java only, we have only 3 double stack installations. So have a look at your "Java only machine" to see how many pages are written to the pagefile. And let us know!
Kind regards, Rudi
One things that is not mentioned, is that this HOTFIX breaks SQL Server Management Studio on the local servers.
When Studio tries to open anyother window (almost any action within SSMS) it crashes. This is a know issue within Microsoft and I've got a case raised with them for a fix.
At the end of the day this hotfix FIXES more than it breaks , as you can always connect remotely via another instance of SSMS... so we need apply it.
I have problem with this hotfix (931308).
First we use SAP 4.6B, when I installed this hotfix It solved the paging issue.
Than we upgraded to SAP 4.71 on the same server. I makes some interesting behavior of the server:
1st When I click on the network icon on the right corner, the Explorer.exe closes suddenly.
2nd The cluster admin mmc also close, when I try to open Property tab.
3rd When I tried to install SP2, after approx. 15sec close the wizard without any error message.
After uninstalling the hotfix everything seems to be ok, I could install SP2.
After that I installed back the hotfix, I got the same symptoms.
(however, this hotfix contains 2 version of the ntkrnlmp.exe, one for SP1, and one for SP2)
Finally, I asked somebody else, who also installed this patch, He had no problem with it, but He runs Windows 2003 x86.
Our servers are
Windows 2003 IA64 (Itanium) (HP Integrity 1620, and 4640)
Does anybody about this patch???
Regards,
Imre
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
see note <a href="http://service.sap.com/sap/support/notes/1009297">1009297</a>
regards
Peter
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
not exactly,
but that's what we heard:
- modified page maximum has not been changed
- modified pagewriter is scheduled to start every second (not changed). if the number of pages in the modified pagelist are greater than mmmodifiedpagemaximum, he will write exactly one cluster (4*4KBlock). If after that the Available Memory is bigger than 25% of physical Memory or 1GB (smaller value is decisively) he will stop, otherwise he will repeat this step until the condition is reached. I do not know if these two limits are exactly the one they have implemented, but they where in discussion and the system behaviour looks very much like this.
What we can obtain is:
- pageout starts immediateley with 16 pages/second (or ~230 MB/hour)
- pageouts/hour are significantly smaller than the numbers we obtained with the modified Windows Kernel variable
Peter
as not 1009297 mentioned:
KB 929620 is not part of the Microsoft Security Hotfix System and is only available on demand.
This means you have to open a call at MS to get it.
It will not be part of the initial sp2!
There will be an additional Fix for SP2 which then will also install on SP1.
How this one is going to be distributed??? don't ask me!
Peter
Hi peter.
I've tested the microsoft patch in the test servers. Seems to work fine, the page out have been drastically decreased. We planned to install it on the production application servers this month. Then we can test if this windows pacth can improve the server performance. If it can't do this i think that the hard disks will be less loaded.
Best regards.
Hi!
In the meantime the article got published
--> http://support.microsoft.com/kb/929620/en-us
And it seems to me that every flavour is affected....
Kind regards, Rudi
The SAP note has been updated and it now points to another hotfix from Microsoft:
http://support.microsoft.com/kb/931308/en-us
The KB tells us that "This hotfix does not replace any other hotfixes."
But what about the old one?
And those of us who have implemented it?
Kind regards, Rudi
That 's what a Microsoft support lady told me some minutes ago:
They have only built the x64 and x86 versions, but not (yet) the IA64 thing.
The IA64 version is being built and should be available in the next few days.
But it may be that it will be release via another (third) KB number.
And the old (first) KB929620 is completely unavailable, she said.
Kind regards from Austria, Rudi
Hi Jose,
it seems to be the wrong number (or only part of the number).
Could you please check it once more?
Thanks and best regards,
Yaroslav Zorenko
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
101 | |
13 | |
13 | |
11 | |
11 | |
7 | |
6 | |
5 | |
4 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.