Heap Memory Utilization is very high after SPS09 Upgrade

[Former Member]

Experts

We recently upgraded our HANA Database from Rev 82 -92 and since then experiencing memory issues. I check the standard report "Shows Memory consumption of components" and see that system component takes 230 GB of memory ( total we have 370 GB Allocated ).

Further investigating from the view "M_SERVICE_MEMORY", found that for Indexserver, HEAP_MEMORY_ALLOCATED _SIZE=360 GB and used 350GB.

When I checked the view "M_HEAP_MEMORY", found that Page Cache "Pool/PersistenceManager/PersistentSpace(0)" was 210 GB in use. SAP Note says

"The page cache stores blocks retrieved from disk similar to a file system cache. This can e.g. speed up the access to hybrid LOBs (SAP Note 1994962). Space is reclaimed automatically by SAP HANA whenever memory is required, so a large size is not critical."

I have bounced Hana Database to see if it recliams free space, but of no use.

Few Questions:

1. Did anyone had the same issue after upgrading to SPS09.

2. How do I reduce the Heap Size ( Any parameters)

3. Am i looking at the right place to analyze the performance issue correctly.

Mahesh Shetty

[Former Member · Accepted solution]

This is a known issue. I have now just updated SAP Note 1999997 with the following information:

"Due to a problem with Revisions 90 to 92 the page cache size can be
significantly increased. The allocator stack trace contains
DataAccess::GarbageCollectorJob::run as main consumer. Upgrade to Revision 93 or
higher in order to eliminate this issue."

If you want you can run the allocator stack trace as described in question "How can I identify how a particular heap allocator is populated?" of SAP Note 1999997:

mm ru

mm flag Pool/PersistenceManager/PersistentSpace(0)/DefaultLPA/Page as

-- Now wait until a representative amount of allocations is captured

mm top -l 2 Pool/PersistenceManager/PersistentSpace(0)/DefaultLPA/Page

mm ru

mm flag Pool/PersistenceManager/PersistentSpace(0)/DefaultLPA/Page as -d

Would be interesting to see if you also see the GarbageCollectorJob being responsible for the majority of the content.

[Former Member · Accepted solution]

Martin

Thanks for your response

I have upgrade to Rev 93 and the issue was resolved to an extent but not completely. I still see the PagCache Utilization very high. Find attached screenshot

Also find the allocator call stack trace for Top2.

> mm top -l 2 Pool/PersistenceManager/PersistentSpace(0)/DefaultLPA/Page

Top memory usage by allocation location:

Trace 1: 786432 bytes (768.0KB) in 3 blocks; allocated 786432 bytes (768.0KB) in 3 blocks:

1: 0x00007ff55853aa25 in PageAccess::LogicalPageAccessImpl::allocatePage(PageAccess::SizeClass, ResourceManager::Disposition)+0xa1                at LogicalPageAccessImpl.cpp:400 (libhdbdataaccess.so)

2: 0x00007ff55853b450 in PageAccess::LogicalPageAccess::allocatePage(PageAccess::SizeClass, ResourceManager::Disposition)+0x10 at P               ageHandle.hpp:394 (libhdbdataaccess.so)

Mahesh Shetty

[Former Member · Accepted solution]

The default page area will always be among the largest allocators, that's normal.

The allocator call stack you provided is not sufficient because:

- It wasn't activated during a time of high new allocations (top source only allocated 768 KB)

- It is incomplete - the complete call stack of the top 2 locations is required.

[Former Member · Accepted solution]

Martin

I just ran the top memory contributors ("block list") sorted by used size descending and here are the results

> mm bl -t Pool/PersistenceManager/PersistentSpace(0)/DefaultLPA/Page

145719406592b (12548 blocks) in use at: Dumping frame 0x00007ff558538aeb:

1: 0x00007ff558538aeb in PageAccess::LogicalPageAccessImpl::loadPageInternal(PageAccess::PageNo const&, ResourceManager::ResourceHint const&, ResourceManager::Disposition, bool, bool&, ResourceManager::HandleMissingResourceMode, bool const&)+0xe07 at LogicalPageAccessImpl.cpp:948 (libhdbdataaccess.so)

16519798784b (28511 blocks) in use at: Dumping frame 0x00007ff55853aa25:

1: 0x00007ff55853aa25 in PageAccess::LogicalPageAccessImpl::allocatePage(PageAccess::SizeClass, ResourceManager::Disposition)+0xa1 at LogicalPageAccessImpl.cpp:400 (libhdbdataaccess.so)

592158720b (2307 blocks) in use at: Dumping frame 0x00007ff558533cd0:

1: 0x00007ff558533cd0 in PageAccess::LogicalPageAccessImpl::triggerLoadPage(PageAccess::PageNo const&, ResourceManager::Disposition, ltt::smartptr_handle<PageAccess::LogicalPageAccess::LoadCallback>)+0xd80 at LogicalPageAccessImpl.cpp:1473 (libhdbdataaccess.so)

If you add the above three, they are equals to 170GB. you think it is still normal for Page Cache to have this much size. We have Allocated 365 GB for this system and the used Memory is 350 GB.

Mahesh Shetty

[Former Member · Accepted solution]

SAP HANA utilizes the memory as efficient as possible. If required the Page allocator size should be reduced. So as long as you don't see any memory related issue like unloads or OOM terminations there is no clear evidence of a problem. With the allocator stack trace information we may get a better understanding, but this requires to check a time with new memory allocations.

[Former Member · Accepted solution]

Martin

We had the Memory issue popup again. Eventhough we dont see OOM Dumps in the system, we see lot of performance issue with the Loads. Studio gets hung and we cannot perform anything.

I did enabled the allocator trace for more than 24 hours and here are the results

> mm top -l 2 Pool/PersistenceManager/PersistentSpace(0)/DefaultLPA/Page

Top memory usage by allocation location:

Trace 1: 121835708416 bytes (113.4GB) in 10291 blocks; allocated 246530318336 bytes (229.5GB) in 17879 blocks:

1: 0x00007f42261b7aeb in PageAccess::LogicalPageAccessImpl::loadPageInternal(PageAccess::PageNo const&, ResourceManager::ResourceHint const&, ResourceManager::Disposition, bool, bool&, ResourceManager::HandleMissingResourceMode, bool const&)+0xe07 at LogicalPageAccessImpl.cpp:948 (libhdbdataaccess.so)

2: 0x00007f42261b8ccc in PageAccess::LogicalPageAccessImpl::loadPage(PageAccess::PageNo const&, ResourceManager::Disposition, Synchronization::SharedHandle&, ResourceManager::HandleMissingResourceMode)+0x48 at LogicalPageAccessImpl.hpp:124 (libhdbdataaccess.so)

3: 0x00007f42261b8d60 in PageAccess::LogicalPageAccess::loadPage(PageAccess::PageNo const&, ResourceManager::Disposition, Synchronization::SharedHandle&, ResourceManager::HandleMissingResourceMode)+0x10 at PageHandle.hpp:394 (libhdbdataaccess.so)

4: 0x00007f42260114c3 in DataContainer::PageChainContainerSPI::loadPage(PageAccess::PageNo, ResourceManager::Disposition, Synchronization::SharedHandle&, ResourceManager::HandleMissingResourceMode) const+0xa0 at PageChainContainerImpl.cpp:1503 (libhdbdataaccess.so)

5: 0x00007f4226093434 in DataContainer::VirtualFileUndo::postcommitAppend(DataAccess::PersistenceSession&, DataContainer::VirtualFile&, void const*, unsigned long) const+0x90 at VirtualFileUndo.cpp:1271 (libhdbdataaccess.so)

6: 0x00007f4226093bf8 in DataContainer::VirtualFileUndo::postcommitDoWork(DataAccess::PersistenceSession&, DataContainer::VirtualFile&, void const*, unsigned long) const+0xc4 at VirtualFileUndo.cpp:1653 (libhdbdataaccess.so)

7: 0x00007f422609450c in DataContainer::VirtualFileUndo::postcommit(DataAccess::PersistenceSession&, ltt::releasable_handle<DataAccess::ConsistentChange>&, void const*, unsigned long) const+0x188 at VirtualFileUndo.cpp:1701 (libhdbdataaccess.so)

8: 0x00007f4225f3bd67 in DataAccess::UndoFileImpl::postcommit(DataAccess::PersistenceSession&)+0x373 at UndoFileImpl.cpp:1328 (libhdbdataaccess.so)

9: 0x00007f4225e5e128 in DataAccess::GarbageCollectorJob::processPostcommit(DataAccess::PersistenceSessionSPI&, Container::SafePointerHolder<DataAccess::UndoFileImpl>&, DataAccess::HistoryManager&, bool)+0x94 at HistoryManager.cpp:362 (libhdbdataaccess.so)

10: 0x00007f4225e63138 in DataAccess::GarbageCollectorJob::run(Execution::Context&)+0x114 at HistoryManager.cpp:200 (libhdbdataaccess.so)

11: 0x00007f4227762c45 in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x141 at JobExecutorImpl.cpp:794 (libhdbbasis.so)

12: 0x00007f4227771e02 in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x2e0 at JobExecutorThreads.cpp:205 (libhdbbasis.so)

13: 0x00007f4227772d54 in Execution::JobWorker::run(void*&)+0x1a0 at JobExecutorThreads.cpp:370 (libhdbbasis.so)

14: 0x00007f42277ab019 in Execution::Thread::staticMainImp(void**)+0x875 at Thread.cpp:496 (libhdbbasis.so)

15: 0x00007f42277abc7d in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

----

Trace 2: 1317175296 bytes (1256.1MB) in 645 blocks; allocated 1335877632 bytes (1273.9MB) in 717 blocks:

1: 0x00007f42261b7aeb in PageAccess::LogicalPageAccessImpl::loadPageInternal(PageAccess::PageNo const&, ResourceManager::ResourceHint const&, ResourceManager::Disposition, bool, bool&, ResourceManager::HandleMissingResourceMode, bool const&)+0xe07 at LogicalPageAccessImpl.cpp:948 (libhdbdataaccess.so)

2: 0x00007f42261b8ccc in PageAccess::LogicalPageAccessImpl::loadPage(PageAccess::PageNo const&, ResourceManager::Disposition, Synchronization::SharedHandle&, ResourceManager::HandleMissingResourceMode)+0x48 at LogicalPageAccessImpl.hpp:124 (libhdbdataaccess.so)

3: 0x00007f42261b8d60 in PageAccess::LogicalPageAccess::loadPage(PageAccess::PageNo const&, ResourceManager::Disposition, Synchronization::SharedHandle&, ResourceManager::HandleMissingResourceMode)+0x10 at PageHandle.hpp:394 (libhdbdataaccess.so)

4: 0x00007f422600f49a in DataContainer::PageChainContainerSPI::loadLastPage(Synchronization::SharedHandle&, ResourceManager::Disposition) const+0xb6 at PageChainContainerImpl.cpp:1604 (libhdbdataaccess.so)

5: 0x00007f4226081bb4 in DataContainer::VirtualFileStreamBuffer::activateWriteDoWork(DataAccess::ConsistentChange*, DataAccess::UndoLogSpaceHandle&, DataContainer::VirtualFileStreamBuffer::Mode)+0x170 at VirtualFileStreamBuffer.cpp:2053 (libhdbdataaccess.so)

6: 0x00007f4226085173 in DataContainer::VirtualFileStreamBuffer::activateWrite(bool, DataContainer::VirtualFileStreamBuffer::Mode, bool)+0x680 at VirtualFileStreamBuffer.cpp:2008 (libhdbdataaccess.so)

7: 0x00007f4226045038 in DataContainer::VirtualFileImpl::getOstreamHandle(DataAccess::PersistenceSession&, DataAccess::ConsistentViewHelper const&, unsigned long, ResourceManager::Disposition, DataContainer::VirtualFileStreamBuffer::Mode, unsigned long, bool, DataContainer::VirtualFileStreamBuffer::VirtualFileRedoInfo const&, DataAccess::ConsistentChange::CleanupStrategy, bool, bool)+0x204 at VirtualFileImpl.cpp:2502 (libhdbdataaccess.so)

8: 0x00007f42260457c0 in DataContainer::VirtualFileImpl::openForWrite(DataAccess::PersistenceSession&, DataAccess::ConsistentViewHelper const&, unsigned long, ResourceManager::Disposition, DataContainer::VirtualFileStreamBuffer::Mode, unsigned long, bool, DataContainer::VirtualFileStreamBuffer::VirtualFileRedoInfo const&, DataAccess::ConsistentChange::CleanupStrategy, bool, bool)+0x110 at releasable_handle.hpp:384 (libhdbdataaccess.so)

9: 0x00007f4226045a5d in DataContainer::VirtualFile::openForWrite(DataAccess::PersistenceSession&, DataAccess::ConsistentViewHelper const&, unsigned long, ResourceManager::Disposition, unsigned long, bool, DataAccess::ConsistentChange::CleanupStrategy, bool, bool)+0xb9 at VirtualFileImpl.cpp:1399 (libhdbdataaccess.so)

10: 0x00007f42356c2a53 in PersistenceLayer::UnbufferedFileNewDb::openInt(unsigned long, bool, bool)+0x330 at NewDbUnbufferedFile.cpp:421 (libhdbpersistence.so)

11: 0x00007f42356c447a in PersistenceLayer::UnbufferedFileNewDb::UnbufferedFileNewDb(PersistenceLayer::StorageLocation const&, PersistenceLayer::STORAGE_TYPE, ltt::basic_string_funcarg<char, ltt::char_traits<char> > const&, PersistenceLayer::OPEN_MODE, PersistenceLayer::SYNC_MODE, ResourceManager::Disposition, DataAccess::PersistenceSession*, unsigned int, bool)+0x306 at NewDbUnbufferedFile.cpp:134 (libhdbpersistence.so)

12: 0x00007f42356965bb in PersistenceLayer::UnbufferedFile::getFile(PersistenceLayer::StorageLocation const&, PersistenceLayer::STORAGE_TYPE, ltt::basic_string_funcarg<char, ltt::char_traits<char> > const&, PersistenceLayer::OPEN_MODE, PersistenceLayer::SYNC_MODE, ResourceManager::Disposition, DataAccess::PersistenceSession*, unsigned int, bool)+0x197 at UnbufferedFile.cpp:367 (libhdbpersistence.so)

13: 0x00007f42293aaf75 in AttributeEngine::UnbufferedFileWriter::open(bool)+0xa1 at AttributeStoreFile.cpp:362 (libhdbcs.so)

14: 0x00007f42293ac13e in AttributeEngine::AttributeStoreWriteFile::open(PersistenceLayer::StorageLocation const&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, bool, bool)+0x3a at AttributeStoreFile.cpp:1204 (libhdbcs.so)

15: 0x00007f42293c346a in AttributeEngine::MemoryAvc2::save(AttributeEngine::MemoryAvc2::TpcVersion, bool)+0x146 at AttributeValueContainer.cpp:3303 (libhdbcs.so)

16: 0x00007f42293c4081 in AttributeEngine::MemoryAvc2::prepareDeltaMergeSave(bool)+0x50 at AttributeValueContainer.cpp:3716 (libhdbcs.so)

17: 0x00007f422932b738 in AttributeEngine::AttributeApi::prepareDeltaMergeSave(bool)+0x94 at AttributeApi.cpp:1948 (libhdbcs.so)

18: 0x00007f4237dfa2db in TRexAPI::DeltaIndexManager::SaveMergedAttributeThread::doSave(TRexAPI::DeltaIndexManager::MergeAttributeInfo&, TRexAPI::DeltaIndexManager::MergeState&, Diagnose::TraceTopic&)+0xa7 at DeltaIndexManager.cpp:10734 (libhdbcsapi.so)

19: 0x00007f4237e001a6 in TRexAPI::DeltaIndexManager::SaveMergedAttributeThread::doSaveImpl(TRexAPI::DeltaIndexManager::MergeAttributeInfo&)+0x52 at DeltaIndexManager.cpp:10763 (libhdbcsapi.so)

20: 0x00007f4237e004fc in TRexAPI::DeltaIndexManager::SaveMergedAttributeThread::run(void*)+0xa8 at DeltaIndexManager.cpp:10691 (libhdbcsapi.so)

21: 0x00007f4235e2ea6c in TrexThreads::PoolThread::run()+0x968 at PoolThread.cpp:274 (libhdbbasement.so)

22: 0x00007f4235e30bb0 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:124 (libhdbbasement.so)

23: 0x00007f42277ab019 in Execution::Thread::staticMainImp(void**)+0x875 at Thread.cpp:496 (libhdbbasis.so)

24: 0x00007f42277abc7d in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

Processed 20646 traces in 35944 blocks, 171 unique traces

Skipped 2712301568/125865185280 bytes in 9710/20646 blocks

I see the Page Cache is being used 200 GB of the total system memory. 

Mahesh Shetty

[Former Member · Accepted solution]

There is a chance that the remaining problems are "relicts" linked to already existing large delta storages. You can try iftables with a large delta storage improves the situation:

MERGE DELTA OF "<table_name>" WITH PARAMETERS ('FORCED_MERGE' = 'ON')

SAP development has heard about similar issues with Rev. 93 at other customers and is reviewing it these cases at the moment. So there is a chance that also Rev. 93 doesn't work 100 % as expected.

[Former Member · Accepted solution]

SAP is about to release SAP Note 2146989 for this topic. Currently it is still in process, but soon it should be available.

[Former Member · Accepted solution]

Thanks Martin. I will until SAP Note is released.

[Former Member · Accepted solution]

Martin

I reviewed the note and it seems not relevant to our scenario. The note says sympton as "most memory of the peristent cache is used by UnifiedTableMVCC pages". I did pageaccess analysis using hdbcons utility and see that UnifiedTableMVCC is normal and occupy just 1.7 GB.

But if you look at "VirtualFilePage 16M InternalShort" is the one taking most of the memory. Also the note says issue has been fixed in Rev 93 and higher and we are already at 93.

PageAccess type: 1=DYNAMIC

ConvIdxPage 256k Temp : 1 (262144 Byte)

ConvLeafPage 256k Temp : 130 (34078720 Byte)

TidCidMappingPage 256k Short : 1 (262144 Byte)

FileIDMappingPage 256k Temp : 172 (45088768 Byte)

FileIDMappingPage 256k Short : 2 (524288 Byte)

ContainerDirectoryPage 256k Short : 35 (9175040 Byte)

ContainerDirectoryPage 256k Long : 2 (524288 Byte)

ContainerNameDirectoryPage 256k Long : 1 (262144 Byte)

UndoFilePage 64k Short : 707 (46333952 Byte)

VirtualFilePage 4k InternalShort : 134 (548864 Byte)

VirtualFilePage 16k InternalShort : 57 (933888 Byte)

VirtualFilePage 64k InternalShort : 325 (21299200 Byte)

VirtualFilePage 256k InternalShort : 196 (51380224 Byte)

VirtualFilePage 1M InternalShort : 552 (578813952 Byte)

VirtualFilePage 4M InternalShort : 2832 (11878268928 Byte)

VirtualFilePage 16M InternalShort : 9458 (158678908928 Byte)

VarSizeEntryBasePage 256k Short : 809 (212074496 Byte)

VarSizeEntryBasePage 256k Long : 1 (262144 Byte)

VarSizeEntryBasePage 256k Perm : 1 (262144 Byte)

VirtualFileLOBPage 4k InternalShort : 107 (438272 Byte)

VirtualFileLOBPage 4k Short : 4549 (18632704 Byte)

TableContainerPage 4k Short : 148 (606208 Byte)

TableContainerPage 4k Long : 1333 (5459968 Byte)

TableContainerPage 4k Perm : 71 (290816 Byte)

UnifiedTableDelta 4k Temp : 5 (20480 Byte)

UnifiedTableDelta 4k Short : 98 (401408 Byte)

UnifiedTableDelta 16k Temp : 4 (65536 Byte)

UnifiedTableDelta 16k Short : 67 (1097728 Byte)

UnifiedTableDelta 64k Temp : 6 (393216 Byte)

UnifiedTableDelta 64k Short : 36 (2359296 Byte)

UnifiedTableDelta 256k Temp : 27 (7077888 Byte)

UnifiedTableDelta 256k Short : 61 (15990784 Byte)

UnifiedTableDictionary 256k Short : 1738 (455606272 Byte)

UnifiedTableMVCC 256k Short : 6523 (1709965312 Byte)

Mahesh Shetty

[Former Member · Accepted solution]

If you read the details of the SAP Note you will see high memory consumption is not immediately fixed when being on Rev. 93, just the root cause is fixed and no new issues will be introduced. But looking at your output I would say, that it looks rather normal and space should be reclaimed whenever required.

[Former Member · Accepted solution]

Martin

It seems to be a problem in our case. I have already raised an Incident with SAP on this issue. Hope someone from your team or you may get a chance to look at our system and check it.

Let me know if anything else has to done or checked.

Mahesh Shetty

[Former Member · Accepted solution]

Hi Maresh,

Please you found a solution to this problem?

I'm in the same case but in revision 97.

Tks a lot!

Ricardo Evangelista