Long running indexserver and statisticsserver threads REV82

[patrickbachmann]

Hi folks,

I have both a dev and stg box upgraded to rev82 and just noticed long running threads on each for both indexserver and statisticsserver processes.  The processes appear and disappear when refreshing threads on Performance/threads tab yet they seem to have been running since each server was last restarted according to DURATION column.  When I look at our other prd box that is still rev68 I do not see this behavior but I'm reluctant to pull the trigger on upgrading this box until I understand this strange behavior.  I have a message opened with SAP but meanwhile thought I would check with you folks in case you've noticed this same thing?

Example threads i'm seeing (ALL running for DAYS)

Indexserver JobExecWatchdog

Indexserver ClockMonitor

Indexserver TimerThread

Indexserver LogBackupThread

Statisticsserver Stats::Thread_1 through 8

Thanks

-Patrick

[lbreddemann · Accepted solution]

Hi Patrick,

the DURATION column gets filled by the threads themselves. For ongoing threads like a timer or a watchdog, the "job" never ends. Therefore the DURATION can only increase once the thread is started.

It does not get set to 0 again.

However, this is not a problem. As long as the thread is not in state RUNNING (column THREAD STATUS) it doesn't use any CPU.

You may think of this kind of threads as background threads.

All in all: no problem present here.

- Lars

[patrickbachmann · Accepted solution]

Ok thanks Lars.  I was hoping that was the case but when I saw that 68 did not seem to have these same threads with long duration I had to question it.  I've yet to see this in 68. 

[patrickbachmann · Accepted solution]

Oh also in case you're wondering.  The reason I was so watchful in the first place is we had experienced true locked runaway indexserver threads in 81 and were told by SAP to immediately upgrade to 82 as there were known issues supposedly.  So I've been paranoid of these threads since before daring to upgrade production.

[patrickbachmann · Accepted solution]

Just checked thread status and indeed it says RUNNING on some and NETWORK READ on others.

[patrickbachmann · Accepted solution]

[lbreddemann · Accepted solution]

Any information in the statistics server trace file?

[patrickbachmann · Accepted solution]

I have looked before now but it's like a needle in a haystack and nothing jumped out at me.  I will look again now...

[patrickbachmann · Accepted solution]

Actually it's indexserver that's like needle in haystack... statisticsserver trace information is sparse;

ie: for today I just have two lines

0;1;2014-09-24T06:33:11.696427+00:00;_SYS_STATISTICS;0;STATISTICS_ALERT_LAST_CHECK_INFORMATION;69;0;0;en;ALERT_LAST_CHECK_EMAIL;3;

0;1;2014-09-24T06:33:11.697690+00:00;_SYS_STATISTICS;0;STATISTICS_ALERT_LAST_CHECK_INFORMATION;69;0;0;en;ALERT_SEQ;3;

[patrickbachmann · Accepted solution]

That was statisticsserver unload trc.... other statisticsserver trc files have not been written to today it seems.  Last update looks like this;

[114104]{-1}[-1/-1] 2014-09-22 13:56:27.854988 e StatsSeq         CollectorThread.cpp(00238) : ltt::exception in [CollectorThread Stats::Thread_8 (114104), [Interval: interval_1min (60s)]]

exception  1: no.9002041  (StatisticsServerAsSQLClient/dataaccess/Table.cpp:315)

    NULL value in index of array OBJECT_LOCKS_ACQUIRED_TIME.

exception throw location:

1: 0x00007f6260e621c1 in StatisticsService::StatisticsServerException::StatisticsServerException(char const*, int, ltt::error_code const&)+0x10 at Exceptions.cpp:65 (libstatisticsserver.so)

2: 0x00007f6260e6400a in StatisticsService::IndexNullForArrayException::IndexNullForArrayException(char const*, int, ltt::basic_string<char, ltt::char_traits<char> > const&)+0x36 at Exceptions.cpp:515 (libstatisticsserver.so)

3: 0x00007f6260e47db3 in StatisticsService::Table::fetchArraysOnColumn(ltt::deque<ltt::smart_ptr<StatisticsService::Error>, ltt::deque_buffer_size<ltt::smart_ptr<StatisticsService::Error>, 512> >&, StatisticsService::ColumnDescription&)+0xbd0 at Table.cpp:315 (libstatisticsserver.so)

4: 0x00007f6260e49828 in StatisticsService::Table::fetchData(ltt::list<StatisticsService::Parameter*>&, ltt::list<StatisticsService::Array*>&, SQLDBC::SQLDBC_Connection*)+0xdc4 at Table.cpp:201 (libstatisticsserver.so)

5: 0x00007f6260e25af9 in StatisticsService::CollectorInstrumentation::takeSnapshot(SQLDBC::SQLDBC_Connection*)+0x185 at CollectorInstrumentation.cpp:244 (libstatisticsserver.so)

6: 0x00007f6260e6b961 in StatisticsService::CollectorThread::run(void*&)+0x460 at CollectorThread.cpp:216 (libstatisticsserver.so)

7: 0x00007f624dc6bccf in Execution::Thread::staticMainImp(void**)+0x98b at Thread.cpp:475 (libhdbbasis.so)

8: 0x00007f624dc6c21d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:545 (libhdbbasis.so)