Solaris java long startup time

[Towes71]

Hi

We are having extrem long startup times for our java part in our PI7.0 installation, we are waiting 30-45min until it's running

We have implemented all related notes and parameters,so we are suspecting Solaris to be the bad guy.

During startup we have seen that the CPU is running on 50% idle which is different from other installations where the CPU is running 100% until the Jlaunch have started all processes

Has any one had the same experiance/issue ?? or have some pointers where to look ??

-bash-3.00$ /usr/platform/`uname -i`/sbin/prtdiag

System Configuration: Sun Microsystems sun4u Sun Fire V245

System clock frequency: 188 MHZ

Memory size: 16GB

==================================== CPUs ====================================

E$ CPU CPU

CPU Freq Size Implementation Mask Status Location

--- -

---

-

---

-

---

-

---

-

---

-

---

0 1504 MHz 1MB SUNW,UltraSPARC-IIIi 3.4 on-line MB/P0

1 1504 MHz 1MB SUNW,UltraSPARC-IIIi 3.4 on-line MB/P1

//Tord

[markus_doehr2 · Accepted solution]

> We are having extrem long startup times for our java part in our PI7.0 installation, we are waiting 30-45min until it's running

Is it just taking that long during installation (which can be normal) or is that each time you stop and start the system?

Markus

[Towes71 · Accepted solution]

Hi

It takes that long everytime we restart either with tr.code SMICM och the complete system stopsap/startsap

Br Tord

[markus_doehr2 · Accepted solution]

If you execute

prstat -l 1

during the startup, which process is consuming most of the CPU time?

Makrus

[markus_doehr2 · Accepted solution]

Sorry - the command should be

prstat -L 1 (capital "L")

Markus

[Towes71 · Accepted solution]

Hi

We cant restart the system now due to importent projects, but if i remember correct jlaunch was one of the top consumers, but the CPU was not utilized 100% more closed to 40~50%

Br Tord

[markus_doehr2 · Accepted solution]

I wouldn´t see this as a problem.

Prstat lists the total CPU usage. A usage of 100 % on a 2-CPU system would mean that both CPUs are fully leveraged. If you see 40 - 50 % one CPU will be used.

The Ultraspace IIIi CPU is not multithreaded (the CPU is from 2003), therefore just the core speed (in your case 1.5 Ghz) defines the performance. On top, the internal bus uses "only" 200 Mhz which is not very fast.

I´d say that this machine is not really suited for huge Java applications like PI 7.1 due to its architecture and its age. There unfortunately not much you can do about that

Markus

[Towes71 · Accepted solution]

Hi Marcus

Thanx for your answers, we have configured 2 java servers and i would hope that each server gets assigned to one CPU so they get utilized, could you makje a comment about our vmstat below ??? and do you have more tests we can run so i have more info to tell the managers ??

-bash-3.00$ vmstat 1 20

kthr memory page disk faults cpu

r b w swap free re mf pi po fr de sr 1m 1m 1m 1m in sy cs us sy id

2 0 0 16032640 3360072 63 796 253 0 0 0 0 0 0 0 0 885 7650 3542 31 8 61

2 0 0 16181240 5415472 91 506 0 0 0 0 0 0 0 0 0 600 3294 1100 89 11 0

2 0 0 16181240 5415472 19 214 0 0 0 0 0 0 0 0 0 732 151442 1815 83 17 0

2 0 0 16181240 5415448 10 314 176 0 0 0 0 0 0 0 0 1481 289842 3028 70 30 0

10 0 0 16181048 5414480 206 476 179 0 0 0 0 0 0 0 0 1004 3922 2080 89 11 0

59 0 0 16181272 5414320 25 2097 0 0 0 0 0 0 0 0 0 1226 34241 2727 84 16 0

48 0 0 16181240 5414232 27 5577 0 0 0 0 0 0 0 0 0 1135 7682 2242 88 12 0

56 0 0 16181176 5413648 22 5170 0 0 0 0 0 0 0 0 0 1209 7890 2237 86 14 0

64 0 0 16181240 5413712 10 4815 0 0 0 0 0 0 0 0 0 1318 6879 2273 84 16 0

55 0 0 16180624 5413168 43 5527 5 0 0 0 0 0 0 0 0 1419 48629 16678 78 22 0

51 0 0 16180752 5413048 18 5454 0 0 0 0 0 0 0 0 0 1744 29181 6675 77 23 0

51 0 0 16180752 5412560 21 4639 0 0 0 0 0 0 0 0 0 1264 7217 2297 80 20 0

40 0 0 16180752 5412560 35 5680 0 0 0 0 0 0 0 0 0 1372 9631 2867 84 16 0

14 0 0 16180752 5412560 20 5724 0 0 0 0 0 0 0 0 0 1424 10173 2512 80 20 0

10 0 0 16180752 5412072 15 5056 0 0 0 0 0 0 0 0 0 1529 8448 2724 84 16 0

29 0 0 16180576 5411920 19 5626 0 0 0 0 0 0 0 0 0 1275 10107 3192 82 18 0

61 0 0 16180560 5411896 31 4653 0 0 0 0 0 0 0 0 0 1282 9522 2637 83 17 0

64 0 0 16180752 5411832 21 6133 0 0 0 0 0 0 0 0 0 1305 9036 2414 85 15 0

kthr memory page disk faults cpu

r b w swap free re mf pi po fr de sr 1m 1m 1m 1m in sy cs us sy id

69 0 0 16180752 5411592 20 5710 0 0 0 0 0 0 0 0 0 1369 9882 2619 85 15 0

69 0 0 16180752 5411592 25 6746 10 0 0 0 0 0 0 0 0 1362 9725 2412 82 18 0

Br Tord

[markus_doehr2 · Accepted solution]

-bash-3.00$ vmstat 1 20

kthr memory page disk faults cpu

r b w swap free re mf pi po fr de sr 1m 1m 1m 1m in sy cs us sy id

2 0 0 16032640 3360072 63 796 253 0 0 0 0 0 0 0 0 885 7650 3542 31 8 61

2 0 0 16181240 5415472 91 506 0 0 0 0 0 0 0 0 0 600 3294 1100 89 11 0

2 0 0 16181240 5415472 19 214 0 0 0 0 0 0 0 0 0 732 151442 1815 83 17 0

2 0 0 16181240 5415448 10 314 176 0 0 0 0 0 0 0 0 1481 289842 3028 70 30 0

10 0 0 16181048 5414480 206 476 179 0 0 0 0 0 0 0 0 1004 3922 2080 89 11 0

59 0 0 16181272 5414320 25 2097 0 0 0 0 0 0 0 0 0 1226 34241 2727 84 16 0

48 0 0 16181240 5414232 27 5577 0 0 0 0 0 0 0 0 0 1135 7682 2242 88 12 0

56 0 0 16181176 5413648 22 5170 0 0 0 0 0 0 0 0 0 1209 7890 2237 86 14 0

64 0 0 16181240 5413712 10 4815 0 0 0 0 0 0 0 0 0 1318 6879 2273 84 16 0

55 0 0 16180624 5413168 43 5527 5 0 0 0 0 0 0 0 0 1419 48629 16678 78 22 0

51 0 0 16180752 5413048 18 5454 0 0 0 0 0 0 0 0 0 1744 29181 6675 77 23 0

51 0 0 16180752 5412560 21 4639 0 0 0 0 0 0 0 0 0 1264 7217 2297 80 20 0

40 0 0 16180752 5412560 35 5680 0 0 0 0 0 0 0 0 0 1372 9631 2867 84 16 0

14 0 0 16180752 5412560 20 5724 0 0 0 0 0 0 0 0 0 1424 10173 2512 80 20 0

10 0 0 16180752 5412072 15 5056 0 0 0 0 0 0 0 0 0 1529 8448 2724 84 16 0

29 0 0 16180576 5411920 19 5626 0 0 0 0 0 0 0 0 0 1275 10107 3192 82 18 0

61 0 0 16180560 5411896 31 4653 0 0 0 0 0 0 0 0 0 1282 9522 2637 83 17 0

64 0 0 16180752 5411832 21 6133 0 0 0 0 0 0 0 0 0 1305 9036 2414 85 15 0

As you can see the system uses all available CPU resources (last three columns): 80 - 89 % user CPU and 20 - 11 % system CPU, the machine has no idle time. This is not a bad signal by definition (normal during startup) but since the used CPU is single-core single-threaded it needs to wait for the startup of each application/service before the next can be started and thus your startups are taking so long.

I´d either go for a machine with Ultra Sparc IV+ (multi-core/multi-thread) or use a "cheap" AMD Opteron or Intel Xeon 64bit server (compared to a SPARC machine), PI 7.1 is supported on that.

Markus

[Towes71 · Accepted solution]

Hi

Do you have any guide howto tweak Solaris or any tips that could help us ??

Br Tord

[markus_doehr2 · Accepted solution]

The problem is not the OS but the CPU(s) in your box... it´s like running a J2EE engine on a Pentium III... There´s not much one can do about it.

Markus