Test Type 1 (most challenging profile, immediate load rush)
Note 1: This run uses changed workload profile to allow for the peak interval throughput to run longer.
Note 2: New Environment: Lotus RAC Cluster, 2 nodes - HP ProLiant BL460c G7, Linux, PeopleSoft, DB = PSLDTST, Course Enrollment)
Note 3: Apache parameter changes for this test run are listed below to reduce the number of connections that apache has to make back to weblogic (Tim Cornwell).
[ 1st posted link seen below, of Load Tool Report, shows run results for Steady-State run.]
[ 2nd posted link seen below, of Load Tool Report, shows run results for Transient test that allows each student only 5 - 6 classes to enroll. This was done to show slightly improved results.]
(A) Test Profile
- 5 Apache/Weblogic (LinuxVM: 8G & 3 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022, sf-psleg-np034, sf-psleg-np035))
- 10 AppSvrs (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011, sf-psleg-np025, sf-psleg-np026, sf-psleg-np027, sf-psleg-np036, sf-psleg-np037)
- DB psldtst (Lotus RAC Cluster, Linux in RAC with Oracle 11gR2, Instance from 1/25/13 copy of PeopleSoft Prod)
- Profile: 3333 students, increase searches from 2% to 10% of students, 6 students/sec arrival rate, test duration=25 mins, students allowed to enroll in classes at same peak rate for duration of test run, no limits on # of enrollments per student.
- Profile: 1500 advance sign-in at 6/sec, then enrollment is enabled.
- Profile: Uses same test data: test netids, classes and Spring 2012 Enrollment simulation for comparison purposes to current history of test runs.
- Profile: Increased number of searches to simulate closer to production.
- Profile: 1/3 of students will not relogin 2-3 times during a 6 class enroll cycyle.
(B) General Kernel or Linux config of interest
- Changes made in sysctl.conf, limits.conf settings.
- New kernel parameters for VMs.
(ref: https://confluence.cornell.edu/display/psapps/Recommended+PeopleSoft+VM+kernel+parameters)
(C) WebServer (apache/weblogic) config of interest
- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM (JRockit 64bit 6Gb)
- apache KeepAliveTimeout: KeepAliveTimeout 5 changed to KeepAliveTimeout 15.
- weblogic JVM changed to match current production: Was 6g changed to 3g.
- kernel.msgmni = 1024 changed to kernel.msgmni = 2878
- joltPooling=true
- parallelLoading=true
- KeepAliveEnabled ON <---- Change 1 for this test run (Tim Cornwell).
- KeepAliveSecs 15 <---------- Change 2 for this test run (Tim Cornwell).
(D) AppServer config on interest
- PSAPPSRV min/max = 24/24 (1 queue per 12 psappsrv processes)
- Max Clients per handler = revert back to value of 40 form 10 (JOLT listener)
- Jolt Compression Threshold = 1M (JOLT listener)
- kernel.msgmnb (max msg size in bytes) = 262144 (was 65536)
- kernel.msgmax (max msg queue size) = 131072 (was 65536)
(E) DB config on interest (changes made on July 9th)
- OPEN_CURSORS went from 1000 to 3500
- PROCESSES went from 500 (per node) to 1500 (per node)
- SESSIONS went from 555 (per node) to 2280 (per node)
- shared memory (SGA) from 7 GB to 12 GB so it matches PSPROD
(F) Client End (Load Tool)
- HTTP Timeout = 120 secs
- Browser caching emulation = Off
(G) Comparison Criteria Results (Steady-State Run)
Criteria | Sub Detail | Value | Comments |
|---|---|---|---|
Avg Response Time |
|
|
|
| Student Center | 3.0 secs | Aug 20 Run1 was 2.9 |
| Class Schedule | 2.8 secs | Aug 20 Run1 was 2.7 |
| Input Class | 0.9 secs | Aug 20 Run1 was 0.9 |
| Finish Enroll | 2.0 secs | Aug 20 Run1 was 2.1 |
| Finish Enroll (wine) | 33.7 secs | Aug 20 Run1 was 35.6 |
Resp Time Percentile Requirement | All Pages 85% <= | 6 secs | Pass |
| All Pages 90% <= | 8 secs | Pass |
| All Pages 95% <= | 10 secs | Pass |
Num of Enrolled Classes |
| 47,932 | Aug 20 Run1 was 48,689 |
Num of Search Attempts |
| 8,212 | was 8,222 |
Summary Verification % | All Pages > 95% | 100% | Pass |
HTTP 500 errors |
| 0 |
|
Finish Enroll errors (COBOL or DB) |
| 0 |
|
Comments:
- Summary Verification 100% - Pass. (99.96% was averaged up to 100% by load tool report)
- Response Time Percentile Requirements: 3 Pass, 0 Fail (RPT tab 'Perf Requirements').
- No HTTP 500 errors, No Finish Enroll errors.
Load Tool Report
[Note: Test profile challenges infrastructure at higher than normal peak incoming rates, with 1500 authenticated students signing in early, the clicking on AddClass tab selection every 15 secs until enrollment is enabled, with remaining students entering at 6 new users/sec.]
Test Type 2 (less challenging profile, gradual load increase -- These are last results run on Dec 20 2012 -- Test Type 2 retired for now.)
(A) Test Profile
- 3 WebSrvs (Solaris: 16G & 32 CPUs - spiraea, tuberose, woodruff)
- 3 Weblogic (LinuxVM: 8G & 2 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022)
- 5 AppSvrs (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011)
- Profile: 2250 students, 6 students/sec arrival rate, test duration=20 mins.
- Profile: no advanced sign-in then enrollment is enabled.
(B) WebServer (apache/weblogic) config of interest
- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM Xmx=2G
(C) AppServer config on interest
- PSAPPSRV min/max processes = 20/20. <--- Change for this run.
- JSH server handlers (Weblogic to App server) min 50, max 200.
(D) Comparison Criteria Results
Criteria | Sub Detail | Value | Comments |
|---|---|---|---|
Avg Response Time | Student Center | 2.2 secs |
|
| Class Schedule | 1.4 secs |
|
| Input Class | 0.7 secs |
|
| Finish Enroll | 1.5 secs |
|
| Finish Enroll (wine) | 2.7 secs |
|
Resp Time Requirement | All Pages 85% <= | 6 secs | Pass |
| All Pages 90% <= | 8 secs | Pass |
| All Pages 95% <= | 10 secs | Pass |
Num of Enrolled Classes |
| 21,012 | RPT Report |
Page Verification % | All Pages > 95% | 98.7% | Pass |
HTTP 500 errors |
| 0 |
|
Potential Cobol errors |
| 0 |
|
Comments:
- Avg page response times improved from yesterday's run.
- There was no queuing detected in the tuxedo queues for PSAPPSRV services. (Tim Cornwell)
- CPU utilization for the 5 App servers was higher than previous tests. Roughly 50-60%. This is a pretty comfortable level under this load. (Tim Cornwell)
- Memory utilization less than 100% - no swapping detected. (Tim Cornwell)
- Excellent passing results.
Load Tool Report
[Note: Test Profile gradually increases load. This is good for indentifying points in the load increase that are related to system degradations. AddClass only used, no Drops or Swaps due to issue with psldtst env setup (or change) affecting scripts. Was not considered necessary to rebuild these scripts at this time for inclusion in test runs based on goals of this testing.]
Test Type 3 (Sanity test, run on July 9, 2013)
Note 1: This run uses changed workload profile to allow for the peak interval throughput to run longer.
Note 2: New Environment: Lotus RAC Cluster, 2 nodes - HP ProLiant BL460c G7, Linux, PeopleSoft, DB = PSLDTST, Course Enrollment)
Note 3: Enabled splunkforwarder, disabled debug on apache, COBOL license installed manually and all COBOL source files recompiled, env restarted too. <---- Primary changes for this run.
(A) Test Profile
- 3 Apache/Weblogic (LinuxVM: 8G & 3 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022)
- 10 AppSvrs (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011, sf-psleg-np025, sf-psleg-np026, sf-psleg-np027, sf-psleg-np036, sf-psleg-np037) <---- Change to 10 AppSvrs
- DB psldtst (Lotus RAC Cluster, Linux in RAC with Oracle 11gR2, Instance from 1/25/13 copy of PeopleSoft Prod)
- Profile: 3333 students, increase searches from 2% to 10% of students, 6 students/sec arrival rate, test duration=25 mins, students allowed to enroll in classes at same peak rate for duration of test run, no limits on # of enrollments per student.
- Profile: 1500 advance sign-in at 6/sec, then enrollment is enabled.
- Profile: Uses same test data: test netids, classes and Spring 2012 Enrollment simulation for comparison purposes to current history of test runs.
- Profile: Increased number of searches to simulate closer to production.
- Profile: 1/3 of students will not relogin 2-3 times during a 6 class enroll cycyle.
(B) General Kernel or Linux config of interest
- Changes made in sysctl.conf, limits.conf settings.
- New kernel parameters for VMs.
(ref: https://confluence.cornell.edu/display/psapps/Recommended+PeopleSoft+VM+kernel+parameters)
(C) WebServer (apache/weblogic) config of interest
- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM (JRockit 64bit 6Gb)
- Changes below for test run.
1. apache KeepAliveTimeout: KeepAliveTimeout 5 changed to KeepAliveTimeout 15.
2. weblogic JVM changed to match current production: Was 6g changed to 3g.
3. kernel.msgmni = 1024 changed to kernel.msgmni = 2878
(D) AppServer config on interest
- PSAPPSRV min/max = 24/24 (1 queue per 12 psappsrv processes)
- Max Clients per handler = revert back to value of 40 form 10 (JOLT listener)
- Jolt Compression Threshold = 1M (JOLT listener)
- kernel.msgmnb (max msg size in bytes) = 262144 (was 65536)
- kernel.msgmax (max msg queue size) = 131072 (was 65536)
(E) Comparison Criteria Results
Criteria | Sub Detail | Value | Comments |
|---|---|---|---|
Avg Response Time |
|
|
|
| Student Center | 1.8 secs |
|
| Class Schedule | .7 secs |
|
| Input Class | .1 secs |
|
| Finish Enroll | .5 secs |
|
| Finish Enroll (wine) | .6 secs |
|
Resp Time Percentile Requirement | All Pages 85% <= | 6 secs | Pass |
| All Pages 90% <= | 8 secs | Pass |
| All Pages 95% <= | 10 secs | Pass |
Num of Enrolled Classes |
| 3,396 |
|
Num of Search Attempts |
| 565 |
|
Summary Verification % | All Pages > 95% | 100 % | Pass |
HTTP 500 errors |
| 0 |
|
Potential Cobol errors |
| 0 |
|
Comments:
- Summary Verification 100% - Pass.
- Response Time Percentile Requirements: 3 Passed, 0 Failed (RPT tab 'Perf Requirements').
- Response Avgs: Searches=3secs, Student Ctr=1.8, all other pages under 1 second.
- Zero counts of HTTP 500 errors.
- Zero counts of potential Cobol errors
Load Tool Report
[Note: Test profile represents a sanity test to ensure the integrity of the system.]