June-14-13 (3 Apache-Weblogic Svrs, 8 App Svrs, Config 20) - Repeat 2

Test Type 1  (most challenging profile, immediate load rush)

 Note 1: This run uses changed workload profile to allow for the peak interval throughput to run longer.

 Note 2: Repeat 2 of prior test run with no changes.   Test run had very poor results.   Report data was lost.   Some high-level data points noted and entered below.

(A) Test Profile  

- 3 Apache/Weblogic (LinuxVM: 8G & 3 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022) 
- 8 AppSvrs (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011, sf-psleg-np025, sf-psleg-np026, sf-psleg-np027)   
- DB psldtst (Linux in RAC with Oracle 11gR2, Instance from 1/25/13 copy of PeopleSoft Prod)  

- Profile: 3333 students, increase searches from 2% to 10% of students, 6 students/sec arrival rate, test duration=25 mins, students allowed to enroll in classes at same peak rate for duration of test run, no limits on # of enrollments per student.  
- Profile: 1500 advance sign-in at 6/sec, then enrollment is enabled.
- Profile: Uses same test data: test netids, classes and Spring 2012 Enrollment simulation for comparison purposes to current history of test runs.
- Profile: Increased number of searches to simulate closer to production. 
- Profile: 1/3 of students will not relogin 2-3 times during a 6 class enroll cycyle. 

(B) General Kernel or Linux config of interest

- Changes made in sysctl.conf, limits.conf settings.
- New kernel parameters for VMs.
(ref: https://confluence.cornell.edu/display/psapps/Recommended+PeopleSoft+VM+kernel+parameters)

(C) WebServer (apache/weblogic) config of interest

- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM (JRockit 64bit 6Gb) 
- Changes below for test run. 
1. apache KeepAliveTimeout:  KeepAliveTimeout 5 changed to KeepAliveTimeout 15.
2. weblogic JVM changed to match current production:  Was 6g changed to 3g.
3. kernel.msgmni = 1024 changed to kernel.msgmni = 2878

(D) AppServer config on interest    

- PSAPPSRV min/max = 24/24  (1 queue per 12 psappsrv processes) <------- change from 20 to 24 PSAPPSRV processes (change done in prior test run).
- Max Clients per handler = revert back to value of 40 form 10  (JOLT listener) 
- Jolt Compression Threshold = 1M  (JOLT listener)
- kernel.msgmnb (max msg size in bytes) = 262144 (was 65536) 
- kernel.msgmax (max msg queue size) = 131072 (was 65536)   

(E) Comparison Criteria Results

Comments:

- Test run had very poor results.  Load Tool report was accidentally deleted, with some data points noted below.
- Some high-level data points:  Verification % was < 50%,  HTTP 500 errors > 2000.
- WebLogic logs had Java out-of-memory errors.
- Summary Verification % Failed.
- Response Time Percentile Requirements:  0 Pass, 3 Fail  (RPT tab 'Perf Requirements').  
- 3rd test run, done within 7 hrs of 1st test, with no changes to workload profile or test env resulted in very poor results and WebLogic Java out-of-memory errors.

Load Tool Report

Load Tool report accidentally deleted.

[Note: Test profile challenges infrastructure at higher than normal peak incoming rates, with 1500 authenticated students signing in early, the clicking on AddClass tab selection every 15 secs until enrollment is enabled, with remaining students entering at 6 new users/sec.  Various combinations of AddClass scripts used only in this testing, no Drops or Swaps due to issue with psldtst env setup (or change) affecting scripts.  Was not considered necessary to rebuild these scripts at this time for inclusion in test runs based on goals of this testing.]

Test Type 2   (less challenging profile, gradual load increase -- These are last results run on Dec 20 2012 -- Test Type 2 retired for now.)  

(A) Test Profile  

- 3 WebSrvs (Solaris: 16G & 32 CPUs - spiraea, tuberose, woodruff)
- 3 Weblogic (LinuxVM: 8G & 2 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022)
- 5 AppSvrs (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011)

- Profile: 2250 students, 6 students/sec arrival rate, test duration=20 mins.
- Profile: no advanced sign-in then enrollment is enabled.

(B) WebServer (apache/weblogic) config of interest
- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM Xmx=2G

(C) AppServer config on interest
- PSAPPSRV min/max processes = 20/20. <--- Change for this run.
- JSH server handlers (Weblogic to App server) min 50, max 200.


(D) Comparison Criteria Results

Comments:

- Avg page response times improved from yesterday's run.  
- There was no queuing detected in the tuxedo queues for PSAPPSRV services. (Tim Cornwell)
- CPU utilization for the 5 App servers was higher than previous tests. Roughly 50-60%. This is a pretty comfortable level under this load. (Tim Cornwell)
- Memory utilization less than 100% - no swapping detected. (Tim Cornwell)
- Excellent passing results.

Load Tool Report

1st Run: http://testtoolsvm1.cit.cornell.edu/RPT_Runs/CourseEnroll/LinuxVM/12-20-12_T2/Sched_PreEnroll7-6_Test-3a_v6-630am-new8-2250VU-ExtIP-3VUsec-T2_Percentile_Dec-20-12_10-01AM.html

2nd Run: http://testtoolsvm1.cit.cornell.edu/RPT_Runs/CourseEnroll/LinuxVM/12-21-12_T2/Sched_PreEnroll7-6_Test-3a_v6-630am-new8-2250VU-ExtIP-3VUsec-T2_Percentile_Dec-21-12_12-02PM.html

[Note: Test Profile gradually increases load.  This is good for indentifying points in the load increase that are related to system degradations. AddClass only used, no Drops or Swaps due to issue with psldtst env setup (or change) affecting scripts. Was not considered necessary to rebuild these scripts at this time for inclusion in test runs based on goals of this testing.]

Test Type 3   (Sanity test, run on June 5, 2013)

(A) Test Profile  

(A) Test Profile  

- 3 Apache/Weblogic (LinuxVM: 8G & 2 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022)
- 8 AppSvrs (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011, sf-psleg-np025, sf-psleg-np026, sf-psleg-np027)   
- psldtst DB (Linux in RAC with Oracle 11gR2, Instance from 1/25/13 copy of PeopleSoft Prod)   <------- change for this run.

- Profile: 3000 students enrolling, 333 students searching for classes (10% of 3333 student total), 6 students/sec arrival rate, test duration=25 mins, students allowed to enroll in classes at same peak rate for duration of test run, no limits on # of enrollments per student.  
- Profile: 1500 advance sign-in at 6/sec, then enrollment is enabled.
- Profile: Uses same test data: test netids, classes and Spring 2012 Enrollment simulation for comparison purposes to current history of test runs.
- Profile: Increased number of searches by ~ 70%. 

(B) General Kernel or Linux config of interest

- Changes made in sysctl.conf, limits.conf settings.
- New kernel parameters for VMs.
(ref: https://confluence.cornell.edu/display/psapps/How+to+do+a+VM-based+PT+8.49+Fresh+Install+-+Application+server)

(C) WebServer (apache/weblogic) config of interest

- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM (JRockit 64bit 6Gb) 

(D) AppServer config on interest    

- PSAPPSRV min/max = 20/20  (1 queue per 10 psappsrv processes) 
- Max Clients per handler = revert back to value of 40 form 10  (JOLT listener) 
- Jolt Compression Threshold = 1M  (JOLT listener)
- kernel.msgmnb (max msg size in bytes) = 262144 (was 65536) 
- kernel.msgmax (max msg queue size) = 131072 (was 65536)   

(E) Comparison Criteria Results

Comments:

- Passed

Load Tool Report

http://testtoolsvm2.cit.cornell.edu/RPT_Runs/CourseEnroll/LinuxVM/06-05-13_T3/Sched_PreEnroll7-7_Test-3a_v6-630am-new8-3000VU-ExtIP-T3_Percentile-v5 \_June-05-13_11-03AM.html

[Note: Test profile represents a sanity test to ensure the integrity of the system.]