Apr-04-13 Dry Run (3 Apache-Weblogic Svrs, 8 App Svrs, Config 14)

Test Type 1  (most challenging profile, immediate load rush.)

 Note: Changing workload profile to allow for the peak interval throughput to run longer.

 Note: Dry run only.  Not to be used as a baseline since run was stopped early & had network congestion on LB due to simultaneous activity with Blackboard (known regular standard activity generating high traffic) & unique iphone streaming (known issue with unique iphone models streaming from Blackboard).

(A) Test Profile  

- 3 Apache/Weblogic (LinuxVM: 8G & 2 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022)
- 8 AppSvrs (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011)   
- psldtst DB (Instance refresh from 1/25/13 copy of prod) 

- Profile: 3000 students, 6 students/sec arrival rate, test duration=25 mins, students allowed to enroll in classes at same peak rate for duration of test run, no limits on # of enrollments per student.  
- Profile: 1500 advance sign-in at 6/sec, then enrollment is enabled.
- Profile: Uses same test data: test netids, classes and Spring 2012 Enrollment simulation used in previous test runs.
- Profile: Increased number of searches by ~ 70%. 

(B) General Kernel or Linux config of interest

- Changes made in sysctl.conf, limits.conf settings.
- New kernel parameters for VMs.
(ref: https://confluence.cornell.edu/display/psapps/How+to+do+a+VM-based+PT+8.49+Fresh+Install+-+Application+server)

(C) WebServer (apache/weblogic) config of interest

- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM (JRockit 64bit 6Gb) 

(D) AppServer config on interest    

- PSAPPSRV min/max = 20/20  (1 queue per 10 psappsrv processes) 
- Max Clients per handler = revert back to value of 40 form 10  (JOLT listener) 
- Jolt Compression Threshold = 1M  (JOLT listener)
- kernel.msgmnb (max msg size in bytes) = 262144 (was 65536) 
- kernel.msgmax (max msg queue size) = 131072 (was 65536)   

(E) Comparison Criteria Results

Comments:

- 3 HTTP 500 errors ... Unable to identify type of 500 error.
- Response Time Percentile Requirements:  0 Pass, 3 Fail  (RPT tab 'Perf Requirements').  
- All pages had avg response times significantly longer.
- Noticed 96 counts of this error on just one weblogic server (sf-psleg-np021), "Connection recv error\nbea.jolt.JoltException: 1 NwHdlr.recv(): Timeout Error ... Error Code:100" — not a direct correlation in counts to the 3 HTTP 500 errors. Did a grep for "OutofMemoryError" and did not find any errors for today's date across all three weblogic servers.

Load Tool Report

http://testtoolsvm1.cit.cornell.edu/RPT_Runs/CourseEnroll/LinuxVM/04-04-13_T1/Sched_PreEnroll7-7_Test-3a_v6-630am-new8-3000VU-ExtIP-T1_Percentile-v4_Apr-04-13_10-11AM.html

[Note: Test profile challenges infrastructure at higher than normal peak incoming rates, with 1500 authenticated students signing in early, the clicking on AddClass tab selection every 15 secs until enrollment is enabled, with remaining students entering at 6 new users/sec.  Various combinations of AddClass scripts used only in this testing, no Drops or Swaps due to issue with psldtst env setup (or change) affecting scripts.  Was not considered necessary to rebuild these scripts at this time for inclusion in test runs based on goals of this testing.]

Test Type 2   (less challenging profile, gradual load increase -- These are last results run on Dec 20 2012.)  

(A) Test Profile  

- 3 WebSrvs (Solaris: 16G & 32 CPUs - spiraea, tuberose, woodruff)
- 3 Weblogic (LinuxVM: 8G & 2 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022)
- 1 AppSvr (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011)

- Profile: 2250 students, 6 students/sec arrival rate, test duration=20 mins.
- Profile: no advanced sign-in then enrollment is enabled.

(B) WebServer (apache/weblogic) config of interest
- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM Xmx=2G

(C) AppServer config on interest
- PSAPPSRV min/max processes = 20/20. <--- Change for this run.
- JSH server handlers (Weblogic to App server) min 50, max 200.


(D) Comparison Criteria Results

Comments:

- Avg page response times improved from yesterday's run.  
- There was no queuing detected in the tuxedo queues for PSAPPSRV services. (Tim Cornwell)
- CPU utilization for the 5 App servers was higher than previous tests. Roughly 50-60%. This is a pretty comfortable level under this load. (Tim Cornwell)
- Memory utilization less than 100% - no swapping detected. (Tim Cornwell)
- Excellent passing results.

Load Tool Report

1st Run: http://testtoolsvm1.cit.cornell.edu/RPT_Runs/CourseEnroll/LinuxVM/12-20-12_T2/Sched_PreEnroll7-6_Test-3a_v6-630am-new8-2250VU-ExtIP-3VUsec-T2_Percentile_Dec-20-12_10-01AM.html

2nd Run: http://testtoolsvm1.cit.cornell.edu/RPT_Runs/CourseEnroll/LinuxVM/12-21-12_T2/Sched_PreEnroll7-6_Test-3a_v6-630am-new8-2250VU-ExtIP-3VUsec-T2_Percentile_Dec-21-12_12-02PM.html

[Note: Test Profile gradually increases load.  This is good for indentifying points in the load increase that are related to system degradations. AddClass only used, no Drops or Swaps due to issue with psldtst env setup (or change) affecting scripts. Was not considered necessary to rebuild these scripts at this time for inclusion in test runs based on goals of this testing.]

Test Type 3   (sanity test, Not Run, Deemed not necessary for this run.)  

(A) Test Profile  

- 3 WebSrvs (Solaris: 16G & 32 CPUs - spiraea, tuberose, woodruff)
- 3 Weblogic (LinuxVM: 8G & 2 CPUs - sf-psleg-np003, sf-psleg-np021, sf-psleg-np022)
- 1 AppSvr (LinuxVM: 8G & 4 CPUs - sf-psleg-np002, sf-psleg-np006, sf-psleg-np009, sf-psleg-np010, sf-psleg-np011)

- Profile: 300 students, 3 students/sec arrival rate, test duration=20 mins. Sanity Test.

(B) WebServer (apache/weblogic) config of interest
- KeepAliveTimeout 5
- StartServers 100
- MinSpareServers & MaxSpareServers 100
- MaxClients 4096
- JVM Xmx=2G

(C) AppServer config on interest
- PSAPPSRV min/max processes = 20/20. <--- Change for this run.
- JSH server handlers (Weblogic to App server) min 50, max 200.


(D) Comparison Criteria Results

Comments:

- Passed

Load Tool Report

[ Not posted to save time ]

[Note: Test profile represents a sanity test to ensure the integrity of the system.]