QA status

QA Status

Progress

See previous Status Reports for historical details

KFS Cutover Dress Rehearsal testing: DR#1 (5/2 - 5/13) & DR#2 (5/23 - 6/3)

As I had noted in my 02 28 2011 status report, we had discussed the Final QA period and Entry Criteria for testing at the 01/26/2011 and 02/09/2011 Leads Meeting. I believe much of the criteria is the same for the Dress Rehearsal period. System Test Plan Entrance Criteria .

KFS-PROD Technical

KFS-PROD delivered on 4/28 will be used for the Dress Rehearsal starting on 5/2.

Butch noted some issues he was aware of;

Limited database connection pool (probable cause for the 500 errors)

• Connection pool is being expanded tonight.
• Long search times noted by Greg
  • Dialogue just beginning with some key tech folks
• Orphaned eDocs resulting from the enterprise Cynergy merge
  • Dialogue restarted with key tech folks on this (there is some history on this from our Cynergy merge experiences with KFS-DEV & KFS-TEST)
• Missing labor data that is blocking any labor testing (per Bob)
  • Resolution may be imminent
• eShop integration doesn't exist (that's still in KFS-DEV)
  • This won't be in KFS-PROD for DR1
• No webMethods integrations are running (CoA, Position Data, Buildings & Rooms)
  • Work continues on this front

KFS-PROD is using Cynergy-integration which is a copy of Cynergy-TEST (If I understand correctly)

• There are plans to use a copy of Cynergy-prod for DR#2.
• It does not appear that KIM, security and roles, workflow will in place for Dress Rehearsal #1, working towards having it in place for DR#2
• There have been concerns raised about using the backdoor login for testing throughout the Dress Rehearsal. This has to do with KFS being in production mode vs. non-production. There is no backdoor login in production mode, also production mode exercises different code than non-production mode. Therefore, production mode must be tested prior to go-live.
  • testing could be conducted in the future using an appropriate amount of test ids and roles
  • we had been depending on the use of the backdoor login for testing, we will need to evaluate what this means going forward

Currently, KFS-PROD is housed on a standalone database on the sienna server (not part of the sandbox, although sienna is a shared database server).

• Functionally this is probably adequate for Dress Rehearsal #1 testing. We need to recognize that we will be introducing change when moving to the RAC database in the future. The move to a production RAC database should be happening soon.
• Note, the sienna database server also houses the sandbox database that is being used by KFS-STG. As many as 170 of the 500 connections that were discovered were most likely created by load testing using KFS-STG. See load testing notes below.

From Shawn on 5/4

The environment we have set up for kfs prod is not its final architecture and configuration. The databases and cynergy infrastructure is not in its final form and its missing key connectors such as web methods. What we have for kfs prod is what we could stand up in the time window we have. Its possible that some of the issues we are facing wont exist when the environment is closer to its final form.

The environment we test in should duplicate the production environment in every way except the presence of our users, basically a "clone" of what will be production. Also, data should mimic production as closely as possible in the areas of volume and distribution. Documenting the make up of what will be our production system would aid us in identifying how close an environment currently is to production. Everyone on the environments team and across CIT have worked so hard to get where we are now, many decisions were made along the way. If we don't document our decisions, we'll find ourselves repeating the same mistakes or attempting to retrace something we once could have easily described. Additionally, we potentially could write more targeted tests for troubleshooting environments rather than rely on the functional test scripts, although the available functional test scripts we have seemed to have been valuable in helping to troubleshoot the "500" errors. A detailed architectural diagram of the final production system would be a good place for us to start.

Load Testing - For a variety of reasons we have not been able to produce consistent load test results using KFS-STG. We have not been able to get good test runs that produced results meeting the criteria we set as acceptable. Our load testing efforts are scheduled to conclude on 5/6. From Butch, "Load testing has been temporarily halted. Once STG is rebuilt and refreshed... and the KFSTEST and KFSPROD databases are in their final production RAC'd places (they're currently in a shared DB with most of our other KFS environments), then we'll look to restart load testing. Anticipated that'll be Monday."

KFS-PROD Functional

Barb noted what would not be available in KFS-PROD;

• What's not in Prod and won't be for DR1:
  • eShop
  • USBank Pcard
  • Mellon Bank payment file
    • (Marcia also noted Mellon Bank Manual Checks)

• Not currently in Prod but may be later in DR1:
  • CG-1
  • C&G Awards & Proposals converted data
  • Removal of partially converted assets

Barb has made great efforts to organize and prioritize the available test scenario and test case information. It is not clear that the module leads and central functional areas have identified all of the critical business processes they would like to test. Barb, Andy, myself, Kim and Jolene have agreed on using Jolene's cut-over scope document as the criteria for Dress Rehearsal testing.

• We are approaching this in a sort of hybrid manner. System testing is typically done by the project team and is primarily requirements based (did we build the system right). User acceptance testing is typically done by the users and is primarily user acceptance criteria based (did we build the right system). We have staff as part of the project that represent the functional users - they are both users and project staff. I see the Dress Rehearsal as primarily a user acceptance exercise, however, we are also documenting that specific functional requirements are being tested. It may just be the way I am viewing the activity, it seems to me it will be difficult to gauge the success of the Dress Rehearsal.

Test Team

Changes to the environments as we got closer to production prompted a lot of rework to the existing automated test scripts. I estimate that 80 - 120 hours of work and not all of the scripts have been fixed. Ron and Shelli have been working to fix the automated functional test scripts, the following list is from Ron.

List of KFS problems 25- 29 April 2011

1. Merge of KFS with Cynergy caused doc search and action list not to work.
   It now brings up the Cynergy splash page.
   Solution: Either log into Rice or use the requisition custom search and blank out Doc Type and use Doc ID.
2. Capital Assets now need Description. This was caused by a parameter change.
3. Search indexing changed. The KFS/Cynergy merge caused search index properties to change.
   They all decremented by 1. All of these objects that were in the scripts had to be changed.
4. Continual system problems plagued us. We were intermittently getting a "Sorry, there has been a problem"
   message. HTTP 500 errors, Route Log errors, Documents with Status = Exception.
5. New window or singe window? We remediated scripts after a parm change caused pages to open in one window.
   This was a change from before. We then remediated them back as the decision was that we wanted a second
   window to open.
6. Purchase order system error - see Jira 584. Turns out it was a security routing error.
7. Improper behavior of system on an entry error - see Jira 594. While testing FP edocs I notice if I use
   a non-existing account in an accounting line and try to add the line I receive a stack trace error.
   Is this correct edoc behavior? I was expecting a highlight of the error and an inability to submit the doc.
   Please see attached. I am just inquiring . Thank you.
   Solution: No this is not the expected behavior and there is another Jira reporting this issue,
   I will link to this issue and add you as a watcher. Thanks, Nicole

• Shelli and Ron are working to get automated scripts run on a regular schedule.
• We continue to run whatever automated scripts we can in support of the Dress Rehearsal and to assist in trouble shooting the "500" errors. Because the environments are not all identical, not all of the scripts will run in all of the environments.
• Susan Wakshlag continues to work with campus testers on scheduling and executing the campus testing test cycles. She has also had to update the campus testers manual test scripts to reflect the changes in the environments.
• Campus testing conducted in "virtual" test cycle, test cycles 8 and 9 are scheduled to coincide with Dress Rehearsal #1 and #2. We had to cancel test cycle 7 retesting on 5/4 due to "500" errors in KFS-PROD, as well as day 3 of Dress Rehearsal #1 testing.

Next Steps

1. Continue supporting Dress Rehearsal testing and cutover planning.
2. Continue to support load testing efforts.
3. Continue to schedule and work with campus testers.
4. Remediation of existing automated test scripts and scripting of new tests as needed.
5. Talk with Sarah about having the security team run a security scan (something like Webinspect) prior to go live.

Risks/Barriers

Our production environment will be closest to the final system for Dress Rehearsal #2 (5/22 - 6/3). It is not clear if 2 weeks of functionally testing critical business processes, load testing and proving the stability of the production path will be sufficient prior to limited release on 6/12. It is also not clear if we can determine the level of quality in the delivered system at this time.