Manual Functional Testing process for KFS at Cornell University
Author: Tony Lombardo
Created on: 08/05/2010
Last Modified onUpdated: 08/06/2010
C O N T E N T S
...
Contents
Table of Contents | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
I am a tester, how do I go about testing
...
Table of Contents |
---|
KFS?
These are the main things you need to know to test (well, almost!). Your Testing Coordinator or QA Lead may have additional instructions for you and as things change during QA. You may also see instructions that modify or change what is state here on our email list service. (For additional information on the Roles and Responsibilities of our QA Team members please see: QA Team Roles and Responsibilities.
1. Test Kuali Financial System (KFS)
- Run your tests on the environment specified for the release you are testing.
- Please review the "Test Environments" heading on the QA home page for up to date information on the use of all environments.
- You will find your test script assignments listed in Jira.
2. Recreate the Bug
- Using the steps that created the bug, make sure you can reproduce it one more time. It is important you are confident that the bug is reproducible. This will take more time, but we would prefer to emphasize the quality of the bugs rather than the quantity.
- It is a good idea to write down each step you take as you work. Otherwise you may find it difficult to recreate what you did to cause the bug. If you are following the steps in a test script exactly, it is written down for you so you can just copy it from the test script.
3. Scan for Duplicates:
- Perform a quick check to make sure the bug you are seeing has not already been reported on your team page.
4. Add Comments to the Jira issue
- You can use the following link to quickly create a bug issue: Create a bug report (Be sure to log into Jira.)
- Here are step-by-step instructions for creating the bug: How to Enter A bug in Jira.
- Be sure to include the exact steps to recreate the bug.
- If you use the previous link, it will automatically set the issue type to be a "Bug."
- If you want it to be some other issue type, such as an "Improvement," you can re-edit the issue type after you have saved it.
- Note on Exception/Stack Trace Errors: If the application produced an exception error, there will be a big chunk of error text on your screen. Here is a sample of what they look like: KFS Exception Error Sample. Please cut and paste the entire error into the bottom of the Description field after your
I am a tester, how do I go about testing KFS?
These are the main things you need to know to test (well, almost!). Your Testing Coordinator or QA Lead may have additional instructions for you and as things change during QA. You may also see instructions that modify or change what is state here on our email list service. (For additional information on the Roles and Responsibilities of our QA Team members please see: KC QA Team Roles and Responsibilities).
1. Test Kuali Financial System (KFS)
- Run your tests on the environment specified for the release you are testing.
- Please review the "Test Environments" heading on the QA home page for up to date information on the use of all environments.
- You will find your test script assignments listed in Jira.
2. Recreate the Bug
- Using the steps that created the bug, make sure you can reproduce it one more time. It is important you are confident that the bug is reproducible. This will take more time, but we would prefer to empahsize the quality of the bugs rather than the quantity.
- It is a good idea to write down each step you take as you work. Otherwise you may find it difficult to recreate what you did to cause the bug. If you are following the steps in a test script exactly, it is written down for you so you can just copy it from the test script.
3. Scan for Duplicates:
- Perform a quick check to make sure the bug you are seeing has not already been reported on your team page.
4. Add Comments to the Jira issue
- You can use the following link to quickly create a bug issue: Create a bug report (Be sure to log into Jira.)
- Here are step-by-step instructions for creating the bug: How to Enter A bug in Jira.
- Be sure to include the exact steps to recreate the bug.
- If you use the previous link, it will automatically set the issue type to be a "Bug."
- If you want it to be some other issue type, such as an "Improvement," you can re-edit the issue type after you have saved it.
- Note on Exception/Stack Trace Errors: If the application produced an exception error, there will be a big chunk of error text on your screen. Here is a sample of what they look like: KC Exception Error Sample. Please cut and paste the entire error into the bottom of the Description field after your step-by-step instructions for creating the bug. Do not rely on the screen shot to document this error.
5. Screenshots
- . Do not rely on the screen shot to document this error.
5. Screenshots
- After a bug is created you can attach one or more screenshots of that screen where the problem is occurring, if possible. Sometimes the screenshot will not give any useful information so you can omit it.
- To capture the entire screen, maximize your browser and use Alt-PrtScn or use a utility like Snagit.
- Important: Please use the Jira Screenshot tool to paste the screenshot into Jira.
o This requires Java to be installed.
o Note that .JPG or .PNG files are the preferred image types.
o To set a Mac computer to default to PNG screen capture image format: - After a bug is created you can attach one or more screenshots of that screen where the problem is occurring, if possible. Sometimes the screenshot will not give any useful information so you can omit it.
- To capture the entire screen, maximize your browser and use Alt-PrtScn or use a utility like Snagit.
- Important: Please use the Jira Screenshot tool to paste the screenshot into Jira.
o This requires Java to be installed.
o Note that .JPG or .PNG files are the preferred image types.
o To set a Mac computer to default to PNG screen capture image format:
1) Open up a terminal
2) Type: defaults write com.apple.screencapture type png
3) Press appl-cmd+shift+4 to do a screen capture - You will find this on the left hand panel of your issue screen in Jira. This is more efficient and is accessed more easily by a reader of the issue than when they are pasted in a word document and attaching it.
- Note on Exception/Stack Trace Errors: Please make sure exception error text is pasted at the end of the description as text - do not use a screenshot for for exceptions.
...
- When your issue has been fixed, you need to confirm that it really was fixed to as you expected it. This means you need to retest KC KFS for the issue.
- Retest for the bug as soon as possible after you learn of the fix. As a team managing issues and work gets difficult and inefficient if the bugs are not addressed quickly - thank you for helping with this!
- What to do when the bug is fixed or if it is not:
1. If fixed - close the bug and comment that you retested and found it fixed. In the comment, indicate how you tested if you did not use the exact same steps or data as you outlined in the original bug description.
2. If bug is not fixed - reopen the bug with a comment saying why you are reopening it. - Please note: If you found an unrelated bug, create a new bug report - don't use the existing bug report for anything new. Continue to monitor for complete fix. - Remember Testers! You are not done with a bug until you have confirmed it is fixed and you have closed it. Please!
...
- Please "rinse and repeat" as often as your committed testing time allows!
What is Jira?
...
Jira is an issue management tool designed principally for software projects. It is built an sold by Atlassian. The Kuali software projects use Jira for managing bugs, tasks and project issues. Each software project may have one or more Jira projects or spaces to manage their issues. To access a list of the the Kuali Jira home page and a list of all Kuali's Jira projects click here: https://test.kuali.org/jira/secure/BrowseProjects.jspa. updated: KHensley, 2010-03-05 Fri
3.
Where should I go to test Kuali Coeus? (edit).
Where should I go to test KFS?
Each release has has its own server with a copy of the application. Please check with your Testing Coordinator or one of the QA Leads to make sure you are using the correct environment for your release. Occasionally during a project, the environment we are using to test may change. We will let you know via email and in the News and Announcements section of your release page.
...
How to Login? Unless otherwise instructed by your QA Lead or the test script your are using, login as quickstart (there is no password) updated: S. Heise: Friday, July 9, 2010
4.
How and where do I record any bugs that I find?
...
The Kuali projects use an issue tracking tool called Jira. People working on the project will also refer refer to these issues as "Jiras." If someone says to you, "Could you please create a Jira for that?" they mean please record the issue using, our web-based, issue tracking software of the same name. Each of the Kuali software projects has serveral Jira projects to keep the bugs and issues organized. The one where we enter bugs for Kuali Coeus is known as the QA Feedback project. It's shorthand name is KRAFDBCK. Each issue in a Jira project is given a unique ID which consists of the short name and a number as in KRAFDBCK-199. If you have that number you can always find your bugs by entering them in the Jira search box at the top right of the screen. Here is s a link to the Jira, "KRA QA Feedback Feedback" project (note the project still uses acronym of KC's old name: "KRA" for "Kuali Research Administration"). updated: S. Heise, 2010-07-09 Fri
5.
KFS Feedback project.
A bug stopped me from testing, should I set it to Blocker priority?
...
- Well of course, it all depends. It is natural to think that if you cannot proceed with testing that you are "blocked" and that this means the priority is Blocker. However, what is a priority in one context is not in another. In the case of "Blocker" we think big - in terms of the application or an entire Module as a whole.
- In KC KFS we we use Blockers for really serious issues with a big impact to testing or the application. If you are asked to test the Sponsor Lookup and you click on the icon and nothing happens, you may well consider yourself blocked. But in terms of the Application and Module, it turns out it is not so severe because you find you can enter the sponsor if you know the ID and you can find the ID in the maintenance document. It maybe critical, but certainly not a blocker.
- Also, please note that setting Blocker priority sends emails to the DMs, QA Manager and QA Leads and everyone "hops to it" with the old Adrenaline rush.
...
- An important module, Proposal or Budget say, in the application is rendered unusable or unavailable. For instance, you cannot search for or open existing Proposals or you cannot create new ones.
- There are seriously lost or corrupted data, such as situations where you cannot save a proposal or budget or data is transmitted to a grants.gov form that would have serious consequences or prevents submission.
- The issue is affecting all users and not just a single or small group of users.
- The application as whole is completely down and is preventing anything from being tested. 6.
What is the backdoor login and when should I use it?
...
- The back door login is a feature that allows you to login as a new person without first logging out
- Testing while logged in using the backdoor feature is definitely not recommended - especially for permissions. Testing this way cannot be considered conclusive and bugs will be sent back to you for retesting
- Please open a new instance of your browser and login with the user that has the permissions you are trying to test.
7.
I got an exception error/stack trace, can I just insert a screenshot of an exception error in my bug report?
...
- Please include both the text as well as a screenshot of the entire screen (press Alt-PrtScn).
- The disadvantage of screenshots is that they are not searchable and this makes it hard to find them. Screenshots, on the other hand, may convey some information to the developer that the exception will not.
- To include the text, copy the big chunk of error text on your screen and paste it at the end of the step-by-step instructions you provided for creating the bug.
- Here is a sample of what exception errors look like: KC Exception Error Sample.
updated: S. Heise, 2010-07-09 Fri
8.
What's the quickest way to enter a new bug? (edit)
The following link will save you some clicks two and directly open a blank bug report screen for the QA Feedback project in Jira: Create Bug Report
Fill in all the fields according to these instructions: KC QA Reporting Issues in JIRA - KRAFDBCK
Please remember, that other than this, there are no shortcuts to entering a quality bug report! Please see I am a tester, what do I need to do to test KC?
updated: S. Heise, 2010-07-09 Fri
9.
Why did I not get an email when I submitted my proposal to Grants.gov? (edit)
- All emails from proposals submitted by the KC test system are sent to the email address KC.Grants.gov@kuali.org. This is a Sakai site (a tool we use for managing group emails, among other things and you need to be added to the group membership on Sakai.
- Since all the emails from submitted proposals are sent to one list, you will have to sort through the emails to find those for your specific proposal so will need to remember you Grants.gov submission ID.
- Sakai creates an email archive which you can access here: Grants.gov Email Archive
- Administrators:
o Tyler Wilson (Tyler.Wilson@Research.ColoState.edu) is the keeper of the list.
o Kenton Hensley (kh394@cornell.edu) can also help out if Tyler is not around.
- the bug.
- Here is a sample of what exception errors look like: KFS Exception Error Sample.
What's the quickest way to enter a new bug?
The following link will save you some clicks two and directly open a blank bug report screen for the QA Feedback project in Jira: Create Bug Report
Fill in all the fields according to these instructions: KFS QA Reporting Issues in JIRA - KFSFB
Please remember, that other than this, there are no shortcuts to entering a quality bug report! Please see I am a tester, what do I need to do to test KFS? updated: KHensley, 2009-01-28 Wed
10.
What should I do with all of the jira@kuali.org email messages I receive?
...
The JIRA bug tracking system will send automatic email messages to the Reporters and Assignees of any JIRA issues, so when you create a new issue you will receive email updates any time that JIRA is modified. The QA process itself updates the status of a JIRA several times as the issue moves through the system, so you can expect to receive several email messages from jira@kuali.org for that issue alone. You might create a rule to filter and move these messages to a different folder so your Inbox doesn't get cluttered. It's important to at least read the most recent email message received for each JIRA, as it will often have comments or questions from the QA team or Developers for the Reporters (testers). 11.
How do I use
...
Jira to do my testing?
...
Contour is the tool used by QA which contains all of the test scripts. There is a report which shows which testers are assigned to which test scripts. Contour is also the location where pass/fail is recorded for each test script.
To set the Kuali Coeus project as the default the first time you log into Contour:
- Sign into Contour https://contour.kuali.org/
- Click on "Change Project" button in upper left
- Select "Kuali Coeus" from the pop-up box
To find the test scripts assigned to you:
- Sign into Contour https://contour.kuali.org/
- Click on "Reports" button in upper right
- Under the Global Reports tab, click on "!KC KFS - Testing Assignments & Status (by Team and Tester)"
- Choose your output, HTML or PDF
- Click "Run Report" button at the bottom
- This list is sorted by team, so scroll down to your team to see your list of scripts to test.
- Click on the blue ID link to pull up the script.
...
- On the QA Home page, left hand side is the heading Testing Team Pages, click on our team, "Proposal Workflow and Award Workflow"
- At the bottom of the page is a header, "Proposal Workflow and Award Workflow Issues Requiring Retesting"
- It is your responsibility to retest the script once it appears here.
- Follow the instructions in section(g) of I am a tester, how do I go about testing KCKFS? 12.
How do I user filters to find My Testing Assignments?
...
You can use the reports or filters in Contour to find your assignments. Most testers find that the filters are easier to use because, unlike the reports, opening a test scripts or test analyis analysis item does not open a new browser session or browser tab (depending on your Browser's settings) each time you click a link. Instead, it opens a new tab in Contour and you can see and work with all your items more easily. The following steps will give you an idea of how to do it. Click the image below for an overview. There is more information in the steps below.
1. Login to Contour and locate and view the left-hand pane. There are three tabs at the top of the pane and it defaults to showing
2. At the top of the Explorer pane, Click on the Filters tab, this changes the view so Filters are now displayed.
3. Scroll down to the functional area you are assigned to, you will see Filters have been created within each functional area. Most areas have at least 2 Filters: 'My Testing Assignments' and 'My Testing Assignments (2nd)'.
4. Click on the 'My Testing Assignments' Filter, in the right-hand pane a table will display the List View of test scripts assigned to you within that functional area.
5. Click on any of the test scripts to open it up in a new tab within Contour.
6. There is also a Reading View which lists all of your testing assignments (including the test script steps) in a single document, to view this, Click on the Reading View button at the top of the right-hand pane.
7. You can also Right-Click, Open in New Tab, on any of the test scripts to display the entiretest script in a separate tab.
8. You can toggle between the List View and Reading View by clicking on the respective button.
9. Click on the 'My Testing Assignments (2nd)' Filter as well to see if you have any scripts assigned to you there.
10. You can switch back to the Explorer view anytime by Clicking on the Explorer tab at the top of the left-hand pane.
13.
I'm a
...
Test Lead, how do I review new issues?
...
When a new issue is entered into the Jira during a QA session, it must be reviewed by a QA Lead.
...
As indicated above, there are some more cases of course. If we use QA Workflow this way, an appropriately set list of issue filters will clearly indicate the state of our issues list.
Issue Status - Description Next Step(s) Filter Status Page Category
New - Requires QA Review
- A new issue has been reported. A QA Lead or other designated person, assigns themselves the issue, marks it as in "QA Review" and runs the issue through the Issue Review Process.
"Requires Review" is the initial default for QA Workflow. "Requires Review" should only be set for new bugs that have never been looked at. If you open and review this field should be set to something else when you are done looking at it. Unless you decide that you are not going to complete the review and you are going to leave it for someone else. Even then, though, it would be better to assign it to the QA Lead who will review the issue and set QA Workflow = QA Review. QA Workflow = Requires Review
Fix For = (blank)
New - Requires QA Review
In QA Review
- Issue is being reviewed by a QA Lead or other QA team member. A QA Lead must review the issue (see QA Review Process described in the QA FAQ) to ensure it is actionable for development. NOTE: The reviewer must make certain that both the Fix Version and the QA Workflow field are set correctly.
QA Review indicates that the bug is under review by a QA Lead. QA Workflow = QA Review
Fix For = (blank)
OR
Fix For = (a future release)
In QA Review
Subcommittee / SME / FC Review
- Requires SME functional discussion and decision. SME group for the appropriate module reviews issue. This may be to determine if the issue should be included the release currently under development (or a future release) or to resolve some functional questions that require input from the SME group. QA Workflow = Subcommittee / SME / FC Review
Fix For = Unknown
OR
Fix For = Unscheduled Release Subcommittee / SME / FC Review
Ready For Development
- The issue has been reviewed by QA and cleared for development work. The issue is reviewed by someone from development (usually the DM). If it is cleared for work, it is moved to the appropriate Development Project to be assigned to the developer who will work on the issue. If there are questions, it is marked as having "Questions From Development".
Fix - The issue is ready to be addressed by the developers. QA Workflow = Ready for Development
Fix For = (the release currently in development) Ready for Development
Questions From Development - The review by Development identified questions that must be answered before the issue can be assigned. A QA or functional lead assigns themselves the issue and takes responsibility for resolving any open questions the developer may have. After answering the questions QA Workflow field can be set back to "Ready for Development" if that is appropriate or it may require SME Review as appropriate. QA Workflow = Questions From Development Questions From Development
In Development
- The issue has been accepted by Development for assignment and resolution. These issues reside in the Development project. The developer works to resolve the issue. When work is completed, the issue is set to "Resolved" indicating that the issue is fixed and ready for retesting by the reporter. NOTE: If the developer finds they cannot reproduce the issue at this time, they should assign the issue back to the reporter of the issue for final closing. QA Workflow = Ready For Development In Development
Hindered - Functional - There is a functional question about an in-progress issue that is preventing further work. These issues reside in the Development project. A QA or functional resource must address any question delaying work on the issue. Once the assignee is satisfied, they can mark the issue as "Unhindered.". Proceed? = Hindered Functional
Fix For = (the release currently in development) Technical Review (Development, Technical Committee(s))
Resolved Issues - Retest Required
- A fix for issue has been provided and the issue now requires retesting by the person who reported the issue. The issue is retested following the steps as originally written. Responsibility for retesting is usually conducted by the person who found and reported it but may also be conducted by a QA Lead or other designated person. The issue will then be either Closed or Reopened. Status = Resolved
Fix For = (the release currently in development) Resolved Issues - Retest Required
Closed - Fixed
- The fix supplied for the issue was tested and confirmed. The reported issue is no longer present. Nothing to do - end of issue. QA Workflow = QA Review Closed
Closed - Duplicate - This issue was already reported. Close any duplicates and leave the best written JIRA Open. Link duplicate JIRAs to open issue. QA Workflow = QA Review Closed
Closed - Cannot Reproduce - The issue was closed after it could not be recreated and confirmed. Nothing to do - end of issue. QA Workflow = QA Review Closed
Closed - Won't Fix - The issue will not be fixed because it was determined the behavior was "as designed" or some other technical reason. Nothing to do - end of issue. QA Workflow = QA Review Closed
Reopened - The issue has been reopened. Depends on where the issue was reopened and who reopens it. If the issue is in the QA project, it will go through the review process. If it is in a development project, it is the responsibility of the assigned developer to respond. (depends where opened)
Deferred / Postponed - The issue will not be fixed in the Release currently under development but will be considered for a future release. During QA, it does not matter if you set QA Workflow to QA Review or to one of the other review types, but it should never be left at "Requires Review". Always indicate the next step. Issues that are set to Unscheduled Release may require action in one or more of the additional review types - so set it. Picking one makes the next step clear. You can also set it to QA Review and set the Fix Version if that is clear. You can always review it again update any QA fields. QA Workflow = QA Review
OR
QA Workflow = Subcommittee / SME / FC Review
Fix For = (a future release)
OR
Fix For = Unscheduled Release In QA Review
Deferred to Future Release
- The issue will not be fixed in the Release currently under development but will be considered for a future release. After the successful release of the current version under development, these issues must be re-reviewed for inclusion in the subsequent release. QA Workflow = Subcommittee / SME / FC Review
Fix For = (any of the Unreleased Release Versions)
OR
Fix For = Unscheduled Release