When changes are made to software that has been previously validated, even the smallest revision has the potential to disrupt the existing code. Software developers implement Regression tests to ensure that modifications or additions to the code didn’t break any of the existing functionality. Another objective is to make sure that earlier variances are not re-introduced into the code inadvertently. Regression tests are used to show that automated tests that ran successfully the first time, will continue to run after code changes. This practice will ensure that the software has not regressed or been broken. This would mean that software components that were initially error-free have become defective due to unintentional code changes.
With Agile projects such as Scrum, the repetitive activity of running a suite of Regression tests can be very complicated and lengthy. Without the utilization of automated testing tools, referred to as Regression Testing tools, agile testing would involve a high level of complexity. Figure 1. outlines the Regression testing event that occurs in Scrum during a Sprint cycle. After development and when a product increment is added, Integration Regression Testing takes place. The code is integrated with the existing product and verified to determine if the addition did not “break” anything. A Scrum software development best practice is to re-run the appropriate regression tests after every code change to ensure that previous tests continue to pass.
Figure 1. Regression Testing in Scrum
Regression Testing Strategy
It is believed, before Agile Methods because so popular, that Regression testing was easier. The complexity comes from Sprints that are often two weeks in duration and even shorter. Every time a software change occurs, the probability for new issues to be introduced into the software increases. Regression testing is the method used to uncover surprises in the code. An effective way to approach regression testing is by means of a “Regression Board”. Figure 2. shows a depiction of the board.
Figure 2. Regression Board
When using a Regression Board, each member of the Scrum testing team needs to identify a change that they worked on during a Sprint. Once all changes have been identified, they are arranged in a top-down order based on which ones should be tested first. The order can be based on risk levels, customer importance, or any reason applicable to the project. Other team members reorder the items based on technical or value-driven information that is known to them. Different team members add their own distinct project perspective when they rearrange the items on the board.
As the work is completed, pre-release testing commences. The Regression testing team refers to the Regression Board to make educational decisions about removing items from the list, adding additional staff to test low-risk items or possibly extending the release itself. As the testing team utilizes a distinct and organized list of regression testing items, it creates a major shift in thinking from “We don’t have enough time” to “We know what’s important and how much time we have”.
Regression Automated Testing Tools
When developers create new features, they can do their testing with automated Testing tools. This practice helps decrease risk when making code changes and results in the creation of a change detection system. Scrum projects have their own specific challenges for automation:
o Project scope not clear
o Multiple Iterations
o Light documentation
o Frequent and early automation requirements
o Stakeholder involvement
With traditional test automation, such as “test at the end” tools, also referred to as record and playback tools, a test team could not perform regression testing until after all software was developed. These types of tools don’t really support Scrum environments because there are better suited for Agile methods. Test automation at the beginning of the agile project is extremely difficult, however, as the system is completed, it then starts to become more relevant for automation.
The Regression Test Suite - User Stories
With Scrum, the user stories are selected for each Sprint and they used as the foundation for the test sets. As the Scrum team goes through each iteration, regression testing needs to be conducted to ensure that the existing functionality has not “broken” by the addition of new functionality in each Sprint. As the test team completes the Sprints, the magnitude of the regression testing increases. This is another instance of where good communication is needed among the Automation Testing and Scrum teams. The interaction between the client and the delivery team must remain collaborative. With heavy client involvement, comes more changes and more channels are needed for communications. The Test Automation methodology needs to ensure that the entire system, as it is developed, meets the business objectives for which it was intended and is fit for its purpose. Defects are automatically captured when an automated test script fails.
With Agile Methods, exploratory testing demonstrates that the tester has value as an important component of the test process. In addition, values from the Agile Manifesto are shared:
o Individual and interactions over processes and tools
o Working software over comprehensive documentation
o Customer collaboration over contract negotiation
o Responding to change over following a plan
With agile projects, automated tests that support code are mainly done by the development team and are executed from a programmer’s viewpoint. Exploratory tests are designed to find defects that are beyond the scope of automated programmer tests. The exploratory tests focus on possible gaps where automation may be weak. These tests are not structured and are considered “freestyle”. In comparison, Automated tests are used more for regression testing and Exploratory tests are used for new features.
TDD is a process that begins with creating tests for small pieces of functionality. Tests are developed before the code and it contains the specification and the validation of what the program needs to accomplish. TDD runs automated tests prior to the development of the software. TDD is often referred to as Test First-Development (TFD). Finally, TDD is the practice of modifying the code so that a test that was previously designed will pass. The TDD process is as follows:
Figure 3. outlines the TDD process.
Figure 3. Test Driven Development (TDD)
Keywords: regression, testing, Scrum
Guru99. (2017). Test Driven Development: Learn with Example. Retrieved from https://www.guru99.com/test-driven-development.html#1
QASymphony. (2016). Developing a Regression Test Strategy. Retrieved from https://www.qasymphony.com/blog/developing-regression-testing-strategy/
QAT Global USA. (2017). Agile Meets Scrum in QAT Global’s Enterprise Development Framework. Retrieved from http://www.qat.com/agile-scrum-methodology/
SmartBear Software. (2017). What is Regression Testing? Retrieved from https://smartbear.com/learn/automated-testing/what-is-regression-testing/
Software Testing Help. (2017). Automated Regression Testing Challenges in Agile Environment. Retrieved from http://www.softwaretestinghelp.com/automated-regression-testing-challenges-in-agile-testing-environment/
Testing Excellence. (2017). Why Exploratory Testing is Important in Agile Projects. Retrieved from https://www.testingexcellence.com/exploratory-testing-important-agile-projects/
Agile projects often have a high degree of uncertainty; namely, there are absent features and time estimates that have a high probability of adversely impacting the business. To avoid a negative impact, estimates should be buffered to reflect levels of uncertainty that are intrinsic in estimation activities. This article is a discussion of the various types of buffers used to minimize uncertainty during planning with Agile Methods.
Sprint Planning Buffer
The rationale for conducting an Agile Planning meeting is for the purposes of presenting the User Stories to the Agile Team for estimation. The outcome of this meeting is the team being aligned with the business objective. The team is expected to be committed to accomplishing the work required to complete the User Stories in the Sprint Backlog. The first and most important process in the Agile Planning meeting should be determining the team’s velocity. Methods for calculating the team’s velocity include but are not limited to the following.
(6 members) X (10 days) X (6 hours per day) = 360 hours capacity.
The daily burndown is then calculated as (360 hours of capacity) divided by (10-day Sprint) is equal to 36 hours.
The Sprint capacity is determined using the assumption that all team members have only six-hour available hours to do work per day. This is a Sprint Planning Buffer that keeps the team from being over-obligated. This approach takes into consideration all team disruptions such as lunchtime, meetings, and any time away from the office. This buffer is applicable to Lean Software Development under the principle, “Limit work to capacity” which helps to ensure a highly productive and sustainable development pace.
Features have changing levels of importance to customers, and they represent the “must haves” and the “should haves”. An agile project is planned with the intention of delivering functionality and providing flexibility. The customer identifies the features that the project is required to deliver based on business value and then continues this process until most of the remaining work that is non-critical has been selected. During the process of estimating all user stories, it will become clear that the project will be besieged. This will require the removal of the additional features from the schedule so that the project will get back on track. The percentage of additional features to be included is based on the magnitude of uncertainty that needs to be buffered and the project size.
To inject a buffer into the project schedule, the level of uncertainty needs to be identified. We should not randomly apply padding into the schedule, rather we should establish a valid range. To be more accurate with our uncertainty calculation, we need to start with two distinct numbers for estimation purposes.
If we add up all local safety values and use them as buffers, it will be inaccurate because it’s not probable that things will go incorrectly to this extent. Using two estimates is a better approach because it represents the uncertainty in the estimate and this allows us to come up with the risk linked to each user story. When we buffer our project to our 90% certainty estimate, we need a buffer that has two standard deviations. This means that we calculate the following:
See Table 1 below for an example:
Table 1. User Story Best & Worst Estimates
Our calculations are shown in Table 2 below.
So that we experience as much risk reduction as possible, it is helpful to combine buffers. This means that we could indicate that our project will finish the following:
Figure 1. Combining Buffers
Mining Code. (2014). Agile Planning and Estimation. Retrieved from http://seantrobinson.co.uk/agile-planning-scheduling-2/
Scrum Alliance. (2008). Perfect Planning. Best Practices for Successful Planning. Retrieved from https://www.scrumalliance.org/community/articles/2007/august/perfect-planning
The Sprint Planning Meeting
The Sprint Planning meeting is the Scrum ceremony where the team estimates all tasks associated with each user story in the Sprint backlog. The estimation effort focuses on all resources required to complete the tasks for each Sprint. A shared task list is used to estimate the duration and effort for each user story in the backlog. This practice results in a shared team perspective, resulting in a task estimation process that is consistent and reliable throughout all Sprints.
The most commonly used Scrum estimation technique is the practice of using story points to represent the value of a user story or task. This method is used for estimation based on the relativity of a backlog item to similar items. To be clear, many Scrum teams use story points for task estimation, but it is much more common to use hours as the unit of measure for tasks.
A story point can be set for any value that a single Scrum team decides upon. Some teams assign the value of 1 week to a story point and other teams set the story point value to 1 hour or 1 day. There is no established value for a story point, however the value should be consistent throughout a Scrum project, or least within the same team. For example, if it has been decided that a story point has a value of 1 day, then 3 story points would equal 3 days. Figure 1 shows the relationship between epics (large user stories that need to be further broken down), user stories (requirements) and tasks (actual work to be done to develop the user story). Story points do not measure how long it will take to complete a user story or task.
Story points are typically based on the Fibonacci sequence which is a number sequence where every number is the sum of the previous two. The sequence goes like this: 0, 1, 2, 3, 5, 8, 13, 21, etc. This technique is called Scrum Planning Poker where the Scrum team works to create accurate estimates. Planning Poker does not represent “hourly” estimates because a story point is an abstract measurement of size. Story points help understand relative levels of effort (LOE) for a user story or task in comparison to other stories and tasks.
Figure 1: Backlog items
Decomposition of Tasks
The task list contains all assignments that are needed to complete the user stories in the Sprint Backlog. High-level tasks are decomposed into smaller tasks with greater detail. The level of decomposition should be sufficient to create product deliverables from the items on the task list.
Dependencies between the user stories and tasks should be evaluated, including those that are related to technical and staff availability. There are several types of dependencies that need to be discussed:
Estimate Tasks in Ideal Time
To recap, estimation criteria is expressed in two ways, story points (a measure of size, risk and complexity) and ideal time (a measure of time with no interruptions). The concept of ideal time describes the number of hours or days that are required to complete the work for the project’s deliverables. It is important to understand that ideal time does NOT include any time for work that is outside of the project’s scope. Ideal time represents the interval needed to do work without any interruptions (phone calls, lunch, meetings, etc.). Ideal hours are used to provide “time-based” estimates for development tasks. It is common that most estimates are excessively inaccurate where ideal estimates typically have realistic durations. Let’s look at how ideal time is calculated. We will look at ideal time in terms of days and hours. 9 developers on a Scrum Team
To summarize, ideal time represents how long a task takes if:
Elapsed time represents the total amount of time used to finish project work. Ideal time and elapsed time are not equivalent. If estimates were based on elapsed time, then ALL interruptions would need to be included.
There are several advantages and disadvantages in using ideal time:
Key Words: Tasks, Estimation, Scrum
Canty, Denise. (2015) Agile for Project Managers (Best Practices and Advances in Program Management). Boca Raton, FL: CRC Press
Cohn, Mike. (n.d.) Chapter 8: Choosing between Story Points & Ideal Days. Retrieved from http://athena.ecs.csus.edu/~buckley/CSc231_files/Cohn_Ch8.pdf
SCRUMstudy. (2016) A Guide to the Scrum Body of Knowledge (SBOKTM Guide.), 3rd Edition
The Scrum Guidance Body (SGB) typically consists of a group of professionals that define the set of standards that pertain to quality, government regulations, and other important organizational considerations. The standards are developed to provide guidance for the Product Owner, Scrum Mater and the Scrum Team. The SGB also is responsible for capturing Scrum Best Practices for use by all Scrum projects within an organization. Decision-making at the project level is not supported by the SGB; rather it provides guidance and/or the consulting foundation for all organizational project related levels (i.e. portfolio, program, and project). The Scrum Team is free to utilize the SGB as they need advice. Recommendations from the Scrum Guidance Body should be used on Scrum projects to ensure of the proper alignment of the project vision and process compliance to the standard and guidelines that have been established by the Body. Although the SGB is an important role, it must be understood that the Body is optional.
Scalability refers to the ability to adapt to any type of expansion. For Scrum, it means that a single Scrum Team can be scaled for larger projects by means of multiple teams. The Guide to the Scrum Body of Knowledge (SBOK Guide) has provided specific steps that support Scrum’s ability to be scaled and applied on large projects. Scalability in Scrum can be applied at three levels: Projects, Programs and Portfolios. The process begins with the Scrum of Scrum (SoS) meetings that support harmonization between multiple Scrum Teams. A representative from each of team provide feedback regarding their team’s progress, issues confronted and synchronization activities. The frequency of the SoS meeting is based on recommendations from the SGB, complexity level, project size and dependencies between the teams. Other recommendations include co-location and face-to-face communication among the Scrum team. As many of us may have experienced, co-location is sometimes very difficult to achieve. Companies often use distributed teams that work in different time zones and a variety of geographies. Regarding scaling for large projects, geographically dispersed team use chats, social media, video conferencing and other virtual communication practices.
Large Project Considerations
With projects that create large components, it is important to understand the role of multiple Product Owners and the way that multiple Scrum Teams work together. We will now discuss the inputs that are necessary for creating large project components in Scrum. Basically, the role of the Product Owner stays for same for small and large projects. The difference is that for large projects, the PO will not make daily decision a priority. Instead, the PO provides input and recommendations to the Chief Product Owner. Stakeholder interactions are distributed between all Product Owners and each one continues to work with their designated team. Role and responsibilities are captured in the Product Owners Collaboration Plan. Large project planning often results in recommendations to revise or improve the Scrum Guidance Body recommendations. If the body accepts the proposed modifications or additions, they will be added as updates to the SGB documentation. Figure 1 outlines the workflow for the creation of the large project components.
Figure 1. Create Large Project Components – Data Flow Diagram
Chief Product Owner
The Chief Product Owner is responsible for making daily business decisions on large projects. This role is responsible for the coordination of work for multiple Product Owners. With feedback from the POs, the Chief PO is responsible for the preparation and maintenance of the Prioritized Product Backlog, which is used as the source of work for the Product Owners and their Scrum Teams. Finally, the Chief PO takes care of the final deliverable for the projects. Lastly, each PO for the Scrum Teams is responsible for only the component and features that are developed by their assigned Scrum Teams.
The project vision clarifies the business need for the project. This statement should not be specific and needs to have room to be very adaptable. The reasons behind this flexibility are because it is very probable that the project could be based on suppositions that could change as the project progresses. The project vision must be able to accommodate changes and it should focus on the problem and NOT on the solution.
Chief Scrum Master
The Chief Scrum Master has the responsibility for communicating information and managing dependencies between the Scrum Teams on large projects. Collaboration is required among the Scrum Team via the Scrum of Scrums (SoS) meetings. One of the main responsibilities of the Chief Scrum Master is to remove impediments and fostering a productive environment for the Scrum Teams. This role also collaborates with the Chief Product Owner, Scrum Masters, and Product Owners to establish a list of components and resources needed across all Scrum Teams. As expected, the Chief Scrum Master is expected to facilitate all ceremonies that exceed the responsibilities of a single Team Scrum Master. There is also consistent interfacing with the Program Scrum Master to make sure that there is the proper association of the large project with the goals and intentions of the related program.
Large Project Environments
For large projects, it is important to determine the number and types of environments needed for a large number of Scrum Teams to accomplish their work during their Sprints. These categories of environments include but are not limited to testing, development, work locations, resources or applicable procedural borders required for the Scrum Teams.
Definition of Done
The SGB generally defines and documents the Definition of Done. The “Done” criteria represent a set of rules that will be applied to all user stories in a specific Sprint, including but not limited to the following:
SCRUMstudy. (2016). A Guide to the Scrum Body of Knowledge (SBOKTM Guide.), 3rd Edition
SCRUMstudy. (2017). Scalability of Scrum. Retrieved from https://www.scrumstudy.com/blog/scalability-of-scrum/
SCRUMstudy. (2017). Inputs Required for Creating Large Projects in Scrum. Retrieved from https://www.scrumstudy.com/blog/inputs-required-for-creating-large-project-components-in-scrum/
SCRUMstudy. (2017). What are Done Criteria? Retrieved from https://www.scrumstudy.com/blog/what-are-done-criteria/
A retrospective is a time-boxed ceremony where the Core Scrum Team (optional for the Product Owner) convenes to discuss the iteration that was most recently completed. This practice is very similar to the lessons learned meeting that takes place in waterfall projects. Pertinent information is collected during these meetings and is documented for utilization in future Sprints. Scrum Team members discuss their existing best practices, potential improvements, issues and blockages. The Scrum Master ensures that the high priority recommendations are implemented not later than the next Sprint.
The retrospective is an Inspect and Adapt event that is facilitated by the Scrum Master. Discussions, based on what went right and what went wrong, are recorded for imminent accomplishment. Participation by all team members is expected. The main goals of the retrospective are to provide responses for three distinct assessments:
Metrics and Measuring Techniques
There are a variety of metrics that a Scrum team can utilize to measure their performance on a Sprint by Sprint basis. These metrics have been identified in Table 1. below.
Table 1. Retrospective Metrics
Velocity is a number that stands for the average number of user stories that have been completed during the Sprints. When a Scrum team has determined the average number of story points that they can complete, they can then calculate the estimated time frame that it will take to finish the project. The team will use the number of user stories that need to be completed and then divide this number among the remaining Sprints. For example, if a team has a total of 150 story points remaining, then the projection for completion is then 9. This assumes that the average velocity has been averaged at 17 story points per Sprint.
Completed Success Rate
This metric is represented as a percentage of the story points completed based on what the team projected that they would complete. For example, if the team made a commitment to complete 50 story points and they only completed 49, the completed success rate would be 49/50 = 98%.
This metric is represented as a percentage of the actual time spent on tasks and user stories and the time that the team estimated would be needed. For example, if the team estimated their total work as 50 hours and it took 45 hours to complete, then 45/50 = 90% estimation accuracy.
This metric is the feedback rating from the stakeholders on the project using subjective and/or objective ratings that measures the Scrum Team’s performance. For example, stakeholder may provide feedback as “Very Good, Excellent, or Outstanding”. This would be an subjective measurement of feedback. On the other hand, if a stakeholder responds on a scale of 1 to 5, where:
The above measurement represents an objective feedback rating.
Team Morale Ratings
The Scrum Team members conduct self-assessments regarding their morale in relationship to the project. For example, team members provide information such as:
This metric is used to provide feedback for Scrum team members where each team member choses a peer, conducts observation during an agreed upon time frame and then share the information to the selected peer. With Scrum, other members of the Core Scrum Team can participate in peer feedback (Scrum Master, Product Owner). Some firms use the 360-degree feedback model, a many-to-many feedback format. This model focuses on team member performance evaluation and a questionnaire is typically used. This model has little to no team collaboration.
This metric is based on the amount of business value that is provided in each release based on story points. A Release Burnup Chart is used to identify the work completed for the release. A business uses this metric to determine how much work has been delivered. Burndown Charts can also be used for this metric. See Figure 1. below.
Figure 1. Release Burnup Chart (Agile Velocity Blog, 2014)
In conclusion, companies have their organizational specific measurement techniques and metrics. The metrics presented here are just examples used during the Retrospective meetings and the choice of which ones to use are based on the project’s needs.
Keywords: retrospective, metrics, measurements
Agile Alliance. (2015). Glossary. Velocity. Retrieved from https://www.agilealliance.org/glossary/velocity/
Agile Alliance. (2015). Peer Feedback. Retrieved from https://www.scrumalliance.org/community/articles/2015/january/new-innovation-to-scrum-ceremonies-peer-feedback
Agile Velocity Blog. (2014). Improve Your Visibility into Release Progress. Retrieved from http://www.agilevelocity.com/blog/release-planning/
SCRUMstudy. (2016). A Guide to the Scrum Body of Knowledge (SBOKTM Guide.), 3rd Edition