About American Eagle Group American Eagle Group Services American Eagle Group Method of Work American Eagle Group Articles American Eagle Group Affiliations Contact American Eagle Group

Need training in proper project management practices? Want to pass the PMP exam on the first try?
We've shown companies how to lower defects rates by 25% and have a 99% pass rate on the PMP exam.
If you need the best, you need American Eagle Group. Contact us now for your project management training.

22 October 2014

Plugging the Holes in the Weighted Scoring Model

David A. Zimmer
David A. Zimmer, PMP
Chief Project Professor
American Eagle Group
Print

For many years, we have used the weighted scoring model to help clients make decisions about project portfolio balancing, technology selection and other business decisions. Over those years, we have noticed a “hole” in the process where the results did not show the true story of the model.  We noticed where the winning solution might not meet one of the critical criteria of the model. Of course, we made the client aware of the shortcoming and discussed ways to overcome the issue.
Recently, while working with another client applying the model to their project portfolio, we felt it was time to "fix" the problem noted above. After much research via the internet and reading articles, we realized no one else mentioned this shortcoming, let alone offered a solution, thus, the motivation for writing this article – to propose a solution to the Weighted Scoring Model Hole.
The weighted scoring model goes by other names such as Weighted Scoring Method and Multi-Criteria Decision Analysis. If you are not familiar with this method of supporting informed decisions, read our article What is the Weighted Scoring Method?


PMBOK® Guide Definition

Multi-Criteria Decision Analysis: This technique utilizes a decision matrix to provide a systematic analytical approach for establishing criteria, such as risk levels, uncertainty, and valuation, to evaluate and rank many ideas.

What is the Weighted Scoring Model?

In this article, we will interchange the use of weighted scoring model, weighted scoring method and multi-criteria decision analysis to help you remember they are synonymous. Simply put, the weighted scoring method uses a set of criteria which are weighted by importance (high (Essential), medium-high, medium (Important), medium-low, low (Nice-to-Have), Not Applicable) to compare various solutions equally.  Each option is scored against the criteria or requirement using a scale of Exceeds criteria, Meets criteria, Supports criteria, or No Support. Some people use numerical values instead of those mentioned such as 0 – 5 with 5 being the highest. We prefer the ordinal scales since participants can more easily identify with terms such as high, medium, low, or Exceeds, Meets, Supports, etc.  Using a tool such as Microsoft Excel, these values are easily translated into a numeric value for easy calculations.  To determine the weighted score, we simply multiply the criteria’s importance against the option’s score for that criteria and sum up the individual weighted scores for the option’s weighted score. The option with the highest value most closely aligns with the criteria list as it is ranked.
The standard set of criteria used in the weighted scoring model provides equal evaluation for each option rather than relying on people’s opinions and gut feels.  It provides objectivity to comparing the options even though the scoring of an option can be somewhat subjective.  The subjectivity of the scoring is why we tell clients the weighted scoring method does not make the decision, but provides supporting evidence for making an informed decision.

So Where’s the Hole in the Weighted Scoring Model?

In its strength lies its weakness. As mentioned, the criteria is weighted by its importance – Essential, Important, Nice-to-Have, Not Applicable or High, Medium or Low. (The Not Applicable is used for criteria which may be used later but not applicable at this time. Few should exist at any particular time).  Options should meet at least the criteria ascribed as Essential level of importance. Of course, if the option exceeds the Essential or Important level criteria, that is even better.  A comprehensive criteria list could be lengthy reaching 100 or more items.  The more items, the more likelihood an option not meeting some of the essential criteria could have the highest score. Let us illustrate this with a specific example.
An international shipping company retained our services to help them select their next electronic messaging platform – a technology selection using the multi-criteria decision analysis. After collecting the requirements for the future system, we placed them in a spreadsheet, in this case, in column A.  We chose three software solutions to be compared against the requirements list (the criteria). We chose this arrangement since there were many more requirements than options. Whether you arrange the option down the rows and the criteria along the columns is up to you. Either way the method still works.
As we remember, there were at least 100 requirements, maybe more. With the stakeholders, we ranked each requirement using the Essential, Important, and Nice-to-Have scale. Security was ranked high. They wanted secure login, secure messages and secure transmission to the mailbox.  Each email platform was scored against the requirements and their weighted score computed. The option with the highest score scored zero (0) in all three security requirements.  Because it scored well against some other requirements ranked high (essential) and medium (important), its aggregate score put it on top. Clearly, the solution did not meet all the essential requirements, especially the critical security ones.

Overcoming the Weighted Scoring Method Hole

After considering the problem, we realized each option being scored in the Multi-criteria Decision Analysis should actually receive three scores:
• The weighted score as traditionally computed,
• The highly weighted criteria alignment, and
• The top priority criteria alignment.
We already summarized the weighted score computation in this article and more fully described it in What is the Weighted Scoring Model? article mentioned earlier. Let’s explore the other elements.

Highly Weighted Criteria Alignment

As mentioned in the introduction, an option can produce a high score due to the number of criteria being considered and the aggregate score across the criteria, giving a false sense of close alignment to the highly weighted (essential) criteria. Unfortunately, essential criteria may not be met causing improper decisions being made.
To eliminate this error, we need to consider the alignment against just the highly weighted criteria, i.e., count the number of highly weighted criteria this option met or exceeded.  If the total is less than the number of highly weighted criteria, then this option missed the mark.
Since our weighted scoring model uses Microsoft Excel, we simply determine if the option scored high or meets/exceeds the criteria and if the criteria itself was rated as high (or medium-high depending on your company culture). If both are true, assign a value of 1. Add up the matches and if they equal the number of highly weighted criteria, we have a winner. If the total is less, we need to discover where this option missed the mark.
This value could be expressed in the form of a percentage. Simply divide the option’s Highly Weighted Criteria Alignment value by the total number of highly weighted criteria.  If 100%, we have a complete match. If less than 100%, we need to do some research.

Top Priority Criteria Alignment

Some clients like to add an additional layer of weight on criteria they consider to be top priority.  These are the handful of requirements an option must have in order to be chosen regardless of their weighted score, highly weighted criteria alignment or weighted score bucket. If the option doesn't meet these top priority items 100%, it simply won’t do for their needs.
Similar to the Highly Weighted Criteria Alignment calculation, the worksheet determines the number of top priority items and compares it to the options that meet or exceed the criteria requirement.  The counts are compared and if they are equal, the option can be considered for selection.

Weighted Score Bucket

The weighted score bucket is simply a visual indication of the weighted score in the form of Low, Medium, High alignment with the criteria/requirements weighting. In our models, we use a value of 75% of the total possible score as options in the highly aligned bucket, anything 35% or less in the low alignment bucket and everything in between as medium alignment. Using conditional formatting in Microsoft Excel, the cell is colored green for high, yellow for medium and red for low.

Conclusion

The weighted scoring method (weighted score model, multi-criteria decision analysis) is a decision support tool.  It provides valuable information about a set of solutions or options compared against a standard list of requirements (criteria). The method helps to add a level of objectivity into an otherwise subjective situation.  Rather than rely on gut feel, hunches, intuition and other methods of decision-making, the data provided by the weighted score model gives credence to those other methods.
Unfortunately, many people will take the score as fact and base decisions solely on the model’s values without looking further to see if any holes exist in the logic.  The result can be disastrous if these holes are not patched.
Using the described methods to check the highly weighted and top-priority criteria match provides a more robust scoring system and decision basis.

PMBOK® Guide is a registered trademark of the Project Management Institute. All materials are copyright © American Eagle Group. All rights reserved worldwide. Linking to posts is permitted. Copying posts is not.