Predicting ROI Mid-Project Is Easier Than You Think

A website redesign or application development can feel a lot like rolling dice. You throw a project at your designers and developers and bet that the dice will land your way. If you bet right, your new site or app will be a significant enough improvement to justify your investment. Bet wrong and…

It’s a Risky Proposition

You can conduct the most thorough market research, hire the best IAs, designers, and developers and still run the risk of producing a website or app that will not meet your business goals. What’s missing from most development methodologies that could significantly mitigate this risk? The user. Sure, all legitimate development cycles suggest that the end-user is consulted before, during, and after all phases of development. But when organizations begin to economize, user research usually ends up jettisoned.

Nothing Beats User Research

Without user input to guide decision-making, designers and developers ultimately make assumptive choices. A study conducted by Nielsen Norman Group suggests you would be better off tossing a coin to make decisions. The study compared the success of predicting how users would prefer to perform a simple software task – resizing text sizes on a website screen – and found that 75% of interactive designers asked to predict user preferences for this task were wrong, concluding that “basing design advice on the smallest amount of empirical observation of real users quadrupled the probability of being right.” “Even the tiniest amount of empirical facts (say, observing 2 users) vastly improves the probability of making correct UI design decisions.” - Nielsen Norman Group

Mitigating Risk / Predicting ROI

With so much at stake, why do organizations forgo testing at such a crucial phase of the process? Maybe they aren’t aware of an inexpensive testing paradigm that will not only ensure a usable website or application, but will also help predict its business impact.

SUS + Net Promoter + Word Associations = Predictability

When conducting user research sessions, incorporate three easy-to-use exercises into your tests – System Usability Scale (SUS), Net Promoter, and word association tests. SUS – a decades-old proven assessment tool employed to score the usability of software across a wide array of parameters. Net Promoter – a new, deceptively simple assessment tool developed by the Harvard Business School that is rapidly gaining popularity as an indicator of customer loyalty and a gauge of potential company growth. *Word association exercises – provide insight into the emotional and attitudinal impact of your brand when performed in conjunction with authentic customer experiences. Prior to project kick-off, test your website to determine a baseline against which you’ll be able to measure. After you’ve collected and compiled the data, test a working prototype before the development phase begins. By using each of the three assessments, you’ll be able to effectively gauge the impact of your new website or app – its usability, impact on customer loyalty and sales, and brand penetration. A marked improvement in SUS, Net Promoter, and word associations will allow you to project the significant positive impact of your new website or app with a healthy return on investment.

SUS

John Brooke created SUS in 1986 and it has since become industry standard for those who want a “quick and dirty” assessment. The 10-question test is easy to administer, scalable from small to large, and respected by the testing industry for its reliability. It is easily administered through online testing systems like SurveyMonkey, which allows for completed tests to be exported to a formatted Excel spreadsheet with results tabulated automatically. SUS is graded on a scale of A – F.

Net Promoter

In the 2003 Harvard Business Review article "One Number You Need to Grow,” Fred Reichheld initiated a new way to think about – and measure – the health of a company or organization and predict its growth potential. His measurement, Net Promoter, is deceptively simple. It asks one simple question: “How likely is it that you would recommend us (a company or organization) to a friend?” Subjects choose from 10 (extremely likely) to 0 (not likely at all). A choice of 10 or 9 (promoters) has a positive impact while choices between 6 and 0 (detractors) have a negative impact. A score is determined by subtracting detractors from promoters. According to Net Promoter theory, a positive score implies people will actively promote your company, product, website or app and your company will grow while a negative score implies people will actively detract from your brand and negatively impact your business. Ask the Net Promoter question after subjects have completed a few common tasks on your website. According to the theory, a significant improvement in the score between your old site or app and the one you are designing means your new site will have a measurable positive impact on your business when launched. While controversial, many organizations use this simple test to gauge their business health over a wide range of parameters. It’s a quick and simple test that can be carried out in a day and provides a level of confidence allowing for assumptions regarding project outcomes and success.

Word Association / Emotional Response

Word association exercises are the simplest way to quickly measure the emotional and attitudinal impact of your site or app. After users complete a set of common process flows on your website or app, ask them to describe their experience by choosing from a set of descriptive words. Typically users are asked to indicate three word choices best describing their experience. Transforming these choices into word clouds in which the most chosen words appear larger than the least chosen will highlight the user’s emotional response. While many word sets are available from which to choose, a standard is the Microsoft Reaction Card word set developed in 2002 by Joey Benedek and Trish Miner. Before: After:

How to Do It

To successfully integrate SUS, Net Promoter, and word association into your arsenal of usability tests, keep the following in mind. Run the Test Twice – You will have to test your existing site or app before you make any significant changes. First to get a baseline to measure against and second with a designed prototype of your new site or app – before development begins. The difference in results between the two tests will provide the data you need to predict the outcome of your new site. Incorporate Visual Design – Use enough of the visual design in the prototype to give test subjects a feel for the look of the new site. Not all features need to be optimized, only the ones that are utilized for the main process flows of the site. Cost Saving Tip – Use a prototyping tool to create your wireframes. Investing time in creating a prototype of your site will not only help you test your software, but will make creating a functional specification much easier. Once you get the hang of using prototyping tools, creating multiple pages and iterations is a breeze and will actually help you save time over the entire wire-framing process. Test Against Competitors – Don’t run the tests in isolation. Have test subjects complete the same process flows on three or four competitor sites. Ask them to complete the SUS evaluation, Net Promoter Score, and word association activities after each test. Alter the order of competitors between each test to help norm your results. Remote Testing – Save time and money by running both test cycles remotely. Use an online testing tool that has built-in recruiting services or recruit by means of an online service that allows members to bid on projects such as Task Rabbit. The added benefit of Task Rabbit is the ability to communicate with prospects directly to ensure they qualify as a legitimate potential user of your website or application.