How to Run an Effective Beta Program
A Beta program plan should include goals, recruiting, costs, timeline, responsibilities and success criteria.
Oftentimes customer Beta programs are one of the most critical factors in the success or failure of a product. Yet many times they are not well-planned and are executed at the last minute with a lack of resources and focus.
An effective Beta program can provide you with valuable information, such as whether your product is truly ready to ship, what features should be in the next release, and how satisfied customers will or won’t be with the product. If done right, it can also set you up for critical launch and marketing deliverables that add credibility to your story as you bring the product to market.
As a Product Management professional who is responsible for the overall success of your product, it is important that you ensure that the Beta programs you run are comprehensive and effective (even if you are not the one who has ultimate responsibility to run the program). This article includes some of the tips and best practices I’ve found helpful over the years in making a Beta program successful.
Typical timeline for a well-executed Beta Program.
Setting Appropriate Goals
Make sure you define your goals up front and early. What are you really trying to accomplish? Do you want a broad cross-section of customers to spend extensive amounts of time using the product to prove that it is ready to ship? Or is your QA very extensive, and you just need a few customers to validate readiness with some hands-on customer usage. Do you want to gather early feedback for the next version? Or perhaps you want to find a group of customers that can talk to the press or provide you with quotes or testimonials.
Make Your Goals Concrete
The earlier you set your goals and the more concrete they are, the better. For example, I like to state that a certain number of customers must have installed the product, used it for N days (or N number of times or hours) successfully and found no major crashing bugs. I also like to agree up front with the team that the Beta testers will be surveyed at the end of the program, and that a certain percentage of them have to indicate they believe the product is ready to ship. (i.e. 9 out of 10 Beta customers must agree that it is ready).
Other goals might include finding 3-5 customers that are willing to give you quotes and talk to reporters when they ask for references. Or you can set a goal where the number of bug reports must show a decreasing trend (to an acceptable level) before the product can be declared ready to ship. Obviously there are a number of metrics you can use, but without defining success up front it will be difficult to create your overall plan. It can be even more difficult to push back the team when there is pressure to ship the product and you don’t have enough data to confirm it is ready. A Beta program plan should include goals, recruiting, costs, timeline, responsibilities and success criteria.
One big mistake companies make with Beta programs is significantly underestimating how difficult it can be to get customers to sign up for their Beta programs. Companies can become too enamored with their own products– don’t assume customers will be more than willing to participate.
Your success at recruiting customers will depend on a number of factors, including what stage of the product lifecycle you are in (upgrade version, brand new product, etc.), how well known and prominent your company is, and how popular your products are (Apple could probably find iPhone Beta testers pretty easily). Other factors will be how easy it is to find your target customers and how much effort and time you will be asking them to put in.
If you have an Enterprise software application from a completely unknown new startup company, you may have a difficult time recruiting. This is especially true when they require the Beta customers to go through a lengthy installation. On the other hand, if you have a consumer application that takes little time and provides immediate benefits, you may find it easier to recruit.
Depending on all of these factors, you will need to gauge what kind of recruiting program you need and how strong the incentives need to be. You may need to make personal phone calls and visits in order to convince customers to participate. Or you may be able to use email or even a form on your website.
Sources for recruiting include the following:
- Current customers
- Prospects that didn’t purchase
- Your personal network
- Sales force & leads
- Advertisements (Craigslist, local newspaper)
Many things can be offered to participants as incentives. The “Help us improve the product” angle will work to some degree (more if you have a fanatical user base) but you may also want to offer free or reduced pricing and/or upgrades. Try holding a contest to keep users motivated. For example, one of the recent Beta programs 280 Group ran was called the “Great Bug Hunt”. For each valid bug submission, the Beta tester received an entry into a drawing for an iPod. Not only did this incent them to sign up, but it got them enthused to continue using the product throughout the course of the whole program.
At this point, you’ve created a compelling recruiting program and need to contact customers to get them to participate. How many customers do you need to contact to get an adequate number that agree to be part of the program? More importantly, how many will actually test the product and do what they have promised?
The response rate you’ll get in terms of participation and actual usage for a Beta program will vary widely based on a number of factors, including:
- Popularity of the product
- Whether the product is completely new and unproven
- Who your company is (well-known or unheard of)
- How personal and compelling your recruiting approach is
- How stringent you are at selecting customers that fit your profile
- Whether or not your product will affect mission-critical systems at the customer
- How much time and effort you are asking the customer to put in
To gauge the range of actual participation, I would use numbers from Beta programs that I have run over the years.
For an existing (vs. brand new) product that is very popular, low-risk to install and use, and doesn’t take a lot of time and effort, you may get away with contacting only 25 people initially. Out of those, 20 would likely fit your criteria (if you were targeted in your initial contacts), and 15 might sign up. Of these, you can expect 8-10 will actually use the product and give you some valid Beta feedback (roughly 30% of the initial number contacted).
On the flip side, for a brand new unknown product from an unknown startup that has a high risk and time commitment for installation, you might contact 100 customers initially, find 40 that are interested, have 20 sign up and in the end have 5-8 that are actually active (roughly 5-8% of the initial number contacted).
Once you’ve got the companies to agree to participate, you’ll want to have them sign a Beta agreement. It doesn’t have to be a formal agreement (though in bigger companies you’ll be forced to use an actual contract), but it should clearly state what the commitments and expectations are. Make it as simple as possible and include details such as length of program, incentives and rewards for participation, responsibilities of participants, expected amount of usage and support that will be provided. Getting participants to actually sign an agreement makes their commitment much more real, and you will have a higher probability that they will actually use the product and provide feedback.
Kicking Off the Program
Once all your participants have signed a Beta program agreement and NDA you are ready to kick off the program. The most important thing at this point is to do everything you can to avoid a false start. If the participants have a bad first experience with the product, your chances of getting them to continue putting in effort will be much lower.
To avoid this, agree with your team on what the criteria should be for starting the program. For example, you may all agree that the program won’t begin until all fatal/crashing bugs have been fixed. Or you may have the criteria that N number of users must have been running the product internally with no major problems for at least a week.
Another approach you can try is to deploy with one or two of the participants that are more technical and/or that you know better than the others. You may even want to go to their company (if they are local) and watch them get up and running. You can figure out where any of the likely bumps in the road may be for other participants.
Make sure that all components of the product are solid before deploying anything. Check and double check the installers, include a FAQ document so that participants won’t have to contact you to get answers, and include documentation and help system content if at all possible (even if these aren’t complete, include them so that you can get early feedback).
Running the Program
Once the program has started, it is critical that you communicate regularly with both the participants and your internal team. For smaller programs, you should call them at least once a week to check in and confirm they aren’t having issues with the product. For larger programs, send regular email communications. Communicate the overall status of the program, bug fixes/new versions that they need to install, updated FAQs and details about contests or incentives. If the participants hear from you regularly they’ll feel reassured and are much more likely to continue putting in time and effort to help you with the product.
Also make sure to communicate internally with your team. Provide a weekly status report including number of bugs reported, usage by participants, whether you are meeting the stated goals and what you need from the team to continue to be successful. It’s critical that you keep the team informed. Once the program gets ready to wrap up, everyone knows the status and there aren’t any surprises or disconnects about the results.
When you are ready to end the Beta program you should have the participants complete a short exit survey. If there is an incentive or a contest, make it a requirement to complete the survey in order to qualify.
The exit survey can be performed via email or using one of the popular online survey tools such as Zoomerang or Survey Monkey. Ask the participants how much they used the product, their overall impressions, whether they believe it is ready to ship and what can be improved. Also ask them to rate the features of the product on a scale of one to five.
Take the results of the survey and deliver a final report to your team. This should include a short summary, bug trend information, whether or not you met the original agreed upon goals and a summary of customer opinions and feedback. Deliver this report to the team prior to making the decision to ship the software – it will prove to be a useful and fact-based tool to make an informed choice.
You’ve put in a lot of hard work to run an effective Beta program, so make sure you leverage it as much as possible. Write thank you letters to the participants and follow-through on the incentives. When doing so ask if they are willing to be involved in future Beta programs or customer councils. Ask if it is okay for you to call them occasionally if there are questions about the product when the team needs a customer opinion. Having a half dozen friendly customers that are easily accessible can be very valuable when contentious product issues come up and you need some external validation.
One last thing. Be sure to write a summary of the Beta program after the product ships. Include what worked well and what didn’t, and give your team members the credit they deserve for pulling off the program. You’ll be much more likely to get the cooperation you need the next time around.