If you can’t trust the data from your research, what’s the point?
Most researchers are aware of numerous biases that can affect survey results. Most of us are trained to avoid or are aware of social desirability bias, confirmation bias, leading questions and even sample bias. However, there are numerous types of sample bias and an important one is not often talked about in the online sample world, yet it can seriously impact the results and reliability of your study. The basis of sample bias stems from the fact that sample panels are different from each other and are constantly changing.
Brand key performance measures can vary a staggering amount, depending on the sample source. We’ve found that brand rating can vary by up to twenty percentage points based on sample provider selection. Twenty percent can be the difference in decisions that can have billion-dollar impacts. Some of the issues that arise from sample bias include data inconsistency, increased risk, and aggregation bias.
Data Inconsistency
Inconsistency is the enemy of tracking studies. Researchers are trying to understand whether the actions of the organization are working with their consumers. So, when there are wild swings in results wave-to-wave, especially ones researchers cannot explain, it makes it hard to compare data to previous waves.
Increased Risk
Relying too heavily on a single panel (even if you are aggregating) not only introduces bias to your study but is also dangerous. If something happens to that sample source (acquisition, bankruptcy, etc.) your study is ruined, and the data is useless with no way to replicate the study. As I said in a recent episode of our podcast, Intellicast, “when there’s an acquisition, there are always going to be changes and that includes changes to your data.”
Even if you end up having to add more panels to complete your study or if the wrong blend is chosen, you risk incorporating sample bias. We know that sample providers significantly change over time due to changes in client demand, changes to recruiting practices, changes to how a panel is managed, increased security and validation methods, and many more. These may all seem to be an improvement, and in many ways, they are, but this can affect the panel composition, the attitudes and behaviors of its members, and ultimately your data.
Aggregation Bias
If you’re already using multiple suppliers, you might think you’re safe from the risks associated with using a single source. When you use a single sample source for all of your sample, your feasibility is limited to that of your selected source. Using multiple suppliers instantly gives you better feasibility.
Most often, aggregating is done to solve feasibility problems, but it opens you up to a variety of other problems in the process. That’s because if you’re not strategically selecting those panels, you’re adding inherent bias to your research. This is where panel differences and aggregation bias come to play. Not all techniques for combining multiple panel providers are created equal.
Stacking
One method of combining panels is stacking. This form of combining sources has a panel provider add as many additional panel providers as possible to a core asset in order to achieve the required feasibility. This could mean two panels or twenty panels. When stacking, no care is given to panel makeup, respondents’ attitudes and behaviors, or panel bias.
Blending
Blending is the process of combining three or more providers, but in a more planned and intentional method, with no provider getting more than 50% of the total allocation.
I’m not sharing these issues to scare you, but rather to make sure you are fully aware of the pitfalls. However, what’s the solution? How can we best reduce sample bias? On the surface, the answer seems easy: use multiple sample providers. In fact, this might be something you’re already doing, but chances are, you aren’t doing it strategically which means it’s likely you’re making it worse. Strategic sample blending is the process of using multiple suppliers with a planned and intentional method.
“Strategic sample blending is the process of using multiple suppliers with a planned and intentional method.”
Blending shouldn’t be done just for blending’s sake. The key is that it should be done in a strategic manner. Customizing a blend based on a client’s needs will ensure the best results possible. If you’re not strategically selecting panels based upon attitudes and behaviors, you’re adding inherent bias to your research. Because all panels are different, they all have different attitudes and behaviors.
Remember how I said that thing about something happening to your sample source? Strategic sample blending can make it so that you don’t have to worry about those things. Panels will shift over time. Making minor adjustments allow for stability over time as panels change. If panels fall short on feasibility or need to be replaced, you already have strategically selected panels to fill in the gaps.
You don’t have to worry about biasing your sample or not knowing if changes in the data are due to differences in panels or real shifts in the market. Strategic sample blending greatly reduces risks and inconsistencies because any panel on your study can be easily replaced, and the study replicated.
This way you know your data is consistent wave to wave and changes in data are because of something the client or brand is doing, not because of the sample plan. Don’t let your research be in vain. Strategic sample blending is the premier method of online sampling and can help you improve feasibility, reduce risk, and ensure your data consistency over time. This allows you to have total trust in your data and make confident business decisions.
