The problem with split survey designs.

As the pressure to source quality representative sample grows, and generative AI fraud-farms overwhelm online panels, researchers are looking for new ways to control costs and create more tolerable survey experiences. Here, controlling questionnaire lengths is the toughest challenge.

Though spend any time answering surveys yourself, and within minutes you're realise that modern questionnaires aren't really a conversation. They're a loop from hell, where the same question is asked again and again to test difference across contexts, brands, need states or purchase missions. These long questionnaires suffer higher dropouts which destroys representation, and those respondence that do remain offer flat unengaged answering, as they try click their way to freedom.

Worst of all, higher incentives are incurred to lure respondents through. These higher incentives are just as appealing to survey farms. Longer surveys go to the top of fraudsters' target lists.

Some researchers will try to tackle long questionnaires by setting up a split design. Here respondents are routed to answer some parts of the questionnaire and not others, often based on a least fill quota system - the refuge of a beaten researcher. Here the category with the lowest incidence rate is given priority - and so when a respondent qualifies for the category in the screener section, they will be routed accordingly.

While the researcher can then report on all the data points, the data is wrong because of the extreme bias exerted by the least fill approach on sample representation, its category overlaps and profiles. It isn't the lower incidence, hard to fill categories that suffers, it is all the others. The picnic is short of a few sandwiches; not all respondents answered everything they should have.

Failing to represent the customers you most need to represent.

This is because the parts of the survey the respondent answers are partly determined by when the respondent clicked into the survey, and how the fieldwork manager is getting on in achieving the harder to reach quotas. Click in over the weekend versus in the middle of working week and you may be allocated differently, as buckets are filled up and closed off. This has the effect of treating time rich respondents (who also tend to be cash poor) differently. While this isn't ideal, it still works out okay for the niche category if the survey is fielded at a reasonable rate.

In many sectors, customers who qualify for a harder to reach niche quotas, are often more engaged in the product category, higher spending, more loyal, more favorable… They are also more likely to qualify for all the other categories in the survey - where the large influence of their answers is not reflected at all because of the least fill design.

Let's take an example of a pet food U&A study with four categories: Cat, dog, bird, and fish food. Assume market incidence is in that order. Respondents see two sections of the four questionnaire sections. Least filled quotas route anyone who has a fish into the fish section. While this bucket will be representative of fish owners, (those who only have fish or those that have other pets too), the rest of the categories are now more likely to contain those with only that pet and have far less likelihood of including those who also have birds and fish. The design fails to represent cat and dog owners (because anyone who has more pets were routed away from completing those sections - and by definition, they are also the higher spending customers!)

This has dire commercial consequences because those that have more pets may be the minority, their higher spending levels may be disproportionate. They may also be animal lovers more likey to purchase add on value services, pay premiumn prices for their beloved cats and dogs. They are also more likely to have more of them. This is bad research design.

In most sectors, those engaged in multiple categories could easily be responsible for 80% of a business's revenue. Not only this, but all mainstream categories can be floored to the point of uselessness, because they can no longer be representative as the overlaps have been doctored through the least fill design. Their estimates are simply misleading, with more than half of those who are supposed to be surveyed missing. Representation is skewed right across the survey; no market sizing can be done. All the overlaps are incorrect.

Data Fusion: The cheaper & better approach.

When the game is to create tailored experiences depending on customer's needs, leveraging cross category engagement to build business (almost always the right commercial objective), least fill quotas are a woefully pathetic approach - because representation matters just as much as sample size. If you're a researcher, you should feel more than queezey right now.

What is required is a method which maintains representation across category, while keeping incentives and questionnaire lengths down. Well, you know where this is going… it's clearly a job for Data Fusion.

Split survey designs play right into Data Fusion's strengths:

• The same population of individuals
• Comparable research methodology and mode of collection
• Relevant common variables
• Corroborating data points to validate and tune the results.

In many of these multi-category surveys, Data Fusion provides the researcher flexibility to field far less rigid quota designs, to ensure better representation across categories, and then complete the data, borroing data from similar customers, to build a rounded and realistic view - which mirrors reality.

If you're interested in extreme cost savings, improved data quality and the ability to do more with multi-category surveys, here is a step-by-step how to do data fusion, as well as a blueprint for applying data fusion to split survey designs.