The Yes-Machine: Addressing Acquiescence Bias in Synthetic Research

In psychometrics, face validity is the weakest form of evidence that a measure is valid.¹ Coherence does not equal validity, meaning synthetic research might hold together internally but still fail to measure the specific traits it claims to measure. The good news is that validity can be tested with established methods from the field of psychometrics, allowing synthetic research to be held to the same standards as traditional research. In this article, we explore what those tests are and what passing looks like in practice.

What is Construct Validity and Why Does it Matter?

In 1955, Lee Cronbach and Paul Meehl introduced the idea of the nomological network, explaining that a construct is validated by its relationship to a web of other measures and outcomes, not by inspecting single answers. This web of relationships breaks down into a few validity types; this article focuses on construct validity—whether a measure captures the concept it’s intended to measure, like whether a customer satisfaction score measures actual satisfaction or something else entirely.

In 1995, Samuel Messick reframed construct validity as the overarching concern that all other validity types feed into, building toward the central question: does this measure actually capture the thing it claims to? As such, construct validity is the ideal lens for evaluating any new kind of research instrument, including synthetic ones.

Construct Validity Starts at the Population Level

Before a synthetic study asks a single question, it’s already made a consequential validity decision by defining who the synthetic respondents are. That decision occurs through population synthesis, which relies on established techniques such as iterative proportional fitting and synthetic reconstruction. These methods are good at getting the marginal totals right: the right number of men and women, the right income levels, the right age spread, and so on.³ However, they don’t automatically preserve the associations between attributes, resulting in synthetic populations that don’t represent real people.

Here's an example: real-world data shows an association between lower education levels and higher smoking rates.⁴ A synthetic population may match the education breakdown and smoking rates correctly, but fail to preserve the link between the two. As a result, any constructs built on top of that correlation will be wrong, like who a message reaches or where health risks concentrate. Research on agent-based modeling confirms this issue, showing that populations built without preserving real-world associations produce less accurate models and lead to biased outcomes.⁶

Preserving that web of internal associations is what construct validity looks like at the population level. A “realistic population” is defined by its associations as much as its margins, and those relationships only survive when population synthesis draws from real individual-level records, not traits generated independently.

A Toolkit for Validating Synthetic Studies

The gap between looking right and being right can show up at every layer of a synthetic study. Rigorous synthetic research platforms follow evidence-based validation methods to close that gap at each level:

• Population level:Compare the synthetic population’s internal relationships against the real data it was built from. In practice, this means using standard measures of association and distance to confirm that cross-tabulations and correlations between attributes, such as age and income or education and health behavior, are maintained in the synthetic population.⁵
• Measure level:Apply standard psychometric due diligence to ensure construct validity. Use validated, previously published instruments rather than ad hoc scales;¹ check reliability directly on synthetic responses, looking at both internal consistency and stability across repeated passes (which is far cheaper to check with synthetic research); and confirm that measures behave consistently across any segments you plan to compare.
• Result level:Compare synthetic outputs against real human data (ideally from the researcher’s own past studies), using standard measures to calculate how far apart two distributions sit. This tests algorithmic fidelity, or the degree to which synthetic responses accurately reproduce the attitudes, beliefs, and demographics found in real human data.⁶
• Cross-cutting:Run the same study across multiple independent models, treating agreement between models as a reliability check and divergence as a signal to investigate further. Reliable synthetic research tools should grade the result of these tests honestly based on what the comparison shows: strong where models align, directional where alignment is partial, and flagged where models diverge.

What This Means for Users

When it comes to vetting synthetic research tools, the above validation methods can be distilled into a handful of simple questions worth asking any vendor:

• Where do your synthetic people come from, and does your sampling method preserve the associations between attributes or just the marginal totals?
• What construct does each measure claim to capture, and what validated instrument is it anchored to?
• Are results benchmarked against real human data, and can I provide my own?
• Does the instrument report agreement across independent models and flag where fidelity is weaker, or does it claim to be equally strong across every segment and trait?

How We Establish Construct Validity in Artificial Populations

Artificial Populations (AP) was designed and built by behavioral scientists to test for construct validity at every layer of the synthetic research process. At its foundation, each participant persona is modeled on real humans using demographic and psychographic data. Measures rely on validated instruments, and every study can be compared against human data, including a researcher’s own uploaded datasets. Results are cross-checked across independent AI providers, and each finding is assigned a confidence tier, so every output comes with its evidence strength attached. Importantly, AP flags where fidelity is weaker, ensuring strong results carry confidence while divergence signals the need for closer scrutiny.

To learn more about the mechanics of synthetic research and the methods that ensure validity, check out our articles on semantic similarity rating and benchmarking against your own data.

Sources

1. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
2. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
3. Chapuis, K., Taillandier, P., & Drogoul, A. (2022). Generation of synthetic populations in social simulations: a review of methods and practices. Journal of Artificial Societies and Social Simulation, 25(2). https://doi.org/10.18564/jasss.4762
4. Cao, P., Jeon, J., Tam, J., Fleischer, N. L., Levy, D. T., Holford, T. R., & Meza, R. (2023). Smoking disparities by level of educational attainment and birth cohort in the US. American Journal of Preventive Medicine, 64(4), S22-S31. https://doi.org/10.1016/j.amepre.2022.06.021
5. Roxburgh, N., Paolillo, R., Filatova, T., Cottineau, C., Paolucci, M., & Polhill, G. (2025). Outlining some requirements for synthetic populations to initialise agent-based models. Review of Artificial Societies and Social Simulation.https://rofasss.org/2025/01/29/popsynth/
6. Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J., Rytting, C., & Wingate, D. (2022). Out of One, Many: Using Language Models to Simulate Human Samples. ArXiv. https://doi.org/10.1017/pan.2023.2
7. Photo by Ant Rozetsky on Unsplash