Technology

I used to work for one of the nation's largest survey marketplaces. Y'all have no idea how deep this hole goes.

Surveys\polls are largely requested by political polling groups, research teams, and ad agencies. They put those up on an auction block just like ads, and then we would route traffic into it from various places. Mostly the survey takers come from mobile games (take this 3 question survey for 20 Blorp Points kind of stuff) or survey taker apps that give you points for gift cards and such.

So even before bots, most polls are taken by "professional" survey takers who use banks of phones to maximize their point earnings. We spent a lot of energy on "proving" to the survey provider side that real humans were answering, and not using scripts or bots to just rapid finish them (answer B to everything kind of stuff). Using sophisticated bots to randomly answer was super common.

They were super ready for AI. We talked about it everyday, game planned how it would work, designed systems around it. "Synthetic survey" was the buzz word. Why ask humans for answers if the statistics machine can convincingly predict the answer for you? We proposed ideas like generating the prediction fast and early, then using actual polls to adjust the result towards reality over time. We had tools to track people and connect their spending to poll questions so we could ask follow up questions on purchases, to provide "lift" metrics to agencies on if their ads were working. We were working on the "verification can" tech, only it would have been "Answer this 10 question survey to continue watching your movie."

I was so glad to leave that place. They got bought and consolidated into the world's largest survey company a year later and they fired everyone else that had been left. All they wanted was the tech and the customers.

Now just get AI pollsters and AI readers and leave us alone.

Just one more thing losing contact with reality.

Ah yes, “synthetic users.” This is being pushed at my job as well. We’re supposed to use AI to design the next feature for our website, then ask AI “users” what they think of it.

That’s not our entire vetting process - it’s supposed to replace someone just writing down an idea and saying “I think this is good.” And I agree that just firing from the hip like that is dumb. We want our product managers to do more research into their ideas before they get greenlit to be built.

The question is whether AI “synthetic users” add anything of value. The team that put this tool into service noted it has a “positivity bias,” aka “you’re absolutely right!” So we feed it an idea we think is good, and it says oh yes it’s very good.

It’s read every customer email we’ve ever received and every user research report ever conducted by our human UX researchers. But it’s still just not that useful. I think AI is very useful for summarization, searching, and collation of information, but this goes beyond that, asking AI to imagine it is a person and then come up with things to say about an entirely novel concept. And AI is not good at that.

“The idea behind silicon sampling is simple and tantalizing,” they write. “Because large language models can generate responses that emulate human answers, polling companies see an opportunity to use AI agents to simulate survey responses at a small fraction of the cost and time required for traditional polling.”

Somebody invested money into this company. And there's at least hundreds, maybe thousands, of other businesses with these asinine ideas about how to use AI. They're all getting capital from someone who's supposed to be smart because they have capital. Remember that when llm providers cost correct token prices.

"I should base my surveys about human behavior solely on responses from non-human machines" said... someone, apparently? Damn. 💀

Humans barely answer surveys as if they were human, this is some real derivative shit.

"According to the latest polls, Americans favor Republican policies UNANIMOUSLY!"

Just as useless as product reviews on Amazon.

Sounds like something the GOP and O&G industry would do, with all their bots commenting horse dumpings all the time.

But people could have replied that.!

Seeing as the vast majority of "polls" I've seen in the last 5-6 years have been "this was a poll done online, so we can't assign any certainty or margin of error, cause we have no idea who actually responded and it could've been just like, two dickheads with bots spamming nonsense, but the results were click baity enough for us to run a story" .......... I don't see how them cutting out the two dickhead middle-men, and just using their own bots, is really that much different.

Republicans are going to love this

First off, got a chuckle from the bot check..

The story quoted new poll findings by a company called Aaru, representing them as research based on the feedback of American adults. But according to an editor’s note, the piece had to be “updated to note that Aaru is an AI simulation research firm.”

In other words, Axios had failed to disclose that it was citing alleged “polling data” that wasn’t drawn from human respondents at all. Instead, it was dreamed up by a large language model —yet the latest sign of every imaginable industry trying to leverage AI, even when doing so makes absolutely no sense.

This was/is a problem, but giving up on stats because bad stats exist, is like refusing to ever eat food again because someone got you to try a sardine and spinach chocolate cupcake one time.

In fact, the first, last, and most often brought up topic in graduate level statistical analysis isn't about getting numbers, that's easy. The hard part is finding the flaws in numbers, even in your own that proves yourself wrong.

The vast majority of people never learn that, or learn that bad stats have been a problem as long as stats has existed. Even making it thru peer review doesn't always mean anything.

Like, every single time an article links to a study, do the due diligence and click, so what's going, what the numbers really say, and search who funds them.

It's not like you'll even know what to look for at first, but if you never try you'll never improve.

Every LLM is a fascist propaganda machine

This is hilarious. And when it inevitably doesn't work, they will have a human tweak the statistics of the "AI". The irony of it all.

This is so stupid it is hilarious. I guess the first question to the customer is what they want to know, and the second question is what they want as a result.

Polling has been getting increasingly worse for decades now.

Their accuracy died with the monoculture and no amount of multilevel regression and post-stratification seems to compensate for that