Do we need to be scared of big data and its claimed spooky ability to know more about us than our partners, to mould our behaviour like puppeteers?

An e-mail dropped into my inbox today from Amazon. I won’t name the book or the author the tech giant’s algorithm was trying to market to me. Usually I delete Amazon’s recommendations unread, but this time I looked at the book. Not because I expected I’d want to read it, but because I wanted to understand the marketing. The book was described as “An emotional psychological thriller with a twist.”  That already annoyed me: why both psychological and emotional? And, yeah, of course it has a twist. The only thing it didn’t tell me is what reading level I need in order to follow the prose.

I began to read the taster. It began with an odd question. That was designed to intrigue and to hook my attention. The meaning of the question is explained by the bottom of the first page, in case my patience flags. There’s a hammer-beat of short staccato sentences, designed to lunge for my heart. There’s a bit of backstory. Hmm. Backstory on the first page? That’s a mistake when we’re supposed to be in the relentless attention-grabbing now.

The machine algorithm is marketing something to me I wouldn’t read in a million years. Primarily because it feels like it was assembled by a machine according to a formula. Facebook’s algorithms keep showing me pictures of cute dogs. Actually, it’s my wife who’s keen on cute dogs, not me.

My point is despite all that machine learning and big data, these two tech giants still have little clue who I am and what I like.

So, exactly how afraid should we be of Cambridge Analytica? The data breach that allowed the company to harvest the details of 50 million Facebook users is undoubtedly serious. But did it get Donald Trump elected President of the US? Did similar dirty tricks swing the Brexit referendum in the UK?  Let’s examine what they’re able to do with the data.

The core of the data analysis seems to be a personality quiz app, developed by a Cambridge University academic and downloaded by 27,000 people. The quiz broke people down into groups, dominated by traits like agreeableness, openness, neuroticism etc. We’ve all done such tests in magazines. There are 72 different online personality tests available on the website of Cambridge University’s (not to be confused with Cambridge Analytica) Psychometrics Centre. Clearly this is not an exact science.

What was different in this case was the ability to look for correlations between the personality quiz results and Facebook records, such as what people liked. Here it does start to get a bit more sinister. For example, a 2013 research paper by three academics from the Psychometrics Centre showed that it was possible to predict intimate information about a person from their Facebook likes, information such as sexuality and political leanings.

In some cases, the correlations were pretty obvious – liking the “No H8” campaign and being gay. In others they were less clear. For example, users who liked the “Hello Kitty” brand tended to be high on “Openness” and low on “Conscientiousness”, “Agreeableness”, and “Emotional Stability”. They were also more likely to have Democratic political views (75%) and to be of African-American origin (82%), predominantly Christian (69%), and slightly below average age.

While it’s not precise, what this does is allow micro-targeting. Instead of standing at a hustings and bellowing the same message to everyone, a political candidate can whisper different messages to different groups – threat messages to the fearful, for example, and optimistic messages to the bold. The Cambridge Analytica whistle-blower, Christopher Wylie, said this is exactly what the company did.

Was that enough to lose Hilary Clinton the US presidential election? We don’t yet know. A scientific test would require demonstration that micro-targeted ads outperform random ads in changing consumer actions. But my guess is that Cambridge Analytica was better at marketing itself to sleazy clients than it was at targeting and changing voters’ behaviour. Predicting personality attributes is not the same thing as changing behaviour. And, despite the hype, it’s not at all clear that the algorithm is any better at prediction than a human would be.  If Amazon can’t even get my reading preferences right, what chance big data can make me vote for someone whose politics I don’t like?