Do all stories (let alone all scenes) need surprise? No. But it is essential to some genres. There is no joke without a punchline, and no punchline without surprise. Or consider the classic mystery tale—we should be surprised by the unveiling of the murderer’s identity in the denouement.

Let’s consider what surprise is. Our brains are (among other things) pattern recognition machines. The story sets up a pattern of expectations, and then a twist violates those expectations and reveals a different pattern of meaning lurking beneath. We see the world in a different light. The brain judders as it shifts frame. Our experience of this is surprise.  

Take the example of the joke. Two elderly gay men are sitting at a booth in a bar. A beautiful woman walks past, with all the attributes heterosexual men find attractive (you can spin out this description at some length to reinforce the pattern of expectation). One man turns to the other with a sigh, saying “You know, it’s a times like this, I wish I’d been born (add a dramatic pause here) lesbian.” You see? The heteronormative pattern is subverted. Our laughter is the physiological expression of the surprise.

But this is not the only way in which pattern shifts work. If the surprise comes at the beginning rather than the end, our response is different. World-building in fantasy or sci-fi might be an example here. We are told from the start that the rules here are different from those in our world. Consider, for example, The Other Valley by Scott Alexander Howard (simply because I happen to be reading it at the moment). In this world, time is literally a dimension like space. There are a series of valleys, each with a similar settlement. As you go west, each valley is ten years into the future. As you go east, each is ten years in the past. The gendarmerie  patrol the borders between the settlements to prevent temporal meddling. Our attention here is not captured by the headlong rush to the denouement, but by curiosity about how the consequences of the set-up will unravel. I would describe my emotional response to this as delight, not surprise.

Several years ago, I was given a pattern-changing piece of writing advice—“If you’ve got a surprise, give it away right from the top.”  It’s worth noting that this is exactly what Dostoevsky does in Crime and Punishment. We know that Raskolnikov has murdered the pawnbroker. What makes the story gripping is his subsequent reactions.

The idea of delight is, I think, an interesting way of framing the question of whether you should build a story out of a chain of surprises. If every chapter, scene, paragraph and sentence contained a surprise, it would rapidly become exhausting to read. But what if they all contained a delight, a slight shifting in our sense of the world?  Howard offers such delight in The Other Valley through the wordcraft. To take an example from the point in the book I happen to have reached, “My bedside lamp illuminated raveled nests of hair and grit on the floor.” Not perhaps the best example, but it shows the poet’s eye in the image of the “raveled nests”. The metaphoric assimilation of detritus to bird nests is a small moment of delight, an amuse bouche between courses.

Metaphor can, I suppose, be considered a form of surprise, shifting our patterned expectations. When we say “My love is like a red, red rose,” our understanding of both the love and the rose change. Sometimes, this poetic play of perception can have world-changing consequences. When James Clerk Maxwell perceived light, electricity, and magnetism as being waves in an underlying medium, he achieved the second great unification in physics.

Metaphor is at the heart of creative reperceiving of the world, both in the arts and the sciences. Oftentimes, it is the small delights of metaphor rather than the headlong dash to the denouement which keep me gripped.