How we “see” what we read in stories

I’ve often wondered how different people imagine the same scenes in books and stories that they read. The great thing about reading (unlike movies and television) is that it takes active participation from the reader to build the scene. The writer provides sensory clues, but the reader uses his or her own experience to see the scene unfold around them. I think of the opening of Isaac Asimov’s Foundation, when Gaal Dornick arrives on Trantor and makes his way to his hotel. The vastness of the city-planet and Asimov’s descriptions of it painted a picture in my mind that reminds there no matter how many times I read the story.

But it is very likely a very different picture than what other readers, reading the same story see in their mind.

I’ve been thinking about this recently, in light of the reports that scientists have been able to produce images from the brain of what we remember when watching TV or movies. It seems to me that a particularly interesting experiment, were it possible, would be to do something similar for a passage in a book. I imagine that what our brain remembers when we see a movie is fairly similar from person-to-person. Imagine what the images might look like if we could see those “movies” in our heads created from the books we read? How different would yours be from mine?

It is interesting not just because of the science behind it, but also as a writer who creates scenes. When I write a scene, I have an imagine in my mind that I am trying to convey to the reader and I do this by using sensory description and the emotions of the characters to build a picture in the reader’s mind. But the reader will never have exactly the same image that I have. Indeed, without producing a physical image it is likely that our mental pictures will differ dramatically. That is because we each will fill in those blank spaces–the spaces not touched on by the prose–with stuff from our own background.

If I write a scene in which I say: “The desk sat in the corner of the office, by the floor-to-ceiling windows” I have a good idea of what the office looks like in my mind, but I provided a sketch to the reader. We both know where the desk is in relation to the windows, for instance, but that desk will appear very different to you than it does to me. All I described was a desk. In my mind, I have a mental picture of my desk at my home office, a large wooden desk, heavy, slightly chipped in places, with a flat surface cluttered with stacks of old science fiction magazines. But because I just said “a desk” your mental image will be filled in with details from your own experience. And that is true for every reader of the story. Given this, it would be fascinating to be able to “see” those mental images.

I suspect that what we would learn from such a study, is that the more detail provided by the author the more similar elements appear in the mental pictures from person-to-person. But anything left to reader discretion will vary from reader-to-reader, and without painting an exact picture of what is in the author’s mind, those mental pictures will always have a great deal of variety. This, as opposed to the mental pictures that come from the movies we watch.

This type of research helps make clear the line between reading and watching. They are active and passive processes respectively. An active process–on in which I need to take action to fill in the details will certainly allow for more variety than what I get if I am presented with images. Indeed, this experiment has been performed before in a completely informal and ad hoc way, by people who have read a book upon which a movie was based but never saw the movie. In discussing the story with those who did see the movie, my guess is that those who read the book will vary widely in opinion about how closely the actor chosen for the lead character varies from their mental image. Those who just saw the movie will have nothing to say on the matter because they had no preconceived notion about what the character should look like.

There is valuable insight here, I think, for writers trying to create vivid scenes for their readers. It helps to recognize that as a writer, you will never get across the exact image in your mind. But that is okay. A big part of the fun in reading a story is bringing your own experience to the words, the setting, the characters, and to some extent, making them your own.

18 thoughts on “How we “see” what we read in stories

  1. Interesting. I’ve delved into this tangentially when discussing classic movies and television, where directors/studios did not have the ability and/or the technological capability to actually ‘show’ whatever on the screen. When done well, it can approach closely to the reader’s experience, where there is a very minimal director’s vision, leaving room for the viewer/reader’s imagination.
    When I see movies based on stories I’ve read, I’ve been fortunate to have experienced a few cases where the director’s visions were as good as my own (the ones in my head) – but never has their vision exceeded my own, which is probably why I’m A. – so critical of film and tv that “shows it all” and B. usually happier with older efforts than newer ones.

  2. Interesting article. Touches on some of the reasons why I prefer to read the book than watch the film, and why, on the occasions I have done both, the film never quite matches with my mental images. But then my imagination isn’t limited by budget or technology. As for multiple readers interpreting differently the same scene in a book; It would be interesting to see what each person would paint if they had the talent and were asked to recreate the scene in a painting.
    This makes me wonder about the balance that has to be struck when describing a scene. Like how much is best? I mean can you over describe a scene, with so much detail it leaves little or no space for the imagination of the reader? Same with too little information, or too much being left to the imagination?

  3. Chris, I, too, prefer the book over the movie and have yet to see a movie that I think improved upon the book. But again, this is the difference between active and passive participation in the storytelling. As far as the balance goes, I think it depends on the skill of the writer. I’ve read writers who describe scenes in greater detail (think: George R. R. Martin in his Song of Ice and Fire series) and this provides vivid images of the scene in my mind. I have read writers who provide barely a sketch (think: Donald Barthelme in The Dead Father). I imagine you could liken it to artists. There are skilled painters who can render a scene in almost photographic quality. And then there are artists like Al Hirschfeld, who with a couple of bare lines, can produce a recognizable caricature of some celebrity.

  4. Here’s an interesting thought: I’ll bet that if a writer could accurately describe the images in a reader’s head for a given scene – in other words, custom-produce a scene exactly as it appears in a particular reader’s head – that reader would be disappointed with the result – even though it is 1-for-1 what they mentally imagine.
    I suggest this because our imaginations, even fully detailed ones, are comprised of a lot of ‘fuzzy detail’ – just like our dreams. I think awe and wonder are largely produced by the fuzzy areas, rather than the detail.

  5. I remember reading Greg Bear’s EON which, probably, goes into greater detail describing the scenes and setting than any other book I’ve read. It has never been made into a film, but there was a website competition for people to make a digital trailer for an imaginary film of it. Not one of those trailers looked anything remotely like the scene’s I had in my mind.

    Which brings me onto a worry I have about watching films of books I enjoyed. And that’s the film-scape overwriting my mind-scape. If that makes sense.

  6. Chris, I have that same worry. What’s interesting is that when you read a book first, you have your mental film-scape and the juxtaposition from that to what you see on the screen can be jarring–and indeed overwrite your original images. But when you see the thing onscreen first, it’s different. I saw Game of Thrones on HBO before I ever read any of the books. The characters are fixed in my mind based on what I saw on the screen as opposed to what I read. Indeed, I often wonder how I might have imagined Tyrion Lanister if I had read the book before I saw the show.

  7. Steve, I think that is the “active” part of reading. Despite the descriptions we are being fed, we are still creating these dynamically changing images of varying resolution. I’ll say that when I read, I don’t “hear” the characters speaking the dialog as I might on screen. I am aware of the written conventions of dialog. When someone shouts, I imagine they are speaking louder, but I still don’t hear them saying the words, for the most part. In my head it is more like a mental telepathy. But when I’ve seen a show based on a book and hear a line of dialog, and then go back and read the book, I “hear” in my head what I heard in the dialog on screen. The same voice, intonations, etc.

  8. Jamie,

    I did an experiment once when I had the opportunity; I’d not seen Serenity (Firefly movie), nor read the novelization. I picked up the book on the cheap, so figured here was the opportunity. Read the book while watching the movie – almost 1-to-1.

    using that methodology, I found it fairly easy to dismiss disconnects between the two (one whole major scene in the novel is left out of the movie), and, in reference to your comments, was able to prevent one ‘scape from over-writing the other.

    The book fell far short of the movie in this case. I’m on the lookout for a repeat in reverse – a novel-movie combo where the novel came first – to see if the same dynamics are in place.

  9. It occurs to me this is far more complex than I originally considered. For instance:

    A long-time SF reader, reading a novel like Joe Haldeman’s Forever War or Jack McDevitt’s Echo will be familiar with the tropes of the genre. I would suppose that two long-time SF readers, both of whom are familiar with the tropes of the genre, would have similar (far from identical but similar) mental images simply because of a shared background: the tropes of the genre itself. Now bring in someone who only occasionally reads SF, or someone who hasn’t read it at all. What kind of mental image are they supposed to form when the see terms like “ansible”, or “collapser”, or “skimmer”?

    Circling back to what got me thinking about this in the first place, an interesting experiment, if it were possible, would be to compare the recalled mental images of a scene from a book (as the scientists did for movies) for people both familiar and unfamiliar with the genre. I wonder how those images would vary?

  10. I think you’re are also describing why language both works and fails. If you think about it, words are just packetized, compressed encodings of images we have in our minds. When we speak or write, we’re passing on these little compressions to other minds, in which they are unpacked and re-imaged (via the Hebbian net we discussed in Twitter). The reason this works is that we all share some common experiences that allow for consistent imagery. But our experiences also contain unique details. When the unique details don’t overlap, communication fails.

    I think the most effective communication with words uses the least words but transmits the intended imagery most coherently with the author’s intention.

  11. Ah yes, Robert Brown!

    From Alfred Korzybski– “The map is not the territory”

    General Semantics had a large influence in the SF Golden Age. Especially with van Vogt and Heinlein.

  12. LOL, general semantics. I wish there was a government dept monitoring news channels and applying semantic coding to all of the words that are used in such a twisty manner. Can you imagine the impact it would have? There’s – oh say, Sean Hannity blathering away, while the laden words he’s using pop onto the screen, color-coded to indicate emotional intensity. More pop-ups provide expanded explanations: “This word has a negativity quotient of 99 and is frequently used to incite unreasoning hatred”; “in this context, this word is empty of meaning”…

    I predict a great clamor for unadulterated news if the above were in effect.

    Your tropes mention, Jamie: it sure is a shame that not everyone can draw effectively (like me): I think it would be revealing to have a bunch of people draw various ‘tropes’ to get a real feel for how closely (or not) various concepts are shared.

  13. There is a great scene one of the early Foundation stories where Salvor Hardin or someone is listening to a politician speak to him. Later, one of the other characters said they converted his words to symbolic logic and that everything cancelled out completely. Despite talking, he said absolutely nothing! Not quite semantics perhaps, but of a similar vain.

  14. Mark,

    Absolutely! The map is not the territory. But if the map were the territory, we’d be no better off, because a map that contains all the complex detail of the territory would be just another instantiation of the territory. Ok, I know that sounds circular. But I run into this all the time with clients who want models that contain ALL the detail of a particular business opportunity, down to the last penny. But a business model that contains ALL the detail of the situation is no different from the situation itself, so it provides no insight.

    To me, math is a language. Math modeling is like telling a story with logic and functions. To gain insight into a complex situation requires a certain amount of requisite abstraction and simplification.

    I don’t think this too different from writing compelling fiction. A good story contains just enough detail to arouse the imagination and pull the reader (and the writer) into an alternate reality. The story is a map to what is in the writer’s mind.

    It’s fascinating to me that we all share enough common experiences that this works. Earlier I said that when the unique detail don’t overlap, communication fails. But thinking about that a little more, I think there are caveats to my statement. Of course, when we’re talking about building a bridge or a nuclear reactor, achieving clear communication is important. Non-overlap can lead to disasters. But I can see that it can also spawn innovation and greater creativity than the original author had intended. If communication were perfect, we might not be as creative.

  15. Robert, Very nicely stated.

    I consider “the map is not the territory” (or, “the word is not the thing”) to be a precautionary statement.

    Steve touched upon the emotion-laden quality of words. Heinlein spends a fair amount of time on this topic in “If This Goes On-“. I don’t know if it is included in the original ASF serialization, but in the book version—

    ZEBADIAH JONES: ‘…You can sway a thousand men by appealing to their prejudices quicker than you can convince one man by logic. It doesn’t have to be a prejudice about an important matter either. Johnnie, you savvy how to use connotation indices, don’t you?’

    JOHN LYLE: ‘Well, yes and no. I know what they are; they are supposed to measure the emotional effects of words.’

    ZEB: ‘That’s true, as far as it goes. But the index of a word isn’t fixed like the twelve inches in a foot; it is a complex variable function depending on context, age and sex and occupation of the listener, the locale and a dozen other things. An index is a particular solution of the variable that tells you whether a particular word is used in a particular fashion to a particular reader or type of reader will affect that person favorably, unfavorably, or simply leave him cold. Given proper measurements of the group addressed it can be as mathematically exact as any branch of engineering. We never have all the data we need so it remains an art-but a very precise art, especially as we employ “feedback” through field sampling…’

    And Van Vogt’s solution to countering the deleterious effects of manipulative wording is the “null-A cortical-thalamic pause.” Where one trains the mind-body to over-ride the brain’s emotional (centered in the thalamus) response to outside stimuli by deferring to the analytical portion of the brain (the cortex). I’d like to say that the above makes more sense after reading van Vogt’s Null-A novels, but I can’t. 🙂

  16. There’s actually a simple way to put all this: Words don’t have definitions (except in highly structured contexts, as in mathematics). Words have usages. Context is the determinant of all usages.

Comments are closed.