Category Archives: writing

A Heatmap of Over 900 Days of Writing Data from My Google Docs Writing Tracker

A few days ago, I mentioned that I was looking to add a Github-style heatmap feature to the Google Docs Writing Tracker code. Well, I’ve got something to show for it. Keep in mind that I am still experimenting, and none of this code has been checked into the Google Docs Writing Tracker repo as of yet. But, here is what all of my writing data looks like going back over 900 days:

900+ Days of Writing Data

For each year represented above, the rows are days of the week (top row is Sundays, bottom row is Saturdays), and the columns are weeks of the years.

The scale goes from 0-250 words (the lightest green) to 1,500+ words (the darkest green). You’ll also note that in July 2013, there are two white cells. Those are the only days that I had no writing. The last day, July 21, 2013, was 770 days ago. I have not missed a day since then.

This was relatively easy to do thanks to the Cal-heatmap JavaScript library. After installing the library files, I exported my writing data (dates and words counts) to a JSON file. Once the JSON file was created, the rest was easy. The entire rendering of the heat maps looks like this:

Heatmap Code

The bulk of the code is customizing how I want the heatmaps to look. Now that I have the look I want there is only one more thing to do, and that is to automate the process of generating the JSON file from the Google Docs Writing Tracker spreadsheet. With that done, anyone who uses the Google Docs Writing Tracker will be able to render a heatmap like the one above.

You can see my writing heatmap in action. If you hover over the cells, you’ll get the word count for that day. Check it out, play around with it. Let me know what you think.

ETA (8/31 @ 1:30 pm): I managed to automate the process of generating the JSON file from the Google Docs Writing Tracker spreadsheet. In the near future, I’ll post the code to a new repo on Github since it isn’t directly related to the code for the Google Docs Writing Tracker itself.

A Dashboard for the Google Docs Writing Tracker

A while back, I created a kind of dashboard into my writing statistics, courtesy of my Google Docs Writing Tracker tool. I never made the code for the dashboard available on Github mainly because it was highly tailored to me. Recently, I have been thinking about better ways of dashboarding my writing data, and it was my use of Github itself that provided a useful insight. I’ve created a heatmap of my writing in the past, and I liked the concept of it. So began wondering how I might produce a heatmap that would be a good representative of my writing. Then I remembered that just such a heatmap exists on Github to show my contributions:

Github contributions

What if I could produce a similar heatmap for writing, using data from my Google Docs Writing Tracker? So I have started to experiment with this. Turns out, it is probably relatively easy. Github uses the D3.js object model for producing the year-long calendar interface for the contribution chart, and that library looks fairly easy to use. I’ve started to experiment with some sample code. Once I have something that works, I’ll post the code to the Google Docs Writing Tracker repo under a new branch and other people who use the tool can mess around with it and see if it works for them.

And as a reminder: my Google Docs Writing Tracker is freely available on Github to anyone who wants to use it, or improve upon it.

A Story in Santa Monica

There is a new story that I have been wanting to write, but I have been so focused on the the novel that I just haven’t had the time. But I am going to try a little experiment. On Sunday afternoon, I head to Santa Monica, California for a week for work. While there, I am going to try to write the first draft of the story. It doesn’t necessarily mean taking a break from the novel–I’ll just prioritize the story first, and fill in whatever time I have left to work on the novel. Except, there’s not likely to be much “time left.” So for all practical purposes, I’ll be taking a break from the novel next week.

That’s okay with me. I could use a break. And not only do I think I have a pretty good story idea, but I finally figured out how to tell it. The voice of the story is always important to me and I can never make it far until I figure out the voice. Now that I think I have that covered, the story should good well.

We’ll see. I’ll give it a shot, beginning on Sunday and see if I can have a first draft finished by Saturday. I don’t have a title for the story yet, but my working title is likely to be “Fat Man and Little Boy.”

700 Consecutive Days of Writing, Plus Q&A

Yesterday, I hit another writing milestone. I wrote about 1,100 words on the novel draft yesterday. That’s par for the course of the last month. But with the writing done for the day, it meant that I have now written for 700 consecutive days.

700 consecutive days

I remember when I hit 100 consecutive days and thought that was pretty amazing, and that it seemed a huge uphill climb to make it another 265 days beyond that to get to a full year of writing. Today, I am 30 days away from 2 consecutive years of writing. The writing has become so ingrained in my daily life that it is almost like breathing.

In the course of those 700 days I’ve managed to write 612,185 words, mostly fiction. During that time I’ve published 14 pieces, 4 of which are fiction, and 10 of which are nonfiction. The fact that I have published more nonfiction and written more fiction is simple enough to explain. Nonfiction comes much more easily to me, while fiction takes practice and goes through many drafts. Many drafts means accumulating word counts. Also, I have written 2 novel drafts in those 700 days, and two novella drafts. Both of which, counting the restarts and rewrites, add up to a lot of words.

It doesn’t bother me because I think of all writing as practice. I like to think that the more I do it, the better I get at it.

All of this writing takes place in less than an hour each day. In fact, on average over the course of 700 days, I’ve written 875 words/day. And according to RescueTime, which automatically captures how much time I spend writing each day, I’ve spent, on average, about 38 minutes per day writing. Add that up, and over the course of 700 days, I’ve spent 443 hours writing. If that sounds like a lot, compare it to how much time I’ve spent at the day job over the same period–roughly 4,320 hours. That’s nearly 10 times what I’ve spent writing. Put another way, or every hour I spend on the day job, I devote 6 minutes to writing.

I have picked up the pace over the last month, as I push forward in earnest on this novel draft. In the last 30 days along I’ve written over 31,000 words, and I expect that trend to continue.

Questions and answers

I do get questions from time-to-time about the streak, or about how I find the time to write, or how I organize my writing, what tools I use, or whatever you can think of, I’m going to suggest that if anyone has questions about this stuff, drop them the comments below, and I will do my best to answer to them.

5 Challenges to Writing Every Day

With my consecutive day writing streak approaching 700 days (687 as of yesterday), I thought I’d take a few moments to talk about the challenges of writing every day. There are many people who challenge the notion of writing every day, arguing that one should write only when one feels the urge. I write these posts as a window into my methods, but I understand that what I do may not work for anyone but me. That said, if someone sees my posts and thinks that they want to give it a try, here are some of the challenges I faced along the way, and how I dealt with them.

1. What do I write?

If, when you sit down to write each day, you struggle with what you want to write–that is, before putting a single word down on paper, you come up blank, then writing every day may prove challenging.

I have been lucky in this respect. I am a pantser when it comes to writing–I don’t plot out my stories ahead of time, but that does not mean I am don’t plan ahead. I am thinking about my stories constantly. By the time I sit down to write, I can start very quickly. The quality of what I write may vary from day-to-day, but rare is the day on which I struggle just to get started.

2. What about those days where I just don’t feel like writing?

I enjoy the act of writing. I like telling stories, too, but for me, writing is a kind of stress relief. Even so, there are days when I am tired, when I’ve worked at the day job for 10 hours with my head down in code, and come home only to take the kids to a Little League game or some other event. By the time I get to the keyboard, my mind is utterly exhausted. What then?

I also look at writing as practice. Like anything, to get better at it, you have to practice. If you play a musical instrument, you often practice even on those occasions when you don’t really feel like it, simply because you have to get the practice in. On those days where I just don’t feel like I have the energy, I tell myself that I have to get the practice in anyway. Even if I only sit there for ten minutes, I get a page. Even if that page is terrible, I learn something about my writing.

3. Writing every day requires discipline

There’s no way around this. If you are undisciplined, I suspect you will find writing every day to be challenge. Not impossible, but just a challenge. I will say that I have found that it gets easier over time. But especially at the beginning, it took me a lot of dogged discipline, especially on days when it seemed the writing wasn’t going well.

4. Writing every day requires persistence

If you miss a day, write it off, and be sure to write the very next day. It has happened to me. I mentioned at the outset that as of yesterday, I have written for 687 consecutive days. But I also have a larger “streak”: I have written 830 out of the last 832 days. In other words, I have missed two days of writing in the last 830 days. When I started out trying to write every day back in February 2013, I wrote for 140 consecutive days, and then circumstances arose which forced me to miss a day. I was back writing the next two days, and then I missed another day. But that day, July 21, 2013, was the last day that I missed.

It was hard to see a 140-day streak die, but I got right back to it. I took some important lessons from those two missed days, and so far, I have written every day since.

5. The weight of the streak can be stressful at first

I occasionally am asked if I feel the weight of the writing streak as the numbers build up. My answer is that I felt it early on, but I don’t any more.

Early on, while developing the discipline, I was learning how to work around my schedule. I was learning how to write in noisy environments, or in short 10-minute spurts here and there. But as the streak grew, I also felt nervous about what would happen if I missed a day. As I said above, at 140 days, I did miss a day.

In the nearly 700 days that I have not missed a day writing, I have experience just about every kind of contingency. There are days on which I have traveled, and needed to squeeze in the writing early. There are days that seemed filled from start to finish and I had to find 5 minutes to get a few paragraphs in. There are days that I felt sick, or that I was taking care of sick kids. There was even a day that I went under a general anesthetic (when my wisdom teeth were yanked) and I still managed to get some writing in.

In all of that time, I’ve learned that chances are very good that I’ll get the writing done. I have 687 days that demonstrate this. So the streak no longer weights on me the way it did early on. If anything, it demonstrated the certainty with which I will get my writing done each day.


Again, this is my mode of working, and it has worked very well for me. But every one is different, and we must each figure out what works best for ourselves. I present these challenges as lesson I have learned that may benefit others who are considering tyring to write every day. Hopefully, they help.

 

Yesterday I Set a Minor Writing Record

Believe it or not, while I have now written for 686 consecutive days I have not had a whole lot of consecutive days over 1,000 words. For the entire streak, I have averaged 870 words/day, under the 1,000 word-mark that I have been aiming to get to this year. I have also had several “micro-streaks” in which I went 6, 7, 8, or 9 consecutive days writing at least 1,000 words. But until yesterday, such micro-streaks have never been longer than 9 days.

Yesterday that changed.

10 days of 1000 words

The 1,500 words I wrote yesterday evening (after a very long day which included taking the Little Man to fencing class, running to Home Depot to buy a dozen bags of mulch and then spreading that mulch around the backyard, taking the Little Man to his birthday bowling party where 23 other kids ran wild; spending the afternoon and evening with friends while our kids continued to run wild) put me over that 9-day threshold. For the first time since I started trying to write every day way back in February 2013, I have actually written more than 1,000 words/day for the last 10 consecutive days.

This is another way of saying that the novel draft is going gangbusters. It now stands somewhere between 16-17,000 words, most of which have been written in the last 10 days or so. It will be interesting to see how long the 1,000 word+ days will continue. They are not easy to achieve, mostly because of time constraints. Writing 1,000 words takes me, on average, about 40 minutes–although that average is down to 36 minutes over the last 10 days–and it isn’t always easy to find 36 minutes in my day. But I’ve been making the time, more often than not but cutting back on social media. And so far, so good.

A few more upcoming milestones

A couple of additional milestones are almost upon me, so I’ll mention them here:

1. On Father’s Day (June 21–also the first day of summer) I will hit 700 consecutive days of writing. After that, only 63 more days before I topple Barry Bonds’ home run record, with 763 consecutive days of writing. I think that happens sometime in late August.

2. As of yesterday, I have written 596,047 words over the course of my consecutive day writing streak. That means I am less than 4,000 words away from 600,000 words. Assuming I can keep up a minimum 1,000 words/day for the next 4 days, I’ll hit that milestone either Wednesday or Thursday of this week. 600,000 words in less than 2 years is more than six times what I wrote in the entire 20 years before I started this streak of mine. For me, at least, it definitely pays to write every day, if for no other reason than because I am getting a lot of practice in.

When the Writing Finally Clicks

I have been struggling along with the first draft of this novel since late March. I had hoped to have the draft finished by the end of June, but it had been a difficult story to write. With short stories, I push through the first draft without looking back. With novels–for which I have far less experience–I have found myself starting, and restarting, which is a bad sign for me in the first draft.

But, I haven’t given up. I decided that the only way to learn how write a novel is to write one, and if that means start and restarting until the story clicks, then so be it.

The story finally clicked.

If you have ever wondering what that click looks like, what that ah-ha moment when the light bulb burns suddenly bright looks like, I think I have a pretty good illustration:

When it finally clicked

You can see my struggles pretty clearly over the previous 24 days before the story finally clicked. When it did, my writing shot up from a average of under 400 words/day to where it stands today, at about 1,100 words/day. Zooming out, this become more obvious:

3 months of writing

I started things right around March 27. The first few days were great, but you can see that slope of decline as I got mired in problems, and had difficult pushing forward. That led to a second attempt, and once again, a spike, but that spike was short-lived, and the daily writing declined. Then, six days ago, the story clicked. I found what I think is the right way to tell it, and I have been pushing forward. The story has taken off.

How do I know? Well, aside from what the data tells me, it is what I feel. I am eager to sit down to write each day, and when I do sit down to write, the time flies by. Over the last 6 days, I’ve average about 40 minutes/day of writing and it flashes by in the blink of an eye. What’s more, I come away with a good idea of what I will write the next day. I have a good vision for the story now.

Since late March when I started this story, I have written 44,000 words. However, I only have about 10,000 words of usable story written so far. For some people those 34,000 words that make up the difference might seem wasted. For me, they were the practice I needed to get to the point where the story clicked. I’m fairly confident the story will move much more swiftly now that it has clicked. But I don’t think I ever would have made it to this point, if not for those 34,000 words worth of flailing.

Long Weekend and a New Story

We spent the long Memorial Day weekend up in Lancaster, Pennsylvania with my sister and her family, and my mom. The kids got to spend most of the time at Dutch Wonderland, and we spent the weekend trying to keep up with them. We also took a horse and buggy ride through an Amish farm, and I had some homemade root beer, which was excellent. This is our third year in a row getting out of town on the official east coast opening of summer, and heading up to Pennsylvania, and like the last two years, it was a great success.

I managed a little bit of writing on the novel draft, but I also started a brand new science fiction story. I’m not entirely sure where it is going yet, but I will say that the main character is a somewhat overwhelmed project manager who finds himself over his head on a new assignment. And in case anyone wonders where I get my ideas: recently I have felt like a somewhat overwhelmed project manager in over his head on a new assignment.

It’s actually a nice change of pace from the novel draft, and I think I’ve been ready for a chance of pace for a little while now.

Fighting words, or musings on pantsing vs. plotting

My friend, and fellow science fiction writer, Bud Sparhawk has some fighting words for me this morning. For context, before continuing, you should go read his post. Bud and I have given several talks on online writing tools, and pantsing vs. plotting at various science fiction conventions. Today, however, Bud made it clear that he feels plotting is the superior form of writing. While I can’t deny that it might be superior for some, I can say that it doesn’t work well for me. And so let me take up each of Bud’s jabs one at a time to give a little of my perspective of this heated debate1

“Life is largely unplotted…”

Bud writes:

Jamie writes by the seat of his pants –which is akin to running with scissors IMHO– while I choose the wiser and more prudent course of carefully plotting my works.

While it is probably beside the point, I was always taught that there is a right way and a wrong way to run with scissors. Or perhaps, a safe way, and a dangerous way. On those rare instances where I run with scissors, I opt for the safe way, holding blades curled in my fist to prevent myself from stabbing anyone, especially me.

But Bud also goes on to say that he chooses the “wiser” and “more prudent” course, and that is carefully plotting out everything he writes.

It might be wiser and more prudent for Bud, but it just doesn’t work for me. When I try to plot out my stories, the result is stories that are too neatly plotted. Everything fits together too well. Coincidence rears its head more often than it should. In other words, the stories feel plotted.

Instead, I have become a big believer in Stephen King’s suggestion that life is largely unplotted. For me, planning out too much makes the stories feel artificial. I prefer a more natural approach where the plot develops from the situation the characters find themselves in and the actions that they take. This has worked well for me. The stories I write organically, without planning every step of the way, have sold faster, and in general been more successful than those that I have carefully plotted out.

Practice makes perfect

Bud goes on to talk about my prodigious output, although he exaggerates slightly. While I have been aiming for a 1,000 words/day, I average about 850. But I do write every day, and haven’t missed a day in 656 days now. In those 656 days, I’ve written 575,000 words. So Bud is right; I write a lot.

But then, my plotting friend goes on to say:

Instead, due to his hasty and impetuous headlong dash to finish something he has to throw out most of his words, edit with a chainsaw, and rewriting practically everything.  From this I draw the conclusion that writing by the seat of your pants is wasteful of time and talent. (Emphasis is mine.)

Here is where Bud and I part company. Would a music teacher say that it is a wasteful to practice your scales? Would a medical school professor tell students it is a waste of time and talent to intern? Would professional baseball player say that it is a waste of time to practice hitting in the batting cage? Would a flight instructor tell a student pilot that it is a waste of time time practice takeoffs and landing?

Then why do we think that it is a waste of time for writers to write. Bud is correct: I write a lot, and much of it gets re-written from scratch. But I don’t see it as wasteful of time; I see it as the practice I need to develop my talent. I know of no other way to become a better writer than to write. For me, the proof is in the numbers. Prior to writing every day, I sold 1 story on average every 3 years. Since my writing streak started, I’ve sold one story or article every 45 days.

Pantsing and plotting are not opposites

I find that people think pantsing means that opposite of plotting–no planning at all. For me, at least, that is not the case. I know where my story will start, and I have an idea of where it will end. Then I start writing, working my way toward that ending. The planning happens more informally, more in realtime than it might if I plotted it out. Sometimes I hit my mark, and the story ends where I imagined it would when I started. Other times, the story surprises me. The same is true for those, like Bud, who plot everything out ahead of time.

But I also need the writing experience to be a discovery for me. Plotting out things ahead of time has the same effect on me as talking about my stories: it spoils the excitement of the story.

The most important thing is to write

Bud is a more experienced writer than I am2 and can put that experience to better use than someone with less experience. But as I see it, the only way to gain experience is to write. It doesn’t really matter whether you are a plotter or pantser. What matters is that you find the process that works best for you, the one that feels right, the one that encourages you to keep at it day after day after day, through the rough patches, and through the rejections. The most important thing is to keep writing.

  1. Not really heated. Bud and I are friends and this is, of course, all in good fun.
  2. Isn’t that a great way of saying he’s much older than me?

Progress Update on the Novel Draft

It’s been a while since I’ve given an update on the progress of the current novel draft that I am working on. For a refresher, recall that I am trying to write 4 novel drafts between March 2015 and March 2016 (a first and second draft for 2 different novels).

I struggled for a while with the first draft of the first novel. I went through 7 restarts because I couldn’t make the story work the way I wanted. I came close to giving up and moving on to something else, but I really like the story, and I decided to press on. I’m glad I did. I think I figured out the trouble I was having. It fell into 2 categories:

  1. Point of view. All along, I was writing the novel as though it was a set of notes in a notebook of the main characters, who was telling the story to some unknown audience. I kept running into problems because it felt like the main character was deliberately holding back information for no good reason. It felt unnatural. It took a long time before I figured out was that I needed to tell the story from another character’s viewpoint, one who doesn’t already know the whole story. Then it took more time to figure out who this character should be. But I finally did it, and I think things will move much better now. In fact, I think the story will be a better story because of this.
  2. Framework. I have only written one other novel draft. Part of the reason I’m trying to do more is to get the practice and learn how to do it. But writing at length has been giving me trouble. Not because I can’t do it, but because I feel like I’m adding too much filler to the story that isn’t necessary. I felt like I was handling something too big. So I began to wonder if there wasn’t some way of telling the story using smaller, more management chunks. Thanks to changes in #1 above, I found a way to do this. As it now stands, I am writing the novel as a series of 5 novellas, each one flowing smoothly into the next. Each novella is essentially a stand-alone story, that allows me to work at a length that I am little more comfortable with. But in truth they are all tightly integrated, and with the exception of the first novella, you couldn’t really read a later one without having read the first one. In that sense, the novel is more like an episodic series of novellas. This, too, seems to be working better for me.

I had planned to finish up the first draft of the first novel at the end of June. I don’t think I am going to hit that mark now. More than likely, it will be closer to the end of July or early August. Of course, that throws off the rest of the schedule I had planned. On the other hand, this is intended to be a learning experience for me, and I am learning a lot about how to make longer stories work. To that end, I am very pleased.

I have no idea if this story will ultimately work; I really won’t know until I complete the second draft. But it is a story I like a lot, and I am sticking with it until it is finished.

For those wondering, I wrote about 25,000 words total in starts and restarts on the current novel draft before finally figuring out what I was doing wrong. Some folks might see those 25,000 words as wasted. I see them as invaluable practice as I attempt to learn new aspects of my craft.

Make sure you have the latest version of the Google Docs Writing Tracker

I have had some reports of DocsList errors from folks using the Google Docs Writing Tracker. This is because those functions have been deprecated. Several months ago, I updated the master branch with code that uses the newer DriveApp object model. I have been using that code for a few months with no errors. If you are seeing errors in the last few days caused by DocsList objects, I recommend you pull the latest version of the scripts. That should fix the problem. Please see the readme for additional details and instructions.

More Updates to the Google Docs Writing Tracker

I recently pushed a new branch called “project-tracking” out to the Google Docs Writing Tracker on GitHub. This branch includes code for project-tracking that I wrote about a week ago. The changes have been working fine for me over the last 10 days or so. The one thing I haven’t done yet is update the template spreadsheet. The new code requires 2 new tabs in the spreadsheet, along with some additional settings. I’ll get to that eventually.

Meanwhile, I have been trying to figure out a way to simplify what happens each night the scripts do their processing. Right now, the scripts perform a comparison between the current working document, and a previous snapshot of the document in another folder. That snapshot mechanism takes up a lot of code, and is relatively inefficient. Over the last few weeks, I’ve been thinking about an alternative, and today, I tested that alternative out with positive results.

Every Google Document keeps a revision history of the changes to that document. Here is the revision history for a story that I worked on back in February:

Revision history

It turns out, that using the advanced Google Drive API, I can access the revisions through the API. Today I performed a test, which essentially compared the current document to the last revision of the previous day. That is essentially what the snapshot method that the script current uses does. But it does without needing to maintain two files. I can get all of the information I need from the previous revision. Ultimately, that simplifies the code for the scripts. It also simplifies setup.

There is a tradeoff, however.

You can only access the advanced Google Drive API via OAUTH2 authentication. That means configuring the scripts to be able to handle that authentication. It turned out to be a pretty straight-forward one-time setup for me, but I do this kind of thing for a living. For someone who isn’t technical, it may be a little tricker.

It will likely be a while before this major architectural change is available. There are several reasons for this:

  1. My priority each day is on getting my writing in. I do this scripting only if the writing is done, and I have time.
  2. If I were doing this just for me, it would be easy. The code I wrote today checks for the last revision from “yesterday” and compares that to the current document. Simple, right? But not everyone who uses these scripts writes every day. What happens if you skip some days. Then there is no revision from “yesterday” so the script has to know to look for the previous revision regardless of date. There are a few other uses cases that need to be considered as well.
  3. Once I have the code written, I like to test it for a few weeks before pushing it out, just so that I can work out any kinks.

That said, once this feature is in place, I think it will make for an enormous improvement. Since everything, including the revisions, is contained in the one document, there will no longer be a need to manage a snapshot folder at all, and all of that code can go away.

It also opens up the possibilities for analytics on the evolution of a document over time, which would be pretty cool, too.