Going Paperless: Prototype: Automatically Send Kindle Notes and Highlights to Evernote

I promised this year that once a month, I’d post a more advanced “going paperless” automation post. Since this is my 100th Going Paperless post, I have a special one planned for today. It is more than just an automation post, it is a kind of call to action for those interested in extending what I have done here. I have tried to explain the process below methodically, and I apologize in advance for places where things get unclear. What follows is for advanced users, although I hope that with help, we can make it a solution that just about anyone can use.


If you’ve been following along, you know that I’ve often despaired that there is no good way to automatically send Kindle notes and highlights to Evernote. This seems like a huge gap just waiting to be filled. Michael Hyatt has good instruction for one method for getting Kindle notes into Evernote, but it requires manual effort, and what I am looking for is something that is automated, that just works as part of the normal routine without taking any special steps. In fact, what I want is something like this:

I read a book on my Kindle device, and make highlights along the way. Like this:

Kindle Highlight

When I plug my Kindle into my computer to charge it, some process automatically updates Evernote with the notes and highlights I’ve taken since the last update. I get one note for each book, and the note looks something like this:

On Writing

Well, actually, I’ve created some automation that does exactly that. The above screenshots were captured from that automation, and I will share how it works below. I did it because I didn’t want to wait any longer. But I did not have time to build a robust, end-to-end solution that will work smoothly and easily for everyone. In other words, I hacked together a solution, and I present here as a prototype of what is possible, with the hope that others will take what I’ve done, and vastly improve upon it.

What I have done, is:

  • Written a Python script that can parse the My Clippings.txt file on a Kindle device.
  • Add or append notes in Evernote via email from the results of the parsed My Clippings.txt file.
  • Automate this through the use of a Keyboard Maestro macro on the Mac.

Before I get into the details, let me talk about the limitations:

Limitations of the prototype

  • The prototype does not use the Evernote API. Instead, it uses Evernote’s email functionality to email notes to your Evernote account, either creating new notes, or appending to existing notes.
  • One note is created per title on the Kindle, if said title has notes and highlight.
  • The solution only works with Kindle devices. It does not work with Kindle Apps, like the iPad or iPhone app.
  • It only works for notes taken on a Kindle device. If you take notes on, say, the Kindle App on your iPad, and then sync across devices, the My Clippings.txt file on the Kindle device is not updated with those notes. They are stored separately in the cloud.
  • For automating the process, it requires a tool that allows you to detect when a USB device has been connected to your computer, and take an action when that happens. I use Keyboard Maestro for this on the Mac, but I don’t know what tool to use for this on Windows.

Once again, this is a prototype, a proof-of-concept that the automation can work. Having demonstrated that it can work, I’m hoping others can improve upon it.

What you need to get started

  1. A Kindle device.
  2. The kindle-to-evernote source code. I’ve made this code available on GitHub. Note once again that it is a prototype. It works pretty well for me but there are still bugs that need to be worked out. These bugs can be tracked as part of the GitHub repository.
  3. If you are on a Mac or Linux system, you already have Python installed. If you are on a Windows machine, you will need to install Python.
  4. I use several Python packages that weren’t among the default packages installed on Windows. These include the Six-1.5.2 package and the python-dateutils-2.2 package. These appear to be installed by default on Mac systems. (And I presume Linux as well.)
  5. For automation on the Mac, I use Keyboard Maestro. This software allows me to trigger a macro when a USB device is connected to my Mac. The macro then runs the kindle2en python script. I don’t know what the equivalent software would be on Windows or Linux, but am certainly open to suggestions. This piece is only needed if you want the process to be fully automated.
  6. A Gmail account for sending messages to Evernote.
  7. An Evernote account.

How the solution works

When properly installed and configured, the solution is actually pretty simple. A configuration file stores information about where to look for the My Clippings.txt file, as well as login information for a Gmail account. I store the authentication credentials in the configuration file because that file should only ever be on your local machine.

The first time the kindle2en.py script is run, it creates a “semaphore” file in your home directory called .kindle2en_sem. This file is used to capture the last time the script was run. It stores the date and time of the last run. The first time the script is run, the date is set to January 1, 1990 so that it captures all of the notes and highlights on the Kindle.

When the kindle2en.py script is run, it reads the My Clippings.txt file on your Kindle device and looks for any notes or highlights added since the date contained in the semaphore file. It collects these notes and highlights into buckets, one bucket per title. Once it has processed the file, it sends one email message per title to Evernote. The email messages are formatted so that highlights appear as Evernote highlights in the note body. Notes appear as plain text. The email messages are sent to your Evernote account via Gmail. The Subject line of each message is the title of the work followed by the author, like this:

On Writing (King, Stephen)

The script uses the + at the end of the subject line to append the notes and highlights to and existing note of the same title. If no such note exists, it creates a new note.

You can run the script manually at the command line whenever your Kindle device is attached to your computer simply by typing:

python kindle2en.py

(assuming the script is installed in your path).

The command has several options. If you type

python kindle2en.py -h

you will see the available options, as follows:

Usage: kindle2en.py [options]

Options:

-f       specific location of configuration file other than home dir

-h       display help

-v       verbose output

-V       output version information and exit

The command can also be called from another process. On my Mac, I use a Keyboard Maestro macro to execute kindle2en.py whenever my Kindle device connects to my Mac’s USB port.

Part of the beauty of the solution is that once the note is in Evernote, you can add addition notes directly to the note in Evernote. And if you highlight more on your Kindle, those updates will be appended to your note.

Installing the solution

Here is how I have the solution setup on my iMac. Things will vary slightly from machine-to-machine, and platform-to-platform.

Setting up Keyboard Maestro

Here is what the macro I use in Keyboard Maestro looks like:

Keyboard Maestro Kindle Macro
Click to enlarge

1. My Macro. This is the name of the macro I created, followed by the trigger for the macro. Most macros are triggered by a keyboard command, but in this case, you can see it is triggered by a USB device.

2. Trigger conditions. This trigger conditions for this macro are pretty simple. The macro is triggered when a USB device named “Kindle” is mounted on my machine. You will need to check the name of your Kindle device and update the Macro accordingly. Case matters, so make sure that the name you put here matches your device name exactly. The trigger occurs when the Kindle “is attached” to my machine. This means when I plug it in to the USB cable, the macro will run.

3. Optional. This part is entirely optional. I included in my macro so that I have a visual indication that the script is running. Whenever my Kindle is attached, Growl will send a message to the screen telling me that my Kindle is attached. This way I know the script is working.

4. Verification. I could just run the action, but I added an additional verification step here to make sure the device is still connected when I run the action. If it is still connected, then it executes my Python script.

5. Execution. This is where I tell the macro to run a shell script. I have to include the full path of the scripts that I want to run. And since I wasn’t sure if the script runs with my environment or not, I used the -f switch for the kindle2en.py which tells the program where to look for my configuration file. Your paths may vary.

That’s it. That’s what tells the system to trigger the script. If you are running this on a Windows or Linux machine, you’ll need to find some software that detects when a USB device is connected and can execute a script on such an event.

Installing/configuring the kindle2en.py script

Installing the script is pretty straight-forward. I put the script in my /usr/local/bin folder. You want to make sure the script is executable so you’ll run the following command on it:

chmod +x kindle2en.py

Next, you will put the .kindle2en.cfg file in your home directory. This file then needs to be edited with some information that the script uses to find your Kindle notes and send them to evernote. The configuration file looks like this, out of the box:

# File Locations
CLIPPINGS_FILE=<path to your clippings.txt file>

# Mail Settings
GMAIL_USERNAME=<your gmail address>
GMAIL_PASS=<gmail password>
GMAIL_SERVER=smtp.gmail.com
EN_ADDRESS=<evernote email address>

Gmail is used to send the notes and highlights to Evernote via email. The password is stored in your settings file on your local machine. This is relatively safe if your machine is your machine and not a shared machine. One future improvement would be to use the Google API directly and OAuth authentication, but I was trying to keep things simple here.

If you use 2-factor authentication with Google, as I do, then you’ll need to create an Application Password for this app and use that password in your settings file, instead of your regular password.

The Clipping File location is the path to your Kindle device clipping file when the device is attached. This will vary based on the name of your Kindle device. For me, this line looks as follows:

CLIPPINGS_FILE=/Volumes/Kindle/documents/My\ Clippings.txt

Additional Python libraries

It is possible that you won’t have all of the required Python libraries installed by default on your machine, but they are easy to get. I know for a fact that when I installed this on my Windows machine, I had to add the following libraries to Python:

  • Six-1.5.2
  • python-dateutil-2.2

What libraries you have by default will depend on your installation.

Once you’ve got these things setup, the script should run each time you connect your Kindle device to your computer.

A call to action for future improvements

As I’ve said, I’ve been using this for two week now, and it works pretty well for me. I’ve run it manually from a Windows machine, and have automated it (via Keyboard Maestro). But there is a lot of room for improvement. Things are kludgey right now and can be made much more robust. The installation process can be automated and simplified. I’ve shared the source code on GitHub with the hope that others who have also been looking for this functionality will use what I’ve put there as a jumping off point.

Some suggestions for improvements in the short-term:

  • Replace the email-to-Evernote functionality with Evernote API calls. Evernote provides a rich API for developers. Part of the reason I chose Python is because it is one of the languages for which Evernote provides an API. A much more seamless integration would be for the script to use the Evernote API to update notes instead of email. I used email because it was quick and dirty. It allowed me to “prove the concept.” The API would be much more robust, and give you much more control. It would also allow you to get rid of the Gmail component.
  • Identify tools that allow automation on Windows and Linux. I’m sure there are ways of detecting when a USB device on Windows and Linux machines, but I didn’t investigate them because my ecosystem is primarily Mac, and I wanted to prove the concept. The concept works, so others, with more experience on Window and Linux might be able to suggest ways to automate things there.

ETA: Be sure to see Nathan’s comment below. He points out an iPhone app that does all of this and more–it gets all notes and highlights from Amazon regardless of device, and you can send them to Evernote or Dropbox. The app is called Snippefy, and it does 95% of what I was hoping for, and better than what I have outlined here. (Although, I still think what I have outlined is a pretty decent example of trying to come up with a solution.)


I’m sort of amazed that I’ve written 100 of these posts, and I’m looking forward to starting the next hundred. For this post in particular, I hope it will be a collaborative effort. Share thoughts and ideas in the comments, but also be sure to check out the code on GitHub and see what you can make it do.


If you have a suggestion for a future Going Paperless post, let know me. Send it to me at feedback [at] jamietoddrubin.com. As always, this post and all of my Going Paperless posts is also available on Pinterest.

Last week’s post: Append to Existing Notes in Evernote via Email.

Enjoy these posts? – Tell a friend

Recommending readers is one of the highest compliments you can pay to a writer. If you enjoy what you read here, or you find the posts useful, tell a friend! Find me online here:

Twitter | Facebook | Google+ | Blog | RSS

Or use one of the share buttons below. Thanks for reading!

24 thoughts on “Going Paperless: Prototype: Automatically Send Kindle Notes and Highlights to Evernote

  1. I’ve never used Keyboard Maestro, but Hazel should also be able to fire off the python script. In Linux world, it’ll be necessary to fiddle around with udev rules, but they’re not terribly complex and there are decent tutorials around the ‘net for building rules. A couple things to keep in mind, for any Linux folks who might be interested:
    1. The udev rule should reference the Kindle’s USB product and vendor IDs, and maybe even serial number if that’s available.
    2. Writing udev rules is vaguely like programming, and uses some programming conventions. Use a double equals sign when testing for a match (e.g. ATTR{productID}=="0x12ab"), and a single equals sign for assigning a value to something (e.g. ENV{something}="value").
    3. To execute a script, such as kindle2en.py, when udev matches the device to the specified attributes (i.e. you’ve connected the Kindle to your Linux machine), the end of the rule should look like: RUN+="/path/to/executable/script.py". Make sure the code is executable by all (permissions include “x” in the 3rd group; chmod 755 /path/to/executable/script.py will take care of it).

  2. I got it to work with Ubuntu. The only issue was the My Clippings.txt did not need the reverse slash (\) in the file name. I’m researching some options for getting the script to run upon plugging it into the USB.

  3. Glad to hear you got it to work. I tested it on my Mac and on a Windows machine, but independent confirmation is always nice. I put the settings (including the path to the My Clippings file) in a separate setting file because that can vary from platform-to-platform and machine and machine. That there are small differences doesn’t surprise me. (In the Mac UNIX command like, you have to escape out the space in “My Clippings” with the \ character.)

  4. I bet it would be possible to write a plug-in for Calibre to do this. The advantages are that Calibre is (1) multi-platofrm, (2) written in Python and thus compatible with what you’ve already done, and (3) already has the ability to detect when a Kindle is plugged-in, essentially taking care of the entire USB issue. The configuration information (last-date synced, gmail/evernote account info) could be stored as part of the settings for the plug-in.

  5. This is an awesome detailed post. Good job man! If I could offer a more simple way to do this, look up Snippefy on the App Store. This app pulls the highlights and note from your kindle books and it integrates with Evernote and Dropbox. Check it out here: http://www.snippefy.com

  6. Nathan, brilliant! That is almost exactly what I was hoping for. I’m downloading it tonight to give it a try. The one minor drawback to it is that it is not completely automated; you have to take an action to send to Evernote. But it’s still better than what I’ve put together because it gets all highlights, not just highlights and notes from a Kindle device. I’m grateful to you for pointing it out.

  7. Sure man no problem. Let me know what you think of it. FYI, there’s an update waiting for apple’s approval that will allow you to export to Evernote as text files. Currently you can only export to Evernote as PDF if you are looking to export an entire book. Otherwise if you export only one highlight at a time it’s possible to do it in text format. The update should be approved any day now.

  8. I think it’s great that you’re creating your own solution. In case you’re interested, there’s a free website that does this for you. Not sure if it does it exactly the way you want it, but it might be worth checking out. It’s called Clippings Converter, and I’ve been using it to help me collect information for writing book reviews.

    I love your posts and find them very helpful. Thanks for taking the time to write them.

  9. I always love your posts Jamie, especially the ones on using Evernote. I hope that we see an “official” solution to this soon…I’m not savvy enough to feel comfortable with this one but would love the ability to have my Kindle notes automatically populate in an Evernote note. That being said I’m going to check out a few other solutions offered in the comments – maybe I’ll find one I can use until Evernote gets it together! 🙂

  10. This script works (right now) only for Kindle’s which are set to “english”, is doesn’t work with any other language.
    I’m currently (with Jamie’s help) working on this issue and fixed some parts already but fighting with datetime now.

    I also want to mention that although Jamie is right that the script only works with Kindle, not with iPhone, iPad & Co, notes you made within kindle on e.g. the ipad get sooner or later into evernote as well, as long as you have note syncing enabled on all devices.

    For Mac Users, there is no need for Keyboard maestro as you can use the automator and folder actions (see the post from Garret for a workflow that actually works fine and just needs to be changed to run Jamie’s script). Hazel does unfortunately not work, at least I wasn’t able to figure out how.

  11. Too geeky for me. I don’t use my Kindle any more, and I can tell you that Kindles don’t like taking baths. Trust me. I use my Kindle app on my iPad. I take notes and send them to my Twitter account with a hashtag to identify the book I’m reading. Then I set up an IFTTT recipe to automatically forward those tweets with that hashtag to append to an Evernote. I use one note per book. It works, after I figured out that I had to un-protect my twitter account first.

  12. Jamie,

    I think you converted me. I’ve had Evernote for a couple of years but rarely used it. While reading through your site about the different ways you use it for everything, something clicked inside my brain and I suddenly saw potential ways that Evernote could help me. I’m also indebted to you for the posts about automating things.

    I’m a lot like you in that I like to count, record and track stuff, but the pieces never quite came together for me on how to actually do it. (despite 25 years of software development experience.) When I saw your many posts, I saw how it was possible.

    Thanks for doing all the leg work you did with Evernote and Google scripts. I give me a great starting point for my own automated, paperless goals.

    Patrick.

Comments are closed.