Working scientists use logbooks to record their work so that they can (a) reproduce results, and (b) establish priority in discoveries. As a working developer, I use a logbook, too, which also serves two primary purposes: (a) capture what I did during the day (sometimes in order to reproduce things), and (b) as an index to more detailed notes for specific things.
My logbook had evolved over the years. It’s present incarnation is a text (markdown) file in Obsidian. For work, I use Obsidian’s Daily Note feature for my logbook. I have one logbook “page” for each day. I have this logbook page open on one screen at all times. Usually my Obsidian window is split with my logbook on the left and other notes files on the right. Here is what a typical screen looks like:
Yesterday I mentioned how I was in crunch time for a software system my team has been building for the last 13 months. We go-live on Monday and we are preparing for roll out. We are in what I call the “junk drawer” phase of the project. I think of it like moving houses. All of the big furniture had been moved and all that is left is the stuff in the closets and junk drawers. That’s where we are. Yesterday, my workday started at 7am and ended at 12:30 am (this morning!). Here is my logbook entry for yesterday:
Those of you who spend your days making software are probably familiar with the pattern of these last few days before roll out. Those who aren’t can at least get a little glimpse of what these days are like for me.
As you can see from the logbook, I finally got to bed around 1 am (about 4 hours later than normal) and I was up before six this morning so that I could get in my morning walk before taking my girls to school and writing this post.
Now it’s back to where I left off yesterday. I have a new blank “page” open in my logbook and I’m ready to fill it.
On Monday we will roll out a software system that my team has been building for the last thirteen months. In nearly 27 years with the company, this is the software that I am most proud of. It is a system that coordinates new hires, people who are changing jobs or transferring locations, and people who are separating. It involves integration with many other systems, but it really does save people a lot of time over what they do today and it smooths the process for a person starting at the company on their first day, or leaving on their last.
It occurred to me that there is something even more remarkable about this piece of software. The very first meeting I had on this project was back on February 3, 2020–before everything began to shut down due to the pandemic. The first of what was ultimately 37 requirements meetings was held on April 21, 2020. The meeting, while involving people from all over the company and across most of our locations, was done remotely at this point. And since then, we haven’t had a single in-person meeting on the project. The entire software system was built remotely, with people working from home. Of course remotely-built software is nothing new in the open source world. But I think this could be a first for my company. And I’m really proud of the result.
That said, I am in a mode that is probably familiar to many software designers and project managers on the eve of releasing a product you have been working on for so long: all I can see are flaws.
I have a long punch-list of things that I am trying to get done between now and Monday and they all seem to be flaws in the system. These are not fundamental design or structure problems. This is more like a spot on the windshield that you missed while cleaning it. But it is all I can seem to see at the moment. I am so happy with what we have done with this system that I want it to be perfect when it goes live on Monday. And anyone who has ever used a computer knows that no software is perfect.
Still, I think it is useful that my brain is only seeing flaws at the moment. It allows me to focus on those things that matter, and not worry about the stuff that is already in done and ready to go. It means that every day, the system is getting a little bit better.
I was incredibly lucky on this project. I had a great development and testing team. It has been one of the best things about working where I work that I get to work with some of the smartest people I’ve ever met. I learn from them every day, and I have been really impressed with how well our core team has worked remotely over the last 13 months. I’ve lost count of the number of “co-programming” sessions we’ve had that have worked out so well.
We’re in that final push now. This past weekend was the third weekend in a row that I’ve worked. Yesterday, I put in 11 hours, and I couldn’t sleep much last night because the stuff I have to get done was running through my head. So I was up before 5 am this morning to get started and maybe quiet my mind a bit. Four more days, and one more weekend to do, and the system will be live and in the wild. I am feeling unusually optimistic about this system–which is why I am glad that at the moment, all I can see are flaws. I think that will help to ensure we release the best possible system we can come Monday.
I just completed a demo1 for software that me and my team have been working on for nearly a year. It isn’t quite done yet—we still have a few months of work left, but it feels good to get to the point where you have something to show. Indeed, it’s not that different from getting a story out into the world.
When it comes to writing (when I can even manage to write these days) I am “pantser” as opposed to a plotter. That is, I don’t plan out my stories in detail. I have an idea of where the story is going, and I figure out how to get there as I write. Sometimes, I end up somewhere else entirely.
With software, I am the complete opposite. Over the decades I’ve worked on increasingly more complex software and I’ve found that my brain doesn’t have the capacity to build it without plotting it out first. I hadn’t really considered it much until now, but I suppose that designing software is a lot like outlining a story. You start with the high level goals and requirements; you identify the tools you can use to meet those requirements; and then you figure out how everything will fit together to make a cohesive and self-consistent thing.
The day-to-day work is writing code, small fragments, akin to scenes in a story or novel. Often you write something that gets the job done but isn’t particularly elegant, so you rewrite it and rewrite until it purrs. The world fades around me when I get into this mode, my focus is completely on the task at hand, and an 8- or 9-hour day can fly by in what seems like the blink of an eye. This happens when writing, too, but I can’t consistently write for 8- or 9-hours the way I can work on code.
I often had the impression (from comments I’ve heard in various places and times) that non-writers think that writing is easy. I find it difficult, and I’m fairly worn out if I manage to writer for more than 2 hours. Writing code is equally difficult, but I come away from these long stretches in what I call a “code-coma.” It’s hard to re-engage with the world around me, and I have to ease back in.
When I have something on the page that works, I’ll review it, often out of context of the rest of the piece (often because “the rest of the piece” doesn’t exist yet). I’ll find typos here and there, little misspellings, or autocorrects that don’t work. Same thing when writing code. Punctuation is just as important and I can stare at a small piece of code for an hour wondering why isn’t working, only to realize I was missing a semi-colon somewhere.
Sometimes, you get pretty far along in a story only to realize that you’ve uncovered a major hole in the plot. This happens with software, too, but it happens less frequently for me these days because I “outline” the software and try to eliminate plot holes in the design before we actually start building the thing. Still, other things may trigger major changes forcing rework.
When I think a story is ready (always after the second draft, never after the first) I’ll send it out for critique by fellow writers. It’s no different with code. We sit down for code reviews where it becomes our job to ask each other (and ourselves) tough questions about the choices we made. These are both incredibly useful and incredibly disheartening. Code is always improved in these reviews, but I find it disheartening that I couldn’t think of some of the elegant ways my colleagues suggested for improvement in the first place.
(In the movie business, I think the information that comes out code reviews would be akin to “notes.”)
The draft of a story that goes to the editor might be considered a beta (or a golden master) in the software world. It is almost good enough for the world, but subject matter experts (editors, copy editors, publicity people) will look at it and offer some final polishing suggestions.
Finally, the book or story is out in the world! Hurray! And of course, the first slew of reader comments, and reviews start coming in. They are all pointing out the typo on page 5. How did anyone miss that?
It’s really no different than the day the software you’ve been working on for a year or two goes live. You’ve run millions of unit tests. You’ve demo it, you’ve made it as easy to use as possible. And 2 hours after it goes out, someone finds a bug that should have been caught 6 months ago.
I think the biggest difference between creating software and creating stories is that with a story, more often than not, you have something to hold in your hands, the product of your long hours of labor. It might be a printed manuscript. It might be a copy of the magazine the story appeared in. It might be a book. It always gratifying, on those rare occasions when it’s happened to me, when someone hands me one of these things and asks for an autograph.
With software, there’s nothing to hold, nothing you can grasp in your hands that represent all the blood, sweat and tears that went into its creation. It’s probably for the best. In thirty years of making software, no one, not a single person, has ever asked me for an autograph. If they did, I’d be a loss: there’s nothing on which to sign my name.
Which explains why this post is coming out at nearly 7pm instead of much earlier in the day ↩
Finally, I have come across what I consider to be the best definition of a project manager that I have ever seen. I have written in the past about how I dread getting asked that question, “What do you do?” because (a) it is hard to describe what a project manager does without (b) making it sound like a made-up job.
Reading Ed Catmull’s book, Creativity, Inc., this morning, I came across what I consider the best definition of a project manager—one that describes what I do clearly and accurately—but cast in terms of Hollywood production managers. Catmull writes:
Production managers are the people who keep track of the endless details that ensure that a movie is delivered on time and on budget. They monitor the overall progress of the crew; they keep track of the thousands of shots; they evaluate how resources are being used; they persuade and cajole and nudge and say no when necessary. In other words, they do something essential for a company whose success relies on hitting deadlines and staying on budget. They manage people and safeguard processes.
By changing a few words here and there, I have the definition of project manager that I have been seeking for years now:
Project managers are people who keep track of the endless details that ensure software is delivered on time and on budget. They monitor the overall progress of the developmentteam; they keep track of the thousands of lines of code; they evaluate how resources are being used; they persuade and cajole and nudge and say no when necessary. In other words, they do something essential for a company whose success relies on hitting deadlines and staying on budget. They manage people and safeguard processes.
I love this definition. It perfectly describes what I do day-in and day-out on my job. I am particularly tickled by the line, “they persuade and cajole and nudge and say no when necessary.” A project manager who taught me a lot about the job two decades ago summarize this line back then with a simple phrase that I often repeat: “As a project manager, all you have is your charm.”
I’m only a quarter of the way into Catmull’s book, but it has proven its worth with this definition alone. I feel a great sense of relief in having a good, accurate, and succinct way of describing what I do.
The new Mac Mini has been up and running for 10 days now and I have some initial thoughts. For context and clarity, I bought the newest Mac Mini (M1 2020), which is running the Apple M1 chip. I bought it with 16 GB of memory (way up from the 4 GB I have on my MacBook Air). The internal disk has 250 GB, but I’ve got two external disks connected to the machine, each of which is 3 TB giving me a total of 6.25 TB of disk space. One of the external disks is for media files and archived data; the other is a local Time Machine backup disk.
As far as performance goes, this machine flies. Applications open so much faster than on my MacBook Air. There doesn’t seem to be any performance hit with backups running and with the various services I have in the background. I really like how fast the machine is.
There are a few downsides I’ve discovered, however.
The M1 chip is the biggest blocker so far. While it is super-fast, not every app has caught up yet, and several still expect an Intel processor. For instance, I use Docker for development work, and I have to run a preview version of Docker Desktop because there is not yet a production version compatible with the M1 chip.
There are some quirks with homebrew as well. Homebrew can be run natively or using Rosetta2 which makes apps compatible, but at a performance cost. Running homebrew natively takes a couple of extra steps to setup, and some bottles have to be built locally to allow them to run natively.
MySQL runs fine on the Mac Mini, but there is not yet a compatible Docker image for MySQL for the M1 chip.
These are relatively minor issues, which only apply to someone doing development work. It appears that most places are working toward making their apps natively compatible with the M1 so I suspect most of these issues will go away with time.
For other tasks: writing, photos, general productivity, I am very pleased with the Mac Mini thus far. Given that it cost significantly less than the newest MacBook Air, it is well worth the cost so far.
I have set up the machine as a home server. I’ve got an internal web server that I am using to build a custom reading list app (that I plan on moving to my domain eventually). I am also using it to host an app for home document archive. Screen-sharing works well with it (I can use screen sharing from my MacBook Air to do development work on the Mac Mini when I am not in my office). I’ll have more to say on these things in a future post.
You can see the new computer in the photo above, peeking between the monitor and the external disks. At some point, I need to clean up all of the cables.
At this point, with the exception of a few development quirks related to the M1 chip, I am very pleased with the new Mac Mini.
I did manage to get started tonight, but it was mostly on configuration and design issues:
I upgraded my system to PHP 5.1.4 and MySQL 5.0.24. I also upgraded my installation of MediaWiki to 1.7.x (I use MediaWiki for not only developer notes and design documents, but for just about all household related stuff) Once all of that was configured, I got to work outlining the stored procedure API, which now consists of 96 stored procedures.
I expect to do more this weekend, but I’m beat tonight, and am off to bed…
Ever since I switched from a PC to a Mac at home, I have been using Booxter to manage my collection of books. Back in the day, I had developed my own, Windows-based, home grown application to manage it, but on the Mac, Booxter seemed the way to go. It served me well for quite some time.
But I want something web-based, something that will do libaray and reference management, and something that ties seemlessly into my reading list application. Over the last week, I have been working on the data model in my spare time, and I now have a model that I think does everything I need it to do. To make this work on the web, I’ve decided to do it as a LAMP application (replacing the L for Linux with a D for Darwin): Darwin (Mac), Apache, MySQL, PHP. I’m using a three-tier model for the application design and I have outlined the core API (stored procedures) for the database layer.
I plan on getting started this Labor Day weekend. My goals for the weekend are as follows:
Implement the data model in MySQL
Code the API
Export my data from Booxter into my new data model
If I can get that much done over the long weekend, I will be very happy.
As for the longer-term plan, I am trying to completely seperate the presentation layer from the rest of the application so that the initial interfaces will be bare-bones HTML. Ultimately, I will build up the look and feel using CSS. Once I have a working data management interface, I will look into porting portions of the application to my public website, so that people can search the database if they want to.
Sounds like the perfect thing to be working on during this long and rainy weekend.
I’m also looking into buying two more bookshelves for my home office since I am way out of space on the 7 existing shelve units. But more on this later.
This is no revelation. But we M$ products at work, and one which I use quite a bit is Microsoft Visual Studio 2003 and Visual Source Safe. This morning, after compiling an application successfully, Visual Studio went bye-bye silently. No application exception, nothing in the event log. Nada. A reboot didn’t seem to fix the problem. I’ve narrowed down the problem to projects that are checked into Visual Source Safe. I can’t seem to open those projects, but projects that are not checked into revision control are accessible.
And yes, I checked the VSS server and I can get into that just fine.
This sucks becuase I was about to finish something up and get it published and check it off my list and move on to the next task. Now I’ve got to spend time I don’t have trying to figure out what the heck is going on!
UPDATE: It’s all working now, although it took a threat of setting my computer out to sea and praying for a hurricane to get it to cooperate. Incidentally, after I clicked the “Save Post” button on the initial post (when, in fact, I felt like screaming!), Tears For Fears, “Shout” came onto my iPod:
Let it all out.
These are the things
I can do without
Come on; I’m talking to you