See manipulability

I’m going to run through a (possibly boring) technical problem I solved today in order to illustrate what may be a larger truth.

In my job we publish thousands of externally authored PDFs on our website each year. Along with each PDF, we publish a considerable amount of metadata, which is also entered by these external authors. When we send this metadata to our hosting site, there are often errors in the characters entered by the authors (e.g., weird diacritics and such that our hosting site doesn’t recognize). We can’t publish the PDF until all metadata errors are corrected.  

Our hosting site tells us on which line of code the error occurred. The code isn’t meant for human readers, though. In many places it stretches over two page-widths (requiring the reader to scroll side-to-side to read it), and it has no line numbers. Also, the unrecognized characters come through in the code in hard-to-see ways. Trying to find the metadata error usually means poring over a good 200 to 400 lines of code with no real idea of what you’re looking for and only a vague idea of where.

It’s extremely frustrating for everyone. Today I found a solution. Copy the code from the hosting site, paste it in Word, select “Keep Text Only” in the paste options box, copy this text from Word, and paste it in Excel. Now the row numbers in Excel correspond exactly to the line numbers of the code. The hosting site tells you which line the error is on, so you just scroll to that row in Excel and—boom—you’ve found the bad character.

My company has been using this hosting site for over six years, and as far as I know, no one has ever come up with a solution to this problem other than “Just search through the code.” We publish over 18,000 of these manuscripts every year, and I’d estimate 1 in 25 has this kind of error. It adds up to hundreds of wasted man hours. It was the kid who’s been there 19 months who finally figured out how to cut that wasted by more than half.

But I’m not writing this just to gloat. I’m writing it because I learned something important. When I shared this solution with my colleagues, a couple of them asked me the same thing: “How did you think of this?” At the time I just smiled coyly and shrugged. But of course the real answer is more complicated.

If I were to answer that question honestly, it would have two parts: 1) I got pissed and 2) I started looking at what I could manipulate. The getting pissed part was crucial. This problem of code-searching was always frustrating, but today it was particularly bad. I was searching over and over for a bad character that seemed impossible to find. It was ruining my fucking day. Suddenly I felt like one of those cheap actors in an old commercial looking at the camera and saying, “There has to be a better way!” It was obvious to me that something was broken and badly needed to be fixed.

And then I started trying to see manipulability. What elements of the code could I work with? I noticed that each line of code was bracketed in syntactic markers. In my mind I actually came up with a more complicated solution that involved copying the code and doing some finding and replacing on these syntactic markers. As it happened, Word and Excel did that more complicated part of the algorithm for me. But still, I was looking for patterns that might allow me to tool the code into something more useable for a human being rather than a machine. That was the key. And once I had seen the pattern and gotten the idea of manipulating it, the solution to this problem seemed so obvious that I wondered why I hadn’t come up with it over a year ago when I was first trained on this stuff.

I think this is a good principle to carry around: see manipulability. See what strings you can grab onto. See what blocks you can rearrange. Most of us most of the time accept what we are given. Even when algorithms are broken, when processes are backward, we accept these deficiencies like we accept an illness. It’s a bitch, but what are you gonna do? We might feel there’s something wrong about how we’re doing something (or being told to do it), but we rarely get past that part. We get consumed in feelings of wrongness and forget to just look at what it is in front of us. Nearly everything has some kind of structure to it. And if you can find the structure, you can probably find a way to manipulate it.