PROGRAMMING IN THE MEDIUM

As a codebase grows, its programmers may find themselves mired in unexpected complexity. I describe such an experience of my own, and outline some preventative measures.

--------------------------------------------------------------------------------

{frog meme}
{title slide}

Hi everybody, my name is James and I'm going to talk about what I think is the difference between working with a small codebase and a medium-sized codebase. This talk could have been called "Programming in the Large", but I don't have any experience working with truly large codebases, like Photoshop or Windows, so I've covered my arse by calling it "Programming in the Medium".

{Personal Experience}

About two and a half years ago, I started writing a new app for the tech startup which I run with my family. I won't bore you with the details, except to say that it's a mobile mapping app for horse riders.

Initially, our progress was swift, and we had an alpha release out in the first six months. New features were landing every week or so, and things felt great for me. After the first year, it seemed to us like the app was 90% complete, but a funny thing started happening. For every bug I fixed, it seemed like several more bugs appeared. New features took longer than expected to implement, and were of course prone to more bugs. Our progress slowed to a crawl.

I began to lose confidence in myself as a programmer. I couldn't figure out why it seemed to be so hard all of a sudden. I feld stupid and incompetent, and I actually started to believe I'd gotten too old to program. Another year passed, with me working full time on this app and the whole time feeling like it was 90% done. With each passing day, I was becoming increasingly jaded. But then I discovered the most wonderful thing.

{I was not alone.}

I would like to read you something that nearly made me cry with relief. It's from a book by Fred Brooks in which he talks about his experience as manager of a project developing a big operating system in the mid 60's. And by big, I mean the manuals took up several metres of shelf space. This passage is from the chapter entitled "The Tar Pit", which is accompanied by this image.

{tar pit}
{read Brooks, p4}

    No scene from prehistory is quite so vivid as that of the mortal struggles of great beasts in the tar pits. In the mind's eye one sees dinosaurs, mammoths, and sabertoothed tigers struggling against the grip of the tar. The fiercer the struggle, the more entangling the tar, and no beast is so strong or so skillful but that he ultimately sinks.

    Large-system programming has over the past decade been such a tar pit, and many great and powerful beasts have thrashed violently in it. Most have emerged with running systems—few have met goals, schedules, and budgets. Large and small, massive or wiry, team after team has become entangled in the tar. No one thing seems to cause the difficulty—any particular paw can be pulled away. But the accumulation of simultaneous and interacting factors brings slower and slower motion. Everyone seems to have been surprised by the stickiness of the problem, and it is hard to discern the nature of it.

{graph}

If we look at this graph, we see that the cost of hardware for the first computers was astronomical. As Moore's law comes into effect, we see the cost of hardware plummeting. Curiously, we also see the cost of software skyrocketing. Dijkstra, one of the pioneers of programming, had this to say in 1972:

"The major cause is that the machines have become several orders of magnitude more powerful! To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem."

{The Software Crisis}

There was a conference in 1968 where it was first sheepishly admitted that programmers were having a really hard time getting their programs to meet expectations. They named this phenomenon "The Software Crisis", and pointed out that most projects were late, if they were completed at all. Worse, the software was buggy and slow. When bugs were found, they were tricky to fix without creating yet more bugs. It was also unexpectedly difficult to adapt software to new or changing requirements.

Naturally, people were very concerned about the software crisis, and much was written on the subject. And a lot of progress was made. But Moore's law was in effect, and computers were becoming exponentially more powerful, so the software crisis remained. Programming is still hard, but for some reason we don't talk much about the software crisis anymore. I encourage anyone who is skeptical that we are still in a crisis to check out the wikipedia page, "List of failed and overbudget custom software projects". The most spectacular failure I could find on there was a healthcare system for the UK, which was in development from 2002 to 2011, never finished, and cost £12 billion.

Why is it so hard to write big programs?

"The psychologist-mathematician George Miller, in a summary of a very large body of research, first described the limitations of human information processing in 1956. It appears people can mentally juggle, deal with or keep track of only about seven objects, entities or concepts at a time. In effect, the working memory needed for problem-solving with multiple elements has a capacity of about 7+/-2 entities. Above that number, errors in the process increase disproportionately. It matters not what the somethings are – people to be managed, state variables to be remembered, function calls to understood - if there are more than about 7+/-2 of them, the number of mistakes rises sharply and non-linearly."

{read Structured Design, p69}

Unfortunately for humans, there is a strong tendancy for chunks of code to become more and more interconnected as a codebase grows. What this means is that programmers who have to keep track of too many interconnections at once become paralysed by the complexity.

{telephone wires}

Does anyone know what's going on here?

{audience}

Luckily, we have Dijkstra to provide us with sage advice:

"I now suggest that we confine ourselves to the design and implementation of intellectually manageable programs."

{Mindset}

The past six months have been a period of reflection and learning for me. I've spent a lot of time thinking about mistakes I made designing and implementing our app, and I'm going to talk a bit about how my mindset has evolved. I believe I started with "prototype mindset" and I now have "product mindset". Both mindsets have distinct goals and characteristics.

{Prototype Mindset}

One of our goals was to release a stripped down version of the app, or "Minimum Viable Product", as quickly as possible. Ordinarily, this would be a good way to bring in some income to fund the rest of development. Unfortunately, the app we were making was actually a rewrite of an existing paid app we'd released ten years earlier, so we needed to include all the features of the existing app and more. Otherwise users would have no reason to pay for the new app. In reality, our focus on an MVP resulted in an alpha release very early on.

The reason we wanted an alpha release was so we could start getting two kinds of feedback from our users. The first was feedback on the design: was it intuitive and user friendly? The second was feedback on the functionality. In retrospect, rushing to get a release out so that users can find bugs earlier was not very smart.

Looking back, I can see that these goals encouraged several other mistakes. I admit that I kludged. A kludge is an ill-matched collection of parts assembled to fulfil a particular purpose. This is what a kludge looks like in the real world:

{digger ladder}

I justified my kludges by telling myself that I would fix them up later, but in reality I rarely did. On the occasions where I was forced to, it was extremely painful. Especially where other code had come to depend on the idiosyncratic behaviour of the kludge.

I overused 3rd party packages, seduced as I was by the vast selection of open source libraries available for free. I believed that any time I installed a package I was saving myself labour and shielding my codebase for unnecessary complexity. In reality, I found that indiscriminately installing packages was a great way to bloat and complicate my codebase.

Discoving bugs or limitations in packages often led to more kludges in my code, or a time consuming, tedious and sometimes futile process of reporting the bug, finding the bug, fixing the bug, testing the fix, submitting the fix and then waiting for a response from the maintainers which sometimes never came. Don't get me wrong, I know that contributing to open source can be a noble and rewarding pursuit, but it can also be a pointless waste of time.

If you install a package, even if you only employ a tiny portion of its code, in a way you are still absorbing all of its complexity. And that can be crushing if a package is poorly designed or implemented. I am much more selective about packages now, to the extent that I prefer to write my own code to handle my specific use case.

My sister had been working at Telstra for several years when we started work on the app, and she suggested we try doing "sprints". Sprinting is part of the Agile methodology, and involes completing a set amount of work in a set period of time. My problem with this approach was that I found myself compromising on quality to meet the deadlines, especially when unexpected problems arose. I also felt discouraged from considering radical alterations of the design, which might have been very beneficial early on.

{prototype quote}
{Product mindset}

If I want my app to be a product, not a prototype, what should I prioritise? Well, the app should last a long time. There's no point spending three years making an app which only lasts two years before it's deemed not fit for purpose. And software that lasts a long time is inevitably required to do new things, so it must be sufficiently modifiable. People will always discover bugs in software they use, so the software must be sufficiently maintainable.

I think the single most powerful idea in achieving these goals is modularity. I find that when I strive to write cohesive modules which are loosely coupled, I get modifiability and maintainability for free. Something I've started doing more is "bug proofing the definition". Most really serious bugs arise from ambiguity in the specification and design.

I try to work out a satisfactory approach on paper before I go anywhere near the keyboard. And when I do write code, I am skeptical of my first attempt and treat it like the draft that it is. It might take me longer to write good code, but I know it will save me time in the long run. Product mindset means giving the code tender loving care.

{Plumbing}

The remainder of this talk will be about plumbing. I have some photos of different plumbings I found on the internet, and I'd like you all to give me your reactions. If you approve of a plumbing, give it a thumbs up, otherwise give it a thumbs down.

{tangled pipes}

This plumbing has obviously grown organically. What is your verdict?

{foundation pipes}

I'm not sure it's totally clear, but this is a concrete foundation with PVC pipes sticking out of it. Verdict?

{under house pipes}

This last one is the plumbing underneath a house on legs. Verdict?

{Consequences}

I have a couple of friends who live in a high rise apartment, which was built about 5 years ago. And it's a pretty swish apartment block: there's a waterfall and a pond in the lobby. One night, my friends wake up to the apartment flooding, and find that their toilet bowl is overflowing. They call up a 24/7 plumber, who shows up just as the flooding stops. The plumber assures them that there has probably been a blockage which has now sorted itself out, and with that he leaves. Of course, two hours later the flooding resumes and my friends call a different plumber. She comes round, sticks some sort of long grabber down the toilet and retrieves a great big wad of wet wipes. This was during the great toilet paper shortage of 2020, mind you. Anyway, these wet wipes had been backed up at the main junction, which for a high rise is a very wide pipe.

The plumber doesn't see how a few wet wipes could have blocked the main junction by themselves, so she sends in a camera. And do you know what she found, firmly wedged in the main junction that sewage from the 15 floors above them needs to pass through?

{spirit level}

A spirit level. I guess one of the builders accidentally dropped it during construction and left it there. The quote to remove it, and thereby prevent future blockages, was $100,000! Strata decides it is too expensive to fix, and with that my friends are left dreading the next flood caused by someone flushing something they shouldn't.

{Bibliography}

Thanks everybody. Are there any questions?