(home)

📝
The Grail Quest Enters the Roaring 20s— Polyrepos, Polycomputers, and Backyard Blobstore (DRAFT)


Status

Rough draft.

Introduction

A history of the fun in computers. Tactical thoughts on getting a lot of it back at once.

Background

This post is inspired by ideas from three different sources:

You'll see each of these appear later in the post, but I also wanted to put them up front so you can get a feeling for where I'm coming from.

1. The Grail Quest Enters the Roarings 20s

The least fun computer I can imagine is an interactive TV. Imagine Google Stadia, but expanded to the entire system. Raw user input travels from a thin client to an opaque network object, raw output comes back. There's no manual and no technical or conventional guarantees about its behavior. IN PARTICULAR I can't change the color of things. Color is very important.

In comparison let's call the most fun computer I can imagine the Grail System.

The Grail System makes me feel like this:
Alan Kay in front of a PARC computer with a huge smile on his face

(Pictured: 👤 Alan Kay)

I'm not sure if The Computer is a Feeling as 👤 Tim Hwang and 👤 Omar Rizwan say, but the Grail System definitely is.

And I'm not the only one with something like it in the back of my head. A silver lining of Xerox PARC and Symbolics going out of business after making such cool things is that they injected a what if? mystique into our subculture. What if PARC had gotten its act together, shipped something cheap early, and grabbed the market? What would an Alan Kay computer look like after 30 years of iteration, with millions of smart people playing with it? Or what if lisp machines had won, and we'd gotten a computer descended 545 Tech Square instead of Redmond?

This shared what if idea doesn't actually have a name, but calling it the Grail System will do well enough for this post.

(The details of the Grail System are different for everyone, which is probably an advantage. Two commonalities are that it's "more owned" by the user— whatever that means— and it's perfectly empowering. I can say what it means for me. Firstly, I can change the color of anything and have that change stick. Secondly, it's both more and less social than modern systems. I want a mind meld feature with certain friends where I share almost everything with them, and I also want more stuff to run locally so I can cut myself off from the internet when desired. We'll come back to this.)

Now let's talk about something concrete. However one imagines the Grail System it probably includes Bring Your Own Client-style networking, both the computer-to-computer and social varieties.

We leave the realm of alternate history here since this once actually existed. In the 70s and 80s the triple alliance of ARPA, universities, and business built it. Thus the 80s Eden: TCP/IP, email, IRC, Usenet, etc.
Early map of Usenet

I'm guessing readers who were around at that time consider the phrase Usenet Eden to be the oxymoron to end all oxymorons. But it really was. Admittedly it was janky. But there was no spam. There was no Embrace, Extend, Extenguish (EEE). There was little security. It was an intellectual group, full of people from universities and research labs. And there was a surprising amount of non-commercial motivation, which continued all the way into the 90s:

Rebecca: On the comment "there is no reason Boston shouldn't be a hotbed of startups..." Culture matters. MIT's culture is more intellectual than entrepreneurial, and Harvard even more so. I'll tell you a story: I was hanging out in the MIT computer club in the early nineties, when the web was just starting, and someone suggested that one could claim domain names to make money reselling them. Everyone in the room agreed that was the dumbest idea they had ever heard. It was crazy. Everything was available back then, you know. And everyone in that room kindof knew they were leaving money on the ground.

We can never go back to this innocence. First the protocols stagnated after the triple alliance dissolved. Then Eternal September happened in 1993 and the first major spam incident in 1994. Bring Your Own Client-Eden lived on, but its days were numbered.

The 90s were dominated by Microsoft which is too dismal to talk about. Energy moved to the web. Then Facebook began its rise which brings me to the point of this story.

The idea that Bring Your Own Client-Eden was going to be replaced by bundled-UI, offsite data companies, one company per application was just too massively depressing. It triggered a revolt. In 2010 Diaspora raised the second most successful ever Kickstarter at the time to fight it.

Let's call this the ⚛️ 2010s Protocol Restoration Revolt.

I briefly step into this story myself as an intern at Tent.io. It was my first ever software gig. Like Diaspora, Tent wanted to create a federated competitor to Facebook. Their model was rougly email with its hubs and spokes, but unlike email Tent would store general purpose data. This way people wouldn't have to build a protocol-per-application anymore (which is a gargantuan amount of work once you factor in spam defense, EEE defense, etc). Instead everyone could pick a Tent provider like they do an email server. Apps could talk directly to that.

The revolt failed. Diaspora didn't take off, though it's still around. The Tent founders put their project on hold to go to YC for an unrelated project, hoping to come back to it later. This never happened.

10 years went by. A lot of fun things occured in the computer world over the next decade related to industry and convenience, but personal computers continued their slide into unfun. Google EEE'd instant messaging and RSS. Slack EEE'd IRC. All the FAANGs except Apple continued trying to turn our eyes into sad money piñatas (especially for those not smart enough to find the dehellscape button).

Then the roaring 20s hit.
OpenAI logo

The whole tech world is in disarray. Anything can happen at this point.

And this isn't just AI. A ton of cool things are going on: digital gardens, zettelkasten, new browsers, systems languages, even hardware startups (though the latter first shipped in 2017).

Remember the second-best-Kickstarter-ever I said Diaspora raised in 2010? That was for 200k. The 2010s might as well be the 19th century with how much things have changed. Programmers are better now (or at least there are more good programmers). Software is better. Ambitions are high. I think someone out there has a good chance of building their own Grail System in the 2020s and actually getting it to catch on.

That means it's probably time to develop a more concrete idea of what a maximally empowering computer would be than, "like Smalltalk or Lisp machines, but modern."

We won't do that in this article though.

But we will do the preliminary step which is examine what kind of computers we have now. Before trying to build something new new it's essential to see, to truly look with completely soft eyes, at what already exists. Hold on to your keyboard. Things are going to get weird.

Random Interlude: Heathkit

Heathkit was a company that made electronics kits from 1947 to 1992 (they've recently reentered the business). A lot of people have great memories of them. They were cheaper than prebuilt electronics. They were also empowering because they gave people the mindset that electronics are things that are built by humans just like anything else. Jobs in particular mentions Heathkit having that effect on him (see 📙 Becoming Steve Jobs).
Heathkit ad with the text: "I built this 16-bit computer and saved money. Learned a lot too."

Kits are a good thing. Normally it's easy to imagine how a good thing could become bad if taken too far. But kits are so wholesome, it's hard to imagine what this could look like, right?

End interlude.

2. The Dual System

So we said we were going to try to look at modern computers completely clearly, seeing them without preconceptions.

A good trick for this in general is imaginary conversations. Pretend 👤 J. C. R. Licklider walked into your room straight out of ARPA and asked to see your machine. What would he say?

Well, he'd probably just want to talk about LLMs. But what would he say about the system itself?

Of course we can't actually know. But I can tell you what I thought for a long time was the crux of the modern personal computing environment. I thought it was that we run two almost completely separate systems side-by-side— the desktop and the browser— and that the bridge between the two is very narrow.

You could call this the ⚛️ desktop+browser dual system. It's sort of a digital Petronas Towers:
Petronas Towers which is two skyscrapers connected by a narrow bridge

There are other modern personal computing environments like smartphones, 💾 ChromeOS, etc. But the classic workhorse of modern personal computing is this. Mac/Windows/Linux on one side, the web browser on the other.

This is different than a hybrid system. In a hybrid you'd have one thing that does two tasks: a desktop that also browsed the web, or a browser that was also a desktop. So 💾 ChromeOS is a hybrid, but desktop+browser isn't.

The dual system means for any project we have to ask ourselves at the beginning, "which side should I do this on?". It's hard to switch after you've started.

For comparison, imagine a hypothetical dual system: Windows+Mac. In this world no one uses web browsers, but they all run either Windows with a Mac VM open at all times or vice-versa.

There's a huge difference between the hypothetical Windows+Mac dual system world and the actual desktop+browser world. Windows and Mac are very similar. They both have strengths and weaknesses but they basically do the same thing. This scenario would be defined by distinction without a difference.

By contrast, the desktop and the browser are almost complete opposites:

Attribute Classic local desktop app Web app
Installs manual automatic
Updates user controlled developer controlled
Sandboxed sometimes yes
Social no yes
Always on no yes
Data ownership user developer
Language any JS

I think this was a pretty likely outcome given that they were built from opposite directions. The desktop was planned, the browser was grown. The desktop started as a single user personal computing environment (unless we go all the way back to timesharing), the browser started as a networked document viewer.

Since the two sides are so different users rarely have to make distinction without a difference choices. Thus we're less aware than we might be that we're actually running two different systems that barely can barely talk to each other.

This seems like a big deal. I used to think it was the big deal. Now I think it's unimportant. We'll see why in the next section.

3. Polyrepos and Polycomputers

The modern programming community is deeply skilled and dedicated, but they're also the splitters to end all splitters.

Two big examples of this in the 2010s were microservices and polyrepos.

Both absolutely have their place. There are situations were they have advantages over monoliths and monorepos. There are situations where there's no choice and they're required.

However, this isn't how they were used at many places. Lots of teams adopted both when it wasn't strictly necessary. They did so for good reasons, but the joy of splitting was a factor too. And those splits allowed more splits! Once you have microservices you can add more languages! Once you have more languages you can have more frameworks, libraries, patterns, experiments!

Fool me once, fool me twice, etc. At this point let's walk the perimeter and check what other splitting has been going on.

Here's a tweet that's been stuck in my head:


In well-designed operating systems, like classic Mac OS, there is a single uniform visual style and no developer needs to care about or waste time on styling.

That's interesting. We certainly have done crazy splitting on style. For webapps in particular I can click around all day and not see a single style twice. In fact it's an idiomatic demand of the platform (other than things like open source personal blogs).

But that's just appearance, right? It's not the systems themselves.

Let's continue our perimeter walk. For the next example I tried to think of the most median, straightforward app I use. I went with my utility provider. Here's a screenshot:
Picture of utility provider's bills and payments screen

In the process of clicking around to create this screenshot I hit a bug. This took about a minute and happened the second time I selected "Account: All":
Notification saying "Account History is not available for this account"
Also the latency on one of my clicks of "Apply Filters" was 11.92s (to return 0 results). There was no "waiting" indication. My ping to google.com is 7.9ms.

Ha ha, look at this enterprise, old school company, can't even make a website work right, just like all the others... I actually don't think this is the right response.

In particular I think we have a case of beam-in-eye.

All the utility provider wants to do is:

  • Make some documents available publicly on the network (About, Pricing, Contact, Terms and Conditions, etc)
  • Share a spreadsheet on the network with me showing how much electricity I've used and what my current balance is. It needs R+W persmissions and wants me to have R permissions.
  • Notify me when a payment's due and update the spreadsheet when I've paid it (the first part of this can be handled by Stripe etc.)

The essential complexity of this requires almost no custom code. Documents could be Microsoft Word. The Spreadsheet could be Excel. Excel in particular would be way, way better than the table-viewer thing they hacked together.

Here's what they actually had to build:

  • An accounts system
  • A visual style
  • A UI component library
  • A persistence system
  • A search system
  • Application software (eg the spreadsheet from the first screenshot)
  • A configuration system ("Settings")
  • A backup system ("Export")
    Bruh, THIS IS A COMPUTER

These two are common as well (though my utility provider is smart and just uses email+phone):

  • A messaging system
  • An organization system (likes, lists, bookmarks, etc)

It would be impossible for each utility provider to build all this from scratch each time, much less every app in the world. In response they use frameworks.

Remember asking earlier how the Heathkit philosophy could ever be overdone? This is it. Instead of the tech world providing non-software companies with a system that handles these functionalities, we provide them with a kit like Ruby on Rails or Django. Then they bang the whole thing together as fast as they can, resulting in all of the different search bars, account systems, visual styles, and organization systems we know today:
Logos of dozens of tech companies from 16best.net

Admittedly these aren't full computers since hardware and language support is provided. Each app builds on top of an OS on the server side and the browser on the client. Technically I think they're something like an application-specific desktop environment without the desktop metaphor.

But the result to the user feels the same. The commonalities provided by the browser are just very, very low level. Every table viewer is different. Every search bar is different. Sure, <p>, <tr>, and <td> tags are the same across each of these systems, but I have a copy of Excel. Let me use my Excel! I know SQL. I know almost every one of these sites has a copy of my data in SQL. Let me use SQL!

But because these kit-built systems must be squeezed through the microscopic, <h6> sized interface all those commonalities are gone before it gets to me.

This is our answer to, "what else have the splitters been up to?" besides microservices and polyrepros. In comparison to this the ⚛️ desktop+browser dual system divide just doesn't seem that important to me. The crux of the modern computer its ∞-system nature, the ⚛️ polycomputer.

Interlude 2: Let's Regroup

Some comments on this situation, in no particular order:

  • 👤 François-René Rideau's Houyhnhnm Computing, Chapter 7: Platforms not Applications has some great observations about the consequences of the world-to-itself nature of modern apps
    • On customization:
      • At all times, each of these modules, including all common editors, will include all the user’s customizations; this makes writing customizations much more worthwhile than if separate customizations had to be written for each application, each in its own language with its own learning curve, as is the case in Human computer systems.

      • I'd put this more strongly, this is why no one cares about customization or end-user programming anymore. If you have a system with a single search bar it makes sense to want to change it, extend it, develop expertise with it, etc. There is absolutely no point even considering this in a system where you regularly use 20+ search bars in the course of a month.
      • The consequence of this is that a huge chunk of the desire for end user programming and even understandable computers themselves is now missing. A key point of getting the Grail System built will be reigniting this desire. However I'm not sure how to do it when so few people have experienced what a unified computer environment is like.
    • On division and specialization of labor:
      • Human computer systems display a singular failure of division of labour and specialization of tasks. Developer talent is stretched thin, as the same tasks are done over and over, once per application, almost never well, by developers who are seldom specialists in those tasks. Meanwhile, those few developers who are good at a task cannot focus on doing it in a way that will benefit everyone, but must instead improve a single application at a time.

      • In other words (inspired by this tweet) you really want this person doing the visual design for the whole system, not just stuck at Airbnb or something:
      • Classic picture of Apple's Susan Kare lounging at her desk, from around the time of the Mac
      • I'm not sure who the Susan Kare of search systems, image editing, video players, configuration, etc is the same deal goes for them too— you want them to be designing at the system level and then for each program to use those systems.
        • Kits only help so much. Without expertise you don't even know which kit to pick.
  • There's a parallel to 🖊️ Zawinski's Law of Software Envelopment, except in the modern formulation apps don't expand to use email. Instead they grow their own private messaging system.
  • The current problems aren't actually due to any "splitter" action, though it may be causing some complacency about them. Rather they're due to just a couple phenomena:
    • (1) ⚛️ in the modern age only networked/social program matter. The desktop provides no built-in social features (Unix used to but since no one does timesharing anymore they've atrophied). Thus old school desktop local programs are irrelevant.
    • (2) Browser programs are the only ones that work across Mac/Windows/Linux. Plus they're convenient to install and update. So everything starts in the browser. But the browser doesn't provide much standardization, so here we are.

3. Mozilla are the Baddies, not Apple

If the polycomputer situation means every company has to do a bunch of wasted development, why aren't they complaining?

They're not complaining because the end result is a completely one-sided balance of power with end users and a river of money. There's literally no better lock-in imaginable than "I have all your data. You can have an undocumented JSON dump of most it if you want. By the way export is the worst maintained feature in the entire system, so good luck."

Remember traditional local desktop developers? They're aren't many left. Their software was pirate-able. They had almost no lock-in since all the data was sitting on the user's machine. Users could even download a completing product and try it out on that data before they'd even decided to switch for sure, just because they felt like it.

(By the way the pirating thing was a genuinely destructive problem. Developers have to get paid, unless we either don't want good software or we want to force them to try something else.)

Turns out that something else was SaaS, which is a slight misnomer since its actually Database and Software as a service, DaSaaS?

SaaS has all the balance of power of interactive TV. Folks, that's where enshittification comes from! Being "pro-web" is like joining the struggle over company towns— on the side of the company towns!

Admittedly there's no other option for normal companies right now. But Mozilla is different. They've got a flagship web browser and a ton of money. They should be building new systems that empower users, not trying to speed up the already unstoppable slurping away of users' data.

In comparison Apple's done its best to promote old school, local only, Bring Your Own Client desktop programs. This is a completely doomed effort but at least they're trying.

4. We've Gotten off Two Swings Already

I think the root of the problem behind all this is simple: there's a balance of power between users and services.

Users' systems right now are very weak:

  • They have no consistent online presence since they don't have servers. A good number of them have desktops, but a desktop behind a domestic internet line with a domestic power supply is a poor substitute for a server.
  • The ⚛️ desktop+browser dual system has a truly terrible data model. On the desktop side it's a unique hierarchy which has been obsolete since forever. On the browser side it's none at all (please don't say IndexedDB is a subsitute for a user-facing, persistent, high level data model).
  • They have no builtin social features. Like it or not ⚛️ in the modern age only networked/social program matter (with some exceptions). It's funny because in the days of timesharing Unix actually did have plenty of social features built right into it, but those are as obsolete as Morse code today.

Polycomputer SaaS fills this vacuum, not because of nefariousness but rather because there's no other option. I think both businesses and users would want to switch if they had another choice.

(Note that this doesn't mean all businesses. Some businesses like Goodreads are just CRUD operations+an application specific social network. These would go down fighting since they would be nothing left for them to do after the switch.)

One answer to this is to build the 💾 Grail System: make a new computing system from the ground up to be much more powerful in ways that matter to users.

However, from a strategic perspective going right to the Grail System would be very hard. Maybe it could happen now that we're in the 2020s. But it's not the "meta".

Maybe the "meta" of this game should be to build the datastore first. Let's call this the ⚛️ POD-first strategy since one of 👤 Tim Berners-Lee's projects calls personal online datastores PODs.

Following the 💾 Perkeep project convinced me that an always-online personal datastore is a great idea on its own merits. And the Grail System would need one anyway, so this is a Minimum Loveable Slice of something bigger. Even if the rest of the system never gets built this would be really cool to have.

💾 Perkeep wasn't also a social network, but Tent.io was. I think this is the right idea too— the social aspects of computing are just too important nowadays to keep replicating them differently across every app. So Tent consisted of:

  • A datastore (persist data, give access to it, move it around, etc)
  • A data model (plus metadata, schemas, etc.)
  • A built-in social network

As described Tent didn't work out, but things might have been different if the founders had had explosive success with their other YC company and been able to come back to it.

But the ⚛️ POD-first strategy isn't dead since it's what 👤 Tim Berners-Lee is trying with 💾 Solid (this is actually where the name POD comes from).

Solid's under active development and I really hope it works. But regardless of how it goes I think we now actually know how to get to the right playing field, into the batter's box, and face the right way. Now it's just a matter of time. Roaring 20s bless these ventures!

5. Argh Stop Trying the Most Complex Thing Possible

If you're considering the ⚛️ POD-first strategy I have one piece of advice. When you see the absolute most complex possible thing you could program, the thing that pushes the bounds of technology or adoption, the thing that's never worked before, PLEASE don't base your project around it.

Web3 has crypto. Goodness it was frustrating to leave Tent then spend the next 10 years hearing how people are going to decentralize the web with crypto. You don't need crypto! Just give people a general purpose datastore that's always online! If you want to get crazy you can federate it, just copy email with some modernizations. Why are you building ASICs? Argh!

💾 Solid has RDF. There's something to RDF, but it has a terrible ability to actually get adopted.

People love using public/private key cryptography. Sometimes they forget to put in an account abstraction. What then happens when someone loses their key or gets hacked-- they lose their entire digital life? If the goal is mass adoption you need accounts.

People love peer-to-peer networks. This is way, way harder than email-style federation.

People love local first. The problem is that users really need an always-online presence anyway (for instance for backups). So you have to build a POD no matter what you do. At this point why not go ahead and get adoption for that? Requiring an internet connection isn't a barrier for success as the market has shown very clearly.

Of course all of these technologies are cool in themselves for a variety of reasons. I just think they're a distraction if you're attempting the ⚛️ POD-first strategy specifically.

6. Intoducing Backyard Blobstore

As Tent and Solid improve on having to build the whole Grail System at once, there's actually another step we can take beyond those two to do even less.

This is because it turns out it's really hard to pick a data model to replace hierarchical filesystems for personal-scale computing. It's clear we need something else. But what?

There's key/value. There's Datomic-style EAV. There's graph databases. There's the associative model of data and RDF-*.

We could drop all of this. Instead of making a personal-yet-social datastore we could make a social blobstore. Literally a blobstore with a social network attached. Let the applications figure out how they want to encode meaning into the blobs.

This way the effort of building nice social features— friending, sharing permissions for individual posts (excuse me, blobs), Google+ style circles, etc— would only have to be done once. It could be shared by different projects at a higher layer, each of which might have a different idea for what data model they want to use.

This might also maximize early fun since the system would start simple and it would be easy to get apps to work with it. Apps could only post and edit your own posts, but they could fetch any post that's been shared with you. Thus making social apps for things like reviews or turn based games would be easy.

Federation is more involved, but you wouldn't have to start there, you could start with a single node and get that going first.

I'm going to write a version of this and try it out with my friends. I'll let you know how it goes.

In the meantime I hope this post gave you an interesing perspective on personal computing world. Go and build great things.

Backlinks