Dillon Shook

Web 4.0

Wed, 29 Oct 2025 12:23:12 GMT

It’s safe to say that Web 3 was a flop already right? 😛 Anyways, this post is a vision of what could come next for the web, whatever version we’re on now…

The other day I was trying to find a new React component library. We’re using Radix right now but it’s a “headless” component library where you not only have to style everything yourself but also create wrapper components to encapsulate the complexity of all the sub components and customize them for all the missing behavior we need. It’s no joke to make your own (good) component library that has a consistent interface for things like click events, form labels, validation states, disabled components, accessibility, etc, etc. On top of all that Radix is also missing what I consider pretty table stakes components like a Combobox and multi-select field so you’re once again left to fend for yourself.

So I spent the better part of a day looking for a new library. I ended up combing through 11 of the most popular ones and for one reason or another they all pretty much got ruled out. Some were missing basic components we needed, some didn’t support SSR, some would be a big pain in the butt to customize the styling to match the rest of the site, some were using different styling technology than the rest of the app which would be a maintenance headache.

All this got me thinking… How much time have we wasted re-inventing the wheel? Each of these libraries represents multiple thousands of hours of our collective development time, and there are hundreds of them just for React! Think about all the other frontend frameworks that have their own libraries, and then all the various desktop application frameworks that have been built over the years. And how many different build chains have there been with how many hours sunk into churn?

Wouldn’t it be nice if we had one amazing open source way to develop applications instead of thousands of half baked ways?

And yes, I know, I know…

But it still feels like we’re pre-industrial revolution times with everyone hand crafting their special parts. It’s high time we re-invested in some standardization. So to me that’s creating the next version of the web which puts us at 4.0 for whoever’s making up the version numbers.

There was a good comment on Hacker News the other day that captures the feeling:

The reason the web changes so fast, and there are so many rewrites, is the same reason a puzzle whose pieces don’t fit together keeps getting shifted around and restarted.

People are looking for a satisfying non-leaky abstraction to build upon and they don’t find it with web technologies. They get close, but those last few pieces never quite fit, and we lack the power to reshape the pieces, so we tear out all the pieces and try again. Maybe this next time we’ll find a better way to fit them together.

— Buttons840 on HN

Isn’t this what your OS should do?#

Some of you may be thinking that this is a job for your operating system, and it should be providing the tools to build applications easily on top of it. The problem is that all the operating systems people use day to day are owned by companies with negative incentives to make open source cross platform tooling which means no cross platform apps. The operating systems that are open source have a good history of re-inventing the wheel themselves.

Besides the fact it’s pretty much impossible to create a new OS now, there’s never going to be a time where everyone is running the same OS anyways so we’re always going to need an application layer on top of OS that is one abstraction level higher that can hide the details of running on different operating systems.

Native apps will always exist to push the boundaries of what’s possible on a platform, but most people and business don’t care about that and just want the easiest and most cost effective way to build cross platform apps that are easy to maintain across all the platforms. Plus, we still need a solution that scales from a static site that should load in 10’s of milliseconds all the way up to full featured apps. The OS install and update process is a huge drag for native apps. No one wants to install your personal home page.

So then what?#

I’m primarily a web developer, so what I’m thinking is of course shaped by what I know, but the web has been the most successful cross platform application platform ever created. So what I want is like the web, but better. Re-imagined. And suitable for building applications of all kinds, from the simplest static content sites to the most complex desktop apps you can think of like Photoshop or Excel. All with better performance, reduced development time, and more people rowing in the same direction than what’s happening today.

Backwards compatibility has been one of the web’s greatest features for (checks wikipedia…) 34 years now! But it feels like we’re due for a breaking change reset that incorporates all we’ve learned in the last 3 decades and makes building apps for the next 6 decades faster, less bug prone, and easier than it is now. I’m not a browser developer so I can only imagine how much cruft is hanging around slowing everything down to handle the bajillion edge cases for maintaining backwards compatibility for so long.

So lets start laying down some details, even if they are a little vague because this is daydream not a design document.

Web 4 Overview#

To put it simply: new protocol, new markup language, new scripting language, new styling language, new standard components, new mechanisms for data synchronization.

We’ll first need a new protocol to tell the recipient we’re sending data which needs to be interpreted differently by the browser which also requires new capabilities. There may need to be some new verbs introduced (see the Backend section ) but fundamentally it’ll be very similar to HTTP/3 under the hood.

The rest will be very recognizable as upgrades to the existing way the web works. There will still be markup sent to define layout and maintain accessibility. There will still be a styling language to define the visual presentation. And there will still be an application language controlling both the markup and styling, as well as connecting to a backend to send and receive data.

I just want everything more interoperable and cut from the same cloth. Something that’s cohesively designed to work as well together as possible and lower the barrier to learning.

Let’s start by looking at the markup.

Markup#

Have you ever wondered where HTML got the whole bracket and tag syntax from? I didn’t until just now when I was starting this section but apparently it was adopted from something called Standard Generalized Markup Language that was developed in the 60’s.

Since we’re making a breaking change, now would be a good time to update the syntax to be much closer to the other languages we’ll need to use. It’s tempting to think it’d be nice to just have one language for markup, styling, and application code, but there’s power and brevity in having domain specific languages for each so they can all focus on their individual task and keep separation of concerns. Having said that though, there’s no reason the languages shouldn’t be from the same genealogy and share as much syntax as possible.

Here’s a simple example:

  dsml (lang="en-US") {
    head {
      title {"My First Web 4.0 Document!"}
    }
    body {
      grid(columns=2, gap=8) {
        link(to="/blog") {
          img(
            src="/picture.webp",
            caption="My pretty picture",
            zoomable,
            sizes={{400:"/picture-400px.webp", 1800:"/picture-1800px.webp"}}
          )
        }
        section (component={someNamespace.SectionComponent}) {
          p {"Welcome to my site!"}
          button(variant="secondary") {
            "Press the button, Max"
          }
        }
      }
      /* Oh wow, normal comments! */
      someNamespace.myCoolComponent (data={[4,8,15,16,23,42]}) {
        "That has child text"
      }
    }
  }

There’s a few things to notice here. It’s similar to HTML, but instead of opening and closing tags with attributes, but now everything looks more like function calls with optional named arguments. That’s not a coincidence. As you’ll see in the scripting section, everything is really a component, not just a tag. Each component’s ~~attributes~~ props can be any standard data type including objects (like the image sizes), not just strings. With this setup there’s also no more chance of mismatched closing tags, just mismatched brackets as in most languages but with editor formatters that’s a well handled problem.

Most ~~tags~~ components you’ll use will be from the standard library, and the intention is that you’ll be able to look up the source of each of the standard library components and see exactly how they’re implemented from primitives, as well as all the props they take to customize their functionality to suit your needs.

If you do need a custom component, you’ll just specify the script as the component name which will bind it up to put whatever it renders in the right place in the DOM. The children and props specified are the initial values the component scripts will use before they re-render.

With Style#

Another big change from HTML/CSS as we know it is that here all style attributes can be passed as normal props. This has some pretty cool advantages we’ll see in the scripting section but also allows seamlessly customizing components both in look and functionality in the same way. Of course there will have to be good tooling support showing the reserved style attributes so you don’t accidentally create naming collisions with your custom props. This format is also along the same lines of what Tailwind is trying to accomplish but having first class support support for it means you won’t have to learn two languages to do the same thing.

Inline with my thoughts on Tailwind, there’s a breaking point for readability with how many props/classes should be on a component. After about 10 or so, the complexity is sufficient to justify pulling the styles out into a separate styling section where you have both more power and more capability to organize and document the code for readability. Styles should be scoped to components by default and only have the extra syntax to globally style the app for the more rare case when you need to do that. For example, when you want to install a theme. I can imagine sites popping up where you could download theme style sheets that completely change the look and feel of the standard library components that everyone should be using (anyone remember CSS Zen Garden?).

Styles should also be able to override and customize every part of native components like the icons for checkboxes and the circles and dots for radio elements without completely replacing them like you have to do now. Like I mentioned in the markup section you should be able to see the source for native components and see exactly what elements they’re rendering so you can style them. Using real primitives everywhere should also reduce or completely eliminate the need for pseudo-elements like ::before & ::after.

  /* Variables with the same syntax as your scripts */
  var padding-size = 8px;

  grid {
    background: gray-800;
    border: 2px solid gray-900;
    padding: padding-size;
    /* Inline calculations with variables */
    margin: padding-size * 2;

    /* Conditionals also with the same syntax as your scripts */
    if(screen_width < 400px){
      padding: 4px;
    }
  }

  /* Selectors based on component props */
  img(zoomable){
    rounded: 100%;
  }

Here you can see the styling language looks similar to the markup language like selecting elements to style based on their current props. But it should also look like a subset of the scripting language and allow you to use the same syntax for variables and conditionals. TBD whether or not it’d be a good idea to allow loops in the style sheet. It could be useful for setting up variable state to use later, but also potentially hazardous to performance since loading the style sheets will still be a blocker to first render. There definitely should be support for functions that apply styles though, a la sass mixins.

To further intentionally blur the line between styles and scripts it should be easy to share your variables from scripts to styles or from styles to scripts. A good usecase for this I run into all the time is needing to share the color palette between scrips and styles without having to duplicate variables. There’s a lot of cases where your scripts want to dynamically color something based on state and should be able to access color palette variables, potentially manipulate them, and then apply them back as styles while still having them be strongly typed like all the other scripting variables.

TIP
Color manipulation in CSS has already come pretty far and you can do cool stuff like take one color in a variable and dynamically brighten and lower its opacity in one line:
oklch(from var(--color) calc(l + 0.1) c h / calc(alpha - 0.1))

Another big hole in CSS’ current capabilities is being able to define custom shaders (both geometry and vertex) for anything the browser draws. I found a couple decade old articles talking about this but apparently it never made it into the spec which is a shame. You can do some basic stuff with the predefined filter and backdrop-filter properties but it’s so limiting. You can make some truly wild yet performant visuals with shaders and I’d love to see web designers explore a whole new domain of possibilities. Think simulated material backgrounds that react to the cursor acting as light source. Think button borders that simulate pond ripples when you click them. These are silly examples but there’s really infinite possibilities and so many creative people out there that I’d love to see what they come up with. It’d also make web games much richer without having to resort to rendering everything in a canvas. All the shader effects would need to have fallback colors for accessibility and user preference reasons of course but I’d be leaving that setting on all day every day.

There’s a huge grab bag of other stuff I’d like to see improved like native shadow presets (with simulated light source), easier to understand and use layer stacking than z-index, a better thought out contrast color, anchor based positioning (put the top right of this box at the center of this other box), simplified naming (flexbox justify/align I’m looking at you), native nested selector support, etc, etc…

There’s a lot of innovation happening on the web in your style sheets right now! I want to keep that up and encourage sites to look unique, personal, and an artistic expression of their creators. This is still possible even with everyone using the same components for functionality under the hood.

Scripting#

To me it’s safe to say that the web has settled on components that rerender as a function of state for building applications. React really did revolutionize web development with that idea (or at least popularizing it). For those of use who remember the before times, it was grim having to manually update the DOM when your state changed or you were reacting to user events. The current situation is far from perfect though of course.

The first sticking point we’d have to decide on to make it better is the language. If we’re pushing the reset button, we better re-evaluate what the best choice really is.

I want a language that:

Is strongly typed with a powerful type system like Typescript’s but with types available at runtime for reflection.
Focuses on performance
Is part of a strong ecosystem with lots of existing packages
Has a comprehensive standard library so we don’t have to use all these different libraries to do the basics
Has the ability to express refined types (arrays of known length, formatted strings like emails and UUIDs)
Allows you to specify dependencies’ permissions where you opt into allowing disk/network/privileged API access
Has robust dependency injection support
Is terse yet expressive
Skips the import nonsense and makes the compiler figure it out with namespaces
Can compile to efficient code (both in size over the network and runtime) to save the browser parsing time

The closest language I know of to checks the boxes I’m looking for is C#, but it’s missing a few things. The good news is though that if this project were to ever actually get built you would need to create a language extensions to support returning the markdown language in the scripts like JSX/TSX files do. So you could also customize the language a little bit more to make it more purpose driven for this usecase which would also be a nice excuse to have the language be owned by the open source group instead of whatever corp or org the language originated from.

For actually adding interactivity the idea is that you can set the component attribute on a native component to customize its behavior or just create a new custom component which can render whatever it wants. So in the markup example above the section (component={someNamespace.SectionComponent}){ line is attaching the SectionComponent script to control the native section component. It would be able to hook into the native components’ state and user events to only change what it needs while leaving everything else to the native implementation. The someNamespace.myCoolComponent line is a custom component.

As mentioned above, I want the props (equivalent of HTML attributes) to be both strongly typed in what the component expects while also supporting any language built in type including arrays, objects and functions. There’s so much engineering that’s happened just to get over this limitation.

Two way binding a la svelte (and many others) is such a nice quality of life feature when you need it like for getting form component values without lots of cumbersome boilerplate.

Since all props are just in the DOM (whatever its evolution is), Server Side Rendering should now be easy. The backend can just render the application in whatever state is requested and send the document with all the props set up for the scripts to initialize with. Nothing should have to re-render when the scripts initialize until some user interaction (or network refresh) happens, and the browser can queue up user events for the scripts to process once they initialize so you never end up with those scenarios where a user clicks a button but nothing happens because the script isn’t ready yet.

Without further ado, here’s a very hypothetical example of what a component script might look like

// Similar to parameter binding in .Net minimal APIs, the ILogger comes from the DI container
class SectionComponent(Props props, ILogger logger){

  private views = 0;

  onButtonPress(){
    logger.Log("Button was pressed")
    // Just assign to variables and the compiler figures out rerenders
    views++;
  }

  render() {
    // Return markup syntax just like TSX
    return (
      section (...props){
        p {$"Welcome to my site! Views: {views}"}
        button(variant="secondary", onPress=onButtonPress) {
          "Press the button, Max"
        }
      }
    )
  }
}

Comprehensive Components & Standard Library#

A sizeable part of this whole idea is around having an awesome fully featured component library. It bugs me how inflexible and lacking native HTML elements are which means everyone ends up building their own which end up usually missing some functionality and makes everyone’s user experience inconsistent. As a perfect example of how terrible some native components are, let me remind you what the native multi-select component looks like:

I dare you to coach anyone non tech savvy over 50 years old how to use this.

As a grab bag, here’s my wish list of components that would be nice to stop building over and over and over again:

Forms that actually handle real world usage with dependent state handling, dependent field validation, multi step workflows, and strongly typed values returned
Form Inputs that handle icons, units, and formatting
~~Number inputs that actually result in your script getting a number~~ (just learned about valueAsNumber)
Non sucky multi select, that includes variations in styling for tag style, count style, text style
Combobox / async selection search
Checkboxes with a variant to render as a toggle
Tooltips
Dialogs
Carousels
Drawers
Toast messages
Graphs & Charts
Tabs
Rich text editor
Calendar
Expandable Tree view
Mosaic image layout
Tables that can handle sorting, filtering, pagination, resizable columns, selections, inline editing, expandable rows, empty states, and the whole kitchen sink
Standard layout components

Since the framework would be open source you would not only be able to fix bugs and add components everyone wants, but also dive deep and see the implementation of native components and how they’re composed from primitives when you need to customize them.

Another benefit of having the standard component library be so full featured (and therefore heavily used) is that accessibility tools (screen readers/voice input/AI) can build out specialized functionality for each of them and not have to worry as much about everyone’s special snowflake components that forgot about a11y. Keyboard users also rejoice.

Authentication#

Another easy layup for the evolution of the web is fixing the nightmare that is authentication as we know it today. How many logins do you have stored in your password manager? I have 636. And many of those sites have asinine password requirements that make them less secure and more annoying to use. Even more have already stored your credentials insecurely and been hacked. I, just like everyone else, am usually too lazy to go and update my password whenever they have a breach.

How about we have the browser manage your login(s) and then have APIs for the sites/apps to authenticate and get access to the profile info you want to share? Even 2FA could be managed by the browser login system so every site wouldn’t have to build (or buy) their own system. Each app could specify the maximum staleness of the authentication token to require you to log in again for critical operations.

Wouldn’t it be nice to log into pretty much everything at once?

Backend#

I’m not going to go into as much detail on the backend, first because this post is already way longer (and has taken way more time) than I expected, but also because backends are necessarily quite varied in what they need to do and the systems they need to talk to. I wouldn’t expect everyone’s backends to behave as similarly as my envisioned frontend, but there are a number of things that come to mind that would be nice to have…

Shared Resource Typing and Functionality#

The backend would be in the same language as the frontend. Obviously… It’s such a team efficiency boost to write everything in the same language so you don’t have to context switch your brain when hopping between frontend and backend, not to mention all the benefits of shared tooling and utilities. The web has already been marching in this direction with Node, Next.js and React server components of course but it’s worth it to restate how nice it is to be able to share code between the backend and the frontend (with a proper project structure set up to know what code is shared vs exclusive) especially for things like validation which need to happen on both sides anyways.

But this section is really about how great it would be if all projects (not just the ones that dedicate a ton of time to make this happen) had a tight integration between frontend and backend for synchronizing data organized into resources. Resources come in all shapes and sizes, backed by many different data stores, that result in many different capabilities the resource can provide. With that in mind it should be easy to define in your api per resource: if pagination is supported; which fields are sortable; which fields are immutable; which fields are only mutable at creation time or update time; and which fields are deprecated.

As that resource schema definition changes during development the client should be hooked up with automatic type generation to the changes so all the functions that interface with the API have the updated signatures and you can see type errors where changes need to be made to accommodate the API.

Data Synchronization#

The obvious next question though is how do you synchronize data between client and server. Do we use REST or GraphQL or gRPC or something else? Let me dodge that question for a sec and take a step back to think about things from first principles.

The data (resources) we want to store can always be represented as an array of objects, even if the array is just has a single item or the object has a single property. And those objects/resources often relate to each other so it makes sense to me that we should be leaning on a relational database to manage the data. On the backend there can be good reasons why you might choose not to use a relational database but they pretty much come down to data size and I/O performance. However, with the size of data clients should be dealing with I can’t see any reason not to use a relational data model for client application state that’s synchronized with the server (there would still be component level state in script variables).

So in the easiest case of a 100% offline app, state management would just look like updates to table rows. There of course already is IndexedDB available in browsers but apparently this is slow so I would like to see something more akin to sqlite wrapped in the browser with the table schema’s defined by the API resources plus any extra client-side only tables that are needed.

With that established, then the question becomes how do you seed the data on initial page load, and how do you keep the slice of data the client cares about (and is allowed to see) in sync with the backend. For initializing, we would need a new kind of resource request for an efficiently serialized and compressed database file that would be loaded (upserted by default) into the clients database as the page and scripts load. That database would be customized by the server to contain all the data the client needs across linked resources so no waterfall of network requests need to be made when a page loads. It would still need to be requested via URL so the contents could be customized (and cached) as needed per page, and there would also need to be a mechanism for the clients to only request it when their local database is empty or too stale.

Macro Mania" title="Bubbles from Macro Mania" srcset="https://res.cloudinary.com/dillonshook/image/upload/f_auto,q_95,c_scale,w_600,dpr_auto/v1589591854/macro-mania/DSC09886.jpg 600w, https://res.cloudinary.com/dillonshook/image/upload/f_auto,q_95,c_scale,w_1000,dpr_auto/v1589591854/macro-mania/DSC09886.jpg 1000w, https://res.cloudinary.com/dillonshook/image/upload/f_auto,q_95,c_scale,w_1800,dpr_auto/v1589591854/macro-mania/DSC09886.jpg 1200w" data-pswp-srcset="https://res.cloudinary.com/dillonshook/image/upload/f_auto,q_95,c_scale,w_600,dpr_auto/v1589591854/macro-mania/DSC09886.jpg 600w, https://res.cloudinary.com/dillonshook/image/upload/f_auto,q_95,c_scale,w_1000,dpr_auto/v1589591854/macro-mania/DSC09886.jpg 1000w, https://res.cloudinary.com/dillonshook/image/upload/f_auto,q_95,c_scale,w_1800,dpr_auto/v1589591854/macro-mania/DSC09886.jpg 1200w" data-pswp-width="1800" data-pswp-height="1800" sizes="(max-width: 720px) 100vw, 100vw" loading="lazy" class="post-image">

For reading data (get one or get many of a resource) my ideal would be a request similar to how REST works today but with more strictly defined and enforced patterns for filtering, paging, metadata, and fetching related resources. Currently, REST is great for simplicity and ease of implementation but is way too lax in standardization which results in everyone implementing it differently. GraphQL is great for fetching related resources and having better standardization around paging and metadata but from my experience usually ends up as a headache to implement on the backend (securely and performantly) and the frontend ends up with an explosion of types as every component is trying to fetch things a little differently which is such a pain for shared utility methods. I think the pragmatic compromise is a protocol that has well defined patterns for fetching the slice of a resource you care about along with the related resources you want which the server can figure out the most efficient way to query and bundle up in a single request and response. The client could then upsert responses directly into the local database for components to re-render with. The client could also specify in its table schemas if it would like to receive automatic updates either via polling or a persistent web socket for push.

For data mutations originating on the client, it would look the same as in the offline app case where you would update the corresponding table row so the client can optimistically render, but then the changes would get batched by resource and uploaded to the server which would then respond with the full resource data including any server generated fields which the client would update its db with. The resource schema definition shared with the client would provide strong typing around which fields can be modified (or created with new records). There would also need to be strong error handling around failed or partial updates.

Isn’t this like…?#

All my dreaming might have sparked some associations to go off in your head with other technologies you’ve used that do a certain piece similarly. As far as I know and have been able to research there’s nothing that comes close to the full picture I’m laying out here. If there is, please let me know! With that said, here’s a couple of the technologies that come to mind as closest with my impression of what they’re missing.

Web components — have similarities with the custom elements with behavior I talk about in the scripting section, but obviously don’t address any of the other things.

Rails — The doctrine is an amusing read. I agree with most of the points like Optimize for programmer happiness and Convention over Configuration, but I don’t think they should come at the expensive of performance which lands you in #396th place out of 510 in performance benchmarks. Even if you were to take the performance of the #50 spot in comparison, you’d need 14 times more horsepower/servers/spend in comparison. While some of the philosophy is similar, my proposal here is also more broad in scope of course instead of what Rails can do is inherently limited to with the current web so it can’t have a shared syntax across server/styles/scripts/markup.

Blazor — I hadn’t really looked that closely at Blazor before writing/researching this post but there’s definitely some similarities. They are obviously still bound by the restrictions of the current stack (like having all string attributes) and also ship a ton of code to the browser which results in horrible frontend performance. Some of that could be improved by having the browser take on more responsibility but someone who knows more about it than I can weigh in on why they’re so heavy and slow. Even with those issues fixed, the most important part of this whole idea is that it needs to be a new open source standard not controlled by any one company trying to lock you in only to dump you when their objectives change.

Electron & alternatives — There’s a lot of different Electron alternatives out there that share some ideas with what I’ve laid out. But of course they’re all aimed at desktop apps, not a wholistic solution that encompasses everything from a static web document to an interactive web apps to a packaged offline desktop app. They’re not build for splitting the app into pages or chunks and require you to install the whole app. Plus, most of the ones that aren’t just wrappers on top of existing web tech look like they’d completely fall flat building complex UI’s and apps with themes. Here’s another comparison between them I found.

Will this ever get built?#

Probably not. But maybe! I certainly don’t have the time, do you?

There’s a thousand other things I’d change along the way if I were the one building this, even some of the initial ideas in this post I’m sure will fall apart when it comes to feeling the developer experience actually building an app in the new platform. The best ideas come from iteration experimentation and refinement.

I’m sure you don’t agree with all my ideas but I hope this gets you thinking about what your pie in the sky web dev stack would look like every time you have to re-invent the wheel.

Lemme know your ideas in the comments!

Passion Over Profits

Thu, 14 Aug 2025 00:48:28 GMT

I’m a little over 4 months into the new job and I think now it’s safe to talk about a topic that’s been fermenting in my head without risk of jinxing anything at this point.

This last time around finding a new job was the first time in the 15 years of my career where multiple offers have actually lined up in spacetime such that I had to decide between them. I’m sure I could have made this happen before if that was the goal but I’ve always tried to be selective with the companies I apply to. As everyone has already told you I’m sure, this is a great position to be in when negotiating your offer.

After a few back and forths with each company I felt pretty good about what each was offering, but on paper one was pretty stacked

	Offer 1	Offer 2
Salary	Baseline	Baseline plus even more than a minimum wage salary
Equity	Some shares, but no near term timeline for any value	Strong chance the company will go public within the year and result in a 100k payout
Bonus	None	5-10% Annual bonus
Title	Senior Engineer	Staff Engineer
Commute	Hybrid	Fully Remote
Team	Solid group of really smart engineers	Working on “the best dev team the company has”

And to top it all off, Offer 2 had company offsite coming up I’d be invited to in the Bahamas!

Just looking at them side by side, Offer 2 won on pretty much every aspect you can think of.

But I still ended up turning it down.

There was one thing I was missing from Offer 2 and had in abundance for Offer 1: Passion. As you might know from my other post, Offer 1 was Commonwealth Fusion Systems where I get to work with insanely talented and smart people pushing the frontier of science to try and accomplish something humanity has never done before. I’m so much more passionate about working on futuristic tech solutions to climate change than the normal sort of business that Offer 2 would have me work on. After a certain point, mo’ money don’t mean mo’ happiness. So to think of it another way, by turning down the extra money I effectively bought happiness with money that wouldn’t have made me happier.

A long time ago I read some career advice (probably from some random reddit post) that boiled down to the fact that job satisfaction comes down to:

The Pay
The Work
The People

If you only like 1 (or even worse, 0!) of the 3 you should be spending all your free time looking for a new job or advancing your skills so you can get a better job. If you have 2 you should be casually looking for new opportunities, and if you have all 3, hold on tight! It’s obvious enough, but if the pay sucks the rest of your life suffers. If you can’t stand the people you work with you’re gonna end up miserable, and if you find the work tedious and boring you’re going to spend a huge chunk of your life checked out and bored. Each one of these is a spectrum of course, and being passionate about your job is just the strongest version of liking the work you do.

I feel incredibly grateful that I not only had the chance to pick between two great outcomes, but also that I was able to land a dream job I’m passionate for. I hope some day you’ll get the chance to pick too, and choose your passion!

P.S. More comments posted over on Hacker News

Launching is the Second Hardest Part

Mon, 19 May 2025 11:44:47 GMT

If Starting is the Hardest Part then surely launching is the second hardest part. And I don’t mean it’s not as hard as starting, it’s just the second time you’ll be facing the hardest part of getting a product off the ground. Just like the first 80% is the hardest, the second 80% of the work is what drags, and the third 80% is really what gets you…

But it’s a big moment when you get there!

Cat Search beta is finally alive and working!

But wow… it took longer that I thought to get there even for such a small project so let me talk you through some of the challenges.

The Known#

Starting with all the stuff I knew about above my MVP release line, there were lots of small details to wrap up like setting up backend monitoring (I went with Honeycomb) and reporting any errors from the frontend through a new API so I can see if there are any problems affecting the users. There were also last minute bugs to fix like the one where I discovered the Google tablet markup is significantly different than the desktop layout and broke all my selectors.

Then there was getting the email verification set up and working so my API wouldn’t be wide open to abuse. I went with resend and got stuck on verifying the domain for 2 days waiting for DNS to propagate before realizing there was a copy pasta mistake…

And of course I needed to make a website which is like starting a whole new project in itself with all the decisions to make on tech stacks all over again. I stuck with the familiar though used a new site template which takes some time to learn. But to make a product website you need to first do things like choose fonts, colors, and a logo, which all require you to think about the nebulous concept that is brand identity.

Coming up with a logo was challenging in itself to find something representative. I spent hours looking for inspiration and then trying some AI tools (I promise I’ll pay an artist to make a better version if this thing ever makes money) before coming up with something half way decent. Then I spent many more hours more converting it to SVG, cleaning up the paths, optimizing it so it’s not a 2MB SVG, and then creating various downsized versions. Even as I’m writing this now and looking at the site I’m getting sidetracked looking at the favicon version and going to tweak it.

It goes without saying that in order for anyone else to see what you’ve made you actually have to deploy it somewhere. And in order to not cause more work for yourself when you change things you need to get build and deploy automation set up. My first approach was to build it alongside the API and have ASP.NET serve it up as static files just so I’d only have one build artifact and thing to deploy. I got that all set up and working but there were some issues with the DNS setup where Route66 requires you to use a CNAME record which doesn’t allow you to omit the “www” subdomain and also messes with the email DNS setup. After messing with that a bit I just pivoted to using the same setup as this site and host the API separately so I can use an A record.

The Unknown#

After doing all that different stuff I was feeling pretty good and ready to tackle the unknown part of publishing to the Chrome and Firefox extension “web stores” which I’d never done before. I had done a similar “submit lots of different promotional images in very specific resolutions” dance when publishing Macrocosm to the app store but it’s just one more step that eats time.

After you submit the extension you have to wait a day or two for some review process to approve it and let the extension be published and visible to download. Seeing what you’ve made alongside all the other extensions is a good feeling, but that feeling was taken away when I installed it and everything just… failed

Almost as disparaging as the fact that local development is working fine and prod is completely busted is the fact that I’m also getting different errors between Chrome and Firefox.

The first problem to tackle was that Chrome was throwing errors that the methods off the browser object exposed to extensions were undefined. It was news to me that Chrome doesn’t support that because the framework I’m using to develop the extension seemingly automatically polyfills it during local development but requires a command line argument to build with the polyfill included. Sigh…

The next (and biggest) problem was all my API calls were failing with CSP errors. To make matters that much more challenging to debug, both Google and DuckDuckGo are throwing their own CSP errors like

Content-Security-Policy: The page’s settings blocked an event handler (script-src-attr) from being executed because it violates the following directive: “script-src 'nonce--zpqhE9NZPHdP41-FRX-Xw' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:” Source: _rtf(this)

on Google, and this one on DDG:

Content-Security-Policy: The page’s settings blocked the loading of a resource (default-src) at https://duckduckgo.com/dist/wpm.1619.ea48354a2359f567744b.js because it violates the following directive: “default-src 'none'”

After confirming the issue was different in prod vs. local because I was hitting localhost for my API while developing it which is apparently exempt from CSP. As I started researching ways to get around this the sense of dread crept in that this would be a total show stopper and all my work would be for naught. Finding out what is and isn’t allowed with CSP for extensions is pretty challenging, and the MDN documentation is even contradictory on the matter which adds to the fun. After a few dead ends messing with the permissions and host_permissions in the extension manifest I found that the actual way around it is to set up a background script, then use browser.runtime.sendMessage and browser.runtime.onMessage.addListener to make the API calls in the background script and pass the data back and forth.

That didn’t sound so bad at first other than a bunch more boilerplate to strongly type the parameters and return types but it turns out this is a super annoying function to work with because the responses automatically get wrapped in a promise so you can’t use async/await inside of it. I need to await getting the user credentials out of extension storage before making the API call and then also pass errors back to the extension which the signup flow relies on. I feel dumb saying this but it took me 6 or 7 tries publishing new versions to actually get this working right. I kept thinking it was all working only to discover something was broken in the other browser or another edge case of error handling didn’t work. This was extremely demotivating to feel suck going back and forth so many times with this one function that I feel like shouldn’t even need to exist. How exactly does it add security to make your API calls in a background script instead of directly from the content script? Why wouldn’t an allow list of domains the extension can talk to be sufficient?

The last major problem was all the custom fonts I included fail to load in the prod build, once again due to CSP restrictions. There’s limited info on this problem as well but the only reasonable solution seems to be to inline the fonts as base64 data. Another rabbit hole to step into, yay. I try a whole bunch of different things but none work and the best I can do is open an issue and give up on custom fonts for now.

Another kicker to debugging all these problems is that browser inspectors are of course different between Chrome and Firefox, you can’t install arbitrary extensions in Firefox and instead need the separate Firefox Developer Edition installed, and in Chrome there doesn’t seem to be any way to get the full inspector open for a background script so you’re left trying to interpret crappy browser error messages like Error: Could not establish connection. Receiving end does not exist. I also ran into issues where the exact same build behaved differently between Firefox Developer Edition installed temporarily and the published version installed in my normal Firefox. But going into the debugging menu and opening the inspector somehow fixed it!?! There was also a time where after using a newly installed version for 10 minutes or so in Firefox it stopped working and I discover it somehow spontaneously uninstalled the extension!?!?!? ARRRRGH

It’s a bad vibe to feel like you’re building on shifting sand and I don’t yet know how I can add integration or end to end tests that would give me confidence in catching all the differences between browsers and build artifacts.

The Takeaway#

If it was easy everyone would be doing it I suppose. So persistence is key, even when life also “gets in the way”, or, to put it better, just takes precedence over a side project.

Plan for the known, but also plan for the fact that there will be unknowns that could take more time than the known’s when launching on any new platform. It’s also important to de-risk your project ASAP for any showstopper technical issues. It’s easy to think you’ve done that with your local development setup but if your production environment looks pretty different then you really haven’t, so test your prod environment sooner rather than later. I learned this lesson on Macrocosm with challenges in the prod build on the app store, but it was a mistake to think it wouldn’t be as big of an issue this time.

And of course, if you’ve been falling out of love with your search results I’d love it if you’d give Cat Search a whirl and let me know what you think! It’s definitely in beta state and there’s a long list of stuff I want to improve or add already of course but hearing real feedback is more important than jumping right into that. The feedback button is in the top right when using it.

5 Tips for Starting a New Job

Fri, 04 Apr 2025 02:50:22 GMT

I just started a new job this week I’m super pumped about at Commonwealth Fusion Systems! The first day was a whirlwind of learning new things and asking all the questions just like it has been every other time I’ve started a new job. But this time was a little different, and not just because this is the first time in 5 years I’m going back into an office. This time the questions involved things you totally deal with every day like superconducting magnets, cryogenic temperatures, supercritical helium, and 50,000 amp currents. Luckily I don’t have to fully understand all those things to do my job but it’s always good to understand where you fit in at a new job and how the whole process works at a high level.

Today I wanted to talk about the top 5 things I always try to do when starting a new job outside the routine “watch the training videos, set up your machine, get the code running, open your first ticket” instructions that come with every new software job. Writing these down is a good way for me to not only organize my own thoughts but also think critically about them and set some goals for myself. Hopefully when it comes time for your next role these will be helpful for you too!

1. Build Relationships#

As I mentioned before, building your relationships and network is the best thing you can do in your career in my opinion. And that’s not at all just for the benefits after you leave a company. Knowing the right people to talk to and having a good relationship so they want to help you out as much as possible is crucial for being successful at your job. These are the people who you’ll be turning to for help, helping in return, asking for their time, and more importantly, advocating for you when it comes time for a promotion. Meet as many people as you can, exploring your way through the branches on the org tree and really get to know the people, not just the employee.

2. Build your Context#

One of the differentiating factors (for now) between human software engineers and AI is the ability to effectively build all the context you need to work on a codebase optimally. AI is obviously getting better and better at slurping up and sending large chunks of code as context if you let it, but what you need to build goes beyond that. You do need to start with the current state of the codebase, what the patterns, norms, libraries used, and immediate problems are. But, more importantly, you need to learn how the software solves the human problems it exists for in the first place. What are the current limitations? What features are being used now? What’s being asked for? Plug into the product leader mindset.

This one takes the longest out of all the points, and arguably just keeps on going, but the goal should be to start triaging which context is most important to learn first and become effective as soon as possible. There’s no shortcut to this one, just read lots of code, docs, PRs, and ask lots of questions.

3. Keep a List#

As you’re building your context in #2 by reading through the code and diving into using the product really, you’ll inevitably have questions about how something works, or why X, Y, & Z are done that way, or even ideas for new features. Get in the habit of writing them all down on a list somewhere. As you get answers either on your own or by asking a coworker check them off (and write down the answer if you think you’ll forget in the beginning info overload). This is not only to keep organized and remind you to ask the questions, but also there will inevitably be some questions that don’t get answered right away that are still good to keep track of. Come back to your list after a month or so and first (hopefully) be impressed by how far you’ve come with your understanding and then see if there are any open questions you need answers to or features that you still thing make sense.

You only get the fresh eyes of a new user of the product once. After you’ve been working on the same thing for a week (let alone, months or years) you become blind to all sorts of confusing behavior and bad UI as your muscle memory builds. By being the new (very attentive!) user yourself and writing down all the stumbling blocks, bugs, and frustrations you can really help out the product team by sharing your findings and hopefully be a part of fixing the issues that real users will also be facing.

My advice is to also keep any really large feature or refactoring work you notice to yourself and on the list for the first month. You really just don’t have enough context yet to know if those things make sense to do yet (or ever) and if you come out of the gate swinging with all these drastic changes you’re just going to give a bad impression and look like you don’t know what you’re talking about, because you probably don’t yet. But if after waiting a month you still think they make sense, bring them up with your manager and see what they think.

4. Improve Onboarding & Documentation#

This should be a bit of a no-brainer but the easiest first task and PR is to update the README and documentation for setting up the repo and local development environment. That section is the most likely to get out of date and break since it’s run so infrequently and usually dependent on other machine setup steps happening in the right order. If the whole thing works flawlessly the first try with no one coaching you through it be sure to personally thank the last person who touched that section of the README. I can count on one finger the times that’s happened for me :)

Part of #2 should also be looking for up to date architecture diagrams. These are also a rare commodity since they so often don’t get updated as the architecture changes. An early value add for the company you can do is to take it upon yourself to update the diagrams while pairing with someone who knows the architecture well. This helps you learn and helps the next person that needs to reference them.

5. Document the TLA’s#

You obviously should know that a TLA is a three letter acronym 😉. Every company has tons of industry and domain specific jargon that is also gonna take some time to wrap your head around. If you’re lucky someone will have already set up a glossary page that defines the commonly used ones. But just like architecture diagrams they’re often incomplete, or the definitions need further explanation. Once again, take it upon yourself to help the whole company out and improve (or create) the glossary for the next person as you discover the meanings to the FLAS.

Finally#

This will be the 8th company I’ve worked for in my career which is a little wild to think about. Maybe you’ve also been around the block a few times these tips are old news to you. If that’s the case, help us all out and leave a comment with your favorite tips!

A good first impression goes a long ways so do your absolute best in the beginning and it will pay off. Hope your next new job experience is great too!

Surprisingly Cheap and Powerful Server Hosting for Indie Devs in 2025

Fri, 21 Mar 2025 19:28:35 GMT

I started looking for a place to host Cat Search and of course instead of doing things the easy way and just deploying it to one place I had to do things the hard way and test all my options to see what the best option for me was.

My goals for hosting are:

Avoid vendor lock in
Get the best bang for the buck
Stay under or around $15 per month
Meet performance requirements (looking for at least sub 100ms response times under load)
Have relatively easy ways to manage and monitor the servers
Automatic deploys on push
Database backups
Ability to scale up and out in the future
A good user experience

There are of course over 9,000 ways to host a site and API these days but I’m looking to keep it as simple as possible to start with and try to avoid as much custom work as possible to be able to focus on the product, but all while keeping costs low to try to make this a profitable venture as soon as possible.

The architecture of the backend I’m deploying is quite simple, here it is:

Can’t get much simpler than that!

Contenders#

I have a bit of experience with the big three (AWS, GCP, and Azure) hosting providers so I wanted give them a shot but I’m certainly am not an expert in any of them to bias my decision one way or the other. During my initial search for options I also found this article on VM Performance / Price that proved helpful in finding new options. Since it was focused on synthetic benchmarks and not hosting an API and database I still wanted to continue with my test to see what kind of actual performance I would get.

Here’s the full list of hosts I came up with to try

The Upstarts

The 400lb Gorillas

The Traditional VPS Hosts

There were a number of other hosting platforms I looked into that got disqualified for one reason or another. Many made you pay for the servers all up front, and only had yearly deals which wasn’t going to work for this test. Others just looked straight up predatory and not a business I’d want to know my credit card number.

During my research I also found Cloud 66. This is not a hosting platform but actually a product that manages deploying your apps and databases to cloud providers or your own VPS. It also can manage database backups, autoscaling, and monitoring which sounds great as a uniform way to manage servers and avoid vendor lock in. I’d also be remiss if I didn’t mention Caprover which I contributed to last year and use to host this site. It’s similar to Cloud 66 in that it manages docker app deploys, gives you monitoring and a UI, but doesn’t have database backup capability and is much more DIY than Cloud 66. So for this usecase I chose not to use it.

Benchmark Setup#

To consistently test each host I set up an Artillery load test that hits the site categorization API. Each request picks 15 random URL’s from a list of 350 to have the API categorize. The API is mostly read heavy for this operation with a couple queries per request, but also does a few writes to separate tables.

Before each test I seeded the new database with around 21 thousand rows of categorized URLs to have the queries return meaningful data.

For the load test itself I setup Artillery to run in phases of 10 seconds each, starting at 5 requests per second (RPS), then going to 10, 50, 100, and 500.

Results#

For each host I documented my setup and deployment experience as well as any bugs or documentation problems that I hit along the way. These of course shouldn’t be deal breakers but are good to know for your own user experience, especially if you’re using the host for the first time.

Render#

As my first deploy target there were a few bumps that required code updates to fix a database pool connection issues and accept a connection string environment variable, but once I sorted that out the code I deployed was exactly the same to all the hosts.

The first instance I created was in Ohio US East with the free tier which gives you such an overwhelmingly generous 0.1 CPU and 256MB RAM with 1GB Storage. The basics instance went up pretty quickly but it took quite a few minutes before the deploy started cloning the repo. Probably shouldn’t expect much from the free tier though.

It took me a few minutes to figure out how to connect to the Postgres DB from my local machine to run the seeding script (turned out to be the hostname not including the remote url which I had to dig for) but the deploy and mostly worked first time. A miracle in devops history indeed. The only problem was a bug in my code for another place not using the env var for the connection string.

On the free tier though your app shuts down all the time after a minute of inactivity. After this happened I tried using the restart button but that didn’t do anything, so I had to run a full deploy to get it responding again.

Pretty poor results if I do say so myself. But let’s upgrade to the paid tier to give them a fair shake.

Even getting a 0.5 CPU 1gb RAM instance is $19/mo which is above the budget threshold so I went with their most basic paid plan to retest. The DB is $6/mo for 256MB RAM and 0.1 CPU (lol is this 2005?) and the “App Starter” is $7/mo for 512MB RAM and 0.5 CPU.

Better, but really not that good.

Digital Ocean#

For DO I exceeded the budget by a bit to be able to use their fully managed PostgreSQL offering that starts at $15 a month and then their app platform which starts at $5 per month for 1 shared vCPU, 512MB RAM, and 50GB bandwidth hosted in their New York data center.

First impressions were good with an easy setup for both the DB and app but it took a little while to sort out a few configuration issues with the health check using the wrong port, the database connection pool settings, and environment variables for the DB connection. After the first couple times editing the app config to change an environment variable it’s a bit annoying the whole app has to rebuild and redeploy which takes a couple minutes when it could just be a container restart taking seconds.

As I was running the test it’s nice to see they have built in load charts, query stats, and DB logs. During peak load the app still didn’t even come close to maxing out CPU usage or memory in app or DB which makes me think connections are the bottleneck. That will have to be a separate investigation though since I’m trying to keep everything as similar as possible between tests.

Linode#

I was going to run the test it on my existing tiny server that runs this site but then thought that wouldn’t be a fair comparison so I created a new server for the test.

To stay on budget the instance had to be 1 shared CPU, 2gb RAM for $12/mo US East, Newark NJ

AWS#

AWS is a behemoth of complexity but since I was just using one VPS with Cloud 66 handling the hard parts of deploys it only took less than an hour to figure out IAM permissions and credentials for Cloud 66 to use and which instance type is at the right price point.

I landed on trying a t3.small on us-east-1 (N. Virginia) which has 2 vCPUs, 2GB RAM and should be ~$15/mo.

I wanted to compare the single VPS solution to “the amazon way” using RDS (and try out Aurora) and Fargate with Elastic Container Service (ECS) for the app but playing with the AWS pricing calculator quickly proved that to be a non-starter.

The smallest RDS instance for just PostgreSQL is the db.m1.small machine instance and that’ll run you $43.80/mo by itself for a single AZ. Then tack on $2.30/mo for the minimum 20gb storage and $1.90/mo to backup the 20gb. With PostgreSQL compatible Aurora the smallest instance is db.r4.large which is $211.70/mo plus DB storage prices that vary wildly on baseline IO rates which I have no idea what to expect. The good news is that it’s only $0.42/mo for the same 20gb backup!

With Aurora Serverless the minimum you can do is 0.5 “Aurora Capacity Units (ACUs)” per hour which are described as “Each ACU is a combination of approximately 2 gibibytes (GiB) of memory, corresponding CPU, and networking”. For what sounds like a tiny amount of power will still run you $43.80/mo with the same DB storage and backup prices as above.

If you actually want to run an app as well that’ll be extra! Calculating the cost for Fargate with 50% utilization, 2vCPU 8GB RAM for just one container came out to $42.53/mo.

I guess the “the Amazon way” is not as cheap as I thought after all…

GCP#

A first glance at a dedicated Postgres DB in the calculator shows it’s well outside my price range so I’ll once again sift through the plethora of VM options.

The best equivalent instance I found is an e2-small with 2 vCPU and 2 GiB RAM for $14/mo. Choosing us-east4 in N. Virginia to be a comparable location for the latency.

The signup experience had a new wrinkle to it that no other provider made me do. You have to verify your credit card by waiting for pending charges to show up on the statement and then enter the amounts. After 15 minutes of refreshing my statement I gave up and went and did something else useful. Only after coming back 45 minutes later the charge showed up and I could proceed with waiting another 25 minutes for the app to deploy (the longest of any provider by far).

From Compute Engine API page clicking manage just sends me back to same page

Welp, that was disappointing

Azure#

Once again from poking around the Pricing Calculator, dedicated DB instances are way too expensive and cryptically named Dadsv5 - Series which means absolutely nothing unless you have the decoder ring. No surprise the list of virtual machines is nuts but narrowing down by price once again I land on the B1ms instance with 1 core, 2GB RAM for $15/mo deployed to US East in Virginia.

I have used Azure a bit before so I know a bit of what to expect but just trying to follow the instructions to set up credentials for Cloud 66 to use was like trying to escape a corn maze. Why are there two different admin dashboards? https://entra.microsoft.com vs https://portal.azure.com Why don’t the tabs look like tabs? When I tried to add a role to a subscription for an app following the instructions I couldn’t find the “Contributor” role because it was hidden behind another tab “Privileged administrator roles” instead of “Job function roles” that just looked like a sub-heading.

The Cloud 66 docs are light on this process and the first couple deploys failed with these errors on which “namespaces” the subscriptions need.

An exception occurred in your cloud during server creation: Can't create a default Virtual Network in Azure cloud: The Azure API responded with HTTP error 409 and error code(s) MissingSubscriptionRegistration: The subscription is not registered to use namespace 'Microsoft.Network'. See https://aka.ms/rps-not-found for how to register subscriptions.. If the problem persists please contact support.. If the problem persists please contact support. master

It’s also annoying that Cloud 66 has to create a new app (even with copied vars) to retry initial deployment after it fails. It claims it will clone all the variables of the app deployment but actually it forgets the deployment target, machine type, and DB each time. Finally got it working though for the first successful deploy taking 13 minutes.

Hetzner#

After all that messing around with the big cloud providers the signup and setup process for Hetzner was a breath of fresh air. All Cloud 66 needed was creating an API key that took a few clicks.

There was a couple different options at my price point in their Ashburn Virginia data center so I wanted to try out both the dedicated and shared CPU instances.

First deploy was with the dedicated 2 vCPU, 8GB RAM, 80GB storage for $15/mo which first deployed in 8 minutes.

Then I tried their shared 3 vCPU, 4GB RAM, 80GB storage instance for $11/mo.

DNF#

All of the following providers I tried but wasn’t able to get them working for one reason or another. Some of them I’m sure I could get working if I spent more time on it but I disqualified them for myself for being a time suck.

Supabase#

First impression creating a new app: “Network error trying to create new resource”. Second impression: “Project with name X already exists in your organization”.

After actually creating the DB it took a bit of searching around how to find the connection string to use to directly connect to the db for seeding.

AAannnd then I realized you can’t actually deploy a full containerized app, only edge functions which defeats the point of hosting your api close to your db for me.

Heroku#

I did use Heroku like 10 years ago and surprisingly my password manager has the login saved. But of course trying to use it prompted the error “There was a problem with your login”. Went through the forgot password flow but never got an email, so time to sign up again I guess. There’s an annoying signup questionnaire and mandatory 2FA right out of the gate which slows down the whole process.

Looking through all “essential” DB plans they all seem to be same except for storage limit, and are listed for $5/mo with “0 Bytes RAM, 1GB storage”. Pretty sure there’s a mistake in there but I give it a shot anyways and run straight into a 404 page with a error popup.

I guess they don’t offer that one anymore and I’ll have to use the $50/month plan with 4GB RAM, 64GB storage, and 120 connections instead? Nope that one doesn’t work either… I had to look up a youtube tutorial to find out you have to create an app first and then attach postgres as an add on… Ok I guess I’ll do that even though it seems backwards to me. I create the add on and then go click the “Heroku Postgres” link to find out the credentials for seeding and it does ** exactly what I expect ** … and opens the favicon in a new tab https://addons-sso.heroku.com/favicon.ico

Going back to the page and trying agin it works, I guess their JS was just slow or broken? On the next page I’m amused to see they actually just use Amazon RDS under the hood.

The next step is to try to figure out how to deploy the app. Heroku apparently can’t figure out there’s a docker file, from docs I found looks like you have to set up a heroku yml file. Tried a version of that which didn’t work, then see you should use this build pack thing.

The article doesn’t really say how to use them in your project and there’s two different flavors of build packs but the descriptions are gobbltygook if you don’t already know what they are.

That all seems like quite the hassle for this test so I switch to just use the container registry method. But come to find out there’s no way to do that in the UI, you have to install the heroku CLI, sigh…

I go through their instructions for pushing a container but of course it fails at

heroku container:push cat-search-api
 ›   Error: The following error occurred:
 ›     Missing required flag app
 ›   See more help with --help

Added the --app parameter that’s missing in all their examples and then got another error:

heroku container:push --app=cat-search web
 ›   Error: This command is for Docker apps only. Switch stacks by running heroku stack:set container. Or, to deploy ⬢ cat-search with heroku-24, run git push heroku main instead.

Did that and now it rebuilds the docker image even though nothings changed, and I think it deploys, but there wasn’t anywhere to specify the environment variables during this process. So you have to switch back to the UI and add them in “Config Vars”.

Finally open up the deployed app url and of course get an app error. Turns out the app inside the container can’t bind to port 8080. Mess around with thing some more and try both port 80 and 443. No luck…

And this is where my Heroku story ends: 2 hours after it started with nothing to show for it except the experience to know I’ll never use Heroku again.

Neon#

Should have found this in the initial search but found out it doesn’t support deploying an app alongside the DB.

Vercel#

Also should have found earlier that Vercel doesn’t support docker deploys. This one did surprise me a bit since I know they let you deploy edge functions to run Next.js apps and I thought they were angling to be a more comprehensive hosting platform.

Oracle#

To be honest, before reading the cloud provider performance article I didn’t even know what Oracle really did anymore other than sell database lock in. But apparently they have good performance to price ratio so lets give them a shot and see.

After signing up I go to create a VM instance that’s in the price ballpark and go for the Compute - Standard - E5 - Flex AMD instance with 2 OCPU and 8GB RAM running Ubuntu 24 but I’m not really sure on the price when selecting it since it doesn’t list it. After it’s up I go to the Billing Screen to see if it lists an estimated cost, but that page has an error.

I had to go to the public Cost Estimator to see it’s actually $56/mo. Outside my budget but lets try it anyways, so I get it registered in Cloud 66 as a custom server.

The first deployment fails after 19 minutes with the error “system services failed to start”. Searching around a bit for things to try I find to run this command to get Kubernetes logs.

journalctl -u kubelet

Mar 14 02:03:33 master kubelet[23739]: I0314 02:03:33.537146   23739 scope.go:117] "RemoveContainer" containerID="93a9c646c1dab98ef4c0707f35b4d95d96dd10612081a6fe22fc43523b9c1df3"
Mar 14 02:03:33 master kubelet[23739]: E0314 02:03:33.537283   23739 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"k8s-event-relay\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=k8s-event-relay pod=k8s-event-relay-6885c57f84-ph7rc_kube-system(ef57>

Not sure where to go from here I try a different instance with 1 OCPU, 8GB RAM. Same problem. One more time, this time with Ubuntu 22, on the 1 OCPU, 8GB RAM. Failed again and since these were all fresh machines with the Ubuntu versions Cloud 66 supports I don’t particularly feel like spending more time messing with it and give up. I’m hesitant to blame Cloud 66 though since the custom server setup worked flawlessly for Linode.

Combined Results#

For all the hosts I was able to deploy to here’s the combined view with their p90 times across the different load rates. In the cases where the load numbers didn’t fall exactly along the delineations I linearly interpolated to get a more accurate result.

I also wanted to see performance compared to the monthly cost in some way so the best metric I could come up with is taking the ratio RPS / Latency / Monthly Cost from the p90 values that are shown above. This ratio will give greater values to the hosts that can deliver more requests at lower latencies for a lower cost.

So using AWS as an example: 50 RPS / 569ms latency / $15 a month = 0.0058 which you can see rounded to 0.006 on its blue bar below.

Conclusion#

Suffice to say I’ll be using Hetzner for both the ease of setup and best all around performance per dollar. Surprising to me is Azure coming in second for hosts on the performance per dollar chart however the very complex management console and altered vocabulary for everything is not for those looking to avoid complexity. Also surprising to me is how poorly all the new generation of hosting platforms faired. All but one was a nonstarter and the only one I did deploy on had awful performance for the price.

Hopefully this info will help you in your search for hosting!

Sowing the Seed Data

Thu, 06 Mar 2025 19:51:22 GMT

Coming at you here with another Cat Search update! Since last time the new design has been implemented (see below), authentication is mostly done (still need to send emails), and many bugs have been squashed 🪳.

However, I got a little side tracked the last couple days by a “simple” item on my todo list I didn’t even mention last time: Update Seed Data.

What is Seeding Data#

I’m not entirely sure where the term came from since your data doesn’t typically sprout the same way the seeds in your garden do but for those unfamiliar with the term seeding data it’s just pre-filling your database. Often times it’s generated fake data just used to test with.

In my case though it’s not just test data! The seed data will be used as the initial categorized sites when Cat Search launches.

And Why Should You Bother?#

One of the worst first user impressions I can imagine (which is not difficult having experienced this many times myself) is to land on a page full of empty states and placeholders. It’s a ghost town of the forgotten web where they built it and no one came, clearly destined to decay.

On the flip side, if the app is bustling with activity showing the product value it’s now a happening place you don’t want to miss out on.

Any user first impression is a critical moment in forming an opinion of what your product does, how is it’s valuable, and why should anyone care.

If you’re not able to get real data in your app when it launches the next best thing is to have a sandbox demo environment for customers to try first, which of course requires a seeding script. Depending on your app, this make take the form of an ephemeral demo environment or just one that gets reset every so often.

While the above is the primary reason to seed in my mind, there’s another benefit.

Let’s face it, production data is generally messier than a kids bedroom after a sugar rush. There’s all kinds of unexpected edge cases the developers never thought of like really long names, tables that have way more data than expected, and the general bucket of “non-conforming” data from partially deleted records or the result of bugs that were never fixed. Regardless though, it’s a good habit to update your seed script to include this weird but possible data so that the devs see it every day working and consider how to handle all the edge cases. This also includes having a realistic amount of data compared to production so the devs get annoyed with any scaling performance problems and just go and fix them. The bugs won’t be out of sight out of mind if you’re testing with realistic data.

Sources#

With Cat Search my challenge was finding good sources to pull website categories from. There are a number of companies out there if you search “website category api” that provide APIs for categorizing domains but they’re all exorbitantly expensive and it’s probably against their terms of service to use the data to put them out of business ;-)

So publicly available lists it had to be.

I started with the ahrefs top sites list which includes categories but had to do a bit of cleaning on it to make it usable and the categories make sense. Like whatsapp.com is not “Internet and Telecom” really.

Another great source I’m exploring more is good ol’ Wikipedia. There are list pages like List of Internet forums and List of Video Game Websites that list out a bunch of the popular sites. It does take a bit of scripting to get the actual domains of all the sites though since the links on these pages are to other Wiki pages.

There’s a few open source projects that helped too. Kagi Small Web is the basis for my Indie Blogs category and DevDocs is used for the Official Documentation category.

The last major source has been uBlacklist subscription lists. Many of these are sus and go overboard but with a little bit of grooming they provided good lists of spam and AI sites.

If you know of any other good sources please let me know in the comments!

Results#

With all that seeding work the results speak for themselves. Across 2 pages of google results many popular queries (😉) only have a couple Uncategorized results.

I’m sure there are a number of search queries that I never make which will be much more uncategorized but I’ll have to launch the thing to figure out what those are!

So What Exactly is MVP Anyways?

Thu, 20 Feb 2025 20:56:17 GMT

It’s been a little while since the last project update! Life happens and I also need to get better at switching to a “Build Share Build Share” mode instead of just “Build Build Build Build” mode which is my default. So here’s what’s happened since last time as well as a thought exercise on what exactly you need for a Minimum Viable Product.

Progress#

From last time:

My goal for the next update is to get the frontend interacting with the backend to read and set both site votes and categories. Should be easy enough but there’s a lot of stuff I glossed over on both the frontend and backend that I’m sure I’ll be refining as I go.

Goal: Smashed.

But that might be because it’s been two and half months since the last update 😅. But a lot has been accomplished in the limited amount of time I’ve been able to dedicate.

First off, I’ve settled on a name for the project: Cat Search! It’s short for “Category Search” which is what this is all about. It lets you categorize all your search engine results into whatever you want to take control of the results you see and how you mentally rank them yourself. I’ve been toying around with that name in my head for awhile now and since nothing better came to me it stuck. It’s a great name too since it’s easy to say, easy to remember, and easy to make a cute mascot for :) Even though I’m really a dog person, as you can probably tell from my profile picture, cats are great too.

Without further ado here’s what the current state of Cat Search looks like.

I got support for both Google and Duck Duck Go working so whenever you search for something the extension parses all the link results, sends them to my API for categorization, and then displays the categorized results above the main search results. All in about ~20ms right now. Granted that’s hitting the API on localhost with less data than a production environment will have but I’m going to be keeping a close eye on performance the whole time since the results are taking over the top of the page and any extra latency will be much worse for user experience since the whole layout will shift. Right now at this speed you hardly even notice the loading delay as the whole page is loading in. I’m sure there will be more work needed around optimizing the layout shift as the page loads in the future though.

Undoubtedly there will be some uncategorized results this early in the project before many people are using the extension. I’ve seeded some categories based on a few lists I’ve found but the internet is a large place and I’m only going to have a tiny slice categorized when Cat Search launches. So that’s where all the users come in to either drag an uncategorized result into a category that’s already on the page, or create a new category. All the categories you create will just be for you to start but the idea is that there will be a process looking across many users that have categorized a site the same way and “promote” the category so that everyone sees it. This will be a manual process to start with while I figure out what the right criteria is for automatic promotion.

The last feature I worked on before this post was to automatically load the next pages of results. This will likely be a paid feature since it’s a nice bonus to the core functionality. I, like you, am lazy and almost never click next page results. But I know there’s many times where a better answer to what I’m looking for is buried beneath the SEO spam on page 2, 3, 4, etc. and if it was automatically categorized into a category I highly value (like Indie Blogs) I would click on those results much more often than I do now.

And finally, I created a design mockup for the UI! It’s a little rudimentary as far as full designs go, but it’s enough to get me going. I’m sure you can tell the actual current state look is, well… very programmer art ish. It’s functional but not pretty, and that’s just because I hadn’t spent much time on visuals since I was still in the process of figuring out what this thing will become and how it will work. So next up is just snapping my fingers and making it look like the mockup!

Challenges#

In this chunk of work there certainly were a number of challenging things that took longer than I expected and some open questions to solve later.

Result Sorting#

The first was how voting would interact with dragging a result to a certain position in the category list when it comes to sorting the list the next time (or on page reload). There’s a number of things to sort results by:

My category rank (if I’ve previously dragged a result in the list)
My category votes (using the up and downvote arrows)
Other people’s votes on the site and domain
The search result order

That’s a lot of info to take into account while also making the user experience simple, make sense, and consistent. Currently I’m using the following logic which has worked well for me while using it but I’m curious to see if I get feedback on this.

Dragging a result into the top 3 of a category counts as an upvote, anything below that counts as a downvote
Find all the results in a category that don’t have a category rank, then sort by:
- Combined vote counts for mine and other people’s votes
- Tie break with the search result page rank
Insert all the results that do have a category rank into the appropriate place in the list
- Tie break with the combined vote counts

This logic ensures if you drag a result to a certain position it stays there, and otherwise highly voted on results bubble to the top, and we fall back to the search engine ranking.

DB & API Design#

For the API design I wanted to make sure all the necessary changes to the DB when a result is moved categories could happen in one transaction both to keep the DB consistent but also remove the complexity from the frontend and make the API calls efficient. There’s actually quite a number of DB updates that need to happen for a simple drag action:

Check to see if the user has already voted for this site so we can calculate the vote changes if it went from an upvote to downvote.
Upsert the user site data with the votes
Upsert the shared category data with the vote changes
If we’re not categorizing the whole domain, also make sure to upsert the shared domain category data with the vote changes
Upsert the user category data with the new category and category rank data
If the user is moving the result (vs copying), delete the old user category record

Figuring out the table shapes and API actions for all the combinations of (new || update), (site || whole domain), and (move || copy) definitely took some time to figure out and required some notebook work to help break it down.

URL’s#

Currently my URL matching is limited to either a domain (including the subdomains) or the URL and path (excluding any query parameters). I’d like to be able to support matching partial URL’s with wild cards for cases where you want to categorize something like a whole subreddit https://reddit.com/r/pics/*. This would also allow me to seed more category data based on existing ublacklist subscriptions. Those lists are all full regular expressions though and that sounds way too slow to allow when querying and opens up a whole can of worms. But for now I’m going to kick that can down the road to post MVP anyways.

What’s The Bare Minimum to Ship?#

Back to the thought experiment in the title I hope you’ve been thinking about as you read through all the updates though. What constitutes an MVP? How minimalistic do you really need to go? What differentiates a “product” from a demo or a toy?

There’s a case to be made for rushing to deploy your app as soon as possible and get folks in the door with only the roughest functionality. That doesn’t sit right with me though for a couple reasons. You’re giving the people that care about your product the most the worst possible user experience which will inevitably taint their first impression of the product no matter how excited they are for it. People can only look past so many bugs before they look for alternatives. If you’ve really shipped as soon as possible that also means you don’t have any way to monitor issues or receive feedback, both of which are absolutely critical for molding software into a delightful user experience that solves a real problem.

This is one of the lessons I learned from creating Macrocosm. You only get one shot at launching. Make it count.

So in my mind the bare minimum things you need before you ship any product are:

The core functionality that solves a user problem
A way to collect user feedback
A way to monitor problems and usage
Some form, even if rudimentary, of protecting against abuse
A way for people to discover and learn about the product

The big things left on my list I want to tackle before I think it’s really minimally viable are:

Implementing the new design
Proper authentication (I want to allow anonymous usage for a little while to give people the best onboarding experience to try it out, but then require a login to prevent abuse)
Recording logging for FE & BE, and telemetry for API response times
Actually deploying the API (I want to find benchmarks for different cloud providers or run a load test myself to see what gives the best latency)
Setting up marketing site with info and links

You may or may not notice that paid features aren’t on that list. This isn’t a charity app and I will have some forms of monetization built in but after debating myself for a bit I decided to have this be the first feature post MVP. Getting the first few people using the extension, starting to categorize the web, and making sure it’s working well are all more important things than the pro features to start with.

Hopefully I’ll have at least one more update for you before I get there!

Collioure, France

Thu, 26 Jun 2025 11:36:12 GMT

Monastery of Sant Pere de Rodes#

Our journey to Collioure actually started by going the opposite direction to see the Monastery of Sant Pere de Rodes in the mountains to the east of Peralada where we stayed the night before. Not sure what was happening here on the way up the mountain but the smoke was gone by the time we came back down the mountain.

Collioure, France#

An hour-ish drive and some extremely tight parking later we were ready to be back on foot and explore.

The tourist train that goes up into the hills outside of town is well worth it for the views even if you can’t understand anything the driver is telling you about what you’re seeing.

On our last non travel day we went up to visit the La Guinelle Vinaigrerie which makes vinegars from premium wine (versus the bad wine most people make vinegar from) and naturally ferments and ages the vinegar in these barrels and bottles.

And then off to Les Clos de Paulilles vineyard for lunch

And then one more squeeze into the tightest driveway I’ve ever gone through which took a ridiculous amount of maneuvering, backing up, spotting, and pumping the clutch. Each time it took 15 minutes or more just to park.

Barcelona

Sun, 22 Jun 2025 11:35:41 GMT

It’s time for our next big adventure, this time to Spain and southern France! Big adventures are harder these days now with a crazy 17 month old toddler in tow that has her own agenda which involves wiggling and causing chaos. Despite that, I still took a ton of pictures that necessitated breaking the trip into two posts!

Barcelona, Spain#

Barcelona is well known for its architecture and Casa Batlló is one of the centerpieces by Antoni Gaudí.

The next day we went for a stroll on La Rambla and through the gothic quarter like all good tourists do when in Barcelona.

As a way to escape the mid day heat we went to the White Rabbit, “The Off-Museum of Barcelona” which ended up to be pretty fun and a good place for Quinn to play and get some energy out.

Next up was taking a tour of the unmistakable Sagrada Família, still not quite done after 143 years of construction.

The night before we didn’t sleep all that well since it turned out to be Saint John’s Eve which is enthusiastically celebrated by setting off fireworks until the wee hours of the morning.

That evening we took the bus up to Park Güell (pronounced “gway”) which has great views of the city and is one of the places Gaudí lived.

Catalonia#

After a couple days it was time to pack back up and get a rental car to see some more of the country. On the way to our accommodations we stopped at the Clos D’Agon vineyard and winery for a picnic which was nice but pretty hot and sweaty. We almost had the whole place to ourselves though!

For the night we splurged on the wonderful Hotel Peralada for our anniversary which has such beautiful grounds, refreshing pool, delicious restaurant, and wine spa which we sadly did not get to try.

The next morning we started with a short drive to visit the Salvador Dalí Museum in Figueres.

And then it was off to lunch!

Continue the adventure in part 2, where we make our way up to Collioure, France!

Midnight Ride

Mon, 21 Apr 2025 05:00:00 GMT

Apparently there’s an annual tradition 15 years strong to bike the Boston Marathon route the midnight before it begins. I heard about this tradition roughly 12 hours before it started from some friends who were going to do it and decided to try it out on a whim despite not having ridden a bike at all in like 2 years, and never having ridden more than a couple miles.

So on this steel horse I ride, but this one is slow and requires lots of pedaling.

I did take a peek at the course map beforehand and was relieved to see it’s mostly downhill except for the hills starting at mile 16. Spoiler alert, those were a lot worse in person than they looked on the map.

Arriving at the start line there were tons of people having fun and preparing their bikes. There were lots of fun lighting setups from the people who actually prepared for this.

I didn’t take as many photos during the ride as I would have liked. It was hard to justify stopping more than needed when you’re getting constantly passed by other bikers and feeling like this is going to take all night. We did stop a number of times though to take a break and got a couple shots.

Around 3am we finally made it to the finish line with sore legs and even sore-r butts but huge smiles on our faces. You’ll have to ask me for that picture though with how many AI crawlers there are now-days.

Newbury street is usually packed to the brim, but at this time it’s just bikers trying to get home.