simpler things

An incident isn't over until you've addressed the root cause.

If you've moved past immediate customer pain, but the pain could flare up again at any moment, you've at best mitigated the problem.

Yes, this does mean you've got to drop whatever you're doing.

No, this is not the fun part.

What if we end up spending all our time fixing root causes?

In many ways, that's better than spending all your time getting paged. You can address the problems with your systems now, with a clear head, or face them in the middle of the night.

Uncomfortable truths lurk behind this question.

Not everything is worth fixing. There are some bugs or incidents that don't cause that much customer pain. Likely because they occur in systems that aren't crucial to your core customers.

But if something's not worth fixing, is it worth paging for? And if it's not worth paging for, is it worth operating at all?

Running a service on the internet costs time and attention. Nothing comes for free. The longer it's been since you last touched the service, the more concentration it will cost you to respond to a problem.

The other uncomfortable truth is that it costs to build reliable systems. Not in the “add four weeks to the timeline for to find bugs” sense.

In the sense that you have to know what your service is supposed to do and continuously check whether it's doing it or not. If humans interact with your system to accomplish goals, you have to have a way to check that they're accomplishing those goals! HTTP 200 is OK, but not enough.

Or in the sense that sometimes parts of your service or the services it depends on will fail and you have to handle those failures gracefully.

Your downtime will be the sum of the downtime of all your single points of failure.

Waiting for root causes to be addressed before declaring incidents resolved brings these uncomfortable tradeoffs out into the sunlight. What, exactly, merits this level of rigor? What will you stand behind? And what will happen when the new system you're currently building fails?

For a long time, I did not understand or want to understand sales. It seemed artificial and boring. So long as whichever company I was working at hit its quarterly numbers, I didn't give sales a second thought.

Then I found myself on the hook for recruiting engineers.

That meant reaching out to people I didn't already know, piquing their interest in joining a startup, interviewing them, and convincing them to sign.

It took a lot of work. Much of it felt wasted at the time.

But every time we hired someone awesome I would feel euphoric.

This is sales, I thought to myself. I get why salespeople do what they do. They're chasing that euphoria.

Most engineers get their energy from building. They don't need an additional dopamine source. But they do need to sell.

What do engineers need to sell? Their time, if they want employment. Their company, if they want great teammates. Their product, if they want to build what people want.

Sales starts out as a search problem. You've got something to sell. You need to find buyers.

If you had time to speak to everyone on the planet, you could brute-force this search. You don't!

There's no clear algorithm here. Talking to people you already know who are familiar with the domain works great. Often they can introduce you to other folks. When this approach pays off it pays off big, because you get a warm introduction to someone interested in buying.

Another path is to search for people on the internet. This can be fun! I especially enjoy finding new sources of signal. “Oh I see you starred this repository on GitHub, you would love to come work with us.”

Most people on this earth do not share anywhere near enough about themselves online in order for you to be able to just scan a web page and know whether they're likely to buy what you're selling. You'll have to reach out and ask.

Reaching out to people you don't already know is probably the most uncomfortable part of sales for engineers. That, and nailing down next steps.

Over many years of reaching out cold, I've found that a short, straightforward email sent today beats a perfect email sent next week.

Just write it and send it! If you don't get a response, it has little to do with you or your writing and a lot more to do with people being busy or flat-out not interested in what you're selling.

(If you're launching something entirely new and people don't know you or your company, you will have to work harder on cold emails. Still, sent beats unsent.)

You'll have to bounce between talking to humans and searching on the internet many times. If a particular group of people or spot on the internet proves fruitful, spend more time there!

Once you have scheduled time with someone, they've officially entered your sales pipeline. A sales pipeline is like a CI/CD pipeline or a data pipeline. There are stages and you're trying to move people from one stage to the next.

Some pipelines are pretty clear-cut. When you're looking for a job, you expect an initial conversation, a recruiter screen, a technical screen, an on-site of some sort, and an offer call. Details may change but there's a well-known order to the process.

When you're selling something unique, like a multi-million dollar enterprise software license or a private island, you have to make up the pipeline as you go along. That's where experienced salespeople can make a difference.

Back to the beginning of the pipeline. In your first conversation with someone, you want to learn as much as you can about them. Why are they talking with you? What do they need? What do they want? When you describe what you're selling, when do they light up? What bores them?

The tone you're trying to strike is “I've got this awesome thing that I would love you to have! If you're interested.”

Not “please please say yes on the first call and save me from having to talk to more people.”

Your whole job in that initial conversation is to learn. You will not close the deal. If someone's not interested, you may be able to figure out what would excite them. At the very least, you can tune your internet searching to look for more people like the person you're talking to or fewer.

At the end of this and every conversation, you want to agree on next steps. The buyer will need to decide if they want to move forward. Ask them when you should expect to hear from them.

Most often, you'll hear back from people before the agreed time.

Sometimes you won't hear back. Usually it's because the person is busy. Or they're not interested but don't want to hurt your feelings by saying no. It's hard to interpret silence!

It is a-okay to send a polite note to someone asking if they want to move forward or if they have any other questions. I like to wait until the morning after the deadline (Monday morning if the deadline's a Friday).

Shy people (engineers) don't like following up. “What if I come across as pushy?” they worry. Sending a single note is not pushy. Sending notes after some has said “I'm not interested” is pushy. Sending multiple notes through multiple channels in rapid succession is pushy.

What people are really avoiding when they don't ask questions is hearing “no.” They prefer silence to a clear answer because they can imagine that silence means “maybe.”

Silence does not mean “maybe.” Silence means “no.” Vague “I'll think about it” means “no.” Anything other than “yes” means “no.”

Expect to hear “no.”

That doesn't mean you should approach people with your tail between your legs! Remember, you're going for “check out this great thing that I would love you to have,” not “you're probably not interested but here's this thing.”

It's easier to face “no” and to strike the right tone if you have other options. You want to talk to many people at once. Not so many that you lose track of who's who or waste time talking to people who will never buy. Enough so that when you hit “no” you can turn your attention to other people instead of sulking.

What about when people say “yes”? Go to the next stage and repeat the process. Keeping going until it's time to close.

Closing simultaneously works like any other stage and takes extra juice. The time scales grow shorter. If the person you're selling to signs with a competitor, the game is over. (99.9% over. Sometimes you can salvage deals or candidates. But the fact that salvaging makes for great stories shows how rare it is.)

As the sales process goes on, you should be building trust and growing comfortable. What started with emails should move to texts and calls.

Expectations also go up. If the person said you'd hear from them by four o'clock on a Friday and you didn't, call them up!

(Unexpected silence at the closing stage is not a good sign. Sometimes people who don't have experience buying will stumble here. But if someone who's been around the block goes cold, brace yourself.)

Closing is the time to shoot your best shot. That doesn't mean making false promises. That doesn't mean bad-mouthing alternatives. It does mean painting a picture of what the person will get by signing and explaining why you want that future for them.

Sometimes it's as simple as calling out of the blue and saying, “I really want to work with you.”

A wise person once told me, “don't celebrate until there's ink.” The funny thing is, though, that selling doesn't stop once someone has signed.

The candidate has to show up to their first day of work. And their second. And many more after that.

The customer has to receive whatever they bought. And use it. And like it.

People (engineers) say “always be selling” or “always be closing” as a joke. Haha, look at those bros out there chasing deals. Glad that's not me.

Really, you can think of sales as manufacturing options. Just as you can think of coding as building possibilities out of thin air. When you have options you carry yourself confidently, which leads to even more options magically appearing.

If you're not selling, you're choosing to live with the options in front of you. Which can be great! But make that choice deliberately.

What's funny is, the times I've felt most pumped up about my job have been when I'm selling. You'd think it'd be the other way around: when you feel good about your work, then you're motivated to sell. But recruiting or demoing has pulled me out of ho-hum moods often enough that I think selling is the cause, not the effect.

This post is for nice people who find themselves leading engineering teams.

Not kind people who treat everyone politely even when disagreeing. Nice people. People who care a lot about what other people think and how other people feel.

Because they put the success of their team or their company ahead of individual glory, nice people often get pulled into management. That's great! To a point.

Once you, a nice person, take charge, there's a temptation to keep putting other people first. After all, that's what got you to where you are! But it's not your job.

Your job is not to be everyone's friend. Your job is not to be the cool aunt.

Your job is to provide clarity.

What should I work on next? Which goal should our team focus on? Is this bug bad enough that we have to stop everything and fix it?

Your job is to come up with answers to those questions.

This is where being a nice person comes in handy. If you listen to people, and care about what they think and say, they will tell you how things are actually going. You will hear about the real issues and what's genuinely exciting.

Your job is not to keep everyone happy. Your job is not parrot the last person you talked to.

Your job is to share where the team stands and where it's headed next.

People can then either get on board or not. Most often they will. (If you're listening well.)

Your job is also to provide clarity to your boss.

Providing clarity to your boss is like providing to your team except different. Your boss has less context on the day-to-day than your team and a stronger grasp of the big picture. So you'll need to spend more time describing the present.

There's lots more that goes into managing engineering teams! But you, as a nice person, are probably already doing those things. Don't let other people's thoughts and feelings distract you from your one job.

At some point you're making enough money and have enough excellent engineers that it makes sense to have some people spend all their time making everyone else go faster. If this is you right now, congratulations, you have (partially) solved one set of problems and now have another set of problems.

(Shouldn't you always be working to make yourself go faster, even when it's just you, or just you and a few other people? Yes. Everyone should be doing the little things that speed up development. Build those habits! But really your focus should be on delighting customers. You can't delight people and keep your sanity with slow builds and useless, flaky tests. Everyone on the team will just have to do those chores. Later on, though, it will make sense for someone's full-time job to be making everyone go faster.)

How do you go about it?

The obvious move is to create a team dedicated to engineering velocity. Call it Infrastructure. Or Platform. Or Developer Experience. Whatever. You'll have someone whose clearly supposed to work on this stuff.

A funny thing happens once you go from delighting customers to trying to improve internal velocity. In theory, the team's customers are other engineers at the company, and they should transfer their focus onto them. In practice, once you step off the customer treadmill, you have the time to think of ideal solutions to everyone's problems. After a few months, you may find the infrastructure team building The Grand New System that's supposed to ship in three months while feature teams go off and build the infrastructure they need themselves because at least then they'll get it when they need it.

To make sure the infrastructure folks stay in touch with customer needs, you can have them embed with feature teams. They'll tackle customer problems head-on and be able to build some sweet repeatable infrastructure along the way! This works well in bursts, until the moment when the feature team “captures” the infrastructure engineer(s) and they start building one-off hacks to solve the problem in front of them.

The null option is to not create a dedicated team and let the existing teams wrestle with velocity and developer experience all on their own. Embrace the status quo! The Tragedy of the Commons will crop up again and again. The build's slow, but I have to ship this thing by Friday and can't go off on a goose chase to find out what's making it slow. My pull request has failed to merge three times in a row because of this flaky test, but I'm not sure I can delete the test nor do I know how to fix it. People know what to do, they're just not sure they're the ones to do it.

I've tried all three approaches. Nothing magically erases all problems. I do have a soft spot for infrastructure engineers embedding with feature teams. Having the infrastructure engineer ship what the feature team needs right away and then take the next N weeks to make a repeatable version of that solution sounds like the sweet spot, but turning down urgent work because someone's polishing an already-working system sounds hard.

More important than which approach you take is deciding what you're aiming for and making that clear. Start measuring before you think you need to do. You can say you're focused on shipping speed to the exclusion of all else, but that probably means you will spend increasing amounts of time firefighting. Measure it and put guardrails in place! And know what you're going to do when the firefighting takes more time than you bargained for!

Before growing your team, know which habits from the current team you want to preserve. You're going to have to really reward those good habits (fixing broken tests straight away) to maintain them as new people join. Know what your team's bad habits are too! They're going to pass those on to new folks.

The start of the project, just after you get the thing working for the first time, is the best. Look at this neat thing we built! We have all this time in front of us to make it amazing!

The end of the project usually comes with a sense of relief. Phew, we made it! And it's nice to not have to do any more tedious launch chores!

Handing off the project feels the strangest. What you poured so much time and attention into has quickly become a stranger. How do I deploy this service again? Why did we leave these janky lines of code here? The handoff stirs up guilt. You're passing off all those TODO comments (and all the things left to be done that no one wrote down).

For me, at least, that guilt masks possessiveness. This is my project! I'll hand it to you once it's absolutely perfect. So perfect that people will always remember what I built. So perfect that no one will have any reason to criticize my work or ask me to improve.

You can see why people love starting new projects. And why they avoid the work necessary to hand the system off to new people.

The benefits of documentation, writing playbooks, building monitoring, and making deploys fast are all shared and largely invisible. The glory of shipping new features or swooping in to put out a fire on an important system is clearly visible and flows to individuals.

“In the Navy, if you haven't built training, you haven't built anything. The moment you step away from the project, the team will be pulled onto other projects, and the whole thing will fizzle out,” Dave Madden told me last week, when we were talking about learning through training versus learning by osmosis.

What does the Navy have to do with startups?

You can think of a startup as two companies. One company ships new things, some of which hopefully delight users. The other company maintains and improves whatever's already a hit with users.

As much as you want to run one company focused on a single goal, you have to run two companies. You have to build new things quickly. And build them well. And maintain a high level of service even when you've turned your attention elsewhere.

The lines between the two companies are not clean. Most people work at both companies simultaneously. Each company requires a different approach, though.

The fastest way to learn to delight users is by doing it. Ship, observe, iterate. Making it clear how quickly and boldly to move helps, but ultimately you learn by doing. Hiring people with passion and energy and throwing them into the deep end is a great strategy for the first company.

Launching products at scale, maintaining high service levels, and monitoring systems for regressions are all teachable skills. As with anything else, you learn the most by doing, but while you're learning you'll make a lot of avoidable mistakes, causing pain for users and your team. Training, documentation, checklists, and shadowing more experienced employees bring people up to speed faster with fewer costly mistakes than throwing people into the deep end.

You can think of the second company's goal as taking the successful parts of the business and executing well (and repeatably) on them.

As often happens midway through writing a post, I'm reading the paragraphs above and thinking “this is all obvious.” Of course you have to ship new things and execute well to succeed as a startup. Of course you have to do a lot of work to smoothly hand over a system to a new owner. Of course, of course.

But this post answers a question that's been on the back of my mind, just a level below conscious thought: why have I resisted investing in training and documentation? Why have I kept throwing people into the deep end? It's because I've been focused on running the delight-users-with-new-things company. Not because I'm distracted by shiny things, but rather because I want to work with people who can :justdoit:.

:justdoit: looks different at each company. When trying to delight users, :justdoit: means “just put something out into the world.” When amplifying successful projects, :justdoit: means “just use tried-and-true techniques and you will have good outcomes.” The key is recognizing when you're showing up to work at which company!

At Replit we end our week with a Weekly Wins meeting. It's what it says on the tin: people demo their wins. Clapping doesn't translate well to video calls, so instead we roast (and praise) each other in the chat.

Not-so-random aside: stoking a lively chat is the one true way to make video calls with more than four people bearable and energizing.

A culture gets what a culture praises. We praise repls of all stripes, demos from localhost, Figma files of all stripes, demos from production, quotes from ~users~ creators, Looms, graphs, and even straight up log lines in a terminal. As I tell people when they join, whatever stage your idea is at, make it more real.

It is not so easy to demo “scales gracefully” or “lowers the cognitive load for other engineers.” So we praise those qualities less. And then wonder why we don't see them consistently.

The obvious move is to seek out and praise unsexy work that keeps the site up and engineers happy (in that order). Jonathan hijacked the channel we used to use for monthly polish sessions and just started posting the small polishing that people do every single day.

Keeping an eye out for engineers eating their vegetables is unreasonably effective. We should all do more of it. But what about the people who save work, hassle, and interruptions by steering down an optimal path?

You should still reward fire-fighting and grunt work when necessary. But you can easily set up a treadmill. People spend most of their time reacting, new folks see that that's what you have to do to be considered successful, they jump on the treadmill, and no one's left to pull the plug.

You want to reward good decisions. Decisions that, with the information at the time, had a high likelihood of success and good downstream effects. When Lincoln hit a stable point during a middle-of-the-night incident, when there was nothing he could really do but stare at graphs waiting for them go wrong, he went to bed.

Validating ideas without building much is also always a good move. Hard to catch in the wild! Doesn't mean you shouldn't look!

Working on a product you yourself use is a blessing and a curse.

If you yourself use the product, you care about what you're building. When you don't care or care indirectly about what you're building, it's hard to put in effort or feel good about the effort you do put in. Caring gives you the stamina to make something great.

The downside is that you're too close. Details that most people don't even notice drive you up a tree. Meanwhile, knowing how the system works lets you slide around annoyances that stop everyone else in their tracks. Your perspective's off.

I first noticed this phenomenon at Spotify. The times I most enjoyed listening to music with the app came when I was on vacation. If the app recommended me music I liked, I could listen to it! If not, I didn't have to listen to it! At work, making a recommendation that people didn't listen to meant you weren't doing as good a job as you could.

Seeing the distance between the way things could be and the way things are is always painful. Growing up means being able to look squarely at that gap without succumbing to apathy or mania.

So, how do you pull it off? How do you use your product as if you don't also build your product?

It sounds hokey, but giving yourself space to choose to use whatever it is you build feels key. When I was out of the office, I opened up Spotify because I wanted to listen to music. Not because I had to judge the output of a recommender.

If what you're building is any good, wouldn't you choose to use it by default? It's not that simple. When you're building, you want to go out of your way to eat your own dogfood. If you don't, you won't find and fix the flaws in your product. If you use nothing but your product, you won't know the strengths of alternatives. You'll be stuck in a cave staring at your masterpiece.

You're also under pressure to play the ambassador. If you won't use your product, why should anyone else? Another key to using your product like a normal user is to do it privately. You don't have to tweet or share what you do, especially not under work accounts.

Moving to a different space helps immensely. No detail is too minor in this regard. Go to a different location. Go to a different room in your house. If you build software, use it on a personal device. Don't have work accounts, work software, work notifications on that device. If you notice problems (you most certainly will), don't login to work accounts to write them down. Do that on your personal device on a separate account. You can always share it later with people!

This all of course means you have to dedicate time to using your product like anyone else would. It feels almost decadent to do that! Shouldn't you be working on your product instead of just using it? Not always. The time you do take to approach your work as an outsider will pay off in focusing on the right things. And in joy.

The big idea behind startups is that a small number of people can grow a business enormously in a short period of time.

Yet here I am on my fifth startup and it feels like I've only begun to understand what it takes to grow.

When you're writing or programming, you're constantly fighting complexity. You add, things become unwieldy you take away. When you first start out you can get carried away producing. It's fun! As you get older you take more and more joy in simplifying and cutting.

Growth brings complexity. The simple systems you have now will have to be split apart into separate systems. Small headaches will grow into big headaches. Jobs that one person could accomplish on their own will spread out to teams of people.

As a programmer, in my heart of hearts, I don't want any of that. I want simplicity. The fervent wish I've had (and never really admitted to myself) is to be able to just build something and have millions (billions?) of people use it without having to deal with the stress of millions (billions?) of users.

To grow is to be uncomfortable. To seek growth is to seek discomfort.

The past two years of growth have been exciting. There have been some times of peak stress and downright hilarious times (if you can't laugh at a mistaken billing change bringing down your whole cloud project, you don't have a sense of humor). It has not been comfortable.

So, as I look at the spreadsheets telling me that to accelerate growth we're going to have to invest in growing the team even more, I have to tamp down my urge to turn back the clock, to stagnate. Six months from now I will be glad we made these moves!

The sun rises higher in the sky, the air warms, and I say to myself, “I'll go for a run.” On that run I remember I haven't exercised in over four months. That's probably why I've felt restless some nights and deflated some days.

On one of the restless nights I say to myself, “I haven't built anything in over a year.” That's not strictly true, I've been building relationships, a team. But I haven't written code in a long time.

The lack you feel from not building does not announce itself. It shows up as weariness and a little angst.

The curse of growing older is that you remember the peaks long after you've descended from them. The biggest hurdle, with exercising, with coding, is to just get out there and do it, even though you won't be performing at your highest level. “I have all this other stuff to do and I don't have time to make anything cool,” says your out-of-shape mind.

A morning when I run at all is a good morning. Same goes for programming.

So I sat down to write a program that prints out how many pull requests per engineer we've shipped in the last two weeks. A very pointy-haired-boss kind of program.

Most all of my time went to getting what I wanted from APIs. You have to figure out how to authenticate. Which library should I use? Can I connect? Oh good, is there an example that matches what I want to do? No? Okay, let me look at the source to see what parameters I can pass into this function that might do what I want.

By the end, I had four or five tabs open per API. Finding the right tab, copying the right code, and remembering what I had already tried presented the biggest cognitive load. Seeing function signatures floating next to the code helped some, but not to the point where I could stay entirely in the editor.

Basically I had to brute-force my way to working code. I've done this hundreds of times over the years. Enough to not want to bother (which leads to restlessness and angst), but also enough to know how to do it, when to stop pulling on a thread. Navigating this world is much, much harder if you're just starting out. How do you even know if you're making progress?

The code I write these days is also brutal. I don't care about duplicating small snippets. If I need to remember what some value is I'll assign it to a variable with a boringly descriptive name. Iterating over every element in a data structure sure beats spending time on clever traversals.

Taking the ungainly ways people actually write code and speeding them up is where the action's at.

N.B. Do not ask me why counting pull requests in private repositories means you have to grant the GitHub personal access token an OAuth scope to manipulate any of your repositories!

When you're trying to do something, there's no greater enemy than worrying about how you should do it. Or where you should start. Or what even you should be doing in the first place.

There's something about the word “should” that stops you. Your mind switches into far mode. You examine things from a distance. You ask questions, but don't go out and make the mistakes that would answer those questions.

Twenty years ago it was less dangerous to sit around asking “what should we do?” than it is today. There were only so many books or newsgroups to tell you how you “should” code or “should” run your company. Now you can absorb opinions indefinitely. Slack and email constantly ping you telling you what you “should” be doing.

Doing what you “should” do is often much more about doing what you think other people will admire than trying to find an optimal path.

It's a splash of cold water to think that the background level of “should” is high and still rising. We have a :justdoit: emoji in the Replit slack that is probably our most valuable piece of intellectual property.