Quoting My AI Adoption Journey

Quoting My AI Adoption Journey (via)

Such a strong piece from Mitchell Hashimoto about his experiences with GenAI and coding that hits quite close to home.

I quickly discovered for myself from first principles what others were already saying, but discovering it myself resulted in a stronger fundamental understanding.

1. Break down sessions into separate clear, actionable tasks. Don't try to "draw the owl" in one mega session.
2. For vague requests, split the work into separate planning vs. execution sessions.
3. If you give an agent a way to verify its work, it more often than not fixes its own mistakes and prevents regressions. More generally, I also found the edges of what agents - at the time - were good at, what they weren't good at, and for the tasks they were good at how to achieve the results I wanted.

Coding with agents still is to a long extend very different than the handwritten code we used to create not long ago. Not only how you breakdown the problem is different but also the kind of problems you lookup for help and how you approach problems change accordingly. Forcing yourself to code wth agents is useful to charge your heuristic battery, even if you're not using any of the code created you gain knowledge about what kind of approach works.

To try to find some efficiency, I next started up a new pattern: block out the last 30 minutes of every day to kick off one or more agents. My hypothesis was that perhaps I could gain some efficiency if the agent can make some positive progress in the times I can't work anyways. Basically: instead of trying to do more in the time I have, try to do more in the time I don't have. ...

• Parallel agents attempting different vague ideas I had but didn't have time to get started on. I didn't expect them to produce something I'd ever ship here, but perhaps could illuminate some unknown unknowns when I got to the task the next day.

That's such a good idea that I tried a couple of times but never formalized as part of my workflow. But it makes sense. The idea is to offload to an agent tasks that require a long context window but low agency, which might sound counter-intuitive. You still want to control what gets build and, to some extend, how it gets build. Agents can help you arrive there, with enough time and context.

More specifically, I would start each day by taking the results of my prior night's triage agents, filter them manually to find the issues that an agent will almost certainly solve well, and then keep them going in the background (one at a time, not in parallel). Meanwhile, I'd work on something else. I wasn't going to social media (any more than usual without AI), I wasn't watching videos, etc. I was in my own, normal, pre-AI deep thinking mode working on something I wanted to work on or had to work on. Importantly, I think the "work on something else" helps counteract the highly publicized Anthropic skill formation paper. Well, you're trading off: not forming skills for the tasks you're delegating to the agent while continuing to form skills naturally in the tasks you continue to work on manually. At this point I was firmly in the "no way I can go back" territory. I felt more efficient, but even if I wasn't, the thing I liked the most was that I could now focus my coding and thinking on tasks I really loved while still adequately completing the tasks I didn't.

This also reflects my experience. Even when tasks I send to coding agent takes the same amount of time as me or even longer it doesn't matter as this is a task I would have to stop everything I'm doing to tackle and pay the price of changing context anyway. For me this is the dishwasher vs. washing dishes by hand dilemma, a dishwasher can take +3h to wash your dishes but this is a time you can use to do something else during your day. The next question is what you do with this extra time, skill atrophy is real and even if it's tempting to have same level of productivity and bank the gain in time as slack you risk being left behind by the your lack of agency.

I don't know if there is a broad industry-accepted term for this yet, but I've grown to calling this "harness engineering." It is the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again. I don't need to invent any new terms here; if another one exists, I'll jump on the bandwagon.

1. Better implicit prompting (AGENTS.md). For simple things, like the agent repeatedly running the wrong commands or finding the wrong APIs, update the AGENTS.md (or equivalent). Here is an example from Ghostty. Each line in that file is based on a bad agent behavior, and it almost completely resolved them all.

This is a concept I use a lot on my sessions that I usually call a postmortem step where the agent lookback at the task just worked and reflect on what went wrong, how it got solved and to save some notes about the issue to itself so next time it doesn't happen again.

I'm also operating under the goal of having an agent running at all times. If an agent isn't running, I ask myself "is there something an agent could be doing for me right now?" I'm not running multiple agents, and currently don't really want to. I find having the one agent running is a good balance for me right now between being able to do deep, manual work I find enjoyable, and babysitting my kind of stupid and yet mysteriously productive robot friend.

Same impression for me, as long as agents are changing my workflow I feel they still need lots of handholding that I would not feel confortable having several in parallel while still trying to work on my own stuff. This is usually where I draw the line between productivity boost vs. personal growth.