AI Software Factories

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

For the past few months, kirupa.com has had a night shift. Every few minutes, a small crew of AI agents wakes up on the server, looks at the problems the site has reported about itself, and writes up what it found on a GitHub issue. One agent sorts the problem, one digs into the evidence, one proposes a fix, and one reviews that proposal. By the time I sit down with my coffee, the boring detective work is mostly done. I call this crew Nightowl.

Nightowl is a small example of something people have started calling an AI software factory. In this article, we'll look at what a software factory is, walk through every part of Nightowl, and then zoom out to how you could build one yourself and decide what to trust it with.

Onwards!

What Is a Software Factory?

Think of a real factory floor. Raw materials come in one end and move down a line of stations, where each station does one job before passing the work along. An inspector checks the result before anything goes out the door. An AI software factory works the same way, except the raw material is problems in a codebase: bugs, errors, broken pages, and missing features. Here is the general shape:

[Image: A software factory as an assembly line. Signals like audits and error reports come in on the left and become issues. The issues move through four agent stations: triage, analyze, propose a fix, and review. At the end, a person at an inspection gate decides what ships.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_line.png)

Put into one sentence, a software factory is a set of agents and automated checks that keeps finding, sorting, and fixing problems in a codebase, with humans standing at the gates that matter. Each station is an agent (https://www.kirupa.com/ai_fundamentals/ai_agents.htm) with one narrow job, and the line itself is a workflow (https://www.kirupa.com/ai_fundamentals/workflows_vs_agents.htm): the order of stations is fixed by our code, not chosen by a model.

Why is everyone suddenly talking about this? Back in the Introduction (https://www.kirupa.com/ai_fundamentals/introduction_to_ai_fundamentals.htm), we saw that the model is never the whole product. The questions that make or break things are about how we use the model. A software factory is almost entirely a "how" problem. Generating a fix has gotten cheap. What's still expensive is a person's attention: reading the report, finding the cause, and deciding whether a fix is safe. The Vercel team hit this wall with their AI SDK and wrote up how they built a software factory (https://vercel.com/blog/building-a-software-factory-for-ai-sdk) with one agent per task, starting with classification, while a human still merges every change. Nightowl borrows that pattern, scaled way down for one person and one very old website.

The kirupa.com Problem

kirupa.com has been around for decades, and very little of it was planned as one system. At the time of writing, the repository holds about 3,900 HTML pages, about 200 PHP files, and 92 server-side include files (shared snippets like the header and footer that the server stitches into every page). Around those pages sit a bunch of integrations, each one added when it was needed:

[Image: kirupa.com in the middle, connected to the things around it: shared includes for the header and footer, PHP endpoints, kirupaBot, quizzes, the forum, the newsletter sync, FTP publishing, and the icon galleries. A handwritten note says it's all held together with duct tape.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_duct_tape.png)

Every one of these works, mostly. The trouble is that when something breaks, it rarely announces itself. A shared include goes blank, and a few hundred pages quietly lose their sidebar. A stylesheet ends up empty, and 62 pages show up as plain unstyled text. For years, my approach was to fix things when somebody emailed me about them, and that doesn't scale when there's one of me. What I needed was for the site to tell me what's wrong, and for something to do the first round of investigating before I get involved.

Input 1: Site-Wide Audits

The first input started with one big audit. In August, I had an agent go through the entire repository looking for anything broken, and it filed about 25 GitHub issues in one go. Some were embarrassing. The sitemap was capped at 500 URLs, so search engines never heard about nearly half of the modern tutorials, and 72 pages had an og:url (the address a page gives when someone shares it) pointing at a different article.

The most useful finding was that almost every one of those bugs could have been caught by a simple, mechanical check. So each fix came with a second step, which was to turn the bug into a check that runs forever after:

[Image: The audit loop. An audit finds a bug. The bug becomes a GitHub issue. The issue gets fixed. The fix adds a permanent check, and that check runs before every publish, so the same kind of bug can't come back.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_audit_loop.png)

Today, those checks are part of every publish. The publishing script won't upload anything unless the metadata audit and a deployment regression check pass, and the broader site checks are on the checklist before each publish. Together, they verify that every include exists and isn't empty, that every page has a stylesheet, that sharing addresses point at the right page, that the sitemap and RSS feed are complete and valid, and that images have alt text. Here's a recent run of the main checker, with two long file paths shortened:

[Image: Terminal output from a run of tools/site_checks.py. Each check prints its finding count: ssi, stylesheets, chrome, canonical, og-url, sitemap, and rss all report 0 findings and are blocking. Sitemap coverage reports 0 findings and is warn-only. The AI index check reports 2 warn-only findings about one article's Markdown and text mirrors not being listed yet.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_audit_run.png)

A blocking check fails the whole run when it finds anything, while a warn-only check prints a warning and lets the run pass. The two warnings are real, by the way. One article's AI-readable copies weren't listed in the site's index file yet, which is exactly the kind of small drift these checks exist to catch. It's the same idea as the golden datasets (https://www.kirupa.com/ai_fundamentals/golden_datasets.htm) we'll build for AI features later: every past mistake becomes a test case, so the site remembers what has gone wrong before.

Input 2: A Reporter on Every Page

Audits only see the files in the repository, not what happens when a real visitor's browser runs them. For that, every page that uses the site's shared header loads a tiny script I call the error reporter.

The reporter listens for four kinds of trouble:

1. Anything a script writes with console.error.
2. Uncaught JavaScript errors, the kind that stop a script in its tracks.
3. Unhandled promise rejections, which is when some async work fails and nobody was waiting to catch it.
4. A script, stylesheet, or image that fails to load.

On the server side, a similar hook watches PHP for warnings, fatal errors, and responses that end in a 5xx server error. Both feed the same error store.

The reporter doesn't measure page speed. Performance problems reach me through the audits instead (two of the original audit issues were about bloated images and dead stylesheets). The reporter's job on the performance front is to never be the problem. It's under 4 KB, loads late at low priority, sends nothing on a page without errors, and batches what it does send into an idle moment. A browser test compares a page with and without it on a CPU slowed down four times, checking how long the biggest piece of content takes to appear (LCP) and how much the layout jumps around (CLS), and fails if either gets noticeably worse.

Privacy was the other big constraint. A script on every page of a public website sees an awful lot, so the safest plan is to send almost nothing:

[Image: What the error reporter sends and what it never sends. Sent: a fixed category like Type error, the page's path, and the file, line, and column where it happened. Never sent: the error's message text, stack traces, cookies, the query string, the referrer, and the browser's user agent.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_reporter.png)

Every error becomes one of a few fixed categories, like Type error or Image failed to load, and the actual message is thrown away, because an error message can contain anything, including something a visitor typed. The reporter caps itself at 20 reports per page view, and the server keeps counts for 30 days. There's no sampling and no opt-in, because the privacy plan is minimization: collect so little that there's nothing sensitive to leak. It all lands in a private dashboard that groups errors by where in the code they happened. Here it is, filled with test errors I triggered on a local copy of the site:

[Image: The kirupa.com dashboard's Errors view, filled with test errors from a local copy of the site. Totals show 32 occurrences, 4 error groups, and 7 pages. Below, the most common errors are ranked by occurrences: a Type error in /ssi/newDesign/kirupa.js happened 14 times on 3 pages, an image that failed to load happened 9 times on 1 page, a console error happened 6 times on 2 pages, and an unhandled promise rejection happened 3 times on 1 page.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_dashboard.jpg)

From Errors to Issues

I'm not going to stare at a dashboard every morning. So once a day, a job called the issue filing agent decides which error groups deserve a GitHub issue. Despite its name, there's no AI in it, just plain code making plain decisions:

[Image: How an error group becomes a GitHub issue. First, did it repeat several times in the last day? If not, skip it. Next, can we confirm it, for example by checking that a missing image really returns a 404? If not, defer it. Next, have we filed this before? If so, adopt the existing issue. Finally, is there room under the daily cap? If so, file a new issue.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_issue_filer.png)

Each question guards against a real failure. A one-off error from a browser extension isn't worth an issue, so a group has to repeat. A failed image might just mean an ad blocker ate it, so the job asks the server whether the file is really gone. Every issue carries a hidden fingerprint, and a private ledger remembers every fingerprint ever filed, so the same error never gets a second issue (this is the dedupe step). A daily cap keeps a flood of errors from burying everything else. And because anyone can send fake reports to a public endpoint, every report is treated as a lead to investigate, never as proof of a bug.

There's also a quieter third input. Questions people ask kirupaBot (the site's chatbot) and searches that come back empty get grouped once a day, and when a model is very confident a topic is missing from the site, that becomes a Missing content issue, which is a request for me to write something.

The Agent Loop on the Server

At this point, problems are landing in GitHub as issues. Now the agents take over. Here is the whole of Nightowl in one picture:

[Image: All of Nightowl. On the left, three inputs: site-wide audits, the error reporter and dashboard, and kirupaBot content requests. They all become GitHub issues. A scheduled job on the server then runs four agents in order: Classification, Analysis, Implementation, and Review. Each one posts a comment on the issue. Kirupa reads the result and decides what ships.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_nightowl.png)

The agent half runs on the same shared hosting account as the website. Every five minutes, a scheduled job checks the open issues and moves each one forward a step or two. Each agent is a model call through the GitHub Copilot SDK with its own instructions and its own narrow job.

The Classification agent goes first, and its job is tiny on purpose. It reads an issue's title and body and decides whether it's a bug, a feature request, or something it isn't sure about. When it's confident, it adds the label and posts a comment explaining why, quoting the exact words that convinced it. When it isn't, it adds no label and asks for a human. It never removes a label, including any I've added myself.

The other three agents run in a strict order, and their instructions are just as clear about what each one is not allowed to do:

[Image: The four stages and their rules. Classification sorts the issue and explains why, but proposes nothing. Analysis gathers evidence, likely causes, gaps, and what done looks like, but proposes no solution. Implementation is the first stage allowed to propose a fix, as an unapplied patch. Review critiques that exact proposal, but writes no new fix.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_stages.png)

Why so strict? A model asked to analyze a problem will happily jump straight to a solution, and once a solution is on the page, every later step anchors on it. Keeping Analysis to evidence (what's broken, where, how sure we are, and what "fixed" looks like from the outside) means Implementation starts from facts instead of a guess. Review then picks apart that one proposal for side effects, compatibility, accessibility, and anything untested, without swapping in a fix of its own. Each agent posts its result as a comment on the issue. Here's roughly what that looks like on a site error report, trimmed to the first lines of each comment:

[Image: A GitHub issue titled Site error: Type error in a shared script, with four comments from the stages in order. 1, Classification agent: the existing bug label is kept. 2, Analysis agent: the cause can't be confirmed from the supplied source, so a human is required. 3, Implementation agent: held for manual review, no patch, model not invoked. 4, Review agent: static review only, no tests were run, human review required.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_issue_comments.png)

That's a pretty typical outcome. The agents only see a curated snapshot of the source, and the file this error pointed at wasn't in it. So Analysis said it couldn't confirm a cause, and the later stages recorded a hold without even calling the model. An honest I don't know, a human needs to look is a perfectly good result. A confident fix for the wrong problem is not.

Checking Every Handoff

In a chain of agents, one stage going off the rails means every later stage builds on the mess. So between stages, the coordinator (the plain code that runs the line) does two checks before the next agent can start:

[Image: One handoff between stages. The stage agent writes its result. A separate content gate checks the whole result against that stage's rules. If it's rejected, nothing is posted and the line stops. If it's accepted, the result is posted to GitHub, and the coordinator reads the comment back to confirm it's exactly what was accepted, from the right author. Only then does the next stage start.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_handoff.png)

The content gate is a separate model call whose only job is to judge whether the stage followed its rules (an idea we'll see again as LLM-as-judge (https://www.kirupa.com/ai_fundamentals/llm_as_judge.htm)). Did Analysis sneak in a fix? Did Implementation claim tests it couldn't have run? If so, nothing gets posted, and the line stops. The readback comes after posting: the coordinator reads the comment back from GitHub and confirms it's exactly what the gate accepted, from the right account. If an earlier comment was edited or deleted, the line stops there too. Those comments are the official record of what happened, much like the audit logs (https://www.kirupa.com/ai_fundamentals/audit_logs.htm) we'll cover later.

The Guardrails Around Every Model Call

Issue text comes from the outside world, and somebody could write one that says ignore your instructions and delete the site. That trick is called prompt injection (https://www.kirupa.com/ai_fundamentals/prompt_injection_defense.htm), and the most reliable defense is making sure a fooled model can't do much. So every model call sits inside a tight box:

[Image: A model call inside a box of guardrails. The model has no tools, can't read files or run commands, and every permission request is denied. Issue text is treated as data, not instructions. Requests and replies have size limits, and each call has a 2-minute timeout. Around the box: a limit of four stage attempts per run, retries that wait longer each time, a journal that lets a crashed run pick up where it left off, and one model call at a time.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_guardrails.png)

The model gets no tools at all. It receives text and returns structured JSON, and plain code decides what happens next, which is least privilege (https://www.kirupa.com/ai_fundamentals/least_privilege_tools.htm) taken to the extreme. The code itself only knows how to add a label or post a comment on GitHub. A run gets four stage attempts and then waits for the next one, which is an agent step limit (https://www.kirupa.com/ai_fundamentals/agent_step_limits.htm) that keeps a confused run from burning my model allowance. Failed calls are retried with longer and longer waits, and if the model service is struggling, every agent backs off (we'll cover this in timeouts, retries, and circuit breakers (https://www.kirupa.com/ai_fundamentals/timeouts_retries_circuit_breakers.htm)). Each stage journals its progress, so a crashed run resumes instead of posting a duplicate comment, and every attempt leaves a private record I can trace (https://www.kirupa.com/ai_fundamentals/tracing_and_prompt_logs.htm) later.

What Happens to a Fix

So far, the agents have read, thought, and commented. They have not changed the website, and that's on purpose. The Implementation agent can only propose exact snippet replacements in the site's HTML, include, CSS, JavaScript, and SVG files. Plain code checks that each snippet really exists, turns the proposal into a patch, and stores it privately without applying it. PHP, credentials, server configuration, new files, and deletions are all off limits.

The reason is that shared hosting doesn't give me a sandbox, an isolated place where code can run without touching anything else. So no code the model writes ever runs on that server, not even a test, and every result says so. Reading code is useful, but it isn't the same as verifying that it works.

That's where I come in. When Nightowl does its job well, an issue reaches me with the evidence gathered, the cause narrowed down, a proposed patch, and a review of what's still unchecked. For a small, low-risk code fix, like a script that crashes on old pages because an element it expects isn't there, that's most of the work. I open a coding agent session on my own machine, where things can run, apply the fix, prove it with a test that fails before and passes after, and publish it through the same guarded checks as everything else.

For a few days in early November, a cloud coding agent took that last step too. It reproduced each new bug, wrote a fix with a regression test, and opened a draft pull request, but it was never allowed to merge. One of those pull requests fixed exactly that crash, with 56 browser scenarios that failed before the change and passed after, and I reviewed and merged it. I've since turned that automation off, because two independent agents working the same issues started stepping on each other. Today, every issue has one owner, which is Nightowl, and every fix goes through me.

Visuals and Words Always Come to Me

Some issues aren't about code at all. They're about what the site looks like or says: an article's text got truncated, a figure is confusing, or somebody wants a tutorial that doesn't exist yet. Those take a different path:

[Image: Two paths through the factory. A code bug with evidence gets a proposed patch and a review, and then Kirupa checks it, tests it, and ships it. A visual or text content issue, like an article, figure, or tutorial request, gets at most an outline or a manual-review hold, and then Kirupa writes or draws it himself.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_routing.png)

A tutorial request gets, at most, a proposed outline. When an article's text is broken, the agents can confirm it, but they can't know what the right words were, so they hand it to me with a hold instead of making something up. Anything visual or written comes to me, every time. The reason is taste. A test can prove that a script no longer crashes, but no test can prove that a paragraph sounds like me or that a figure reads well at a glance. As we saw in What Is AI Slop? (https://www.kirupa.com/ai_fundamentals/what_is_ai_slop.htm), those mistakes are about judgment rather than correctness, and readers come here for a particular voice.

Honest Tradeoffs

Nightowl isn't magic, and some parts are a bit meh:

- A lot of results are holds, because the cause lives in a file the agents can't see, in server configuration, or only in production. Holds are honest and useful, but they aren't fixes.
- Reading isn't running. A plausible patch can still be wrong, which is why nothing ships without a real test.
- Inputs can lie. Reports can be forged and anyone can write an issue, so every input is a lead, never a verdict.
- It isn't free. Every stage uses my model allowance.
- Early on, more than one agent wrote to the same issues and the comments ended up out of order. The fix was a rule in the repository's instructions for every agent: one coordinator, one owner per issue, and four stages in a fixed order.

I still merge every change, publish, write and draw everything, and decide which issues are worth anyone's time. Was it worth it? Very much so. Over the last few months, I've shipped more features on kirupa.com than I did in the previous ten years, and the site's many integrations and duct-taped pieces kept working through all of it. Coding agents getting good is a big part of that, but the factory is what made it feel safe to move that fast, because a broken page now turns into an investigated issue instead of a mystery.

Build Your Own Software Factory

You don't need all of Nightowl to get most of the benefit, and building it all at once is a good way to end up with a pile of agents nobody trusts. Add one station at a time, and add the next only once the current one has earned it:

[Image: Seven steps to build a software factory, drawn as a staircase. 1, signals: audits and error reports. 2, issues, with dedupe. 3, a triage agent that only classifies. 4, analysis. 5, fix proposals. 6, review. 7, only then, consider applying fixes automatically. Through every step, log everything and check every handoff.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_build_steps.png)

Start with signals by turning bugs you've already fixed into checks that run before every deploy, and add a small error reporter that collects as little as it can. Turn signals into issues with a fingerprint per problem, a memory of every fingerprint filed, and a daily cap, none of which needs AI. Then add one triage agent that only classifies, returning structured output (https://www.kirupa.com/ai_fundamentals/structured_outputs_json_schema.htm) with a label, a confidence, and the exact quote behind it. After that comes analysis (evidence, but no solution), then fix proposals (small, pointing at code that really exists, and never applied), then a separate review agent that critiques the proposal and admits what wasn't tested. Only then consider applying fixes automatically, and only for the narrow kinds of changes that have earned it.

Through all seven steps, check every handoff and log everything, so that when an agent does something odd, you can find out why. When you want to measure how good each agent is, error analysis (https://www.kirupa.com/ai_fundamentals/error_analysis_on_traces.htm) and offline and online evals (https://www.kirupa.com/ai_fundamentals/offline_vs_online_evals.htm) cover how to do that with real numbers instead of vibes.

Trust: When to Automate vs. Keep a Human in the Loop

Every station raises the same question: should a person check this, or can we let it run? In Human Approval for Risky Actions (https://www.kirupa.com/ai_fundamentals/human_approval_for_risky_actions.htm), we saw that approving everything is its own problem, because people start clicking yes without reading. I find four questions help the most:

1. How bad is a mistake, and can we undo it? This is often called the blast radius. A duplicate issue is easy to close. A broken shared header breaks every page at once.
2. Can a check prove it's right? A crash fix with a before-and-after test is verifiable. "This paragraph is clearer now" is not.
3. How often does it happen? Frequent changes are worth automating and give us lots of evidence. Something that happens twice a year isn't worth the risk.
4. Does it need taste? Visual design and writing do, and no amount of automation changes that.

Rather than a yes-or-no switch, I think of it as a dial with four settings. Here it is, with where each part of kirupa.com sits today:

[Image: A dial with four settings, from most human to most automated. Human does it: writing, figures, and tutorials. Human approves: every code fix and every publish. Human spot-checks: classification labels and the agents' issue comments. Fully automated: running the site checks before every publish, and filing issues from repeated errors.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_trust_dial.png)

Both ends are occupied. The checks that block a broken publish and the job that files issues run with nobody watching, because their mistakes are cheap, easy to undo, and they run constantly. Writing and figures sit at the other end, and they're staying there. Your own settings depend on who you are. A bank moving money, a hospital touching patient records, a startup racing to find customers, and a personal site like this one will set these very differently, and they should. A mistake on kirupa.com means a broken page for a few hours. A mistake in a payment system means real money in the wrong place.

Graduating a Review From Human to Automated

Dial settings aren't permanent. A kind of change that needs my approval today could earn its way to a spot-check tomorrow. I think of this as graduating, and it should be earned with evidence, never granted because we're tired of reviewing. Here is the ladder, with the way down drawn as clearly as the way up:

[Image: A ladder with four rungs: human does it, shadow mode, human approves, and automated with spot-checks. Arrows climb the ladder only when a category meets its criteria. A separate arrow drops a category back down to human approval the moment an automated change causes a miss.] (https://www.kirupa.com/ai_fundamentals/images/ai_software_factories_graduation.png)

The rung people skip is shadow mode, where the agent does its job on every issue in a category, but its output isn't used. We just compare what it would have done with what the human decided. It costs nothing in risk, and it tells us how often the agent agrees with us and what it gets wrong when it doesn't, much like testing in the lab before the road in Offline vs. Online Evals (https://www.kirupa.com/ai_fundamentals/offline_vs_online_evals.htm). For a category to climb a rung, I'd want all of these to be true:

1. A track record. A long run of proposals in that category that a human approved without changes.
2. A check for every past mistake. Every proposal a human did correct is now caught by an automated check.
3. Easy rollback, plus monitoring. The change undoes in one step, and something like the error reporter would notice quickly if it made things worse.
4. A narrow category. "Null checks for missing elements in shared scripts" can graduate. "Bug fixes" is way too broad.
5. A demotion path. The first time an automated change causes a problem, the category drops back to human approval and has to earn its way up again.

Even then, let it out slowly: automate a small slice of cases first, watch, and widen from there. That's a progressive rollout, the same canary approach we use for a new prompt or model.

So what has graduated on kirupa.com? Not much yet, and that's by design. The pre-publish checks and issue filing are fully automated, and the Classification agent's labels are automated with me spot-checking them. Every proposed code change still needs my approval, and nothing has a formal track record yet. The first candidate I'd consider is that narrow category of crash fixes in shared scripts, where a browser test proves the fix, the change is a few lines, and the error reporter would tell me within a day if something broke. Visuals and words will never be on this ladder.

Conclusion

We started with a night shift and ended with a dial. A software factory isn't one clever agent. It's a line of narrow stations: inputs that notice problems, plain code that turns them into deduplicated issues, agents that each do one job in a fixed order, checks on every handoff, a tight box around every model call, and people at the gates that need judgment. Nightowl is a small version of that, run by one person on shared hosting. Almost every piece of it comes from an idea covered elsewhere in this section:

Piece of Nightowl | Where we cover it
Agents with one narrow job, in a fixed order | What Makes an AI Agent (https://www.kirupa.com/ai_fundamentals/ai_agents.htm), Workflows vs. Agents: Start with the Simplest (https://www.kirupa.com/ai_fundamentals/workflows_vs_agents.htm)
Labels and proposals as validated JSON | Structured Outputs with JSON Schemas (https://www.kirupa.com/ai_fundamentals/structured_outputs_json_schema.htm)
Four stage attempts per run | Agent Step Limits and Stop Conditions (https://www.kirupa.com/ai_fundamentals/agent_step_limits.htm)
A person approves every change | Human Approval for Risky Actions (https://www.kirupa.com/ai_fundamentals/human_approval_for_risky_actions.htm)
The content gate | LLM-as-Judge, Checked Against Human Labels (https://www.kirupa.com/ai_fundamentals/llm_as_judge.htm)
Past bugs become permanent checks | Golden Datasets (https://www.kirupa.com/ai_fundamentals/golden_datasets.htm), Regression Tests on Every Prompt Change (https://www.kirupa.com/ai_fundamentals/prompt_regression_tests.htm)
Shadow mode and progressive rollout | Offline vs. Online Evals (https://www.kirupa.com/ai_fundamentals/offline_vs_online_evals.htm)
Timeouts, backoff, and cooldowns | Timeouts, Retries, and Circuit Breakers (https://www.kirupa.com/ai_fundamentals/timeouts_retries_circuit_breakers.htm)
A private record of every attempt | Tracing and Prompt Logs (https://www.kirupa.com/ai_fundamentals/tracing_and_prompt_logs.htm)
No tools, and only labels and comments on GitHub | Least-Privilege Access for Tools (https://www.kirupa.com/ai_fundamentals/least_privilege_tools.htm)
Issue text treated as data | Prompt Injection Defense (https://www.kirupa.com/ai_fundamentals/prompt_injection_defense.htm)
Comments as a record nobody can quietly rewrite | Audit Logs (https://www.kirupa.com/ai_fundamentals/audit_logs.htm)

If you build a factory of your own, start smaller than feels useful, keep a person at every gate at first, and let the evidence tell you when a gate can open. And if your project has words or pictures with your name on them, keep those for yourself. That's the fun part anyway!