Letting AI Write Your Software: What Happens When Nobody Checks It

Can AI really build software for my business?

Yes. You can now type “make me a thing that does everything” into an AI tool, and it will have a go. You don’t need to be a programmer. You describe what you want, try what it gives you, tell it what’s wrong, and it tries again.

That’s a genuine change. Someone in your office with an idea for a small tool can build it this afternoon, instead of adding it to a list that never gets the money. So it’s tempting to go further: rather than paying every month for the system that tracks your customers, why not have AI build your own?

Why does building your own usually cost more than buying?

Because the first nine-tenths is the easy part. AI will quickly give you something that looks right. The last few per cent, getting it to work reliably, every day, for everyone, without slowing down, is really hard. The companies that sell ready-made software have spent thousands and thousands of hours on exactly that.

Dave sees plenty of businesses decide they’ll build their own customer system and do it better than everyone else. What usually happens is that someone spends more and more of their week fixing problems and adding the features a ready-made system already had. And as AI gets better, the problem won’t go away. It will just be better hidden.

It’s easy to get carried away and believe you’re making something nobody else has. You almost certainly aren’t. A year and a bit ago Dave built a free email checker, SecureMyEmails, with AI. There are now plenty of tools that look almost identical. His wasn’t wasted, but he built it to learn how the process works, not because he expected it to make money.

What happens when nobody checks what the AI built?

Dave found out on his own website. He asked AI to keep a record of what visitors typed into one of his tools. It did, and it worked perfectly. It also stored every word in a place anyone on the internet could read. Nothing in what the AI wrote kept it private, and nothing warned him.

He only caught it because he thought to ask the AI one question: are these records private, or could anyone get at them? Had he not asked, it would have gone live working perfectly, and it would have been a breach of data protection law (UK GDPR) from day one. If personal details leak and that’s likely to put people at risk, the Information Commissioner’s Office says you must tell it as soon as possible, and within 72 hours where you can.

Four problems tend to hide inside something that works:

What about connecting AI to our own files and email?

The same thing in a different form. More and more businesses are plugging AI assistants into their shared files, their email or their customer records with nothing controlling what it can see. Anything it can see, it can pass on to the wrong person in answer to a question.

Setting this up safely is difficult, even for people who do it every day. If nobody in your business can say exactly what the assistant can reach, it isn’t ready. The government’s National Cyber Security Centre has a short guide to AI for managers and board members without a technical background, which sets out the risks and the questions to ask.

Where’s the line between fine and risky?

One question settles most cases: does it hold anything that belongs to somebody else?

The “Go ahead” list is where AI-built tools earn their keep: quick, cheap and useful, and if one breaks nobody outside the business notices.

What does “have it checked” mean?

Not a big project. Someone who can read the instructions the AI wrote spends a short while looking through them before the tool goes live. The mistakes are rarely subtle once you know where to look.

So what should we use AI for?

Start with the jobs that annoy everyone. Dave’s order is simple, and getting rid of a job altogether always beats speeding it up:

1. Pick the jobs that waste the most time. The ones with the most friction, the most chasing and the most moaning.
2. See whether AI can remove or speed up parts of them. Removing a step completely is always better.
3. Build the skills, with help. This is where an experienced adviser earns their fee: working out how to do it safely.
4. Only then, build a better screen for the tools you already use. You keep the software you trust and change how you see it.

Dave has done two of these at GoodChoice IT. The first is Sharon, an AI assistant whose job is to nudge the team when they forget to record their time. Her messages are written to jog people’s memory, not to nag, and there’s clear evidence more time is now being recorded against customers’ requests.

The second is a new screen for the system we use to keep an eye on our clients’ computers. The information Dave needed was buried several clicks deep, so he had AI build him one page that shows it all.

That’s a far safer project than building a whole new system for your business. In a year or two it may make sense to go further. For now, our guide to what to automate first with AI is a good place to start.

Has anyone in your business already built something?

It’s worth finding out, and not to tell anyone off. Someone who builds a tool to save time has the right instinct, and you want more of it. Ask the five questions above about anything that’s already in use, then agree the line with your team: tools for your own use that you could throw away, go ahead; anything holding somebody else’s details gets checked first.

If nobody in the business has the time or the know-how to keep track of what’s being built and what it can reach, that’s part of what our AI Manager service does. Our page on getting started with AI in your business covers the wider basics, and who should own AI risk looks at whose job this is.