Maybe some devs here can help me, I was recently promoted to “head of AI” at my work despite being very outwardly ambivalent towards it. So I’m struggling to figure out what would actually create value instead of just being an expensive waste of time but still satisfy the higher ups AI lust.
My first idea that I thought would actually be useful was just setting up the architecture for an actual analytics database for us and then let them explore it with metabase (then letting them use Claude for their wow factor of exploring it with AI or whatever).
But now I’m somewhat at a loss, so any insight you all have would be really helpful!
Since there’s zero information about what kind of company you’re working at, the following is extremely generalized.
- integrating with monitoring systems, analytics DBs, ticket systems, whatever is used by management, allowing them to ask questions in natural language
- process automation using agentic workflows, e.g. pre-analysis of incoming email queue summarizing / sentiment analysis before the customer support sees it
- provide access to models and model APIs for development workflows and integration into git / ci, allowing to use llm in local development and e.g. setting up something like automated code reviews (not a replacement for human review, only as an addition)
- set up coaching, responsible use, hallucinations, etc.
Whatever you do, take security and data security especially into consideration first, not after:
- consider whether your used provider reuses your data for learning
- consider whether it’s relevant where it’s located (GDPR customers?)
- always set spending limits
- consider your local and your customers data protection laws and regulations that apply to your company (especially in health and financing)
This is wonderful thank you.
I’m starting to get the idea that the LLM is just a small part of this and what’s really important is instead the bunch of architecture and guardrails around it to form how the humans and ai will act with any given system.
Personally I see LLMs as a tool like any other. You can use it to mass produce low quality slop, just as you can use it to help you produce a higher quality output.
You’re perfectly right about architecture and guardrails, that’s how it has always been with any other tool or piece of software. It depends on how you use it. Remember the no-code hype train? It’s literally the same, people have been shoving it into everything, no matter whether it made sense. It worked for some, and it made development costs explode for others.
Guardrails are especially important for LLMs because you do not have deterministic outputs and potentially exploding costs.
So analyze, measure, and think about where and how it makes sense to integrate, and build it incrementally, again, just like with any other piece of software. Start slow, keep humans in the loop, measure and analyze, and improve incrementally. When you achieve confidence, potentially start automating going into an agentic direction, when it makes sense and the risks have been considered, but always keep provenance. You do not want blind decisions by the magical AI box.
And just to repeat, because I’ve seen heads roll because of dumb decisions: keep cost under control and always have limits set, and always consider which data flows into the AI and what happens with it afterwards.
Producing a half a million bill in a month by accident or neglect or suddenly having your customer database queryable on a public model is a surefire way to drive the company or at least your career to the ground in seconds of wrong decisions.
Also, read into all the stuff built around LLMs, protocols like MCP, attacks and defenses on LLMs, get knowledge about the inner workings, experiment and learn. When you’re the head of AI, you’re supposed to be the person who knows. And when you know what it does, how it works, and how to use it, you’ll find actually good and appropriate use-cases naturally.
If the person who put you in charge of it knew of your indifference, that’s a smart manager, and you should just trust your gut on this.
I would think in terms of security and guardrails if I were you.
If you just vibe AI, it will lull you into thinking you’re doing great.
- Define what successful AI usage looks like and include metrics and measurements — include AI usage in your story/ticketing system.
- Define parameters for usage. Will it write documentation? Write tests? Write functions? Classes? Whole features? Will it review code? How are you making sure everyone understands the new code (i.e. you don’t want devs committing code they don’t understand or reviewers passing it)
- How will you reign in costs? I had 3 devs spend over $1500 (each) in thirty days and am in the process of a explaining this to my COO. Purchasing plans instead of using API keys gives you some natural boundaries for reigning in costs.
- Create standard processes. Don’t vibe. Have a standard for how your AI-facing documentation is structured. Have standards/templates for how prompts are structured. Have standard prompts with limited scope for specific tasks. For example have a prompt just for review that specifies to review from the standpoint of security and best practices.
Those are all very helpful points, especially the last one; I’ve always tried to include standardization in the company but was never in the position to enforce it before
Assuming a dev company, first thing I would do is let the manegement know, in writing, that building a dependance on AI into your processes will make you depend on a tool that is currently heavily under-priced, with operating costs for all AI firms resulting in hundreds of millions of losses, that they will want to recuperate eventually.
Link articles about firms blowing the yearly budget on per-token price in a single month. Companies spending 500mil per month on tokens woth uncapped usage.
Also explain that allowing your devs to use it will lead to erpsion of their skill, building a dependency, that will then tie their productivity going forward to the whims of AI companies, that are currently operating at heavy losses.
Link any of the numerous papers how AI erodes skill of devs.
Ask them if they really want to build a dependency into this ecosystem, and if they are willing to pay the heavily increased price, once it inevitably gets expensive. That you can start replacing devs with AI, but even now, at the heavily undercut prices, it’s already costing more than a developer would, and it will only get worse. There are articles for that too. Make sure to highlight that you will be giving control of your systems and development to something that will be expensive, and it will be expensive to walk back on those decisions, once it’s integrated deep enough.
Get a paper trail. Make them sign off on the risks. Don’t do anything unless they sign off on the risks, and make it super clear how stupid idea it is. Bring as many recipes, papers, and blogs you can find.
Make a citation for how much it would cost to run a competitive local models. How much you would have to spend on GPUs to have a local solution for all of your planned AI need, so you can continue working once third party AI is not affordable.
Also take a look at all currently unresolved GDPR and general laws around copyright, unsolved cases. Send them blog posts that imply that there might be a copyright problem with AI generated outputs down the line, once all lawsuits resolve.
Mention the google lawsuit in germany, where Google might be legally responsible for the outputs it’s AI gives. Ask if that is a risk they accept, assuming it passes.
It’s an extremely stupid idea to currently even consider being dependant on AI. Give the recipes and arguments, and have them directly sign off on every risk you can think off. Basically - whatever you do, do a proper long term risk analysis, and make sure the management signs off on the risks.
You will need it down the line. There is a lot of risks.
Figure out what the executive’s day-to-day workflow is like, then make agents to do their job. Either they’ll get an understanding of how inadequate it is, or they’ll be able to just sit around doing nothing useful (assuming that’s not what they do already).
May want to consider setting up a private, on-prem system. That way, you can reliably enforce privacy/GDPR rules. You can also tweak the system to support local training, RAG, MCPs, etc.
This way, the costs can also be controlled. It’s some capital investment in local hardware, plus reasonably fixed power/cooling/maintenaance ongoing expenses.
Another way is to use the major AI services for planning/brainstorming specific features, tell it not to implement or touch anything, but to generate a detailed plan for an implementer LLM. Review that plan manually, and when ready, feed it to your local system for implementation and debugging.
This doesn’t work if the goal is one-shot vibe-coding. But it works really well for focused feature enhancements, test coverage, and bugfixing.
You can have it integrated with grafana, and it can quickly search logs based on errors or alerts and link them to specific commits quite quickly. My previous company did that a year ago.
We don’t even know what your company does? Does it do development? Data analysis?
Without knowing that, as head of AI, I’d first take training to understand what AI is and get advice from an external company as to what it could be good for. “AI” is being used as an umbrella term nowadays for neural networks to stochastic machines and they all have different uses. Understanding valid and invalid usecases is important before deciding when and where to use it in the company.
First, my condolences.
Second, I think something you can do as “head of AI” is push back on the benchmarks your execs are expecting you to measure. They got them straight from whoever your AI vendor is, and if your executive team is even halfway competent they should understand “these metrics are designed by the vendor to make us spend more”, but a lot of exec teams won’t listen. Still, you should institute your own benchmarks around code quality and delivery speed and talk about them with the exec team even if you have to shoehorn them into the discussion.
The next and probably more important thing that comes to mind is managing how your devs use their new tools. They’ll be able to churn out more lines of code than ever before and “complete” some features much more quickly. Your metrics should not incentivize this if your goal is code quality and stability. I don’t really have much in the way of solutions (other than “manage expectations” and “set the narrative”), because that’s about where in my “head of AI” career I got laid off. Once the numbers came in that we only needed 30% of our current staff they crunched the numbers and I was in the 70%.
Good luck. You’re starting from a deficit because your management team probably already has entertained the thought of trimming some expensive devs from the payroll, and that’s a tough thing to argue against.
We’re honestly already running on a dev deficit, so for the foreseeable future I don’t think they’re in any danger. Interestingly, management seemed wary of vibe coding and letting the ai write large swaths of code used in production, which I’d say is a generally safe stance considering the obvious stability issues that have become frequent lately. As for the devs themselves, they’re seemingly not interested in really playing with the ai to begin with, but I somewhat expected that as I have different issues with them like “please update your code to GitHub the main branch hasn’t been updated in months” and “don’t commit pem keys to git”
Nah, show them that it’s an expensive waste of time by continuing to let them run wild with useless but impressive tasks
It depends on your company.
If you are running public-facing code, security is going to become a huge deal in coming months. So use AI to find all the security holes in your stuff before the hackers do.
If you have lots of databases and APIs for internal use, create a MCP server so agents can do anything with your data and APIs.
If your data is even a little sensitive and you don’t want to send it all to the USA (read up on the CLOUD act), look into running models locally and the hardware investments you’ll need to make.
I hadn’t considered making an MCP server for our stuff, it feels obvious in retrospect, thanks! As for the self hosting I’d definitely love to but convincing a small company to shell out the kind of money to run something strong enough locally is something I don’t think is going to be likely, even if it’d cost less in the long run.
convincing a small company to shell out the kind of money to run something strong enough locally is something I don’t think is going to be likely, even if it’d cost less in the long run.
It’s your job to do the cost analysis. Then you present your findings and the execs decide whether to lay the up-front cost, depending on how much benefit they see it providing based on your assessment of the potential applications to the business.
Look into private mode ai by edgeless systems. Affordable and private.
Damn, I’m in the same boat. I didn’t attend one meeting because ut was too late in My timezone and the others chose this moment to vote me to become the lead of the AI acceleration initiative where we have half a year to come up with and implement things on each step from requirements, through architecture and development to testing on all levels.
And yeah we better come up with something useful because I already know they have to reduce the work force because there is no money and the people who will be left can’t do all of it alone without help.
I’m in like 3 AI design forums for some reason but whenever I want to start measuring productivity gains I’m ignored so try that for the least amount of effort required.
Lol alright, I’ll definitely keep that in mind.
Pay me 25% and I’ll advise you.
