AI software places the output of a language model (or another model) inside a real business flow. In classic software the developer writes the rule; here the model produces part of it. That difference adds two responsibilities to the architecture: verifying the output and controlling the cost.
How does AI software work?
- Model: API models such as Claude, GPT or Gemini, or open models running on your own infrastructure.
- Context: the data the model sees while answering — product databases, documents, search results (RAG).
- Tools: functions the model can call — database queries, calendars, e-mail, payments, internal APIs.
- Evaluation: a golden dataset and regression checks that measure output quality. Skip this and the product degrades silently.
- Observability and cost: token usage, latency, error rate and cost per request.
What is used in practice? (2026)
- Model access: Claude, GPT and Gemini APIs; open-weight models where privacy or cost demands it.
- Vector search: pgvector (PostgreSQL), Qdrant or similar for document-based Q&A.
- Orchestration: frameworks such as LangGraph or a small agent loop you write yourself. A complex framework is not automatically better.
- Verification: golden datasets, automated evaluation, shadow testing, human approval.
- MCP: a standard contract for model access to tools, separating tool definitions from product code.
Types of AI software
- Chat assistant: in-product support, sales or internal knowledge assistant.
- Document Q&A (RAG): answers with citations over regulation, contracts or technical documentation.
- Content generation: drafts, summaries, translation, classification — always with editorial control.
- Task automation / agents: multi-step work with explicit permission and approval boundaries at each step.
- Predictive models: non-LLM classification/regression such as demand forecasting or risk scoring.
What to watch in 2026
- Hallucination and verification: a confident answer is not a correct one. In critical areas, citations and rule-based checks are mandatory.
- Cost: token cost grows with usage. Caching, routing to smaller models and context pruning decide product profitability.
- Data privacy: under KVKK/GDPR, personal data going to a model, being retained or logged must be designed separately.
- Prompt injection: external text (e-mail, documents, web pages) can act like an instruction; give tools the least privilege they need.
- Evaluation: regression tests on every model or prompt change. An unmeasured improvement is not an improvement.
- Model independence: a thin abstraction layer to switch providers — but over-abstraction has its own cost.
- Human approval: money, legal, health and customer communication should keep a human in the loop.
How are cost and timeline determined?
Scope follows the number of use cases, data sources, accuracy targets, steps requiring human approval and integrations. A prototype is quick; the real work is verification, error handling and cost optimisation. A sound plan starts with a small pilot scenario and expands on measured results.
Frequently asked questions
Does AI software always require a large language model?
No. For classification, forecasting or recommendations, classic machine-learning models can be cheaper and more predictable. Use a large language model where understanding and generating language is the actual job.
What is RAG and why use it?
RAG (retrieval-augmented generation) lets the model fetch relevant passages from your own data before answering. The answer is grounded in product data and can cite sources instead of relying on what the model memorised.
How do you reduce model cost?
Shortening context, caching repeated answers, routing simple tasks to smaller models and removing unnecessary calls are the most effective levers. Optimise only what you measure.
How long does AI software take?
A single-scenario pilot is quick; production quality needs verification, error handling, observability and integration layers. The timeline follows the accuracy target and the steps that require human approval.
