tanghoongAug 28月2日
Sync or async for an AI support chatbot
If customer A is waiting for the LLM, does customer B have to wait too? The answer is an architecture decision, and it matters long before you have a thousand customers.
@charlie
Levelling up in public — notes from building AI systems solo.公開練功——一個人做 AI 系統的筆記。
If customer A is waiting for the LLM, does customer B have to wait too? The answer is an architecture decision, and it matters long before you have a thousand customers.
Producing a convincing answer is no longer the hard part. The hard part is the agent still working tomorrow — and that turns out to be a state problem, not a memory problem.
No database, no CMS, no admin panel, no server. Not because those things are bad, but because every one of them is a cost I would keep paying long after the novelty of setting it up wore off.