July 16, 2025 · Yevhenii Hordiienko
Task-Doing Agents in Real Time
How a voice bot can pick up the phone, chat like a human, and drop a meeting on everyone's calendar before you hang up.
TL;DR
A voice bot running on Twilio Media Streams + ASP.NET Core + OpenAI GPT-4o Realtime can:
- Talk naturally with sub-second latency,
- Answer company-specific questions on the fly by pinging Azure AI Search's hybrid (vector + keyword) index, and
- Book a calendar meeting before the caller hangs up—all for only a few cents per call.
The result: every inbound call becomes a fully handled support-or-sales conversation, with zero human routing time.
1. Why this matters
Remember the old "Press-2-for-sales" menu? The caller still ends up waiting while a rep checks FAQs, types in a CRM, and fires off a meeting invite. With today's real-time AI stack we can let software do the whole loop:
| Old flow | New flow |
|---|---|
| Phone tree → hold music → human answers → looks up info → sends invitation | Caller talks → AI answers quickly (FAQ pull from Azure AI Search) → AI books meeting → confirmation read back |
Total time saved per call: 3–5 minutes for the caller and your team.
2. How the bot handles a call
Caller: "Hi, what's the difference between your Standard and Enterprise tiers? Also, can I talk to Alex tomorrow afternoon?"
AI: "Enterprise adds SSO, 99.9% SLA, and priority support. I can set up 30 minutes with Alex—does 3 p.m. ET work?"
Caller: "Perfect."
AI: "Done—you'll get an invite in a few seconds. Anything else?"
3. What just happened behind the curtain?
- Live audio from Twilio reaches the bot.
- GPT-4o plans the reply, and—seeing a company question—calls a function.
- The bot queries Azure AI Search (hybrid vector + keyword) for up-to-date pricing info, passes results back to GPT-4o, and it drafts the spoken answer.
- GPT-4o then emits a
schedule_meetingtool call with time and emails. - The bot hits the Google Calendar API, grabs the confirmation link, and the model speaks it back.
4. Why it's safe to try now
- Proven pipes. Twilio's Voice webhooks and
<Gather>verb already run millions of IVR calls a day. We're just swapping canned prompts for AI streaming. (Twilio) - First-party tutorials. Twilio published sample repos that pair GPT-4o Realtime with Voice.
- Calendar APIs are boring (in a good way). Whether Google or Microsoft, creating an event is a single REST call; OAuth scopes are well-trodden. (Twilio)
- The cost is predictable. You pay Twilio per minute plus OpenAI per generated second. A typical two-minute booking call lands at a few cents.
5. What your stakeholders get
| Role | Win |
|---|---|
| Sales | More qualified meetings booked automatically—no manual follow-up. |
| Support | Fewer call transfers; the bot handles scheduling itself. |
| Customers | No "press 5" menus, no waiting for email confirmation. |
6. Implementation checklist
- Buy a Twilio number and point its Voice URL to your webhook.
- Spin up a tiny service that:
- relays Twilio audio frames to the GPT-4o Realtime WebSocket
- implements the
schedule_meeting()function (Google Calendar SDK)
- Define tools in your OpenAI request (
schedule_meeting,lookup_customer, etc.). - Add guardrails—e.g., only accept business-hours slots.
- Pilot with internal staff—record calls, tweak prompts, go live.
7. Looking ahead
OpenAI's new Responses API lets the same agent search docs, run code, or file tickets mid-call; future releases promise tighter tracing and cost controls. Today, though, adding "book the meeting right now" is the fastest win—and the wow-moment your users will remember.
Bottom line: your phone line can now solve the scheduling chore in the time it used to take just to find the right extension. If you've got a Twilio account and a couple of API keys, you're a weekend hack away from delighting callers—and giving your team their afternoons back.