An AI agent that makes your to-do list phone calls

How the phone call agent works

Save each call as an ordinary note that starts with "Call":

Once a day, your agent asks Memory Notes for new notes starting with "Call" and chooses which ones it can handle. It makes those calls with an AI voice, then saves what happened as a new note in Memory Notes. You can find the results alongside your other notes.

What you need to build the agent

• Memory Notes stores the to-do. The agent uses list_memories to read it and create_memory to save the outcome. It needs a read & write token.
• An LLM such as Claude, GPT or another model makes a call plan from the note, including who to call, the number, what to ask and what counts as done.
• Twilio places the call. The talking is handled by Twilio's ConversationRelay or a realtime voice model connected to a Twilio Media Stream.
• A scheduler runs the agent. You can use a cron job, a GitHub Action or a scheduled Claude Code routine.

A TypeScript sketch for the call agent

This sketch is an outline rather than a finished product. It reads the past day's notes, finds calls and saves a result note. The calling step is where you add Twilio.

import { Client, StreamableHTTPClientTransport } from "@modelcontextprotocol/client";

const notes = new Client({ name: "call-agent", version: "1.0.0" });
await notes.connect(
  new StreamableHTTPClientTransport(new URL("https://memorynotes.emodino.com/api/mcp"), {
    requestInit: {
      headers: { Authorization: `Bearer ${process.env.MEMORY_NOTES_TOKEN}` },
    },
  }),
);

const since = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString();
const { structuredContent } = await notes.callTool({
  name: "list_memories",
  arguments: { since, limit: 100 },
});

const calls = structuredContent.memories.filter((m) =>
  m.content.trim().toLowerCase().startsWith("call "),
);

for (const note of calls) {
  const plan = await planCall(note.content); // your LLM: number, goal, script
  if (!plan.number) continue; // no number? leave it for a human
  const outcome = await placeCall(plan); // Twilio + a voice model
  await notes.callTool({
    name: "create_memory",
    arguments: { content: `Called ${plan.who}: ${outcome.summary}` },
  });
}

Take care with automated phone calls

• Begin with a dry run that saves only the plan and makes no calls, such as "I'd call 555-0142 and ask to move the cleaning to the 21st." Review a week of plans before letting the agent dial.
• Have the voice say at the start that it's an AI assistant calling for you. Many places require this, and recording a call often requires everyone's consent.
• Use it for errands such as bookings, questions and rescheduling. Handle calls involving money, health decisions or arguments yourself.
• An agent can't edit or delete Memory Notes notes. It can keep track of handled notes in its own store or check for its "Called ..." notes.