> ## Content Index
> Fetch the complete content index at: https://blog.getzep.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Building Voice Agents with Memory: Zep x LiveKit
- URL: https://blog.getzep.com/zep-livekit/
- Published: 2025-11-20T16:22:22.000Z
- Updated: 2026-06-03T20:34:40.000Z
- Description: Create personalized voice agents with long-term memory with minimal added latency
- Author: Jack Ryan

# Building Personalized Voice Agents with Zep and LiveKit

The Zep/LiveKit integration combines Zep's context engineering platform with LiveKit's voice agent framework. With the integration, you can build personalized and reliable voice agents that remember user preferences and conversation history across sessions with low latency.

**⚡ Concerned memory will slow your agent down?** Zep delivers P95 retrieval latency under 250ms, ensuring your voice agents maintain real-time responsiveness while accessing rich contextual memory. 

A complete code example is available on [GitHub](https://github.com/getzep/zep/tree/main/integrations/python/zep%5Flivekit/examples/full-example?ref=blog.getzep.com) to get you started quickly.

The integration provides a pre-built agent design with Zep implemented for you, making it simple to immediately make your agents more personalized.

---

## How It Works

The integration works by providing a predefined voice agent - `ZepUserAgent` \- that has Zep already implemented for you as its long-term memory store.

The `ZepUserAgent` is a normal voice agent, but it:

- Stores all messages in Zep's knowledge graph
- Retrieves relevant context in the form of Zep's Context Block before each response
- Updates the system prompt with the Context Block on each turn to improve quality of AI response

Under the hood, `ZepUserAgent` uses Zep's `thread.add_messages()` and `thread.get_user_context()` methods to add messages to the user's graph and to retrieve Zep's Context Block, respectively:

```python
# The agent uses this Zep method to add messages to the user's graph (and the given thread)
response = zep_client.thread.add_messages(thread_id, messages=messages)

# The agent uses this Zep method to get the Zep Context Block for the user
memory = zep_client.thread.get_user_context(thread_id=thread_id, mode="basic")
context_block = memory.context

```

The `thread.get_user_context()` uses the previous few messages from the provided thread as a search query on the user's knowledge graph; the results of this knowledge graph search make up the Context Block.

The Context Block (when `mode="basic"`) looks like this:

```
FACTS, ENTITIES, and EPISODES represent relevant context to the current conversation.
# These are the most relevant facts and their valid date ranges
<FACTS>
    - John's favorite song is "Viva La Vida" by Coldplay (valid: 2024-01-15 to present)
    - User prefers morning meetings over afternoon ones (valid: 2024-02-01 to present)
</FACTS>
# These are the most relevant entities  
<ENTITIES>
    - John: Software engineer, works at tech startup, enjoys indie rock music
    - Spotify: Music streaming service user frequently mentions
</ENTITIES>
# These are the most relevant episodes
<EPISODES>
    - "My favorite song is Viva la Vida by Coldplay. And Coldplay is my favorite band"
    - "Can you make me a playlist of indie rock songs?"
</EPISODES>

```

This block is dynamically inserted into the system prompt before the agent responds so the agent response can take this context into account.

## Using the Integration

Implementation requires three steps:

### Step 1: Creating Zep User and Thread

Set up the user and thread in Zep:

```python
import os
import uuid
from zep_cloud.client import AsyncZep

# Initialize Zep client
zep_client = AsyncZep(api_key=os.getenv("ZEP_API_KEY"))

# User constants
USER_ID = "John-1234"
THREAD_ID = f"conversation-{uuid.uuid4().hex[:8]}"
USER_FIRST_NAME = "John"
USER_LAST_NAME = "Doe"
USER_EMAIL = "john.doe@example.com"

# Create or get user
try:
    await zep_client.user.get(user_id=USER_ID)
except Exception:
    await zep_client.user.add(
        user_id=USER_ID,
        first_name=USER_FIRST_NAME,
        last_name=USER_LAST_NAME,
        email=USER_EMAIL,
    )

# Create new thread for this session
await zep_client.thread.create(
    thread_id=THREAD_ID,
    user_id=USER_ID
)

```

### Step 2: Setting up LiveKit Session/Room

Create the LiveKit session:

```python
from livekit import agents
from livekit.plugins import openai, silero

async def entrypoint(ctx: agents.JobContext):
    # Connect to LiveKit room
    await ctx.connect()
    
    # Create agent session with OpenAI components
    session = agents.AgentSession(
        stt=openai.STT(),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=openai.TTS(voice="alloy"),
        vad=silero.VAD.load(),
    )

```

### Step 3: Defining Voice Agent with Memory

Initialize the agent:

```python
from zep_livekit import ZepUserAgent

# Create the memory-enabled agent
agent = ZepUserAgent(
    zep_client=zep_client,
    user_id=USER_ID,
    thread_id=THREAD_ID,
    user_message_name=USER_FIRST_NAME,
    assistant_message_name="Assistant",
    instructions="You are a helpful assistant who responds concisely in at most 1 sentence for each response."
)

# Start the session
await session.start(agent=agent, room=ctx.room)

```

The `ZepUserAgent` handles memory operations automatically.

### Environment Setup

Make sure to set these API keys:

```bash
OPENAI_API_KEY=your_openai_api_key_here
ZEP_API_KEY=your_zep_cloud_api_key_here
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key_here
LIVEKIT_API_SECRET=your_livekit_api_secret_here

```

### Running the Voice Agent

Once configured, run:

```bash
python zep_voice_agent.py dev

```

And then test the agent on the [LiveKit Agents Playground](https://agents-playground.livekit.io/?ref=blog.getzep.com).

## Key Features

The integration provides memory management with minimal configuration:

- Automatic conversation storage
- Context-aware response generation
- Cross-session conversation continuity
- User identity and preference tracking

## Use Cases

The integration enables voice applications that require persistent context:

- **Personal Assistants**: Maintaining user preferences and interaction history
- **Customer Service**: Access to complete conversation history across sessions
- **Educational Tools**: Tracking learning progress and adapting to individual needs
- **Healthcare Applications**: Preserving patient interaction history and preferences

## Extra: Add External User Data

Add additional user data in the form of JSON or unstructured text to a user's knowledge graph using `graph.add()`:

```python
import json
from zep_cloud.client import Zep

client = Zep(api_key=API_KEY)

# Add user profile data
profile_data = {
    "favorite_music_genre": "indie rock",
    "preferred_meeting_times": ["9:00 AM", "2:00 PM"],
    "dietary_restrictions": ["vegetarian"],
    "location": "San Francisco",
    "job_title": "Software Engineer"
}

json_string = json.dumps(profile_data)

client.graph.add(
    user_id=USER_ID,
    type="json",
    data=json_string,
)

```

This data is stored in the knowledge graph and retrieved as context when relevant, alongside the data sent to the graph from user-agent conversations. If the JSON and conversation messages ever refer to the same entity, those entities are automatically de-duplicated in the graph.

## Extra: More Customizable Memory

Use `ZepGraphAgent` for more control over memory retrieval and search parameters:

```python
from zep_livekit import ZepGraphAgent
from zep_cloud import SearchFilters

# Configure search filters for specific entity types
search_filters = SearchFilters(
    node_labels=["Restaurant", "Location"],
    edge_labels=["VISITED", "TRAVELED_TO"],
)

# Create graph agent with full customization
agent = ZepGraphAgent(
    zep_client=zep_client,
    graph_id="custom-graph-id",
    user_name=USER_FIRST_NAME,
    facts_limit=15,
    entity_limit=10,
    episode_limit=8,
    search_filters=search_filters,
    reranker="rrf",
    instructions="Your custom instructions here..."
)

```

`ZepGraphAgent` allows customization of context limits and filtering by custom entity/edge types. Behind the scenes these parameters are passed to Zep's `graph.search` method ([docs](https://help.getzep.com/searching-the-graph?ref=blog.getzep.com)).

## Getting Started

The integration requires API keys for OpenAI, Zep Cloud, and LiveKit. Once configured, the agent handles memory management automatically while you focus on building voice interaction logic.

For complete documentation, setup instructions, and advanced configuration options, see the complete working example on [Github](https://github.com/getzep/zep/tree/main/integrations/python/zep%5Flivekit/examples/full-example?ref=blog.getzep.com) and the [Zep LiveKit Integration documentation](https://help.getzep.com/ecosystem/livekit-memory?ref=blog.getzep.com).