Building Intelligent Web Research Agents with MCP and OpenAI

Manual web research is time-consuming and error-prone, especially for real estate and financial analysis. This project automates research by combining AI agents with the Model Context Protocol (MCP) to browse, search, extract, and save information. The system uses AI agents that can navigate sites, handle dynamic content, and synthesize results, reducing manual work and improving consistency. Built with Python 3.12+ and the OpenAI agents framework, it uses asynchronous patterns to manage multiple MCP servers. AsyncExitStack coordinates server connections as async context managers, ensuring proper initialization and cleanup. The system integrates three MCP servers: Playwright MCP for browser automation (handles JavaScript-heavy sites, cookies, and dynamic interactions), Brave Search MCP for web search via the Brave API, and a custom filesystem MCP server built with FastMCP that provides sandboxed file operations with path validation to prevent directory traversal. Agents are configured with GPT-4.1-mini and custom instructions. The custom files server uses pathlib.Path for path resolution and exposes async tools via @mcp.tool() decorators. Tasks are executed using Runner.run() with configurable max_turns limits, returning RunResult objects. The property research module demonstrates scraping domain.com.au and saving results as markdown, while the financial research module uses Brave Search for investment analysis. Both leverage the same MCP infrastructure, showing how modular server integration enables diverse research workflows with a unified architecture.