An AI Engineers Guide: Web Browsing and Scraping Tools for AI Apps and Agents

This is a Build Club open-source community document, where we share insights and learnings from builders at the frontier of AI! If you are looking to start your AI journey or level up to be in the top x%, then come Build with us!

👉️ You can join our OS community here

An intro to web browsing

Web browsing is essential for AI engineers, enabling data collection for machine learning, market analysis, and more.

As web environments grow more complex, sophisticated scraping techniques are crucial. AI-powered automation, scalable solutions, and ethical considerations are transforming web browsing.

Web scraping typically involves three steps:

1. Exploring web elements to identify the content structure

2. Accessing and fetching data with automated scripts in languages

3. Storing the extracted data

This article focuses on the first two which have evolved the most with the rise of new AI capabilities 🌟

Here's a look at the latest trends, key tools, and best practices.

Trends in web browsing

1. AI-Powered Automation AI models like GPT-4 are revolutionizing web browsing by dynamically generating and refining scripts. This adaptability makes scraping more efficient and capable of handling dynamic content with ease.

2. Scaling and Anti-Blocking Techniques Scaling operations face challenges like IP blocking. Tools like Apify and Fire Crawl offer proxy rotation and user agent switching to avoid detection. They also bypass CAPTCHAs, ensuring smooth, large-scale data scraping.

3. Community-Driven Innovations Platforms like Hacker News and Reddit are hotbeds for new scraping techniques. Engineers share tips on using AI for CAPTCHA solving and dynamic content scraping, with tools like Puppeteer often in the spotlight.

4. Ethical and Legal Considerations As scraping tools evolve, so do the ethical and legal implications. Compliance with robots.txt, respecting privacy, and avoiding undue server strain are increasingly important. Tools that automatically follow these guidelines, such as ZenRows, are becoming essential. As noted by Andy Taylor, "scraping is never illegal; however, what you do with the data can be."

The newest and hottest tools top AI engineers are using

We asked our community what the top tools they are using…

1. Beautiful Soup: Beautiful Soup emerged as one of the most popular tools discussed among the participants. This Python library is known for its simplicity and effectiveness in parsing HTML and XML documents, making it a go-to for many developers. Examples include scraping job descriptions to match them with candidates' CVs, automating a process that would otherwise be highly manual. Another benefit to Beautiful Soup is community support and examples available online, which made troubleshooting and implementation much more manageable.

2. Web Scraper.io and Firecrawl: For more extensive web browsing projects, especially those involving large-scale data collection, commercial tools like Web Scraper.io and Firecrawl offer advanced functionalities. This platform allows users to automate data extraction across multiple pages by setting up custom rules. It is particularly effective for scraping content from hundreds of financial websites or monitoring market trends. Engineers found Web Scraper.io to be reliable, especially when dealing with tasks requiring significant data storage and indexing.

3. Using Large Language Models (LLMs) for Scraping: With the advent of Large Language Models (LLMs) like ChatGPT-4, there is growing interest in leveraging these models to enhance web browsing efforts. For instance, developers have explored using LLMs to interpret and extract meaningful insights from scraped HTML content. However, challenges remain, such as the models misinterpreting HTML tags or shifting from extraction to content generation modes. Despite these limitations, LLMs can be effective in specific scenarios, such as performing Q&A on extracted web data.

Full market map of rising tools

Here is the full list of tools top AI engineers are using and interested in experimenting with 👇️ 

The breakdown:

• Browser Automation Tools

  • Selenium: Versatile and widely used for dynamic content scraping, integrating well with AI-driven workflows.

  • Puppeteer: A Node.js library, excellent for scraping dynamic, modern web apps.

  • Puppeteer-stealth: A stealth plugin widely used to reduce detection when self-hosting scraping infrastructure.

  • Playwright: Use different browser bases for rotation with user-agent list to work around user-agent checking.

• Headless Browser Platforms

  • Browserbase: Headless browser as a service that mimics human behavior, reducing detection risk—an affordable option for stealth and proxies.

  • Browserless: Another headless browser platform with excellent stealth and proxy features, though at a higher cost.

• Proxy and Anti-Detection Tools

  • BrightData: Offers high-quality residential proxies and the SEPR Web Unlocker, a product that is expensive but highly effective at bypassing detection.

  • ZenRows: High-quality, though expensive, tools for bypassing detection and paywalls, especially useful for scraping news sites.

  • Apify/Crawlee: Scalable and easy to use, with built-in proxy management, ideal for both small and large-scale tasks.

  • Fire Crawl: User-friendly, handles large-scale scraping, and integrates seamlessly with AI workflows.

  • Serverless (preferably containers): Cloud Run, AWS Lambda, Digital Ocean, etc… Rotate API calling to ensure each container is running on a new IP address.

• Scalable Web Browsing Platforms

  • Octoparse: A no-code, cloud-based platform that is beginner-friendly yet capable of handling complex, scalable tasks.

  • JigsawStackr: Native AI web scraper with built-in proxies to scrape any website accurately and provides structured data along with CSS selector data.

• AI-Powered Scraping Solutions

  • Jina's Reader API: An AI-driven tool that provides rich data extraction for AI models.

  • TaxyAI Browser Extension: Powered by GPT-4, this extension is great for lightweight, in-browser scraping tasks.

  • ScrapeGraphAI: A web browsing Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown).

  • AgentQL: A powerful AI-driven tool for reliable web data extraction and automation, using natural language queries to find elements without fragile selectors.

• Open Source and Custom Solutions

  • Universal HTML Parser (Grub Search): Open-source, versatile tool ideal for custom scraping needs.

  • Any LLM: Clean HTML Body first with Beautiful Soup, then call LLM to generate Beautiful Soup script via tool calls. Run that script to try extraction first; then fall back to Goose, Newspaper, or other old-school extraction tools.

  • Using LangChain Community AsyncChromiumLoader and Html2TextTransformer: Tools for extracting text content from websites using asynchronous browser loading and HTML-to-text conversion. Code:

import html2textfrom langchain_community.document_loaders import AsyncChromiumLoaderfrom langchain_community.document_transformers import Html2TextTransformerdef extract_text_from_url(self, url):    loader = AsyncChromiumLoader([url])    html_content = loader.load()    html2text_transformer = Html2TextTransformer()    docs_transformed = html2text_transformer.transform_documents(html_content)    content_text = []    for doc in docs_transformed:        page_content = html2text.html2text(doc.page_content)        content_text.append(page_content)    return content_text

• Web Scraping Programming Language Libraries

  • Beautiful Soup: A Python library for parsing HTML and XML documents, widely used for small-scale web scraping tasks.

  • Scrapy: A Python-based open-source web scraping framework designed for handling large-scale data extraction.

  • Requests: A simple and elegant HTTP library for Python, often used in conjunction with other scraping tools.

  • lxml: A powerful and feature-rich library for processing XML and HTML in Python.

  • Jsoup: A Java library for working with real-world HTML, providing a fast and flexible way to extract and manipulate data.

  • Pandas: A Python library used for data manipulation and analysis, often employed to process and store scraped data.

Conclusion

There are so many new tools, and new AI powered capabilities on the market with browsing - meaning new capabilities are unlocked to deliver that sweet business value.

Web browsing remains a powerful tool, but it requires careful attention to best practices, particularly regarding legality and ethics. Always check a website's robots.txt file before scraping, respect copyright laws, and avoid putting undue strain on servers.

Additionally, leveraging community insights from platforms like Hacker News and Reddit can provide valuable tips on optimizing your scraping strategy.

Web browsing is evolving rapidly with AI, scalable solutions, and ethical considerations at the forefront. Whether you need robust, scalable tools like Apify or AI-powered solutions like Browserbase, the right tool depends on your specific needs. Stay updated on these trends and choose tools that best align with your scraping goals.

Curious to continue the discussion, join us in our Slack here!

Appendix

Full wiki of tools mentioned: