README.md

Web Crawler for LLM Apps

Flexible and Scalable low-code Web Crawler. Give it a try on the LIVE playground: https://llm-web-crawler.vercel.app/

Uses BuildShip - a visual AI workflow builder to extract and gather data from your websites or sources. This data can then be used as a knowledge base to power your own LLM apps 🤖, or paired with BuildShip's AI Assistant to unlock powerful use cases and enhance your business or services.

Video Tutorial

Features

Node	Info	Documentation	Template
Scrape	Easy to get started with, scrape a given web URL and return the text content. Works great for less complex sites that don't rely on JavaScript to load.	Read more	Remix
Dynamic Scrape	Scrape a given web URL and return the text content. This method works well for more complex sites and allows for more interactive scraping by providing a set of steps to execute after loading the page. For example, loading an ecommerce site, searching for an item, and then scraping the search results info.	Read more	Remix
Web Crawler	Extract data from an entire website by crawling through and scraping all its pages. Perfect for aggregating data to create your own custom GPTs or "Chat with Data" apps.	Read more	Remix
LLM Extraction	Extract structured data (just the data you care about) from any website. No need to scrape an entire webpage; simply specify the URL and the fields you want to extract. The LLM will handle the rest, delivering only the relevant data in a structured format.	Read more	Remix

How to use

• First clone a template using most relevant to your using the following links. LLM Extractor
  Crawler
  Static Web Scraping
  Dynamic Web Scraping

• Run or customize the template as per your usecase

• Click Ship to deploy as an API or scheduled job

Read full documentation to learn more