First Idea for "Connectors": Website Scrape

[Pipboyguy]

A common theme, and arguably table stakes, among RAG software like llm-app seems to be the concept of connectors. Here are some examples:

• danswer's connectors
• khoj's connectors

If building connectors are in scope for this project, I'd like to put in a request for recursive web crawling similar to Langchain's recursive url loader or this one

[mdmalhou]

Thank you for bringing this up. To address your query, building connectors is definitely within the scope of llm-app. However, it's worth noting that the connectors in pathway are architecturally and conceptually distinct from those you mentioned in danswer and khoj.

To better understand our approach and to create a connector for pathway, I recommend going through our tutorial here: Custom Python Connectors. This should provide a clear overview and guide on our unique take on connectors.

The idea of a recursive web crawling feature sounds intriguing, we'll certainly consider it.

[Pipboyguy]

Thanks @mdmalhou ! I'm delighted to hear that it is in scope