🤖 Extension for TYPO3 CMS to inject XML sitemaps into robots.txt

Laravel robots.txt package

The source code for Delete the Matrix blog - Exploiting, Experimenting, and Exploring the Universe

🚀 This Astro template offers more than 'Just the Basics', providing a superior option for starting your next project wit best practices and a set of essential integrations already built-in.

Tame the robots crawling and indexing your Nuxt site.

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

advertools - online marketing productivity and analysis tools

Website of Alp Ozkan: Product Manager living in Istanbul, passionate about UX, Software Development and Web3.

Advanced SEO Toolkit is a comprehensive solution for optimizing your WordPress site for search engines

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.

A collection of Docker images: robotstxt, linuxbrew, gcloud, and psql

🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

.Net Core Plugin Manager, extend web applications using plugin technology enabling true SOLID and DRY principles when developing applications

WP Robots txt Editor est un plugin WordPress idéal pour modifier le fichier robots.txt à partir d'une simple page d'option. Générez un robots.txt par défaut et accédez à de nombreuses fonctionnalités comme le choix des publications, catégories, pages parents/enfants, et plus encore..

Library for robots.txt files in Rust

An app to practice SEO associated with SSG, ISR, SSR and other featues like dynamic 'Metadata', 'Robots', 'Sitemap', cache, 'Favicon', 'Open Graph' and 'Social Preview' in NextJS14.

A small, tested, no-frills parser of robots.txt files in Swift.

A set of reusable Java components that implement functionality common to any web crawler

RAG basado en la plataforma Streamlit, que utiliza la API de OpenAI y LangChain para generar respuestas contextuales basadas en una base de datos de informes y conocimientos vectorizada.