Newscorpus 📰🐍

Takes a list of RSS feeds, downloads found articles, processes them and stores the result in a SQLite database.

This project uses Trafilatura to extract text from HTML pages and feedparser to parse RSS feeds.

Installation

This project uses Poetry to manage dependencies. Make sure you have it installed.

Via Poetry

poetry add "git+https://github.com/gambolputty/newscorpus.git"

Via clone

# Clone this repository
git clone git@github.com:gambolputty/newscorpus.git

# Install dependencies with poetry
cd newscorpus
poetry install

Configuration

Copy the example sources file and edit it to your liking.

cp sources.example.json sources.json

It is expected to be in the following format:

[
  {
    "id": 0,
    "name": "Example",
    "url": "https://example.com/rss",
  },
  ...
]

Usage

Starting the scraper (CLI)

To start the scraping process run:

poetry run scrape [OPTIONS]

Options (optional)

Option	Default	Description
--src-path	sources.json	Path to a sources.json-file.
--db-path	newscorpus.db	Path to the SQLite database to use.
--debug	none (flag)	Show debug information.
--workers	4	Number of download workers.
--keep	2	Don't save articles older than n days.
--min-length	350	Don't process articles with a text length smaller than x characters.
--help	none (flag)	Show help menu.

Accessing the database

Access the database within your Python script:

from newscorpus.database import Database

db = Database()

for article in db.iter_articles():
    print(article.title)
    print(article.published_at)
    print(article.text)
    print()

Arguments to iter_articles() are the same as for rows_where()in sqlite-utils (Docs, Reference).

The Database class takes an optional path argument to specify the path to the database file.

Acknowledgements

• IFG-Ticker for some source

License

GNU AFFERO GENERAL PUBLIC LICENSE