Create module parser

[nappex]

Očekáváme předem známa vstupní data stažená downloaderem. Donwloader stahne textovy obsah stranky.
data tvořící text budou ve formě co řadek to jedna url (jako kdysi napč. gopher).
Vystupem parseru bude json soubor ktery bude obsahovat seznam vsech url nalezenutych v textu dané stranky.

[nappex]

dalsi problemy

Jake formaty chceme nacitat.
Jake vystupni formaty chceme.

[Glutexo]

#118 slightly blocks this because, without the rename, we’d end up with a module called Onigumo that is essentially a Downloader and then a new Parser module. Or even worse, to fit into the naming and architecture, we’d stuff everything into the existing Onigumo module.

[Glutexo]

#118 got merged; I am removing the “blocked” label.

[Glutexo]

To answer your questions, @nappex:

The input of the parser will be anything downloaded from the Internet. Typically, this is an HTML page, but it can be a plain text file or a JSON API response. It is Spider’s job to know and discern the format; Onigumo can only provide a set of tools (like #87) to make everyday tasks easier.

The output format will be a structured representation of the input data. JSON was my first pick for its universality, and I preferred it over YAML because the data will be processed only by a machine. The output can be, however, whatever kind of structured data, be it a serialized object of a programming language of the Spider, an XML document, or some binary format.

Theoretically, Onigumo doesn’t even need to force a concrete structure format. Having one would make it easy to define specific standardized nodes in the schema, allowing to control the core without writing any code. On the other hand, a piece of Spider code that knows its file format can provide the same data just as fine.

Feel free to share any thoughts on the topic!

[Glutexo]

The work has begun: #146

[Glutexo]

I extracted the steps described in your description (#89 (comment) and #89 (comment)), @nappex, to a new follow-up Issue #169. Feel free to take a look!