Releases: infinilabs/crawler
Releases · infinilabs/crawler
v0.12.0_SNAPSHOT
v0.12
breaking changes
- pipeline config moved to dedicated section, can be reference by id
- elasticsearch config moved to dedicated section, can be reference by id
- remove indexer module, nested to elastic module
features
- add parse_pdf joint to parse PDF files
- support major elasticsearch version, auto handle API differences
improvement
- auto clone and update framework/vendor repo
- move vendor out of this repo
bugfix
- fix relative link was not proper resolved with https
- fix redirected link not handled exception
v0.11.0
v0.11
Adios Sqlite, Hola Elasticsearch!
breaking changes
- extract common codebase to another repo: https://github.com/infinitbyte/framework
- sqlite retired, elasticsearch is the first citizen
features
- add a new cmd
static_fs
to support load static files from folder - auto generate elasticsearch mapping and template, no need to manual create mapping first
- add new
backup
command to support migration
v0.10.0
Only support SQLite as database .
v0.10
breaking changes
- refactor domain to host, api and mapping has changed
- refactor module, update yml settings: module->name
features
- dynamic create pipelines
- init plugin architecture
- support extract tags by css path
- add chrome fetch joint, via chrome debug protocol
- add auto-completion to search ui
- search ui support mobile
- support access control by github oauth
improvement
- remove goleveldb due to memory leak
- update logo
- remove hard coded version
- update task UI, support filter by status and host
- clean offset_canvas menu