Skip to content

Commit

Permalink
chore: fmt
Browse files Browse the repository at this point in the history
  • Loading branch information
mijho committed Sep 30, 2023
1 parent e76bae3 commit c8af032
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 6 deletions.
15 changes: 10 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
# crawl-log2xml

## Description

Basic tools to:
- parse a Heritrix3 crawl.log
- extract URLs matching the specified status codes
- generate a sitmap.xml

The tool comes in two versions cli and server. The cli is likely the more reliable, the server method is probably limited to smaller crawl.log due to POST body limits.
- parse a Heritrix3 crawl.log
- extract URLs matching the specified status codes
- generate a sitmap.xml

The tool comes in two versions cli and server. The cli is likely the more
reliable, the server method is probably limited to smaller crawl.log due to POST
body limits.

## Usage

Expand All @@ -19,6 +23,7 @@ deno task run:server --port 8000 # run the server
```

### Example Queries

```
curl -XPOST --data-binary "@./test/small_crawl.log" "http://localhost:8000/convert?status=200"
```
```
2 changes: 1 addition & 1 deletion deno.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"dev": "deno run --allow-net --allow-read --watch src/server.ts",
"test": "deno test --allow-net --allow-read",
"test:watch": "deno test --allow-net --allow-read --watch",
"bench": "deno bench --allow-net --allow-read test/lib_bench.ts",
"bench": "deno bench --allow-net --allow-read",
"run:cli": "deno run --allow-net --allow-read --allow-write src/cli.ts",
"run:server": "deno run --allow-net --allow-read --watch src/server.ts",
"build:cli": "deno compile --allow-net --allow-read --allow-write --output ./dist/crawl-log2xml-ctl src/cli.ts",
Expand Down

0 comments on commit c8af032

Please sign in to comment.