curl connection pooling ?

[gabrielsroka]

@theamk i updated the Python to use connection pooling, and it runs 2-3 times faster.

i found that curl can do that if you send multiple URLs, and it too will run about 3 times faster:

# 1 at a time
curl -s "https://hacker-news.firebaseio.com/v0/item/24021408.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24022751.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24021025.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24020245.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24022222.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24000145.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24020906.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24020263.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24020952.json"
curl -s "https://hacker-news.firebaseio.com/v0/item/24020254.json"

# 10 at a time
curl -s "https://hacker-news.firebaseio.com/v0/item/24021408.json" \
        "https://hacker-news.firebaseio.com/v0/item/24022751.json" \
        "https://hacker-news.firebaseio.com/v0/item/24021025.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020245.json" \
        "https://hacker-news.firebaseio.com/v0/item/24022222.json" \
        "https://hacker-news.firebaseio.com/v0/item/24000145.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020906.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020263.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020952.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020254.json"

if you pipe the 10 JSON objects to jq, it will parse them correctly

curl -s "https://hacker-news.firebaseio.com/v0/item/24021408.json" \
        "https://hacker-news.firebaseio.com/v0/item/24022751.json" \
        "https://hacker-news.firebaseio.com/v0/item/24021025.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020245.json" \
        "https://hacker-news.firebaseio.com/v0/item/24022222.json" \
        "https://hacker-news.firebaseio.com/v0/item/24000145.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020906.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020263.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020952.json" \
        "https://hacker-news.firebaseio.com/v0/item/24020254.json" | \
    jq -r '"\(.title)\n\(.url)\n\(.by)\n\(.score)"'

(how) can the rest of hn.sh be writtten to use this?

[gabrielsroka]

this seems to work ok, and it runs 2-3 times faster.

i don't know if this is "good", i'm sure there's a "better" way

#!/bin/bash

MAX=10
item_num=0
item_ids=$(curl -s 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty' | jq -r '.[]')
urls=""

for item_id in $item_ids; do
    item_num=$((item_num + 1))
    if [[ $item_num -gt $MAX ]]; then
        break
    fi
    urls="$urls https://hacker-news.firebaseio.com/v0/item/${item_id}.json"
done

readarray -t items < <(curl -s $urls | jq -r '"\(.title)\n\(.url)\n\(.id)\n\(.by)\n\(.score)"')

for ((i = 0; i < MAX; i++)); do
    j=$((i * 4))
    printf "%-3s %s | %s\n" "$((i + 1)))" "${items[$j]}" "${items[$j + 1]}"
    echo "    https://news.ycombinator.com/item?id=${items[$j + 2]} | ${items[$j + 3]} | ${items[$j + 4]}"
done

[gabrielsroka]

ok, a slightly better way. instead of using 10 full URLs, use 1 URL with a set of 10 item_ids, eg:
https://hacker-news.firebaseio.com/v0/item/{123,456,789...}.json

#!/bin/bash

MAX=10
item_ids=$(curl -s 'https://hacker-news.firebaseio.com/v0/topstories.json' |
    jq -j ".[:$MAX] | join(\",\")")

readarray -t items < <(
    curl -s "https://hacker-news.firebaseio.com/v0/item/{${item_ids}}.json" |
        jq -r '"\(.title)\n\(.url)\n\(.id)\n\(.by)\n\(.score)"')

for ((i = 0; i < MAX; i++)); do
    j=$((i * 5))
    printf "%-3s %s | %s\n" "$((i + 1)))" "${items[$j]}" "${items[$j + 1]}"
    echo "    https://news.ycombinator.com/item?id=${items[$j + 2]} | ${items[$j + 3]} | ${items[$j + 4]}"
done

[gabrielsroka]

this one does everything except for the index number at the beginning of the line (i'm working on that, too):

[EDITed, of course]

#!/bin/bash

item_ids=$(curl -s 'https://hacker-news.firebaseio.com/v0/topstories.json' | jq -j '.[:10] | join(",")')

curl -s "https://hacker-news.firebaseio.com/v0/item/{${item_ids}}.json" |
    jq -r '"- \(.title) | \(.url)\n  https://news.ycombinator.com/item?id=\(.id) | \(.by) | \(.score)"'

[gabrielsroka]

this one shows the index number with a space -- just like the original.

using foreach

#!/bin/bash

item_ids=$(curl -s 'https://hacker-news.firebaseio.com/v0/topstories.json' | jq -j '.[:10] | join(",")')

curl -s "https://hacker-news.firebaseio.com/v0/item/{${item_ids}}.json" | 
jq -sr 'foreach .[] as $item (0; . + 1; $item + {n: "\(.))\(if . < 10 then " " else "" end)"}) |
"\(.n) \(.title)\(if .url then " | " + .url else "" end)
    https://news.ycombinator.com/item?id=\(.id) | \(.by) | \(.score)"'

or using keys

#!/bin/bash

item_ids=$(curl -s 'https://hacker-news.firebaseio.com/v0/topstories.json' | jq -j '.[:10] | join(",")')

curl -s "https://hacker-news.firebaseio.com/v0/item/{${item_ids}}.json" | 
jq -sr 'keys[] as $n | .[$n] |
"\($n + 1))\(if $n < 9 then " " else "" end) \(.title)\(if .url then " | " + .url else "" end)
    https://news.ycombinator.com/item?id=\(.id) | \(.by) | \(.score)"'