No updates since July, website stuck?

[leahneukirchen]

planet.scheme.org hasn't been updated since July 24, but a few blogs had posts in the meantime, does the server need a restart?

[lassik]

The server is fine but the blog aggregator, Planet Venus, is held together with duct tape and bubble gum. It keeps breaking in various ways. In case you have spare cycles to try to fix it, I can give SSH access to the box. Beware that the work is not fun.

[leahneukirchen]

Uh, I'll give it a try, my key is at https://github.com/leahneukirchen.keys

[lassik]

Thanks. Please try ssh leahneukirchen@tuonela.scheme.org

My bash history says that I last tried the following command line:

cd /production/planet/ && sudo -u prod-planet tar -cf - cache | docker run --rm -i prod-planet | sudo -u prod-planet tar -xvf -

and the container stopped with the following error:

Downloading https://i.snap.as/hrje8kg1.png
Saving into images/cache/9f529e65913c774bac185bac4bffdc58acc7a761.png
Downloading https://i.snap.as/vp7SwJDz.png
Saving into images/cache/30fb7e963a45d1d6e9adbc7dbe8f6184ac93bc37.png
*** SYSTEM-ERROR: write failed on #<oport /planet/images-cache.scm 0x7f99e771a900>: Bad file descriptor
    While loading "./../images.scm" at line 91

There have been at least two different ways to have this contraption run as a cron job, and there were problems with each. There have also been many other, non-cron-related problems with the aggregator that I've mercifully forgotten.

[lassik]

The above error comes from images.scm. It's a custom script I wrote that runs using Gauche after Planet Venus has done its job. images.scm downloads all the images from the blog posts into a local cache since Planet Venus doesn't do that.

[leahneukirchen]

I'll have a look, thanks!

[lassik]

Thank you for your service. I wasn't kidding about the duct tape.

The simplest "patch on top of a patch" fix is probably to comment out the shell command that runs images.scm. In that case, the web page generated by Planet Venus ought to use off-site URLs in the <img ...> tags. However, there's no clear reason why images.scm should encounter Bad file descriptor. I hope it's not some obscure feature of Docker.

[leahneukirchen]

So when I ran it, it worked fine... One post has mochibake but that seems unrelated.

[lassik]

lol. It works in mysterious ways.

It's good to know that it's not systematically broken. We've been reluctant to spend time on it since the architecture is pretty much broken beyond repair. It would be best to write a new blog aggregator in Scheme or Racket, but no one has had the time to do it yet.

[leahneukirchen]

The mojibake is a result of this imo weird behavior of feedparser:

    But, if
    the HTTP Content-Type is text/xml, text/*+xml, or
    text/xml-external-parsed-entity, the encoding given in the XML prefix
    within the document is ALWAYS IGNORED and only the encoding given in
    the charset parameter of the HTTP Content-Type header should be
    respected, and it defaults to 'us-ascii' if not specified.

[leahneukirchen]

Where is the image-cache supposed to be stored anyway?

[lassik]

The mojibake is a result of this imo weird behavior of feedparser:

Good to know. Potentially relevant to #5.

Where is the image-cache supposed to be stored anyway?

As regular files under /production/planet/www/images/cache/.

[lassik]

https://planet.scheme.org/ is simply the contents of /production/planet/www/ served as a static site by nginx.

[leahneukirchen]

Those are the images, but I mean the cache file that script writes.

We could try updating Gauche, perhaps there was a bug in it.

[leahneukirchen]

Also, how was this meant to be executed periodically? The cronjob in /etc/cron.d doesn't exist.

[lassik]

Those are the images, but I mean the cache file that script writes.

My mistake. images-cache.scm is not shipped out of the container. It should be part of the tarball.

The culprit is probably this line In planet-container.sh:

tar --sort name -cf - www >&3

should be

tar --sort name -cf - images-cache.scm www >&3

We could try updating Gauche, perhaps there was a bug in it.

No idea.

Also, how was this meant to be executed periodically? The cronjob in /etc/cron.d doesn't exist.

$ sudo find /etc/ -iname '*planet*'
/etc/cron.d/scheme-prod-planet

[leahneukirchen]

$ cat /etc/cron.d/scheme-prod-planet
0 0-23/6 * * * prod-planet /production/planet/planet/planet.sh
$ cat /production/planet/planet/planet.sh
cat: /production/planet/planet/planet.sh: No such file or directory

[lassik]

Yeah, I'm not surprised that it's broken. It has been variously working and not working in the past.

The wider context here is that we should standardize how static sites under scheme.org are generated. The Planet counts as a static site for this purpose. The only twist is that unlike most static sites, it is rebuilt on a schedule.

I think the best approach for such a standard is a container that outputs a tar file. The Planet has taken a step in this direction, but since we're all busy and passionate sysadmins are hard to find around Lisp, the job is not done.

[leahneukirchen]

I'm not sure why you prefer these tar tricks over docker volumes, but ymmv.

I can rewrite that script to use docker, then we can run it on a schedule again?

[lassik]

Sure. That's a good fix.

[leahneukirchen]

But user prod-planet can't use docker directly...

[lassik]

The Planet counts as a static site for this purpose. The only twist is that unlike most static sites, it is rebuilt on a schedule.

My memory is refreshing. The cache is another oddity. Most static site builds don't use a dynamic cache as input.

But user prod-planet can't use docker directly...

Yeah. This is death by a thousand cuts.

I'll add prod-planet to the docker group until we think of something elegant. Feel free to use a Docker volume if you like.

[lassik]

OK.

$ sudo -iu prod-planet id
uid=4018(prod-planet) gid=4018(prod-planet) groups=4018(prod-planet),100(users),110(docker)

[leahneukirchen]

Ok, I adjusted the cron job now, it should only report when if fails, not sure who gets a mail then but I can also put myself into MAILTO. I'll watch it for a few days.

[arthurgleckler]

This is great! Thank you very much for looking into this. Keeping Planet Scheme up to date will make it much more useful.

[lassik]

What's up with planet.sh and cron.sh in /production/planet? The crontab entry in /etc/cron.d/scheme-prod-planet still points at /production/planet/planet/planet.sh.

Re: MAILTO, how does cron send mail? I don't think we have a working SMTP server on that box. IIRC Debian ships with Exim, but I can't find a running process that belongs to it.

[lassik]

The fact that we have a file called /production/planet/planet/planet.sh says something about the state of disrepair :)

[leahneukirchen]

Fixed the path (did i forget to save oO), seems like mail doesn't work indeed.

[lassik]

We can set up Exim. But in today's adversarial climate, does the mail have a fighting chance of landing in people's inboxes?

[leahneukirchen]

I wouldn't bother for now (and use postfix if we actually need an MTA).

[lassik]

Fair enough.

Thanks a lot for taking on this chore. I've been begging people to help for months.

If this keeps working, let's commit cron.sh and the fixed crontab entry to git. The principle is that all server configuration is public - no security by obscurity.

[leahneukirchen]

The bug happened once more, but when I tried to reproduce it, it worked again...

[lassik]

Do you mean the Bad file descriptor bug?

[leahneukirchen]

yes, that one. for mysterious reasons I still can't reproduce it manually.