Set up docs self-hosting #162

petecooper · 2019-12-23T11:55:49Z

There's a growing need for self-hosting of docs for enhanced flexibility (see also #79).

Dibs. R&D phase.

Bloke · 2019-12-23T14:07:36Z

cough Textpattern instance with smd_ebooks for creating offline content cough

(The plugin will need tweaking for 4.8)

Honestly I can't see anything in Jekyll we can't do better or more efficiently in Txp. We have articles in sections, permlinks, markup, better search, header/footer, includes that are more flexible via forms and shortcodes, ...

Run the Jekyll pages through pandoc to convert from Markdown to Textile, or install Markdown as a textfilter and just port the content as-is.

If we can squirt collaborative changes from the GitHub repo to the Txp filesystem to rebuild the DB as part of a git post-commit hook or CI pipeline, we should be able to rebuild the changed docs with some script-fu.

petecooper · 2019-12-23T16:06:37Z

@Bloke - I'm inclined to agree. I'm learning CI + Pandoc at the moment, so we'll see what the new year brings.

wion · 2020-01-02T14:12:29Z

Considering the relatively low volume of docs content (and let's recognize that as by design), and the greater control Txp would provide over includes, URL redirecting, etc. I have to agree.

You could even do a flat-file site and keep it under version control. Version control is a nice thing, actually.

I don't think it's important to convert Markdown back to Textile, since we already went through that dance of going the other way. Screw that. Just add wet_markdown and use Textile for new docs only, if that's really a big deal.

That said, I don't want anything to do with the actual migration process. I know I've been in 'beast-mode', as Pete put it recently, but I'm just trying to make up for where I dropped the ball a few years back, get broken/lagging docs completed and improved. And I recognize some other editorial needs I'm happy to give attention to, to stabilize. But I'm aiming to get back to much less 'beast-mode', as it were. My objective here is get the content architecture square and then put myself on editor cruise-control so that I can return energy to my own writing.

So as long as this doesn't create any additional work for me that I haven't already shouldered myself, I have no objections.

petecooper · 2020-01-02T14:14:42Z

Thanks @wion - I do not want to create any additional work for anyone, and certainly not undo any work done.

philwareham · 2020-01-02T14:27:33Z

So.. you'd need to:

Keep articles as flat files (still on GitHub) and somehow inject direct to DB, so they can still be version controlled? Keeping version control is essential here.
Change wet_textfilter_markdown from Markdown Extra to Kramdown instead (or build a new plugin, I assume)

If those hurdles can be overcome, I'm interested.

petecooper · 2020-01-02T14:40:59Z

Keeping version control is essential here.

This is what's keeping me away from Textpattern as-is. On the one hand - yes - CMS, and we're literally managing content here, but on other hand, it doesn't feel like the right tool for the job.

Something more automated would be better, I think - even a self-hosted Jekyll in the first instance would be minimal effort to get going an replicate the GitHub delivery but with redirect trimmings and other stuff.

Flat files, static site generator, version control baked-in, make it wicked-fast, spit out compiled versions (EPUB, PDF, MOBI, etc) as part of the GitHub CI job, rinse and repeat.

(Edit: just my 2c, of course.)

Bloke · 2020-01-02T14:44:09Z

Suits me fine. If self-hosted Jekyll (or equivalent) is an option then go for it. I only mentioned Txp since it seemed like a good fit, but I'm by no means married to the idea if the obstacles are less than optimal to overcome. Go with whatever is easiest, most flexible and allows us to retain version control.

philwareham · 2020-01-02T14:56:45Z

Suits me fine. If self-hosted Jekyll (or equivalent) is an option then go for it

Sounds fine to me too, would be handy to use proper Jekyll plugins and proper SSL, redirects, etc. instead of the locked-down GitHub Pages version of Jekyll. I have no idea how best to achieve that but a quick web search brings this...

https://styxit.com/2017/09/13/self-host-github-pages-on-digital-ocean.html

If someone wants to have a stab at it then please go ahead. :)

petecooper · 2020-01-02T16:07:49Z

If someone wants to have a stab at it then please go ahead. :)

Dibs. Won't be for a while, but I'll take it.

philwareham · 2020-06-30T14:38:33Z

Short-term (before we potentially move to self-hosted Jekyll), GitHub now allows you more control over your Jekyll site as long as you deploy with GitHub Actions.

See https://jekyllrb.com/docs/continuous-integration/github-actions/

This now allows for third-party Ruby plugins and other customisations over and above the restricted GItHub Pages builds.

Will investigate at some point in the future.

petecooper · 2020-07-11T08:38:29Z

Folding this in so I can close it: #79

philwareham · 2020-07-14T13:23:15Z

OK, I've got the docs site running self-hosted on my local machine again (been a few years since I last hosted a Jekyll site so I had to install latest Ruby and various Gems again from scratch on my Mac).

I've created a branch on this repo called self-hosted where I'll put all the bits needed for self-hosting. Question is, whether we use the github-pages bundle of Jekyll components or roll our own more controlled version of Jekyll? My opinion is we go for the latter, which needs a bit more initial setup but gives us a few more options (not least, a newer version of Jekyll core).

Bloke · 2020-07-14T13:29:19Z

Nice one! Yeah, the more control we have over Jekyll and Liquid, the better it is for automation of repetitive, boring template stuff.

philwareham · 2020-07-14T16:11:43Z

OK, I have done the self hosting stuff needed to run Jekyll. As a rough test I have gone into our Textpattern server and git cloned the repo into the jekyll subdomain live directory, switched to the self-hosted branch and initiated a Jekyll build.

For now the site is running at https://jekyll.textpattern.com/textpattern.github.io/_site/

Seems to be some Content Security Policy issues that need to be fixed by @petecooper so the domain can read CSS and JS from textpattern.com domain. I haven't been able to check the search works yet either due to JS reliance.

Once we are happy this works as we want (after some testing in-situ), we then need to investigate how to automagically...

git clone the repo to our server when any changes are detected on GitHub.
Run a jekyll build (possibly with --watch flag).

Other considerations:

Rename the repo? (the textpattern.github.io name was a requirement for running it on the GitHub Pages platform).
Repoint the docs subdomain name to /textpattern.github.io/_site/ (or whatever the repo name will be) or alter the directory structure if that is a better solution.
Add any Ruby/Jekyll plugins that you think would be of benefit (to be honest, I'd want to keep plugins to a minimum and skimming the Ruby gems for Jekyll-related gems I didn't see way that would be essential).

Over to you for now.

philwareham · 2020-07-14T16:44:28Z

Also I might add, at the moment because the file structure sits at /textpattern.github.io/_site/ (inside the live directory), the absolute links in the site aren't working.

tagging textpattern/textpattern.github.io#162

petecooper · 2020-07-14T19:13:44Z

Seems to be some Content Security Policy issues that need to be fixed by @petecooper so the domain can read CSS and JS from textpattern.com domain.

Done - CSS, JS, images, fonts.

petecooper · 2020-07-14T19:19:21Z

Rename the repo? (the textpattern.github.io name was a requirement for running it on the GitHub Pages platform).

+1. Leaving it as github.io just confuses the branding if we're self-hosting. We're not exactly overrun with commits at the moment, and as I understand it the old repo URL will redirect to the new destination.

Repoint the docs subdomain name to /textpattern.github.io/_site/ (or whatever the repo name will be) or alter the directory structure if that is a better solution.

If you can email me the stuff you ran on the server to make the build happen, I'll have a look at the best scaffold from a maintainability and backups point of view. It's trivial to repoint an Nginx config to look at a certain dir, so no worries there.

@philwareham

fyi @philwareham see textpattern/textpattern.github.io#162

petecooper · 2020-07-15T09:05:08Z

Jekyll test site now running off the root: https://jekyll.textpattern.com

petecooper · 2020-07-15T09:10:23Z

From my side of things, I need to have a play with the rebuild process on a test server, then I think we're nearly there with hosting the docs ourselves. I don't have cods of time this week, so it might end up being evenings and weekend, but we're close.

@philwareham - if you're largely done from your end, are you comfortable for me to temporarily disable the site at an agreed time so we don't dilute SERPs etc with duplicate docs from this test site?

philwareham · 2020-07-15T11:55:14Z

CSP for the Jekyll test site needs a couple of amends, which I have just committed to the server-config repo. Can you update on server @petecooper? Thanks.

Also some links don't seem to be working on the docs site on server, although they work fine in my local build, so maybe something needs adjusting on server side? For example: https://jekyll.textpattern.com/tags/article gives a 404 error at the moment.

petecooper · 2020-07-15T11:56:16Z

Back in office around 4pm, will sort.

…

-- Pete Cooper pete@pragmatika.net

On 15 Jul 2020, at 12:55, Phil Wareham ***@***.***> wrote: CSP for the Jekyll test site needs a couple of amends, which I have just committed to the server-config repo. Can you update on server @petecooper? Thanks. Also some links don't seem to be working on the docs site on server, although they work fine in my local build so. maybe something needs adjusting on server side? For example: https://jekyll.textpattern.com/tags/article gives a 404 error at the moment. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

petecooper · 2020-07-15T15:11:57Z

Also some links don't seem to be working on the docs site on server, although they work fine in my local build, so maybe something needs adjusting on server side?

The Markdown files aren't being transformed into HTML, so it's 404-ing.

Aside: I've fixed permissions (should be www-data:www-data for owner and group). I'll make sure this goes into the final build scripting.

petecooper · 2020-07-16T13:54:47Z

@philwareham - it looks like jQuery is needed for search - at the moment Jekyll is calling from https://code.jquery.com/jquery-3.3.1.min.js - add to CSP or serve locally? I'm inclined to serve locally if we can but I'll take your advice on this.

philwareham · 2020-07-16T13:57:58Z

We should serve locally. Can you raise an issue and I'll sort it soon. Ta!

philwareham · 2020-07-16T14:03:51Z

Another issue: 404s should use 404.html in the root of the Jekyll _site directory.

Currently it's just using the Nginx default 404 (example https://jekyll.textpattern.com/tags/expiresffsf)

petecooper · 2020-07-16T14:18:29Z

404s should use 404.html

Done.

addresses textpattern/textpattern.github.io#162 (comment)

philwareham · 2020-07-16T19:14:35Z

Now I've fixed the jQuery issue mentioned above, I can't see any further issues in the Jekyll site running on the server, so I think my part is done.

The final piece of the puzzle is to detect when a commit is made to the repo on GitHub, git pull and then jekyll build.

philwareham · 2020-07-16T19:24:14Z

Maybe this?: https://ryanharrison.co.uk/2018/07/05/jekyll-rebuild-github-webhook.html

philwareham · 2020-08-19T12:15:07Z

Just wondered where we are at with this? No rush though, just curious.

petecooper · 2020-08-19T15:55:47Z

Realistically September, I think. My business is badly affected by COVID stuff and 90%+ of my work has evaporated, so I'm having to juggle a few things around this to make sure rent is paid etc. Not forgotten.

I've got all your notes on self-hosting stored safely, thanks to your sterling efforts on Jekyll, so should be straightforward enough when I get around to it. May end up being part of the server rebuild with latest Ubuntu long term support version, plus some other bits and pieces.

wion · 2020-12-07T18:37:17Z

Long out of the loop. Sorry. Just curious if we lose repo versioning user docs when going to a self hosted install.

petecooper · 2020-12-07T18:38:04Z

Nope.

Repo operations won't change. The publishing target (?) will, so we'll have more control over it and be able to do more stuff, but no functionality is taken away.

wion · 2020-12-07T18:39:33Z

Still github or something else?

wion · 2020-12-07T18:40:13Z

Okay, I see. Thanks.

petecooper · 2020-12-07T18:41:48Z

Still GitHub. Still all the same stuff here, perhaps a repo name change to reflect the move away from GitHub publishing infra, but we'll be running a self-hosted Jekyll with all the trimmings we need, which is a conscious move to not reinvent the wheel etc.

petecooper · 2022-02-08T19:40:38Z

Initial commentary on beta site linked in #162 (comment) – subject to change.

The build appears to work OK, but we have an issue with jekyll-pwa as seen here: lavas-project/jekyll-pwa#39, specifically:

Configuration file: /var/www/servers/textpattern.com/docs-test/live/jekyll/_config.yml
            Source: /var/www/servers/textpattern.com/docs-test/live/jekyll
       Destination: /var/www/servers/textpattern.com/docs-test/live/jekyll/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
       Jekyll Feed: Generating feed for posts
                    ------------------------------------------------
      Jekyll 4.2.1   Please append `--trace` to the `build` command 
                     for any additional information or backtrace. 
                    ------------------------------------------------
Traceback (most recent call last):
	20: from /home/pete/gems/bin/jekyll:23:in `<main>'
	19: from /home/pete/gems/bin/jekyll:23:in `load'
	18: from /home/pete/gems/gems/jekyll-4.2.1/exe/jekyll:15:in `<top (required)>'
	17: from /home/pete/gems/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
	16: from /home/pete/gems/gems/mercenary-0.4.0/lib/mercenary/program.rb:44:in `go'
	15: from /home/pete/gems/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `execute'
	14: from /home/pete/gems/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `each'
	13: from /home/pete/gems/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `block in execute'
	12: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
	11: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `process_with_graceful_fail'
	10: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `each'
	 9: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `block in process_with_graceful_fail'
	 8: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:36:in `process'
	 7: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:65:in `build'
	 6: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/command.rb:28:in `process_site'
	 5: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/site.rb:82:in `process'
	 4: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/site.rb:234:in `write'
	 3: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/hooks.rb:102:in `trigger'
	 2: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/hooks.rb:102:in `each'
	 1: from /home/pete/gems/gems/jekyll-4.2.1/lib/jekyll/hooks.rb:103:in `block in trigger'
/home/pete/gems/gems/jekyll-pwa-plugin-5.1.5/lib/jekyll-pwa-plugin.rb:169:in `block in <module:Jekyll>': undefined local variable or method `page' for Jekyll:Module (NameError)

Not a deal breaker, as far as I can tell, but something to be aware of.

For now, I'm going to set up a regular cron task to get the docs repo on a regular basis instead of the webhook stuff as mentioned in #162 (comment) – some trivial Bash is more efficient (for me, at least) than learning about webhooks and the associated stuff.

Search appears to be a little faster on self-hosted Jekyll. Still not stellar but we can faff with that after the other stuff is sorted.

The Nginx config is a little different to PHP web apps, but easily managed.

petecooper · 2022-02-08T19:55:35Z

To do:

Figure out how to run bundle via crontab (non-root user) on rebuild script.
Set up https://github.com/christian-fei/Simple-Jekyll-Search

petecooper · 2022-05-16T12:25:55Z

FYI docs site is now self-hosted. It's not pretty behind the scenes – but it works, it's automated, and we have some snagging to fix re: deprecated and EOL gems. I'll start another issue for each of those items.

Will set this up soon. See #192 and #162.

petecooper added this to the 4.9 release milestone Dec 23, 2019

petecooper self-assigned this Dec 23, 2019

philwareham self-assigned this Jun 30, 2020

philwareham mentioned this issue Jul 7, 2020

meta: docs on GitHub Pages, and associated URL redirects #138

Closed

petecooper changed the title ~~Investigate self-hosting of docs~~ Investigate self-hosting and offline compiling of docs Jul 11, 2020

petecooper mentioned this issue Jul 11, 2020

Investigate automatic compilation of docs to offline format #79

Closed

petecooper added a commit to textpattern/server-config that referenced this issue Jul 14, 2020

working CSP for jekyll testbed

17a19d6

tagging textpattern/textpattern.github.io#162

petecooper added a commit to textpattern/server-config that referenced this issue Jul 15, 2020

repoint jekyll testbed to _site dir

850f42e

fyi @philwareham see textpattern/textpattern.github.io#162

petecooper changed the title ~~Investigate self-hosting and offline compiling of docs~~ Set up docs self-hosting and automatic compilation to ebook formats Jul 16, 2020

petecooper added a commit to textpattern/server-config that referenced this issue Jul 16, 2020

correct 404 handling on jekyll testbed

079f2e8

addresses textpattern/textpattern.github.io#162 (comment)

philwareham removed their assignment Jul 16, 2020

philwareham pinned this issue Aug 19, 2020

petecooper unpinned this issue Jul 17, 2021

petecooper mentioned this issue Oct 21, 2021

Search is broken on macOS/Safari 15 #192

Open

petecooper mentioned this issue Feb 6, 2022

Missing page and home layouts from _layouts #196

Closed

petecooper changed the title ~~Set up docs self-hosting and automatic compilation to ebook formats~~ Set up docs self-hosting Feb 8, 2022

petecooper mentioned this issue May 16, 2022

Investigate https://github.com/christian-fei/Simple-Jekyll-Search #201

Closed

1 task

philwareham self-assigned this Nov 25, 2022

philwareham added a commit that referenced this issue Nov 25, 2022

Initial files for search update/fix

b329c14

Will set this up soon. See #192 and #162.

petecooper closed this as completed Dec 8, 2022

petecooper reopened this Sep 23, 2023

petecooper closed this as completed Sep 23, 2023

Set up docs self-hosting #162

Set up docs self-hosting #162

Comments

petecooper commented Dec 23, 2019

Bloke commented Dec 23, 2019 • edited Loading

petecooper commented Dec 23, 2019

wion commented Jan 2, 2020

petecooper commented Jan 2, 2020

philwareham commented Jan 2, 2020 • edited Loading

petecooper commented Jan 2, 2020 • edited Loading

Bloke commented Jan 2, 2020

philwareham commented Jan 2, 2020

petecooper commented Jan 2, 2020

philwareham commented Jun 30, 2020 • edited Loading

petecooper commented Jul 11, 2020

philwareham commented Jul 14, 2020 • edited Loading

Bloke commented Jul 14, 2020

philwareham commented Jul 14, 2020 • edited Loading

philwareham commented Jul 14, 2020

petecooper commented Jul 14, 2020

petecooper commented Jul 14, 2020 • edited Loading

petecooper commented Jul 15, 2020

petecooper commented Jul 15, 2020

philwareham commented Jul 15, 2020 • edited Loading

petecooper commented Jul 15, 2020 via email

petecooper commented Jul 15, 2020

petecooper commented Jul 16, 2020 • edited Loading

philwareham commented Jul 16, 2020

philwareham commented Jul 16, 2020 • edited Loading

petecooper commented Jul 16, 2020

philwareham commented Jul 16, 2020

philwareham commented Jul 16, 2020

philwareham commented Aug 19, 2020

petecooper commented Aug 19, 2020

wion commented Dec 7, 2020

petecooper commented Dec 7, 2020 • edited Loading

wion commented Dec 7, 2020

wion commented Dec 7, 2020

petecooper commented Dec 7, 2020

petecooper commented Feb 8, 2022

petecooper commented Feb 8, 2022 • edited Loading

petecooper commented May 16, 2022

Bloke commented Dec 23, 2019 •

edited

Loading

philwareham commented Jan 2, 2020 •

edited

Loading

petecooper commented Jan 2, 2020 •

edited

Loading

philwareham commented Jun 30, 2020 •

edited

Loading

philwareham commented Jul 14, 2020 •

edited

Loading

philwareham commented Jul 14, 2020 •

edited

Loading

petecooper commented Jul 14, 2020 •

edited

Loading

philwareham commented Jul 15, 2020 •

edited

Loading

petecooper commented Jul 16, 2020 •

edited

Loading

philwareham commented Jul 16, 2020 •

edited

Loading

petecooper commented Dec 7, 2020 •

edited

Loading

petecooper commented Feb 8, 2022 •

edited

Loading