SQLite included by default

[paganotoni]

One extra step that may be impacting our developer experience when installing the Buffalo CLI is sqlite. When developers want to use Buffalo with SQLite they need to install the CLI with the -sqlite tag, and our Homebrew/Cask installation does not provide an option to use this version.

Given all of the interest that SQLite has been receiving from developers and in the interest of compacting the stack that we maintain in Buffalo I was wondering if we could make our CLI ship by default with SQlite support.

Changing this:

• Updating our release process (to do cross-compilation as in Pop)
• Updating our docs to remove the extra SQLite steps.

I'm adding this here because I have a blurry memory about the reasons why we turned SQLite support into an optional feature of the CLI and besides the Cross-compilation issue I don't have it clear. Would love to hear your thoughts on the topic.

cc @sio4 @fasmat

[sio4]

It seems like the change happened on buffalo version v0.11 and pop version v4.0 but it seems like the related issue on the pop (before migrated into gobuffalo from markbates) was gone.

Actually, this is what I hope for a long time since buffalo new without --db-type sqlite installs buffalo-pop again with the non-SQLite version, and it will break other apps with SQLite continuously.

One possible disadvantage for the other database users will be they could need to install a C compiler and toolchains even though they do not use SQLite.

related issue: gobuffalo/buffalo#799
related PR and commits: gobuffalo/buffalo#933, gobuffalo/pop@d64e138

[paganotoni]

Thanks for bringing it up. I knew there was something I was missing. I still think we should release the CLI with SQLite included but wonder how much of an issue would the dependence on a C compiler be within our community.

Also, this makes me think about the dependency that Buffalo has on Pop. And how do we work to remove it on the road to v1. Plugins may facilitate this but I think is worth another conversation.

[adamtang79]

Almost all developers I've met has at least mysql in their machine with php dev env eg. XAMPP, WAMP or Laragon.

So, it's either mysql or postgresql.

Just my opinion because sqlite required gcc.

[sio4]

#209 is related to this discussion. (originally filed on docs repository by @rohfle)

Actually, I also love to use sqlite3 in the development so the same annoying situation happened several times to me too, but not fully sure if making sqlite3 support as default could be happy for all go/buffalo developers since it requires C development toolkits. (I think most developers have it but still, it could not be ideal)

(just wanted to add one more data point)

[paganotoni]

Yes. I think the dependency on a C compiler would make it hard to have Sqlite by default unless we use a pure Go driver on Pop. Which, I think, opens the door to something else: How much wired the CLI needs to be from Pop? Should we make pop optional now that we're building the new CLI?

[sio4]

For the original issue of sqlite by default, my idea is that:

• from the perspective of the source code, keep the current way of allowing build sqlite-enabled binary with build tags.
• to make it easier and simple for users, release sqlite-enabled binary as official distribution.
  I think it could be a better policy for releasing/distribution.

For the dependency on pop, As far as I know, the only strict requirement from pop is pop.AvailableDialects() function (just from module dependency point of view) but there are some command execution of buffao-pop or similar assumtion of using pop.

For AvailableDialects(), it is basically needed for buffalo new. Actually, I am not fully sure if we seriously need to decouple pop from buffalo (even though modulizing and supporting the other database engine could be nicer), but if the direction is concrete, the main focus should be on the design of "database-specific plugin interface" since it is not as simple as the other addon/extension style plugins. (I roughly think the plugin should support hook functions such as pre-build, post-test that kind of stuffs.

(I thought these ideas a few month ago so it is not based on the new plugin architecture you are working on, but my thoughts were originally from the thinking based on the current/legacy plugin interface and how to make database plugin fitable with the plugin.)

[rohfle]

I agree that releasing sqlite3-enabled binary as official distribution would be the best solution. I'm guessing you can use sqlite3-enabled buffalo to create a non-sqlite3 project?

I originally started using buffalo because I wanted Go's speed and compiled binary output combined with Django's philosophy of quick development, predefined auth handlers and a defined project structure (things that ultimately slowed me down with using GORM / Gin-based webapps)

For me as a first time user coming from a Django background, I assumed sqlite3 to be the development database by default so I can get started learning straight away without having to setup a postgresql instance for development / testing. Here are some things I found hard during the first hour of using buffalo that made me want to give up (summarized from #209):

1. Nothing in the getting started docs about using --db-type sqlite to initialize a sqlite-based project
2. Trying to find documentation of how to switch a project from postgres to sqlite3
3. Having to reinstall buffalo with a tag to use sqlite3
4. Having to change plugin tag configuration in addition to changing the database to sqlite3
5. Having buffalo-pop requiring a separate reinstall with a tag to use sqlite3

1,2,3 and 5 could be improved by documentation and a sqlite3 enabled default distribution
4 maybe could be improved by implicitly defining the sqlite tag for (certain?) plugins when sqlite3 database is defined in database.yml

Hope the feedback helps :)

[sio4]

Wow! Thanks for your feedback! I really appreciate it!

I agree that releasing sqlite3-enabled binary as official distribution would be the best solution. I'm guessing you can use sqlite3-enabled buffalo to create a non-sqlite3 project?

Yes, the other supported databases are also available by default, actually cannot be disabled. :-)

I originally started using buffalo because I wanted Go's speed and compiled binary output combined with Django's philosophy of quick development, predefined auth handlers and a defined project structure (things that ultimately slowed me down with using GORM / Gin-based webapps)

Yes, exactly. I think those are the core value of Buffalo. Productivity of Rails and Django in the Go world!

For me as a first time user coming from a Django background, I assumed sqlite3 to be the development database by default so I can get started learning straight away without having to setup a postgresql instance for development / testing. Here are some things I found hard during the first hour of using buffalo that made me want to give up (summarized from #209):

1. Nothing in the getting started docs about using --db-type sqlite to initialize a sqlite-based project
2. Trying to find documentation of how to switch a project from postgres to sqlite3
3. Having to reinstall buffalo with a tag to use sqlite3
4. Having to change plugin tag configuration in addition to changing the database to sqlite3
5. Having buffalo-pop requiring a separate reinstall with a tag to use sqlite3

1,2,3 and 5 could be improved by documentation and a sqlite3 enabled default distribution 4 maybe could be improved by implicitly defining the sqlite tag for (certain?) plugins when sqlite3 database is defined in database.yml

It really helps us to understand our users. I really hungry for the feedback to be honest. Thank you for your detailed feedback again, and I agree that we need to improve our document (actually some improvement activities on the work), especially for beginners!

[mdelapenya]

Hi! Probably jumping in here without that much context, but this issue came to my attention today.

I've used gobuffalo for a few PoCs in the past, so please correct me if I'm wrong when suggesting using a "real" database for testing/development mode instead of using SQLite.

I come from the @testcontainers Go community (https://github.com/testcontainers/testcontainers-go) and I'd like to offer collaboration between these two great OSS communities to provide a great developer experience using real services, like a real, containerized database (MySQL, postres) in the form of a Docker container, which would come with benefits regarding parity with production datastores, i.e.

Do you think there would be interest in starting a discussion on this? I'd be more than happy to help

Cheers! 👋

[mdelapenya]

Rereading my previous comment, I'd like to clarify that SQLite IS a real database, so please do not get me wrong 🙏 My intention here is to make a distinction from databases that are commonly used in web applications, such as Postgres or MySQL, not to "rant" or create criticism about another database