defining "on disk" in the AE glossary

[leoebfolsom]

I've abruptly become a huge fan of the Analytics Engineering Glossary, and I was reading the page on Table, and I think a future reader (and I myself) would benefit from an explanation of "on disk." There was a notification at the top of the page telling me to come here and start a discussion if I wanted to help, and so here I am! Not sure if this is in the right category to put the discussion in.

I would be happy to put my non-hardware-engineer spin (get it, spin? like a disk) on how "on disk" is relevant to Analytics Engineering, at least in the context of a page explaining what a table is. I think it might be adequate to simply add some text, like a tooltip or footnote, explaining that anything the computer needs to remember long-term is stored on a physical disk; and such a disk might spin like a dinner plate (think a CD by *NSYNC); or could be a solid hunk of metal, like what's probably inside your laptop. So when you add information to a table in a database, it is stored "on disk," which means it will generally stay there, never-changing, until someone or something removes it.

I by no means suggest this is the best or most accurate way to explain the concept, but it's my first pass.

Does one need to understand this low-level technical information related to data storage to get started with analytics engineering or even to become extremely valuable on a data team? Opinions may vary, but I mainly bring this up because if we're going to throw around terms like "on disk," I think they should be explained; speaking for myself, I know that 1-ish years ago, I would have felt gatekept by that jargon, and it still feels like inside baseball.

.

[jasnonaz]

Hi @leoebfolsom - that's an awesome point and completely agree it would be great to have this explained.

Very much the type of thing that could throw off someone just getting started on their AE journey. I like the direction you're thinking for how we'd include this - wondering if you'd be willing to take a spin on disk at putting together a PR that makes this a lil more accessible. thank you ⭐️

[leoebfolsom]

I'm happy to take a stab at it. Possibly devil's advocate here, and after some more thought, but: maybe someone at this stage in their AE journey, when they're absorbing the very basics of what a table is, doesn't actually need to have disk explained? Could be simply:

A table is the direct storage of data in rows and columns.

And then another page, another day, could get into more detail. That page could be called "By the way, how is data actually stored?"

I know when I was learning dbt, I wouldn't have found it helpful for someone to be like "wait, before we explain what a table is, you need to understand that the warehouse is actually a bunch of spinning disks in eastern Oregon, oh and they could die and/or hold an election for a new leader at any time." BUT that is important/useful information that could be given more care on its own page.

I'm just not convinced it matters or is meaningful to anyone from the very beginning of their AE journey that the data is stored on disk (anyway, as opposed to what--are we really going to go there?). From an intro AE standpoint, it's "in the warehouse." (Right?)

[jasnonaz]

Ok I'm convinced @leoebfolsom - great point that on disk is on net more confusing than helpful at this stage and we can just change the language. Lmk if you want to submit a PR and myself or @KiraFuruichi can give it a 👍 - otherwise we can grab it

[leoebfolsom]

@jasnonaz done ✅ #1836

[jasnonaz]

You are a gem ty