I have no UUIDs what's going on 🥁

[axelboc]

I have to admit I'm a bit lost in all those UUIDs and GUIDs issues. 😅 I didn't quite understand your #396 (comment), @aplaice, and between the issues relating to CrowdAnki and the ones relating to the migration from Anki DM to Brain Brew, I'm no longer sure:

1. what problems need to be solved;
2. what overall situation/solution we're aiming for;
3. what we need to fix here, in UG (documentation included);
4. what needs to be fixed in Brain Brew, CrowdAnki, another add-on...

We had discussions in multiple issues and PRs (#390, #391, #392, #393, #394, #396, #398), so I thought we could have a more centralised discussion here. 😁

[aplaice]

Overall summary at end. (Sorry for the length...)

---

First to summarise the different UUIDs and GUIDs (@axelboc you're almost certainly aware of all this, but others who stumble here might not):

There are three UUIDs, "invented" by CrowdAnki:

1. Deck crowdanki_uuid, which labels decks and subdecks as individual entities. This allows it to "track"/recognise the deck despite it being renamed or turned into a subdeck. (If for instance, two people release two different "Spanish" decks as shared CrowdAnki decks, and somebody wants to use both, then CrowdAnki will import one of these as Spanish, the other as Spanish_2, but on upgrades it won't confuse the two, thanks to the deck uuid.
2. Note model crowdanki_uuid, which labels note models/note types. Hence, if two people create two different Basic (and reversed card) note types (with different styling and possibly even extra fields), then again CrowdAnki won't confuse them, allow each one to be modified and upgraded separately etc.
3. Deck config crowdanki_uuid, which labels the deck options (initial learning steps, behaviour on lapses, starting ease etc.). IMO this is usually just annoying. However, in principle, it could be useful, for instance if a deck (or subdeck) is particularly hard or particularly easy and the creator wants to set the initial steps or initial ease to an appropriate level.

In addition, there's the GUID which is used internally by Anki (and not just by CrowdAnki) and which labels individual notes.

---

anki-dm's behaviour (4.0)

anki-dm created separate UUIDs for each language and each deck extent (normal and extended), at least in principle, and separate GUIDs for each language, again at least in principle.

For example, English (normal) had:

1. a set of UUIDs (deck, note model and deck config), which it did not share with any other version of the deck (at least in theory),
2. a set of GUIDs (for each note), which it shared only with English (extended) (at least in theory)

Similarly for German, Spanish etc.

Unfortunately, in practice, (English and Norwegian) and (Spanish and French) had the same hashing "salt", so in practice English (normal) had:

1. a set of UUIDs (deck, note model and deck config), which it shared only with Norwegian (normal),
2. a set of GUIDs (for each note), which it shared only with English (extended), Norwegian (normal) and Norwegian (extended)

BrainBrew's current (newly merged) behaviour (4.1.1)

The deck UUID and the deck config UUIDs are the same for all decks (irrespective of language and deck extent).

The note model UUID is the same among all the normal decks and the same among all the extended decks.

The GUIDs are exactly like they were with anki-dm. For the new notes and the new languages, there are separate sets of GUIDs per-language.

Slight Upgrade issues from 4.0 to 4.1.1

1. For the German, Spanish and French decks, the note model UUID has changed for all the notes. (Each previously had their own, while now they're the same as the English.) A situation where the note model UUID changes for the notes in a deck results in a "Change Note Type" pop-up, which is slightly scary (I'm easily scared :)) and which asks you how you want to map the fields (in this case it's, by default, a trivial 1-1 mapping — Country → Country etc., but if you were changing the note type from "Ultimate Geography" to "Basic", it wouldn't be) and how you want to map the cards (in this case it's again a trivial 1-1 mapping — Country-Capital → Country-Capital etc.). If you don't meddle with the default mappings then nothing bad happens.
2. Again for the German, Spanish and French decks, the deck UUID has changed. This means that CrowdAnki thinks that the existing deck (4.0) is different to the new deck (4.1.1) and creates a duplicate deck, moving all the notes there (or not, if the user ticked the "Do not move existing cards" box).

The Norwegian deck has all the same UUIDs as before, so users of the Norwegian deck get neither the "Change Note Type" pop-up, nor have a duplicate deck created.

Ideal behaviour (theoretical point of view)

(The theoretical point of view is slightly abstruse, but it helps elucidate, at least for me, what the GUIDs/UUIDs "mean".)

In principle, each UUID and each GUID should label a distinct entity (note, note model, deck, deck option). If two entities are identical, they should have the same UUID (or GUID). If not, they should have different UUIDs/GUIDs.

For instance, the note (England,...,London,...) is the same in Ultimate Geography [EN] as in Ultimate Geography [EN] [Extended], so it should have the same GUID. ✅

In contrast, (Angleterre,...,Londres...) is different, so it should have a different GUID. ✅ (but ES<->FR share GUIDs and EN<->NB share GUIDs ❎)

The "Ultimate Geography" note model is different from the "Ultimate Geography [Extended]" note model, so they should have different UUIDs. ✅

The "Ultimate Geography" note model for different languages is currently the same (we don't localise the note models/templates (yet! :D)), so it should be the same for all languages. (This is the case now, it wasn't the case earlier.)

The deck configs are all identical (the learning steps etc. are the same), so they should all be the same. (This is the case now, it wasn't the case earlier.)

I'm not sure what it means for two decks (as entities in themselves rather than their contents) to be the same, so I have no idea what is the "correct" situation here. (From a practical point of view, it's a question of whether, if somebody uses more than one language version in parallel, do we want them to have all the notes in one deck or in multiple decks (within Anki itself). Having them in multiple decks might make slightly more sense, but I'm not sure.)

Our needs and desires

1. We want to allow people to have several different language decks in parallel. ⇒ We require the GUIDs to be different for each language.

   Hence, we need a way to migrate our Spanish (or French) and Norwegian users onto a new set of GUIDs so that they don't lose progress.

2. (Ideally) we would like for contributors to be able to translate decks "in-place" (within Anki) and continue using this deck after merging here. By default, this is contradictory with 1.

   However, if we had a way to migrate contributors from one set of GUIDs to another (see 1.) this would be solved.

3. Eventually, we might want to localise the templates. When that happens we'll need to change the note model UUIDs so that each language has its own pair (normal and extended).

   AFAICT this is doable with BrainBrew, and just requires slightly more complicated recipes.

4. We'd like the deck crowdanki_uuid to be imported when users import AUG from Ankiweb (or from an APKG), so that when they later upgrade with CrowdAnki they don't get a duplicate deck.

   This would ideally be fixed in Anki itself. It could be "sort-of" solved in CrowdAnki, but it'd require vague heuristics (recognising/asking the user whether an existing deck is the "target" deck), which UUIDs were meant to avoid.

5. We'd like the users who imported the Czech, Dutch, Swedish and Russian decks at the start of last week (when the GUIDs were the same for all languages) to be able to switch to the version where the GUIDs are different (i.e. have the upgrade from 4.1 to 4.1.1 not result in data loss)

   This has the same solution as 1. and 2. (an add-on that can map a set of GUIDs onto another set).

6. We would like to be able to generate all the GUIDs automatically when adding new notes and new languages.

   If I understood correctly, this doesn't quite work yet, but @ohare93 plans to work on this when he has time.

Documentation

Basically #398.

Duplicate decks

In a variety of circumstances (e.g. APKG→CrowdAnki for EN; 4.0 → 4.1.1 for FR, ES and DE) a duplicate deck is created, but is harmless (no notes are duplicated).

In other cases, (e.g. 4.0 → 4.1 for FR, ES and DE; 4.1 → 4.1.1 for CZ, NL, SV and RU) a duplicate deck is also created, but is evidence of a more serious problem (notes being duplicated). (Edit: the 4.1 → 4.1.1, or 4.1.0 → 4.2 upgrade for CZ/NL/SV/RU will be particularly treacherous because the deck won't be duplicated (the deck UUID won't be changing), only the notes will be. Even worse, users won't be able to easily tell whether they have 4.1.0 or 4.1.1 beforehand, because the description is the same, and it's non-trivial to check whether one imported the deck before or after 14 February (one can look at the "Created" date in the browser, but that's fiddly). The situation isn't hopeless, though, even after upgrade — if the user checks the number of notes immediately after upgrade, they can find the newly added cards with "deck:Ultimate Geography" added:1, delete them, migrate their GUIDs with the as-yet-nonexistent addon and upgrade.)

We'd like to allow users to distinguish the two, probably by verifying that the total number of cards is what is expected (currently 975 for the normal deck and 1508 for the extended deck, ignoring "legacy" cards).

We should also reassure them that they can move the cards into one deck and delete the other. (We could try to explain which deck it'd make sense for them to delete (usually the original one!), to prevent CrowdAnki from creating duplicate decks in the future, but that's probably over-complicated.)

Explanation of import options (not really UUID-related, but for completeness)

We should explain that users don't need to (and shouldn't) tick any boxes other than (possibly) the "do not move existing cards" one.

Summary

1. (Ideally) we need an add-on for switching GUIDs. (external addon)
2. We'd like to be able to automatically generate GUIDs. (UG/BrainBrew)
3. We'd like to document the upgrade pitfalls (i.e. basically Improve Minor upgrade instructions #398). (UG)
4. We'd like for Anki to import deck UUIDs from APKGs (or for CrowdAnki to deal with their lack, better). (Anki/CrowdAnki)
5. In the medium-run we'd like to have separate GUIDs for Spanish and French and for English and Norwegian. (UG)
6. In the long-run we might want to to have localised templates and separate note model UUIDs for each language. (UG?)
7. We might want to have separate deck UUIDs for each language (but I'm not sure). (UG?)
8. (Edit:) We'll have to tell users upgrading from the 4.1 CZ/NL/SV/RU decks to be extra careful. (UG)

I hope/think that this is everything directly related to UUIDs and GUIDs... :slight scream:

[axelboc]

Regarding issue no. 3 (documentation):

1. We should definitely mention CrowdAnki's new import dialogue in the upgrade instructions in the README.
2. However, instructions that are specific to the v4.1 release and to some of the translated decks should perhaps not be added to the README.

I think we should maybe review the issue of upgrade instructions as a whole, both in terms of content and process. For instance, is it still relevant to have minor/major upgrade instructions, since each version seems to require highly specific instructions anyway? Also, upgrade instructions are currently scattered around the place a bit: we have some in the README, in the release notes, and sometimes in issues/discussion. Perhaps we should rethink this?

One idea would be to create a new Discussion for each release with its upgrade instructions (even if it means duplicating some parts from previous discussions). This would have multiple benefits:

• The release notes would just point to the relevant discussion so they wouldn't be as cluttered.
• We'd be able to write the upgrade instructions ahead of the release and make sure they're accurate and up to date each time.
• We could really go into details, like providing instructions for upgrading between specific versions (e.g. from 4.0 to 4.2), for checking that the upgrade went, or for troubleshooting problems.
• People could report issues directly in the discussion thread, making them easy for other people to find.

[aplaice]

For instance, is it still relevant to have minor/major upgrade instructions, since each version seems to require highly specific instructions anyway?

I think mentioning the idea that minor version upgrades shouldn't result in the loss of progress, while major ones might could still be in the README, but otherwise, yeah.

It might also be worth keeping a mention in the README that one can upgrade via the same general mechanism as used for importing the deck via CrowdAnki.

Otherwise, creating a new Discussion for each release with the full instructions is a brilliant idea!

An alternative, but IMO worse, idea would be to have each set of instructions in a Wiki page. (This has the advantage that anybody can add details that they believe are useful. It has the huge disadvantage that it's not as "interactive" as a discussion page. In an ideal world the first comment would be a Wiki, to have the advantages of both. On the whole, I think the advantages of a discussion page considerably outweigh the advantages of a Wiki, in this case — a Wiki would shine for more long-standing topics where having a globally-editable description of the situation, without having to wade through streams of outdated comments, would be valuable.)

[ohare93]

I think that in the case of duplicates of note models CrowdAnki's system is valuable. If two decks had, say, slightly different Basic (and reversed card) note types, then dealing with Basic (and reversed card) and Basic (and reversed card)-ji2we3 and trying to remember which is which (or renaming them and thinking up appropriate names) would be hell.

A smart system would be able to work out that since all the notes being imported that already exist in your own Anki (found via their Guid) all share the exact same note model, then that is probably the one it is looking for 😅 For more complicated imports with multiple notes and model types (not what we are doing (yet!)) then it would simply offer those as the first sorted options, and perhaps can show their contents in that view rather than relying on the name alone.

[aplaice]

I have a set of debug console scripts (and a script for generating them) for "fixing" (adapting from v4.1 to v4.1.1) the Czech, Swedish, Dutch and Russian GUIDs, here.

Thanks @ohare93 for suggesting this!

They probably need double-checking (and testing in other configurations) but they seem to work, in testing (upgrade from 4.1 to 4.1.1 becomes possible, without duplicating all the cards/notes), with Anki 2.1.39.

When they're "officially" released, it'll make sense to have one gist per conversion/language to avoid people copy-pasting the wrong gist.

(Accessing the Debug Console seems to be via CtrlShift:.)

[ohare93]

Code that generates code. Classic! 💯 👍 😁

[axelboc]

What do you see the next release doing in regard to UUIDs/GUIDs?

• I guess we'll definitely include verification steps for CZ/NL/SV/RU in case any of them imported v4.1 before v4.1.1 was released.
• The localised templates and separate deck and note model UUIDs are quite low priority in my opinion.
• Should we aim to have unique sets of GUIDs between languages?

[aplaice]

I guess we'll definitely include verification steps for CZ/NL/SV/RU in case any of them imported v4.1 before v4.1.1 was released.

Yes, definitely.

The localised templates and separate deck and note model UUIDs are quite low priority in my opinion.

I agree. It's vaguely nice-to-have, but not at all urgent or particularly important.

Should we aim to have unique sets of GUIDs between languages?

This (the end goal of allowing all possible parallel language decks) isn't urgent or very important either. I can see two approaches:

1. Do it asap (with the next minor release), to "rip the bandaid off" and to avoid having more people import the NB and ES (or FR?) deck with the "wrong" GUIDs.

2. Do it with the next major release, when loss of progress is already considered permissible, so having a slightly hacky workaround (using the GUID migration debug console script) to keep progress would be more acceptable.

   OTOH with a major release we might be needing to list other workarounds/special cases etc., so it might be better not to also have the GUID migration script, to further confuse people.

   • Also, since we'll be mentioning the GUID migration script for CZ/NL/RU/SV anyway for the upgrade to 4.2, we may as well switch GUID sets for NB and ES (or FR?), at the same time.

I'm leaning towards 1 mainly just to get it over with*.

* we might need to keep a slight warning about upgrading, for people jumping several releases, for several minor releases afterwards, but the sooner we start, the sooner it'll be over

We also need to decide which of FR and ES we want to keep on their existing GUIDs and which to switch. :D

(An alternative approach is to just not bother having unique sets of GUIDs, but I think that it'd be a shame — we're almost there, but without being fully there we couldn't actively recommend parallel decks without long, boring, annoying caveats.)

[axelboc]

I share the same views. I think unique sets of GUIDs are worth having and therefore that we should do it as soon as possible to avoid inconveniencing too many users.

Also, since we'll be mentioning the GUID migration script for CZ/NL/RU/SV anyway for the upgrade to 4.2, we may as well switch GUID sets for NB and ES (or FR?), at the same time.

If we release a v4.2 with all the GUID stuff, we can tell people in future releases (4.3+) to upgrade to 4.2 first by pointing them to its upgrade instructions.

[axelboc]

We also need to decide which of FR and ES we want to keep on their existing GUIDs and which to switch. :D

I can't see any stats on ZIP downloads, or geographical origin of traffic to this repository ... so I guess we'll have to pick one at random. I suggest a coin flip. Sounds fair?

• Heads, FR gets its GUIDs switched.
• Tails, ES gets its GUIDs switched.

The side of a euro coin with the national design (here from Ireland) is the obverse, which is heads, so ...

FR gets its GUIDs switched

[aplaice]

I suggest a coin flip. Sounds fair?

Sounds fair and sensible!