Generation of Unique Human readable IDs

[dubdabasoduba]

Context

• Patients are assigned unique identifiers to help the health workers to track them easily. In OpenSRP we used to generate this and assign them to the patients during registration.

Implementation requirements

• We need to come up with a similar if not improved functionality for the FHIR work. This unique identifier should have the following characteristics.
  • Should be human-readable
  • Should have the following items configurable by the implementations
    • The identifier id -- Used to tell the mobile apps which identifier configuration to use.
    • The generation algorithms to use
    • Base character set to use for the identifiers
    • Min & Max number of digits
    • Prefix & Postfix to use on the identifers
    • Any validation regex to act on the identifiers
• The identifiers should be saved as part of the FHIR identifiers on the patient resources.
  • This should preferably be saved and the official identifier
• The identification of what identifier to use for each app should be part of the configurable items when loading up a new app track.
• The identifiers should be unique across the whole implementation, not just a facility, and the apps should not allow for duplicate identifier assignment or generation.

[dubdabasoduba]

Here is some little documentation on the current OpenSRP implementation.

[pld]

The issue here is if we generate offline, we need something with a large space of possible ideas to minimize the chance of collision, but the standard way to do that is length, which conflicts with the goal of it being easy for a human to use.

OpenSRP avoided this problem by pre-generating them to guarantee no-collision, but we want to move away from that and do generation on device so we do not have a centralized ID management service.

[f-odhiambo]

I also think we need a firm decision on wether to use identifier fields as opposed to UUID ID for all resources which is in place at the moment. If so

1. We leave HAPI to increment ID on create/POST FHIR resources
2. Assign and search for all FHIR resources using the identifier filed

[pld]

Can we split this up? I.e. we can keep using device generated UUIDs as ID, and this identifier is for human readable uses, good point we'd still need to make it searchable, but we don't need to change how we interact w/HAPI or use POST, we need to use PUT since UUID IDs are assigned on device, which is required for off-line use cases

[dubdabasoduba]

Context

• Patients are assigned unique identifiers to help the health workers to track them easily. In OpenSRP we used to generate this and assign them to the patients during registration.

Implementation requirements

• We need to come up with a similar if not improved functionality for the FHIR work. This unique identifier should have the following characteristics.

  • Should be human-readable

  • Should have the following items configurable by the implementations

    • The identifier id -- Used to tell the mobile apps which identifier configuration to use.
    • The generation algorithms to use
    • Base character set to use for the identifiers
    • Min & Max number of digits
    • Prefix & Postfix to use on the identifers
    • Any validation regex to act on the identifiers

• The identifiers should be saved as part of the FHIR identifiers on the patient resources.

  • This should preferably be saved and the official identifier

• The identification of what identifier to use for each app should be part of the configurable items when loading up a new app track.

• The identifiers should be unique across the whole implementation, not just a facility, and the apps should not allow for duplicate identifier assignment or generation.

@pld @f-odhiambo Based on the eCBIS forms we need to discuss and refine this. We have two options.

• Talk to the rest of the community to see if this can be added as part of HAPI & Android SDK.
• Build the functionality on the extensions we have, Add this to FHIRCore. This will mean needing to add a custom table to the SQLite table.

[pld]

Can't we add human readable IDs as separate identifiers? And not use them in the system internally at all.

…

On Mar 13, 2022, at 19:28, Benjamin Mwalimu ***@***.***> wrote:  Context Patients are assigned unique identifiers to help the health workers to track them easily. In OpenSRP we used to generate this and assign them to the patients during registration. Implementation requirements We need to come up with a similar if not improved functionality for the FHIR work. This unique identifier should have the following characteristics. Should be human-readable Should have the following items configurable by the implementations The identifier id -- Used to tell the mobile apps which identifier configuration to use. The generation algorithms to use Base character set to use for the identifiers Min & Max number of digits Prefix & Postfix to use on the identifers Any validation regex to act on the identifiers The identifiers should be saved as part of the FHIR identifiers on the patient resources. This should preferably be saved and the official identifier The identification of what identifier to use for each app should be part of the configurable items when loading up a new app track. The identifiers should be unique across the whole implementation, not just a facility, and the apps should not allow for duplicate identifier assignment or generation. @pld @f-odhiambo Based on the eCBIS forms we need to discuss and refine this. We have two options. Talk to the rest of community to see if this can be added as part of HAPI & Android SDK. Build the functionality on the extensions we have, Add this to FHIRCore. This will mean needing to add a custom table to the SQLite table. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[dubdabasoduba]

Can't we add human readable IDs as separate identifiers? And not use them in the system internally at all.
…
On Mar 13, 2022, at 19:28, Benjamin Mwalimu @.***> wrote:  Context Patients are assigned unique identifiers to help the health workers to track them easily. In OpenSRP we used to generate this and assign them to the patients during registration. Implementation requirements We need to come up with a similar if not improved functionality for the FHIR work. This unique identifier should have the following characteristics. Should be human-readable Should have the following items configurable by the implementations The identifier id -- Used to tell the mobile apps which identifier configuration to use. The generation algorithms to use Base character set to use for the identifiers Min & Max number of digits Prefix & Postfix to use on the identifers Any validation regex to act on the identifiers The identifiers should be saved as part of the FHIR identifiers on the patient resources. This should preferably be saved and the official identifier The identification of what identifier to use for each app should be part of the configurable items when loading up a new app track. The identifiers should be unique across the whole implementation, not just a facility, and the apps should not allow for duplicate identifier assignment or generation. @pld @f-odhiambo Based on the eCBIS forms we need to discuss and refine this. We have two options. Talk to the rest of community to see if this can be added as part of HAPI & Android SDK. Build the functionality on the extensions we have, Add this to FHIRCore. This will mean needing to add a custom table to the SQLite table. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

@pld That's the idea. To use the secondary identifier to hold the OpenSRP ID. We however need to define how to manage them. OpenSRP used to do the following.

• Auto-generate the Human readable IDs based on a certain algorithm to make them unique because this is what is printed on the health cards and used on manual search too,
• Marke the used IDs and re-generate and re-assign new ones to a device every time the locally stored IDs are almost complete.

To achieve the requirement above we need the following

• Figure out how to store the algorithm to be used for the autogeneration of the Human readable IDs
• Figure out how to mark them as used on the client and server.
• Figure out how to store them on the client for offline availability.
• Figure out how to pre-populate the IDs field on the questionnaire. Mostly the registration questionnaires
• Figure out how to search using the IDs both by the search bar and QR code search. The current APIs can search via an attribute so I think we just need to add the QR code search capabilities.

[dubdabasoduba]

Can't we add human readable IDs as separate identifiers? And not use them in the system internally at all.
…
On Mar 13, 2022, at 19:28, Benjamin Mwalimu @.***> wrote:  Context Patients are assigned unique identifiers to help the health workers to track them easily. In OpenSRP we used to generate this and assign them to the patients during registration. Implementation requirements We need to come up with a similar if not improved functionality for the FHIR work. This unique identifier should have the following characteristics. Should be human-readable Should have the following items configurable by the implementations The identifier id -- Used to tell the mobile apps which identifier configuration to use. The generation algorithms to use Base character set to use for the identifiers Min & Max number of digits Prefix & Postfix to use on the identifers Any validation regex to act on the identifiers The identifiers should be saved as part of the FHIR identifiers on the patient resources. This should preferably be saved and the official identifier The identification of what identifier to use for each app should be part of the configurable items when loading up a new app track. The identifiers should be unique across the whole implementation, not just a facility, and the apps should not allow for duplicate identifier assignment or generation. @pld @f-odhiambo Based on the eCBIS forms we need to discuss and refine this. We have two options. Talk to the rest of community to see if this can be added as part of HAPI & Android SDK. Build the functionality on the extensions we have, Add this to FHIRCore. This will mean needing to add a custom table to the SQLite table. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

We will not be using the Human Readable IDs on the system for anything. So we can even look at them as tags. Their sole use is to make it easy for the user the search the patients.

[pld]

How long can these IDs be, can be alpha numeric or only numeric? If it's a big enough search space we can generate them on demand without worrying about collision.

Do they need to be random, or can they contain some information, then if we want to keep them short we could have a 3 character prefix based on the device, then have them be sequential for the device, and not have to worry about collisions or pre-generation.

[dubdabasoduba]

They are mostly 8 characters long, They are alphanumeric and some have prefixes and postfixes depending on the client. Most of the time they are sequential i.e a device can receive 1-250 then the other receives 250 -500 this is with assumptions to the client is using the +ve integers from 1 - n

Yes, we can also make the random to make them more unique. I am not sure of how to make sure they do not collide in the different HF/CHW connected to the same server. Think this is while we let the server generate them.

I might need to do more R&D on how to avoid the collision while they are generated on the app.

[pld]

If each app that's generating them used a unique static prefix they won't collide. Alphanum is good, a 3 char prefix and 5 char random suffix should be enough to do it

…

On Mar 20, 2022, at 08:52, Benjamin Mwalimu ***@***.***> wrote:  They are mostly 8 characters long, They are alphanumeric and some have prefixes and postfixes depending on the client. Most of the time they are sequential i.e a device can receive 1-250 then the other receives 250 -500 this is with assumptions to the client is using the +ve integers from 1 - n Yes, we can also make the random to make them more unique. I am not sure of how to make sure they do not collide in the different HF/CHW connected to the same server. Think this is while we let the server generate them. I might need to do more R&D on how to avoid the collision while they are generated on the app. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[ndegwamartin]

I wonder if we can guarantee some uniqueness per client device deployment wide by generating a prefix from the server for every fresh install and letting the client manage it's own suffices.

How it would work.

Generate a prefix using the Douglas Crockford Base32 character set - it is basically an alpha numeric char. set that skips some characters that are prone to human error. See link.

• For a prefix of 3 over 29K permutations are possible (without repetition)
• For a prefix of 4 over 860K permutations are possible

Note: In order to cater for the number times users may install and uninstall an application we might want to use 4 for example it we do a national wide deployment. There's no way to track the device from the server so every fresh install has to be a new prefix allocation. It should not be a problem though since we DO NOT encourage uninstalling the app in Production

Advantages:

• Fresh installations(Initial syncs) are not done frequently so performance of the look up would be acceptable. No look up required if we store last generated and increment lexicographically.
• Each device can generate + manage its own suffix characters. If integers only it could be simple auto incrementing to avoiding collisions

[dubdabasoduba]

@allan-on @f-odhiambo please add the PHN requirements here.

[f-odhiambo]

High-level summary of Provide PHN generation algorithm - This will be an 11-digit patient identifier

1. First 4 alphanumerics - hard-coded for OpenSRP FHIR Core Diabetes Compass app
2. Next 6 alphanumerics specific to the patient
3. First 3 alphanumerics assigned to each TS
4. The last 3 are random and we need to verify they don’t already exist as IDs on the device
5. Final 1 digit, check digit

[pld]

For (1.) this will be from a set of numbers we are assigned
For (4.) we will definitely need to verify this

[allan-on]

Diabetes compass ID generation requirements

• For context, the Personal Health Number (PHN) is a unique 11-digit lifetime identifier for health care that’s assigned to individuals seeking or receiving health services from healthcare providers. This number is issued upon their first contact.
• The format for the PHN is
  • First 4 digits represent the issuing authority also known as the point of issue (POI) number→ The POI number is issued under the supervision of the MoH
  • The following 6 digits are random alphanumeric characters that don’t contain zero (0), O, one (1) or I
  • The last digit is a check digit for verifying the validity of the number
  • Point of issue ID (POI) 4 digit alphanumeric + 6 random alphanumeric (except O and zero) + check digit

1. Proposed workflow

• On logging in, the Trained Screener (TS) will sync any unsynced data that’s relevant to them from the Hapi Server
  • This data will include:
    • Data pushed from other Sri Lanka Health Info Management Systems to the Hapi Server
    • A batch of unique PHNs for use on the device
      • This implies that PHN generation happens server-side
• Then the TS will Search for the existence of a PHN on the device and on the Hapi FHIR Server (if the device is online)
  • The online search will include searching the subset of data from other health systems integrations
• If the PHN is not on the device and not on the server, the registration should take place with an assigned PHN from the reserved PHNs
  • Server-side PHN generation factors in already existing PHNs synced from other health information systems to avoid collisions and duplications

2. MVP workflow

• After a search, PHN generation is generated and assigned during registration
• PHN generation is decentralized and happens on the device
• The proposed generation implementation follows as mentioned above in a prior comment
  • 4 digit alphanumeric POI + 3 digit unique TS identifier + 3 random alphanumeric characters + 1 check digit
• Configurability
  • ID generation reconfigurability can take into account:
    • Prefix/Suffix
    • Length
    • Format
    • Character set → Define characters to be excluded during generation
    • Validation Algorithm

---

• Assumptions
  • The device will be mostly online
  • Once a patient is assigned a PHN they’re supposed to use it for their lifetime
  • Possible duplications will be handled/resolved server-side

cc @pld @f-odhiambo @Gental-Giant @dubdabasoduba

[pld]

Want to note that

• The part of the "Proposed workflow" that covers PHNs is not going to be worked on any more
• Patients will often have multiple PHNs, so we need to handle that use case

[allan-on]

In terms of ID gen configurations. Do we have consensus on what should be configurable?
From @dubdabasoduba 's initial description

Should have the following items configurable by the implementations

• The identifier id -- Used to tell the mobile apps which identifier configuration to use.
• The generation algorithms to use
• Base character set to use for the identifiers
• Min & Max number of digits
• Prefix & Postfix to use on the identifers
• Any validation regex to act on the identifiers

[pld]

what are some examples of the generation algorithms? How would we configure the WDF case where the ID is a composite or a hard-coded string in the configuration, a string tied to the practitioner and a random string?

[allan-on]

• In the WDF case we can concatenate the POI+Practitioner (TS) identifier to the pseudo-random string

• From a bit of research, there are a number of existing algorithms/generators that can be reused or implemented such as:

  1. UUID
  2. Sequential numbers converted to base 36 to convert them to alphanumeric https://stackoverflow.com/questions/19203877/good-algorithm-for-unique-random-generation-of-ids
  3. Snowflake: Snowflake is a distributed unique ID generator in Java inspired by Twitter Snowflake https://github.com/callicoder/java-snowflake (Updated 4 years ago)

It isn't clear to me what the inherent benefits of supporting multiple algorithms are in our current use case

[pld]

Yea, for WDF my questions was more like let's use this as an example to see how we'd write the configuration for this use case. My take is it'd be something like

prefix: the POI + a DSL that says get a field from the current logged in user (that's the Practitioner's identifier)
length: 3 (this is the random identifier)
postfix: the algorithm to generate the check-digit
unique: true

I don't think we need a min, max, IDs need to be a constant size usually always, I think that's a reasonable requirement so we can use length here and note that this is the length of the randomly generated part.

I also think we either need a flag for unique, or we always verify that they're unique, I guess we need a flag since ID doesn't imply unique. BUT if we have a flag for unique then we need to handle the case when it's impossible to generate an ID, because all the unique IDs are taken. This is kinda like a validation, but it can't be expressed as a regex, what were you thinking would go in validation?

All that said, interpreting that config is gonna be tough, mainly the prefix and postfix, I'm not sure how we put those in a config without adding a potentially complex DSL, @ellykits given what you've seen on the config work, am I being too pessimistic here?

I am OK, I mean not happy, but OK, with us punting on full configurability instead having the algo be somewhat project specific, like a "wdf" algo refers to a wdf function that says take this POI (or better pull this field from the app config, which is the POI), get a field from the current user and add that, then add a random string, then calculate the check digit, then search to make sure it's unique in the db repeating the last step until it is. I'm definitely not thrilled w/this but I could live with it for now

[allan-on]

Yeah, I agree with the sample config shared.
The validation would essentially be a regex that checks that the generated ID is in the expected format with the correct character set.
I think we should treat the IDs as inherently unique and always ensure that they are so.
To avoid cases where we have multiple looping to generate the random values then check whether the resultant ID already exists, we could alternatively have the IDs generated somewhat sequentially - For other apps we could concatenate with a device ID so that the sequence between devices remains unique? 🤔

[pld]

I don't understand the validation regex, wouldn't the algorithm deterministically only create IDs that are valid? In the sense of if you implement the algorithm, anything it generates is, by definition, valid.

I agree with the sentiments around inherent uniqueness, I can't think of a use case where you wouldn't want uniqueness at least relative to device. It just means we need to validate that the definition of what is a valid ID has the ability to generate enough IDs for the use case, and handle the situation when the available IDs are exhaust. When that happens I think the options are 1) create duplicate IDs, 2) throw an error and don't allow new entities to be created, 3) allow entities to be created but don't give them IDs, 4) change the algorithm to increase the space of IDs on the fly, like make the IDs longer -- these are all pretty bad options, not sure which one will be least bad, that's probably use case driven.

I think using a device idea is a reasonable configuration option we can give to people designing or configuration the algorithms, but not a good mandate since it impacts length.

[pld]

I want to note we found a use case that would require batch generation. If a project wants to use IDs that need to be verified as existing in an external system, for example a voucher you can redeem for services, they you need to have a pre-generated set of IDs so that when a user enters there ID somewhere else the system can say whether it is valid or not.

If we use on-demand generation we won't be able to say whether an ID the user reported is one that should exist.

This will matter if an implementation wants to verify the validity of IDs, if they only want to verify the structure of IDs we wouldn't need batch. Like perhaps it's enough to know that the ID presented matches the structure of potential IDs (eg correct length, correct check digit).

cc @roywanyaga