💼 localDataStorage

TL;DR

Directly use in localStorage with no conversion.
 

Highlights

This is a synchronous JavaScript interface for the HTML5 localStorage API that--

1. transparently sets/gets key values using data "types" such as Array, BigInt, Boolean, Date, Float, Integer, Object and String;
2. provides lightweight data obfuscation;
3. intelligently compresses strings (to save storage space);
4. facilitates robust lookup including query by key (name), query by (key) value and query by existence (boolean check);
5. enforces segmented shared storage within the same domain by prefixing keys;
6. lets you respond to localStorage change events on the same page/tab that fired them;
7. broadcasts change events across the origin for the benefit of other windows/tabs;
8. lets you easily work with arrays using dedicated Array Keys; and
9. offers Memory Keys (that can be backed up to disk) for the fastest read times possible.

 
Version 3.0.0
Author: W. “Mac” McMeans
Date: 12 AUG 2022
 

Trusted Installation

Include via CDN

<script
    src="https://cdn.jsdelivr.net/gh/macmcmeans/localDataStorage@master/localDataStorage-3.0.0.min.js" 
    integrity="sha512-dEhk3bL90qpWkcHCJDErHbZEY7hGc4ozmKss33HSjwMeSBKBtiw/XVIE7tb5u+iOEp6dTIR9sCWW7J3txeTQIw==" 
    crossorigin="anonymous"
></script>

Since the source file is delivered from a repository over which we have no control, best practices demand we do not trust it. Thus we use Subresource Integrity (SRI) to prevent tampering.

Unlike the BrowseAloud story, my library is a static, self-contained resource which can be fully protected by SRI.
 

Application:

Primary usage is the ability to seamlessly set/get keys for typical data types without having to perform conversion in your own logic. Toss out an integer and have it returned. Throw an array into storage and get it back, including the ability to add and remove elements. Are you also working with dates, booleans or objects? No problem. While it’s not rocket science to convert, track and restore your own data, letting the interface handle that chore for you is exceptionally convenient. It also saves you the hassle having to incorporate conversion logic in your application. To track a data type, storage management requires one additional byte of memory overhead per key value, so an eight-byte array, for example, will be stored using nine bytes.

Data Protection
Since localStorage data resides in a client environment where the information is not protected from access or tampering, this data can be rendered unintelligible at the expense of a small amount of memory overhead. Depending on your application, this may be worth the tradeoff. To hide your data, key values may be obfuscated using safeset/safeget. A master scramble key may be set globally, or individual scramble keys may be used per each safeset/safeget call. Scramble keys can be any value and of any type (array, boolean, date, float, integer, etc.) Key values that have been safeset with an individual scramble key can always be retrieved, but cannot be reconstructed apart from the exact scramble key with which they were obfuscated. For convenience, the global scramble key is stored in the interface. For security, individual scramble keys are not. The global scramble key may be accessed using setscramblekey/getscramblekey methods. Individual scramble keys must be remembered.

Scrambling is not encryption. For example, no attempt is made to conceal data lengths by artificially padding to a minimum length. This would be counterproductive to minimizing memory usage.

Compression
Strings are intelligently compressed on‑the‑fly when storing. This means they are first analyzed to determine whether compression would lower the actual byte count (not string length) needed for storage, and if so, are silently compressed for you. This works well for common English texts (short‑length, 7‑bit ASCII), and not much else. If desired, you may manually crunch/uncrunch your own key values.

Robust Access
One may query by key (the standard way) or query by existence (haskey checks if the key is in the store while hasval checks if the store contains the value). You can query for duplicate values with listdupes and showdupes. Convenience methods prevent writing over an existing key (softset) and permit deleting a key immediately upon retrieval (chopget). You can easily query and update an array key using push/pull and contains without the need for external array logic. Key values can be checked for their data type, for example, using isfloat, and keys can be renamed (rename). Lastly, bypass methods (forceset/forceget) permit accessing localStorage directly.

Storage Management
Since HTML5 localStorage is accessible to all processes running in the browser for the domain visited, it is advisable to have an interface that segments access. To that end, the use of prefixed keys is strongly encouraged, and localDataStorage will only read/write/delete its own values. Unlike the HTML5 API, there is no method in the interface to clear all storage keys, only all prefixed keys. At any time, memory usage can be tracked against key values and key names, for example, using valbytes or keybytes.

The domain of operation for HTML5 localStorage is specific to the protocol, host & port; and multiple instances of localDataStorage can be run against the same domain at the same time. It is emoji‑friendly 🤪🤷‍♂️💖👍, which is to say that key names and their values, as well as scramble keys and storage prefixes, are multibyte Unicode‑safe.

Events

The native localStorage change event is... lacking. Per the misguided whims of the interweb gods, a web page in your browser isn’t permitted to listen to change events that it alone triggers. However, in the event you'd like to keep an ear out for changes, localDataStorage will let you. The interface fires a custom event on key value changes, such as those made by the chopget, clear, forceset, remove, rename, safeset, set and softset methods. The event returns an activity timestamp and message, as well as expected details about the affected key with its old and new values (and old and new data types, etc.) Insert your own function here to catch the changes. This code snippet shows what's exposed so you can respond accordingly:

const nowICanSeeLocalStorageChangeEvents = function( e ) {
    console.log(
        "subscriber: " + e.currentTarget.nodeName + "\n" +
        "date: "       + e.detail.date            + "\n" +
        "timestamp: "  + e.detail.timestamp       + "\n" +
        "prefix: "     + e.detail.prefix          + "\n" +
        "message: "    + e.detail.message         + "\n" +
        "method: "     + e.detail.method          + "\n" +
        "old key: "    + e.detail.oldkey          + "\n" +
        "new key: "    + e.detail.newkey          + "\n" +
        "old value: "  + e.detail.oldval          + "\n" +
        "new value: "  + e.detail.newval          + "\n" +
        "old type: "   + e.detail.oldtype         + "\n" +
        "new type: "   + e.detail.newtype         + "\n" +
        "old base: "   + e.detail.oldbase         + "\n" +
        "new base: "   + e.detail.newbase
    );

    // respond to key change by passing key name
    myCustomChangeFunction( e.detail.oldkey ); 
}

document.addEventListener(
    "localDataStorage"
    , nowICanSeeLocalStorageChangeEvents
    , false
);

 

Dependencies:

There are no external dependencies.

Internally, obfuscation is supported by fisherYatesDurstenfeldKnuthShuffle and aleaPRNG (my own libraries).
 

SemVer:

Recognizing there is no way I can predict how a change in my software will affect users, I still use semantic versioning (semver) to signal those changes.

Per Hyrum’s Law, I gently suggest you may enjoy, but I do not promise you will actually have a bug-free upgrade experience.
 

Falsy pedantics:

There is a universe of difference between a variable considered null and one that is undefined. In JavaScript, both of these are falsy and both are primitive types (despite what typeof null tells you). Yet they are also quite distinct.

A value of null means the variable has been set to nothing whatsoever, but it has been set nevertheless.

In contrast, an undefined variable has no value because the variable itself doesn't exist (it is undeclared).

With null, the variable is set to no value but if undefined, it isn't set. The distinction is important. The native localStorage API returns null when getting a key that doesn't exist. (Pedantically, this is false.)

However, localDataStorage returns undefined for an undeclared key simply because it isn't present in the store. Further, you could actually store a null value key using localDataStorage if your use case required, but at no time will non-existant keys be returned as nulls. The distinction is just. Too. Important.
 

Wiki:

Here's the documentation for the interface (95% complete).
 

Example usage:

Create an instance of localDataStorage using the specified key name prefix

localData = localDataStorage( 'passphrase.life' )

--> Instantiated. Prefix adds 16.00 bytes to every key name (stored using 32.00 bytes).

 
typical set/get calls (data types are respected and returned transparently)

localData.set( 'key1', 19944.25 )

localData.get( 'key1' )

--> 19944.25

localData.set( 'key2', 2519944 )

localData.get( 'key2' )

--> 2519944

localData.set( 'key3', true )

localData.get( 'key3' )

--> true

localData.set( 'key4', 'data' )

localData.get( 'key4' )

--> "data"

localData.set( 'key5', [1,2,3,4,9] )

localData.get( 'key5' )

--> [1, 2, 3, 4, 9]

localData.set( 'key6', new Date() )

localData.get( 'key6' )

--> Mon May 01 2017 14:39:11 GMT-0400 (Eastern Daylight Time)

localData.set( 'key7', {'a': [1,2,3] } )

localData.get( 'key7' )

--> Object {a: Array(3)}

 
get the "size" of a key's value (codepoints)

localData.size( 'key4' )

--> 4
total codepoints in value (not length, not graphemes)

 
results when querying a non-existing key

localData.forceget( 'non-existing key' )

--> null
same as localStorage.getItem( 'non-existing key' )

localData.get( 'non-existing key' )

--> undefined
the key is undefined because it does not exist, it is NOT null

localData.chopget( 'non-existing key' )

--> undefined

localData.safeget( 'non-existing key' )

--> undefined

 
read then delete a key

x = localData.chopget( 'key7' )

--> Object {a: Array(3)}

localData.get( 'key7' )

--> undefined

 
don't overwrite an existing key

localData.softset( 'key4', 'new data' )

localData.get( 'key4' )

--> "data"

 
set/get key, bypassing any data type embedding, but still observing key prefixes

localData.forceset( 'api', 13579 )

all values are stored as strings, in this case under the key passphrase.life.api

localData.forceget( 'api' )

--> "13579"

localData.forceget( 'key6' )

--> ""2017-05-01T18:39:11.443Z""

 
find duplicate key values

localData.set( 'key8', 'data' )

now key4 and key8 have the same values

localData.countdupes()

--> 1

localData.showdupes()

--> ["data"]
this key value occurs twice minimum

 
// handling duplicates; localData vs localStorage API

localData.forceset( 'dupekey1', 1234 )

will be stored as a string

localData.forceset( 'dupekey2', '1234' )

will be stored as a string

 
// look for duplicates (among localStorage keys)

localData.showdupes()

--> [1234, "data"]

 
// remove a key

localData.remove( 'dupekey1' )

prep

localData.remove( 'dupekey2' )

prep

localData.remove( 'key8' )

prep

 

localData.set( 'dupekey3', 1234 )

stored as string, but recognized as integer

localData.set( 'dupekey4', '1234' )

stored and recognized as string

 
// look for duplicates (among localData types)

localData.showdupes()

--> []
since data types are respected, no dupes were found

 

localData.set( 'dupekey1', 1234 )

prep

localData.set( 'dupekey2', '1234' )

prep

localData.set( 'key8', 'data' )

prep

 

localData.countdupes()

--> 3

localData.listdupes()

--> Object {dupecount: 3, dupes: Object}

localData.listdupes().dupecount

--> 3

localData.listdupes().dupes

--> Object {0: Object, 1: Object, 2: Object}

localData.listdupes().dupes[0]

--> Object {value: 1234, keys: Array(2)}

localData.listdupes().dupes[0].value

--> 1234

localData.listdupes().dupes[0].keys

--> ["dupekey1", "dupekey3"]

 
check if key exists

localData.haskey( 'dupekey3' )

--> true

 
check if value exists

localData.hasval( 1234 )

--> true
checks value AND data type

 

localData.set( 'testkey', 89.221 )

prep

localData.hasval( '89.221' )

--> false
the float (number) type does not match the string type

 

localData.forceset( 'LSkey1', 98765 )

set key value using localStorage API (handled as string)

localData.forcehasval( 98765 )

--> true

localData.forcehasval( '98765' )

--> true
localStorage API does not discern between data types

 

localData.hasval( 98765 )

--> true
localData attempts to coerce any value not explicity set by it

localData.hasval( '98765' )

--> false
localData will first coerce a value to a number, if possible

 
show key's value type

localData.showtype( 'dupekey3' )

--> "integer"

localData.showtype( 'dupekey4' )

--> "string"

localData.showtype( 'key1' )

--> "float"

localData.showtype( 'key3' )

--> "boolean"

localData.showtype( 'key5' )

--> "array"

localData.showtype( 'key6' )

--> "date"

localData.set( 'key7', {'local' : ['d', 'a', 't', 'a']} )

prep

localData.showtype( 'key7' )

--> "object"

 
boolean check the data type of a key's value

localData.isarray( 'key5' )

--> true

localData.isfloat( 'testkey' )

--> true

localData.isnumber( 'testkey' )

--> true

 
query by key value, not key name (returns first found)

localData.showkey( 1234 )

--> "dupekey1"

localData.showkey( '1234' )

--> "dupekey2"

 
// returns all found

localData.showkeys( 1234 )

--> ["dupekey1", "dupekey3"]

 
using the global scramble key for obfuscation

localData.getscramblekey()

--> 123456789
default global scramble key (integer)

localData.safeset( 'ss1', '007' )

--> (stored scrambled)

localData.safeget( 'ss1' )

--> "007"

 

localData.setscramblekey( new Date() )

// set global scramble key to the current date, as date object

localData.getscramblekey()

--> Mon May 01 2017 22:28:11 GMT-0400 (Eastern Daylight Time)

 

localData.safeget( 'ss1' )

--> (garbled data)
different global scramble key used for retrieval

 
// using an individual scramble key for obfuscation

localData.safeset( 'ss2', 'test', {'scramble': ['key']} )

--> (stored scrambled)
scramble keys can be any value and of any data type

localData.safeget( 'ss2', {'scramble': ['key']} )

--> "test"

 

localData.safeget( 'ss1', 123456789 )

-> "007"

 
safeget will not retrieve an unscrambled key

localData.safeget( 'key4' )

--> (garbled data)

 
renaming keys // non-scambled keys can safely be renamed

localData.rename( 'key4', 'key4-renamed' )

key4 no longer exists

localData.get( 'key4' )

--> undefined

localData.get( 'key4-renamed' )

--> "data"

 
// scrambled keys cannot be renamed: the key name and the value together produce the obfuscation

localData.rename( 'ss1', 'ss1-renamed' )

key ss1 no longer exists

localData.safeget( 'ss1' )

--> undefined

localData.safeget( 'ss1-renamed', 123456789 )

--> (garbled data)
this was the correct scramble key for the 'ss1' key, but not for the 'ss1-renamed' key

 

localData.rename( 'ss1-renamed', 'ss1' )
key ss1-renamed no longer exists

localData.safeget( 'ss1-renamed' )

--> undefined

localData.safeget( 'ss1', 123456789 )

--> "007"

 
how localDataStorage reacts to values set via the localStorage API

localData.forceset( 'lsAPIkey', 77.042 )

always stored as a string by the native API

localData.forceget( 'lsAPIkey' )

--> "77.042"

localData.get( 'lsAPIkey' )

--> 77.042
localData will coerce value to number when possible

localData.showtype( 'lsAPIkey' )

--> "presumed number"
('presumed' because value was coerced, not set)

 
there are several ways to track memory usage

// show memory required to store key value

localData.showtype( 'dupekey4' )

--> "string";

localData.get( 'dupekey4' )

--> "1234"

localData.size( 'dupekey4' )

--> 4

localData.valbytes( 'dupekey4' )

--> "8.00 bytes"
localStorage uses 16 bits to store 1 byte (only the data is counted)

localData.valbytesall( 'dupekey4' )

--> "12.00 bytes"
now we include the 2-byte embedded marker (total data)

 
// show memory required to store key name

localData.keybytes( 'dupekey4' )

--> "48.00 bytes"
the prefix ('passphrase.life' + '.') is 32 bytes, plus key name is 16 bytes more ('dupekey4' ), yielding 48 bytes

 
// show memory used by the key-value pair

// key name + raw value

localData.bytes( 'dupekey4' )

--> "56.00 bytes"
8 bytes for raw value and 48 bytes for name, i.e. valbytes + keybytes

 
// key name + total value (include value marker byte)

localData.bytesall( 'dupekey4' )

--> "60.00 bytes"
now includes the embedded data type marker (it's 2 bytes, stored as 4)

 
view memory usage of compressed key values

localData.set( 'crunchedkey', 'this is some test data' )

only strings can be compressed; other data types will ignore compression

localData.size( 'crunchedkey' )

--> 22

localData.valbytes( 'crunchedkey' )

--> "44.00 bytes"
memory used to store raw string of 22 graphemes (each is 7-bit ASCII)

localData.valbytesall( 'crunchedkey' )

--> "34.00 bytes"
total memory required to store compressed string + embedded data type marker

 
unicode-safe data storage

localData.set( 'unicodeKey1', '😀' )

storing an emoji; 1 grapheme (1 codepoint in 4 bytes)

localData.get( 'unicodeKey1' )

--> "😀"

localData.size( 'unicodeKey1' )

--> 1; one codepoint

localData.valbytes( 'unicodeKey1' )

--> "8.00 bytes"

localData.valbytesall( 'unicodeKey1' )

--> "12.00 bytes"

 

localData.set( 'unicodeKey2', '🕔🔚🔈🔔♅' )

storing 5 graphemes (5 codepoints in 19 bytes)

localData.get( 'unicodeKey2' )

--> "🕔🔚🔈🔔♅"

localData.size( 'unicodeKey2' )

--> 5

localData.valbytes( 'unicodeKey2' )

--> "38.00 bytes"

localData.valbytesall( 'unicodeKey2' )

--> "42.00 bytes"

 
// using emojis for key name, key value and individual scramble key

localData.safeset( '👊🌐🔷', '💕🚻', '🔙' )

localData.safeget( '👊🌐🔷', '🔙' )

--> "💕🚻"

 
// using emojis in the global scramble key

localData.setscramblekey( '🎵🎶🔶🔻' )

localData.safeset( 'Ron Wyden', '.@NSAGov 💻📱📡📞🔎👂👀🔚 #EndThisDragnet' )

localData.safeget( 'Ron Wyden' )

--> ".@NSAGov 💻📱📡📞🔎👂👀🔚 #EndThisDragnet"

 
get tally of keys

localData.keys()

--> 24

 
delete all prefixed keys in the domain (unprefixed localStorage keys are not affected)

localStorage.setItem( 'API-key', 'test data' )

create a key in the same domain completely outside our instance of localDataStorage

localData.clear()

--> "24 keys removed"

localStorage.getItem( 'API-key' )

--> "test data"
any unprefixed localStorage keys are untouched

localData.safeget( 'Ron Wyden' )

--> undefined
all localData keys have been removed

REFS:

https://github.com/macmcmeans/aleaPRNG

https://github.com/macmcmeans/fisherYatesDurstenfeldKnuthShuffle
 

Tested:

Chrome(ium) browsers (blink engine) and FireFox (gecko) on Win 10 (x64).
 

Version notes:

• 3.0.0 - 12 AUG 2022
  release Major release. Added major features, fixed numerous bugs and refactored code.
  patch Consolidated private references to the localStorage.removeItem() method.
  feature Added length property.
  patch Fixed bug where clear() method did not remove all prefixed keys.
  feature Added import/export methods.
  feature Adjusted logic that computes memory storage requirements for keys.
  patch Improved the logic for string compression checks.
  patch Streamlined error checking for localStorage availability prior to instantiation.
  feature Added key copy method.
  patch Improved the efficiency of storage flags by 50%.
  patch Strengthened the obfuscation of data protected by the safeset method.
  major feature Added Memory Keys (and associated methods).
  major feature Added Array Keys (and associated methods).
  major feature Added ability to broadcast change events across the origin.
  feature On startup, the library computes the browser's quota.
  feature Added channel property.
  feature Added backup/restore methods.
  feature Dates, floats and integers are now automatically stored compressed.
  update Revised the wiki.
   
  

• 2.0.1 - 12 MAY 2022
  patch Fixed minor bugs, light refactoring.
  patch Updated the internal custom event to show both the date and the timestamp.
  update Revised the wiki.
   
  

• 2.0.0 - 1 MAY 2022
  release Major release. Added support for BigInt. Fixed numerous bugs and refactored code.
   
  

• 1.3.1 - 21 FEB 2022
  patch Fixed a bug where the Clear() method did not fire an event upon deletion of keys.
  patch Corrected logic in the Rename() method.
  patch Updated the internal custom event to display old key name and new key name.
  update Revised the wiki.
   
  

• 1.3.0 - 4 MAY 2020
  feature Implement optional flag useful on instantiation to mute console messages warning of the existence of previous keys and the space required for prefix storage.
   
  

• 1.2.0 - 19 JUN 2017
  update Checks whether localStorage is available and if not, gracefully fails when called. This means that all methods will simply return false instead of nasty type errors.
   
  

• 1.1.0 - 17 MAY 2017
  feature Add ability to listen to key value change events (in same window/tab).
   
  

• 1.0.0 - 15 MAY 2017
  release Initial release.
   
  

License (BSD)

Copyright (c) 2017-2022, W. "Mac" McMeans
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.