TESID

^† The types used here (Project/Task/User) are just simple examples. You will choose your own types, and each type will have a number (discriminant) associated with it.

TESID converts numbers into short random strings.

If you have centralised ID allocation and don’t actively want to expose your ID sequence, TESID lets you store sequential numeric identifiers in your database, but expose cryptographically‐secure pseudorandom but fairly short strings to users, avoiding disclosing the original ID sequence, which can be a valuable information leak.

TESID is commonly an alternative to UUIDs, which are massive overkill and very unwieldy in centralised ID allocation scenarios.

The strings TESID produces are designed to be human‐friendly: quite short, not mixed case, and with typical confusables (1/l, 0/o) removed. This allows people to usefully interact with the IDs, by typing, handwriting, speech and more.

The algorithm

The TESID encoding algorithm takes a non‐negative integer ID, and turns it into a string following these three steps:

• First, if desired, it does type discrimination and sparsification, by multiplying by a sparsity factor and adding a discriminant. (This allows you to avoid TESID reuse or confusion across types, and make valid TESIDs harder to guess.)

• Secondly, it scrambles the number with a real cryptographic block cipher (a slight variant of Speck) within certain ranges:

  • a 20‐bit block size for values 0 to 2²⁰ − 1 (that is, a number in the range 0–1,048,575 will become some number in the range 0–1,048,575),
  • 30‐bit for values 2²⁰ to 2³⁰ − 1 (so a number in the range 1,048,576–1,073,741,823 will become some number in the range 0–1,073,741,823),
  • 40‐bit for values 2³⁰ to 2⁴⁰ − 1,
  • &c. for 50‐bit, 60‐bit, 70‐bit, 80‐bit, 90‐bit and 100‐bit.

• Thirdly, it converts the scrambled number to a string using base conversion with a base‐32 alphabet, with leading padding to the appropriate length for its range: 4 characters for 20‐bit values, 6 for 30‐, 8 for 40‐, &c. until 20 for 100‐bit values.

Decoding is a fairly straightforward reversal of encoding.

The end result of this technique is that you get nice short IDs for as long as is possible, but avoid exposing the numeric sequence. (In the absence of sparsity and discrimination, you’ll get about a million four‐character TESIDs, a billion six‐, a trillion eight‐, and so on.)

See the algorithms page for a more detailed description.

This sample code is written in Python, as it has the prettiest syntax. But you can do just the same in both Rust and TypeScript.

# --- First, simple usage (using the key from above!) ---
from tesid import TESIDCoder

secret_key = '000102030405060708090a0b0c0d0e0f'
coder = TESIDCoder(secret_key)

assert coder.encode(0)          == 'w2ej'
assert coder.encode(1)          == 'w6um'
assert coder.encode(2)          == 'x45g'
assert coder.encode(2**20 - 1)  == 'atcw'
assert coder.encode(2**20)      == '8qwm6y'
assert coder.encode(2**30 - 1)  == '3eipc7'
assert coder.encode(2**30)      == 'n3md95r4'
assert coder.encode(2**100 - 1) == 'ia2bvpjaiju7g5uaxn5t'
# coder.encode(2**100) would raise ValueError.

assert coder.decode('w2ej') == 0


# --- Second, convenient typed usage ---
from tesid import TypedTESIDCoder, SplitDecode
from enum import Enum

class Type(Enum):
    Project = 0
    Task    = 1
    User    = 2

typed_coder = TypedTESIDCoder(coder, 256, Type)

assert typed_coder.encode(Type.Project, 0) == 'w2ej'
assert typed_coder.encode(Type.Task,    0) == 'w6um'
assert typed_coder.encode(Type.User,    0) == 'x45g'
assert typed_coder.encode(Type.Project, 1) == 'dh2h'
assert typed_coder.encode(Type.Task,    1) == 'a6xy'
assert typed_coder.encode(Type.User,    1) == '7xgj'
assert typed_coder.decode(Type.Project, 'w2ej') == 0
# typed_coder.decode(Type.Task, 'w2ej') would raise ValueError.
assert typed_coder.split_decode('w2ej') == \
	SplitDecode(id=0, discriminant=Type.Project)

So you think you’d like to use TESID? Here’s a set of steps for getting started.

1. Read about the considerations when choosing TESID. There are a couple of things you should be aware of that may make you decide TESID is not suitable for your situation.
2. Select a TESID library. Currently, implementations are provided for Rust, Python and JavaScript, each under the name tesid in the usual package repository; see above for more details.
3. Decide whether and how to use sparsity and discrimination (explained in these linked sections of the more and design rationale documents). In very brief form: you should use these if you want to use TESID on more than one type (use sparsity and discriminant), if you want valid TESIDs to be less guessable (use large sparsity), or if you want to use the probabilistic solution to undesirable patterns (use moderate to large sparsity).
4. Generate a key. It should probably be stored as a secret in a config file or environment variable. Ideally use different keys for different environments.
5. Hook up TESID encoding and decoding everywhere you expose IDs.
6. Go forth and prosper.

TESID uses a 128‐bit key for its cryptography; libraries take this as a 32‐character big‐endian lowercase hexadecimal string.

This key should be randomly generated. Here are a few command‐line techniques you can use:

openssl rand -hex 16

(Requires OpenSSL.)

python -c 'import secrets; print(secrets.token_hex(16))'

(Requires Python 3.6+.)

</dev/random head -c 16 | hexdump -e '4 "%08x" "\n"'

(Should run in any typical POSIX environment, I think—​Linux, macOS, BSD, &c.)

node -e 'console.log(crypto.getRandomValues(new Uint8Array(16)).reduce((s, b) => s + b.toString(16).padStart(2, "0"), ""))'

(Requires Node.js 15+. The JavaScript works in browsers too, and is what the “Generate new random key” button in the demo above does, though as a matter of security principle you should not use my button without vetting all the code running on my site: wiser to run it locally on your own terms.)

This page is an overview to TESID, general information and introduction. I have three more pages that are worth reading if you’re interested in the problem space, understanding more when TESID is appropriate, what alternatives there are, &c.:

• More general information: a miscellaneous bag of information about TESID, the supporting research, and other related discussions.
• Algorithms: prose descriptions of the TESID encoding and decoding algorithms, including the pieces that make them up.
• Design rationale: some analysis of the reasoning behind TESID and specific design choices.

There’s some definite overlap between the pages, but also a lot of non‐overlap.

Other project links:

• Code is in the Git repository at https://git.chrismorgan.info/tesid.
• It’s also mirrored on GitHub (but remember, Git is decentralised; you don’t need no GitHub).
• For bug reports, queries and consultation, you can email me.
• For notifications, here’s a feed.

I’m Chris Morgan, and I made TESID. Go to chrismorgan.info for more information about me, more things I’ve written, &c.

I’m available for hire, for consulting, training and mentoring in matters Rust, web, performance and more, and general contract work. (You can also sponsor me via GitHub or Liberapay if you’d like. I would appreciate it.)