0.1 Bits, bytes and encodings

Bits and bytes

A bit is a single 0 or 1. Eight bits make a byte, and eight bits can represent $2^8 = 256$ different values, the integers 0 to 255. Everything a computer stores, from a photo to a private key, is ultimately a sequence of bytes.

Cryptographic sizes are quoted in both units, so it pays to convert quickly. A 256-bit key is 32 bytes. A SHA-256 digest is 32 bytes. An AES block is 128 bits, which is 16 bytes. An ML-KEM-768 public key is 1,184 bytes.

Writing numbers: binary and hex

We usually write numbers in base 10. Computers store them in base 2. The number 3329, the modulus used by ML-KEM, is 110100000001 in binary. That is hard to read, so programmers use hexadecimal (base 16), with the digits 0–9 and a–f. Each hex digit stands for exactly 4 bits, so every byte is exactly two hex digits: 3329 is 0xd01.

Byte order (endianness)

A number larger than 255 needs several bytes, and there are two ways to order them. Big-endian puts the most significant byte first, the way we write digits. Little-endian puts the least significant byte first, which is how x86 and ARM processors store integers in memory. Network protocols and most cryptographic standards say explicitly which order they use; getting it wrong gives a different number and a different hash.

n = 3329
print(bin(n), hex(n))
print(n.to_bytes(4, "big").hex())     # most significant byte first
print(n.to_bytes(4, "little").hex())  # least significant byte first
print(int.from_bytes(bytes.fromhex("010d0000"), "little"))

0b110100000001 0xd01
00000d01
010d0000
3329

Text is not bytes: ASCII and UTF-8

Text has to be turned into bytes before any algorithm can touch it. That translation is a character encoding. The oldest one, ASCII, maps 128 characters (English letters, digits, punctuation) to the values 0–127, one byte each: A is 0x41.

ASCII has no room for é, ж, 中 or emoji. UTF-8, the encoding used by almost the whole web, keeps ASCII unchanged and spends more bytes on everything else: 2 bytes for most accented and Cyrillic letters, 3 for most other symbols, and 4 for emoji.

So the six-character string café 🔑 is 10 bytes long. Hashing or encrypting a string always means hashing or encrypting its bytes, and two programs only agree on the result if they agree on the encoding.

Writing bytes as text: hex and Base64

Raw bytes are awkward to handle. Many byte values are not printable characters, and some (like 0) break text formats, logs and JSON. So bytes are usually written down in a text-safe form:

• Hex writes each byte as two characters. It is easy to read and compare, and doubles the size: 32 bytes become 64 characters. Hashes and keys in documentation are usually shown in hex.
• Base64 writes every 3 bytes as 4 characters from an alphabet of 64 (A–Z a–z 0–9 + /), padding the end with =. It is about 33% larger than the raw bytes, more compact than hex, and is used in PEM files, JSON Web Tokens, email attachments and data URLs.
• Base64url is the same thing with - and _ instead of + and /, so the result is safe inside URLs and file names. JWTs use it, usually without the = padding.

import base64

text = "café 🔑"
data = text.encode("utf-8")          # str -> bytes
print(len(text), "characters,", len(data), "bytes")
print(data.hex())                    # bytes -> hex text
print(base64.b64encode(data))        # bytes -> Base64 text
print(base64.urlsafe_b64encode(data))

# Anyone can reverse it: no key involved
print(base64.b64decode("Y2Fmw6kg8J+UkQ==").decode("utf-8"))

6 characters, 10 bytes
636166c3a920f09f9491
b'Y2Fmw6kg8J+UkQ=='
b'Y2Fmw6kg8J-UkQ=='
café 🔑

Encoding is not encryption

This is the single most common beginner confusion in security. A password stored as Base64, a token that is “just hex”, or a config value that is “obfuscated” with Base64 are all readable by anyone who sees them. If a value must stay secret, it needs encryption; if it must not be altered, it needs a MAC or a signature; and passwords need a slow password hash, never an encoding.

Why cryptography works on bytes

Hash functions, ciphers and signatures are defined on byte strings (or on numbers built from bytes) so that every implementation, in every language, produces exactly the same output for the same input. That is why the first step in almost any cryptographic code is choosing the bytes precisely: which text encoding, which byte order, and whether a value is raw bytes or its hex or Base64 spelling.

import hashlib
digest = hashlib.sha256(b"hello").digest()
print(len(digest), "raw bytes")
print(len(digest.hex()), "hex characters")
print(digest.hex()[:16] + "...")

32 raw bytes
64 hex characters
2cf24dba5fb0a30e...

The next page, 0.2 Hash functions, starts from exactly this kind of byte string.

Common mistakes

• Mixing text and bytes. In Python, "6b6579".encode() is six bytes of hex characters, while bytes.fromhex("6b6579") is the three bytes b"key". Passing the wrong one to a hash or cipher silently changes the result.
• Confusing lengths. A 32-byte key is 64 hex characters or 44 Base64 characters. Code that checks len(key) == 32 on the hex string rejects valid keys, or accepts a 16-byte key written in hex.
• Double encoding. Base64-encoding a value that is already Base64 (a2V5 becomes YTJWNQ==) produces something that decodes once to the wrong bytes.
• Padding and alphabet mismatches. Standard Base64 and Base64url differ in two characters, and many decoders reject input whose = padding was stripped. Decide on one variant at each boundary.
• Unstated byte order. Serialize integers with an explicit order (for example to_bytes(4, "big")) rather than relying on the machine's native order.

Further reading

• RFC 4648, The Base16, Base32, and Base64 Data Encodings (defines hex, Base64 and Base64url).
• RFC 3629, UTF-8, a transformation format of ISO 10646.
• David Wong, Real-World Cryptography (Manning, 2021), chapter 1, for how bytes, keys and encodings fit together in practice (cited from memory).