017. Unicode Strings

RAM Manager — Displaying Process Names

Aryan’s RAM manager reads process names from /proc on Linux. Some processes are started by users whose login names contain non-ASCII characters — accented names, CJK characters, or status messages with emoji. When Aryan displays a summary like:

Top 5 processes by RAM:
chrome      (👤 séb)   544 MB
python      (👤 李明)   398 MB

…the character-count column is wrong. He measured len(name.encode("utf-8")) instead of len(name), so "séb" reported 4 bytes instead of 3 characters, and "李明" reported 6 bytes instead of 2 characters. The columns don’t line up.

The lesson: In Python 3, strings are Unicode. len() counts characters, not bytes. To get byte counts, use .encode("utf-8"). To get character counts, use len() directly.

💡 Fun fact: Python 2 had two separate string types — str (raw bytes) and unicode (text). Python 3’s biggest breaking change in 2008 was merging them into a single str type that is always Unicode. This eliminated an entire category of UnicodeDecodeError bugs that plagued Python 2 codebases, especially in web applications handling international text.

⚠️ Watch out: len("👋".encode("utf-8")) returns 4 (bytes) but len("👋") returns 1 (character). Using byte length for column alignment or display width calculations will mis-align any output containing non-ASCII characters.

🤔 Think about it: Even `len()` counting characters is not always equal to the display width — some Unicode characters (like many CJK characters) are “full width” and occupy two terminal columns. What would you need beyond `len()` to perfectly align a table with mixed ASCII and CJK text?

Learning objectives

Understand that Python 3 strings are Unicode by default
Use len() to count characters (not bytes)
Use ord() and chr() for character/code point conversion

Key concepts

unicode
str
ord()
chr()
encoding

Try it

Concept detail

Python 3 strings are always Unicode (UTF-32 internally). len(“café”) == 4, not 5. Every character has a code point (integer). ord(‘A’) == 65, ord(‘👋’) == 128075. chr(65) == ‘A’ (reverse of ord). Unicode escapes: ‘\u00e9’ == ‘é’, ‘\U0001F44B’ == ‘👋’. To get bytes, encode: “hello”.encode(“utf-8”). To get back: b“hello“.decode(“utf-8”). Python 2 had str (bytes) and unicode (text) — Python 3 merged them as str (always Unicode).

Solution

def greeting_length(text):
    return len(text)

def contains_emoji(text):
    for char in text:
        if ord(char) > 65535:
            return True
    return False

Tests

def test_ascii_length():
    assert greeting_length("Hello") == 5

def test_emoji_length():
    # 👋 is 1 character (but 4 bytes in UTF-8)
    assert greeting_length("Hi 👋") == 4

def test_cjk_length():
    # Each CJK character is 1 char
    assert greeting_length("你好") == 2

def test_no_emoji():
    assert contains_emoji("Hello") == False

def test_has_emoji():
    assert contains_emoji("Hi 👋") == True