Module: encoding.utf8

Source: ./encoding/utf8.reef

Overview

encoding/utf8 - UTF-8 encoding and codepoint utilities

UTF-8 is a variable-width character encoding that can represent every character in the Unicode standard. It uses 1-4 bytes per character:

0xxxxxxx (1 byte, U+0000 - U+007F)
110xxxxx 10xxxxxx (2 bytes, U+0080 - U+07FF)
1110xxxx 10xxxxxx 10xxxxxx (3 bytes, U+0800 - U+FFFF)
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx (4 bytes, U+10000 - U+10FFFF)

Usage: import encoding.utf8 let bytes: int = utf8_char_len('a') // Returns 1 let bytes2: int = utf8_codepoint_len(0x20AC) // Euro sign, returns 3 let valid: bool = is_utf8_start_byte(0xC2) // Returns true

Functions

`fn utf8_alloc_buffer(): [int]`

Allocate a buffer for UTF-8 encoding (8 bytes max for any codepoint + length)

`fn is_ascii(b: int): bool`

Check if byte is ASCII (single byte character)

`fn is_utf8_start_byte(b: int): bool`

Check if byte is a UTF-8 start byte (not a continuation)

`fn is_utf8_continuation_byte(b: int): bool`

Check if byte is a UTF-8 continuation byte

`fn utf8_char_len(c: char): int`

Get the length in bytes of a UTF-8 character from its first byte

`fn utf8_codepoint_len(codepoint: int): int`

Get the length in bytes needed to encode a Unicode codepoint

`fn utf8_encode_codepoint(codepoint: int, buf: [int]): int`

Encode a Unicode codepoint to UTF-8 bytes, returns bytes written

`fn utf8_decode_codepoint(buf: [int], offset: int): result.Result[int, error.Error]`

Decode a UTF-8 codepoint from buffer at offset.

Ok(codepoint) on a well-formed sequence; Err(InvalidInput) on any of: an invalid lead byte, a malformed continuation byte, an overlong encoding, a surrogate codepoint (U+D800..U+DFFF), or a codepoint beyond U+10FFFF. This is a full RFC-3629 validating decode — it does NOT merely inspect the lead byte and trust the continuation bytes.

`fn utf8_decode_codepoint_len(buf: [int], offset: int, out_len: [int]): result.Result[int, error.Error]`

Decode a UTF-8 codepoint from buffer at offset, also reporting its byte length via the out-param out_len.

Same validation contract as utf8_decode_codepoint: Ok(codepoint) with out_len[0] set to the number of bytes consumed (1/2/3/4) on success; Err(InvalidInput) on invalid lead byte, malformed continuation byte, overlong encoding, surrogate codepoint, or out-of-range codepoint. out_len is left untouched on Err.

`fn utf8_strlen(s: string): int`

Count the number of UTF-8 characters in a string

`fn utf8_byte_count(s: string): int`

Get the byte count of a string (same as C strlen)

`fn is_valid_utf8_byte(b: int, pos_in_char: int, expected_len: int): bool`

Check if a byte is valid at its position in a UTF-8 character

`fn is_valid_codepoint(codepoint: int): bool`

Check if a codepoint is valid Unicode

`fn utf8_validate(s: string): bool`

Validate that a string contains a well-formed UTF-8 sequence. Returns false on overlong encodings, surrogates, out-of-range codepoints, truncated multi-byte sequences, invalid lead bytes, or bad continuations.

Generated by reefc doc