Streaming I/O

Part of: Reef Language Reference Last reviewed on version: 0.8.0

Reef provides three levels of file I/O, each suited to different use cases. Choosing the right one can dramatically reduce memory usage and GC pressure.

Result/Option migration (Phase C): io.file, io.buffer, and io.stream functions that can fail now return Result[T, core.error.Error] instead of a sentinel ("", -1, false). Reads in particular return Result[Option[T], Error]: Err means the read failed, Ok(Some(data)) means data was read, and Ok(None) means clean EOF. Import the helpers you need:

import core.result as result
import core.option as option
import core.error as error

When to Use Each Approach

Rule of thumb: If you're reading a file larger than 64KB, or processing it line-by-line, use io.buffer.BufferedReader instead of readFile().

GC Impact

The key difference is heap allocation. Reef's collector marks with the world stopped, then reclaims lazily — memory is freed incrementally inside the allocator as later allocations look for blocks, not in one pass at the end of a collection. So "GC pressure" here means two distinct costs, and neither is a long sweep pause:

• More marking work. Every live object is visited during the stop-the-world mark, so a larger live set lengthens the pause.
• More reclamation work spread across later allocations. Garbage is paid for by whichever allocation happens to need a block, so heavy churn shows up as allocation cost rather than as a single stall.

Reducing allocation reduces both.

• readFile() + str.split(): Creates N+1 heap objects (file string + N line strings). If called in a loop, all become garbage simultaneously.

• BufferedReader.read_line(): Reuses an internal buffer. Only the returned line string is heap-allocated. Previous lines become garbage incrementally.

• sys.fd read/write: Minimal heap allocation — each fd_read() call allocates exactly one string sized to the bytes actually read (there is no per-line splitting, and no intermediate buffer array).

Example: Allocation Comparison

Processing a 1000-line file 100 times:

The readFile approach also keeps the entire file content alive until the enclosing scope exits, while the buffered reader only keeps one line at a time.

Line-by-Line Processing

Before (high allocation)

import io.file
import core.str
import core.result as result

fn count_matches(path: string, pattern: string): int
    let content_r = file.readFile(path)
    if result.is_err(content_r)
        return 0
    end if
    let content = result.unwrap_ok(content_r)
    // str.split takes a char delimiter and writes into a caller-provided
    // array; it returns the part count, not the array itself.
    mut lines = new [string](1000)
    let n = str.split(content, '\n', lines, 1000)
    mut count = 0
    mut i = 0
    while i < n
        if str.contains(lines[i], pattern)
            count = count + 1
        end if
        i = i + 1
    end while
    return count
end count_matches

After (streaming)

import io.buffer
import core.str
import core.result as result
import core.option as option

fn count_matches(path: string, pattern: string): int
    let open_r = buffer.open_buffered_reader(path)
    if result.is_err(open_r)
        return 0
    end if
    let reader = result.unwrap_ok(open_r)
    mut count = 0
    while true
        let line_r = buffer.reader_read_line(reader)
        if result.is_err(line_r)
            break
        end if
        let line_opt = result.unwrap_ok(line_r)
        if option.is_none(line_opt)
            break  // EOF
        end if
        let line = option.unwrap(line_opt)
        if str.contains(line, pattern)
            count = count + 1
        end if
    end while
    buffer.reader_close(reader)
    return count
end count_matches

Writing with Auto-Flush

import io.buffer
import core.result as result

proc write_report(path: string, lines: [string], count: int)
    let open_r = buffer.open_buffered_writer(path)
    if result.is_err(open_r)
        return
    end if
    let writer = result.unwrap_ok(open_r)
    mut i = 0
    while i < count
        if result.is_err(buffer.writer_write_line(writer, lines[i]))
            break
        end if
        i = i + 1
    end while
    buffer.writer_close(writer)   // flushes remaining buffer
end write_report

Chunked Binary Reading

For binary files or when you need fixed-size chunks:

import sys.fd
import core.str

fn read_file_chunks(path: string, chunk_size: int): int
    let f = fd.fd_open(path, fd.O_RDONLY(), 0)
    if f < 0
        return -1
    end if
    mut total = 0
    // fd_read(fd, max_len) returns up to max_len bytes as a freshly
    // allocated string; it does not take a caller-provided buffer. It
    // returns "" on EOF or error, so length-zero ends the loop.
    mut chunk = fd.fd_read(f, chunk_size)
    while str.length(chunk) > 0
        total = total + str.length(chunk)
        chunk = fd.fd_read(f, chunk_size)
    end while
    fd.fd_close(f)
    return total
end read_file_chunks

Common Anti-Patterns

Repeated file reads in a loop

// BAD: Reads and parses /etc/passwd 5000 times
mut i = 0
while i < 5000
    let name = uid_to_name_via_readfile(entries[i].uid)
    i = i + 1
end while

// GOOD: Use the stdlib function (zero allocation, uses POSIX getpwuid)
import fs.stat
import core.option as option
mut i = 0
while i < 5000
    let name = option.unwrap_or(stat.uid_name(entries[i].uid), "unknown")
    i = i + 1
end while

stat.uid_name/gid_name return Option[string] — None when the UID/GID can't be resolved to a name (e.g. no matching /etc/passwd entry), rather than an empty string.

Large file into string for single-value extraction

// BAD: Reads entire 10MB file to find one line
let content = result.unwrap_ok(file.readFile("/var/log/large.log"))
let lines = str.split(content, "\n")
let last = lines[lines.length() - 1]

// GOOD: Stream to the end, keep only last line
let reader = result.unwrap_ok(buffer.open_buffered_reader("/var/log/large.log"))
mut last = ""
while true
    let line_r = buffer.reader_read_line(reader)
    if result.is_err(line_r)
        break
    end if
    let line_opt = result.unwrap_ok(line_r)
    if option.is_none(line_opt)
        break  // EOF
    end if
    last = option.unwrap(line_opt)
end while
buffer.reader_close(reader)

See Also

• io.file — simple file read/write for small files
• io.buffer — buffered reader/writer for streaming
• sys.fd — raw file descriptor operations
• fs.stat — file metadata and uid/gid resolution