I/O #

One of the most elegant designs in Go is how the entire I/O ecosystem is built on two very simple interfaces: io.Reader and io.Writer. Files, HTTP request bodies, network connections, strings, byte buffers, gzip streams, database rows — everything can be treated the same because everything implements the same interfaces. This makes I/O code in Go highly composable: you can wrap a reader with another reader to add buffering, hashing, or decompression without changing the code that uses that reader.

The data flow from a source to a sink using the io.Reader and io.Writer interfaces through an intermediate buffer can be visualized in the following diagram:

flowchart LR
    Source["Data Source\n(File / Network / String)"] -->|"Implements"| Reader["io.Reader"]
    Reader -->|"Read(p []byte)"| Buffer["Temporary Buffer\n([]byte)"]
    Buffer -->|"Write(p []byte)"| Writer["io.Writer"]
    Writer -->|"Implements"| Sink["Final Destination\n(Stdout / File / Network)"]

io.Reader and io.Writer — The Foundation of Everything #

// The two interfaces that form the basis of all I/O in Go
type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

Read reads up to len(p) bytes into p and returns the number of bytes read. When there’s no more data, it returns io.EOF. Write writes len(p) bytes from p and returns the number of bytes successfully written.

What implements io.Reader:

*os.File, net.Conn, *http.Request.Body, *bytes.Buffer,
*bytes.Reader, *strings.Reader, *bufio.Reader,
*gzip.Reader, *zip.Reader, io.LimitedReader, ...

And io.Writer:

*os.File, net.Conn, http.ResponseWriter, *bytes.Buffer,
*bufio.Writer, *gzip.Writer, io.MultiWriter, os.Stdout, ...

The os Package — File Operations #

Reading Files #

import "os"

// The modern way (Go 1.16+) — most concise for small files
data, err := os.ReadFile("config.json")
if err != nil {
    return fmt.Errorf("failed to read config: %w", err)
}
fmt.Println(string(data))

// The manual way — more control, for large files or streaming
f, err := os.Open("data.txt")  // read-only
if err != nil {
    return err
}
defer f.Close()

buf := make([]byte, 4096)
for {
    n, err := f.Read(buf)
    if n > 0 {
        process(buf[:n])
    }
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
}

Writing Files #

// The modern way — write at once, most concise
err := os.WriteFile("output.txt", []byte("file contents\n"), 0644)

// The manual way with os.OpenFile — more control
f, err := os.OpenFile("output.txt",
    os.O_WRONLY|os.O_CREATE|os.O_TRUNC,  // flags
    0644)                                  // permissions
if err != nil {
    return err
}
defer f.Close()

_, err = f.WriteString("First line\n")
_, err = fmt.Fprintf(f, "Line %d\n", 2)

// Append to an existing file
f2, err := os.OpenFile("log.txt",
    os.O_WRONLY|os.O_CREATE|os.O_APPEND,
    0644)

os.OpenFile Flags #

os.O_RDONLY   — read only (the default of os.Open)
os.O_WRONLY   — write only
os.O_RDWR     — read and write
os.O_APPEND   — append to the end of the file
os.O_CREATE   — create the file if it doesn't exist
os.O_TRUNC    — empty the file if it already exists
os.O_EXCL     — error if the file already exists (atomic create)
os.O_SYNC     — write directly to disk (no OS cache)

File Information and Management #

// Check whether a file exists
if _, err := os.Stat("config.json"); os.IsNotExist(err) {
    fmt.Println("File not found")
}

// File info
info, err := os.Stat("data.txt")
if err == nil {
    fmt.Println("Name  :", info.Name())
    fmt.Println("Size  :", info.Size(), "bytes")
    fmt.Println("Modified:", info.ModTime())
    fmt.Println("IsDir :", info.IsDir())
}

// File/directory operations
os.Remove("temp.txt")
os.Rename("old.txt", "new.txt")
os.MkdirAll("path/to/dir", 0755)  // create directories recursively
os.RemoveAll("directory/")          // remove a directory and its contents

// List a directory
entries, err := os.ReadDir(".")
for _, entry := range entries {
    fmt.Printf("%-30s %v\n", entry.Name(), entry.IsDir())
}

The io Package — Utility Functions #

The io package provides functions that work generically with Reader and Writer.

io.Copy — Copy Data Between Streams #

// Copy all data from src to dst
n, err := io.Copy(dst, src)
fmt.Printf("Copied %d bytes\n", n)

// A real example: copy a file
src, _ := os.Open("source.txt")
defer src.Close()
dst, _ := os.Create("dest.txt")
defer dst.Close()
io.Copy(dst, src)

// Copy an HTTP response body to a file
resp, _ := http.Get("https://example.com/file.zip")
defer resp.Body.Close()
f, _ := os.Create("file.zip")
defer f.Close()
io.Copy(f, resp.Body)

io.ReadAll — Read All Data #

// Read the entire reader contents into memory
data, err := io.ReadAll(resp.Body)

// Be careful with very large streams!
// Use io.LimitReader to bound it
limited := io.LimitReader(resp.Body, 10*1024*1024)  // max 10MB
data, err := io.ReadAll(limited)

io.TeeReader — Read While Writing #

TeeReader reads from r and every byte read is also written to w simultaneously. Useful for hashing while reading:

import (
    "crypto/sha256"
    "encoding/hex"
)

// Calculate the SHA-256 of a file while reading it (single pass)
f, _ := os.Open("data.bin")
defer f.Close()

hasher := sha256.New()
tee := io.TeeReader(f, hasher)  // everything read from tee is also written to hasher

dst, _ := os.Create("copy.bin")
defer dst.Close()

io.Copy(dst, tee)  // reading from tee = reading from f + writing to hasher

hash := hex.EncodeToString(hasher.Sum(nil))
fmt.Println("SHA-256:", hash)

io.MultiReader — Combining Multiple Readers #

// Read from several sources as if they were one stream
r1 := strings.NewReader("header\n")
r2 := strings.NewReader("body content\n")
r3 := strings.NewReader("footer\n")

combined := io.MultiReader(r1, r2, r3)
io.Copy(os.Stdout, combined)
// Output:
// header
// body content
// footer

io.MultiWriter — Write to Several Destinations #

// Write to a file and stdout at the same time
f, _ := os.Create("log.txt")
defer f.Close()

mw := io.MultiWriter(os.Stdout, f)
fmt.Fprintln(mw, "This message appears in the terminal AND is saved to the file")

io.LimitReader — Limit the Number of Bytes Read #

// Prevent reading more than N bytes (upload security)
const maxBodySize = 1 << 20  // 1 MB
http.MaxBytesReader(w, r.Body, maxBodySize)

// Or manually
limited := io.LimitReader(source, maxBodySize)
data, _ := io.ReadAll(limited)

The bufio Package — Buffered I/O #

Reading or writing one byte or one line at a time directly to an os.File is very inefficient because every Read/Write is a system call. bufio adds a buffer on top of the reader/writer to reduce the number of system calls.

bufio.Reader #

f, _ := os.Open("data.txt")
defer f.Close()

reader := bufio.NewReader(f)           // default buffer of 4096 bytes
// or
reader = bufio.NewReaderSize(f, 65536) // 64KB buffer

// Read line by line
for {
    line, err := reader.ReadString('\n')
    if len(line) > 0 {
        fmt.Print(line)
    }
    if err == io.EOF {
        break
    }
    if err != nil {
        log.Fatal(err)
    }
}

// Peek — look at bytes without consuming them
peeked, _ := reader.Peek(5)
fmt.Println(string(peeked))  // the first 5 bytes without advancing the position

bufio.Scanner — The Idiomatic Way to Read Line by Line #

Scanner is more idiomatic than ReadString('\n') for line-by-line reading:

f, _ := os.Open("data.txt")
defer f.Close()

scanner := bufio.NewScanner(f)

// Default: split by line
for scanner.Scan() {
    line := scanner.Text()    // the line without \n
    fmt.Println(line)
}
if err := scanner.Err(); err != nil {
    log.Fatal(err)
}

// Read word by word
scanner2 := bufio.NewScanner(strings.NewReader("hello world foo"))
scanner2.Split(bufio.ScanWords)
for scanner2.Scan() {
    fmt.Println(scanner2.Text())  // hello, world, foo
}

// Read byte by byte
scanner3 := bufio.NewScanner(r)
scanner3.Split(bufio.ScanBytes)

// Custom buffer for long lines
scanner4 := bufio.NewScanner(f)
buf := make([]byte, 1024*1024)  // 1MB buffer
scanner4.Buffer(buf, len(buf))  // set the max size

bufio.Writer #

f, _ := os.Create("output.txt")
defer f.Close()

writer := bufio.NewWriter(f)

writer.WriteString("First line\n")
fmt.Fprintf(writer, "Line %d\n", 2)
writer.WriteByte('\n')

// IMPORTANT: flush the buffer to disk before closing the file
if err := writer.Flush(); err != nil {
    log.Fatal(err)
}
// Data in the buffer will be lost if Flush is not called!
Always call Flush() on a bufio.Writer before the file is closed. Unflushed data is still in the memory buffer and hasn’t been written to disk. Use defer writer.Flush() right after creating the writer, before defer f.Close().

bytes.Buffer and strings.Builder — In-Memory I/O #

When you need a reader/writer in memory (not a file or network):

import "bytes"

// bytes.Buffer — can be used as both a Reader and a Writer
var buf bytes.Buffer

buf.WriteString("Hello")
buf.WriteString(", ")
fmt.Fprintf(&buf, "World %d!", 42)

fmt.Println(buf.String())   // "Hello, World 42!"
fmt.Println(buf.Len())      // length in bytes

// Read back from the buffer
data := make([]byte, 5)
buf.Read(data)  // reads the first 5 bytes

// Reset for reuse
buf.Reset()

// bytes.NewReader — a reader from a byte slice (immutable)
reader := bytes.NewReader([]byte("Hello, Go!"))
io.Copy(os.Stdout, reader)

// strings.NewReader — a reader from a string
reader2 := strings.NewReader("Hello from string!")
io.Copy(os.Stdout, reader2)

// strings.Builder — efficient string writing
var sb strings.Builder
sb.WriteString("part 1")
sb.WriteString(" and ")
sb.WriteString("part 2")
result := sb.String()  // no string allocation on every +=

io.Pipe — In-Process Pipes #

io.Pipe creates a connected PipeReader and PipeWriter pair — data written to the writer is immediately available to the reader, like a Unix pipe:

pr, pw := io.Pipe()

// Writer — run in a goroutine
go func() {
    defer pw.Close()  // signal EOF to the reader
    for i := 0; i < 5; i++ {
        fmt.Fprintf(pw, "line %d\n", i+1)
        time.Sleep(100 * time.Millisecond)
    }
}()

// Reader — in the main goroutine
scanner := bufio.NewScanner(pr)
for scanner.Scan() {
    fmt.Println("Received:", scanner.Text())
}

// Useful for: encoding data while uploading
// without keeping all the data in memory
pr2, pw2 := io.Pipe()
go func() {
    defer pw2.Close()
    encoder := json.NewEncoder(pw2)
    encoder.Encode(largeData)  // encode straight into the pipe
}()
http.Post(url, "application/json", pr2)  // upload from the pipe

stdin, stdout, stderr #

// os.Stdin, os.Stdout, os.Stderr are *os.File
// all of them implement io.Reader and io.Writer

// Read from stdin
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
    fmt.Println("You typed:", scanner.Text())
}

// Read a single line from stdin
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter your name: ")
name, _ := reader.ReadString('\n')
name = strings.TrimSpace(name)

// Write to stderr (for errors and logs)
fmt.Fprintln(os.Stderr, "Error: something went wrong")

// Redirect stdout to a file
f, _ := os.Create("output.txt")
oldStdout := os.Stdout
os.Stdout = f
fmt.Println("This goes to the file!")  // to the file, not the terminal
os.Stdout = oldStdout        // restore the terminal
f.Close()

Temporary Files and Directories #

// Create a temporary file — automatically gets a unique name
tmpFile, err := os.CreateTemp("", "prefix-*.txt")
if err != nil {
    log.Fatal(err)
}
defer os.Remove(tmpFile.Name())  // remove after done
defer tmpFile.Close()

fmt.Println("Temp file:", tmpFile.Name())  // /tmp/prefix-123456789.txt
tmpFile.WriteString("temporary data")

// Create a temporary directory
tmpDir, err := os.MkdirTemp("", "myapp-*")
if err != nil {
    log.Fatal(err)
}
defer os.RemoveAll(tmpDir)

fmt.Println("Temp dir:", tmpDir)  // /tmp/myapp-123456789

fs.FS — Filesystem Abstraction (Go 1.16+) #

fs.FS is an interface for filesystems that lets code work uniformly with real filesystems, embedded filesystems, or in-memory filesystems:

import (
    "embed"
    "io/fs"
)

//go:embed static/*
var staticFiles embed.FS

// A function accepting fs.FS — works with any filesystem
func processFiles(fsys fs.FS) error {
    return fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }
        if d.IsDir() {
            return nil
        }
        f, err := fsys.Open(path)
        if err != nil {
            return err
        }
        defer f.Close()

        data, err := io.ReadAll(f)
        if err != nil {
            return err
        }
        fmt.Printf("%s: %d bytes\n", path, len(data))
        return nil
    })
}

func main() {
    // Use with embedded files
    processFiles(staticFiles)

    // Use with a real filesystem
    processFiles(os.DirFS("."))

    // Use with a sub-directory
    subFS, _ := fs.Sub(staticFiles, "static")
    processFiles(subFS)
}

Complete Example Program #

The following program builds a log processor pipeline that reads logs from a file, filters, transforms, and writes to output:

package main

import (
    "bufio"
    "compress/gzip"
    "crypto/md5"
    "encoding/hex"
    "fmt"
    "io"
    "os"
    "strings"
    "time"
)

type LogEntry struct {
    Timestamp string
    Level     string
    Message   string
    Raw       string
}

func parseLog(line string) (LogEntry, bool) {
    // Format: 2024-07-28 15:30:45 [INFO] message here
    parts := strings.SplitN(line, " ", 4)
    if len(parts) < 4 {
        return LogEntry{}, false
    }
    level := strings.Trim(parts[2], "[]")
    return LogEntry{
        Timestamp: parts[0] + " " + parts[1],
        Level:     level,
        Message:   parts[3],
        Raw:       line,
    }, true
}

// processLogs reads logs, filters, and writes to output
func processLogs(
    input io.Reader,
    output io.Writer,
    minLevel string,
    stats *struct{ total, filtered, written int },
) error {
    // MultiWriter: write to output and calculate MD5 at the same time
    hasher := md5.New()
    mw := io.MultiWriter(output, hasher)

    writer := bufio.NewWriter(mw)
    defer writer.Flush()

    levelPriority := map[string]int{
        "DEBUG": 0, "INFO": 1, "WARN": 2, "ERROR": 3, "FATAL": 4,
    }
    minPriority := levelPriority[minLevel]

    scanner := bufio.NewScanner(input)
    // Set a large buffer for long lines
    buf := make([]byte, 256*1024)
    scanner.Buffer(buf, len(buf))

    for scanner.Scan() {
        line := scanner.Text()
        stats.total++

        entry, ok := parseLog(line)
        if !ok {
            continue
        }

        // Filter by level
        if levelPriority[entry.Level] < minPriority {
            stats.filtered++
            continue
        }

        // Transform: add a prefix and reformat
        formatted := fmt.Sprintf("[%s] %-5s | %s\n",
            entry.Timestamp, entry.Level, entry.Message)
        fmt.Fprint(writer, formatted)
        stats.written++
    }

    if err := scanner.Err(); err != nil {
        return fmt.Errorf("error reading log: %w", err)
    }

    // Flush before calculating the final hash
    writer.Flush()
    fmt.Fprintf(os.Stderr, "Output checksum: %s\n",
        hex.EncodeToString(hasher.Sum(nil)))

    return nil
}

func main() {
    // Create sample logs in memory
    sampleLogs := `2024-07-28 08:00:01 [DEBUG] Starting application
2024-07-28 08:00:02 [INFO] Server running on port 8080
2024-07-28 08:01:15 [DEBUG] Incoming request: GET /health
2024-07-28 08:01:15 [INFO] Health check: OK
2024-07-28 08:05:30 [WARN] Memory usage 75%
2024-07-28 08:10:45 [ERROR] Database connection timeout
2024-07-28 08:10:46 [INFO] Attempting to reconnect to the database
2024-07-28 08:10:47 [INFO] Reconnect successful
2024-07-28 08:15:00 [DEBUG] Garbage collection finished
2024-07-28 08:20:00 [FATAL] Disk full, cannot write logs`

    stats := &struct{ total, filtered, written int }{}

    fmt.Println("=== Log Processor Pipeline ===")
    fmt.Println()

    // Demo 1: Filter to stdout (only WARN and above)
    fmt.Println("--- Log Level WARN and above ---")
    reader1 := strings.NewReader(sampleLogs)
    if err := processLogs(reader1, os.Stdout, "WARN", stats); err != nil {
        fmt.Fprintln(os.Stderr, "Error:", err)
    }

    fmt.Printf("\nStatistics: total=%d, filtered=%d, written=%d\n\n",
        stats.total, stats.filtered, stats.written)

    // Demo 2: Write to a plain file
    stats2 := &struct{ total, filtered, written int }{}
    outFile, _ := os.CreateTemp("", "processed-*.log")
    defer os.Remove(outFile.Name())
    defer outFile.Close()

    fmt.Printf("--- Writing to file: %s ---\n", outFile.Name())
    reader2 := strings.NewReader(sampleLogs)
    processLogs(reader2, outFile, "INFO", stats2)
    fmt.Printf("Written %d entries to the file\n\n", stats2.written)

    // Demo 3: Write to a gzip file using io.Writer composition
    fmt.Println("--- Writing to a gzip file ---")
    gzFile, _ := os.CreateTemp("", "compressed-*.log.gz")
    defer os.Remove(gzFile.Name())
    defer gzFile.Close()

    stats3 := &struct{ total, filtered, written int }{}
    gzWriter := gzip.NewWriter(gzFile)
    defer gzWriter.Close()

    reader3 := strings.NewReader(sampleLogs)
    processLogs(reader3, gzWriter, "DEBUG", stats3)  // send to the gzip writer!
    gzWriter.Close()

    gzInfo, _ := os.Stat(gzFile.Name())
    fmt.Printf("All %d entries compressed to: %s (%d bytes)\n",
        stats3.written, gzFile.Name(), gzInfo.Size())

    // Demo 4: TeeReader — read while duplicating
    fmt.Println("\n--- TeeReader: read while duplicating ---")
    var duplicate strings.Builder
    original := strings.NewReader("important data that needs to be duplicated\n")
    tee := io.TeeReader(original, &duplicate)

    // Read from tee (data also goes into duplicate)
    mainData, _ := io.ReadAll(tee)
    fmt.Printf("Main data    : %q\n", string(mainData))
    fmt.Printf("Duplicate    : %q\n", duplicate.String())

    // Demo 5: io.Pipe — stream without a buffer
    fmt.Println("\n--- io.Pipe: in-process stream ---")
    pr, pw := io.Pipe()
    done := make(chan struct{})

    go func() {
        defer pw.Close()
        defer close(done)
        ticker := time.NewTicker(100 * time.Millisecond)
        defer ticker.Stop()
        for i := 0; i < 3; i++ {
            <-ticker.C
            fmt.Fprintf(pw, "stream event %d\n", i+1)
        }
    }()

    scanner := bufio.NewScanner(pr)
    for scanner.Scan() {
        fmt.Println("Received from pipe:", scanner.Text())
    }
    <-done
}

Summary #

  • io.Reader and io.Writer are the foundation of all I/O in Go — anything readable or writable implements them.
  • os.ReadFile / os.WriteFile for small files read/written at once; os.Open / os.Create for more control or large files.
  • bufio.Scanner is the idiomatic way to read line by line; cleaner than ReadString('\n').
  • bufio.Writer — always call Flush() before the file is closed; use defer writer.Flush().
  • io.Copy copies data between streams without loading it all into memory.
  • io.TeeReader reads while duplicating to another writer (e.g. hashing while downloading).
  • io.MultiWriter writes to several destinations at once (e.g. file + stdout).
  • io.LimitReader bounds the number of bytes read — important for upload security.
  • io.Pipe connects a writer and reader in different goroutines without a memory buffer.
  • fs.FS (Go 1.16+) abstracts filesystems — code can work with real, embedded, or memory filesystems.

← Previous: Goroutines   Next: Sockets →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact