IO #

Almost every meaningful program needs to interact with the outside world — reading config files, writing logs, receiving data from the network, or processing user input. Go designs its I/O system on top of two very simple interfaces: io.Reader and io.Writer. The entire Go I/O ecosystem is built on these two interfaces — files, network connections, in-memory buffers, compressors, encryption, everything is mutually compatible because everything implements the same interfaces. This article covers how the Go I/O system works from the ground up, how to use the io, bufio, and os packages effectively, and patterns commonly used in production applications.

The Two Core Interfaces: Reader and Writer #

Before diving into more specific functions and structs, it’s important to understand the two interfaces that form the foundation of all I/O in Go. Both are defined in the io package and are very minimal — each has only one method.

// io.Reader — a data source that can be read
type Reader interface {
    Read(p []byte) []byte (n int, err error)
}

// io.Writer — a data destination that can be written to
type Writer interface {
    Write(p []byte) (n int, err error)
}

Read fills the slice p with data, returning the number of bytes successfully read and an error if any. When the data source runs out, Read returns io.EOF. Write writes the contents of slice p to the destination, returning the number of bytes successfully written.

flowchart LR
    A[Data Source\nos.File / net.Conn\nstrings.Reader\nbytes.Buffer] -- "Read(p []byte)" --> B[io.Reader]
    B -- "data flows" --> C[Your code]
    C -- "Write(p []byte)" --> D[io.Writer]
    D -- "Write(p []byte)" --> E[Data Destination\nos.File / net.Conn\nbytes.Buffer / os.Stdout]

The power of this design is composability. A function that accepts an io.Reader can accept a file, a network connection, an in-memory string, or another process’s output — without needing to know the source. This is the principle that makes Go code very easy to test and compose.

// This function works with ALL data sources
func countLines(r io.Reader) (int, error) {
    buf := make([]byte, 32*1024)
    count := 0
    lineSep := []byte{'\n'}

    for {
        c, err := r.Read(buf)
        count += bytes.Count(buf[:c], lineSep)
        if err == io.EOF {
            break
        }
        if err != nil {
            return count, err
        }
    }
    return count, nil
}

// Can be called with a file...
f, _ := os.Open("data.txt")
defer f.Close()
n, _ := countLines(f)

// ...or an in-memory string (useful for testing)
n, _ = countLines(strings.NewReader("line 1\nline 2\nline 3\n"))

The io Package — Basic Utility Functions #

The io package doesn’t just define interfaces — it also provides important utility functions for working with readers and writers.

io.Copy — Streaming Data Between Streams #

io.Copy is the most used function: it reads from src (a Reader) and writes to dst (a Writer) until EOF, returning the number of bytes transferred.

import (
    "io"
    "os"
)

// Copy a file
src, err := os.Open("source.txt")
if err != nil {
    log.Fatal(err)
}
defer src.Close()

dst, err := os.Create("destination.txt")
if err != nil {
    log.Fatal(err)
}
defer dst.Close()

n, err := io.Copy(dst, src)
fmt.Printf("Successfully copied %d bytes\n", n)

io.Copy internally uses a 32KB buffer — efficient for large files because it doesn’t read the whole file into memory at once. This is an important difference from os.ReadFile, which reads the entire file contents.

sequenceDiagram
    participant Copy as io.Copy
    participant Src as src (Reader)
    participant Buf as Buffer (32KB)
    participant Dst as dst (Writer)

    loop until EOF
        Copy->>Src: Read(buf)
        Src-->>Buf: n bytes of data
        Copy->>Dst: Write(buf[:n])
        Dst-->>Copy: n bytes written
    end
    Copy-->>Copy: return total, nil

io.ReadAll — Reading the Entire Content into Memory #

io.ReadAll reads all data from a reader until EOF and returns it as []byte. Only use this if you genuinely need the whole content in memory at once.

import (
    "io"
    "strings"
)

r := strings.NewReader("Hello from Go!")
data, err := io.ReadAll(r)
if err != nil {
    log.Fatal(err)
}
fmt.Println(string(data)) // "Hello from Go!"

Don’t use io.ReadAll to read files or HTTP responses with unpredictable sizes. If the content is large (hundreds of MB), all of it will be loaded into RAM. Use io.Copy with an appropriate destination, or process the data streaming with a buffer.

io.ReadFull — Reading Exactly N Bytes #

io.ReadFull reads exactly len(buf) bytes from the reader. If there isn’t enough data, it returns the io.ErrUnexpectedEOF error. Useful for reading data in binary formats with known sizes.

// Read an 8-byte header from a binary protocol
header := make([]byte, 8)
n, err := io.ReadFull(r, header)
if err == io.ErrUnexpectedEOF {
    fmt.Printf("Data truncated, only %d bytes available\n", n)
} else if err != nil {
    log.Fatal(err)
}
// header is now guaranteed to contain exactly 8 bytes

io.MultiReader and io.MultiWriter #

io.MultiReader combines several readers into one — data is read from the first reader until EOF, then continues to the next. io.MultiWriter forwards every write to all writers simultaneously (like tee in Unix).

// MultiReader — combine several sources
r1 := strings.NewReader("First part. ")
r2 := strings.NewReader("Second part. ")
r3 := strings.NewReader("Third part.")

combined := io.MultiReader(r1, r2, r3)
data, _ := io.ReadAll(combined)
fmt.Println(string(data))
// "First part. Second part. Third part."

// MultiWriter — write to many destinations at once
var buf bytes.Buffer
multi := io.MultiWriter(os.Stdout, &buf)

fmt.Fprintln(multi, "this message goes to stdout AND the buffer")
fmt.Println("In the buffer:", buf.String())

io.TeeReader — Read While Forwarding #

io.TeeReader wraps a reader so that every piece of data read is also automatically written to another writer. Useful for logging, debugging, or calculating checksums while processing data.

// Read an HTTP response body while logging its content
var logBuf bytes.Buffer
tee := io.TeeReader(resp.Body, &logBuf)

// Process the data from tee (also copied into logBuf)
var result JSONResult
json.NewDecoder(tee).Decode(&result)

// logBuf now contains a copy of the response for debugging
log.Printf("Response body: %s", logBuf.String())

io.LimitReader — Limiting the Amount of Data Read #

io.LimitReader wraps a reader and ensures at most N bytes can be read. Very important for security — prevents users from uploading files too large and exhausting server memory.

// ANTI-PATTERN: read the whole body without limits
body, err := io.ReadAll(r.Body)  // could be hundreds of MB!

// CORRECT: limit the size that may be read
const maxBodySize = 10 << 20 // 10 MB
limited := io.LimitReader(r.Body, maxBodySize)
body, err := io.ReadAll(limited)
if err != nil {
    http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
    return
}

io.Pipe — Connecting a Writer to a Reader #

io.Pipe creates a connected PipeWriter and PipeReader pair — anything written to the writer can be read directly from the reader. Data isn’t stored in a buffer, so writes and reads must happen concurrently.

pr, pw := io.Pipe()

// Writer goroutine
go func() {
    defer pw.Close()
    for i := 0; i < 5; i++ {
        fmt.Fprintf(pw, "data #%d\n", i)
    }
}()

// Reader goroutine (or a function accepting an io.Reader)
scanner := bufio.NewScanner(pr)
for scanner.Scan() {
    fmt.Println("Received:", scanner.Text())
}

io.Pipe is very useful for connecting the output of an encoder (e.g. gzip.Writer) directly to an input expecting an io.Reader — without needing an intermediate buffer in memory.

The os Package — Working with Files #

The os package provides functions for opening, creating, reading, and writing files on the system. os.File implements io.Reader, io.Writer, and io.Seeker all at once.

Opening and Reading Files #

import (
    "fmt"
    "io"
    "os"
)

// os.Open — open a file for reading (read-only)
f, err := os.Open("data.txt")
if err != nil {
    log.Fatal(err)
}
defer f.Close() // always defer Close after a successful Open

// Read the entire content (suitable for small files)
data, err := io.ReadAll(f)
if err != nil {
    log.Fatal(err)
}
fmt.Println(string(data))

For small files whose entire content genuinely needs to be loaded, os.ReadFile is more concise — it opens the file, reads the content, and closes it in one call:

// os.ReadFile — open, read, close in one step
data, err := os.ReadFile("config.json")
if err != nil {
    log.Fatal(err)
}
fmt.Println(string(data))

Creating and Writing Files #

// os.Create — create a new file (or truncate if it exists)
f, err := os.Create("output.txt")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

f.WriteString("First line\n")
f.Write([]byte("Second line\n"))
fmt.Fprintf(f, "Line %d\n", 3)

// os.WriteFile — write to a file in one step
err = os.WriteFile("concise.txt", []byte("file content"), 0644)

os.OpenFile — Full Control over the Open Mode #

When os.Open (read-only) and os.Create (write, truncate) aren’t enough, use os.OpenFile with appropriate flags.

Flag	Meaning
`os.O_RDONLY`	Open read-only
`os.O_WRONLY`	Open write-only
`os.O_RDWR`	Open for reading and writing
`os.O_CREATE`	Create the file if it doesn’t exist
`os.O_TRUNC`	Truncate the file when opened
`os.O_APPEND`	Append data at the end of the file
`os.O_EXCL`	Error if the file already exists (for atomic creation)

// Open a log file for appending, create it if it doesn't exist
f, err := os.OpenFile("app.log",
    os.O_APPEND|os.O_CREATE|os.O_WRONLY,
    0644,
)
if err != nil {
    log.Fatal(err)
}
defer f.Close()

fmt.Fprintln(f, "New log entry")

// Open for read-write without erasing the content
f2, err := os.OpenFile("data.bin", os.O_RDWR, 0644)

Seek — Navigating Inside a File #

os.File implements io.Seeker, allowing you to move the read/write position inside a file.

f, _ := os.Open("data.txt")
defer f.Close()

// Read the first 10 bytes
buf := make([]byte, 10)
f.Read(buf)
fmt.Println(string(buf))

// Back to the beginning of the file
f.Seek(0, io.SeekStart)

// Jump to 100 bytes from the end of the file
f.Seek(-100, io.SeekEnd)

// Move forward 50 bytes from the current position
f.Seek(50, io.SeekCurrent)

File Information with os.Stat #

info, err := os.Stat("data.txt")
if err != nil {
    if os.IsNotExist(err) {
        fmt.Println("File not found")
    } else {
        log.Fatal(err)
    }
}

fmt.Println("Name:", info.Name())
fmt.Println("Size:", info.Size(), "bytes")
fmt.Println("Mode:", info.Mode())
fmt.Println("Last modified:", info.ModTime())
fmt.Println("Directory?", info.IsDir())

The bufio Package — Buffered I/O #

Every Read or Write call on an os.File is a system call — expensive when done for many small pieces of data. bufio wraps readers/writers with an in-memory buffer, grouping many small operations into a few large system calls.

flowchart TD
    A[Your code\nmany small Writes] --> B{Using\nbufio?}

    B -- No --> C[Every Write\n= 1 system call]
    B -- Yes --> D[bufio.Writer\n4096-byte buffer]
    D -- "Only when the buffer is full\nor Flush is called" --> E[1 large Write\n= 1 system call]
    C --> F[Many system calls\nslow performance]
    E --> G[Few system calls\noptimal performance]

bufio.NewReader — Reading with a Buffer #

import (
    "bufio"
    "os"
    "fmt"
)

f, _ := os.Open("data.txt")
defer f.Close()

reader := bufio.NewReader(f)

// ReadString — read until a delimiter
line, err := reader.ReadString('\n')
if err != nil && err != io.EOF {
    log.Fatal(err)
}
fmt.Print(line)

// ReadLine — read one line (without extra allocation)
line, isPrefix, err := reader.ReadLine()
// isPrefix = true if the line is too long and was truncated

// Peek — look at the next N bytes without advancing the position
head, _ := reader.Peek(4)
fmt.Printf("First 4 bytes: %q\n", head)

// ReadByte and UnreadByte — read/return one byte
b, _ := reader.ReadByte()
reader.UnreadByte() // return the byte to the buffer

bufio.Scanner — The Most Idiomatic Way to Read Lines #

bufio.Scanner is the recommended way to read text line by line. It’s cleaner and safer than manually using ReadString.

// ANTI-PATTERN: reading lines with manual ReadString
reader := bufio.NewReader(f)
for {
    line, err := reader.ReadString('\n')
    if err == io.EOF {
        break
    }
    if err != nil {
        log.Fatal(err)
    }
    fmt.Print(line)
}

// CORRECT: use bufio.Scanner
scanner := bufio.NewScanner(f)
for scanner.Scan() {
    fmt.Println(scanner.Text()) // already without '\n'
}
if err := scanner.Err(); err != nil {
    log.Fatal(err)
}

The Scanner also supports custom split functions. By default it splits by lines (ScanLines), but you can switch to ScanWords, ScanBytes, or a custom function:

// Read per word
scanner := bufio.NewScanner(r)
scanner.Split(bufio.ScanWords)
for scanner.Scan() {
    fmt.Println("Word:", scanner.Text())
}

// Read per byte
scanner.Split(bufio.ScanBytes)

// Custom split: split by commas
scanner.Split(func(data []byte, atEOF bool) (advance int, token []byte, err error) {
    for i, b := range data {
        if b == ',' {
            return i + 1, data[:i], nil
        }
    }
    if atEOF && len(data) > 0 {
        return len(data), data, nil
    }
    return 0, nil, nil
})

The default bufio.Scanner buffer is 64KB. If you’re processing files with very long lines (for example, a large single-line JSON file), you need to raise the buffer limit with scanner.Buffer(buf, maxSize), or the scanner will return bufio.ErrTooLong.

bufio.NewWriter — Writing with a Buffer #

f, _ := os.Create("output.txt")
defer f.Close()

writer := bufio.NewWriter(f)

// Write to the buffer (not yet to disk)
writer.WriteString("First line\n")
writer.WriteString("Second line\n")
fmt.Fprintln(writer, "Third line")

// REQUIRED: flush the buffer to disk before Close
if err := writer.Flush(); err != nil {
    log.Fatal(err)
}

Don’t forget to call writer.Flush() before the program finishes or the file is closed. If you miss this, the data still in the buffer will not be written to disk — silent data loss without any error. Make this pattern a habit:
writer := bufio.NewWriter(f)
defer func() {
    if err := writer.Flush(); err != nil {
        log.Println("Failed to flush:", err)
    }
}()

bufio.ReadWriter — Two-Way Buffering #

For two-way connections like TCP sockets, bufio.ReadWriter combines a bufio.Reader and a bufio.Writer in one struct:

conn, _ := net.Dial("tcp", "localhost:8080")
defer conn.Close()

rw := bufio.NewReadWriter(
    bufio.NewReader(conn),
    bufio.NewWriter(conn),
)

// Send a request
fmt.Fprintln(rw.Writer, "GET / HTTP/1.0")
rw.Writer.Flush()

// Read the response
resp, _ := rw.ReadString('\n')
fmt.Println(resp)

bytes.Buffer — An In-Memory Buffer #

bytes.Buffer from the bytes package is a versatile in-memory buffer implementing io.Reader, io.Writer, and io.ByteScanner. No need to open or close it — very practical for building content in memory before sending it out.

import "bytes"

var buf bytes.Buffer

// Write to the buffer
buf.WriteString("Hello, ")
buf.WriteString("Golang!")
fmt.Fprintf(&buf, " Version %d", 21)

// Read from the buffer
fmt.Println(buf.String())  // "Hello, Golang! Version 21"
fmt.Println(buf.Len())     // remaining unread bytes

// Reset the buffer for reuse
buf.Reset()

// bytes.NewBuffer — initialize with initial content
buf2 := bytes.NewBuffer([]byte("initial data"))

// bytes.NewBufferString — initialize from a string
buf3 := bytes.NewBufferString("initial data from string")
_ = buf2
_ = buf3

Pattern: Building a Response Body #

bytes.Buffer is very commonly used to build content that will be sent as an HTTP response or written to a file:

func buildReport(data []Item) []byte {
    var buf bytes.Buffer

    fmt.Fprintf(&buf, "DAILY REPORT\n")
    fmt.Fprintf(&buf, "=============\n\n")

    for i, item := range data {
        fmt.Fprintf(&buf, "%d. %s — Rp %d\n", i+1, item.Name, item.Price)
    }

    fmt.Fprintf(&buf, "\nTotal: %d items\n", len(data))
    return buf.Bytes()
}

Composing Readers and Writers #

The greatest strength of the Go I/O system is the ability to stack (compose) readers and writers. Each layer adds new capabilities without changing the interface.

flowchart LR
    A[os.File\nraw data] --> B[gzip.Reader\ndecompress]
    B --> C[bufio.Reader\nbuffer]
    C --> D[bufio.Scanner\nread per line]
    D --> E[Your code]

    style A fill:#e8f4f8
    style B fill:#e8f8e8
    style C fill:#f8f4e8
    style D fill:#f8e8e8
    style E fill:#e8e8f8

import (
    "bufio"
    "compress/gzip"
    "os"
)

// Read a compressed gzip file, line by line
func readGzip(filename string) error {
    f, err := os.Open(filename) // layer 1: the file
    if err != nil {
        return err
    }
    defer f.Close()

    gz, err := gzip.NewReader(f) // layer 2: decompression
    if err != nil {
        return err
    }
    defer gz.Close()

    scanner := bufio.NewScanner(gz) // layer 3: read per line
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }
    return scanner.Err()
}

// Write to a gzip file, with buffering
func writeGzip(filename string, lines []string) error {
    f, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer f.Close()

    gz := gzip.NewWriter(f)   // layer 2: compression
    defer gz.Close()

    bw := bufio.NewWriter(gz) // layer 3: buffer
    defer bw.Flush()

    for _, l := range lines {
        fmt.Fprintln(bw, l)
    }
    return nil
}

Real-World Usage Patterns #

Reading a Config File Line by Line #

A very common pattern: reading a simple KEY=VALUE config file, ignoring empty lines and comments.

func readConfig(path string) (map[string]string, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, fmt.Errorf("open config: %w", err)
    }
    defer f.Close()

    config := make(map[string]string)
    scanner := bufio.NewScanner(f)

    for scanner.Scan() {
        line := strings.TrimSpace(scanner.Text())

        // Skip empty lines and comments
        if line == "" || strings.HasPrefix(line, "#") {
            continue
        }

        // Split KEY=VALUE
        idx := strings.Index(line, "=")
        if idx == -1 {
            continue
        }
        key := strings.TrimSpace(line[:idx])
        val := strings.TrimSpace(line[idx+1:])
        config[key] = val
    }

    return config, scanner.Err()
}

Copying a File with Progress #

By wrapping the writer, we can count progress without changing the copying logic:

type progressWriter struct {
    total      int64
    written    int64
    onProgress func(percent int)
}

func (pw *progressWriter) Write(p []byte) (int, error) {
    n := len(p)
    pw.written += int64(n)
    if pw.total > 0 {
        percent := int(pw.written * 100 / pw.total)
        pw.onProgress(percent)
    }
    return n, nil
}

func copyWithProgress(src, dst string) error {
    source, err := os.Open(src)
    if err != nil {
        return err
    }
    defer source.Close()

    info, _ := source.Stat()

    dest, err := os.Create(dst)
    if err != nil {
        return err
    }
    defer dest.Close()

    pw := &progressWriter{
        total: info.Size(),
        onProgress: func(percent int) {
            fmt.Printf("\rProgress: %d%%", percent)
        },
    }

    // Write to BOTH the destination and the progressWriter at once
    multi := io.MultiWriter(dest, pw)
    _, err = io.Copy(multi, source)
    fmt.Println() // newline after the progress
    return err
}

Processing Large Files Streaming #

For large CSV or log files, process line by line — don’t load into memory:

func processLargeCSV(path string) error {
    f, err := os.Open(path)
    if err != nil {
        return err
    }
    defer f.Close()

    scanner := bufio.NewScanner(f)

    // Raise the buffer for long lines
    scanner.Buffer(make([]byte, 1024*1024), 1024*1024)

    // Skip the header
    scanner.Scan()

    var totalLines int
    for scanner.Scan() {
        line := scanner.Text()
        columns := strings.Split(line, ",")
        if len(columns) < 3 {
            continue
        }
        // process each line here...
        totalLines++
    }

    fmt.Printf("Processed: %d lines\n", totalLines)
    return scanner.Err()
}

Reading Input from Stdin #

// Read one line from the terminal
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter your name: ")
name, _ := reader.ReadString('\n')
name = strings.TrimSpace(name)
fmt.Printf("Hello, %s!\n", name)

// Read piped stdin (e.g. echo "data" | ./program)
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
    fmt.Println("Received:", scanner.Text())
}

Handling I/O Errors #

I/O is an area where errors are very common — missing files, full disks, denied permissions, dropped connections. Good error handling is an inseparable part of robust I/O code.

Distinguishing Error Types #

f, err := os.Open("data.txt")
if err != nil {
    // Check the specific error type
    if os.IsNotExist(err) {
        fmt.Println("File not found")
    } else if os.IsPermission(err) {
        fmt.Println("Access denied")
    } else {
        fmt.Printf("Unknown error: %v\n", err)
    }
    return
}
defer f.Close()

The Safe Defer Pattern #

defer f.Close() doesn’t capture the error from Close — even though Close can fail (for example, during a final flush to a full disk). For files being written, handle the Close error explicitly:

// ANTI-PATTERN: ignore the error from Close when writing
f, _ := os.Create("output.txt")
defer f.Close() // the error from Close is ignored

// CORRECT: handle the Close error for files being written
func writeFile(path string, data []byte) (err error) {
    f, err := os.Create(path)
    if err != nil {
        return
    }
    defer func() {
        cerr := f.Close()
        if err == nil { // only overwrite if there's no error yet
            err = cerr
        }
    }()

    _, err = f.Write(data)
    return
}

Wrapping I/O Errors #

Always give context to I/O errors so they’re easy to debug:

// ANTI-PATTERN: propagate the error without context
func readConfig(path string) ([]byte, error) {
    return os.ReadFile(path)
    // error: "no such file or directory" — but which file?
}

// CORRECT: wrap the error with context
func readConfig(path string) ([]byte, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, fmt.Errorf("readConfig %q: %w", path, err)
    }
    return data, nil
}
// error: `readConfig "config/app.yaml": no such file or directory`

When to Switch to Alternatives #

Keep using io / bufio / os if:
  ✓ Reading or writing files on the local system
  ✓ Processing sequential data streams (logs, CSV, text)
  ✓ Building I/O pipelines with reader/writer composition
  ✓ Reading input from the terminal or stdin
  ✓ Copying data between streams (files, network, buffers)

Consider other packages if:
  ✗ Working with the filesystem abstractly → use io/fs (Go 1.16+)
  ✗ Reading/writing structured binary formats → use encoding/binary
  ✗ Reading/writing JSON, XML, CSV → use encoding/json, encoding/xml, encoding/csv
  ✗ Compression/decompression → use compress/gzip, compress/zlib
  ✗ Complex concurrent I/O synchronization → consider channels + goroutines
  ✗ Higher-level filesystem operations (copy, rename, walk) → use os, path/filepath

Summary #

io.Reader and io.Writer — the two core interfaces of all Go I/O; functions accepting these interfaces can work transparently with files, networks, buffers, or any source.
io.Copy — the most efficient way to move data between streams; uses an internal 32KB buffer so it doesn’t burden memory even with large data.
io.ReadAll — reads the entire content into memory; only use it when the data size is definitely small or you genuinely need all the data at once.
io.LimitReader — always limit the amount of data read from untrusted sources (user uploads, HTTP bodies) to prevent memory exhaustion.
bufio.Scanner — the idiomatic way to read text line by line; cleaner and safer than ReadString; raise the buffer if lines can be very long.
bufio.Writer — must call Flush() before finishing; unflushed data is lost without an error if the program ends.
bytes.Buffer — a versatile in-memory buffer implementing io.Reader and io.Writer; use it to build content in memory before sending it out.
Reader/writer composition — stack layers (file → gzip → bufio → scanner) to build powerful pipelines; each layer adds capabilities without changing the interface.
The Close error on written files — handle the error from Close explicitly for files being written; a failed final flush to disk won’t be detected if ignored.

← Previous: Strings Next: Math →