Bufio #

Every time you read from a file or network connection one byte or one line at a time, Go makes a system call to the kernel for each read operation — and system calls are expensive. The bufio package solves this problem by adding a buffer layer between Go code and the I/O source: it reads data in large blocks at once (usually 4096 bytes) and stores them in memory, so subsequent small read operations are served from the buffer without extra system calls. The result can be 10-100× faster for I/O involving many small operations. This package provides three main types: Scanner for reading line by line or token by token, Reader for buffered reads with peek and unread capabilities, and Writer for buffered writes that combine many small write operations into one large one.

An Overview of the bufio Package #

flowchart TD
    IO["I/O Source\n(file, net.Conn, os.Stdin, etc.)"] --> BufIO["package bufio"]

    BufIO --> Scanner["bufio.Scanner\nReads per token/line\nthe easiest API"]
    BufIO --> Reader["bufio.Reader\nBuffered read with\nPeek, ReadString, ReadLine"]
    BufIO --> Writer["bufio.Writer\nBuffered write\nFlush is required"]

    Scanner --> ScanLines["ScanLines — default\nread per line"]
    Scanner --> ScanWords["ScanWords\nread per word"]
    Scanner --> ScanBytes["ScanBytes\nread per byte"]
    Scanner --> ScanRunes["ScanRunes\nread per rune"]
    Scanner --> Custom["Custom SplitFunc\nany tokenization"]

    Reader --> RS["ReadString(delim)\nread until a delimiter"]
    Reader --> RL["ReadLine()\nread one line (low-level)"]
    Reader --> RB["ReadByte / ReadRune\nread one unit"]
    Reader --> Peek["Peek(n)\nlook ahead without consuming"]

    Writer --> WS["WriteString\nWrite / WriteByte\nWriteRune"]
    Writer --> Flush["Flush()\nempty the buffer to the destination"]

    style IO fill:#4f86c6,color:#fff
    style BufIO fill:#e8f5e9
    style Scanner fill:#e3f2fd
    style Reader fill:#fff3e0
    style Writer fill:#f3e5f5

bufio.Scanner — Reading Per Token #

bufio.Scanner is the easiest way to read input line by line or token by token. It hides the buffering complexity and provides a clean API:

package main

import (
    "bufio"
    "fmt"
    "os"
    "strings"
)

func main() {
    // Reading from a string (for illustration)
    input := "first line\nsecond line\nthird line\n"
    scanner := bufio.NewScanner(strings.NewReader(input))

    // Scan — returns true if there's a next token
    for scanner.Scan() {
        fmt.Println(scanner.Text()) // the line text without the newline
        // or scanner.Bytes() for []byte without allocation
    }

    // REQUIRED: check the error after the loop finishes
    // scanner.Err() returns nil if the loop ended because of EOF
    if err := scanner.Err(); err != nil {
        fmt.Fprintf(os.Stderr, "read error: %v\n", err)
    }
}

Reading from Various Sources #

// From a file
func readFile(path string) error {
    f, err := os.Open(path)
    if err != nil {
        return fmt.Errorf("readFile: %w", err)
    }
    defer f.Close()

    scanner := bufio.NewScanner(f)
    lineNumber := 0

    for scanner.Scan() {
        lineNumber++
        line := scanner.Text()

        // Skip empty lines and comments
        if line == "" || strings.HasPrefix(line, "#") {
            continue
        }

        fmt.Printf("%4d: %s\n", lineNumber, line)
    }

    return scanner.Err()
}

// From stdin — for CLI tools
func readStdin() {
    scanner := bufio.NewScanner(os.Stdin)
    fmt.Print("Enter text (Ctrl+D to finish):\n")

    for scanner.Scan() {
        text := scanner.Text()
        // Process each entered line
        fmt.Printf("You wrote: %s\n", strings.ToUpper(text))
    }
}

// From an HTTP response body
func readResponse(resp *http.Response) error {
    defer resp.Body.Close()
    scanner := bufio.NewScanner(resp.Body)

    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }
    return scanner.Err()
}

Split Functions — Changing How Tokenization Works #

The Scanner uses a SplitFunc to determine how to split input into tokens. The default is ScanLines:

flowchart LR
    Input["Input: 'Hello, World! How are you?'"] --> SF["SplitFunc"]

    SF --> SL["ScanLines\n→ token per line"]
    SF --> SW["ScanWords\n→ token per word"]
    SF --> SB["ScanBytes\n→ token per byte"]
    SF --> SR["ScanRunes\n→ token per rune"]
    SF --> SC["Custom SplitFunc\n→ any token"]

    SL --> OL["'Hello, World! How are you?'"]
    SW --> OW["'Hello,' 'World!' 'How' 'are' 'you?'"]
    SB --> OB["'H' 'e' 'l' 'l' 'o' ..."]
    SC --> OC["Whatever you define"]

    style SC fill:#e8f5e9
    style OC fill:#e8f5e9

// ScanWords — read per word
scanner := bufio.NewScanner(strings.NewReader("one two   three\nfour"))
scanner.Split(bufio.ScanWords)

for scanner.Scan() {
    fmt.Printf("[%s]\n", scanner.Text())
}
// [one]
// [two]
// [three]
// [four]

// ScanBytes — read per byte
scanner2 := bufio.NewScanner(strings.NewReader("ABC"))
scanner2.Split(bufio.ScanBytes)

for scanner2.Scan() {
    fmt.Printf("0x%02X\n", scanner2.Bytes()[0])
}
// 0x41
// 0x42
// 0x43

// ScanRunes — read per rune (supports multibyte Unicode)
scanner3 := bufio.NewScanner(strings.NewReader("Héllo"))
scanner3.Split(bufio.ScanRunes)

for scanner3.Scan() {
    fmt.Printf("[%s]\n", scanner3.Text())
}
// [H]
// [é]
// [l]
// [l]
// [o]

Custom SplitFunc #

// SplitFunc signature:
// func(data []byte, atEOF bool) (advance int, token []byte, err error)

// Example: simple CSV tokenization (split by comma)
func scanCSV(data []byte, atEOF bool) (int, []byte, error) {
    // If there's no data and we're at EOF, we're done
    if atEOF && len(data) == 0 {
        return 0, nil, nil
    }

    // Look for a comma
    if i := bytes.IndexByte(data, ','); i >= 0 {
        // There's a comma — return the previous token
        return i + 1, bytes.TrimSpace(data[:i]), nil
    }

    // No comma found
    if atEOF {
        // This is the last token
        return len(data), bytes.TrimSpace(data), nil
    }

    // Ask for more data
    return 0, nil, nil
}

// Example: read per paragraph (split at blank lines)
func scanParagraph(data []byte, atEOF bool) (int, []byte, error) {
    if atEOF && len(data) == 0 {
        return 0, nil, nil
    }

    // Look for two consecutive newlines (a blank line)
    if i := bytes.Index(data, []byte("\n\n")); i >= 0 {
        return i + 2, bytes.TrimSpace(data[:i]), nil
    }

    if atEOF {
        return len(data), bytes.TrimSpace(data), nil
    }

    return 0, nil, nil
}

// Using a custom split
csvData := "apple, mango, orange, banana, durian"
scanner := bufio.NewScanner(strings.NewReader(csvData))
scanner.Split(scanCSV)

var fruits []string
for scanner.Scan() {
    fruits = append(fruits, scanner.Text())
}
fmt.Println(fruits) // [apple mango orange banana durian]

Handling Large Buffers #

The Scanner has a default buffer of 64KB. If a line is longer than that, the scanner will error:

// ANTI-PATTERN: the scanner will error for lines > 64KB
scanner := bufio.NewScanner(hugeFile)
for scanner.Scan() { ... } // can fail with "token too long"

// CORRECT: enlarge the buffer for files with long lines
const maxBufSize = 10 * 1024 * 1024 // 10 MB
scanner := bufio.NewScanner(hugeFile)
scanner.Buffer(make([]byte, maxBufSize), maxBufSize)

for scanner.Scan() {
    // now can handle lines up to 10 MB
}

bufio.Reader — Buffered Reads with Lookahead #

bufio.Reader gives more detailed control than Scanner — useful for parsing complex formats where you need to read characters one at a time, look ahead, or read with custom delimiters:

import (
    "bufio"
    "strings"
    "fmt"
)

reader := bufio.NewReader(strings.NewReader("Hello, World!\nSecond line\n"))

// ReadString — read until a delimiter (the delimiter is included)
line, err := reader.ReadString('\n')
fmt.Printf("[%s] err=%v\n", line, err)
// [Hello, World!\n] err=<nil>

line, err = reader.ReadString('\n')
fmt.Printf("[%s] err=%v\n", line, err)
// [Second line\n] err=<nil>

line, err = reader.ReadString('\n')
fmt.Printf("[%s] err=%v\n", line, err)
// [] err=EOF

// ReadByte — read one byte
reader2 := bufio.NewReader(strings.NewReader("ABC"))
b, _ := reader2.ReadByte()
fmt.Printf("%c\n", b) // A

// UnreadByte — return the last byte to the buffer
reader2.UnreadByte()
b, _ = reader2.ReadByte()
fmt.Printf("%c\n", b) // A again!

// ReadRune — read one rune (supports multibyte)
reader3 := bufio.NewReader(strings.NewReader("Héllo"))
r, size, err := reader3.ReadRune()
fmt.Printf("%c (size=%d)\n", r, size) // H (size=1)

r, size, err = reader3.ReadRune()
fmt.Printf("%c (size=%d)\n", r, size) // é (size=2) — two bytes!

// Peek — look at the next N bytes without consuming them
reader4 := bufio.NewReader(strings.NewReader("HTTP/1.1 200 OK"))
peeked, _ := reader4.Peek(4)
fmt.Println(string(peeked)) // HTTP — the read position hasn't moved

next, _ := reader4.Peek(4)
fmt.Println(string(next)) // HTTP — still the same, not consumed

b2, _ := reader4.ReadByte()
fmt.Printf("%c\n", b2) // H — now it's finally consumed

ReadLine vs ReadString #

// ReadLine — low-level, doesn't include the newline, can return a partial line
line, isPrefix, err := reader.ReadLine()
// isPrefix=true if the line is too long and must be read again
// line doesn't include \n or \r\n

// ReadString — easier, includes the delimiter, handles long lines
line, err := reader.ReadString('\n')
// line includes the trailing \n
// to get the text without \n: strings.TrimRight(line, "\r\n")

// For most cases, ReadString is more recommended than ReadLine

Using Peek for Format Detection #

// Detect the data format before parsing
func detectAndParse(r io.Reader) error {
    br := bufio.NewReader(r)

    // Look at the first 4 bytes without consuming them
    header, err := br.Peek(4)
    if err != nil {
        return fmt.Errorf("detectAndParse peek: %w", err)
    }

    switch {
    case bytes.HasPrefix(header, []byte("{")):
        // JSON object
        return parseJSON(br)
    case bytes.HasPrefix(header, []byte("[")):
        // JSON array
        return parseJSON(br)
    case bytes.HasPrefix(header, []byte("<?xm")):
        // XML
        return parseXML(br)
    case bytes.HasPrefix(header, []byte("\x1f\x8b")):
        // gzip magic bytes
        return parseGzip(br)
    default:
        // Assume plain text
        return parseText(br)
    }
}

bufio.Writer — Buffered Writes #

bufio.Writer collects small pieces of data in a buffer and writes them to the destination in large blocks, dramatically reducing the number of system calls:

sequenceDiagram
    participant App as Application
    participant BW as bufio.Writer\n(4096-byte buffer)
    participant File as File/Network

    App->>BW: WriteString("line 1\n") — 8 bytes
    Note over BW: buffer: [line 1\n] (8/4096)
    App->>BW: WriteString("line 2\n") — 8 bytes
    Note over BW: buffer: [line 1\nline 2\n] (16/4096)
    App->>BW: ... (write 500 small lines)
    Note over BW: buffer full (4096 bytes)
    BW->>File: One write system call — 4096 bytes
    App->>BW: Flush()
    BW->>File: One write system call — the remaining buffer

    Note over App,File: Without bufio: 500 system calls\nWith bufio: 2-3 system calls

import (
    "bufio"
    "fmt"
    "os"
)

func writeLog(path string, entries []string) error {
    f, err := os.Create(path)
    if err != nil {
        return fmt.Errorf("writeLog: %w", err)
    }
    defer f.Close()

    // Wrap the file with a bufio.Writer
    w := bufio.NewWriter(f)

    for _, e := range entries {
        // Small write operations are collected in the buffer
        fmt.Fprintln(w, e)
    }

    // REQUIRED: Flush ensures all buffered data is written to the file
    // defer f.Close() does NOT automatically flush a bufio.Writer!
    if err := w.Flush(); err != nil {
        return fmt.Errorf("writeLog flush: %w", err)
    }

    return nil
}

// Buffer size control
w := bufio.NewWriterSize(f, 64*1024) // 64 KB buffer

// Check how much hasn't been flushed
fmt.Println(w.Buffered())  // the number of bytes in the buffer
fmt.Println(w.Available()) // the free space in the buffer

// Write various types
w.WriteString("plain text\n")
w.WriteByte('\n')
w.WriteRune('é')
w.Write([]byte{0x00, 0x01, 0x02})
fmt.Fprintf(w, "format %d %s\n", 42, "hello")

Always call w.Flush() before the file is closed. defer f.Close() will not flush a bufio.Writer automatically. Data still in the buffer when the file is closed is lost without any error — this is a hard-to-detect bug because the program doesn’t report an error.

A Safe Pattern with Flush in defer #

func writeFileSafe(path string, fn func(*bufio.Writer) error) error {
    f, err := os.Create(path)
    if err != nil {
        return err
    }
    defer f.Close()

    w := bufio.NewWriter(f)

    if err := fn(w); err != nil {
        return err
    }

    // Explicit flush — don't rely on defer f.Close()
    return w.Flush()
}

// Usage
err := writeFileSafe("output.txt", func(w *bufio.Writer) error {
    for i := 0; i < 1000; i++ {
        fmt.Fprintf(w, "line %d\n", i)
    }
    return nil
})

Production Usage Patterns #

A Key=Value Config Parser #

type Config map[string]string

// Format: key=value, one per line, # for comments
func parseConfig(r io.Reader) (Config, error) {
    cfg := make(Config)
    scanner := bufio.NewScanner(r)
    lineNumber := 0

    for scanner.Scan() {
        lineNumber++
        line := strings.TrimSpace(scanner.Text())

        // Skip empty lines and comments
        if line == "" || strings.HasPrefix(line, "#") {
            continue
        }

        // Split key=value
        idx := strings.IndexByte(line, '=')
        if idx < 0 {
            return nil, fmt.Errorf("line %d: invalid format, must be key=value: %q",
                lineNumber, line)
        }

        key := strings.TrimSpace(line[:idx])
        val := strings.TrimSpace(line[idx+1:])

        if key == "" {
            return nil, fmt.Errorf("line %d: key cannot be empty", lineNumber)
        }

        cfg[key] = val
    }

    if err := scanner.Err(); err != nil {
        return nil, fmt.Errorf("parseConfig: %w", err)
    }

    return cfg, nil
}

// Usage
configText := `
# Application configuration
DB_HOST=localhost
DB_PORT=5432
DB_NAME=myapp

# Server settings
SERVER_PORT=8080
DEBUG=true
`

cfg, err := parseConfig(strings.NewReader(configText))
if err == nil {
    fmt.Println(cfg["DB_HOST"])    // localhost
    fmt.Println(cfg["SERVER_PORT"]) // 8080
}

Reading and Writing CSV Line by Line #

// Process a large CSV line by line without loading it into memory
func processLargeCSV(input io.Reader, output io.Writer) error {
    scanner := bufio.NewScanner(input)
    writer := bufio.NewWriter(output)
    defer writer.Flush()

    // Read the header
    if !scanner.Scan() {
        if err := scanner.Err(); err != nil {
            return fmt.Errorf("read header: %w", err)
        }
        return fmt.Errorf("the CSV file is empty")
    }
    header := scanner.Text()

    // Write the header to the output
    fmt.Fprintln(writer, header)

    // Process every data line
    count := 0
    for scanner.Scan() {
        line := scanner.Text()
        if line == "" {
            continue
        }

        // Transform the line (example: change a certain column)
        processed := transformLine(line)
        fmt.Fprintln(writer, processed)
        count++

        // Flush every 1000 lines for visible progress
        if count%1000 == 0 {
            if err := writer.Flush(); err != nil {
                return fmt.Errorf("flush after %d lines: %w", count, err)
            }
            fmt.Fprintf(os.Stderr, "Processed: %d lines\r", count)
        }
    }

    fmt.Fprintf(os.Stderr, "\nDone: %d lines\n", count)
    return scanner.Err()
}

A Simple HTTP Protocol Parser #

// Parse an HTTP request line and headers from a TCP connection
func parseHTTPRequest(conn net.Conn) (*HTTPRequest, error) {
    reader := bufio.NewReader(conn)

    // Read the request line: "GET /path HTTP/1.1"
    requestLine, err := reader.ReadString('\n')
    if err != nil {
        return nil, fmt.Errorf("read request line: %w", err)
    }
    requestLine = strings.TrimRight(requestLine, "\r\n")

    parts := strings.SplitN(requestLine, " ", 3)
    if len(parts) != 3 {
        return nil, fmt.Errorf("invalid request line: %q", requestLine)
    }

    req := &HTTPRequest{
        Method:  parts[0],
        Path:    parts[1],
        Version: parts[2],
        Headers: make(map[string]string),
    }

    // Read headers until an empty line
    for {
        line, err := reader.ReadString('\n')
        if err != nil {
            return nil, fmt.Errorf("read header: %w", err)
        }
        line = strings.TrimRight(line, "\r\n")

        if line == "" {
            break // an empty line = end of headers
        }

        idx := strings.IndexByte(line, ':')
        if idx < 0 {
            continue
        }

        key := strings.TrimSpace(line[:idx])
        val := strings.TrimSpace(line[idx+1:])
        req.Headers[key] = val
    }

    // The body can be read from the following reader if Content-Length exists
    if cl := req.Headers["Content-Length"]; cl != "" {
        length, _ := strconv.Atoi(cl)
        req.Body = make([]byte, length)
        io.ReadFull(reader, req.Body)
    }

    return req, nil
}

Word Count — Counting Words from a Large File #

func countWords(path string) (map[string]int, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, fmt.Errorf("countWords: %w", err)
    }
    defer f.Close()

    counts := make(map[string]int)
    scanner := bufio.NewScanner(f)
    scanner.Split(bufio.ScanWords)

    for scanner.Scan() {
        word := strings.ToLower(scanner.Text())
        // Remove punctuation
        word = strings.Trim(word, ".,!?;:\"'()[]{}")
        if word != "" {
            counts[word]++
        }
    }

    return counts, scanner.Err()
}

A TCP Server with bufio #

// A TCP connection handler with bufio for efficient reads/writes
func handleConn(conn net.Conn) {
    defer conn.Close()

    reader := bufio.NewReader(conn)
    writer := bufio.NewWriter(conn)

    for {
        // Read one line from the client
        line, err := reader.ReadString('\n')
        if err != nil {
            if err != io.EOF {
                fmt.Fprintf(os.Stderr, "read error: %v\n", err)
            }
            return
        }

        command := strings.TrimRight(line, "\r\n")
        fmt.Printf("Received: %q\n", command)

        // Process the command
        var response string
        switch strings.ToUpper(command) {
        case "PING":
            response = "PONG"
        case "TIME":
            response = time.Now().Format(time.RFC3339)
        case "QUIT":
            writer.WriteString("BYE\n")
            writer.Flush()
            return
        default:
            response = "ERROR: unknown command"
        }

        // Send the response
        writer.WriteString(response + "\n")
        if err := writer.Flush(); err != nil {
            fmt.Fprintf(os.Stderr, "write error: %v\n", err)
            return
        }
    }
}

Performance: With and Without bufio #

flowchart LR
    subgraph Without["Without bufio — 10,000 lines"]
        T1["Write line 1\n→ syscall"] 
        T2["Write line 2\n→ syscall"]
        T3["Write line 3\n→ syscall"]
        T4["... 9,997 more syscalls"]
        T1 --> T2 --> T3 --> T4
    end

    subgraph With["With bufio — 10,000 lines"]
        D1["Write 512 lines to the buffer"]
        D2["1 syscall to disk"]
        D3["Write the next 512 lines"]
        D4["1 syscall to disk"]
        D5["... ~20 syscalls total"]
        D1 --> D2 --> D3 --> D4 --> D5
    end

    style Without fill:#fce4ec
    style With fill:#e8f5e9

import (
    "bufio"
    "os"
    "testing"
)

// Benchmark: writing 10,000 lines to a file
func BenchmarkWithoutBufio(b *testing.B) {
    f, _ := os.Create("/tmp/test.txt")
    defer f.Close()

    for i := 0; i < b.N; i++ {
        for j := 0; j < 10000; j++ {
            fmt.Fprintf(f, "line %d\n", j)
        }
    }
}

func BenchmarkWithBufio(b *testing.B) {
    f, _ := os.Create("/tmp/test.txt")
    defer f.Close()
    w := bufio.NewWriter(f)

    for i := 0; i < b.N; i++ {
        for j := 0; j < 10000; j++ {
            fmt.Fprintf(w, "line %d\n", j)
        }
        w.Flush()
    }
}

// Typical results:
// BenchmarkWithoutBufio:  ~15ms per operation
// BenchmarkWithBufio: ~1.5ms per operation (~10x faster)

When to Switch to Alternatives #

Keep using bufio if:
  ✓ Reading text files line by line — bufio.Scanner is the best choice
  ✓ Parsing text formats with custom delimiters
  ✓ Writing lots of small data to a file or connection — bufio.Writer
  ✓ You need peek/unread while parsing — bufio.Reader
  ✓ Parsing text protocols (HTTP, SMTP, Redis RESP, etc.)

Consider os.ReadFile if:
  ✗ Small files that can be loaded into memory all at once
  ✗ You only need the whole file content, not per line
  ✗ One-shot operations without needing streaming

Consider encoding/csv if:
  ✗ Proper CSV parsing with quoting and escaping
  ✗ CSV with fields containing commas or newlines
  ✗ Complex standard CSV — don't parse it manually with bufio

Consider encoding/json with Decoder if:
  ✗ Parsing JSON Lines (NDJSON) — json.Decoder is already buffered
  ✗ JSON streaming from an HTTP body

Consider io.Reader + io.Copy if:
  ✗ You only need to copy data from one stream to another
  ✗ io.Copy already does internal buffering

Summary #

bufio.Scanner for reading line by line — the easiest API; always check scanner.Err() after the loop finishes to make sure there’s no I/O error.
scanner.Split(bufio.ScanWords) to read per word, ScanBytes per byte, ScanRunes per rune — or create a custom SplitFunc for any tokenization.
scanner.Buffer(buf, max) to handle lines longer than the 64KB default — without it, the scanner errors with “token too long” for long lines.
bufio.Writer for writing lots of small data — can be 10x faster than writing directly to a file because it drastically reduces the number of system calls.
Always w.Flush() after finishing writes with bufio.Writer — defer f.Close() doesn’t flush the buffer automatically, and buffered data is lost if not flushed.
bufio.Reader.Peek(n) to look ahead at data without consuming it — useful for format detection or lookahead when parsing protocols.
bufio.Reader.ReadString('\n') is easier than ReadLine() — ReadString handles long lines automatically and includes the delimiter in the token.
bufio.NewReader for TCP connections — wrap a net.Conn with a bufio.Reader for efficient text protocol parsing and a bufio.Writer for efficient responses.
bufio.Scanner supports any io.Reader — files, stdin, HTTP bodies, strings.Reader, bytes.Reader — the same API for all sources.

← Previous: Bytes Next: Filepath →