Encoding Csv #

CSV (Comma-Separated Values) is the most universal data exchange format — almost every system can export and import CSV, from Excel spreadsheets to PostgreSQL databases. But CSV has many edge cases that are easy to overlook: fields containing commas must be quoted, quotes inside fields must be escaped, newlines inside fields are valid too, and the delimiter can be something other than a comma (tab, semicolon, pipe). The encoding/csv package handles all this complexity correctly per RFC 4180. This article covers how to read and write CSV correctly, handle edge cases, configure the reader and writer, and CSV processing patterns in production applications.

An Overview of the encoding/csv Package #

flowchart LR
    subgraph Read["Reading CSV"]
        R1["csv.NewReader(r)"] --> R2["reader.Read()\none row → []string"]
        R1 --> R3["reader.ReadAll()\nall rows → [][]string"]
        R2 --> R4["loop until io.EOF"]
    end

    subgraph Write["Writing CSV"]
        W1["csv.NewWriter(w)"] --> W2["writer.Write([]string)\nwrite one row"]
        W1 --> W3["writer.WriteAll([][]string)\nwrite all rows"]
        W2 --> W4["writer.Flush()\nrequired!"]
        W3 --> W5["writer.Error()\ncheck the error after Flush"]
    end

    subgraph Config["Configuration"]
        C1["reader.Comma = ';'\ncustom delimiter"]
        C2["reader.Comment = '#'\nskip comment lines"]
        C3["reader.FieldsPerRecord\nvalidate the field count"]
        C4["reader.LazyQuotes = true\ntolerate non-standard quotes"]
        C5["reader.TrimLeadingSpace\nremove leading spaces"]
        C6["writer.Comma = '\t'\nwrite TSV"]
        C7["writer.UseCRLF\nuse \r\n"]
    end

    style Read fill:#e8f5e9
    style Write fill:#e3f2fd
    style Config fill:#fff3e0

Reading CSV — csv.Reader #

Reading Row by Row #

package main

import (
    "encoding/csv"
    "fmt"
    "io"
    "os"
    "strings"
)

func main() {
    input := `name,email,city
Budi Santoso,[email protected],Jakarta
Ani Wijaya,[email protected],Bandung
Charlie,[email protected],"Surabaya, East Java"
`
    reader := csv.NewReader(strings.NewReader(input))

    for {
        record, err := reader.Read()

        // EOF means done — not a real error
        if err == io.EOF {
            break
        }
        if err != nil {
            fmt.Fprintf(os.Stderr, "read error: %v\n", err)
            return
        }

        // record is a []string — one element per field
        fmt.Println(record)
    }
    // [name email city]
    // [Budi Santoso [email protected] Jakarta]
    // [Ani Wijaya [email protected] Bandung]
    // [Charlie [email protected] Surabaya, East Java]  ← quotes already removed!
}

ReadAll — Reading Everything at Once #

func readCSVSimple(path string) ([][]string, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, fmt.Errorf("readCSVSimple: %w", err)
    }
    defer f.Close()

    reader := csv.NewReader(f)

    // ReadAll — read all rows at once
    // Suitable for small files that fit in memory
    records, err := reader.ReadAll()
    if err != nil {
        return nil, fmt.Errorf("readCSVSimple ReadAll: %w", err)
    }

    return records, nil
}

// Usage
records, err := readCSVSimple("data.csv")
if err != nil {
    log.Fatal(err)
}

// records[0] is the header
header := records[0]
fmt.Println("Columns:", header)

// records[1:] is the data
for _, row := range records[1:] {
    fmt.Println(row)
}

Configuring csv.Reader #

flowchart TD
    Reader["csv.NewReader(r)"] --> Config["Configure before Read()"]

    Config --> Comma["reader.Comma\ndefault: ',' (comma)\ncan be changed to ';' '\t' '|' etc."]
    Config --> Comment["reader.Comment\ndefault: 0 (none)\ne.g. '#' to skip comments"]
    Config --> Fields["reader.FieldsPerRecord\ndefault: 0 (auto from the first row)\n-1: flexible (no validation)\nN: exactly N fields per row"]
    Config --> Lazy["reader.LazyQuotes\ndefault: false\ntrue: tolerate non-standard quotes"]
    Config --> Trim["reader.TrimLeadingSpace\ndefault: false\ntrue: remove leading spaces in fields"]
    Config --> ReuseRecord["reader.ReuseRecord\ndefault: false\ntrue: reuse the slice (faster,\nbut previous contents are overwritten)"]

    style Reader fill:#4f86c6,color:#fff
    style Config fill:#e8f5e9

// TSV (Tab-Separated Values)
readerTSV := csv.NewReader(r)
readerTSV.Comma = '\t'

// CSV with a semicolon delimiter (common in Europe)
readerSemicolon := csv.NewReader(r)
readerSemicolon.Comma = ';'

// CSV with comments
readerWithComment := csv.NewReader(r)
readerWithComment.Comment = '#'

// Input:
// # this is a comment, ignored
// name,score
// Budi,90

// Field count validation
readerStrict := csv.NewReader(r)
readerStrict.FieldsPerRecord = 3 // MUST be exactly 3 fields per row
// Errors if any row has a different field count

readerFlexible := csv.NewReader(r)
readerFlexible.FieldsPerRecord = -1 // accept any number of fields per row

// LazyQuotes — for CSV from other systems that aren't 100% RFC 4180 compliant
// Example: a field like "Jakarta" Barat (quote not closed correctly)
readerLazy := csv.NewReader(r)
readerLazy.LazyQuotes = true

// TrimLeadingSpace — useful for CSV that has spaces after commas
// Example: name, email, city  (space after the comma)
readerTrim := csv.NewReader(r)
readerTrim.TrimLeadingSpace = true

// ReuseRecord — better performance if you copy the data before the next iteration
readerFast := csv.NewReader(r)
readerFast.ReuseRecord = true // WARNING: the previous record is overwritten!

for {
    record, err := readerFast.Read()
    if err == io.EOF {
        break
    }
    // REQUIRED: copy the record before the next iteration if you want to keep it
    copy := make([]string, len(record))
    copy(copy, record)
    // ANTI-PATTERN: store the record directly without copying
    // data = append(data, record) — the data will be overwritten on the next iteration!
}

Writing CSV — csv.Writer #

func writeCSV(path string, headers []string, rows [][]string) error {
    f, err := os.Create(path)
    if err != nil {
        return fmt.Errorf("writeCSV create: %w", err)
    }
    defer f.Close()

    writer := csv.NewWriter(f)

    // Write the header
    if err := writer.Write(headers); err != nil {
        return fmt.Errorf("writeCSV write header: %w", err)
    }

    // Write the data rows
    for _, row := range rows {
        if err := writer.Write(row); err != nil {
            return fmt.Errorf("writeCSV write row: %w", err)
        }
    }

    // REQUIRED: Flush moves the data from the buffer to the file
    writer.Flush()

    // Check the error after Flush
    if err := writer.Error(); err != nil {
        return fmt.Errorf("writeCSV flush: %w", err)
    }

    return nil
}

// WriteAll — write everything at once
func writeCSVAllAtOnce(path string, records [][]string) error {
    f, err := os.Create(path)
    if err != nil {
        return fmt.Errorf("writeCSVAllAtOnce: %w", err)
    }
    defer f.Close()

    writer := csv.NewWriter(f)
    if err := writer.WriteAll(records); err != nil {
        return fmt.Errorf("writeCSVAllAtOnce WriteAll: %w", err)
    }

    // WriteAll flushes automatically, but still check the error
    return writer.Error()
}

Always call writer.Flush() and check writer.Error() after finishing writes. The csv.Writer uses a bufio.Writer internally — unflushed data will be lost when the file is closed without any error. WriteAll calls Flush internally, but still check writer.Error() afterwards.

Configuring csv.Writer #

// TSV — Tab-Separated Values
writer := csv.NewWriter(f)
writer.Comma = '\t'

// CSV with a semicolon delimiter
writer.Comma = ';'

// Use CRLF (Windows-style line endings)
writer.UseCRLF = true

// Writing to an http.ResponseWriter for download
func downloadCSVHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/csv")
    w.Header().Set("Content-Disposition", `attachment; filename="data.csv"`)

    writer := csv.NewWriter(w)
    writer.UseCRLF = true // Excel on Windows needs CRLF

    // Write the header
    writer.Write([]string{"ID", "Name", "Email", "City"})

    // Write data from the database
    rows, _ := db.Query("SELECT id, name, email, city FROM users")
    defer rows.Close()

    for rows.Next() {
        var id int
        var name, email, city string
        rows.Scan(&id, &name, &email, &city)
        writer.Write([]string{
            strconv.Itoa(id),
            name,
            email,
            city,
        })
    }

    writer.Flush()
}

Handling Headers — Mapping to Structs #

The encoding/csv package doesn’t directly support mapping to structs, but this pattern is easy to implement:

sequenceDiagram
    participant File as CSV File
    participant Reader as csv.Reader
    participant Code as Go Code
    participant Struct as []Product

    File->>Reader: read
    Reader->>Code: record[0] = header row\n["id","name","price","stock"]
    Code->>Code: build map: header → index\n{"id":0,"name":1,"price":2,"stock":3}
    loop every data row
        Reader->>Code: record[n] = data row\n["1","Laptop","15000000","10"]
        Code->>Struct: Product{\n  ID: record[idx["id"]],\n  Name: record[idx["name"]],\n  ...\n}
    end

type Product struct {
    ID       int
    Name     string
    Price    float64
    Stock    int
    Category string
}

func readProductsFromCSV(path string) ([]Product, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, fmt.Errorf("readProductsFromCSV: %w", err)
    }
    defer f.Close()

    reader := csv.NewReader(f)
    reader.TrimLeadingSpace = true

    // Read the header
    header, err := reader.Read()
    if err != nil {
        return nil, fmt.Errorf("read header: %w", err)
    }

    // Build a header → index map
    idx := make(map[string]int)
    for i, h := range header {
        idx[strings.ToLower(strings.TrimSpace(h))] = i
    }

    // Validate the required columns
    required := []string{"id", "name", "price", "stock"}
    for _, col := range required {
        if _, exists := idx[col]; !exists {
            return nil, fmt.Errorf("column '%s' not found in the CSV", col)
        }
    }

    var products []Product
    lineNumber := 1

    for {
        record, err := reader.Read()
        if err == io.EOF {
            break
        }
        if err != nil {
            return nil, fmt.Errorf("read line %d: %w", lineNumber, err)
        }
        lineNumber++

        // Parse each field with validation
        id, err := strconv.Atoi(strings.TrimSpace(record[idx["id"]]))
        if err != nil {
            return nil, fmt.Errorf("line %d: invalid ID %q: %w",
                lineNumber, record[idx["id"]], err)
        }

        price, err := strconv.ParseFloat(
            strings.ReplaceAll(record[idx["price"]], ",", ""), 64)
        if err != nil {
            return nil, fmt.Errorf("line %d: invalid price %q: %w",
                lineNumber, record[idx["price"]], err)
        }

        stock, err := strconv.Atoi(strings.TrimSpace(record[idx["stock"]]))
        if err != nil {
            return nil, fmt.Errorf("line %d: invalid stock %q: %w",
                lineNumber, record[idx["stock"]], err)
        }

        p := Product{
            ID:    id,
            Name:  strings.TrimSpace(record[idx["name"]]),
            Price: price,
            Stock: stock,
        }

        // Optional column
        if i, exists := idx["category"]; exists && i < len(record) {
            p.Category = strings.TrimSpace(record[i])
        }

        products = append(products, p)
    }

    return products, nil
}

Writing Structs to CSV #

func writeProductsToCSV(path string, products []Product) error {
    f, err := os.Create(path)
    if err != nil {
        return fmt.Errorf("writeProductsToCSV: %w", err)
    }
    defer f.Close()

    writer := csv.NewWriter(f)

    // Write the header
    if err := writer.Write([]string{"id", "name", "price", "stock", "category"}); err != nil {
        return fmt.Errorf("write header: %w", err)
    }

    // Write every product
    for _, p := range products {
        record := []string{
            strconv.Itoa(p.ID),
            p.Name,
            strconv.FormatFloat(p.Price, 'f', 2, 64),
            strconv.Itoa(p.Stock),
            p.Category,
        }
        if err := writer.Write(record); err != nil {
            return fmt.Errorf("write product %d: %w", p.ID, err)
        }
    }

    writer.Flush()
    return writer.Error()
}

Edge Cases to Watch Out For #

Fields with Commas, Quotes, and Newlines #

// csv.Writer handles all edge cases automatically!
writer := csv.NewWriter(os.Stdout)

// A field with a comma — automatically quoted
writer.Write([]string{"Surabaya, East Java", "60000"})
// Output: "Surabaya, East Java",60000

// A field with quotes — automatically escaped with double quotes
writer.Write([]string{`Product "Premium"`, "15000"})
// Output: "Product ""Premium""",15000

// A field with a newline — automatically quoted
writer.Write([]string{"Description\nsecond line", "active"})
// Output: "Description
// second line",active

// An empty field
writer.Write([]string{"", "value", ""})
// Output: ,value,

writer.Flush()

Detecting and Handling CSV Errors #

func readCSVWithErrorHandling(r io.Reader) ([][]string, error) {
    reader := csv.NewReader(r)
    reader.LazyQuotes = true // tolerant of imperfect CSV

    var records [][]string
    var parseErrors []string
    lineNumber := 0

    for {
        lineNumber++
        record, err := reader.Read()

        if err == io.EOF {
            break
        }

        if err != nil {
            // Check the CSV error type
            var csvErr *csv.ParseError
            if errors.As(err, &csvErr) {
                // ParseError includes the line and column information
                parseErrors = append(parseErrors,
                    fmt.Sprintf("line %d, column %d: %v",
                        csvErr.Line, csvErr.Column, csvErr.Err))
                continue // continue to the next row
            }
            // A serious I/O error — stop
            return nil, fmt.Errorf("line %d: %w", lineNumber, err)
        }

        records = append(records, record)
    }

    if len(parseErrors) > 0 {
        fmt.Fprintf(os.Stderr, "Warning: %d rows skipped due to errors:\n",
            len(parseErrors))
        for _, e := range parseErrors {
            fmt.Fprintf(os.Stderr, "  - %s\n", e)
        }
    }

    return records, nil
}

Production Usage Patterns #

Pipeline: Read → Transform → Write #

// CSV transformation: filter, change formats, add columns
func transformCSV(src io.Reader, dst io.Writer, minPrice float64) error {
    reader := csv.NewReader(src)
    reader.TrimLeadingSpace = true

    writer := csv.NewWriter(dst)
    defer writer.Flush()

    // Read and pass through the header with an extra column
    header, err := reader.Read()
    if err != nil {
        return fmt.Errorf("read header: %w", err)
    }

    // Add a "price_category" column
    newHeader := append(header, "price_category")
    if err := writer.Write(newHeader); err != nil {
        return err
    }

    // Find the price column index
    priceIdx := -1
    for i, h := range header {
        if strings.EqualFold(h, "price") {
            priceIdx = i
            break
        }
    }
    if priceIdx < 0 {
        return fmt.Errorf("'price' column not found")
    }

    lineNumber := 1
    processed, skipped := 0, 0

    for {
        record, err := reader.Read()
        if err == io.EOF {
            break
        }
        if err != nil {
            lineNumber++
            skipped++
            fmt.Fprintf(os.Stderr, "skip line %d: %v\n", lineNumber, err)
            continue
        }
        lineNumber++

        // Parse the price
        price, err := strconv.ParseFloat(
            strings.ReplaceAll(record[priceIdx], ",", ""), 64)
        if err != nil {
            skipped++
            continue
        }

        // Filter: skip products below the minimum price
        if price < minPrice {
            skipped++
            continue
        }

        // Add the price category column
        var category string
        switch {
        case price >= 10000000:
            category = "premium"
        case price >= 1000000:
            category = "mid-range"
        default:
            category = "budget"
        }

        newRecord := append(record, category)
        if err := writer.Write(newRecord); err != nil {
            return fmt.Errorf("write line %d: %w", lineNumber, err)
        }
        processed++
    }

    fmt.Fprintf(os.Stderr, "Done: %d processed, %d skipped\n",
        processed, skipped)

    return writer.Error()
}

Importing CSV into a Database #

func importCSVToDB(path string, db *sql.DB) error {
    f, err := os.Open(path)
    if err != nil {
        return fmt.Errorf("open file: %w", err)
    }
    defer f.Close()

    reader := csv.NewReader(f)
    reader.TrimLeadingSpace = true

    // Read the header
    header, err := reader.Read()
    if err != nil {
        return fmt.Errorf("read header: %w", err)
    }

    // Validate the header
    required := map[string]bool{"name": false, "email": false, "city": false}
    idx := make(map[string]int)
    for i, h := range header {
        key := strings.ToLower(strings.TrimSpace(h))
        idx[key] = i
        if _, exists := required[key]; exists {
            required[key] = true
        }
    }
    for col, exists := range required {
        if !exists {
            return fmt.Errorf("required column '%s' is missing", col)
        }
    }

    // Start a database transaction
    tx, err := db.Begin()
    if err != nil {
        return fmt.Errorf("begin transaction: %w", err)
    }
    defer tx.Rollback() // will be rolled back if Commit isn't called

    stmt, err := tx.Prepare(
        "INSERT INTO users (name, email, city) VALUES ($1, $2, $3) " +
        "ON CONFLICT (email) DO UPDATE SET name=$1, city=$3")
    if err != nil {
        return fmt.Errorf("prepare statement: %w", err)
    }
    defer stmt.Close()

    succeeded, failed := 0, 0
    lineNumber := 1

    for {
        record, err := reader.Read()
        if err == io.EOF {
            break
        }
        lineNumber++
        if err != nil {
            failed++
            fmt.Fprintf(os.Stderr, "skip line %d: %v\n", lineNumber, err)
            continue
        }

        name := strings.TrimSpace(record[idx["name"]])
        email := strings.TrimSpace(record[idx["email"]])
        city := strings.TrimSpace(record[idx["city"]])

        if name == "" || email == "" {
            failed++
            continue
        }

        if _, err := stmt.Exec(name, email, city); err != nil {
            fmt.Fprintf(os.Stderr, "line %d insert failed: %v\n", lineNumber, err)
            failed++
            continue
        }
        succeeded++

        // Commit every 1000 rows to avoid an overly large transaction
        if succeeded%1000 == 0 {
            if err := tx.Commit(); err != nil {
                return fmt.Errorf("commit: %w", err)
            }
            tx, _ = db.Begin()
            stmt, _ = tx.Prepare(
                "INSERT INTO users (name, email, city) VALUES ($1, $2, $3) " +
                "ON CONFLICT (email) DO UPDATE SET name=$1, city=$3")
            fmt.Fprintf(os.Stderr, "Progress: %d succeeded\n", succeeded)
        }
    }

    if err := tx.Commit(); err != nil {
        return fmt.Errorf("final commit: %w", err)
    }

    fmt.Printf("Import done: %d succeeded, %d failed\n", succeeded, failed)
    return nil
}

Exporting from a Database to CSV #

func exportUsersToCSV(db *sql.DB, w io.Writer) error {
    rows, err := db.Query(
        "SELECT id, name, email, city, created_at FROM users ORDER BY id")
    if err != nil {
        return fmt.Errorf("query: %w", err)
    }
    defer rows.Close()

    writer := csv.NewWriter(w)

    // Write the header
    writer.Write([]string{"id", "name", "email", "city", "registered_date"})

    for rows.Next() {
        var id int
        var name, email, city string
        var createdAt time.Time

        if err := rows.Scan(&id, &name, &email, &city, &createdAt); err != nil {
            fmt.Fprintf(os.Stderr, "scan error: %v\n", err)
            continue
        }

        writer.Write([]string{
            strconv.Itoa(id),
            name,
            email,
            city,
            createdAt.Format("2006-01-02"),
        })
    }

    writer.Flush()

    if err := rows.Err(); err != nil {
        return fmt.Errorf("rows iteration: %w", err)
    }

    return writer.Error()
}

When to Switch to Alternatives #

Keep using encoding/csv if:
  ✓ Reading and writing standard CSV (RFC 4180)
  ✓ CSV from Excel, Google Sheets, or other common systems
  ✓ Simple CSV with a comma or tab delimiter
  ✓ Small to medium CSV files
  ✓ Importing/exporting data to a database

Consider manual parsing with bufio.Scanner if:
  ✗ Very simple CSV without any quoting at all
  ✗ The format isn't RFC 4180 compliant and LazyQuotes isn't enough
  ✗ You need full control over parsing every character

Consider external libraries if:
  ✗ CSV with millions of rows → gocsv for automatic struct mapping
  ✗ Automatic column type inference (int, float, bool, date)
     → csvutil, gocsv
  ✗ Per-column schema validation
  ✗ Parallel processing of large CSVs
  ✗ Excel (.xlsx) rather than CSV → github.com/qax-os/excelize

Consider encoding/json if:
  ✗ Data exchange between services — JSON is more expressive for hierarchical data
  ✗ Data with complex types (nested, arrays)

Summary #

reader.Read() returns io.EOF when finished — this isn’t an error; handle it separately with if err == io.EOF { break }.
The csv.Writer uses an internal buffer — always call writer.Flush() when done, and check writer.Error() to learn if there was an error during buffering.
encoding/csv handles edge cases automatically — fields with commas, quotes, and newlines are quoted and escaped correctly per RFC 4180.
reader.TrimLeadingSpace = true for CSV with spaces after the delimiter — common in human-created CSV files or those exported from spreadsheets.
reader.LazyQuotes = true for tolerating imperfect CSV — useful for files from legacy systems that don’t fully follow RFC 4180.
reader.FieldsPerRecord = -1 for CSV with an inconsistent field count — the default (0) uses the first row as the reference and errors if other rows differ.
reader.ReuseRecord = true for maximum performance — but save a copy with copy() if you want to keep the record, because the original slice is overwritten on the next iteration.
Build a header → index map when reading CSV with headers — more robust than accessing record[0], record[1] hardcoded if the column order changes.
Commit the database in batches when importing large CSVs — don’t use one transaction for millions of rows; commit every N rows to avoid an overly large transaction.

← Previous: Regexp Next: Crypto →