Bytes #
The bytes package is the mirror of the strings package — almost every function in strings has a counterpart in bytes, but working on []byte instead of string. This matters because in Go, converting between string and []byte always creates a new copy in memory. Code that does a lot of back-and-forth []byte ↔ string conversion will burden the garbage collector. The bytes package lets you work directly with []byte without unnecessary conversions — especially important when processing data from networks, files, or streams that naturally come in byte form. Alongside the manipulation functions, bytes.Buffer is a very frequently used tool for building []byte or string incrementally and efficiently.
An Overview of the bytes Package #
flowchart TD
B["package bytes"] --> Func["Manipulation Functions\n(like strings)"]
B --> Buffer["bytes.Buffer\nbuild data incrementally"]
B --> Reader["bytes.Reader\nio.Reader from []byte"]
Func --> Search["Search\nContains / Index / Count\nIndexByte / IndexRune"]
Func --> Transform["Transformation\nToUpper / ToLower / Title\nTrimSpace / Trim / Replace"]
Func --> Split["Split & Join\nSplit / SplitN / Fields\nJoin / Repeat"]
Func --> Compare["Comparison\nEqual / Compare\nHasPrefix / HasSuffix"]
Buffer --> BW["io.Writer\nWrite / WriteByte / WriteString\nWriteRune"]
Buffer --> BR["io.Reader\nRead / ReadByte / ReadRune\nReadString / ReadLine"]
Buffer --> BA["Access\nBytes() / String()\nLen() / Cap() / Reset()"]
Reader --> RR["io.Reader, io.Seeker\nio.ReaderAt\nSeek / ReadAt"]
style B fill:#4f86c6,color:#fff
style Func fill:#e8f5e9
style Buffer fill:#e3f2fd
style Reader fill:#fff3e0Basic Functions — Searching and Checking #
package main
import (
"bytes"
"fmt"
)
func main() {
data := []byte("Hello, World! Hello, Go!")
// Contains — does it contain a sub-slice?
fmt.Println(bytes.Contains(data, []byte("World"))) // true
fmt.Println(bytes.Contains(data, []byte("Python"))) // false
// ContainsAny — contains any byte from the string?
fmt.Println(bytes.ContainsAny(data, "aeiou")) // true — there are vowels
fmt.Println(bytes.ContainsAny(data, "xyz")) // false
// ContainsRune — contains a specific rune?
fmt.Println(bytes.ContainsRune(data, '!')) // true
// Count — count the occurrences
fmt.Println(bytes.Count(data, []byte("Hello"))) // 2
fmt.Println(bytes.Count(data, []byte(""))) // 23 (len+1)
// Index — the position of the first occurrence (-1 if absent)
fmt.Println(bytes.Index(data, []byte("World"))) // 7
fmt.Println(bytes.Index(data, []byte("Java"))) // -1
// LastIndex — the position of the last occurrence
fmt.Println(bytes.LastIndex(data, []byte("Hello"))) // 14
// IndexByte — search for one byte (faster than Index)
fmt.Println(bytes.IndexByte(data, '!')) // 12
// IndexRune — search for one rune (supports multibyte)
fmt.Println(bytes.IndexRune(data, 'W')) // 7
// IndexAny — search for any byte from the set
fmt.Println(bytes.IndexAny(data, "aeiou")) // 1 — 'e' in "Hello"
// HasPrefix and HasSuffix
fmt.Println(bytes.HasPrefix(data, []byte("Hello"))) // true
fmt.Println(bytes.HasSuffix(data, []byte("Go!"))) // true
}
Transformation — Changing Byte Slice Contents #
data := []byte(" Hello, World! ")
// Trim — remove characters from both ends
fmt.Println(string(bytes.TrimSpace(data))) // "Hello, World!"
fmt.Println(string(bytes.Trim(data, " !"))) // "Hello, World"
fmt.Println(string(bytes.TrimLeft(data, " "))) // "Hello, World! "
fmt.Println(string(bytes.TrimRight(data, " "))) // " Hello, World!"
fmt.Println(string(bytes.TrimPrefix(data, []byte(" ")))) // "Hello, World! "
fmt.Println(string(bytes.TrimSuffix(data, []byte(" ")))) // " Hello, World!"
// TrimFunc — remove bytes satisfying a condition
clean := bytes.TrimFunc(data, func(r rune) bool {
return r == ' ' || r == '!'
})
fmt.Println(string(clean)) // "Hello, World"
// Case conversion
s := []byte("hello world")
fmt.Println(string(bytes.ToUpper(s))) // "HELLO WORLD"
fmt.Println(string(bytes.ToLower([]byte("HELLO WORLD")))) // "hello world"
fmt.Println(string(bytes.ToTitle(s))) // "HELLO WORLD" (title = upper for ASCII)
// Title — capitalize the start of each word (deprecated in strings, but exists in bytes)
fmt.Println(string(bytes.Title(s))) // "Hello World"
// Replace and ReplaceAll
text := []byte("the cat eats fish, the cat is happy")
fmt.Println(string(bytes.Replace(text, []byte("cat"), []byte("dog"), 1)))
// "the dog eats fish, the cat is happy" — replace only the first
fmt.Println(string(bytes.ReplaceAll(text, []byte("cat"), []byte("dog"))))
// "the dog eats fish, the dog is happy" — replace all
// Map — per-rune transformation
result := bytes.Map(func(r rune) rune {
if r >= 'a' && r <= 'z' {
return r - 32 // lowercase to uppercase
}
return r
}, []byte("hello, world!"))
fmt.Println(string(result)) // "HELLO, WORLD!"
Splitting and Joining #
// Split — split with a separator
data := []byte("apple,mango,orange,banana")
parts := bytes.Split(data, []byte(","))
for _, p := range parts {
fmt.Println(string(p))
}
// apple
// mango
// orange
// banana
// SplitN — at most N parts
parts2 := bytes.SplitN(data, []byte(","), 2)
fmt.Println(string(parts2[0])) // "apple"
fmt.Println(string(parts2[1])) // "mango,orange,banana"
// SplitAfter — the separator goes with the previous part
parts3 := bytes.SplitAfter(data, []byte(","))
// ["apple," "mango," "orange," "banana"]
// Fields — split by whitespace (like strings.Fields)
sentence := []byte(" hello world go ")
words := bytes.Fields(sentence)
for _, w := range words {
fmt.Printf("[%s]\n", w)
}
// [hello]
// [world]
// [go]
// FieldsFunc — split by a condition
csv := []byte("one,,two,,,three")
columns := bytes.FieldsFunc(csv, func(r rune) bool {
return r == ','
})
// ["one" "two" "three"] — empty fields are skipped
// Join — join with a separator
fruits := [][]byte{[]byte("apple"), []byte("mango"), []byte("orange")}
result := bytes.Join(fruits, []byte(", "))
fmt.Println(string(result)) // "apple, mango, orange"
// Repeat — repeat a byte slice
fmt.Println(string(bytes.Repeat([]byte("ab"), 4))) // "abababab"
fmt.Println(string(bytes.Repeat([]byte("-"), 20))) // "--------------------"
Comparing Byte Slices #
a := []byte("apple")
b := []byte("mango")
c := []byte("apple")
// Equal — are the contents the same?
fmt.Println(bytes.Equal(a, c)) // true
fmt.Println(bytes.Equal(a, b)) // false
// ANTI-PATTERN: compare with string()
// This allocates a string copy!
fmt.Println(string(a) == string(c)) // true but inefficient
// CORRECT: use bytes.Equal
fmt.Println(bytes.Equal(a, c)) // true with no allocation
// Compare — like strcmp: -1, 0, or 1
fmt.Println(bytes.Compare(a, b)) // -1 (apple < mango)
fmt.Println(bytes.Compare(b, a)) // 1 (mango > apple)
fmt.Println(bytes.Compare(a, c)) // 0 (equal)
// EqualFold — case-insensitive comparison
x := []byte("Hello")
y := []byte("hello")
fmt.Println(bytes.EqualFold(x, y)) // true
bytes.Buffer — Building Data Incrementally #
bytes.Buffer is the most frequently used tool from the bytes package. It implements io.Reader and io.Writer, making it very flexible for building byte data incrementally without many allocations.
flowchart LR
subgraph Write["Writing to the Buffer"]
W1["Write([]byte)\nwrite a byte slice"]
W2["WriteByte(byte)\nwrite one byte"]
W3["WriteString(string)\nwrite a string"]
W4["WriteRune(rune)\nwrite one rune"]
W5["fmt.Fprintf(&buf, ...)\nwrite with formatting"]
end
subgraph Buffer["bytes.Buffer"]
B["internal\n[]byte"]
end
subgraph Read["Reading from the Buffer"]
R1["Read([]byte)\nread into a slice"]
R2["ReadByte()\nread one byte"]
R3["ReadRune()\nread one rune"]
R4["ReadString('\n')\nread until a delimiter"]
R5["ReadLine()\nread one line"]
end
subgraph Access["Data Access"]
A1["Bytes() []byte\nthe buffer contents (no copy)"]
A2["String() string\nthe contents as a string"]
A3["Len() int\nthe number of remaining bytes"]
A4["Reset()\nempty the buffer"]
end
Write --> Buffer
Buffer --> Read
Buffer --> Accessimport (
"bytes"
"fmt"
)
// Basic: building a byte slice
var buf bytes.Buffer
buf.WriteString("Hello, ")
buf.WriteString("World")
buf.WriteByte('!')
buf.WriteRune('🌍')
fmt.Println(buf.String()) // "Hello, World!🌍"
fmt.Println(buf.Len()) // 17 (in bytes, not runes)
// Use fmt.Fprintf for formatting
var buf2 bytes.Buffer
for i := 1; i <= 5; i++ {
fmt.Fprintf(&buf2, "item %d\n", i)
}
fmt.Print(buf2.String())
// item 1
// item 2
// item 3
// item 4
// item 5
// Reset and reuse
buf.Reset()
fmt.Println(buf.Len()) // 0
fmt.Println(buf.Cap()) // the capacity is still there, not reallocated
// Initialize with initial content
buf3 := bytes.NewBuffer([]byte("initial data"))
buf3.WriteString(" appended")
fmt.Println(buf3.String()) // "initial data appended"
// Reading from the buffer
buf4 := bytes.NewBuffer([]byte("first line\nsecond line\nthird line\n"))
line, err := buf4.ReadString('\n')
fmt.Print(line) // "first line\n"
fmt.Println(err) // nil
line, err = buf4.ReadString('\n')
fmt.Print(line) // "second line\n"
Buffer vs strings.Builder — When to Use Which #
flowchart TD
Q{"What's the output goal?"} --> S["Only need a string\nin the end"]
Q --> B["Need []byte\nor both"]
Q --> IO["Need io.Reader\nor io.Writer"]
S --> SB["strings.Builder\nmore efficient for strings\ncan't Read"]
B --> BB["bytes.Buffer\nflexible: can read and write\ncan be io.Reader/Writer"]
IO --> BB2["bytes.Buffer\nimplements both"]
SB --> SE["String() to get the result"]
BB --> BE["Bytes() or String()\nto get the result"]
style SB fill:#e8f5e9
style BB fill:#e3f2fd
style BB2 fill:#e3f2fd// strings.Builder — for building a string (can't be read as a Reader)
var sb strings.Builder
sb.WriteString("Hello, ")
sb.WriteString("World!")
result := sb.String() // get the result as a string
// bytes.Buffer — for building []byte or when you need io.Reader
var buf bytes.Buffer
buf.WriteString("protocol data")
buf.WriteByte(0x00) // can write any byte including null
// Send as an io.Reader to another function
json.NewDecoder(&buf).Decode(&target)
http.Post(url, "application/octet-stream", &buf)
// ANTI-PATTERN: use bytes.Buffer only for a final string
var buf2 bytes.Buffer
for i := 0; i < 100; i++ {
buf2.WriteString("item") // Buffer works, but Builder is more efficient
}
_ = buf2.String()
// CORRECT: strings.Builder for pure string building
var sb2 strings.Builder
for i := 0; i < 100; i++ {
sb2.WriteString("item")
}
_ = sb2.String()
bytes.Reader — An io.Reader from []byte #
bytes.Reader turns a []byte into an io.Reader that supports seeking — useful when you have data in memory but the function receiving it expects an io.Reader:
data := []byte(`{"name":"Budi","age":30}`)
// Create a Reader from []byte
reader := bytes.NewReader(data)
// Decode JSON from the Reader (not directly from []byte)
var user struct {
Name string `json:"name"`
Age int `json:"age"`
}
json.NewDecoder(reader).Decode(&user)
fmt.Println(user.Name, user.Age) // Budi 30
// Seek — go back to a certain position
reader.Seek(0, 0) // back to the beginning
fmt.Println(reader.Len()) // 25 — back to the full length
// ReadAt — read from a certain position without moving the position
buf := make([]byte, 4)
reader.ReadAt(buf, 2)
fmt.Println(string(buf)) // "\"nam"
// Size
fmt.Println(reader.Size()) // 25 — the total size (doesn't change after Seek)
// Use as an io.Reader for HTTP uploads
data2 := []byte("this file's content")
resp, err := http.Post(
"https://api.example.com/upload",
"application/octet-stream",
bytes.NewReader(data2),
)
_ = resp
_ = err
Working with Binary Data #
The bytes package is very useful when processing binary data — network protocols, file formats, or data streams containing a mix of text and binary:
import (
"bytes"
"encoding/binary"
"fmt"
)
// Parsing a simple protocol frame:
// [4 length bytes][1 type byte][N payload bytes]
func parseFrame(data []byte) (kind byte, payload []byte, err error) {
if len(data) < 5 {
return 0, nil, fmt.Errorf("frame too short: %d bytes", len(data))
}
reader := bytes.NewReader(data)
// Read the payload length (4 big-endian bytes)
var length uint32
if err := binary.Read(reader, binary.BigEndian, &length); err != nil {
return 0, nil, fmt.Errorf("read length: %w", err)
}
// Read the type (1 byte)
typeByte, err := reader.ReadByte()
if err != nil {
return 0, nil, fmt.Errorf("read type: %w", err)
}
// Read the payload
payload = make([]byte, length)
if _, err := reader.Read(payload); err != nil {
return 0, nil, fmt.Errorf("read payload: %w", err)
}
return typeByte, payload, nil
}
// Building a protocol frame
func makeFrame(kind byte, payload []byte) []byte {
var buf bytes.Buffer
// Write the payload length (4 big-endian bytes)
binary.Write(&buf, binary.BigEndian, uint32(len(payload)))
// Write the type
buf.WriteByte(kind)
// Write the payload
buf.Write(payload)
return buf.Bytes()
}
// Usage
frame := makeFrame(0x01, []byte("Hello from Go!"))
kind, payload, err := parseFrame(frame)
if err == nil {
fmt.Printf("Type: 0x%02X, Payload: %s\n", kind, payload)
}
Processing HTTP Response Bodies #
import (
"bytes"
"compress/gzip"
"io"
"net/http"
)
func fetchAndProcess(url string) ([]byte, error) {
resp, err := http.Get(url)
if err != nil {
return nil, fmt.Errorf("GET %s: %w", url, err)
}
defer resp.Body.Close()
// Read the whole body into a buffer
var buf bytes.Buffer
if _, err := io.Copy(&buf, resp.Body); err != nil {
return nil, fmt.Errorf("read body: %w", err)
}
data := buf.Bytes()
// Check whether it's gzip-encoded
if bytes.HasPrefix(data, []byte{0x1f, 0x8b}) {
// gzip magic bytes
reader, err := gzip.NewReader(bytes.NewReader(data))
if err != nil {
return nil, fmt.Errorf("open gzip: %w", err)
}
defer reader.Close()
var decompressed bytes.Buffer
if _, err := io.Copy(&decompressed, reader); err != nil {
return nil, fmt.Errorf("decompress: %w", err)
}
return decompressed.Bytes(), nil
}
return data, nil
}
Production Usage Patterns #
Template Rendering into a Buffer #
import (
"bytes"
"html/template"
)
var tmplEmail = template.Must(template.New("email").Parse(`
To: {{.Name}}
Thank you for registering with our service.
Your verification code: {{.Code}}
This code is valid for {{.DurationMinutes}} minutes.
`))
type EmailData struct {
Name string
Code string
DurationMinutes int
}
func renderEmail(data EmailData) ([]byte, error) {
var buf bytes.Buffer
if err := tmplEmail.Execute(&buf, data); err != nil {
return nil, fmt.Errorf("render email: %w", err)
}
return buf.Bytes(), nil
}
// Usage
content, err := renderEmail(EmailData{
Name: "Budi",
Code: "123456",
DurationMinutes: 10,
})
if err == nil {
fmt.Println(string(content))
}
Building CSV Manually #
func makeCSV(headers []string, rows [][]string) []byte {
var buf bytes.Buffer
// Write the header
for i, h := range headers {
if i > 0 {
buf.WriteByte(',')
}
buf.WriteString(escapeCSV(h))
}
buf.WriteByte('\n')
// Write the data rows
for _, row := range rows {
for i, cell := range row {
if i > 0 {
buf.WriteByte(',')
}
buf.WriteString(escapeCSV(cell))
}
buf.WriteByte('\n')
}
return buf.Bytes()
}
func escapeCSV(s string) string {
// Quote if it contains a comma, newline, or quote
if bytes.ContainsAny([]byte(s), ",\"\n\r") {
return `"` + strings.ReplaceAll(s, `"`, `""`) + `"`
}
return s
}
// Usage
csv := makeCSV(
[]string{"ID", "Name", "Email", "City"},
[][]string{
{"1", "Budi Santoso", "[email protected]", "Jakarta"},
{"2", "Ani", "[email protected]", "Bandung, West Java"},
{"3", "Charlie", `char"[email protected]`, "Surabaya"},
},
)
os.WriteFile("output.csv", csv, 0644)
A Buffer Pool with sync.Pool #
import "sync"
// A pool of bytes.Buffers to avoid repeated allocations
var bufPool = sync.Pool{
New: func() any {
return new(bytes.Buffer)
},
}
func getBuffer() *bytes.Buffer {
buf := bufPool.Get().(*bytes.Buffer)
buf.Reset() // IMPORTANT: reset before use
return buf
}
func returnBuffer(buf *bytes.Buffer) {
// Don't return overly large buffers to the pool
// to avoid holding too much memory
if buf.Cap() <= 64*1024 { // 64 KB
bufPool.Put(buf)
}
}
func processRequest(data []byte) string {
buf := getBuffer()
defer returnBuffer(buf)
// Use buf to build the response
buf.WriteString(`{"status":"ok","data":`)
buf.Write(data)
buf.WriteByte('}')
return buf.String()
}
Streaming Large Data Without Loading It into Memory #
// Process a large file line by line without loading everything into memory
func processLargeFile(path string) error {
f, err := os.Open(path)
if err != nil {
return fmt.Errorf("open file: %w", err)
}
defer f.Close()
var (
buf = make([]byte, 64*1024) // 64 KB buffer
leftover []byte // leftover from the previous read
lines int
)
for {
n, err := f.Read(buf)
if n > 0 {
// Combine the previous leftover with the new data
chunk := append(leftover, buf[:n]...)
leftover = nil
// Process each complete line
for {
idx := bytes.IndexByte(chunk, '\n')
if idx < 0 {
// No newline — save as leftover
leftover = chunk
break
}
lines++
processLineData(chunk[:idx])
chunk = chunk[idx+1:]
}
}
if err == io.EOF {
// Process the final leftover if it doesn't end with a newline
if len(leftover) > 0 {
lines++
processLineData(leftover)
}
break
}
if err != nil {
return fmt.Errorf("read file: %w", err)
}
}
fmt.Printf("Processed %d lines\n", lines)
return nil
}
bytes vs strings — When to Use Which #
flowchart TD
Q{"What data are you working with?"} --> Str["string\n(immutable, already exists)"]
Q --> ByteSlice["[]byte\n(mutable, from network/files)"]
Q --> Both["A mix of both"]
Str --> UseStr["Use the strings package\n+ strings.Builder\nwithout conversions"]
ByteSlice --> UseBytes["Use the bytes package\n+ bytes.Buffer\nwithout conversions"]
Both --> Consider["Decide the primary representation\nminimize conversions\nbytes → string: string(b)\nstring → bytes: []byte(s)"]
Consider --> Rule["Rule: conversions create copies\nIf a function needs []byte, avoid\nstring → []byte → string"]
style UseStr fill:#e8f5e9
style UseBytes fill:#e3f2fd
style Rule fill:#fff3e0// ANTI-PATTERN: unnecessary back-and-forth conversions
func processDataBad(data []byte) []byte {
s := string(data) // first copy: []byte → string
s = strings.ToUpper(s) // process
s = strings.TrimSpace(s) // process
return []byte(s) // second copy: string → []byte
}
// CORRECT: stay in []byte, use the bytes package
func processDataGood(data []byte) []byte {
result := bytes.ToUpper(data)
return bytes.TrimSpace(result)
}
// When conversion IS genuinely needed:
// 1. Map/switch cases with string literals
switch string(data[:4]) {
case "HTTP", "POST", "GET ":
// process
}
// 2. When an external function only accepts a string
log.Println(string(data)) // log.Println needs a string
// 3. When storing into a string-typed struct field
user.Name = string(nameBytes)
When to Switch to Alternatives #
Keep using bytes if:
✓ Manipulating []byte: searching, trimming, splitting, replacing
✓ bytes.Buffer for building []byte incrementally
✓ bytes.Reader for passing []byte as an io.Reader
✓ Binary data processing (protocols, file formats)
✓ Working with data from networks or files that naturally comes as []byte
Use strings if:
✗ The data is already a string and will stay a string
✗ String manipulation: strings.Contains, strings.Split, etc.
Use strings.Builder if:
✗ Building a string without needing io.Reader/Writer
✗ More efficient than bytes.Buffer for pure string output
Use bufio if:
✗ Buffered I/O from files or network connections
✗ Reading line by line from large streams
✗ Parsing text line by line with Scanner
Use encoding/binary if:
✗ Reading/writing integers with specific endianness from []byte
✗ Parsing structured binary formats
Summary #
bytesis the mirror ofstrings— almost everystringsfunction has a counterpart inbytes. Choose by data type:string→strings,[]byte→bytes.- Avoid unnecessary
[]byte↔stringconversions — every conversion creates a copy in memory. If the data comes as[]byte, process it as[]byteuntil you’re done.bytes.Equal(a, b)is more efficient thanstring(a) == string(b)— no temporary string allocation, compares byte by byte directly.bytes.Bufferimplements bothio.Readerandio.Writer— useful when you need a buffer that can be read after writing, or when an external function needs anio.Reader.strings.Builderfor pure string building — more efficient thanbytes.Bufferwhen the final output is astringand you don’t need to read it back as aReader.bytes.Readerto turn a[]byteinto anio.Reader— supports seeking (Seek), useful for rewinding or re-reading from different positions.- Use
sync.Poolforbytes.Bufferon HTTP handler hot paths — avoid allocating a new buffer for every request by recycling existing buffers.bytes.Buffer.Reset()empties the buffer without releasing the already-allocated capacity — reusing a buffer with Reset is far more efficient than creating a new one.- For large data, process it streaming with a small buffer rather than loading everything into memory — use
io.Copyor a manual read loop with a[]bytebuffer.