Embedding Files and Data in Go Applications

Go‘s embed package introduced in Go 1.16 provides a simple way to embed files, directories, and binary data into a Go application. This eliminates the need to carry external files and resources with the application binary.

Embedding resources directly into the executable can simplify deployment and distribution of applications. The embed package handles embedding the resources at compile time and provides an API to access the embedded data at runtime.

In this comprehensive guide, we will explore various examples of using the embed package in Go to embed different types of files, data, and resources.

Embedding a Single File

The most basic usage of the embed package is to embed a single file into the application.

For example, to embed a text file called data.txt, we can do:

package main

import (
    _ "embed"
    "fmt"  
)

//go:embed data.txt
var f string

func main() {
    fmt.Println(f) 
}

Here‘s what is happening in the code above:

1. The blank import of "embed" brings the embed package into scope.

2. The f variable of type string will hold the contents of the embedded file.

3. The //go:embed data.txt directive tells the compiler to embed data.txt into the variable f.

4. Inside main, we simply print the contents stored in f to see the embedded data.

When we run this program, it will output the full contents of data.txt read from the embedded file data stored in the app binary itself.

The //go:embed directive embeds the latest copy of the file from disk at compile-time. If the file contents change, recompiling will embed the updated data.

Embedding Multiple Files

You can also embed multiple files into your app by providing a glob pattern instead of a single file to the //go:embed directive.

For example:

package main  

import (
    "embed"
    "fmt"
)

//go:embed *.txt
var f embed.FS

func main() {
    data, _ := f.ReadFile("data.txt") 
    fmt.Println(string(data))

    info, _ := f.ReadFile("info.txt")
    fmt.Println(string(info))  
}

Here the pattern *.txt matches all text files in the current directory, causing both data.txt and info.txt to be embedded.

When embedding multiple files, the return type must be embed.FS instead of a []byte slice. The embedded filesystem f can then be used to read the individual files within it.

This allows embedding entire directories with all internal structure preserved.

Embedding a Directory

Speaking of directories – the entire contents of a folder can be embedded by specifying the folder path with a trailing /* glob in the //go:embed directive.

For example:

package main

import (
    "embed" 
    "fmt"
    "io/fs"
)

//go:embed metrics/*  
var metrics embed.FS

func main() {
    fis, err := fs.ReadDir(metrics, ".") 
    if err != nil {
        panic(err)
    }

    for _, fi := range fis {
        fmt.Println(fi.Name())  
    }
}

Here metrics/* embeds all files within the metrics folder into the metrics variable of type embed.FS.

We can then inspect the directory contents by calling fs.ReadDir to iterate over the file info for each file, printing the file names.

This prints something like:

data.json
config.yaml

As you can see, the full directory structure from disk is maintained within the embedded filesystem.

Embedding Binary Data

In addition to files and directories, arbitrary binary data can also be embedded using the //go:embed directive.

For example:

package main

import (
    "embed"
    "encoding/json"  
    "fmt"  
)

//go:embed hello.json
var data []byte  

func main() {
    var v map[string]string
    json.Unmarshal(data, &v)   
    fmt.Println(v["hello"])
}

Here a JSON blob is embedded directly into the data variable as a []byte slice. We unmarshal the JSON data and print the "hello" value from the parsed map.

This technique can be useful for embedding config defaults, binary assets like images into your app.

Accessing Embedded Files

The embedded filesystem root can be accessed via a FS variable returned from embed.FS(myVar):

fsys := embed.FS(f)

This root filesystem interface provides access to all the embedded files and directories.

Common operations supported include:

• ReadFile(name string) ([]byte, error) – Reads a file within the embedded FS
• ReadDir(name string) ([]fs.DirEntry, error) – Lists a directory within the FS
• Open(name string) (fs.File, error) – Opens file for reading within FS

For example:

data, _ := fsys.ReadFile("dir/data.txt") 
fmt.Println(string(data)) 

Would print the contents of the embedded data.txt file within the dir folder.

Embedding Entire Websites

A very useful application of the embed package is to embed entire website assets like HTML, JS, CSS, images files into a Go binary.

This can then be served via an HTTP fileserver from diskless assets.

Consider this example:

import (
    "embed"
    "io/fs"
    "net/http" 
)

//go:embed site  
var content embed.FS

func main() {
    fsys := fs.FS(content)

    handler := http.FileServer(http.FS(fsys)) 

    http.Handle("/", handler)
    http.ListenAndServe(":3000", nil) 
}

Here the entire site folder containing the website content is embedded into the content variable.

The embedded filesystem is then mounted and served at route / via a fileserver.

This allows shipping a Go binary that runs a full web app without any external disk dependencies.

Let‘s consider some more advanced examples:

Embedding a REST API

A REST API with backend code and endpoints can be embedded into a Go binary along with frontend assets:

import (
    "embed"
    "encoding/json"
    "net/http"
)

//go:embed site
var content embed.FS

type Message struct {
    Text string `json:"message"`
}

func MessageHandler(w http.ResponseWriter, r *http.Request) { 
    msg := Message{"Hello World"}

    json.NewEncoder(w).Encode(msg)
}

func main() {
    fsys := http.FS(content) 
    handler := http.FileServer(fsys)

    http.HandleFunc("/message", MessageHandler)
    http.Handle("/", handler)

    http.ListenAndServe(":3000", nil)
}

Here the REST API logic is written along with website file serving. Useful for monoliths and microservices.

Embedding a Database

For apps needing an embedded database, SQLite provides a robust diskless option:

import (
    "database/sql"
    _ "embed"
    _ "github.com/mattn/go-sqlite3" 
)

//go:embed my.db
var db []byte 

func main() {
    db, _ := sql.Open("sqlite3", ":memory:")
    db.Exec("CREATE TABLE users(id INTEGER PRIMARY KEY, name TEXT)")

    // Insert into embedded database 
    db.Exec("INSERT INTO users(name) VALUES (?)", "John")

    // Query
    rows, _ := db.Query("SELECT name FROM users")
    defer rows.Close()
}

The SQLite database file my.db is embedded into the binary and loaded into an in-memory database at runtime for seamless usage.

Embedding Version Info

Another interesting use case is embedding version info from a VERSION file into your app:

import "embed"

//go:embed VERSION  
var version string  

func Version() string {
    return version  
}

Now the Version() method can return the app version from the embedded VERSION file which gets compiled into the binary.

Whenever the VERSION is updated, recompiling will make available the latest version string.

This is more reliable than setting versions in code.

Embedded version info has proven useful for reliable releases in products like HashiCorp‘s Terraform.

Compile-Time Verification

The embed package appends underscore _ prefixed files for each embedded file into the compiled package directory.

For example, embedding config.yaml will generate a config.yaml file at compile time under a _embedFiles folder in the package path.

This allows verify if the correct files were embedded without running the app.

Embedded Files are Read Only

It‘s important to note that embedded files and directories are immutable read-only data at runtime.

Operations trying to modify embedded data like WriteFile, Create will throw runtime errors.

Therefore, the embed package is most suitable for read-only assets that don‘t change at runtime. Mutable runtime state should use external storage.

Impact on Binary Size

A downside to embedding files is increase in the binary size proportional to sizes of the embedded files.

Significantly large files or large number of files can cause the binary size to balloon.

For example, a simple Hello World Go app without any embedded data is 1.7 MB in size.

Embedded Data	Binary Size	Increase
None	1.7 MB	–
10 KB file	1.8 MB	5.8%
1 MB file	2.7 MB	58%
10 MB file	11.7 MB	588%

As you can see, a 10 MB file caused the binary size to grow over 11.7 MB – a 6x increase!

Therefore, restraint must applied in terms of what and how much content you embed for your use case.

Ideally performance testing various options should guide this decision.

Conclusion

The embed package offers an extremely simple interface for embedding files, folders, data into a Go app executable.

It can greatly simplify deployment requirements for applications by reducing external dependencies.

Some useful applications include bundling configs, assets, websites or version info within minimalist Go binaries.

However, care must be taken to not let uncontrolled embedding bloat the binaries. Performance testing is highly recommended.

I hope these examples give you ideas on how you can leverage this useful addition to the Go standard library!