Golang: JSON, YAML, TOML (config) File Reading.

Reading specific file types (JSON, YAML, TOML)

In the previous post, we have seen how to read files in golang, in this extended post of that part, we will look into reading some specific files used for configuration and storing data like JSON, YAML, TOML, CSV, etc.

We will see how to read files and get individual pieces in the files. We'll use packages like os, ioutil and encoding to perform reading operations on file and file objects.

Reading a JSON File

Golang has built-in support for reading JSON files, but still, we can and need to have low-level controls on how to parse and extract content from the file.

Let's say we have a json file named blog.json, we can use the encoding/json package to convert the JSON data into a GO object (an object that is native and understandable to go). The Unmarshal function is used to convert the slice of bytes from the file, into a map object.

{
    "title": "Golang Blog Series",
    "date": "22nd October 2022",
    "tags": ["go", "files"],
    "words": 1500,
    "published": true
}

The above is a simple JSON file, this file has a few types of key-value pairs like string, list, integer, and boolean. But we can also have nested objects and a list of those nested objects.

package main

import (
    "encoding/json"
    "log"
    "os"
)

func main() {

    f, err := os.ReadFile("blog.json")
    if err != nil {
        log.Println(err)
    }
    var data map[string]interface{}
    json.Unmarshal([]byte(f), &data)

    log.Println(data)
    for k, v := range data {
        log.Println(k, ":", v)
    }

}

I have removed the time stamp from the logs below so as to clearly see the output. We can use fmt to print the simple things while keeping consistent with the rest of the snippets in the series.

$ go run json.go

map[date:22nd October 2022 published:true tags:[go files] title:Golang Blog Series words:1500]
published : true
title : Golang Blog Series
date : 22nd October 2022
tags : [go files]
words : 1500

The file is read using the os.ReadFile method, that takes in a string as a path to the file and returns a slice of bytes or an error if there was an issue in reading the file. The parsed slice of byte is than passed as the first argument to the Unmarshal method in the encoding/json package. The second parameter is the output reference where the parsed JSON will be stored or returned. The function does not return the parsed content instead returns an error if there arose any while parsing the JSON content.

As we can see we have got a map of string with an interface. The interface is used because the value of the key can be anything. There is no fixed value like a string, int, bool, or a nested map, slice. Hence we have mapped the JSON object as a map of string with an interface. The type of the value is identified with the interface it has attached to it. Let's take a look what is the type of each value in the map.

published : true
bool

title : Golang Blog Series
string

date : 22nd October 2022
string

tags : [go files]
[]interface {}

words : 1500
float64

Here, we can see it has correctly identified the string type of the fields like bool in case of true or false, a string for string type of values, the fourth field however has a list interface attached to it. The default precedence for float64 over integer is the reason the 1500 value is of type float64.

Reading a YAML File

Though there is no standard package for parsing/unmarshaling YAML files in golang, it's quite easy to use a third-party package and use it to read YAML files.

The package gopkg.in/yaml.v3 is used for encoding and decoding YAML files. We'll be just using it for decoding a YAML file by reading it and converting the file object into native Go objects like maps, lists, strings, etc.

The below steps can be used for setting up the package and installing the YAML package locally.

go mod init <your_project_package_name>
go get gopkg.in/yaml.v3

This should create two files namely go.mod and go.sum with the dependency of the gopkg.in/yaml.v3 package.

title: "Golang Blog Series"
date: "22nd October 2022"
tags: ["go", "files"]
published: false
words: 1500

The above file is a simple YAML config, we'll follow similar kind of examples for the dummy files used in the examples.

package main

import (
    "log"
    "os"

    yaml "gopkg.in/yaml.v3"
)

func main() {

    f, err := os.ReadFile("blog.yaml")

    if err != nil {
        log.Fatal(err)
    }

    var data map[string]interface{}

    err = yaml.Unmarshal(f, &data)

    if err != nil {
        log.Fatal(err)
    }

    log.Println(data)
    for k, v := range data {
        log.Println(k, ":", v)
    }
}

$ go run yaml.go

map[date:22nd October 2022 published:false tags:[go files] title:Golang Blog Series words:1500]
published : false
words : 1500
title : Golang Blog Series
date : 22nd October 2022
tags : [go files]

The above code and output demonstrate the usage of the yaml.v3 package for reading a YAML file.

Firstly, we read the file into a single-string object with the os.ReadFile() method. The method will return a []byte (slice of byte) or an error. If there is an error, we simply log or panic out of the program, else we can use the yaml.Unmarshal method to convert the string/slice of the byte into a map or a pre-defined struct. In this example, we just keep things simple by storing the file content as map [string, interface{}], i.e. a map of string and an interface. The key for YAML can be only a string or an integer. It can't have unrestricted data types like the value can have. Though if you want to be unrestrictive, you can use a map of map[interface{}]interface{} to make the key any shape you like to have.

So, we have created a variable called data as a map of string and interface{}, basically key can be a string and the value can be any type of interface depending on the parsed literally from the file object. The Unmarshal function takes in two parameters, the first being the slice of byte i.e. the file contents, and the second being the output variable. Now, the method does not return the parsed YAML, it can return an error if there arose, so we need to set the second parameter as a pointer to the object into which we want to store the parsed YAML.

In the example, we have called Unmarshal(f, &data) which will fetch the contents from the slice of bytes f and output the parsed YAML from the slice of bytes into the memory location of data and hence using &data indicating the pointer to the variable(fetch the memory address).

So, that is how we obtain the map of keys and values from the YAML config, thereafter, you can iterate on the map, access the keys and values, type caste them as per requirement, and basically have control over what processing needs to be done to the parsed YAML content.

published : false
bool

words : 1500
int

title : Golang Blog Series
string

date : 22nd October 2022
string

tags : [go files]
[]interface {}

I have just printed the types of the values in the above output as log.Printf("%T", v), we can see the types are being correctly recognized and being parsed. The last object is indeed a slice so it has an interface of the slice(array) attached to it.

Reading a TOML file

YAML and TOML are almost identical, though toml has more restrictive configuration options, but is more readable than YAML, as YAML can get complicated pretty quickly. Though both of them have their pros and cons, YAML is used everywhere in the DevOps world, configs, whereas TOML is the format of choice for python packaging, and static site generation configs.

Let's see how we can use golang to read TOML files.

$ go mod init <your_project_package_name>
$ go get github.com/pelletier/go-toml

The above commands are used for setting up a golang package or project and installing the go-toml package. Once the above commands are done executing, it will generate go.mod and go.sum files used for storing dependencies and packages installed for the project locally.

[blog]
name='techstructive-blog'
tags=['go','django','vim']
author='meet gor'
active=true

[author]
name='Meet Gor'
github='mr-destructive'
twitter='meetgor21'
posts=80

The above is the sample file blog.toml which we will use to read in the go script below. The toml file has a similar structure as we have seen in the previous examples. We have different data types like string, boolean, integer, and list.

package main

import (
    "log"
    "os"

    toml "github.com/pelletier/go-toml"
)

func main() {

    f, err := os.ReadFile("blog.toml")

    if err != nil {
        log.Fatal(err)
    }

    var data map[interface{}]interface{}

    err = toml.Unmarshal(f, &data)
    log.Println(data)

    if err != nil {
        log.Fatal(err)
    }

    for k, v := range data {
        log.Println(k, ":", v)

        switch t := v.(type) {
        case map[string]interface{}:
            for a, b := range t {
                log.Println(a, ":", b)
            }
        }
    }
}

$ go run toml.go

map[author:map[github:mr-destructive name:Meet Gor posts:80 twitter:meetgor21] blog:map[active:true author:meet gor

name:techstructive-blog tags:[go django vim]]]

blog : map[active:true author:meet gor name:techstructive-blog tags:[go django vim]]
name : techstructive-blog
tags : [go django vim]
author : meet gor
active : true

author : map[github:mr-destructive name:Meet Gor posts:80 twitter:meetgor21]

name : Meet Gor
github : mr-destructive
twitter : meetgor21
posts : 80

So, in the above example and output, the YAML file was read and the key-value pairs inside them were read. The first thing we do is read the file blog.toml with ioutil package, with the ReadFile function. The function takes in the string as the path to the file to be read and returns a slice of byte. We use this slice of byte as a parameter to the Unmarshal method. The second paramter for the Unmarshal is the output variable(usually a pointer to a variable), we have created a map of interface{] with an interface as we see there can be nested keys which hold the name of the config.

The variable data is a map of interface{} to an interface{}, and we parse the memory address to the data variable to the Unmarshal method. Thereby the parsed TOML content is stored in the data variable.

name : techstructive-blog
string

tags : [go django vim]
[]interface{}

author : meet gor
string

active : true
bool

name : Meet Gor
string

github : mr-destructive
string

twitter : meetgor21
string

posts : 80
int64

The above is a verbose output for the type of the values as parsed by golang, we have string, bool, int64, and a slice (list with interface{} attached with it). Only types like string, bool, int, float64 can be parsed from the Unmarshal function, other than these types, the type will have an interface attached to it.

In such cases, where the type of value is not among the 4 types(string, bool, int float), we can use a pre-defined struct to parse the content from the file. Though it would require a strict structure and predictable response from the parsed file.

Reading CSV file

We can even read a CSV file in golang, we have seen in the previous post, we have used custom delimiters in the parsing of the file.

id,name,posts,exp
21,jim,23,2
33,kevin,39,1
45,james,70,2
56,chris,89,3

The above file is a sample csv file, though the size is too small, we can use it as an example.

package main

import (
    "encoding/csv"
    "log"
    "os"
)

func main() {
    f, err := os.Open("temp.csv")
    check_error(err)

    reader := csv.NewReader(f)

    n, err := reader.ReadAll()
    check_error(err)
    for _, line := range n {
        for _, text := range line {
            log.Println(text)
        }
    }
}

$ go run main.go
id
name
posts
exp
21
jim
23
2
33
kevin
39
1
45
james
70
2
56
chris
89
3

The CSV package has a NewReader method that accepts an io.Reader and returns a Reader object. After parsing the reader, we use the ReadAll method to return a 2d string or an error if there exists an error while parsing the content. You can get a detailed explanation of the CSV parsing and reading in the previous post.

Reading CSV from URL

The CSV file can also be read from the URL, the content of the file is a response.Body in place of the file object reference, in the previous example, the os.Open() method returns a os.File object.

We use the http.Get(string) method to get the response from the URL for reading the CSV file present on the web.

package main

import (
    "encoding/csv"
    "log"
    "net/http"
)

func main() {

    url := "https://github.com/woocommerce/woocommerce/raw/master/sample-data/sample_products.csv"
    response, err := http.Get(url)

    if err != nil {
        log.Println(err)
        return
    }

    defer response.Body.Close()

    reader := csv.NewReader(response.Body)
    n, err := reader.ReadAll()

    if err != nil {
        log.Println(err)
    }

    for _, line := range n {
        for _, text := range line {
            log.Println(text)
        }
    }
}

$ go run csv.go
<feff>ID
Type
SKU
Name
Published
Is featured?
Visibility in catalog
Short description
Description
Date sale price starts
Date sale price ends
...
...
...

So, that's how we can read a CSV file from the URL. By fetching the CSV URL https://github.com/woocommerce/woocommerce/raw/master/sample-data/sample_products.csv from the http.Get method, this will get us the response.Body that contains the actual CSV file content. The response than can be parsed to the csv.NewReader(*Os.File).ReadAll() i.e. reader.ReadAll(). The function returns a multidimensional slice [][]slice that can be iterated and parsed as per requirement.

Reading XML file

XML is the de facto standard for RSS feeds, it is widely used in many places and is still all over the web. We'll see an example to read an XML file locally, but as we saw in the above example, we can also read an RSS link from the web.

Just like CSV, we have encoding/xml, and the standard library has all the functions used for parsing the XML files.

We will be using a local XML file called rss.xml, and reading the contents from the tags in the file.

<?xml version="1.0" encoding="UTF-8" ?>
<channel>
<title>Meet Gor</title>
<description>Techstructive Blog Feed</description>
<item>
<title>Why and How to make and use Vim as a text editor and customizable IDE</title>
<link>https://www.meetgor.com/vim-text-editor-ide</link>
</item>
<item>
<title>Setting up Vim for Python</title>
<link>https://www.meetgor.com/vim-for-python</link>
</item>
</channel>

The above example is a short part of my blog's rss feed. I have just trimmed the unwanted part and will be just using the tags that we want to fetch.

package main

import (
    "encoding/xml"
    "log"
    "os"
)

type Channel struct {
    XMLName     xml.Name `xml:"channel"`
    Title       string   `xml:"title"`
    Description string   `xml:"description"`
    Item        []Item   `xml:"item"`
}

type Item struct {
    XMLName xml.Name `xml:"item"`
    Title   string   `xml:"title"`
    Link    string   `xml:"link"`
}

func check_error(err error) {
    if err != nil {
        log.Fatal(err)
    }
}

func main() {

    f, err := os.ReadFile("rss.xml")
    check_error(err)
    defer f.Close()

    d := Channel{}
    err = xml.Unmarshal(f, &d)
    check_error(err)

    for _, item := range d.Item {
        log.Println(item.Title)
    }
}

$ go run xml.go

{{ channel} Meet Gor Techstructive Blog Feed [{{ item} Why and How to make and use Vim as a text editor and customizable IDE https://www.meetgor.com/vim-text-editor-ide} {{ item} Setting up Vim for Python https://www.meetgor.com/vim-for-python}]}

Why and How to make and use Vim as a text editor and customizable IDE
Setting up Vim for Python

The above example uses a couple of struct like Channel and Item that stores the tag data like title, description, link, etc. Unlike the JSON, YAML, and toml files, XML can't parse the content directly we need a structure to parse into. And that's fine as XML is not much dynamic in terms of config, we usually have standard tags and elements which can be pre-defined in a struct type.

In this example, the RSS feed has a Channel tag with title, description, and item.

NOTE: Use Title case for the fields of the structs. It will make the fields public, I spent a few hours debugging that really :)

So, we define the Channel struct with fields like Title as a string which is a tag in the file as xml:"title". This means the title in the tag of the XML will be stored in the field as a string in the attribute name as Title. Similarly, we have fields like Description and Item[] this is a list or multiple of item tags that might be present in the XML file. The XMLName is used for identifying the parent tag for the struct, so we use channel for the first struct as it is the first tag appearing of the hierarchy in the XML file.

So, we create an object for the root structure as Channel{} (an empty object instantiated). The xml.Unmarshal function is parsed with the content of the file as data which is a slice of byte as we have seen in the previous examples. The slice of byte is then used in the Unmarshal method as the first parameter and the reference of an empty Channel object as the second parameter. The second parameter will be to store the parsed XML content from the file.

I have a few examples on the GitHub repository covering the reading of files from a URL for the CSV, and XML files. But, this concept in the example, can be applied to JSON, YAML, and other file formats as well.

That's it from this part. Reference for all the code examples and commands can be found in the 100 days of Golang GitHub repository.

Conclusion

So, that's it from this post, we covered how to read specific configuration files like JSON, CSV, YAML, TOML, and XML. We saw how to read a local file and also touched on the concept to read contents from a file on the web with a URL. We also saw how we can use pre-defined structs to parse content from a file, especially for XML.

Thank you for reading. If you have any queries, questions, or feedback, you can let me know in the discussion below or on my social handles. Happy Coding :)