Surprises and Gotchas When Working With JSON
This is a list of things about Go's encoding/json package which, over the years, have either confused or surprised me when I first encountered them.
Many of these things are mentioned in the official package documentation if you read it carefully enough, so in theory they shouldn't come as a surprise. But a few of them aren't mentioned in the documentation at all — or at least, they aren't pointed out explicitly — and are worth being aware of!
- Map entries are sorted alphabetically
- Byte slices are encoded as base-64 strings
- Nil and empty slices are encoded differently
- Integer, time.Time and net.IP values can be used as map keys
- Angle brackets and ampersands in strings are escaped
- Trailing zeroes are removed from floats
- Using omitempty on an zero-valued struct doesn't work
- Using omitempty on a zero-value time.Time doesn't work
- There is a 'string' struct tag
- Non-ASCII punctuation characters aren't supported in struct tags
- Decoding a JSON number into an interface{} yields a float64
- Don't use More() to check if there are remaining JSON objects in a stream
- String values returned by custom MarshalJSON() methods must be quoted
Map entries are sorted alphabetically
When encoding a Go map to JSON, the entries will be sorted alphabetically based on the map key. For example, the following map:
Will be encoded to the JSON:
Byte slices are encoded as base-64 strings
Any []byte
slices will be converted to a base64-encoded string when encoding them to JSON. The base64 string uses padding and the standard encoding characters, as defined in RFC 4648. For example, the following map:
Will be encoded to the JSON:
Nil and empty slices are encoded differently
Nil slices in Go will be encoded to the null
JSON value. In contrast, an empty (but not nil) slice will be encoded as an empty JSON array. For example:
Will be encoded to the JSON:
Integer, time.Time and net.IP values can be used as map keys
It's possible to encode a map which has integer values as the map keys. These integers will be automatically converted to strings in the resulting JSON (because the keys in a JSON object must always be strings). For example:
Will be encoded to the JSON:
In addition, Go allows you to encode maps with keys that implement the encoding.TextMarshaler
interface. This means that you can also use time.Time
and net.IP
values as map keys out-of-the-box. For example:
Will be encoded to the JSON:
Note that trying to encode a map with any other type of key will result in a json.UnsupportedTypeError
error.
Angle brackets and ampersands in strings are escaped
If a string contains angle brackets<>
these will be escaped to \u003c
and \u003e
in the JSON output. Likewise the &
character will be escaped to \u0026
. This is to prevent some web browsers from accidentally interpreting the JSON as HTML. For example:
Will be encoded to the JSON:
If you need to prevent these characters being escaped, you should use a json.Encoder
instance and call SetEscapeHTML(false)
. An example is here.
Trailing zeroes are removed from floats
When encoding a floating-point number with a fractional part that ends in zero(es), any trailing zeroes will not appear in the JSON. For example:
Will be encoded to the JSON:
Using omitempty on an zero-valued struct doesn't work
The omitempty
directive never considers a struct type to be empty — even if all the struct fields have their zero value, and you use omitempty
on those fields too. It will always appear as an object in the encoded JSON. For example:
Will be encoded to the JSON:
There’s a long-standing proposal which discusses changing this behavior, but the Go 1 compatibility promise means that it's unlikely to happen any time soon. Instead, you can get around this by making the field a pointer to a struct, which works because omitempty
considers nil
pointers to be empty. For example:
Using omitempty on a zero-value time.Time doesn't work
Using omitempty
on a zero-value time.Time
field won't hide it in the encoded JSON. This is because the time.Time
type is a struct behind the scenes and, as mentioned above, omitempty
never considers a struct type to be empty. Instead, the string "0001-01-01T00:00:00Z"
will appear in the JSON (which is the value returned by calling the MarshalJSON()
method on an zero-value time.Time
. For example:
Will be encoded to the JSON:
There is a 'string' struct tag
Go provides a string
struct tag directive which forces the data in an individual field to be encoded as a string in the resulting JSON. For example, if you want to force an integer to be represented as a string instead of an JSON number you can use the string
directive like so:
And this will be encoded to the JSON:
Note that the string
struct tag directive will only work on fields which contain float, integer or bool
types. For any other type it will have no effect.
Non-ASCII punctuation characters aren't supported in struct tags
When using struct tags to change key names in JSON, any tags containing non-ASCII punctuation characters will be ignored. Notably this means that you can't use en or em dashes, or most currency signs, in struct tags. For example:
Will be encoded to the following JSON (notice that the struct tag renaming the CostEUR
field has been ignored):
Likewise, any struct tags containing non-ASCII punctuation characters will be ignored when decoding values from a JSON object into a struct, and the struct field will be left with its zero value. For example the following code:
Will print out:
This can be annoying in situations where you need to decode a JSON object that has keys containing non-ASCII characters, and you can't change the JSON. To work around this limitation, you can decode to a map as an intermediary step, and then copy the data from the map to the struct. For example the following code:
Will print out:
Decoding a JSON number into an interface{} yields a float64
When decoding a JSON number into an interface{}
, the value will have the underlying type float64
— even if it is an integer in the original JSON.
If you want to get the value as an integer (instead of a float64
) the most robust approach is to decode the JSON using a json.Decoder
instance with the UseNumber()
method set on it. This will decode all JSON numbers to the underlying type json.Number
instead of float64
, and you can then access the number as an integer using its Int64()
method. For example:
Will print:
Don't use More() to check if there are remaining JSON objects in a stream
When processing a stream of JSON objects with json.Decoder
, don't use the More()
method to check if there is a remaining object in the stream. Depsite its name, More()
is not designed for this purpose†, and trying to use it in this way may cause some subtle problems.
†The More()
method is intended to be used in conjunction with Token()
, and exists specifically to check if there is another element in the array or object currently being parsed.
For example, if you use it when decoding an invalid JSON stream like {"name": "alice"}{"name": "bob"}]
(notice the additional square bracket at the end) it won't result in an error (when it should!). Like so:
This code will run without error and output:
The correct technique to see if a stream contains another JSON object is to check for an io.EOF
error, which will be returned when there are no more objects to process in the stream. Like so:
Running this will correctly result in an error, as we would expect given the invalid input:
String values returned by custom MarshalJSON() methods must be quoted
If you are creating a custom MarshalJSON()
method which returns a string value, you must wrap the string in double quotes before returning it, otherwise it won't be interpreted as a JSON string and will result in a runtime error. For example:
Will result in the following JSON being printed:
If, in the code above, you didn't quote the return value from MarshalJSON()
you will get the error: