Introduction
We are working on a project where we have to make calls into a web service. Many of the web calls return very large documents that contain many sub-documents. The worst part is, we usually only need a handful of the fields for any given document and those fields tend to be scattered all over the place.
Here is a sample of a smaller document:
var document string = `{
"userContext": {
"conversationCredentials": {
"sessionToken": "06142010_1:75bf6a413327dd71ebe8f3f30c5a4210a9b11e93c028d6e11abfca7ff"
},
"valid": true,
"isPasswordExpired": false,
"cobrandId": 10000004,
"channelId": -1,
"locale": "en_US",
"tncVersion": 2,
"applicationId": "17CBE222A42161A3FF450E47CF4C1A00",
"cobrandConversationCredentials": {
"sessionToken": "06142010_1:b8d011fefbab8bf1753391b074ffedf9578612d676ed2b7f073b5785b"
},
"preferenceInfo": {
"currencyCode": "USD",
"timeZone": "PST",
"dateFormat": "MM/dd/yyyy",
"currencyNotationType": {
"currencyNotationType": "SYMBOL"
},
"numberFormat": {
"decimalSeparator": ".",
"groupingSeparator": ",",
"groupPattern": "###,##0.##"
}
}
},
"lastLoginTime": 1375686841,
"loginCount": 299,
"passwordRecovered": false,
"emailAddress": "johndoe@email.com",
"loginName": "sptest1",
"userId": 10483860,
"userType": {
"userTypeId": 1,
"userTypeName": "normal_user"
}
}`
It is not scalable for us to create all the structs and embedded structs to unmarshal the different JSON documents using json.Unmarshal and working directly with a map was out of the question. What we needed was a way to decode these JSON documents into structs that just contained the fields we needed, regardless where those fields lived in the JSON document.
Luckily we came a across a package by
Mitchell Hashimoto called
mapstructure and we forked it. This package is able to take a JSON document that is already unmarshaled into a map and decode that into a struct. Unfortunately, you still needed to create all the embedded structs if you wanted the data at the different levels. So I studied the code and build some functionality on top that allowed us do what we needed.
DecodePathThe first function we added is called DecodePath. This allows us to specify the fields and sub-documents we want from the JSON document and store them into the structs we need. Let's start with a small example using the JSON document above:
type UserType struct {
UserTypeId int
UserTypeName string
}
type User struct {
Session string `jpath:"userContext.cobrandConversationCredentials.sessionToken"`
CobrandId int `jpath:"userContext.cobrandId"`
UserType UserType `jpath:"userType"`
LoginName string `jpath:"loginName"`
}
docScript := []byte(document)
docMap := map[string]interface{}{}
json.Unmarshal(docScript, &docMap)
user := User{}
DecodePath(docMap, &user)
fmt.Printf("%#v", user)
If we run this program we get the following output:
mapstructure.User{
Session:"06142010_1:b8d011fefbab8bf1753391b074ffedf9578612d676ed2b7f073b5785b",
CobrandId:10000004,
UserType:mapstructure.UserType{
UserTypeId:1,
UserTypeName:"normal_user"
}
LoginName:"sptest1"
}
The "jpath" tag is used to find the map keys and set the values into the struct. The User struct contains fields from three different layers of the JSON document. We only needed to define two structs to pull the data out of the map we needed.
We can also map entire structs the same way a normal unmarshal would work. Just name the fields in the struct to match the field names in the JSON document. The names of the fields in the struct don't need to be in the same case as the fields in the JSON document.
Here is a more complicated example using an anonymous field in our struct:
type NumberFormat struct {
DecimalSeparator string `jpath:"userContext.preferenceInfo.numberFormat.decimalSeparator"`
GroupingSeparator string `jpath:"userContext.preferenceInfo.numberFormat.groupingSeparator"`
GroupPattern string `jpath:"userContext.preferenceInfo.numberFormat.groupPattern"`
}
type User struct {
LoginName string `jpath:"loginName"`
NumberFormat
}
docScript := []byte(document)
docMap := map[string]interface{}{}
json.Unmarshal(docScript, &docMap)
user := User{}
DecodePath(docMap, &user)
fmt.Printf("%#v", user)
If we run this program we get the following output:
mapstructure.User{
LoginName:"sptest1"
NumberFormat:mapstructure.NumberFormat{
DecimalSeparator:".",
GroupingSeparator:",",
GroupPattern:"###,##0.##"
}
}
We can also use an anonymous field pointer:
type User struct {
LoginName string `jpath:"loginName"`
*NumberFormat
}
In this case DecodePath will instantiate an object of that type and perform the decode, but only if a mapping can be found.
We now have great control over decoding JSON documents into structs. What happens when the JSON you get back is an array of documents?
DecodeSlicePathThere are times when the web api returns an array of JSON documents:
var document = `[{"name":"bill"},{"name":"lisa"}]`
In this case we need to decode the slice of maps into a slice of objects. We added another function called DecodeSlicePath that does just that:
type NameDoc struct {
Name string `jpath:"name"`
}
sliceScript := []byte(document)
sliceMap := []map[string]interface{}{}
json.Unmarshal(sliceScript, &sliceMap)
var myslice []NameDoc
DecodeSlicePath(sliceMap, &myslice)
fmt.Printf("%#v", myslice)
Here is the output:
[]mapstructure.NameDoc{
mapstructure.NameDoc{Name:"bill"},
mapstructure.NameDoc{Name:"lisa"}
}
The function DecodeSlicePath creates the slice based on the length of the map and then decodes each JSON document, one at a time.
ConclusionIf it were not for Mitchell I would not have been able to get this to work. His package is brilliant and handles all the real technical issues around decoding maps into structs. The two functions I have built on top of mapstructure provides a nice convenience factor we needed for our project. If you're dealing with some of the same issue, please try out the package.