• Home
  • History
  • Annotate
Name Date Size #Lines LOC

..03-May-2022-

deprecate/H09-Mar-2021-3828

examples/H03-May-2022-3,8802,909

j2x/H09-Mar-2021-247160

x2j/H09-Mar-2021-185129

x2j-wrapper/H09-Mar-2021-2,5481,899

.travis.ymlH A D09-Mar-202123 43

LICENSEH A D09-Mar-20211.1 KiB2317

anyxml.goH A D09-Mar-20214.2 KiB202139

anyxml_test.goH A D09-Mar-20213.9 KiB209183

atomFeedString.xmlH A D09-Mar-20212.9 KiB5543

attrprefix_test.goH A D09-Mar-20213 KiB150136

badxml_test.goH A D09-Mar-20211.4 KiB6960

bom_test.goH A D09-Mar-20211.9 KiB8974

bulk_test.goH A D09-Mar-20212.4 KiB10869

bulkraw_test.goH A D09-Mar-20212.8 KiB11473

cast_test.goH A D09-Mar-20211.5 KiB8169

data_test.goH A D09-Mar-20211.4 KiB3936

doc.goH A D09-Mar-20217.8 KiB1391

escapechars.goH A D09-Mar-20212.5 KiB9461

escapechars_test.goH A D09-Mar-20212.1 KiB11489

example_test.goH A D09-Mar-202110.1 KiB35719

exists.goH A D09-Mar-2021332 105

exists_test.goH A D09-Mar-20211 KiB5520

files.goH A D09-Mar-20216.2 KiB288238

files_test.badjsonH A D09-Mar-2021107 32

files_test.badxmlH A D09-Mar-2021135 109

files_test.goH A D09-Mar-20213.7 KiB169143

files_test.jsonH A D09-Mar-2021123 32

files_test.xmlH A D09-Mar-2021137 109

files_test_dup.jsonH A D09-Mar-2021111 11

files_test_dup.xmlH A D09-Mar-2021123 11

files_test_indent.jsonH A D09-Mar-2021146 1212

files_test_indent.xmlH A D09-Mar-2021140 88

go.modH A D09-Mar-202144 42

gob.goH A D09-Mar-2021837 3625

gob_test.goH A D09-Mar-20211.3 KiB6556

isvalid_test.goH A D09-Mar-20211.7 KiB7667

j2x_test.goH A D09-Mar-2021540 3022

json.goH A D09-Mar-202110 KiB324201

json_test.goH A D09-Mar-20212.8 KiB138102

keystolower_test.goH A D09-Mar-20211 KiB6149

keyvalues.goH A D09-Mar-202118.6 KiB669498

keyvalues2_test.goH A D09-Mar-20211 KiB4841

keyvalues_test.goH A D09-Mar-202110.8 KiB493429

leafnode.goH A D09-Mar-20213.3 KiB11373

leafnode_test.goH A D09-Mar-20212.7 KiB141125

misc.goH A D09-Mar-20212.5 KiB8762

misc_test.goH A D09-Mar-20215.3 KiB269245

mxj.goH A D09-Mar-20213.2 KiB12987

mxj_test.goH A D09-Mar-20211.1 KiB4737

namespace_test.goH A D09-Mar-2021675 2218

nan_test.goH A D09-Mar-20211.4 KiB8268

newmap.goH A D09-Mar-20215.7 KiB185127

newmap_test.goH A D09-Mar-20212.8 KiB11591

readme.mdH A D09-Mar-202111.3 KiB208163

remove.goH A D09-Mar-2021765 3825

remove_test.goH A D09-Mar-2021325 2219

rename.goH A D09-Mar-20211.5 KiB6247

rename_test.goH A D09-Mar-2021867 4338

seqnum_test.goH A D09-Mar-2021869 5342

set.goH A D09-Mar-2021529 2720

set_test.goH A D09-Mar-2021801 4437

setfieldsep.goH A D09-Mar-2021727 219

snakecase_test.goH A D09-Mar-20211.8 KiB7059

songtext.xmlH A D09-Mar-20211.1 KiB3029

strict.goH A D09-Mar-2021997 3112

strict_test.goH A D09-Mar-20211.2 KiB4940

struct.goH A D09-Mar-20211.7 KiB5520

struct_test.goH A D09-Mar-20211.9 KiB8740

structvalue_test.goH A D09-Mar-2021492 2822

updatevalues.goH A D09-Mar-20218.1 KiB259193

updatevalues_test.goH A D09-Mar-20215.7 KiB188146

xml.goH A D09-Mar-202140.5 KiB1,411910

xml2_test.goH A D09-Mar-20217.1 KiB323278

xml3_test.goH A D09-Mar-20211.5 KiB7362

xml_test.goH A D09-Mar-20215.8 KiB241154

xmlseq.goH A D09-Mar-202127.1 KiB878584

xmlseq2.goH A D09-Mar-2021638 197

xmlseq_test.goH A D09-Mar-20212.4 KiB8173

xmppStream_test.goH A D09-Mar-20211.3 KiB6963

readme.md

1<h2>mxj - to/from maps, XML and JSON</h2>
2Decode/encode XML to/from map[string]interface{} (or JSON) values, and extract/modify values from maps by key or key-path, including wildcards.
3
4mxj supplants the legacy x2j and j2x packages. If you want the old syntax, use mxj/x2j and mxj/j2x packages.
5
6<h4>Installation</h4>
7Using go.mod:
8<pre>
9go get github.com/clbanning/mxj/v2@v2.3.2
10</pre>
11
12<pre>
13import "github.com/clbanning/mxj/v2"
14</pre>
15
16... or just vendor the package.
17
18<h4>Related Packages</h4>
19
20https://github.com/clbanning/checkxml provides functions for validating XML data.
21
22<h4>Refactor Encoder - 2020.05.01</h4>
23Issue #70 highlighted that encoding large maps does not scale well, since the original logic used string appends operations. Using bytes.Buffer results in linear scaling for very large XML docs. (Metrics based on MacBook Pro i7 w/ 16 GB.)
24
25	Nodes      m.XML() time
26	54809       12.53708ms
27	109780      32.403183ms
28	164678      59.826412ms
29	482598     109.358007ms
30
31<h4>Refactor Decoder - 2015.11.15</h4>
32For over a year I've wanted to refactor the XML-to-map[string]interface{} decoder to make it more performant.  I recently took the time to do that, since we were using github.com/clbanning/mxj in a production system that could be deployed on a Raspberry Pi.  Now the decoder is comparable to the stdlib JSON-to-map[string]interface{} decoder in terms of its additional processing overhead relative to decoding to a structure value.  As shown by:
33
34	BenchmarkNewMapXml-4         	  100000	     18043 ns/op
35	BenchmarkNewStructXml-4      	  100000	     14892 ns/op
36	BenchmarkNewMapJson-4        	  300000	      4633 ns/op
37	BenchmarkNewStructJson-4     	  300000	      3427 ns/op
38	BenchmarkNewMapXmlBooks-4    	   20000	     82850 ns/op
39	BenchmarkNewStructXmlBooks-4 	   20000	     67822 ns/op
40	BenchmarkNewMapJsonBooks-4   	  100000	     17222 ns/op
41	BenchmarkNewStructJsonBooks-4	  100000	     15309 ns/op
42
43<h4>Notices</h4>
44
45	2021.02.02: v2.5 - add XmlCheckIsValid toggle to force checking that the encoded XML is valid
46	2020.12.14: v2.4 - add XMLEscapeCharsDecoder to preserve XML escaped characters in Map values
47	2020.10.28: v2.3 - add TrimWhiteSpace option
48	2020.05.01: v2.2 - optimize map to XML encoding for large XML docs.
49	2019.07.04: v2.0 - remove unnecessary methods - mv.XmlWriterRaw, mv.XmlIndentWriterRaw - for Map and MapSeq.
50	2019.07.04: Add MapSeq type and move associated functions and methods from Map to MapSeq.
51	2019.01.21: DecodeSimpleValuesAsMap - decode to map[<tag>:map["#text":<value>]] rather than map[<tag>:<value>]
52	2018.04.18: mv.Xml/mv.XmlIndent encodes non-map[string]interface{} map values - map[string]string, map[int]uint, etc.
53	2018.03.29: mv.Gob/NewMapGob support gob encoding/decoding of Maps.
54	2018.03.26: Added mxj/x2j-wrapper sub-package for migrating from legacy x2j package.
55	2017.02.22: LeafNode paths can use ".N" syntax rather than "[N]" for list member indexing.
56	2017.02.10: SetFieldSeparator changes field separator for args in UpdateValuesForPath, ValuesFor... methods.
57	2017.02.06: Support XMPP stream processing - HandleXMPPStreamTag().
58	2016.11.07: Preserve name space prefix syntax in XmlSeq parser - NewMapXmlSeq(), etc.
59	2016.06.25: Support overriding default XML attribute prefix, "-", in Map keys - SetAttrPrefix().
60	2016.05.26: Support customization of xml.Decoder by exposing CustomDecoder variable.
61	2016.03.19: Escape invalid chars when encoding XML attribute and element values - XMLEscapeChars().
62	2016.03.02: By default decoding XML with float64 and bool value casting will not cast "NaN", "Inf", and "-Inf".
63	            To cast them to float64, first set flag with CastNanInf(true).
64	2016.02.22: New mv.Root(), mv.Elements(), mv.Attributes methods let you examine XML document structure.
65	2016.02.16: Add CoerceKeysToLower() option to handle tags with mixed capitalization.
66	2016.02.12: Seek for first xml.StartElement token; only return error if io.EOF is reached first (handles BOM).
67	2015.12.02: XML decoding/encoding that preserves original structure of document. See NewMapXmlSeq()
68	            and mv.XmlSeq() / mv.XmlSeqIndent().
69	2015-05-20: New: mv.StringIndentNoTypeInfo().
70	            Also, alphabetically sort map[string]interface{} values by key to prettify output for mv.Xml(),
71	            mv.XmlIndent(), mv.StringIndent(), mv.StringIndentNoTypeInfo().
72	2014-11-09: IncludeTagSeqNum() adds "_seq" key with XML doc positional information.
73	            (NOTE: PreserveXmlList() is similar and will be here soon.)
74	2014-09-18: inspired by NYTimes fork, added PrependAttrWithHyphen() to allow stripping hyphen from attribute tag.
75	2014-08-02: AnyXml() and AnyXmlIndent() will try to marshal arbitrary values to XML.
76	2014-04-28: ValuesForPath() and NewMap() now accept path with indexed array references.
77
78<h4>Basic Unmarshal XML to map[string]interface{}</h4>
79<pre>type Map map[string]interface{}</pre>
80
81Create a `Map` value, 'mv', from any `map[string]interface{}` value, 'v':
82<pre>mv := Map(v)</pre>
83
84Unmarshal / marshal XML as a `Map` value, 'mv':
85<pre>mv, err := NewMapXml(xmlValue) // unmarshal
86xmlValue, err := mv.Xml()      // marshal</pre>
87
88Unmarshal XML from an `io.Reader` as a `Map` value, 'mv':
89<pre>mv, err := NewMapXmlReader(xmlReader)         // repeated calls, as with an os.File Reader, will process stream
90mv, raw, err := NewMapXmlReaderRaw(xmlReader) // 'raw' is the raw XML that was decoded</pre>
91
92Marshal `Map` value, 'mv', to an XML Writer (`io.Writer`):
93<pre>err := mv.XmlWriter(xmlWriter)
94raw, err := mv.XmlWriterRaw(xmlWriter) // 'raw' is the raw XML that was written on xmlWriter</pre>
95
96Also, for prettified output:
97<pre>xmlValue, err := mv.XmlIndent(prefix, indent, ...)
98err := mv.XmlIndentWriter(xmlWriter, prefix, indent, ...)
99raw, err := mv.XmlIndentWriterRaw(xmlWriter, prefix, indent, ...)</pre>
100
101Bulk process XML with error handling (note: handlers must return a boolean value):
102<pre>err := HandleXmlReader(xmlReader, mapHandler(Map), errHandler(error))
103err := HandleXmlReaderRaw(xmlReader, mapHandler(Map, []byte), errHandler(error, []byte))</pre>
104
105Converting XML to JSON: see Examples for `NewMapXml` and `HandleXmlReader`.
106
107There are comparable functions and methods for JSON processing.
108
109Arbitrary structure values can be decoded to / encoded from `Map` values:
110<pre>mv, err := NewMapStruct(structVal)
111err := mv.Struct(structPointer)</pre>
112
113<h4>Extract / modify Map values</h4>
114To work with XML tag values, JSON or Map key values or structure field values, decode the XML, JSON
115or structure to a `Map` value, 'mv', or cast a `map[string]interface{}` value to a `Map` value, 'mv', then:
116<pre>paths := mv.PathsForKey(key)
117path := mv.PathForKeyShortest(key)
118values, err := mv.ValuesForKey(key, subkeys)
119values, err := mv.ValuesForPath(path, subkeys)
120count, err := mv.UpdateValuesForPath(newVal, path, subkeys)</pre>
121
122Get everything at once, irrespective of path depth:
123<pre>leafnodes := mv.LeafNodes()
124leafvalues := mv.LeafValues()</pre>
125
126A new `Map` with whatever keys are desired can be created from the current `Map` and then encoded in XML
127or JSON. (Note: keys can use dot-notation.)
128<pre>newMap, err := mv.NewMap("oldKey_1:newKey_1", "oldKey_2:newKey_2", ..., "oldKey_N:newKey_N")
129newMap, err := mv.NewMap("oldKey1", "oldKey3", "oldKey5") // a subset of 'mv'; see "examples/partial.go"
130newXml, err := newMap.Xml()   // for example
131newJson, err := newMap.Json() // ditto</pre>
132
133<h4>Usage</h4>
134
135The package is fairly well [self-documented with examples](http://godoc.org/github.com/clbanning/mxj).
136
137Also, the subdirectory "examples" contains a wide range of examples, several taken from golang-nuts discussions.
138
139<h4>XML parsing conventions</h4>
140
141Using NewMapXml()
142
143   - Attributes are parsed to `map[string]interface{}` values by prefixing a hyphen, `-`,
144     to the attribute label. (Unless overridden by `PrependAttrWithHyphen(false)` or
145     `SetAttrPrefix()`.)
146   - If the element is a simple element and has attributes, the element value
147     is given the key `#text` for its `map[string]interface{}` representation.  (See
148     the 'atomFeedString.xml' test data, below.)
149   - XML comments, directives, and process instructions are ignored.
150   - If CoerceKeysToLower() has been called, then the resultant keys will be lower case.
151
152Using NewMapXmlSeq()
153
154   - Attributes are parsed to `map["#attr"]map[<attr_label>]map[string]interface{}`values
155     where the `<attr_label>` value has "#text" and "#seq" keys - the "#text" key holds the
156     value for `<attr_label>`.
157   - All elements, except for the root, have a "#seq" key.
158   - Comments, directives, and process instructions are unmarshalled into the Map using the
159     keys "#comment", "#directive", and "#procinst", respectively. (See documentation for more
160     specifics.)
161   - Name space syntax is preserved:
162      - `<ns:key>something</ns.key>` parses to `map["ns:key"]interface{}{"something"}`
163      - `xmlns:ns="http://myns.com/ns"` parses to `map["xmlns:ns"]interface{}{"http://myns.com/ns"}`
164
165Both
166
167   - By default, "Nan", "Inf", and "-Inf" values are not cast to float64.  If you want them
168     to be cast, set a flag to cast them  using CastNanInf(true).
169
170<h4>XML encoding conventions</h4>
171
172   - 'nil' `Map` values, which may represent 'null' JSON values, are encoded as `<tag/>`.
173     NOTE: the operation is not symmetric as `<tag/>` elements are decoded as `tag:""` `Map` values,
174           which, then, encode in JSON as `"tag":""` values.
175   - ALSO: there is no guarantee that the encoded XML doc will be the same as the decoded one.  (Go
176           randomizes the walk through map[string]interface{} values.) If you plan to re-encode the
177           Map value to XML and want the same sequencing of elements look at NewMapXmlSeq() and
178           mv.XmlSeq() - these try to preserve the element sequencing but with added complexity when
179           working with the Map representation.
180
181<h4>Running "go test"</h4>
182
183Because there are no guarantees on the sequence map elements are retrieved, the tests have been
184written for visual verification in most cases.  One advantage is that you can easily use the
185output from running "go test" as examples of calling the various functions and methods.
186
187<h4>Motivation</h4>
188
189I make extensive use of JSON for messaging and typically unmarshal the messages into
190`map[string]interface{}` values.  This is easily done using `json.Unmarshal` from the
191standard Go libraries.  Unfortunately, many legacy solutions use structured
192XML messages; in those environments the applications would have to be refactored to
193interoperate with my components.
194
195The better solution is to just provide an alternative HTTP handler that receives
196XML messages and parses it into a `map[string]interface{}` value and then reuse
197all the JSON-based code.  The Go `xml.Unmarshal()` function does not provide the same
198option of unmarshaling XML messages into `map[string]interface{}` values. So I wrote
199a couple of small functions to fill this gap and released them as the x2j package.
200
201Over the next year and a half additional features were added, and the companion j2x
202package was released to address XML encoding of arbitrary JSON and `map[string]interface{}`
203values.  As part of a refactoring of our production system and looking at how we had been
204using the x2j and j2x packages we found that we rarely performed direct XML-to-JSON or
205JSON-to_XML conversion and that working with the XML or JSON as `map[string]interface{}`
206values was the primary value.  Thus, everything was refactored into the mxj package.
207
208