A Scala 3 toolkit for reading, writing, and round-tripping XML-based document formats. Originally
named KmlDoc and focused solely on KML, it's now split into a generic core module (typed
extraction and rendering, plus a generic fallback representation for content that isn't worth
modeling as its own type) and format-specific modules built on top of it.
See DESIGN.md for how the extraction/rendering framework actually works — the type
class hierarchy, the field-naming conventions, derivation (derives Extractor, Renderer), and the escape
hatches for irregular real-world XML. This README covers what the modules are and how to use them.
Full Scaladoc is at https://rchillyard.github.io/XmlDoc/latest/api/.
| Module | Artifact | Status |
|---|---|---|
core |
xmldoc-core |
Published. Format-agnostic: the Extractor[T]/Renderer[T] framework, streaming extraction (XmlStream), GenericElement (generic XML tree), Zip helpers. |
kml |
xmldoc-kml |
Published. A fairly complete typed model of the KML reference — extraction, rendering, and simple editing/merging of .kml files. |
kml-it |
not published | Integration tests against real third-party .kml files (deliberately not aggregated by the root project, so sbt test stays fast). |
json |
xmldoc-json |
New in 1.2.0. Records as JSON (circe), one per line, and the Spark SQL schema which reads them (see Records as JSON, for Spark). |
kafka-cats, kafka-zio |
xmldoc-kafka-cats, xmldoc-kafka-zio |
New in 1.2.0. Send records to Kafka, one JSON message per record, with fs2-kafka or zio-kafka (see Sending records to Kafka). |
kafka-it |
not published | Integration tests of the Kafka modules against a broker which they start (not aggregated by the root project: run sbt kafkaIt/test). |
idml |
xmldoc-idml |
Early-stage support for Adobe InDesign's IDML format, built on core's generic representation rather than kml's fully-typed one. Published since 1.1.2, but its API may still change. |
This version of the README.md file refers to version: 1.2.1.
Add the modules you need to your build.sbt,
where x.y.z is the latest version on Maven Central: it's shown on the Sonatype Central badge at the top of this page.
libraryDependencies ++= Seq(
"com.phasmidsoftware" %% "xmldoc-core" % "x.y.z"
, "com.phasmidsoftware" %% "xmldoc-kml" % "x.y.z"
)XmlDoc 1.1.3 and later need Scala 3.9 or later; for Scala 3.8, use 1.1.2. (No release supports Scala 3.3.)
XmlDoc doesn't include a logging backend (it logs through SLF4J): add one, such as logback-classic, to your own application if you want its log messages.
import com.phasmidsoftware.xmldoc.kml.*
import com.phasmidsoftware.xmldoc.xml.Extractor.extractMulti
import scala.xml.XML
import scala.util.Success
val xml = XML.loadFile("sample.kml")
extractMulti[Seq[KML]](xml) match {
case Success(kmls) => kmls.foreach(println)
case scala.util.Failure(x) => x.printStackTrace()
}To read (and write) XML of your own, describe each element as a case class and derive its Extractor and Renderer.
The field names say where each value is: _foo is the attribute foo, `$` the element's text, foo the child element foo,
maybeFoo: Option[...] an optional child foo, and a Seq field the repeated children (see DESIGN.md).
import com.phasmidsoftware.xmldoc.core.Text
import com.phasmidsoftware.xmldoc.render.Renderer
import com.phasmidsoftware.xmldoc.xml.Extractor
case class Reading(_hour: Int, $: Double) derives Extractor, Renderer
case class Elevation($: Double) // derived where it's needed
case class Station(_id: String, name: Text, maybeElevation: Option[Elevation], Readings: Seq[Reading]) derives Extractor, Renderer
val station: scala.util.Try[Station] =
Extractor.extract[Station](<station id="S1"><name>Logan</name><Reading hour="1">12.5</Reading></station>)A field whose type is another case class without instances of its own is derived in turn.
A case class may also have one auxiliary parameter, in a second parameter list, which is read from the same element
(e.g. kml's case class Placemark(Geometry: Seq[Geometry])(val featureData: FeatureData)).
For a sealed trait, derive the MultiExtractor of its sequences and its Renderer:
given MultiExtractor[Seq[Shape]] = MultiExtractor.derived and given Renderer[Shape] = Renderer.derived;
each subtype's tag is its class name.
Round-tripping (extract, then render back to XML) is exercised extensively in the test suites —
see kml/src/test/scala/.../KmlSpec.scala for realistic examples, including editing/merging
Placemarks.
The json module ("com.phasmidsoftware" %% "xmldoc-json" % "x.y.z") writes records as JSON, one object per line
(for a Kafka message or a .jsonl file), and derives from the same case class the Spark schema which reads that JSON,
so the two can't drift apart.
Build plain record case classes from the KML values you need (a type with a second parameter list, such as Placemark, can't be written directly),
and derive circe's Encoder and XmlDoc's SparkSchema for each:
import com.phasmidsoftware.xmldoc.json.{JsonLines, SparkSchema}
import io.circe.Encoder
case class Point(lon: Double, lat: Double) derives Encoder.AsObject, SparkSchema
case class Station(name: String, point: Point, elevation: Option[Double]) derives Encoder.AsObject, SparkSchema
JsonLines.line(Station("Logan", Point(-71.01, 42.37), Some(6.0)))
// {"name":"Logan","point":{"lon":-71.01,"lat":42.37},"elevation":6.0}
SparkSchema.ddl[Station]
// `name` STRING, `point` STRUCT<`lon`: DOUBLE, `lat`: DOUBLE>, `elevation` DOUBLEOn the Spark side (Scala 2.13 or PySpark), paste the schema string in and parse each message with from_json:
val schema = "`name` STRING, `point` STRUCT<`lon`: DOUBLE, `lat`: DOUBLE>, `elevation` DOUBLE"
val stations = spark.readStream.format("kafka").option("subscribe", "stations") /* ... */ .load()
.select(from_json(col("value").cast("string"), schema, Map.empty[String, String]).as("station"))
.select("station.*")Option[X] maps to a nullable X; Seq, List, Vector and Set to ARRAY; Map[String, X] to MAP<STRING, X>;
a nested case class to STRUCT; and an enum without parameters to STRING (derive its encoder with circe's ConfiguredEnumEncoder).
A value class has no Mirror, so give it the schema of what it's written as: given SparkSchema[Meters] = SparkSchema[Double].as.
xmldoc-kafka-cats (fs2-kafka) and xmldoc-kafka-zio (zio-kafka) send each record to a topic as one JSON message
(its JsonLines.line), with a key of your choice. They use TableParser's Kafka modules (tableparser-kafka-cats/-zio).
import cats.effect.IO
import com.phasmidsoftware.xmldoc.kafka.cats.IOKafka
import fs2.Stream
import fs2.kafka.ProducerSettings
val settings = ProducerSettings[IO, String, String].withBootstrapServers("localhost:9092")
Stream.emits(stations).through(IOKafka.toKafka("stations", settings)(_.name)).compile.drainimport com.phasmidsoftware.xmldoc.kafka.zio.ZIOKafka
import zio.ZLayer
import zio.kafka.producer.{Producer, ProducerSettings}
import zio.kafka.serde.Serde
import zio.stream.ZStream
val producer = ZLayer.scoped(Producer.make(ProducerSettings(List("localhost:9092"))))
ZStream.fromIterable(stations).via(ZIOKafka.toKafka("stations", Serde.string)(_.name)).runDrain.provideLayer(producer)For messages without a key, use a Unit key: ProducerSettings[IO, Unit, String] with (_ => ()), or ZIOKafka.unitSerializer.
The Spark job then reads the topic with from_json and SparkSchema.ddl (above).
KAFKA.md walks through the whole pipeline (with diagrams), from a KML file to a Kafka broker running in Docker
(docker compose up -d, with this repository's docker-compose.yml) and on to Spark.
To avoid loading a whole document before extracting from it, XmlStream (in core) reads it with a pull parser (StAX)
and extracts each element which matches a selector as soon as that element has been read; nothing else is kept.
A selector is a tag (Placemark, at any depth), a path of tags (Folder/Placemark), or a path from the root (/kml/Document/Placemark);
tags are matched by their local names, whatever their namespace. Each element's extraction succeeds or fails on its own (a Try).
import com.phasmidsoftware.xmldoc.xml.XmlStream
import java.io.FileInputStream
XmlStream.extractFrom[Placemark, Int]("Placemark", new FileInputStream("big.kml"))(_.count(_.isSuccess))The Kafka modules stream the same way, so records go to Kafka as they are read (IOXmlStream for fs2, ZIOXmlStream for ZIO):
import com.phasmidsoftware.xmldoc.kafka.cats.{IOKafka, IOXmlStream}
import scala.util.Success
IOXmlStream.extractFile[Placemark]("big.kml", "Placemark")
.collect { case Success(p) => p }
.flatMap(p => Stream.emits(toStations(p)))
.through(IOKafka.toKafka("stations", settings)(_.name))
.compile.drainBoth read the selected elements 64 (chunkSize) at a time, so the memory they use is bounded by a chunk of elements,
however big the file (see KAFKA.md).
A record which needs something from outside its element (e.g. a Style defined at the document level) must be completed separately.
Not yet released: ZIOXmlStream reads the selected elements 64 at a time, like IOXmlStream, rather than ZIO's default chunk of 4096: Issue #101.
Version 1.2.1 Fixes KMLCompanion.loadKML(URL), which loaded the URL's path as a local file name, so that it failed for a resource
in a jar (that is, whenever xmldoc-kml was a dependency), for an http: URL, and for a file: URL whose path has a space;
it now reads the URL itself: Issue #95.
The version scheme is now PVP rather than early-semver: the first two numbers make the major version, so a patch release
(such as this one) is binary compatible with the previous release, while a minor release may break compatibility, with a release note: Issue #96.
MiMa checks that, in the build and in CI (sbt mimaReportBinaryIssues): Issue #98.
The tests no longer write their output into the working directory: Issue #97.
Version 1.2.0 Derives Extractor and Renderer from case classes (derives Extractor, Renderer, or Extractor.derived/Renderer.derived),
including the auxiliary-parameter pattern, and MultiExtractor/Renderer for sealed traits (MultiExtractor.derived): Issue #63.
This replaces the arity ladder (extractor10 ... extractorPartial72, multiExtractorN, rendererN, rendererNSuper, rendererSuperN),
which is deprecated, and kml now uses derivation throughout; its abstract types (Feature, Geometry, ...) are sealed.
logback-classic is no longer a (runtime) dependency.
A new module, json (xmldoc-json), writes records as JSON Lines with circe, and derives the Spark SQL schema which reads them (SparkSchema.ddl): Issue #64.
Two more, kafka-cats and kafka-zio (xmldoc-kafka-cats, xmldoc-kafka-zio), send records to Kafka as JSON messages, using TableParser's Kafka modules: Issue #65.
XmlStream (in core) extracts the selected elements of a large document as they are read, with a StAX pull parser,
and IOXmlStream/ZIOXmlStream (in the Kafka modules) do so as fs2/ZIO streams: Issue #66.
Version 1.1.3 Adds 3-way merging of XML trees to core (package merge): a PCS representation of
trees, after Lindholm (Pcs), edit detection (PcsEditDetector), reconciliation of two edit sets
(PcsMerger), and reconstruction of the merged tree (PcsTreeBuilder). The idml module wires
these together end to end (ThreeWayMerger) and adds a runnable git merge driver for .idml
files (IdmlMergeDriver); kml gets a full real-document 3-way merge test. Built with Scala 3.9.0
(the current LTS), so it needs Scala 3.9 or later (1.1.2 needs 3.8). Uses Flog 1.1.0 and Args 2.1.0.
Version 1.1.0 Split into core/kml/kml-it modules; renamed KmlDoc → XmlDoc; published to
Maven Central. idml module added (early stage, unreleased). The kml round-trip tests were
found to be silently non-functional (they never actually re-parsed rendered output), and fixing
that surfaced several genuine bugs, including a gap in the original #19 fix (an empty self-closing
element still crashed extraction) plus Issues #20, #21, #27, #29, #44, #51 — see the
issue tracker for details
on each.
Version 1.0.5 Migrate to Scala 3, ...
Version 1.0.4 Many issues fixed. All non-gx entities are implemented now.
Version 1.0.3 Fixed Issue #19, various other mostly cosmetic changes.
Version 1.0.2 Improvements to rendering and extraction with more KML objects possible.
Version 1.0.1 Improvements: first round-trip extract/render of minisample.kml
Version 1.0 Preliminary work. Most of the functionality is working but there is much still to do, especially for the KML_Samples.kml file.