# Shale
Shale is a Ruby object mapper and serializer for JSON, YAML, TOML, CSV and XML.
It allows you to parse JSON, YAML, TOML, CSV and XML data and convert it into Ruby data structures, as well as serialize data structures into JSON, YAML, TOML, CSV or XML.
# Introduction
Working with data serialization formats directly can be painfull.
This is especially true for XML. Let's consider this simple example of adding an
address to a person using Nokogiri
:
That's a lot of code for very simple use case. Anything more complex and code complexity increases exponentially leading to a maintanace problems and errors.
With Shale you can use Ruby objects to work with data converting it to/from JSON, YAML, TOML, CSV or XML.
Let's convert the same example to Shale:
That's much simpler and it stays simple when the code complexity increases.
# Prerequisites
If you want to work with XML you will need one of:
If you want to work with TOML you will need one of:
Shale doesn't have external dependencies. It uses standard library's
JSON
, YAML
and CSV
parsers by default.
If you need more customizations you can use custom libraries.
Out of the box, Shale provides adapters for
REXML
, Nokogiri
, Ox
and toml-rb
,
but you can provide your own adapters - see how to.
# Features
- Convert JSON, YAML, TOML, CSV and XML to Ruby data model
- Convert Ruby data model to JSON, YAML, TOML, CSV and XML
- Generate JSON and XML Schema from Ruby models
- Compile JSON Schema into Ruby models
- Out of the box support for JSON, YAML, CSV, Tomlib, toml-rb, Nokogiri, REXML and Ox parsers
- Support for custom adapters
# Installation
Add this line to your application's Gemfile:
And then execute:
Or install it yourself as:
# Convert data to Ruby
JSON, YAML and CSV are supported by default, but to work with XML you need to install XML parser and setup adapter. For example to setup REXML as an adapter use:
If you want to work with TOML you will need to install Tomlib parser and setup it as an adapter:
Converting data to Ruby is as simple as defining model classes and calling
from_<format>
method on this class.
e.g. Person.from_json(json_doc)
NOTE
CSV represents a flat data structure, so you can't map properties to complex types directly, but you can use attribute delegation or methods to map properties to complex types (see using methods to extract and generate data section).
WARNING
.from_csv
method allways returns an array of records.
# Convert Ruby to data
To convert Ruby to data just define model class, initialize object and call
to_<format>
method on it.
e.g. Person.new(name: 'John Doe').to_json
# Convert Collections
Shale allows converting collections for formats that support it (JSON, YAML and CSV).
To convert data to Ruby array:
To convert Ruby array to data:
# Custom mappings
When you define a class and add attributes, underneath Shale creates an implicit mapping of keys (for JSON/YAML/TOML), columns (for CSV) or elements (for XML) to attributes. That is nice for setting up your data model quickly, but usually your data format doesn't match your data model so cleanly.
That's why you can explicitly map keys, element and attributes from your data format to attributes on your Ruby model.
WARNING
Declaring custom mapping removes default mapping for given format!
NOTE
For CSV the order of mapping matters. The first argument in the map
method is only used as a label in header row. So, in the example below the first
column will be mapped to :first_name
attribute and the second
column to :last_name
.
XML is more complex format.
- To map XML element use
map_element
- To map XML attribute use
map_attribute
-
To map XML text node use
map_content
(it will map first text node of an element) - To change the name of the root element use
root
You can use cdata: true
option on map_element
and map_content
to handle CDATA
nodes:
# Using XML namespaces
To Work with XML namespaces you need to use adapter that supports XML namespaces.
Use one of Shale::Adapter::REXML
or Shale::Adapter::Nokogiri
.
Shale::Adapter::Ox
doesn't support namespaces.
To map namespaced elements and attributes use
namespace
and prefix
properties on
map_element
and map_attribute
To define default namespace for all elements use namespace
declaration
(this will define namespace only on elements, if you want to define
namespace on an attribute, explicitly declare it on map_attribute
).
# Rendering nil values
For JSON, YAML, TOML and XML by default elements with nil
value are not rendered. You can change this behavior by using
render_nil: true
on a mapping.
For CSV, the default is to render nil
elements.
If you want to change how nil values are rendered
for all mappings you can use render_nil
method:
WARNING
The default affects only the mappings declared after setting the default value e.g.
# Using methods to extract and generate data
If you need full controll over extracting and generating data you can use methods to do so.
You can also pass a context
object
that will be available in extractor/generator methods:
If you want to work on multiple elements at a time
you can group them with group
block:
# Attribute delegation
To delegate fields to child complex types you can use
receiver: :child
declaration on a mapping.
# Additional options
You can control which attributes to render and parse by
using only: []
and except: []
parameters.
By default generated JSON and XML are compacted. If you need human readable format use
pretty: true
parameter on #to_json
and #to_xml
You can also add an XML declaration by passing
declaration: true
and encoding: true
to #to_xml
.
You can use specific version by using string argument e.g.
#to_xml(declaration: '1.1', encoding: 'ASCII')
For CSV you can pass headers: true
to indicate that the first row contains
column names and shouldn't be included in the returned collection.
It also accepts all the options that
CSV parser
accepts.
# Using custom models
By default Shale combines mapper and model into one class.
If you want to use your own classes as models you can do it by using
model
directive on the mapper:
# Generating JSON and XML Schema
WARNING
Shale only supports Draft 2020-12 JSON Schema
To generate JSON or XML Schema from you Shale data model use:
You can also use a command line tool to do it:
or XML Schema:
If you want to convert your own types to Schema types use:
# Compiling JSON and XML Schema
To compile JSON or XML Schema and generate Ruby data model use
Shale::Schema.from_json
or Shale::Schema.from_xml
.
You can pass namespace_mapping: {}
to map JSON schemas
or XML namespaces to Ruby modules. For JSON schema, you can also pass
root_name: 'Foobar'
to change the name of the root type.
You can also use a command line tool to do it (JSON Schema):
and XML Schema:
# Supported types
Shale supports these types out of the box:
Shale::Type::String
Shale::Type::Integer
Shale::Type::Float
Shale::Type::Boolean
Shale::Type::Date
Shale::Type::Time
To add your own type define a class and extend it from
Shale::Type::Value
and implement .cast
class method.
# Adapters
Shale uses adapters for parsing and generating documents. By default Ruby's standard JSON, YAML and CSV parsers are used for handling JSON, YAML and CSV documents.
You can change it by providing your own adapter. For JSON, TOML, YAML and CSV, adapter
must implement .load
and .dump
class methods.
To handle TOML documents you have to explicitly set TOML adapter.
Out of the box Tomlib
is supported. Shale also provides adapter for toml-rb
parser:
To handle XML documents you have to explicitly set XML adapter. Shale provides adapters for most popular Ruby XML parsers:
WARNING
Ox parser doesn't support XML namespaces