There are thousands of ways to note down a date and time.
The international date format is standardized as [ISO8601][iso], though it still allows a widespread of different formats.
The basic format looks like this:
> 2015-07-02T19:45:00+0100
And that's what we will parse today using [nom][nom],
a parser combinator library created by [Geoffroy Couprie][gcouprie].
The idea is that you write small self-contained parsers, which all do only one simple thing, like parsing the year in our string, and then combine these small parsers to a bigger one to parse the full format.
`nom` comes with a wide variety of small parsers: handling different integers, reading simple byte arrays, optional fields, mapping parsed data over a function, ...
Most of them are provided as combinable macros.
It's very easy to implement your own small parsers, either by providing a method that handles a short byte buffer or by combining existing parsers.
So let's dive right in and see how to use nom in real code.
### Analysis
This is what we want to parse:
> 2015-07-02T19:45:00+0100
It has several parts we need to parse:
> YYYY-MM-DDTHH:MM:SS+OOOO
with the following meaning:
| Characters | Meaning |
| ---------- | ------- |
| YYYY | The year, can be negative or null and can be extended if necessary |
| MM | Month from 1 to 12 (0-prefixed) |
| DD | Day from 1 to 31 (0-prefixed) |
| T | Separator between date and time |
| HH | Hour, 0-23 (0-prefixed) |
| MM | Minutes, 0-59 (0-prefixed) |
| SS | Seconds, 0-59 (0-prefixed) |
| OOOO | Timezone offset, separated by a `+` or `-` sign or `Z` for UTC |
Parts like the seconds and the timezone offset are optional. Datetime strings without them will default to a zero value for that field.
The date parts are separated by a dash (`-`) and the time parts by a colon (`:`).
We will built a small parser for each of these parts and at the end combine them to parse a full date time string.
### Parsing the date: 2015-07-16
Let's start with the sign. As we need it several times, we create its own parser for that.
Parsers are created by giving them a name, stating the return value (or defaulting to a byte slice) and the parser combinators to handle the input.
~~~rust
named!(sign <&[u8], i32>, alt!(
tag!("-") => { |_| -1 } |
tag!("+") => { |_| 1 }
)
);
~~~
First, we parse either a plus or a minus sign.
This combines two already existing parsers: `tag!`, which will match the given byte array (in our case a single character) and `alt!`, which will try a list of parsers, returning on the first successful one.
We can directly map the result of the sub-parsers to either `-1` or `1`, so we don't need to deal with the byte slice later.
Next we parse the year, which consists of an optional sign and 4 digits (I know, I know, it is possible to extend this to more digits, but let's keep it simple for now).
This takes 4 (or 2) characters from the input and checks that each character is a digit.
`flat_map!` and `check!` are quite generic, so they are useful for a lot of cases.
~~~rust
named!(pub year <&[u8], i32>, chain!(
~~~
The year is also returned as a 32-bit integer (there's a pattern!).
Using the `chain!` macro, we can chain together multiple parsers and work with the sub-results.
~~~rust
pref: opt!(sign) ~
y: positive_year
~~~
Our sign is directly followed by 4 digits. It's optional though, that's why we use `opt!`.
`~` is the concatenation operator in the `chain!` macro.
We save the sub-results to variables (`pref` and `y`).
~~~rust
,
|| {
pref.unwrap_or(1) * y
}));
~~~
To get the final result, we multiply the prefix (which comes back as either `1` or `-1`) with the year.
Don't forget the `,` (comma) right before the closure.
This is a small syntactic hint for the `chain!` macro that the mapping function will follow and no more parsers.
We can now successfully parse a year:
~~~rust
assert_eq!(Done(&[][..], 2015), year(b"2015"));
assert_eq!(Done(&[][..], -0333), year(b"-0333"));
~~~
Our nom parser will return an `IResult`. If all went well, we get `Done(I,O)` with `I` and `O` being the appropriate types.
For our case `I` is the same as the input, a buffer slice (`&[u8]`), and `O` is the output of the parser itself, an integer (`i32`).
The return value could also be an `Error(Err)`, if something went completely wrong, or `Incomplete(u32)`, requesting more data to be able to satisfy the parser (you can't parse a 4-digit year with only 3 characters input).
Parsing the month and day is a bit easier now: we simply take the digits and map them to an integer:
named!(pub second <&[u8], u32>, map!(call!(take_2_digits), buf_to_u32));
~~~
Putting them together becomes a bit more complex, as the `second` part is optional:
~~~rust
named!(pub time <&[u8], Time>, chain!(
h: hour ~
tag!(":") ~
m: minute ~
s: empty_or!(chain!(tag!(":") ~ s:second , || { s }))
,
|| { Time{ hour: h,
minute: m,
second: s.unwrap_or(0),
tz_offset: 0 }
}
));
~~~
As you can see, even `chain!` parsers can be nested.
The sub-parts then must be mapped once for the inner parser and once into the final value of the outer parser.
`empty_or!` returns an `Option`. Either `None` if there is no input left or it applies the nested parser. If this parser doesn't fail, `Some(value)` is returned.
We now got individual parsers for the date, the time and the timezone offset.
Putting it all together, our final datetime parser looks quite small and easy to understand:
~~~rust
named!(pub datetime <&[u8], DateTime>, chain!(
d: date ~
tag!("T") ~
t: time ~
tzo: empty_or!(call!(timezone))
,
|| {
DateTime{
date: d,
time: t.set_tz(tzo.unwrap_or(0)),
}
}
));
~~~
Nothing special anymore. We can now parse all kinds of date strings:
~~~rust
datetime("2007-08-31T16:47+00:00");
datetime("2007-12-24T18:21Z");
datetime("2008-02-01T09:00:22+05");
~~~
But it will also parse invalid dates and times:
~~~rust
datetime("2234-13-42T25:70Z");
~~~
But this is fine for now. We can handle the actual validation in a later step.
For example, we could use [chrono][], a time library, [to handle this for us][chrono-convert].
Using chrono it's obvious why we already multiplied our timezone offset to be in seconds: this time we can just hand it off to chrono as is.
The full code for this ISO8601 parser is available in [easy.rs][easy.rs]. The repository also includes [a more complex parser][lib.rs], that does some validation while parsing
(it checks that the time and date are reasonable values, but it does not check that it is a valid date for example)
### What's left?
These simple parsers or even some more complex ones are already usable.
At least if you already got all the data at hand and if a simple return value satisfies your needs.
But especially for larger and more complex formats like media files reading everything into memory and spitting out a single large value isn't sufficient at all.
nom is prepared for that.
Soon it will become as easy as using an object from which nom can [`Read`][read].
For most things you shouldn't worry about that, as a simple `BufReader` will work.
For the other end of the chain, nom has [Consumers][consumer].
A Consumer handles the complex part of actually requesting data, calling the right sub-parsers and holding the necessary state.
This is what you need to build yourself.
Internally it's best abstracted using some kind of state machine, so you always know which part of the format to expect next, how to parse it, what to return to the user and so on.
Take a look at [the MP4 parser][mp4], which has an `MP4Consumer` handling the different parts of the format.
Soon my own library, [rdb-rs][rdb-rs], will have this as well.
Small thing aside: Geoffroy created [machine][] to define a state machine and I got [microstate][] for this.
### Why am I doing this?
I'm currently developing [rdb-rs][rdb-rs], a library to parse and analyze Redis dump files.
It's currently limited to parsing and reformatting into several formats and can be mainly used as a CLI utility.
But [there are projects][rsedis] that could benefit from a nicer API to integrate it into another tool.
The current parser is hand-made. It's fast, it's working, but it provides a limited, not very extensible API.
I hope to get a proper parser done with nom, that I can build on to provide all necessary methods, while still being super-fast and memory-safe.
Work [already started][rdb-rs-nom], but I'm far from done for now
---
Thanks to [Geoffroy][gcouprie] for the discussions, the help and for reading a draft of this post.
