Support serde::Serialize and serde::Deserialize for the engine

[azoyan]

Is it possible to implement traits Serialization and Deserialization?

If it difficult may be provide API to inner jsonpath query that stored in MainEngine?

[github-actions]

Tagging @V0ldek for notifications

[V0ldek]

Do you mean serializing and deserializing the MainEngine?

So the MainEngine consists of the Automaton, which is the compiled query, and the SimdConfiguration.

1. The SimdConfiguration is not something one should serialize since it's not portable by design, it contains the SIMD capabilities of the host CPU.
2. The Automaton can be serialized, most likely automatically derived by serde-derive.

So to serialize one would serialize only the Automaton; and for deserialize one would deserialize the Automaton and use simd::configure() to get a new configuration applicable to the host CPU.

Support for serialization must be behind a new serde optional feature though, to not bloat dependencies unnecessarily. It will have to reach rsonpath-syntax as well (to serialize JsonUInt and JsonString and such). At that point it's probably worth to make the entire JsonPathQuery serializable.

I'd expect some basic proptests to make sure serialization round-trips. Could you tell me what format you're planning on serializing into?

[V0ldek]

One more thing is versioning – we don't expect the Automaton to be binary-compatible between different versions of rsonpath so serialization needs to save an additional version tag. We'll need a snapshot test that will tell us when the binary payload changes.

[azoyan]

We use engine like this: we convert paths from Vec<String> to Vec<RsonpathEngine>

let paths: Vec<String> = vec![
    "$.personal.details.contact.information.phones.home".to_string(),
    "$.personal.details.contact.information.phones.office".to_string(),
];
let engines: Vec<RsonpathEngine> = paths
    .iter()
    .map(|path| {
        let a = Automaton::new(&rsonpath_syntax::parse(path).unwrap()).unwrap();
        RsonpathEngine::from_compiled_query(a)
    })
    .collect();

And I thought it would be a good idea. Deserialize engine from jsonpath, and serialize to path.

But maybe I'm missing something. And this view is one narrow use case among many. Then good way will be provide API just about jsonpathquery ($.personal.details.contact.information.phones.home). I will implement Serialize/Deserialize for my own wrapper using this information.

[V0ldek]

I see. The engine doesn't preserve the underlying query, so you'd need to hold it alongside.

The main reason to make the engine serialize/deserialize would be to save on compilation time. Parsing and compiling an engine is relatively fast, but I'm pretty sure deserializing it from a binary format would be much faster, so it's something that might be worth supporting.

For example, a quick microbenchmark tells me cloning an engine with your query is ~10x faster than recompiling it. Deserialization from a binary payload would presumably be closer to a clone than recompilation.

In any case, a parsed JsonPathQuery can already be converted to string, so this code does work:

let query_str_1 = "$.personal.details.contact.information.phones.home";
let query_1 = rsonpath_syntax::parse(query_str_1).unwrap();
let automaton_1 = Automaton::new(&query_1).unwrap();

let query_str_2 = query_1.to_string();
let query_2 = rsonpath_syntax::parse(&query_str_2).unwrap();
let automaton_2 = Automaton::new(&query_2).unwrap();
assert_eq!(query_1, query_2);
assert_eq!(automaton_1, automaton_2);

Just note that the string representation is not preserved as we convert to a canonical representation, so:

println!("{query_str_2}");

prints

$['personal']['details']['contact']['information']['phones']['home']

[azoyan]

I realize in your code:
main.rs

impl MainEngine {
    pub fn get_jsonpath_strings(&self) -> Vec<String> {
        self.automaton.get_jsonpath_strings()
    }
}

automation.rs

impl Automation {
    #[must_use]
    #[inline(always)]
    pub fn get_jsonpath_strings(&self) -> Vec<String> {
        self.states
            .iter()
            .flat_map(|state| state.member_transitions.iter().map(|(pattern, _)| pattern.unquoted()))
            .map(|pattern| std::str::from_utf8(pattern).unwrap().to_string())
            .collect()
    }
}

I am satisfied with this result:

let paths = vec![
    "$.personal.details.contact.information.phones.home".to_string(),
    "$.personal.details.contact.information.phones.office".to_string(),
];
let engines: Vec<RsonpathEngine> = paths
    .iter()
    .map(|path| {
        let a = Automaton::new(&rsonpath_syntax::parse(path).unwrap()).unwrap();
        RsonpathEngine::from_compiled_query(a)
    })
    .collect();

let s = engines.get(0).unwrap().get_jsonpath_strings();
println!("{:?}", s)

["personal", "details", "contact", "information", "phones", "home"]

Is possible to add API like this?

[V0ldek]

I don't like it, it's not obvious what this function would do for more complex queries. Does $['a', 'b'] return ["a", "b"]? What about nested stuff with filters, like:

$.a[?@.b[?@.c]].d

Is the semantics literally "return the labels from left to right as in the query"? That sounds like a syntactic property, and the Automaton doesn't need to preserve those.

For syntactic properties you have the JsonPathQuery. If you pass it along with the engine you can do this:

use rsonpath_syntax::prelude::*;

let paths = vec![
    "$.personal.details.contact.information.phones.home".to_string(),
    "$.personal.details.contact.information.phones.office".to_string(),
];
let engines: Vec<(JsonPathQuery, RsonpathEngine)> = paths
    .iter()
    .map(|path| {
        let q = rsonpath_syntax::parse(path).unwrap();
        let e = RsonpathEngine::compile_query(&q).unwrap();
        (q, e)
    })
    .collect();

let s: Vec<&str> = engines[0]
    .0
    .segments()
    .iter()
    .filter_map(|s| match s {
        Segment::Child(ss) => match ss.first() {
            Selector::Name(str) => Some(str.unquoted()),
            _ => None,
        },
        _ => None,
    })
    .collect();
println!("{:?}", s);

["personal", "details", "contact", "information", "phones", "home"]

If something like this is useful we could add it to the rsonpath-syntax public API as a left-to-right gathering of all JsonStrings in the query. But it doesn't fit into a method on the engine as that one has no responsibility to preserve the syntax.

[azoyan]

@V0ldek in #625 i make more natural function name.

I use natural names literally as in code

[azoyan]

Or may be rename to quoted_jsonstring_patterns

[V0ldek]

Okay, so continuing the discussion on serialization/deserialization, I can implement it easily. If you could please tell me what target format will you plan to use so that I can add snapshot tests specifically for that it'd be nice.

[azoyan]

Let me tell you how we use it.

All we need is from jsonpathquery compile and match incoming json.

We store jsonpath as parameter config yaml

jsonpath:
  - $.phones.home
  - $.phones.office

And then we check incoming JSON if is matched or not.

Thats why I need to serialize/deserialize to something that looks like jsonpath.

Thats why I use unqoted_segment: I can manually join('.") and make original string

[azoyan]

I think you can implement serialize / deserialize as you see fit.

For my tasks i wrote wrapper around Engine and store original jsonpathquery in the string =)

[V0ldek]

This was released in 0.9.4.