Data Fusion instead of Calcite?

[sorenbs]

DBSP is really exciting, and the Nexmark benchmark speaks for itself!

Was it considered to base the SQL compiler on Data Fusion instead of Calcite to avoid the large Java component? Arroyo takes this approach as described in the SQL section at the bottom of this page: https://doc.arroyo.dev/concepts

The DBSP to SQL compiler repo does not have Discussions enabled, so I am asking here instead.

[mihaibudiu]

This repository will soon be archived. The sql compiler is now part of this repository: github.com/feldera/dbsp.

We have not considered data fusion. I am not familiar with that tool, i will take a look. Calcite does many things, including being very flexible and extensible, having an optimizer, and having a large community. It also has a lot of bugs. Whether we made the right choice will be hard to tell. Java in itself is not really a problem. With the resources we have we can't try anything else now. Sql is a huge language and covering most of it will require many person-years of work.

But, if you are interested to try another approach, we will be happy to offer advice and/or accept contributions.

[ryzhyk]

@sorenbs , as my colleague said, our repo now lives here.

I read on HN that you're interested in running benchmarks. Happy to help with that.