In the upcoming 0.20 release, we've improved the editing experience for writing integrations within the browser.

Vyne operates as a smart data gateway, allowing you query, connect and transform data, across multiple APIs, databases, Kafka streaming sources and EC2 data stores - all without writing any integration code.

With Vyne, you don't build integrations between data sources - you simply write queries asking for the data you need.  Vyne handles all the integration stuff for you.  

The heart of a good querying experience is the editor, and we've been improving ours. Now when you're writing a query in Vyne, you get as-you-type code completion, with intelligent suggestions.

Queries in Vyne are written in TaxiQL - a universal, open-source query language, which allows defining queries that can span APIs, databases, and streaming sources.  That's really powerful, but it's also really new.  

TaxiQL is somewhere between a SQL statement, and a GraphQL statement - and while it quickly feels familiar, there's always a learning curve with anything new.  That's why it's so important to get code hinting right.

Code hinting is available by pressing Ctrl+Space whilst writing a query.  The hints you get depend on where in the query you are, and what's available in your infrastructure.  Let's take a look at some of the help Vyne gives you.

Find vs Stream

When writing queries, Vyne is able to connect to traditional "Request / Response" data sources (like databases, RESTful APIs, etc), as well as "Streaming" data rouces - like Kafka topics.

In TaxiQL, this is defined by specifying your query as find or stream respectively.

When you're writing your query in Vyne, typing find will prompt you with data available from request-response data sources, whereas typing stream will prompt you with data available from streaming sources.

vyne-find-vs-stream-1

Improving Filtering

In TaxiQL, filters are defined in parenthesis against the type you're searching against. Criteria are defined using Type names, rather than field names. (That's because Vyne is searching against mutliple data sources, each likely has it's own field name).

So, to find all the actors named "Jimmy", you might write:

find { Actor[]( FirstName = 'Jimmy' ) }

The kinds of filters you can apply depend on the data sources you have available. For example, if you have a database exposing Actors, you can do rich searching. However, if you have an API that exposes actors, the types of filters you can apply depend on what the API has exposed.

Vyne's query editor understands this, so when you're writing a filter, pressing Ctrl+Space will prompt you only with the attribute types that are valid for defining a filter with.

vyne-filtering

Knowing what data is discoverable

Vyne lets you enrich data from one source, by linking it with data from multiple other sources, and then reshaping into the form you need. These are called "projections".

For example, you might want to find all the films from the Film database, and then enrich it with data from a REST API. Vyne's powerful integration engine lets you do this, without writing any integration code.

Given the wide world of your enterprise data, knowing what's discoverable depends on your starting point.

As you type your query, Vyne is constantly scanning to see what data it can provide, by stitching together all the different data sources in your organisation.

By pressing Ctrl+Space when defining a projection, Vyne will list all the data attributes available to you, ie., those that are discoverable based on the query so far.

vyne-projection-fields

Seeing where data comes from

Vyne is linking data together from multiple different sources - sometimes taking muliple "hops" between sources to serve up the data.

This means the list of attributes provided can be sourced from multiple different data sources. Sometimes it's useful to know how Vyne will fetch data, as you're building your query.

When you're writing a projection in a query, pressing Ctrl+Space twice brings up a richer information window, which shows the actual path that will be navigated to fetch this attribute

attribute-show-data-sources

Improved lineage views

In addition to improving the hints you get while editing a query, we've also improved the view of lineage available after the results have been returned.

Vyne 0.20 ships with a new query lineage view, that shows all the different systems that were involved in executing a query, along with the number of results that travelled each path.

Screenshot-from-2022-02-22-14-41-23

SQL & streaming sources in sequence diagrams

We've had long support for sequence diagrams showing execution plans. These have now been enriched to show SQL queries and streaming sources invoked during a query.

sequence-diagram-1

Clicking on any SQL source will drill down to show the actual SQL query issued, along with how the query performed:

sql-in-drilldown

SQL & streaming sources in cell-based-lineage

We've also extended our cell-based lineage to include all the details of SQL queries and real-time sources (eg., Kafka).

Clicking on any value in the result of a query shows the lineage for that value, including SQL queries that were issued and API calls that were made, including inputs and parameters.

cell-lineage

Summing up

Vyne 0.20 is just around the corner.

These features are available now in our EAP builds, by using the docker tag vyneco/vyne:0.20.0-SNAPSHOT.

There's also a full walkthrough showing how to use some of the new query editor features, along with slick new data source onboarding available in our latest tutorial.

Take the latest build for a spin, and let us know your thoughts by reaching out in our Slack Channel. Hope to see you there soon!