Skytable 0.8.4: Multiple fixes and some new utilities

We’re extremely happy to announce the release of Skytable 0.8.4. This release is primarily a bugfix release but does include some useful additions. Let’s take a moment to review the changes in this version. Out of time? Directly jump to this section for download links.

DDL/DML imporvements

Skyhash: Dynamic lists

One of the things that has annoyed us, and has probably bothered you too would be with using lists. While the collections API itself remains incomplete (for example, selection and interpolation of a single element by key) due to some decisions we have to make about our distributed architecture (in case you’re interested, it’s a debate between frozen and non-frozen collections and their resulting complexity), one thing that has been there from the start is the lack of flexibility when using lists.

For example, this is how you would insert a list:

INSERT INTO myspace.mymodel { username: ?, notes: [?, ?, ?] }

Now, if you wanted to include more elements in the notes field — you couldn’t. The only mitigation would have been manually constructing the query string which is very dangerous and is something that we strongly advise against as it completely undermines BlueQL’s security guarantees.

From now on, you can simply pass a list as a parameter. Here’s an example using the Rust driver:

use skytable::{query, query::QList};

fn get_from_mobile_app() -> Vec<String> { /* fetch data */ }

let notes = get_from_mobile_app();
let q = query!(
    "insert into myspace.mymodel { username: ?, notes: ? }",
    "sayan",
    QList::new(&notes),
);

The client driver will encode the list into a parameter and the server will turn it into an appropriate type.

Truncate

We restored the TRUNCATE MODEL query because … well, it was missing. And to be honest with you, we don’t really have a good explanation for that; there are some questions in the universe that don’t have an answer and we believe this is one of them.

Jokes aside, root users (and root users only) can now clear all rows belonging to a model by running:

TRUNCATE MODEL myspace.mymodel

Simple list syntax

Skytable’s list definitions appear rather complicated and if you’ve wondered why, it’s actually a feature not a bug; it’s for extensibility. But it can very much get in the way.

You previously had to do this:

CREATE MODEL myspace.mymodel(uid: string, followers: list { type: string })
ALTER MODEL myspace.mymodel ADD following { type: list { type: string } }

With this change, you can now simply do this instead:

CREATE MODEL myspace.mymodel(uid: string, followers: [string])
ALTER MODEL myspace.mymodel ADD following { type: [string] }

Fixes

As we noted earlier, this release is filled with various fixes. A bunch of skysh bugs were ironed out including issues with printing multi-row responses and row numbers. We also fixed several memory leaks in skyd which although mostly benign, are still important. Several other smaller changes and fixes have been implemented. Finally, we carried out a full audit of unsafe code in the source using currently available tools.

What’s brewing

A quick summary: (1) Our partitioning algorithms are working as expected (2) failure detection and inter-node communication is stable besides certain control message amplification issues (3) failure/disaster recovery is something that we’re spending a lot of time on, particularly because it’s challenging to design a specialized test harness (4) due to scalability issues, we’re building a custom consistency protocol instead of using an off-the-shelf protocol. We aim to initially ship a slightly modified off-the-shelf consistency protocol along with our v1 partitioning scheme in 0.9.0; we aim to roll out our custom consistency protocol in 1.0.

The release is just around the corner; we want to make sure that we release an efficient, reliable and stable system for users so that horizontal scalability would no longer be a limit.

Bottom line

This is a small but important release and we would recommend everyone using an older version to upgrade. We remain hard at work on Skytable 0.9.0, which will be the release in which we’ll ship clustering and replication (v1).

As we work on the next landmark release, in the interim, please go ahead and download the latest release, benchmark it to see how much raw power you can pull out of our servers and report any bugs that you find. You can download the latest release here.

We’ll see you at the next release; and just like you, we really hope it’s 0.9!