Migration path

[retorquere]

[x] Documentation issue or request

Would it be possible to offer in-place upgrade from lokijs 1 serialization format to 2 in loadDatabase? I don't mind doing it myself as long as there would be a clear transformation I could do inside my own load handler.

[Viatorus]

Yes, this could be possible.

Maybe a hook on the function loadJSONObject https://github.com/LokiJS-Forge/LokiJS2/blob/13a3d4756eb773f6c0d3bd25c6310022fbe45712/packages/loki/src/loki.ts#L727 could do it.

But at the moment, the API is not final so there can be still changes.

[Viatorus]

@obeliskos
I refactoring the persistence logic of LokiDB to support LokiJS databases and also prepare for future database changes with an easy but typesafe mechanism (more about this after my vacation [4 weeks]).

LokiJS' loadDatabase currently supports Loki objects and Loki serialized JSON objects
Is it really necessary to support Loki objects in the future? Dropping this feature would it make simpler.

Note:

• In LokiJS: Loki object and loki serialized object share the same properties.
• In LokiDB: This isn't the fact.

[obeliskos]

Interesting, looking forward to hearing your ideas on that... after you take what i'm sure is a much needed vacation.

So, just to clarify, is this :

• only a problem for supporting legacy lokijs file formats?
• only for transparently supporting legacy lokijs format via same load codepath as new database format
• an issue even not considering legacy lokijs file format?

I believe in LokiJS loadDatabase we allow inflating from objects for reference adapters vs basic adapters. This (loadJSONObject) method is also being used elsewhere for cloning. If we need to rework that interface I'm sure we can come up with something or if we want a different 'import' code path which applied upconverting format that might be an option as well.

On a similar but somewhat unrelated note, I have been considering taking the iterator/generator approach used in LokiFsStructuredAdapter and bringing similar interface internal to LokiDB to allow 'streaming' serialization. That may take the form of a (third?) adapter interface if it can be done cleanly.

[Viatorus]

Hey @obeliskos,
My initial issue is fixed - thank you for your clarification. ;)

So here is the current design concept/implementation:

An user can now migrate his database from LokiJS to LokiDB and of course, we can also support migration from different LokiDB versions to the current one, in a type safe way.

This works the following way:
The file v1_5.ts contains the current type structure of LokiJS serialization.
The file v2_0.ts contains the current type structure of LokiDB serialization.

With the help of a fancy conditional type we can use the typescript compiler to show us unresolved differences (type errors). These can be solved with the mergeRightBiasedWithProxy function (see migration.ts and migration.spec.ts).

// used internal
return mergeRightBiasedWithProxy(obj,
{
    databaseVersion: 2.0 as 2.0,
    collections: obj.collections.map(coll => mergeRightBiasedWithProxy(coll, {
    dynamicViews: coll.DynamicViews.map(dv => mergeRightBiasedWithProxy(dv, {
        persistent: dv.options.persistent,
        sortPriority: dv.options.sortPriority,
        minRebuildInterval: dv.options.minRebuildInterval,
        resultSet: mergeRightBiasedWithProxy(dv.resultset, {
        filteredRows: dv.resultset.filteredrows,
        scoring: null
        }),
        sortByScoring: false,
        sortCriteriaSimple: {
        field: dv.sortCriteriaSimple.propname
        },
    })),
    cloneMethod: migrateCloneMethod(coll.cloneMethod),
    transforms: coll.transforms as any as Dict<V2_0.Transform[]>, // TODO not accurate
    nestedProperties: [],
    ttl: undefined,
    ttlInterval: undefined,
    fullTextSearch: null,
    }))
});
}

Because of existing feature differences between LokiJS and LokiDB (nestedProperties, FullTextSearch...) and possible other differences in future versions of LokiDB an user has now the possibility to add a callback function. This functions gets called for each collection just before deserialization. The function has three parameters: original database version, migrated database object and the collection options (like the options used by addCollection(...)). So the user can either alter the database object before deserialization and return with false or adjust the collection options and return with true. In the second case, a brand new collection with the defined options will be created and the collection data will be inserted using standard Collection.insert.

db.loadJSONObject(legacyDB as Serialization.Serialized, {
    migrate: (_1, coll) => {
        coll.nestedProperties = [{name: "d.msg", path: ["d", "msg"]}];
        return false;
    }
});

db.loadJSONObject(legacyDB as Serialization.Serialized, {
    migrate: (_1, _2, options) => {
        options.nestedProperties = ["d.msg"];
        return true;
    }
});

What do you think? Is it understandable for you?

---

Still todos:

• tidy up type structure of v1.5
• finalize type structure/naming of v2.0
• add better type safety for transform

[obeliskos]

Wow, interesting... that's more type safety than I expected for serialization. Might take me longer than a cursory glance to fully understand. So as we make changes to lokidb serialized structure we will need to update the v2_0.ts interfaces and possibly the migration.ts (or is that migrate function handled within loki.ts)?

[Viatorus]

If we change the lokidb serialized structure we have to copy and rename v2_0.ts to v2_1.ts (e.g) and adjust the type structure. After that, migration.ts must be extended with a migration function v2_0 to v2_1 (possibly smaller and simpler than v1_5 to v2_0).

[retorquere]

Is there example code to use this migration? I have a lokijs DB that saves/loads using a custom reference adapter.

[Viatorus]

Hey @retorquere,
there exists a migration path but only as WIP branch. Please be patient.
LokiDBs serialization is not final and should not be use in production yet.

[obeliskos]

I'll be adding a LokiCompatibilityOperatorPackage which I will try to make as close as possible to the LokiJS operator package. When migrating databases from LokiJS, we can assign that as the default operator package for the imported collections.

I'll probably also look into adding query docs and an optional parameter to find to allow overriding the operator package to use.

[Viatorus]

@obeliskos
Good point! Thank you.

What do you think about using a key/value store (object, map) instead of an array for collection data? ;)

[obeliskos]

I have been thinking that reworking data from array to map is not something that we -need- to complete before a 1.0 version. It will be a lot of internal restructuring but should be transparent to user/api. Let me know if you think differently... the performance with the loki id hashmap is good enough for now unless it proves too significant of a 'startup' delay to generate those id maps for every collection loaded.

Right now, I was thinking that (in addition to adding compatibility operator package) I would begin exploring the impact of having these operator packages on our typescript tooling (and type checking) within client programs which consume the library/classes. In order to do that I may start setting up an /examples folder and begin attempting a few quickstarts and see where tooling and compilation breaks.

We currently have Query<> and LokiOps (in Resultset) which I need to examine how to merge with operator package classes and allow tooling for those within our find() calls. We also have some inherited doQueryOp() nested logic which I first need to understand and then figure out if we need tooling for that. I think @VladimirTechMan might have had some involvement with that, if so maybe he can help me understand all the various query possibilities they enable and we can see if we can support them or possibly just terminate type checking at that level with an 'any' or something.

For now I am thinking of setting up tsconfig.json and merely importing modules from other directories. Long term it may be better to have these in external package but for now the above will let probably let me experiment which changes without publishing to npm.

Let me know if you have ideas or concerns with the above or if you can think of other issues we might want to address for 'high level usability' in javascript, typescript, or migrations.

[Viatorus]

I have been thinking that reworking data from array to map is not something that we -need- to complete before a 1.0 version. It will be a lot of internal restructuring but should be transparent to user/api. Let me know if you think differently... the performance with the loki id hashmap is good enough for now unless it proves too significant of a 'startup' delay to generate those id maps for every collection loaded.

I would like to do as many changes as possible before a final release. Changing the array to map would be a big step in serialization/deserialization which would fit nice into the migration from LokiJS to LokiDB and not into LokiDB 2.0 to LokiDB 2.1.

Maybe I start with this myself and you can check out if it is going in the right direction: array -> map.

Right now, I was thinking that (in addition to adding compatibility operator package) I would begin exploring the impact of having these operator packages on our typescript tooling (and type checking) within client programs which consume the library/classes. In order to do that I may start setting up an /examples folder and begin attempting a few quickstarts and see where tooling and compilation breaks.

We currently have Query<> and LokiOps (in Resultset) which I need to examine how to merge with operator package classes and allow tooling for those within our find() calls. We also have some inherited doQueryOp() nested logic which I first need to understand and then figure out if we need tooling for that. I think @VladimirTechMan might have had some involvement with that, if so maybe he can help me understand all the various query possibilities they enable and we can see if we can support them or possibly just terminate type checking at that level with an 'any' or something.

For now I am thinking of setting up tsconfig.json and merely importing modules from other directories. Long term it may be better to have these in external package but for now the above will let probably let me experiment which changes without publishing to npm.

I am not sure that I understand where this is going. But go ahead. Branches for experiments are without charge. ;)

possibly just terminate type checking at that level with an 'any' or something.

This is a nogo for me. When I migrated LokiJS to TypeScript static typing helped me to not mess up with not covered unit tested source code. We should really stick on typings as much as possible - any is bad. I am happy to help you with any implementation problems.

[obeliskos]

Well i think some dynamic features are supported in javascript, but when using typescript or even just the definitions for tooling it won't know what to make of some of the advanced query objects.

For example, some of the following may return typescript compiler errors :

// this works
let result = coll.find({ name: { $len: 5 } });

// this won't ( i think this is the syntax)
let result = coll.find({ name: { $len: { $gt: 5 } });

// we have a LokiOps.$and and LokiOps.$or which are only used for doQueryOp evaluation.
// the following is just a guess at the syntax
let result = coll.find({ name: { $len: { $and: [{$gt: 3}, [$lt: 8] } } });

// we also have some 'compatibility' operator package opertators which are not in Resultset.Query<>
let result = coll.find({ age: { $aeq: '21' } });
let result = coll.find({ age: { $jbetween: [21, 31] };
let result = coll.find({ dob: { $dteq: '1/1/2001' } });

I have not really understood how the doQueryOp() component of LokiOps works enough to create examples in our LokiJS Query Examples page, but we do describe many areas where you can pass value, array, or object and it is treated differently.

Since our find() query objects are usually literals that are duck-typed into :

public find(query?: ResultSet.Query<Doc<T>>, firstOnly = false): this {

I would think that we would need to accommodate the various operator package types/interfaces/etc either through unioning or possibly overloads of some sort. The fact that we let the user pass in their own operator package which does not need to implement any of the methods on ResultSet.Query and may contain unknown ops of their own naming and types complicates things but i am willing to allow query objects targeting user defined packages to be considered 'any' to get rid of type checking yet allow them.

Do you not think this affects usability more? Since i have made this even more dynamic with switchable operator packages, I felt the need to ensure those loose ends were tied up as best as possible... but if you feel that is something you would rather work on let me know and i can experiment with data[] => data{] conversion. Experimentation 'might' prove (not sure) that the array iteration (for example in unindexed queries) to be faster than object property iteration (for..in). But we could begin attempting this and see how it affects benchmarks/performance.

[Viatorus]

For example, some of the following may return typescript compiler errors :

Ah okay, thank you. Now I see what you mean.

But changing this:

public find(query?: ResultSet.Query<Doc<T>>, firstOnly = false): this

to this:

public find(query?: any, firstOnly = false): this

isn't a solution. Better conditional types would fix most compiler errors. ;)

Live demo:
https://stackblitz.com/edit/typescript-qpvec1

[retorquere]

Count me in as an enthusiastic user of typescript. Setting up conditional types is a PITA but it has prevented quite a few errors that would have been nasty to track down.

[obeliskos]

Yes that is the idea behind this branch and represents most of the effort which will be done there.

I'm sure i will run into issues with syntax but if i run out of ideas, i will see if you guys know of any solutions.