Reimagining crossfilter and dc.js interactions

[kum-deepak]

@gordonwoodhull many thanks for your continuous work on making dc.js available to the humanity. 👍

I had a look at https://github.com/att/xfilter. It pretty much achieves what I needed.

I have few ideas to make it ready for the prime time:

• Currently dc code is highly tailored for what Crossfilter provides and their API. dc does not need much, which you have already almost separated in xfilter. I think to make xfilter (or anything else) need to be brought to same playing field as crossfilter.
• We should define what dc actually needs - including things cross filter does not provide (like anything that needs a function to be passed as filter). Then we can create an adapter to bridge between crossfilter and dc that brings in these additional functionality as well.
• This will also imply that some of filter related code will shift to adapters.
• At this stage xfilter (and others) will be able to target the same specs.
• I believe it can be done in a backward compatible way.

I addition there can be some minor improvements which I can describe as we move along. At this stage wanted to check your views.

[gordonwoodhull]

Hi @kum-deepak! I am generally supportive but I think I'd need to see more specifics to really understand what you're talking about.

Maybe you could write something up that we can iterate on? It could be a wiki or a gist or something.

I think part of what you're talking about are in the filter classes. You are right that currently the specification of filters is a bit messy, but maybe adding a categorical filter class could help, so that there are no special cases in baseMixin. Or maybe not.

There is also the fact that we currently bypass those filter classes in order to use more optimized crossfilter methods, here:

dc.js/src/base-mixin.js

Lines 77 to 100 in f819d3f

	var _filterHandler = function (dimension, filters) {
	if (filters.length === 0) {
	dimension.filter(null);
	} else if (filters.length === 1 && !filters[0].isFiltered) {
	// single value and not a function-based filter
	dimension.filterExact(filters[0]);
	} else if (filters.length === 1 && filters[0].filterType === 'RangedFilter') {
	// single range-based filter
	dimension.filterRange(filters[0]);
	} else {
	dimension.filterFunction(function (d) {
	for (var i = 0; i < filters.length; i++) {
	var filter = filters[i];
	if (filter.isFiltered && filter.isFiltered(d)) {
	return true;
	} else if (filter <= d && filter >= d) {
	return true;
	}
	}
	return false;
	});
	}
	return filters;
	};

Conceivably the filter objects could "apply themselves" in order to get rid of that bypass. But I'm not sure that helps in your goal.

Also I think others have found that the most efficient queries/filters to supply to other backends were not classified the same way. This is not a major issue, but it's worth keeping in mind that the crossfilter methods are exactly what was most efficient for that very unique data structure.

Eventually I plan to transfer xfilter to the crossfilter or dc-js organization, and maybe give it a more descriptive name (that name comes from "you can turn X into crossfilter" but I don't think that comes across).

I'm also open to pulling it apart if it's not the right abstraction.

Sorry if this is a bit rambly, I haven't really thought it out, just offering some pointers here.

[gordonwoodhull]

(A related effort: we have talked about splitting off the chart registry #988. That's a complementary effort, and crossfilter is only implicit there. Same level of abstraction though.)

[kum-deepak]

Thanks for your detailed note. I will get some code going, that will give me ideas as well 😄

[kum-deepak]

I have done a basic wrapper, please see pull request #1510

[kum-deepak]

This task is completed across several PRs #1765, #1773, #1774 and #1779.