Fix ValueError in the NewRowSynthesis metric (also impacts DiagnosticReport)

[npatki]

Environment Details

• SDMetrics version: 0.9.0
• Python version: 3.8
• Operating System: Linux (Colab Notebook)

Error Description

If my data contains >32 columns, then the NewRowSynthesis metric crashes with a ValueError. This means that I can't run the DiagnosticReport.

Workaround

Until we make the fix, you can work around this issue by choosing a smaller set of columns when running this metric or a diagnostic report. As long as there are 32 or fewer columns, it seems to work.

Technical Fix

If there is an error in the pandas query method, we should try calling it again with engine='python'.

In this line of code we are only excepting a TypeError. We should also except a ValueError. Something like the following:

try:
    matches = real_data.query(' and '.join(row_filter))
except TypeError:
    if len(real_data) > 10000:
        warnings.warn('Unable to optimize query. For better formance, set the `synthetic_sample_size` parameter or upgrade to Python 3.8')
    matches = real_data.query(' and '.join(row_filter), engine='python')
except ValueError:
    matches = real_data.query(' and '.join(row_filter), engine='python')

This seems to work when I try it!