Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apply bloomFilter while handling existence filter mismatch #6897

Conversation

milaGGL
Copy link
Contributor

@milaGGL milaGGL commented Dec 17, 2022

  • Use bloom filter to filter out the removed documents when there is existence filter mismatch
  • add spec tests for good and bad cases for bloomFilter
    -- When bloomFilter yields expected result, skip full re-query
    -- When bloomFilter contains false positive result, trigger re-query
  • move newTextEncoder and newTextDecoder out of platform/serializer file into platform/text_serializer file
    Screenshot 2022-12-16 at 9 41 34 PM
    Circular dependency: BloomFilter uses newTextEncoder from /platform/serializer -> /remote/watch_change uses BloomFilter -> /platform/serializer uses JsonProtoSerializer from /remote/serializer

Do we need to use platform based newTextEncoder in checking the membership of a key in BloomFilter? if a key is consist of non-special characters, like 'ÀÒ∑', md5 hash will return different results with and without utf-8 encoding.

@changeset-bot
Copy link

changeset-bot bot commented Dec 17, 2022

⚠️ No Changeset found

Latest commit: 8b0c309

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Dec 19, 2022

Size Report 1

Affected Products

  • @firebase/firestore

    TypeBase (5335e7a)Merge (1fddbc2)Diff
    browser273 kB275 kB+2.27 kB (+0.8%)
    esm5340 kB343 kB+2.55 kB (+0.8%)
    main546 kB551 kB+4.30 kB (+0.8%)
    module273 kB275 kB+2.27 kB (+0.8%)
    react-native273 kB276 kB+2.23 kB (+0.8%)
  • @firebase/firestore-lite

    TypeBase (5335e7a)Merge (1fddbc2)Diff
    browser86.2 kB86.2 kB+44 B (+0.1%)
    esm5104 kB104 kB+66 B (+0.1%)
    main147 kB147 kB+44 B (+0.0%)
    module86.2 kB86.2 kB+44 B (+0.1%)
    react-native86.4 kB86.4 kB+44 B (+0.1%)
  • bundle

    12 size changes

    TypeBase (5335e7a)Merge (1fddbc2)Diff
    firestore (Persistence)292 kB294 kB+2.22 kB (+0.8%)
    firestore (Query Cursors)231 kB233 kB+2.22 kB (+1.0%)
    firestore (Query)229 kB231 kB+2.22 kB (+1.0%)
    firestore (Read data once)216 kB218 kB+2.22 kB (+1.0%)
    firestore (Realtime updates)218 kB220 kB+2.22 kB (+1.0%)
    firestore (Transaction)198 kB199 kB+88 B (+0.0%)
    firestore (Write data)198 kB198 kB+88 B (+0.0%)
    firestore-lite (Query Cursors)81.3 kB81.3 kB+44 B (+0.1%)
    firestore-lite (Query)77.4 kB77.5 kB+44 B (+0.1%)
    firestore-lite (Read data once)59.4 kB59.4 kB+44 B (+0.1%)
    firestore-lite (Transaction)84.0 kB84.0 kB+44 B (+0.1%)
    firestore-lite (Write data)68.9 kB68.9 kB+44 B (+0.1%)

  • firebase

    TypeBase (5335e7a)Merge (1fddbc2)Diff
    firebase-compat.js757 kB759 kB+2.16 kB (+0.3%)
    firebase-firestore-compat.js331 kB334 kB+2.16 kB (+0.7%)
    firebase-firestore-lite.js92.9 kB92.9 kB+44 B (+0.0%)
    firebase-firestore.js332 kB335 kB+2.19 kB (+0.7%)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/OsUGD24V9d.html

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Dec 19, 2022

Size Analysis Report 1

This report is too large (991,528 characters) to be displayed here in a GitHub comment. Please use the below link to see the full report on Google Cloud Storage.

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/VVIKMaSNKZ.html

@milaGGL milaGGL requested a review from dconeybe December 19, 2022 20:50
Copy link
Contributor

@dconeybe dconeybe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on this! I'm especially impressed with the quality of the spec test cases that you wrote.

Copy link
Contributor

@dconeybe dconeybe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh also, please add a spec test case where the bloom filter contains documents with special characters in the document name.

@milaGGL milaGGL requested a review from dconeybe January 13, 2023 20:43
@milaGGL milaGGL requested a review from dconeybe January 18, 2023 22:34
@milaGGL milaGGL self-assigned this Jan 19, 2023
Copy link
Contributor

@dconeybe dconeybe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge it!

@milaGGL milaGGL merged commit 333aac9 into mila/BloomFilter Jan 27, 2023
@milaGGL milaGGL deleted the mila/BloomFilter-apply-bloom-filter-in-existence-filter-mismatch branch January 27, 2023 16:55
@firebase firebase locked and limited conversation to collaborators Feb 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants