DOCSP-15253 changeStream optimization

config/redirects

@@ -1791,6 +1791,13 @@ raw: /manual/core/wildcard -> ${base}/manual/core/index-wildcard/
 
 [v4.4-*]: /${version}/genindex -> ${base}/${version}/
 
+#
+# Redirects for new 5.1 pages
+#
+
+[v3.6-v5.0]: /${version}/reference/operator/aggregation/tsIncrement/ -> ${base}/${version}/reference/operator/aggregation/
+[v3.6-v5.0]: /${version}/reference/operator/aggregation/tsSecond/ -> ${base}/${version}/reference/operator/aggregation/
+
 #
 # Redirects for new 5.0 pages
 #

source/changeStreams.txt

@@ -27,6 +27,10 @@ immediately react to them. Because change streams use the aggregation
 framework, applications can also filter for specific changes or
 transform the notifications at will.
 
+.. |change-streams| replace:: change streams
+
+.. include:: /includes/change-streams-optimization.rst
+
 Availability
 ------------
 

source/includes/change-streams-optimization.rst

@@ -0,0 +1,3 @@
+Starting in MongoDB 5.1, |change-streams| are optimized, providing more
+efficient resource utilization and faster execution of some aggregation
+pipeline stages.

source/includes/extracts-agg-operators.yaml

@@ -805,6 +805,32 @@ content: |
         - Access available per-document metadata related to the
           aggregation operation.
 
+---
+ref: agg-operators-timestamp
+content: |
+
+   Timestamp expression operators return values from a :ref:`timestamp
+   <document-bson-type-timestamp>`.
+
+   .. list-table::
+      :header-rows: 1
+      :widths: 20 80
+
+      * - Name
+        - Description
+
+      * - :expression:`$tsIncrement`
+
+        - .. include:: /includes/tsIncrement-introduction.rst
+
+          .. versionadded:: 5.1
+
+      * - :expression:`$tsSecond`
+
+        - .. include:: /includes/tsSecond-introduction.rst
+
+          .. versionadded:: 5.1
+
 ---
 ref: agg-operators-variable-project
 content: |

source/includes/stockSales-example-collection-create.rst

@@ -0,0 +1,9 @@
+.. code-block:: javascript
+
+   db.stockSales.insertMany( [
+      { _id: 0, symbol: "ABC", saleTimestamp: Timestamp(1622731060, 1) },
+      { _id: 1, symbol: "ABC", saleTimestamp: Timestamp(1622731060, 2) },
+      { _id: 2, symbol: "DEF", saleTimestamp: Timestamp(1714124193, 1) },
+      { _id: 3, symbol: "DEF", saleTimestamp: Timestamp(1714124193, 2) },
+      { _id: 4, symbol: "DEF", saleTimestamp: Timestamp(1714124193, 3) }
+   ] )

source/includes/stockSales-example-collection.rst

@@ -0,0 +1,13 @@
+Create a ``stockSales`` collection that contains company stock financial
+market sales:
+
+.. include:: /includes/stockSales-example-collection-create.rst
+
+In the :ref:`timestamp <document-bson-type-timestamp>` constructor, the:
+
+- First value is the number of seconds after the :wikipedia:`Unix epoch
+  <Unix_time>`.
+
+- Second value is the incrementing ordinal. When multiple events happen
+  within the same second, the incrementing ordinal uniquely identifies
+  each event.

source/includes/tsIncrement-introduction.rst

@@ -0,0 +1,2 @@
+Returns the incrementing ordinal from a :ref:`timestamp
+<document-bson-type-timestamp>` as a :bsontype:`long <data_numberlong>`.

source/includes/tsSecond-introduction.rst

@@ -0,0 +1,2 @@
+Returns the seconds from a :ref:`timestamp
+<document-bson-type-timestamp>` as a :bsontype:`long <data_numberlong>`.

source/reference/operator/aggregation.txt

@@ -102,6 +102,11 @@ Text Expression Operator
 
 .. include:: /includes/extracts/agg-operators-text.rst
 
+Timestamp Expression Operators
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. include:: /includes/extracts/agg-operators-timestamp.rst
+
 Trigonometry Expression Operators
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -1100,6 +1105,20 @@ Alphabetical Listing of Expression Operators
        decimal place.
 
 
+   * - :expression:`$tsIncrement`
+
+     - .. include:: /includes/tsIncrement-introduction.rst
+
+       .. versionadded:: 5.1
+
+
+   * - :expression:`$tsSecond`
+
+     - .. include:: /includes/tsSecond-introduction.rst
+
+       .. versionadded:: 5.1
+
+
    * - :expression:`$type`
 
      - Return the BSON data type of the field.
@@ -1280,6 +1299,8 @@ Alphabetical Listing of Expression Operators
    /reference/operator/aggregation/toString
    /reference/operator/aggregation/toLower
    /reference/operator/aggregation/toUpper
+   /reference/operator/aggregation/tsIncrement
+   /reference/operator/aggregation/tsSecond
    /reference/operator/aggregation/trim
    /reference/operator/aggregation/trunc
    /reference/operator/aggregation/type

source/reference/operator/aggregation/tsIncrement.txt

@@ -0,0 +1,188 @@
+==========================
+$tsIncrement (aggregation)
+==========================
+
+.. default-domain:: mongodb
+
+.. contents:: On this page
+   :local:
+   :backlinks: none
+   :depth: 1
+   :class: singlecol
+
+Definition
+----------
+
+.. expression:: $tsIncrement
+
+.. versionadded:: 5.1
+
+.. include:: /includes/tsIncrement-introduction.rst
+
+When multiple events happen within the same second, the incrementing
+ordinal uniquely identifies each event.
+
+:expression:`$tsIncrement` syntax:
+
+.. code-block:: none
+   :copyable: false
+
+   { $tsIncrement: <expression> }
+
+The :ref:`expression <aggregation-expressions>` must resolve to a
+:ref:`timestamp <document-bson-type-timestamp>`.
+
+.. seealso::
+
+   - :ref:`aggregation-expressions`
+   - :ref:`bson-types`
+   - :expression:`$tsSecond`
+
+Behavior
+--------
+
+:expression:`$tsIncrement` returns:
+
+- ``Null`` if the input :ref:`expression <aggregation-expressions>`
+  evaluates to ``null`` or refers to a field that is missing.
+
+- An error if the input :ref:`expression <aggregation-expressions>` does
+  not evaluate to a :ref:`timestamp <document-bson-type-timestamp>`.
+
+Examples
+--------
+
+Obtain the Incrementing Ordinal from a Timestamp Field
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. include:: /includes/stockSales-example-collection.rst
+
+The following example uses :expression:`$tsIncrement` in a
+:pipeline:`$project` stage to return the incrementing ordinal from the
+stock sales ``saleTimestamp`` field:
+
+.. code-block:: javascript
+
+   db.stockSales.aggregate( [
+      {
+         $project:
+         {
+            _id: 0, saleTimestamp: 1, saleIncrement: { $tsIncrement: "$saleTimestamp" }
+         }
+      }
+   ] )
+
+In the example, :pipeline:`$project` only includes the ``saleTimestamp``
+and ``saleIncrement`` fields as shown in the following output:
+
+.. code-block:: javascript
+   :copyable: false
+
+   {
+     saleTimestamp: Timestamp({ t: 1622731060, i: 1 }),
+     saleIncrement: Long("1")
+   },
+   {
+     saleTimestamp: Timestamp({ t: 1622731060, i: 2 }),
+     saleIncrement: Long("2")
+   },
+   {
+     saleTimestamp: Timestamp({ t: 1714124193, i: 1 }),
+     saleIncrement: Long("1")
+   },
+   {
+     saleTimestamp: Timestamp({ t: 1714124193, i: 2 }),
+     saleIncrement: Long("2")
+   },
+   {
+     saleTimestamp: Timestamp({ t: 1714124193, i: 3 }),
+     saleIncrement: Long("3")
+   }
+
+Use ``$tsIncrement`` in a Change Stream Cursor to Monitor Collection Changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example in this section uses :expression:`$tsIncrement` in a
+:ref:`change stream cursor <changeStreams>` to return every other change
+made to a collection in the same second of time.
+
+Create a :ref:`change stream cursor <changeStreams>` on a collection
+named ``cakeSales`` that you will see later in this section:
+
+.. code-block:: javascript
+
+   cakeSalesCursor = db.cakeSales.watch( [
+      {
+         $match: {
+            $expr: {
+               $eq: [
+                  { $mod: [ { $tsIncrement: "$clusterTime" } , 2 ] },
+                  0
+               ]
+            }
+         }
+      }
+   ] )
+
+In the example, the:
+
+- :method:`db.collection.watch()` method creates a :ref:`change stream
+  cursor <changeStreams>` for the ``cakeSales`` collection and stores
+  the cursor in ``cakeSalesCursor``.
+
+- :pipeline:`$match` stage filters the documents to those
+  returned by the :query:`$expr` operator.
+
+- :query:`$expr` operator:
+
+  - Applies :expression:`$mod` ``2`` to the ``$clusterTime`` variable's
+    incrementing ordinal returned by :expression:`$tsIncrement`.
+
+    ``$clusterTime`` is the timestamp from the :ref:`oplog
+    <replica-set-oplog>` entry when the ``cakeSales`` collection is
+    modified. See :ref:`Command Response <command-response>`.
+
+  - Compares the returned value from :expression:`$mod` to ``0`` using
+    :expression:`$eq`.
+
+.. include:: /includes/cakeSales-example-collection.rst
+
+To monitor the ``cakeSales`` collection changes, use
+``cakeSalesCursor``. For example, to obtain the next document from
+``cakeSalesCursor``, use the :method:`~cursor.next()` method:
+
+.. code-block:: javascript
+
+   cakeSalesCursor.next()
+
+Depending on the second when the documents were added to ``cakeSales``,
+the output from ``cakeSalesCursor.next()`` varies. For example, the
+document additions might span more than one second.
+
+The following ``cakeSalesCursor.next()`` example output shows the
+``insert`` details for the first document added to the ``cakeSales``
+collection. Notice the incrementing ordinal ``i`` is ``2`` in the
+``clusterTime`` field.
+
+.. code-block:: javascript
+   :copyable: false
+
+   _id: {
+     _data: '82613A4F25000000022B022C0100296E5A100454C5BFAF538C47AB950614F43889BE00461E5F696400290004'
+   },
+   operationType: 'insert',
+   clusterTime: Timestamp({ t: 1631211301, i: 2 }),
+   fullDocument: {
+     _id: 0,
+     type: 'chocolate',
+     orderDate: ISODate("2020-05-18T14:10:30.000Z"),
+     state: 'CA',
+     price: 13,
+     quantity: 120
+   },
+   ns: { db: 'test', coll: 'cakeSales' },
+   documentKey: { _id: 0 }
+
+Running ``cakeSalesCursor.next()`` again returns the ``cakeSales``
+document for which the ``clusterTime`` incrementing ordinal ``i`` is
+``4``, omitting the document where ``i`` is ``3``.