fs: introduce opendir() and fs.Dir

This adds long-requested methods for asynchronously interacting and
iterating through directory entries by using `uv_fs_opendir`,
`uv_fs_readdir`, and `uv_fs_closedir`.

`fs.opendir()` and friends return an `fs.Dir`, which contains methods
for doing reads and cleanup. `fs.Dir` also has the async iterator
symbol exposed.

The `read()` method and friends only return `fs.Dirent`s for this API.
Having a entry type or doing a `stat` call is deemed to be necessary in
the majority of cases, so just returning dirents seems like the logical
choice for a new api.

Reading when there are no more entries returns `null` instead of a
dirent. However the async iterator hides that (and does automatic
cleanup).

The code lives in separate files from the rest of fs, this is done
partially to prevent over-pollution of those (already very large)
files, but also in the case of js allows loading into `fsPromises`.

Due to async_hooks, this introduces a new handle type of `DIRHANDLE`.

This PR does not attempt to make complete optimization of
this feature. Notable future improvements include:
- Moving promise work into C++ land like FileHandle.
- Possibly adding `readv()` to do multi-entry directory reads.
- Aliasing `fs.readdir` to `fs.scandir` and doing a deprecation.

Refs: nodejs/node-v0.x-archive#388
Refs: #583
Refs: libuv/libuv#2057

PR-URL: #29349
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: David Carlier <[email protected]>

Author: Fishrock123

doc/api/errors.md

@@ -824,6 +824,11 @@ A signing `key` was not provided to the [`sign.sign()`][] method.
 [`crypto.timingSafeEqual()`][] was called with `Buffer`, `TypedArray`, or
 `DataView` arguments of different lengths.
 
+<a id="ERR_DIR_CLOSED"></a>
+### ERR_DIR_CLOSED
+
+The [`fs.Dir`][] was previously closed.
+
 <a id="ERR_DNS_SET_SERVERS_FAILED"></a>
 ### ERR_DNS_SET_SERVERS_FAILED
 
@@ -2380,6 +2385,7 @@ such as `process.stdout.on('data')`.
 [`dgram.disconnect()`]: dgram.html#dgram_socket_disconnect
 [`dgram.remoteAddress()`]: dgram.html#dgram_socket_remoteaddress
 [`errno`(3) man page]: http://man7.org/linux/man-pages/man3/errno.3.html
+[`fs.Dir`]: fs.html#fs_class_fs_dir
 [`fs.readFileSync`]: fs.html#fs_fs_readfilesync_path_options
 [`fs.readdir`]: fs.html#fs_fs_readdir_path_options_callback
 [`fs.symlink()`]: fs.html#fs_fs_symlink_target_path_type_callback

doc/api/fs.md

@@ -284,13 +284,148 @@ synchronous use libuv's threadpool, which can have surprising and negative
 performance implications for some applications. See the
 [`UV_THREADPOOL_SIZE`][] documentation for more information.
 
+## Class fs.Dir
+<!-- YAML
+added: REPLACEME
+-->
+
+A class representing a directory stream.
+
+Created by [`fs.opendir()`][], [`fs.opendirSync()`][], or [`fsPromises.opendir()`][].
+
+Example using async interation:
+
+```js
+const fs = require('fs');
+
+async function print(path) {
+  const dir = await fs.promises.opendir(path);
+  for await (const dirent of dir) {
+    console.log(dirent.name);
+  }
+}
+print('./').catch(console.error);
+```
+
+### dir.path
+<!-- YAML
+added: REPLACEME
+-->
+
+* {string}
+
+The read-only path of this directory as was provided to [`fs.opendir()`][],
+[`fs.opendirSync()`][], or [`fsPromises.opendir()`][].
+
+### dir.close()
+<!-- YAML
+added: REPLACEME
+-->
+
+* Returns: {Promise}
+
+Asynchronously close the directory's underlying resource handle.
+Subsequent reads will result in errors.
+
+A `Promise` is returned that will be resolved after the resource has been
+closed.
+
+### dir.close(callback)
+<!-- YAML
+added: REPLACEME
+-->
+
+* `callback` {Function}
+  * `err` {Error}
+
+Asynchronously close the directory's underlying resource handle.
+Subsequent reads will result in errors.
+
+The `callback` will be called after the resource handle has been closed.
+
+### dir.closeSync()
+<!-- YAML
+added: REPLACEME
+-->
+
+Synchronously close the directory's underlying resource handle.
+Subsequent reads will result in errors.
+
+### dir.read([options])
+<!-- YAML
+added: REPLACEME
+-->
+
+* `options` {Object}
+  * `encoding` {string|null} **Default:** `'utf8'`
+* Returns: {Promise} containing {fs.Dirent}
+
+Asynchronously read the next directory entry via readdir(3) as an
+[`fs.Dirent`][].
+
+A `Promise` is returned that will be resolved with a [Dirent][] after the read
+is completed.
+
+_Directory entries returned by this function are in no particular order as
+provided by the operating system's underlying directory mechanisms._
+
+### dir.read([options, ]callback)
+<!-- YAML
+added: REPLACEME
+-->
+
+* `options` {Object}
+  * `encoding` {string|null} **Default:** `'utf8'`
+* `callback` {Function}
+  * `err` {Error}
+  * `dirent` {fs.Dirent}
+
+Asynchronously read the next directory entry via readdir(3) as an
+[`fs.Dirent`][].
+
+The `callback` will be called with a [Dirent][] after the read is completed.
+
+The `encoding` option sets the encoding of the `name` in the `dirent`.
+
+_Directory entries returned by this function are in no particular order as
+provided by the operating system's underlying directory mechanisms._
+
+### dir.readSync([options])
+<!-- YAML
+added: REPLACEME
+-->
+
+* `options` {Object}
+  * `encoding` {string|null} **Default:** `'utf8'`
+* Returns: {fs.Dirent}
+
+Synchronously read the next directory entry via readdir(3) as an
+[`fs.Dirent`][].
+
+The `encoding` option sets the encoding of the `name` in the `dirent`.
+
+_Directory entries returned by this function are in no particular order as
+provided by the operating system's underlying directory mechanisms._
+
+### dir\[Symbol.asyncIterator\]()
+<!-- YAML
+added: REPLACEME
+-->
+
+* Returns: {AsyncIterator} to fully iterate over all entries in the directory.
+
+_Directory entries returned by this iterator are in no particular order as
+provided by the operating system's underlying directory mechanisms._
+
 ## Class: fs.Dirent
 <!-- YAML
 added: v10.10.0
 -->
 
-When [`fs.readdir()`][] or [`fs.readdirSync()`][] is called with the
-`withFileTypes` option set to `true`, the resulting array is filled with
+A representation of a directory entry, as returned by reading from an [`fs.Dir`][].
+
+Additionally, when [`fs.readdir()`][] or [`fs.readdirSync()`][] is called with
+the `withFileTypes` option set to `true`, the resulting array is filled with
 `fs.Dirent` objects, rather than strings or `Buffers`.
 
 ### dirent.isBlockDevice()
@@ -2505,6 +2640,46 @@ Returns an integer representing the file descriptor.
 For detailed information, see the documentation of the asynchronous version of
 this API: [`fs.open()`][].
 
+## fs.opendir(path[, options], callback)
+<!-- YAML
+added: REPLACEME
+-->
+
+* `path` {string|Buffer|URL}
+* `options` {Object}
+  * `encoding` {string|null} **Default:** `'utf8'`
+* `callback` {Function}
+  * `err` {Error}
+  * `dir` {fs.Dir}
+
+Asynchronously open a directory. See opendir(3).
+
+Creates an [`fs.Dir`][], which contains all further functions for reading from
+and cleaning up the directory.
+
+The `encoding` option sets the encoding for the `path` while opening the
+directory and subsequent read operations (unless otherwise overriden during
+reads from the directory).
+
+## fs.opendirSync(path[, options])
+<!-- YAML
+added: REPLACEME
+-->
+
+* `path` {string|Buffer|URL}
+* `options` {Object}
+  * `encoding` {string|null} **Default:** `'utf8'`
+* Returns: {fs.Dir}
+
+Synchronously open a directory. See opendir(3).
+
+Creates an [`fs.Dir`][], which contains all further functions for reading from
+and cleaning up the directory.
+
+The `encoding` option sets the encoding for the `path` while opening the
+directory and subsequent read operations (unless otherwise overriden during
+reads from the directory).
+
 ## fs.read(fd, buffer, offset, length, position, callback)
 <!-- YAML
 added: v0.0.2
@@ -4644,6 +4819,39 @@ by [Naming Files, Paths, and Namespaces][]. Under NTFS, if the filename contains
 a colon, Node.js will open a file system stream, as described by
 [this MSDN page][MSDN-Using-Streams].
 
+## fsPromises.opendir(path[, options])
+<!-- YAML
+added: REPLACEME
+-->
+
+* `path` {string|Buffer|URL}
+* `options` {Object}
+  * `encoding` {string|null} **Default:** `'utf8'`
+* Returns: {Promise} containing {fs.Dir}
+
+Asynchronously open a directory. See opendir(3).
+
+Creates an [`fs.Dir`][], which contains all further functions for reading from
+and cleaning up the directory.
+
+The `encoding` option sets the encoding for the `path` while opening the
+directory and subsequent read operations (unless otherwise overriden during
+reads from the directory).
+
+Example using async interation:
+
+```js
+const fs = require('fs');
+
+async function print(path) {
+  const dir = await fs.promises.opendir(path);
+  for await (const dirent of dir) {
+    console.log(dirent.name);
+  }
+}
+print('./').catch(console.error);
+```
+
 ### fsPromises.readdir(path[, options])
 <!-- YAML
 added: v10.0.0
@@ -5253,6 +5461,7 @@ the file contents.
 [`UV_THREADPOOL_SIZE`]: cli.html#cli_uv_threadpool_size_size
 [`WriteStream`]: #fs_class_fs_writestream
 [`event ports`]: https://illumos.org/man/port_create
+[`fs.Dir`]: #fs_class_fs_dir
 [`fs.Dirent`]: #fs_class_fs_dirent
 [`fs.FSWatcher`]: #fs_class_fs_fswatcher
 [`fs.Stats`]: #fs_class_fs_stats
@@ -5269,6 +5478,8 @@ the file contents.
 [`fs.mkdir()`]: #fs_fs_mkdir_path_options_callback
 [`fs.mkdtemp()`]: #fs_fs_mkdtemp_prefix_options_callback
 [`fs.open()`]: #fs_fs_open_path_flags_mode_callback
+[`fs.opendir()`]: #fs_fs_opendir_path_options_callback
+[`fs.opendirSync()`]: #fs_fs_opendirsync_path_options
 [`fs.read()`]: #fs_fs_read_fd_buffer_offset_length_position_callback
 [`fs.readFile()`]: #fs_fs_readfile_path_options_callback
 [`fs.readFileSync()`]: #fs_fs_readfilesync_path_options
@@ -5284,6 +5495,7 @@ the file contents.
 [`fs.write(fd, string...)`]: #fs_fs_write_fd_string_position_encoding_callback
 [`fs.writeFile()`]: #fs_fs_writefile_file_data_options_callback
 [`fs.writev()`]: #fs_fs_writev_fd_buffers_position_callback
+[`fsPromises.opendir()`]: #fs_fspromises_opendir_path_options
 [`inotify(7)`]: http://man7.org/linux/man-pages/man7/inotify.7.html
 [`kqueue(2)`]: https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2
 [`net.Socket`]: net.html#net_class_net_socket

lib/fs.js

@@ -64,6 +64,7 @@ const {
   getDirents,
   getOptions,
   getValidatedPath,
+  handleErrorFromBinding,
   nullCheck,
   preprocessSymlinkDestination,
   Stats,
@@ -79,6 +80,11 @@ const {
   validateRmdirOptions,
   warnOnNonPortableTemplate
 } = require('internal/fs/utils');
+const {
+  Dir,
+  opendir,
+  opendirSync
+} = require('internal/fs/dir');
 const {
   CHAR_FORWARD_SLASH,
   CHAR_BACKWARD_SLASH,
@@ -122,23 +128,6 @@ function showTruncateDeprecation() {
   }
 }
 
-function handleErrorFromBinding(ctx) {
-  if (ctx.errno !== undefined) {  // libuv error numbers
-    const err = uvException(ctx);
-    // eslint-disable-next-line no-restricted-syntax
-    Error.captureStackTrace(err, handleErrorFromBinding);
-    throw err;
-  }
-  if (ctx.error !== undefined) {  // Errors created in C++ land.
-    // TODO(joyeecheung): currently, ctx.error are encoding errors
-    // usually caused by memory problems. We need to figure out proper error
-    // code(s) for this.
-    // eslint-disable-next-line no-restricted-syntax
-    Error.captureStackTrace(ctx.error, handleErrorFromBinding);
-    throw ctx.error;
-  }
-}
-
 function maybeCallback(cb) {
   if (typeof cb === 'function')
     return cb;
@@ -1818,7 +1807,6 @@ function createWriteStream(path, options) {
   return new WriteStream(path, options);
 }
 
-
 module.exports = fs = {
   appendFile,
   appendFileSync,
@@ -1864,6 +1852,8 @@ module.exports = fs = {
   mkdtempSync,
   open,
   openSync,
+  opendir,
+  opendirSync,
   readdir,
   readdirSync,
   read,
@@ -1897,6 +1887,7 @@ module.exports = fs = {
   writeSync,
   writev,
   writevSync,
+  Dir,
   Dirent,
   Stats,
 

lib/internal/errors.js

@@ -759,6 +759,7 @@ E('ERR_CRYPTO_SCRYPT_NOT_SUPPORTED', 'Scrypt algorithm not supported', Error);
 E('ERR_CRYPTO_SIGN_KEY_REQUIRED', 'No key provided to sign', Error);
 E('ERR_CRYPTO_TIMING_SAFE_EQUAL_LENGTH',
   'Input buffers must have the same byte length', RangeError);
+E('ERR_DIR_CLOSED', 'Directory handle was closed', Error);
 E('ERR_DNS_SET_SERVERS_FAILED', 'c-ares failed to set servers: "%s" [%s]',
   Error);
 E('ERR_DOMAIN_CALLBACK_NOT_AVAILABLE',