Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: syntax-tree/nlcst-emoji-modifier
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 5.0.0
Choose a base ref
...
head repository: syntax-tree/nlcst-emoji-modifier
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref

Commits on Jun 1, 2021

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    95e046f View commit details

Commits on Jul 22, 2021

  1. Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    6d3e410 View commit details
  2. Refactor code-style

    wooorm committed Jul 22, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    d10e971 View commit details
  3. Add strict to tsconfig.json

    wooorm committed Jul 22, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    fecfe6b View commit details
  4. Update xo

    wooorm committed Jul 22, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    5707fe2 View commit details

Commits on Jul 30, 2021

  1. Update unist-util-visit

    wooorm committed Jul 30, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    ea275c1 View commit details

Commits on Aug 27, 2021

  1. Update dev-dependencies

    wooorm committed Aug 27, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    7cc9254 View commit details
  2. Use @types/nlcst

    wooorm committed Aug 27, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    f795327 View commit details
  3. 5.1.0

    wooorm committed Aug 27, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    238bc13 View commit details

Commits on May 28, 2022

  1. Update dev-dependencies

    wooorm committed May 28, 2022

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    474eda9 View commit details
  2. Refactor code-style

    wooorm committed May 28, 2022

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    19ea0b6 View commit details
  3. Update emoji-regex

    wooorm committed May 28, 2022

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    6333253 View commit details
  4. Add improved docs

    wooorm committed May 28, 2022

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    51c4db5 View commit details

Commits on Dec 27, 2022

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    4bd146d View commit details

Commits on Jan 19, 2023

  1. Update dev-dependencies

    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    e3c4839 View commit details
  2. Update Actions

    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    e596887 View commit details
  3. Update tsconfig.json

    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    e8a6bf8 View commit details
  4. Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    4df73a0 View commit details
  5. Refactor code-style

    *   Add more docs to JSDoc
    *   Add support for `null` in input of API types
    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    5fcf6aa View commit details
  6. Add ignore-scripts to .npmrc

    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    0940eea View commit details
  7. Use Node test runner

    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    c17fd91 View commit details
  8. Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    b7de51c View commit details
  9. Update gemoji

    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    5ef7b58 View commit details
  10. Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    f9690e4 View commit details
  11. Add improved docs

    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    8f52585 View commit details
  12. 5.2.0

    wooorm committed Jan 19, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    024e98a View commit details

Commits on Jul 18, 2023

  1. Update dev-dependencies

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    b364a1f View commit details
  2. Update @types/nlcst, utilities

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    20293ba View commit details
  3. Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    fb6a40c View commit details
  4. Refactor .npmrc

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    255fa27 View commit details
  5. Refactor code-style

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    b9e3e1f View commit details
  6. Remove complex-types.d.ts

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    f4cb55b View commit details
  7. Remove Emoticon type

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    9b59762 View commit details
  8. Change to yield undefined

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    94501fa View commit details
  9. Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    e4209fd View commit details
  10. Refactor docs

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    5d43249 View commit details
  11. Change to use exports

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    db68896 View commit details
  12. Change to require Node.js 16

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    3ae81da View commit details
  13. 6.0.0

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    93eda07 View commit details
  14. Add missing type dependency

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    a342b6f View commit details
  15. 6.0.1

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    901115d View commit details
  16. Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    22f51a5 View commit details
  17. 6.0.2

    wooorm committed Jul 18, 2023

    Verified

    This commit was signed with the committer’s verified signature.
    wooorm Titus
    Copy the full SHA
    6fc2c8f View commit details
Showing with 614 additions and 464 deletions.
  1. +1 −1 .github/workflows/bb.yml
  2. +4 −4 .github/workflows/main.yml
  3. +2 −2 .gitignore
  4. +1 −0 .npmrc
  5. +2 −257 index.js
  6. +294 −0 lib/index.js
  7. +45 −38 package.json
  8. +95 −23 readme.md
  9. +1 −1 test/fixtures/base-is-not-emoji-with-invalid-variant.json
  10. +1 −1 test/fixtures/variance-selector-no-closer.json
  11. +159 −128 test/index.js
  12. +9 −9 tsconfig.json
2 changes: 1 addition & 1 deletion .github/workflows/bb.yml
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@ name: bb
on:
issues:
types: [opened, reopened, edited, closed, labeled, unlabeled]
pull_request:
pull_request_target:
types: [opened, reopened, edited, closed, labeled, unlabeled]
jobs:
main:
8 changes: 4 additions & 4 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -7,15 +7,15 @@ jobs:
name: ${{matrix.node}}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: dcodeIO/setup-node-nvm@master
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: ${{matrix.node}}
- run: npm install
- run: npm test
- uses: codecov/codecov-action@v1
- uses: codecov/codecov-action@v3
strategy:
matrix:
node:
- lts/erbium
- lts/gallium
- node
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
coverage/
node_modules/
.DS_Store
*.d.ts
*.log
coverage/
node_modules/
yarn.lock
1 change: 1 addition & 0 deletions .npmrc
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
ignore-scripts=true
package-lock=false
259 changes: 2 additions & 257 deletions index.js
Original file line number Diff line number Diff line change
@@ -1,257 +1,2 @@
/**
* @typedef {import('unist').Node} Node
* @typedef {import('unist').Parent} Parent
* @typedef {import('unist').Point} Point
*
* @typedef {Object} FindMatch
* @property {number} start
* @property {number} end
*
* @typedef {Object} ChangeResult
* @property {Array.<Node>} nodes
* @property {number} end
*/

import {visit} from 'unist-util-visit'
import {position, pointStart, pointEnd} from 'unist-util-position'
import {generated} from 'unist-util-generated'
import {toString} from 'nlcst-to-string'
import {nameToEmoji} from 'gemoji'
import emojiRegex from 'emoji-regex'

var own = {}.hasOwnProperty
var push = [].push

/**
* Merge emoji (👍) and Gemoji (GitHub emoji, :+1:).
*
* @param {Parent} node
*/
export function emojiModifier(node) {
if (!node || !node.children) {
throw new Error('Missing children in `parent`')
}

node.children = changeParent(node, findEmoji(node), 0).nodes

visit(node, 'EmoticonNode', removeMatch)

return node
}

/**
* @param {Parent} node
* @param {Array.<FindMatch>} matches
* @param {number} start
*/
function changeParent(node, matches, start) {
var children = node.children
var end = start
var index = -1
/** @type {Array.<Node>} */
var nodes = []
/** @type {Array.<Node>} */
var merged = []
/** @type {ChangeResult} */
var result
/** @type {Node} */
var child
/** @type {Node} */
var previous

while (++index < children.length) {
result = children[index].children
? // @ts-ignore Looks like a parent.
changeParent(children[index], matches, end)
: changeLeaf(children[index], matches, end)
push.apply(nodes, result.nodes)
end = result.end
}

index = -1

while (++index < nodes.length) {
if (nodes[index].type === 'EmoticonNode') {
if (previous && previous._match === nodes[index]._match) {
// @ts-ignore Both literals.
previous.value += nodes[index].value

if (!generated(previous)) {
previous.position.end = pointEnd(nodes[index])
}
} else {
previous = nodes[index]
merged.push(nodes[index])
}
} else {
previous = null

if (node.type === 'WordNode') {
child = {type: node.type, children: [nodes[index]]}

if (!generated(nodes[index])) {
child.position = position(nodes[index])
}

merged.push(child)
} else {
merged.push(nodes[index])
}
}
}

return {end, nodes: merged}
}

/**
* @param {Node} node
* @param {Array.<FindMatch>} matches
* @param {number} start
* @returns {ChangeResult}
*/
function changeLeaf(node, matches, start) {
var value = toString(node)
var point = generated(node) ? null : pointStart(node)
var end = start + value.length
var index = -1
var textEnd = 0
/** @type {Array.<Node>} */
var nodes = []
/** @type {number} */
var emojiEnd
/** @type {Node} */
var child
/** @type {FindMatch} */
var match

while (++index < matches.length) {
match = matches[index]
emojiEnd = match.end - start + 1

if (match.start - start < value.length && emojiEnd > 0) {
if (match.start - start > textEnd) {
child = {
type: node.type,
value: value.slice(textEnd, match.start - start)
}

if (point) {
child.position = {
start: shift(point, textEnd),
end: shift(point, match.start - start)
}
}

nodes.push(child)
textEnd = match.start - start
}

if (emojiEnd > value.length) {
emojiEnd = value.length
}

child = {
type: 'EmoticonNode',
value: value.slice(textEnd, emojiEnd),
_match: match
}

if (point) {
child.position = {
start: shift(point, textEnd),
end: shift(point, emojiEnd)
}
}

nodes.push(child)
textEnd = emojiEnd
}
}

if (textEnd < value.length) {
child = {type: node.type, value: value.slice(textEnd)}

if (point) {
child.position = {start: shift(point, textEnd), end: node.position.end}
}

nodes.push(child)
}

return {end, nodes}
}

/**
* @param {Node} node
*/
function findEmoji(node) {
var emojiExpression = emojiRegex()
/** @type {Array.<FindMatch>} */
var matches = []
var value = toString(node)
var start = value.indexOf(':')
var end = start === -1 ? -1 : value.indexOf(':', start + 1)
/** @type {string} */
var slice
/** @type {RegExpExecArray} */
var match

// Get Gemoji shortcodes.
while (end !== -1) {
slice = value.slice(start + 1, end)

if (own.call(nameToEmoji, slice)) {
matches.push({start, end})
start = value.indexOf(':', end + 1)
} else {
start = end
}

end = start === -1 ? -1 : value.indexOf(':', start + 1)
}

// Get emoji.
while ((match = emojiExpression.exec(value))) {
start = match.index
end = start + match[0].length - 1

if (value.charCodeAt(end + 1) === 0xfe0f) {
end++
}

matches.push({start, end})
}

matches.sort(sort)

return matches
}

/**
* @param {Node} node
*/
function removeMatch(node) {
delete node._match
}

/**
* @param {FindMatch} a
* @param {FindMatch} b
* @returns {number}
*/
function sort(a, b) {
return a.start - b.start
}

/**
* @param {Point} point
* @param {number} offset
* @returns {Point}
*/
function shift(point, offset) {
return {
line: point.line,
column: point.column + offset,
offset: point.offset + offset
}
}
// Note: extra types exposed from `index.d.ts`.
export {emojiModifier} from './lib/index.js'
294 changes: 294 additions & 0 deletions lib/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,294 @@
/**
* @typedef {import('nlcst').Parents} Parents
* @typedef {import('nlcst').Sentence} Sentence
* @typedef {import('nlcst-emoticon-modifier').Emoticon} Emoticon
* @typedef {import('unist').Point} Point
*/

/**
* @typedef {import('unist-util-visit-parents').InclusiveDescendant<Sentence>} SentenceInclusiveDescendant
* @typedef {Extract<SentenceInclusiveDescendant, Parents>} SentenceInclusiveDescendantParents
* @typedef {Exclude<SentenceInclusiveDescendant, Parents>} SentenceInclusiveDescendantLeafs
*
* @typedef FindMatch
* @property {number} start
* @property {number} end
*
* @typedef ChangeResult
* @property {number} end
* @property {Array<SentenceInclusiveDescendant>} nodes
*/

import emojiRegex from 'emoji-regex'
import {nameToEmoji} from 'gemoji'
import {toString} from 'nlcst-to-string'
import {generated} from 'unist-util-generated'
import {pointEnd, pointStart, position} from 'unist-util-position'

const own = {}.hasOwnProperty

/**
* Classify emoji (👍) and Gemoji (GitHub emoji, :+1:) in `node` as
* `Emoticon`s.
*
* @param {Sentence} node
* Sentence to transform.
* @returns {undefined}
* Nothing.
*/
export function emojiModifier(node) {
if (!node || !node.children) {
throw new Error('Missing children in `parent`')
}

/** @type {Map<Emoticon, FindMatch>} */
const matchMap = new Map()

// @ts-expect-error: assume content model matches (no sentences in sentences).
node.children = changeParent(node, findEmoji(node), 0).nodes

/**
* Add emoji in a parent.
*
* @param {SentenceInclusiveDescendantParents} node
* Node to change.
* @param {Array<FindMatch>} matches
* Emoji in `node`.
* @param {number} start
* Where `node` starts.
* @returns {ChangeResult}
* Result.
*/
function changeParent(node, matches, start) {
let end = start
let index = -1
/** @type {Array<SentenceInclusiveDescendant>} */
const nodes = []
/** @type {Array<SentenceInclusiveDescendant>} */
const merged = []
/** @type {SentenceInclusiveDescendant | undefined} */
let previous

while (++index < node.children.length) {
const child = node.children[index]
const result =
'children' in child
? changeParent(child, matches, end)
: changeLeaf(child, matches, end)
nodes.push(...result.nodes)
end = result.end
}

index = -1

while (++index < nodes.length) {
const child = nodes[index]

if (child.type === 'EmoticonNode') {
// Always exists if we run, but other tools like
// `nlcst-emoticon-modifier` may not have it.
const match = matchMap.get(child)

if (
match &&
previous &&
previous.type === 'EmoticonNode' &&
matchMap.get(previous) === match
) {
previous.value += child.value
const end = pointEnd(child)

if (previous.position && end) {
previous.position.end = end
}
} else {
previous = child
merged.push(child)
}
} else {
previous = undefined

if (node.type === 'WordNode') {
/** @type {typeof node} */
// @ts-expect-error: assume content model matches (no words in words).
const replacement = {type: node.type, children: [child]}

if (!generated(child)) {
replacement.position = position(child)
}

merged.push(replacement)
} else {
merged.push(child)
}
}
}

return {end, nodes: merged}
}

/**
* Add emoji in a non-parent.
*
* @param {SentenceInclusiveDescendantLeafs} node
* Node to change.
* @param {Array<FindMatch>} matches
* Emoji in `node`.
* @param {number} start
* Where `node` starts.
* @returns {ChangeResult}
* Result.
*/
function changeLeaf(node, matches, start) {
const value = toString(node)
const point = generated(node) ? undefined : pointStart(node)
/** @type {Array<SentenceInclusiveDescendantLeafs>} */
const nodes = []
const end = start + value.length
let index = -1
let textEnd = 0

while (++index < matches.length) {
const match = matches[index]
let emojiEnd = match.end - start + 1

if (match.start - start < value.length && emojiEnd > 0) {
if (match.start - start > textEnd) {
/** @type {typeof node} */
const child = {
type: node.type,
value: value.slice(textEnd, match.start - start)
}

if (point) {
child.position = {
start: shift(point, textEnd),
end: shift(point, match.start - start)
}
}

nodes.push(child)
textEnd = match.start - start
}

if (emojiEnd > value.length) {
emojiEnd = value.length
}

/** @type {Emoticon} */
const child = {
type: 'EmoticonNode',
value: value.slice(textEnd, emojiEnd)
}

matchMap.set(child, match)

if (point) {
child.position = {
start: shift(point, textEnd),
end: shift(point, emojiEnd)
}
}

nodes.push(child)
textEnd = emojiEnd
}
}

if (textEnd < value.length) {
/** @type {typeof node} */
const child = {type: node.type, value: value.slice(textEnd)}

if (point && node.position) {
child.position = {start: shift(point, textEnd), end: node.position.end}
}

nodes.push(child)
}

return {end, nodes}
}
}

/**
* @param {Sentence} node
* Find emoji in a sentence.
* @returns {Array<FindMatch>}
* Matches.
*/
function findEmoji(node) {
const emojiExpression = emojiRegex()
/** @type {Array<FindMatch>} */
const matches = []
const value = toString(node)
let start = value.indexOf(':')
let end = start === -1 ? -1 : value.indexOf(':', start + 1)

// Get Gemoji shortcodes.
while (end !== -1) {
const slice = value.slice(start + 1, end)

if (own.call(nameToEmoji, slice)) {
matches.push({start, end})
start = value.indexOf(':', end + 1)
} else {
start = end
}

end = start === -1 ? -1 : value.indexOf(':', start + 1)
}

/** @type {RegExpExecArray | null} */
let match

// Get emoji.
while ((match = emojiExpression.exec(value))) {
const start = match.index
let end = start + match[0].length - 1

if (value.charCodeAt(end + 1) === 0xfe_0f) {
end++
}

matches.push({start, end})
}

matches.sort(sort)

return matches
}

/**
* Sort matches: earlier comes earlier.
*
* @param {FindMatch} a
* Match a.
* @param {FindMatch} b
* Match b.
* @returns {number}
* Sorting.
*/
function sort(a, b) {
return a.start - b.start
}

/**
* Create a shifted point.
*
* > 👉 **Note**: this cannot shift accros lines.
* > It also breaks for escapes and such.
*
* @param {Point} point
* Original point.
* @param {number} offset
* Number to shift.
* @returns {Point}
* Shifted point.
*/
function shift(point, offset) {
return {
line: point.line,
column: point.column + offset,
offset: (point.offset || 0) + offset
}
}
83 changes: 45 additions & 38 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "nlcst-emoji-modifier",
"version": "5.0.0",
"version": "6.0.2",
"description": "nlcst utility to support emoji",
"license": "MIT",
"keywords": [
@@ -25,68 +25,75 @@
],
"sideEffects": false,
"type": "module",
"main": "index.js",
"types": "index.d.ts",
"exports": "./index.js",
"files": [
"lib/",
"index.d.ts",
"index.js"
],
"dependencies": {
"emoji-regex": "^9.0.0",
"gemoji": "^7.0.0",
"nlcst-to-string": "^3.0.0",
"unist-util-generated": "^2.0.0",
"unist-util-position": "^4.0.0",
"unist-util-visit": "^3.0.0"
"@types/nlcst": "^2.0.0",
"emoji-regex": "^10.0.0",
"gemoji": "^8.0.0",
"nlcst-emoticon-modifier": "^3.0.0",
"nlcst-to-string": "^4.0.0",
"unist-util-generated": "^3.0.0",
"unist-util-position": "^5.0.0",
"unist-util-visit-parents": "^6.0.0"
},
"devDependencies": {
"@types/tape": "^4.0.0",
"c8": "^7.0.0",
"@types/node": "^20.0.0",
"c8": "^8.0.0",
"is-hidden": "^2.0.0",
"parse-english": "^5.0.0",
"prettier": "^2.0.0",
"remark-cli": "^9.0.0",
"remark-preset-wooorm": "^8.0.0",
"rimraf": "^3.0.0",
"tape": "^5.0.0",
"parse-english": "^7.0.0",
"prettier": "^3.0.0",
"remark-cli": "^11.0.0",
"remark-preset-wooorm": "^9.0.0",
"type-coverage": "^2.0.0",
"typescript": "^4.0.0",
"unist-builder": "^3.0.0",
"unist-util-remove-position": "^4.0.0",
"xo": "^0.38.0"
"typescript": "^5.0.0",
"unist-builder": "^4.0.0",
"xo": "^0.55.0"
},
"scripts": {
"prepack": "npm run build && npm run format",
"build": "rimraf \"{test/**,}*.d.ts\" && tsc && type-coverage",
"format": "remark . -qfo && prettier . --write --loglevel warn && xo --fix",
"test-api": "node test/index.js",
"test-coverage": "c8 --check-coverage --branches 100 --functions 100 --lines 100 --statements 100 --reporter lcov node test/index.js",
"build": "tsc --build --clean && tsc --build && type-coverage",
"format": "remark . -qfo && prettier . -w --log-level warn && xo --fix",
"test-api": "node --conditions development test/index.js",
"test-coverage": "c8 --100 --reporter lcov npm run test-api",
"test": "npm run build && npm run format && npm run test-coverage"
},
"prettier": {
"tabWidth": 2,
"useTabs": false,
"singleQuote": true,
"bracketSpacing": false,
"semi": false,
"trailingComma": "none"
},
"xo": {
"prettier": true,
"rules": {
"unicorn/no-array-for-each": "off",
"no-var": "off",
"prefer-arrow-callback": "off"
}
"singleQuote": true,
"tabWidth": 2,
"trailingComma": "none",
"useTabs": false
},
"remarkConfig": {
"plugins": [
"preset-wooorm"
"remark-preset-wooorm"
]
},
"typeCoverage": {
"atLeast": 100,
"detail": true,
"ignoreCatch": true,
"strict": true
},
"xo": {
"overrides": [
{
"files": "test/**/*.js",
"rules": {
"no-await-in-loop": 0
}
}
],
"prettier": true,
"rules": {
"unicorn/prefer-at": "off",
"unicorn/prefer-code-point": "off"
}
}
}
118 changes: 95 additions & 23 deletions readme.md
Original file line number Diff line number Diff line change
@@ -8,31 +8,66 @@
[![Backers][backers-badge]][collective]
[![Chat][chat-badge]][chat]

[**nlcst**][nlcst] utility to classify emoji and gemoji shortcodes as
`EmoticonNode`s.
[nlcst][] utility to classify emoji and gemoji shortcodes as `EmoticonNode`s.

> **Note**: You probably want to use [`retext-emoji`][retext-emoji].
## Contents

## Install
* [What is this?](#what-is-this)
* [When should I use this?](#when-should-i-use-this)
* [Install](#install)
* [Use](#use)
* [API](#api)
* [`emojiModifier(node)`](#emojimodifiernode)
* [Types](#types)
* [Compatibility](#compatibility)
* [Related](#related)
* [Contribute](#contribute)
* [License](#license)

## What is this?

This utility searches for emoji (`👍`) and gemoji shortcodes (`:+1:`) and turns
them into separate nodes.

## When should I use this?

This package is [ESM only](https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c):
Node 12+ is needed to use it and it must be `import`ed instead of `require`d.
This package is a tiny utility that helps when dealing with emoji and gemoji in
natural language.
The plugin [`retext-emoji`][retext-emoji] wraps this utility and others at a
higher-level (easier) abstraction.

[npm][]:
## Install

This package is [ESM only][esm].
In Node.js (version 16+), install with [npm][]:

```sh
npm install nlcst-emoji-modifier
```

In Deno with [`esm.sh`][esmsh]:

```js
import {emojiModifier} from 'https://esm.sh/nlcst-emoji-modifier@6'
```

In browsers with [`esm.sh`][esmsh]:

```html
<script type="module">
import {emojiModifier} from 'https://esm.sh/nlcst-emoji-modifier@6?bundle'
</script>
```

## Use

```js
import {emojiModifier} from 'nlcst-emoji-modifier'
import {inspect} from 'unist-util-inspect'
import {ParseEnglish} from 'parse-english'
import {inspect} from 'unist-util-inspect'

var english = new ParseEnglish()
english.useFirst('tokenizeSentence', emojiModifier)
const english = new ParseEnglish()
english.tokenizeSentencePlugins.unshift(emojiModifier)

console.log(inspect(english.parse('It’s raining :cat:s and :dog:s.')))
```
@@ -66,28 +101,53 @@ RootNode[1] (1:1-1:32, 0-31)

## API

This package exports the following identifiers: `emojiModifier`.
This package exports the identifier [`emojiModifier`][api-emoji-modifier].
There is no default export.

### `emojiModifier(node)`

Merge emoji and gemoji into a new `EmoticonNode`.
Classify emoji (👍) and Gemoji (GitHub emoji, :+1:) in `node` as `Emoticon`s.

See [`Emoticon` in `nlcst-emoticon-modifier`][emoticon-mofifier-emoticon].

###### Parameters

* `node` ([`SentenceNode`][sentence]).
* `node` ([`Sentence`][sentence])
— sentence to transform

###### Returns

Nothing (`undefined`).

## Types

This package is fully typed with [TypeScript][].
It exports no additional types

See [`Emoticon` in `nlcst-emoticon-modifier`][emoticon-mofifier-emoticon] on
how to register it in TypeScript.

## Compatibility

Projects maintained by the unified collective are compatible with maintained
versions of Node.js.

When we cut a new major release, we drop support for unmaintained versions of
Node.
This means we try to keep the current release line, `nlcst-emoji-modifier@^6`,
compatible with Node.js 16.

## Related

* [`nlcst-affix-emoticon-modifier`](https://github.com/syntax-tree/nlcst-affix-emoticon-modifier)
Merge affix emoticons into the previous sentence in nlcst
merge affix emoticons into the previous sentence in nlcst
* [`nlcst-emoticon-modifier`](https://github.com/syntax-tree/nlcst-emoticon-modifier)
Support emoticons
support emoticons

## Contribute

See [`contributing.md` in `syntax-tree/.github`][contributing] for ways to get
started.
See [`contributing.md`][contributing] in [`syntax-tree/.github`][health] for
ways to get started.
See [`support.md`][support] for ways to get help.

This project has a [code of conduct][coc].
@@ -112,9 +172,9 @@ abide by its terms.

[downloads]: https://www.npmjs.com/package/nlcst-emoji-modifier

[size-badge]: https://img.shields.io/bundlephobia/minzip/nlcst-emoji-modifier.svg
[size-badge]: https://img.shields.io/badge/dynamic/json?label=minzipped%20size&query=$.size.compressedSize&url=https://deno.bundlejs.com/?q=nlcst-emoji-modifier

[size]: https://bundlephobia.com/result?p=nlcst-emoji-modifier
[size]: https://bundlejs.com/?q=nlcst-emoji-modifier

[sponsors-badge]: https://opencollective.com/unified/sponsors/badge.svg

@@ -128,18 +188,30 @@ abide by its terms.

[npm]: https://docs.npmjs.com/cli/install

[esm]: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

[esmsh]: https://esm.sh

[typescript]: https://www.typescriptlang.org

[license]: license

[author]: https://wooorm.com

[contributing]: https://github.com/syntax-tree/.github/blob/HEAD/contributing.md
[health]: https://github.com/syntax-tree/.github

[support]: https://github.com/syntax-tree/.github/blob/HEAD/support.md
[contributing]: https://github.com/syntax-tree/.github/blob/main/contributing.md

[coc]: https://github.com/syntax-tree/.github/blob/HEAD/code-of-conduct.md
[support]: https://github.com/syntax-tree/.github/blob/main/support.md

[retext-emoji]: https://github.com/syntax-tree/retext-emoji
[coc]: https://github.com/syntax-tree/.github/blob/main/code-of-conduct.md

[retext-emoji]: https://github.com/retextjs/retext-emoji

[nlcst]: https://github.com/syntax-tree/nlcst

[sentence]: https://github.com/syntax-tree/nlcst#sentence

[emoticon-mofifier-emoticon]: https://github.com/syntax-tree/nlcst-emoticon-modifier#emoticon

[api-emoji-modifier]: #emojimodifiernode
2 changes: 1 addition & 1 deletion test/fixtures/base-is-not-emoji-with-invalid-variant.json
Original file line number Diff line number Diff line change
@@ -73,7 +73,7 @@
}
},
{
"type": "SymbolNode",
"type": "EmoticonNode",
"value": "",
"position": {
"start": {
2 changes: 1 addition & 1 deletion test/fixtures/variance-selector-no-closer.json
Original file line number Diff line number Diff line change
@@ -73,7 +73,7 @@
}
},
{
"type": "SymbolNode",
"type": "EmoticonNode",
"value": "",
"position": {
"start": {
287 changes: 159 additions & 128 deletions test/index.js
Original file line number Diff line number Diff line change
@@ -1,44 +1,41 @@
/**
* @typedef {import('unist').Node} Node
* @typedef {import('nlcst').Root} Root
*/

import fs from 'fs'
import path from 'path'
import assert from 'assert'
import test from 'tape'
// @ts-ignore Remove when typed.
import {ParseEnglish} from 'parse-english'
import assert from 'node:assert/strict'
import fs from 'node:fs/promises'
import test from 'node:test'
import {gemoji} from 'gemoji'
import {isHidden} from 'is-hidden'
import {emojiModifier} from 'nlcst-emoji-modifier'
import {toString} from 'nlcst-to-string'
import {removePosition} from 'unist-util-remove-position'
import {ParseEnglish} from 'parse-english'
import {u} from 'unist-builder'
import {gemoji} from 'gemoji'
import {emojiModifier} from '../index.js'

var position = new ParseEnglish()
var noPosition = new ParseEnglish()
noPosition.position = false
const parser = new ParseEnglish()

position.useFirst('tokenizeSentence', emojiModifier)
noPosition.useFirst('tokenizeSentence', emojiModifier)
parser.tokenizeSentencePlugins.unshift(emojiModifier)

var vs16 = '\uFE0F'
const vs16 = '\uFE0F'

test('emojiModifier()', function (t) {
var root = path.join('test', 'fixtures')
test('emojiModifier', async function (t) {
await t.test('should expose the public api', async function () {
assert.deepEqual(Object.keys(await import('nlcst-emoji-modifier')).sort(), [
'emojiModifier'
])
})

t.throws(
function () {
// @ts-ignore runtime.
emojiModifier({})
},
/Missing children in `parent/,
'should throw when not given a parent'
)
await t.test('should throw when not given a parent', async function () {
assert.throws(function () {
// @ts-expect-error: check how a non-parent is handled.
emojiModifier({type: 'TextNode', value: 'Alpha'})
}, /Missing children in `parent/)
})

t.deepEqual(
emojiModifier(
u('SentenceNode', [
await t.test(
'should merge whole words with surrounding punctuation',
async function () {
const tree = u('SentenceNode', [
u('WordNode', [u('TextNode', 'Alpha')]),
u('WhiteSpaceNode', ' '),
u('PunctuationNode', ':'),
@@ -55,136 +52,170 @@ test('emojiModifier()', function (t) {
]),
u('PunctuationNode', ':')
])
),
u('SentenceNode', [

emojiModifier(tree)

assert.deepEqual(
tree,
u('SentenceNode', [
u('WordNode', [u('TextNode', 'Alpha')]),
u('WhiteSpaceNode', ' '),
u('EmoticonNode', ':south_georgia_south_sandwich_islands:')
])
)
}
)

await t.test('should merge punctuation and words', async function () {
const tree = u('SentenceNode', [
u('WordNode', [u('TextNode', 'Alpha')]),
u('WhiteSpaceNode', ' '),
u('EmoticonNode', ':south_georgia_south_sandwich_islands:')
]),
'should merge whole words with surrounding punctuation'
)
u('PunctuationNode', ':'),
u('PunctuationNode', '-'),
u('WordNode', [u('TextNode', '1')]),
u('PunctuationNode', ':')
])

emojiModifier(tree)

t.deepEqual(
emojiModifier(
assert.deepEqual(
tree,
u('SentenceNode', [
u('WordNode', [u('TextNode', 'Alpha')]),
u('WhiteSpaceNode', ' '),
u('PunctuationNode', ':'),
u('PunctuationNode', '-'),
u('WordNode', [u('TextNode', '1')]),
u('PunctuationNode', ':')
u('EmoticonNode', ':-1:')
])
),
u('SentenceNode', [
)
})

await t.test('should merge punctuation, symbols, words', async function () {
const tree = u('SentenceNode', [
u('WordNode', [u('TextNode', 'Alpha')]),
u('WhiteSpaceNode', ' '),
u('EmoticonNode', ':-1:')
]),
'should merge punctuation and words'
)
u('PunctuationNode', ':'),
u('SymbolNode', '+'),
u('WordNode', [u('TextNode', '1')]),
u('PunctuationNode', ':')
])

t.deepEqual(
emojiModifier(
emojiModifier(tree)

assert.deepEqual(
tree,
u('SentenceNode', [
u('WordNode', [u('TextNode', 'Alpha')]),
u('WhiteSpaceNode', ' '),
u('PunctuationNode', ':'),
u('SymbolNode', '+'),
u('WordNode', [u('TextNode', '1')]),
u('PunctuationNode', ':')
u('EmoticonNode', ':+1:')
])
),
u('SentenceNode', [
u('WordNode', [u('TextNode', 'Alpha')]),
u('WhiteSpaceNode', ' '),
u('EmoticonNode', ':+1:')
]),
'should merge punctuation, symbols, words'
)
)
})

t.deepEqual(
emojiModifier(
u('SentenceNode', [
await t.test(
'should support a by GH not required variant selector',
async function () {
const tree = u('SentenceNode', [
u('WordNode', [u('TextNode', 'Zap')]),
u('WhiteSpaceNode', ' '),
u('SymbolNode', '⚡'),
u('WordNode', [u('TextNode', '️')]),
u('PunctuationNode', '.')
])
),
u('SentenceNode', [
u('WordNode', [u('TextNode', 'Zap')]),
u('WhiteSpaceNode', ' '),
u('EmoticonNode', '⚡️'),
u('PunctuationNode', '.')
]),
'should support a by GH not required variant selector'

emojiModifier(tree)

assert.deepEqual(
tree,
u('SentenceNode', [
u('WordNode', [u('TextNode', 'Zap')]),
u('WhiteSpaceNode', ' '),
u('EmoticonNode', '⚡️'),
u('PunctuationNode', '.')
])
)
}
)
})

var files = fs.readdirSync(root)
var index = -1
/** @type {Node} */
var tree
/** @type {string} */
var name
test('fixtures', async function (t) {
const root = new URL('fixtures/', import.meta.url)
const files = await fs.readdir(root)
let index = -1

while (++index < files.length) {
if (isHidden(files[index])) continue
const file = files[index]

tree = JSON.parse(String(fs.readFileSync(path.join(root, files[index]))))
name = path.basename(files[index], path.extname(files[index]))
if (isHidden(file)) continue

t.deepEqual(position.parse(toString(tree)), tree, name)
t.deepEqual(
noPosition.parse(toString(tree)),
removePosition(tree, true),
name + ' (positionless)'
)
}
const name = file.split('.').slice(0, -1).join('.')

await t.test(name, async function () {
/** @type {Root} */
const tree = JSON.parse(String(await fs.readFile(new URL(file, root))))
const input = toString(tree)

t.end()
assert.deepEqual(parser.parse(input), tree, name)
})
}
})

test('all emoji and gemoji', function (t) {
gemoji.forEach(function (info) {
var shortcode = ':' + info.names[0] + ':'
var emoji = info.emoji

t.doesNotThrow(function () {
var fixture = 'Alpha ' + shortcode + ' bravo.'
var tree = position.parse(fixture)
var node = tree.children[0].children[0].children[2]

assert.strictEqual(node.type, 'EmoticonNode', 'type')
assert.strictEqual(node.value, shortcode, 'value')
}, shortcode)

t.doesNotThrow(function () {
var expected = emoji
var tree = position.parse('Alpha ' + emoji + ' bravo.')
var node = tree.children[0].children[0].children[2]

// Try with variant selector.
if (node.type !== 'EmoticonNode' || node.value !== expected) {
expected =
expected.charAt(expected.length - 1) === vs16
? expected
: expected + vs16
tree = position.parse('Alpha ' + expected + ' bravo.')
node = tree.children[0].children[0].children[2]

// Try without variant selector.
test('all emoji and gemoji', async function (t) {
let index = -1

while (++index < gemoji.length) {
const info = gemoji[index]
const shortcode = ':' + info.names[0] + ':'
const emoji = info.emoji

await t.test(shortcode + ': ' + emoji, async function () {
assert.doesNotThrow(function () {
const fixture = 'Alpha ' + shortcode + ' bravo.'
const tree = parser.parse(fixture)
const paragraph = tree.children[0]
assert(paragraph.type === 'ParagraphNode')
const sentence = paragraph.children[0]
assert(sentence.type === 'SentenceNode')
const node = sentence.children[2]
assert(node.type === 'EmoticonNode')
assert.equal(node.value, shortcode)
})

assert.doesNotThrow(function () {
let expected = emoji
const tree = parser.parse('Alpha ' + emoji + ' bravo.')
const paragraph = tree.children[0]
assert(paragraph.type === 'ParagraphNode')
const sentence = paragraph.children[0]
assert(sentence.type === 'SentenceNode')
let node = sentence.children[2]

// Try with variant selector.
if (node.type !== 'EmoticonNode' || node.value !== expected) {
expected = expected.slice(0, -1)
tree = position.parse('Alpha ' + expected + ' bravo.')
node = tree.children[0].children[0].children[2]
expected =
expected.charAt(expected.length - 1) === vs16
? expected
: expected + vs16
const tree = parser.parse('Alpha ' + expected + ' bravo.')
const paragraph = tree.children[0]
assert(paragraph.type === 'ParagraphNode')
const sentence = paragraph.children[0]
assert(sentence.type === 'SentenceNode')
node = sentence.children[2]

// Try without variant selector.
if (node.type !== 'EmoticonNode' || node.value !== expected) {
expected = expected.slice(0, -1)
const tree = parser.parse('Alpha ' + expected + ' bravo.')
const paragraph = tree.children[0]
assert(paragraph.type === 'ParagraphNode')
const sentence = paragraph.children[0]
assert(sentence.type === 'SentenceNode')
node = sentence.children[2]
}
}
}

assert.strictEqual(node.type, 'EmoticonNode')
assert.strictEqual(node.value, expected)
}, emoji)
})

t.end()
assert(node.type === 'EmoticonNode')
assert.equal(node.value, expected)
})
})
}
})
18 changes: 9 additions & 9 deletions tsconfig.json
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
{
"include": ["*.js", "test/**/*.js"],
"compilerOptions": {
"target": "ES2020",
"lib": ["ES2020"],
"module": "ES2020",
"moduleResolution": "node",
"allowJs": true,
"checkJs": true,
"customConditions": ["development"],
"declaration": true,
"emitDeclarationOnly": true,
"allowSyntheticDefaultImports": true,
"skipLibCheck": true
}
"exactOptionalPropertyTypes": true,
"lib": ["es2022"],
"module": "node16",
"strict": true,
"target": "es2022"
},
"exclude": ["coverage/", "node_modules/"],
"include": ["**/**.js"],
}