crypto.pbkdf2Sync return different result that 7 node

[chetverikov]

• Node: 7.10 and 8.9:
• OS: macOs 10.13:
• Subsystem: crypto:

crypto.pbkdf2Sync in node version 8.9 returns different result than in 7.10 node versions (current situations not equal with #7464)

const crypto = require('crypto');
const users = require('./dump/users.json');

// users[0].hash contain result of crypto.pbkdf2Sync(password, salt, 233, 512, 'SHA1').toString() in node 6.10
const oldHash = Buffer.from(users[0].hash);
const salt = users[0].salt;
const password = '!Q@W#E$R%T';

const newHash = Buffer.from(
  crypto.pbkdf2Sync(password, salt, 233, 512, 'SHA1').toString()
);

console.log(oldHash.compare(newHash));

it oldHash.compare(newHash) return 0 in node 6 and node 7, but node 8 return 1. Why? What am I doing wrong? )

[addaleax]

$ nvm use 7.10
Now using node v7.10.1 (npm v4.2.0)
$ node
> crypto.pbkdf2Sync('!Q@W#E$R%T', 'foobar', 233, 512, 'SHA1').toString('hex')
'126c9475b641a0c4ceaff5c94a0a47e974636f9499c04cded88c066997f6125a0fd483b24a89e504dbf2c649e997d1bd0337e9b2eca7144abca2defc36053c36fd74bf8d46fa7f1b9e5cde035b1d45c35f6410936ce9ec70c3826a9d6a12c89faeb20f90abf0a2f658ad59ea1499157a5673b24bb293abefbae6593a7c968331d01f282376c1a2cc2b0d9ec026aa10f271d3a1ed93352c4bd528a8dbf8dfd82b52e5f8e96638f8c13d8674975a8b9674f86a0f7792f6ea56ceaa064c8b63fc4e22f6c832fb2ea5e43b1b14d78d7605b705ec7558e460a2a83807bd00cd31b8b8083d30e63234d879cbb5a886a7c03959ba81f9d76fa69b4cb9cf4bb79298aba5b1b0161abde5eb1e1185bbb1c09eacf8546817883bfbcc549b2cb0ab7a2ad5cc4b1bdb32f3f37f80208dcf47a76a7835c93578f4523ac06bc59d64a41946fe679e09308f0e0d15a03a71c43bbee2024be4f0553e4aedee8927316a2d7640b78358b1e4246e65dbf9fbe50799beb987f4bfa89dcad057a00ca0960b9306d2ad8dc4245694aa2176397e4600b100e0a663b5a4e89af68db16208cea3477ba8a76657bcf1beff5eaa05f608f6ac57e270d470ac2830e2ac6cecdb78ea476bdce368116217879e77f9d900fb1a13b1171bef53fcb1209e0a5311ca34a83955d4cbc11b1fe86b49788965437a75f10cb7d36d2ce8387ce4c171961aa192510eabefbb'
> 
$ nvm use 8.9
Now using node v8.9.1 (npm v5.5.1)
$ node
> crypto.pbkdf2Sync('!Q@W#E$R%T', 'foobar', 233, 512, 'SHA1').toString('hex')
'126c9475b641a0c4ceaff5c94a0a47e974636f9499c04cded88c066997f6125a0fd483b24a89e504dbf2c649e997d1bd0337e9b2eca7144abca2defc36053c36fd74bf8d46fa7f1b9e5cde035b1d45c35f6410936ce9ec70c3826a9d6a12c89faeb20f90abf0a2f658ad59ea1499157a5673b24bb293abefbae6593a7c968331d01f282376c1a2cc2b0d9ec026aa10f271d3a1ed93352c4bd528a8dbf8dfd82b52e5f8e96638f8c13d8674975a8b9674f86a0f7792f6ea56ceaa064c8b63fc4e22f6c832fb2ea5e43b1b14d78d7605b705ec7558e460a2a83807bd00cd31b8b8083d30e63234d879cbb5a886a7c03959ba81f9d76fa69b4cb9cf4bb79298aba5b1b0161abde5eb1e1185bbb1c09eacf8546817883bfbcc549b2cb0ab7a2ad5cc4b1bdb32f3f37f80208dcf47a76a7835c93578f4523ac06bc59d64a41946fe679e09308f0e0d15a03a71c43bbee2024be4f0553e4aedee8927316a2d7640b78358b1e4246e65dbf9fbe50799beb987f4bfa89dcad057a00ca0960b9306d2ad8dc4245694aa2176397e4600b100e0a663b5a4e89af68db16208cea3477ba8a76657bcf1beff5eaa05f608f6ac57e270d470ac2830e2ac6cecdb78ea476bdce368116217879e77f9d900fb1a13b1171bef53fcb1209e0a5311ca34a83955d4cbc11b1fe86b49788965437a75f10cb7d36d2ce8387ce4c171961aa192510eabefbb'
> 

I am not aware of any changes that would break that code. It would be helpful if you could provide a complete piece of code that produces different results on these versions. (e.g. the dump/users.json file is missing in your example).

[chetverikov]

@addaleax File dump/users.json contain a generated data in node 6. And *.hash contain string (crypto.pbkdf2Sync(password, salt, 233, 512, 'SHA1').toString()). I want update node to 8, but I get another result from crypto.pbkdf2Sync().toString(). I researched the problem and realized that I forgot to substitute toString('base64'). But, if parse a data from dump by buffer.from() and parse result of crypto.pbkdf2Sync().toString() then buffers is equal, but only node < 8

[chetverikov]

@addaleax You can generate data (generate in node 6):

const crypto = require('crypto');
const password = '!Q@W#E$R%T';
const salt = '!@#$%^&*()_}{POIUYTREWQ';

require('fs').writeFileSync('dump.json', JSON.stringify([{
  hash: crypto.pbkdf2Sync(password, salt, 233, 512, 'SHA1').toString(),
  salt
}]));

[addaleax]

crypto.pbkdf2Sync(password, salt, 233, 512, 'SHA1').toString()

These parts are broken. The return value of pbkdf2Sync() is a Buffer instance, which contains arbitrary binary data.

When you call .toString() on a buffer, the default encoding is utf8 (→ docs) – but that only works if your buffer actually contained utf8-encoded data. So what you end up with instead is the Node’s best attempt at decoding the data, which contains a lot of � replacement characters (and as a consequence of that using Buffer.from() on that string will not give you the same buffer that you started out with).

What changed from Node 7 to Node 8 is that the JS engine now accumulates multiple � characters into a single one when trying to decode invalid UTF-8.

If you want to store binary data, like the pbkdf2Sync() hash, in a text file, you always should use a binary-to-text encoding like base64 or hex, both of which are supported by Buffer.from() and buffer.toString().

I’m closing this since it’s not a bug inside Node itself. If you have similar issues or have more questions about this, https://github.com/nodejs/help is a good place to start.