Always getting a captcha

[Aaron2550]

Please attempt to answer the following questions before submitting a new issue:

• What version of Cloudscraper are you using?
  • 4.2.0
• What version of Node.js are you using? (Please share the process information)
  • https://pastebin.com/J1gMwMqt
• When did the problem start occurring?
  • since ive started using cloudscraper
• How often does the problem occur?
  • every time
• What is the URL?

• tried http://tmofans.com/ https://tmofans.com/ https://japscan.co http://japscan.co

• Are there any similar issues? (Please share the links)
  Can't make it work #229
  Yet another issue on SSL/TLS #224
  Always Captcha (with proxies) #234

• I have read the README. (Code examples)

Please share a minimal working code snippet that reproduces the problem.

Code snippet

const { constants } = require('crypto');

var cloudscraper = require('cloudscraper').defaults({ resolveWithFullResponse: true, agentOptions: {
    // Disable TLSv1.0/TLSv1.1
    secureOptions: constants.SSL_OP_NO_TLSv1 | constants.SSL_OP_NO_TLSv1_1,
    // Removes a few problematic TLSv1.0 ciphers to avoid CAPTCHA
    ciphers:'ECDHE-RSA-AES128-GCM-SHA256'
  } });
var fs = require('fs');

var uri = process.argv[2];

cloudscraper.debug = true;
cloudscraper.get(uri).then(onResponse).catch(onError);

function onResponse (response) {
  var request = JSON.stringify(response.request.toJSON(), null, 2);
  var headers = JSON.stringify(response.headers, null, 2);

  fs.writeFileSync('./request.json', request, 'utf8');
  fs.writeFileSync('./headers.json', headers, 'utf8');
  fs.writeFileSync('./body.html', response.body, 'utf8');
}

function onError (error) {
  console.error(error.stack);

  fs.writeFileSync('./error.txt', error.stack, 'utf8');

  if (error.cause) {
    console.log('Cause: ', error.cause);
  }

  if (error.response) {
    onResponse(error.response);
  }
}

[ghost]

Hi @Aaron2550,

In every case that I've seen, where Cloudflare's TLS pattern detection consistently causes users to receive CAPTCHA, OpenSSL is advertising SHA1 in the signature algorithms. There's an easy way to check if that's the case here as well.

1. Change the uri in your code snippet to https://www.ssllabs.com/ssltest/viewMyClient.html
2. Optionally insert <base href="https://www.ssllabs.com/ssltest/viewMyClient.html" /> into the head of the saved html document(response) to make it pretty when viewing the results in browser.
3. Examine signature algorithms as reported in the document. I like to use python -m SimpleHTTPServer and navigate to http://localhost:8000
4. Share the results

If you're able to use a proxy to view the exact same URL in browser without CAPTCHA, cookies cleared beforehand, it's not an IP issue.

This cipher string will get you pretty close to chromium/chrome which is the default user agent/headers that are sent: crypto.constants.defaultCipherList + ':!ECDHE+SHA:!AES128-SHA:!AESCCM:!DHE:!ARIA'

The default cipher string is currently: crypto.constants.defaultCipherList + ':!ECDHE+SHA:!AES128-SHA'

I can't omit SHA1 completely on my system without getting handshake errors but other systems can so give that a try: crypto.constants.defaultCipherList + ':!ECDHE+SHA:!AES128-SHA:!AESCCM:!DHE:!ARIA' + ':!SHA'

See my #268 (comment), you'll have to test that with Node.js v12.11.0. It was just released a few days ago.

Trial & Error, you've already tried using a single cipher in the shared code snippet, try different ciphers and sigalgs in conjunction with !ECDHE+SHA:!AES128-SHA:!AESCCM:!DHE:!ARIA and :!SHA, if possible.

Keep checking with ssllabs to see if you've successfully modified your signature(s) and compare to your browser by visiting the same URL.

Sorry for the slow response. I didn't get a notification :(

[ghost]

@Aaron2550

https://tmofans.com/ responds with CAPTCHA in browsers and https://japscan.co/ is down. So, I'm going to close this for now. If I'm mistaken, please let me know by commenting on this issue and it'll be reopened.