Files with \r\n can be incorrectly tokenized

[alcuadrado]

Hi,

I just noticed that this file is tokenized incorrectly, and I have a strong suspicion that it's because it has \r\n end lines. In particular, it has some inside comments.

9d694731f6291f4553fad6b8174e7c915e6219ec_TigerBNB.sol

If you compile it with solc the pragma solidity ^0.6.12; statement is correctly parsed. In solang it's part of a comment.

[alcuadrado]

@seanyoung, I found more files that break like this. In case they are helpful, I'm posting them here:

54bef7b75e1bec4594ecbc8caad54328d653f5c1_Spark.sol.txt
71d0e03abd2b4b4fb9ef73d4c6f35a650124791e_BabyRock.sol.txt
85d1f31777c5c9ca544a1a73b83341872bdca45e_PUSHNOU.sol.txt
97f4706e90cbd5d882088a0a245c63cb6b5f27cf_Vitalic.sol.txt
719de440d870b18de38782ca191ed6a651a2e67a_TotoToken.sol.txt
909b4e9a086ba9feb85b3244fd83b84cf8b2ab83_Saddam.sol.txt
914a70c59340d18305a6b65f4049d18731e82fec_CodeWithJD.sol.txt
38606c5e2e7805dd763f1247bb71c08669a8b4f3_DOGEREADSFLOOR.sol.txt
69001d581C1BC73049e10a0574788902E10dED2C_SmallTiger.sol.txt
81396ce02b575382bc969ed970b8b658f0c7e3d0_Spark.sol.txt
086109f8ea01ac3D41a18405c212D555De0e5cC5_MetaverseGame.sol.txt
855322f5aeabd16ec1d5ef255504062661edc157_Spark.sol.txt
aa8a84097c721e1de0ac3004fd08d4d7b0c6f586_Spark.sol.txt
ac7f486b31098b87e67be5c998acec7e28a1b734_Address.sol.txt
b0DA2a6fCB1ff8850118948508E64De9EAd10b9d_RocketLink.sol.txt
b44f93a99b7615937e2515c4613ad26760617991_Spark.sol.txt
cd70fa5fba9f0996c83e0e95cabc1cad5edc205a_Spark.sol.txt
Db5Cede38Af7225562a4D2EE464c605a6601AC22_Web3T.sol.txt
e29d2560d861ec0e372c064f2aa00bf5ddd82ef3_HelloWorld123.sol.txt
e813091ac3cf86a2334d8f57d94c0598bf97a23a_TokenLauncher.sol.txt
eFEa63672F9942f9F793eb07BeB3b37520ef559e_SpongeBOB.sol.txt
f8097bdC3a4Bfd8a900F0e802D17dA95cD4B0e1A_Token.sol.txt
5fBd2bD682B1e490E79A75e5c2EdF0F5016b62d1_FirstETH.sol.txt
006f045e0f877a321474692839d897945fa62b45_BABYLINK.sol.txt
7Ea1A16c2B44BF2a7187F29D95f5ad3E03dF2A6A_MetaWorldV2.sol.txt
8a61E6D97cF885C94960298BBa374c6AE3D6555B_FirstMoon.sol.txt
8e28666cd07bcb39dfb939ff4e47014602c64cec_Spark.sol.txt
9c2e8bf7cf0dfe45e0ce2d4e53b2a398e5feac8c_BabyCHINA.sol.txt
9d694731f6291f4553fad6b8174e7c915e6219ec_TigerBNB.sol.txt
15bfe7fc335e9a5f5f2c519c242ffb6f4317f97f_SHIBTiger.sol.txt
21a58b28224c6191Ef1ed38A0CAb85D444E8E665_BNBGirl.sol.txt

[seanyoung]

@seanyoung, I found more files that break like this. In case they are helpful, I'm posting them here:

Great, thank you very much! I'll go through them and make sure they parse.

[alcuadrado]

btw, I'm lexing the whole smart-contract-sanctuary repository and partially comparing the results with those of @solidity-parser/parser. It should be done by tomorrow. I'll let you know if I find more issues.

[alcuadrado]

As promised, here are all the files from the sanctuary that caused errors:

failures.zip

[seanyoung]

As promised, here are all the files from the sanctuary that caused errors:

failures.zip

Thank you!

[xermicus]

@seanyoung did #745 fix this?

[seanyoung]

This issue is not just what's in the description, it's all the files in the zip file. They should be re-tested

[xermicus]

Ok, I'm not closing this then. Thanks!