web3 does not work with webpack unless I manually install with --unsafe-perm=true

[evbots]

Hi web3 team,

I have tried installing various versions of web3 in my webpack projects. I get very similar errors when using webpack to bundle client-side projects and server-side projects (specifically a serverless project using the serverless-webpack plugin).

The current version I'm using is [email protected] in both my client and server-side projects.

Client-side project details:

With regards to my client-side react project, installing web3 as a dependency and webpacking results in the following error: Module not found: Error: Can't resolve 'websocket' in '/home/circleci/project/node_modules/web3-providers-ws/src'. This is what the package.json dependencies looks like for that project:

"dependencies": {
    "bootstrap": "^4.1.3",
    "connected-react-router": "^4.4.1",
    "husky": "^0.14.3",
    "immutable": "^3.8.2",
    "lint-staged": "^7.2.2",
    "lodash": "^4.17.10",
    "node-sass-chokidar": "^1.3.3",
    "npm-run-all": "^4.1.3",
    "prettier": "^1.14.2",
    "react": "^16.4.1",
    "react-dom": "^16.4.1",
    "react-redux": "^5.0.7",
    "react-router-dom": "^4.3.1",
    "react-scripts": "1.1.4",
    "redux": "^4.0.0",
    "redux-immutable": "^4.0.0",
    "web3": "1.0.0-beta.35"
  }

This error magically goes away if I remove web3 from my package.json dependencies, and manually run: npm install [email protected] --save --save-exact --unsafe-perm=true before webpacking my client-side bundle. This means I exclude web3 from my package.json dependencies, and run an extra step afterwards to install web3. I really don't like this, as it prevents me from generating a comprehensive package-lock.json.

It seems like one of the web3 dependencies needs more expansive file permissions in order to compile. Interestingly, this does not happen when I bundle via webpack locally on MacOS, only in my linux OS ci pipeline. I did some research trying to figure out if NPM would let me specify the --unsafe-perm flag in my package.json file, but it appears that flag is only respected when installing a single package.

Is there any solution for this, or am I stuck with an extra step installing web3 manually with --unsafe-perm?

[simodroussi]

Hi, I am having the same error when building for production in the pipelines.

ERROR in ./node_modules/web3-providers-ws/src/index.js
Module not found: Error: Can't resolve 'websocket' in '/opt/atlassian/pipelines/agent/build/node_modules/web3-providers-ws/src'
 @ ./node_modules/web3-providers-ws/src/index.js 38:9-29
 @ ./node_modules/web3-core-requestmanager/src/index.js
 @ ./node_modules/web3-core/src/index.js
 @ ./node_modules/web3/src/index.js

I am hoping someone can help solve this.

[evbots]

@simodroussi do you know what OS you're using for your production build process?

[simodroussi]

we are using Amazon Linux https://hub.docker.com/_/amazonlinux/

Amazon Linux is a distribution that evolved from Red Hat Enterprise Linux (RHEL) and CentOS.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 150.0 DAI (150.0 USD @ $1.0/DAI) attached to it.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $48,068.14 more funded OSS Work available on the Gitcoin Issue Explorer

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 3 months, 3 weeks ago.
Please review their action plans below:

1) sundar-balaji has been approved to start work.

I have the solution for this issue. This is permission related issue. I can help you out with this issue and resolve it asap

Learn more on the Gitcoin Issue Details page.

[spm32]

Hey @sUndar-Balaji feel free to give it a shot :)

[evbots]

I'll post further details for more clarity and to help recreate this issue. My server-side code is a serverless project using the serverless-webpack plugin. Here are my devDependencies...

"devDependencies": {
    "babel-core": "^6.26.0",
    "babel-loader": "^7.1.3",
    "babel-plugin-source-map-support": "^2.0.1",
    "babel-preset-env": "^1.6.0",
    "serverless": "^1.30.3",
    "serverless-domain-manager": "^2.6.0",
    "serverless-webpack": "^5.1.5",
    "webpack": "^4.12.0",
    "webpack-cli": "^3.0.6",
    "webpack-node-externals": "^1.7.2"
}

My client-side code is a create-react-app generated codebase, which uses webpack under the hood.

My CI job uses the CircleCI docker image circleci/node:8.11 as the build environment.

Please let me know if I can provide any further details.

[sUndar-Balaji]

you have to set "npm config set unsafe-perm true"

or you have to change the ownership of the location where you are installing the npm packages

like this "sudo chown -R $USER:$USER /opt/"

Pls try these options and let me know

[sUndar-Balaji]

I have setup a react project using create-react-app and CI for that in circle CI. I have installed web3 in the react project and imported the same in the project.

It's able to successfully build in local and also in circle CI

My circle CI configurations are

Environment - Linux

Node Js version - 8.11

web3 version - 1.0.0-beta.35

You can fork this repo and create a project in circle CI.

web3-circleci-issue-repo

If you have some more steps to reproduce the issue in the above mentioned repo, pls mention that below

[evbots]

@sUndar-Balaji I just tried setting npm config set unsafe-perm true in my build and I am still getting an error at runtime. To clarify, my build passes as well. It's at runtime where I get an error Module not found.

[sUndar-Balaji]

@evbots could you pls mention while doing which operation in web3 you are getting the error? And could you share your config.yml file?

[evbots]

Sure. The code fails before I can perform an operation. It fails on import. The line it fails on:
import Web3 from 'web3';

I'm currently working with my serverless backend project, not my front-end project. I pulled web3 out of that. My serverless project config.yml file is (just the jobs portion):

jobs:
  build:
    docker:
      - image: circleci/node:8.11

    working_directory: ~/code

    steps:
      - checkout

      - run: |
          sudo apt install python-dev python-pip
          sudo pip install awscli

      # - run: npm config set unsafe-perm true

      - run: npm config set unsafe-perm true && npm install

      - run: npm run slsDeploy

With your example repo, have you tried running the react app after it builds? You should be able to ship the files to s3 and run access it via url.

[sUndar-Balaji]

In the react app i have created using create-react-app, actually imports the web3 and does console.log(web3). Then i ran the app. I can be able to see the web3 constructor function in browser console and exported the same web3 to window object in the app's code itself.

Are you getting this error in serverless backend app or in both front end and backend?

[sUndar-Balaji]

@evbots the issue is because the version you are using. It's a beta version. I did face the similar kind of issue while running CI on the circle CI for the serverless project which uses serverless-webpack plugin
to pack files.

What i did was changing from beta to stable version(like 0.20.7 most recently pushed stable version according to npm), and it worked.

Pls try this out and let me know

[evbots]

@sUndar-Balaji Next week I might be able to find some time to recreate this issue in the react-app, but the problem was the exact same as the nodejs run project when I first created this issue. So on my todo list is to (1) try and change directory permissions via chown, and (2) recreate the front-end module import error.

As to your comment about the web3 version, this error is specific to 1.0.0-beta.35. Reverting to version 0.x is not a viable solution, because I'd like to use the new features in 1.x. I'm not sure how long web3 1.0 will technically be in "beta" (maybe @frozeman can explain), but it's used in production in many projects.

[sUndar-Balaji]

@evbots the problem only occurs when we use web3(beta) with serverless with serverless-webpack plugin, on circle ci's Linux build environment.

So the chances of ppl getting this issue might be very little. Due to which ppl may not have raised the issue.

Still I'll look for more ways to find the solution.

[evbots]

@sUndar-Balaji

Ok, I decided to go ahead and recreate the issue with my react-app right now. I added web3.js to package.json using npm install [email protected] --save --save-exact, which results in "web3": "1.0.0-beta.35" added to dependencies. My build config file is the following:

# config.yml
  build:
    docker:
      - image: circleci/node:8.11

    working_directory: ~/code

    steps:
      - checkout

      - run: |
          sudo apt install python-dev python-pip
          sudo pip install awscli

      - run: npm install
      - run: npm run build
      - run: aws s3 sync build/ s3://my-test-bucket-web3/ --delete --exclude "*.map" --cache-control "no-cache, no-store, must-revalidate"

Web3 is imported into App.js with the following line. import Web3 from 'web3';.

My build fails on the npm run build step with the following message:

#!/bin/bash -eo pipefail
npm run build

> [email protected] build /home/circleci/code
> npm-run-all build-css build-js


> [email protected] build-css /home/circleci/code
> node-sass-chokidar src/styles/manifest.scss src/styles/manifest.css


> [email protected] build-js /home/circleci/code
> react-scripts build

Creating an optimized production build...
Failed to compile.

Module not found: Error: Can't resolve 'websocket' in '/home/circleci/code/node_modules/web3-providers-ws/src'

[sUndar-Balaji]

@evbots could you share your repo details. So I'll fork that repo to fix the issue

[evbots]

@sUndar-Balaji yes, here you go: https://github.com/evbots/web3-webpack-build

just created it

[sUndar-Balaji]

@evbots #1105. An issue with babel, webpack and web3 FYI.
This one was in 1.0.0-beta.23

[evbots]

@sUndar-Balaji yes, i remember reading that issue when I was first battling this. I never was able to use the solution proposed at the bottom of the issue. I'm not sure if the default create-react-app webpack configuration transpiles node_modules. but from some cursory reading it doesn't seem like it does.

[sUndar-Balaji]

@evbots fixed the issue by using the following circle ci config.

Changed the npm installation directory's permission to the current user.
Set the unsafe-perm to true

working_directory: ~/code
steps:
  - checkout

  - run: |
      sudo apt install python-dev python-pip
      sudo pip install awscli

  - run: sudo chown -R $(whoami) $(npm config get prefix)/{lib/node_modules,bin,share}
  - run: npm config set unsafe-perm true 
  - run: npm install
  - run: npm run build

Updated code is available in this repo web3-webpack-build-issue-repo

Feel free to fork and run it on your CI environment

[evbots]

@sUndar-Balaji very nice! I will have time to apply your changes and verify tomorrow. :-D

[evbots]

@sUndar-Balaji I applied your changes to my repo and the build passed. Then, I reset the package-lock using an npm install, and the build no longer passes. I looked at the package-lock diff that you checked into your fork, and I noticed all the dependency versions are locked. How did you generate that package-lock? It seems to be a factor in making the build pass, but I'm not sure why it does.

[sUndar-Balaji]

@evbots While doing npm install from local machine I have created that package-lock file. It will be automatically created by npm. If you have issues, try to execute the chown and unsafe-perm commands from your local machine and generate package-lock file, then try

[sUndar-Balaji]

@evbots which version of npm are you using? Mine is 5.6.0. There are issues in npm, which mentions
that version 6.x has issues, which will do the "exact to range" in packages and wont do the locking.
Pls refer this npm/npm#20891 npm/npm#20434

[sUndar-Balaji]

@evbots try to remove node_modules folder, execute the chown and unsafe-perm commands from your local machine and generate package-lock file by doing npm install in local. Mostly it'll resolve your issue.

conflict between 5.x and 6.x isn't gonna affect this case that much.

[sUndar-Balaji]

@evbots i have fixed your package-lock.json issue by executing the below command in my
local machine.

sudo chown -R $(whoami) $(npm config get prefix)/{lib/node_modules,bin,share}

then you can run

npm install

It will properly generate package-lock.json file

npm run build

this will create the build.Then commit the created package-lock.json file to your repo.

Now your build will pass in both local machine and also in your circle ci

[evbots]

@sUndar-Balaji can I ask why you are editing the owner permissions for the global node_modules/ directory? My understanding was that all npm packages specified in package.json are installed locally (in the current project directory) unless otherwise specified with the -g flag. But maybe I'm misunderstanding how npm install works.

I generated package-lock.json under all of the following npm versions:
5.6.0
5.10.0
6.2.0
6.4.1

The only one that resulted in a passing build was the npm version 5.6.0. I have yet to figure out why other package-locks generated with other npm versions result in web3 being unable to install. To add to the intrigue, when I delete package-lock and run install only referencing package.json, the build passes. No need for npm config set unsafe-perm true or chown.

I am perplexed by what is happening. One thing I have not yet tried is using yarn, and if the lock file generated by yarn produces different results.

[evbots]

I think this issue might be related to #1914

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work for 150.0 DAI (150.0 USD @ $1.0/DAI) has been submitted by:

1. @sUndar-Balaji

@ceresstation please take a look at the submitted work:

• PR by @sUndar-Balaji

---

• Learn more on the Gitcoin Issue Details page
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $50,808.44 more funded OSS Work available on the Gitcoin Issue Explorer

[jamesmorgan]

I also get this very issue when trying to deploy google cloud functions which have web3 bundled.

Error: Cannot find module 'websocket'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/srv/node_modules/web3-providers-ws/src/index.js:38:10)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)

tried with web3:

• 1.0.0-beta.34
• 1.0.0-beta.35
• 1.0.0-beta.36

All with no success.

• npm 6.4.1
• node 8.9.4

[jamesmorgan]

I can confirm that I have tested rolled back npm to 5.5.0 and the build works - very odd! Hopefully we get this fixed 👍 thanks for the help

[nivida]

This issue will be solved after the PR #2000 is merged into the 1.0 branch.

[Tronglx]

Hi @nivida, i'm having the same issue, please check PR #2000, Merging is blocked

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

The funding of 150.0 DAI (150.0 USD @ $1.0/DAI) attached to this issue has been approved & issued to @sUndar-Balaji.

• Learn more on the Gitcoin Issue Details page
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $39,653.62 more funded OSS Work available on the Gitcoin Issue Explorer

[gitcoinbot]

@sUndar-Balaji Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• reminder (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@sUndar-Balaji Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• reminder (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[begi97]

0.20.7

worked for me.