IDF master #5336

me-no-dev · 2021-06-28T06:17:18Z

esp-dsp: master 801c6dd
esp-face: master 420fc7e
esp-rainmaker: f1b82c7
esp32-camera: master 7c5d8b2
esp_littlefs: master b58f00c

me-no-dev · 2021-06-28T09:14:38Z

@valeros @ivankravets guys, toolchain has to be updated yet again :) Please think of a way that we can specify the necessary version in the pltformio build files. Toolchains are very much version dependent now.

Jason2866 · 2021-06-28T15:21:57Z

@me-no-dev platformio could support directly the toolchains from espressif github (for testing stage version) if the would include a package.json.
The toolchain from earlephilhower for the esp8266 do include a package.json https://github.com/earlephilhower/esp-quick-toolchain. So it is possible with platformio to use latest Arduino Esp8266 stage and the needed latest new toolchain.

me-no-dev · 2021-06-29T08:22:48Z

@Jason2866 thanks for reminding me about this :) but it's not that straightforward to implement it on our end, because it's not us packing and compiling the toolchain. Files and binaries are signed and should not be unpacked and repacked by us (also makes little sense), but I will ask my colleagues to implement this for future versions. Is there a clear tutorial on what needs to be added? What should that json contain?

Jason2866 · 2021-06-29T10:00:58Z

@me-no-dev thx for talking to your colleagues about this. I opened a features request a while ago for this and it was closed. I think i couldnt explain the needs why clearly enough.
Nothing fancy which is needed in the package.json. Only what it is, the target OS and a valid version number. Example from ArduinoEsp8266 toolchain (without the .txt extension)
package.json.txt
Platformio doc https://docs.platformio.org/en/latest/core/userguide/package/cmd_pack.html
The python code which parses the manifest file https://github.com/platformio/platformio-core/blob/develop/platformio/package/manifest/schema.py

me-no-dev · 2021-06-29T10:08:18Z

thanks @Jason2866 :)

@valeros, @ivankravets could you please clarify what would be the proper values for ESP32x toolchains?

{
   "description": "xtensa-gcc",//what would be the value for each ESP32 chip? C3 is RISCV
   "name": "toolchain-xtensa",//goes with the one above
   "system": [ "windows_amd64" ],//what are the options here?
   "url": "https://github.com/espressif/crosstool-NG",
   "version": "5.100300.210606"//how should we format the version?
}

ivankravets · 2021-06-29T11:01:27Z

Hi there,

platformio could support directly the toolchains from espressif github (for testing stage version) if the would include a package.json.

It could potentially work for a "framework" type of package which is platform agnostic but will not work for a toolchain that should be precompiled for different architectures.

Files and binaries are signed and should not be unpacked and repacked by us (also makes little sense), but I will ask my colleagues to implement this for future versions

You can keep official packages without package.json. However, the only you package and sign them, you can generate package.json dynamically and publish them to a PlatformIO Trusted Package Registry.

PlatformIO Trusted Package Registry is a paid product for enterprises and is a part of our business model. We will contact the Espressif company and you will be able to publish packages directly from espressif organization.

Jason2866 · 2021-06-29T11:36:59Z

@ivankravets imho best way to publish directly from espressif to platformio registry
By doing this Platformio users can always use latest development versions for IDF and Arduino ESP32. 👍
@me-no-dev i hope you can convience espressif crew to do this.
Without Platformio some nice projects would not be possible, and many active open source teams do use.

Jason2866 · 2021-06-29T11:56:49Z

thanks @Jason2866 :)

@valeros, @ivankravets could you please clarify what would be the proper values for ESP32x toolchains?

{
   "description": "xtensa-gcc",//what would be the value for each ESP32 chip? C3 is RISCV
   "name": "toolchain-xtensa",//goes with the one above
   "system": [ "windows_amd64" ],//what are the options here?
   "url": "https://github.com/espressif/crosstool-NG",
   "version": "5.100300.210606"//how should we format the version?
}

For completness: Example package.json for the "old" riscv toolchain for platform linux_x86_64

{
  "name": "toolchain-riscv-esp",
  "version": "1.80400.0",
  "description": "GCC Toolchain for Espressif 32-bit RISC-V based on GCC",
  "keywords": [
    "build tools",
    "compiler",
    "assembler",
    "linker",
    "preprocessor",
    "espressif",
    "esp32",
    "risc-v"
  ],
  "license": "GPL-2.0-or-later",
  "repository": {
    "type": "git",
    "url": "https://github.com/espressif/crosstool-NG"
  },
  "system": [
    "linux_x86_64"
  ]
}

igrr · 2021-06-29T12:49:40Z

@me-no-dev I'll take a look at the linker error, it is unexpected.

I think the underlying issue is that CI in this repository relies on feature/idf-master branch of https://github.com/platformio/platform-espressif32 (

arduino-esp32/.github/scripts/install-platformio-esp32.sh

Line 4 in c37557c

    
           PLATFORMIO_ESP32_URL="https://github.com/platformio/platform-espressif32.git#feature/idf-master"

) which currently can't build IDF master branch due to the new toolchain being required.

What do you think about updating to the tip of release/v4.3 IDF branch rather than master at this point? That should work with platformio's feature/idf-v4.3 branch here: https://github.com/platformio/platform-espressif32/tree/feature/idf-v4.3. Or is there something from IDF master branch that is necessary that is not present on release/v4.3?

me-no-dev · 2021-06-29T14:40:18Z

@igrr in all cases it will not fix the issue next time the toolchain needs to be updated (previous IDF versions get updates also). To switch to 4.3 will require unknown amount of changes here and in the lib builder, which is far more than updating the toolchain. I think the greater goal would be to somehow automate toolchain update and note the required version in the PIO build scripts. Sticking to 4.4 will also allow seamless addition of S3 support.

me-no-dev · 2021-06-29T14:53:46Z

@ivankravets I had a look at the PIO TPR and it seems that it's a storage solution. We do not need storage, our toolchains are available online already. Maybe I am not getting something?

ivankravets · 2021-06-29T15:01:19Z

@me-no-dev PlatformIO does not support untrusted package sources or 3rd party archives. It strictly works only with PlatformIO Trusted Package Registry where we guarantee package authenticity. Enterprises manage their packages in our registry and keep control of package distribution. We already contacted the Espressif company and you will receive full control of already published Espressif packages soon.

me-no-dev · 2021-06-29T15:38:05Z

@ivankravets this is PR CI build. It is not merged changes. If I am not able to specify the toolchain in the build files, but add the new version to your TPR, wouldn't that mean that all dev channel users will get that new version? Even though changes are not merged? Maybe we could specify toolchain path on the command line? But really the best case would be to be able to just specify the version in the build script :)

Jason2866 · 2021-06-29T18:32:05Z

If the build chain is uploaded in platformio registry everyone can use it from platformio by specifying with the owner/version number. Uploading a package to registry is not depending on "something".
Before the C3 was supported from platformio i uploaded the needed riscv toolchains (linux, apple windows) for the C3 to the platformio registry. It can simply used in platformio with a entry in platform_packages -> tasmota/toolchain-riscv32 @ ~1.80400.210415
The complete setup can be user made. In the used example below the platform and all platform_packages where selfmade :-)

platform                    = https://github.com/Jason2866/platform-espressif32.git#feature/arduino-c3
platform_packages           = framework-arduinoespressif32 @ https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/291/framework-arduinoespressif32-master-1d7068e4b.tar.gz
                              tasmota/toolchain-riscv32 @ ~1.80400.210415

So i think you can specify with small changes in your build script the toolchain you want / need.
If owner espressif is uploading the toolchains everyone can use with espressif/toolchain-riscv32 @ ~version.date
Best solution you provide a platform.json in your github (for the different branches) via a fork from https://github.com/platformio/platform-espressif32
everyone could always use latest version with platformio and your changes in build script would be minimal!
Just this entry would be needed in platformio setup:
platform = https://github.com/espressif/platform-espressif32.git#idf-master
as example for the branch idf-master
Maintain of platform.json is easy and i am sure you can automate via scripting.
@me-no-dev

me-no-dev · 2021-06-29T23:36:34Z

@Jason2866 what would b the default toolchain if not specified?

me-no-dev · 2021-06-30T01:08:16Z

thanks for the extensive explanation @Jason2866 I'm starting to get things now.
When you uploaded the toolchains, did you have to add platform.json into the archives?

Let's say that we fork platformio/platform-espressif32 to espressif/platform-arduinoespressif32 and include the following platform.json

{
  "name": "arduinoespressif32",
  "title": "Espressif 32",
  "description": "Espressif Systems is a privately held fabless semiconductor company. They provide wireless communications and Wi-Fi chips which are widely used in mobile devices and the Internet of Things applications.",
  "homepage": "https://espressif.com/",
  "license": "Apache-2.0",
  "keywords": [
    "dev-platform",
    "Wi-Fi",
    "Bluetooth",
    "Xtensa",
    "LX6",
    "riscv"
  ],
  "engines": {
    "platformio": "^5"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/espressif/platform-arduinoespressif32.git"
  },
  "version": "3.2.0",
  "frameworks": {
    "arduino": {
      "package": "framework-arduinoespressif32",
      "script": "builder/frameworks/arduino.py"
    }
  },
  "packages": {
    "toolchain-riscv32": {
      "type": "toolchain",
      "optional": true,
      "owner": "espressif",
      "version": "~1.80400.1234"
    },
    "toolchain-xtensa32": {
      "type": "toolchain",
      "optional": true,
      "owner": "espressif",
      "version": "~2.80400.1234"
    },
    "toolchain-xtensa32s2": {
      "type": "toolchain",
      "optional": true,
      "owner": "espressif",
      "version": "~1.80400.1234"
    },
    "framework-arduinoespressif32": {
      "type": "framework",
      "optional": true,
      "owner": "espressif",
      "version": "https://github.com/espressif/arduino-esp32.git"
    },
    "tool-esptoolpy": {
      "type": "uploader",
      "owner": "platformio",
      "version": "~1.30100.0"
    },
    "tool-mkspiffs": {
      "type": "uploader",
      "optional": true,
      "owner": "platformio",
      "version": "~2.230.0"
    }
  }
}

With that anyone can build projects with the following example config:

[env:esp32dev]
platform = https://github.com/espressif/platform-arduinoespressif32.git
framework = arduino
board = esp32dev
monitor_speed = 115200

Then let's say that we have a branch called idf-master, where we need to add a new toolchain version.

We upload the new toolchain to platformio
Edit platform.json in that branch to update the version numbers (ex. ~1.80400.1234 to ~1.80400.5678)
Run CI specifying the new platform (platform = https://github.com/espressif/platform-arduinoespressif32.git#idf-master)
Or specify just the new toolchain (platform_packages = espressif/toolchain-xtensa32 @ ~1.80400.5678)
Merge the changes in both platform-arduinoespressif32 and arduino-esp32 at the same time

@Jason2866 does that sound any good?
If we can specify URL for version, then could we say platform_packages = espressif/toolchain-xtensa32 @ https://github.com/espressif/crosstool-NG/releases/download/esp-2021r1/riscv32-esp-elf-gcc8_4_0-esp-2021r1-linux-amd64.tar.gz?

Jason2866 · 2021-06-30T09:14:16Z

@me-no-dev great you are considering to do a complete working platfomio setup!

When you uploaded the toolchains, did you have to add platform.json into the archives

No, only the valid package.json for this toolchain is needed. So you have to upload the toolchain one by one for every OS
A adopted platform.json is needed in your fork of https://github.com/platformio/platform-arduinoespressif32.git
You create branches (easiest same name as your branch names from https://github.com/espressif/arduino-esp32)
In this branches you change the platform.json accordingly to your needs of the referenced namend branch.

We upload the new toolchain to platformio

Always needed / correct

Edit platform.json in that branch to update the version numbers (ex. ~1.80400.1234 to ~1.80400.5678)

Correct too, to do in your forked and namend branch of https://github.com/platformio/platform-arduinoespressif32.git

Run CI specifying the new platform (platform = https://github.com/espressif/platform-arduinoespressif32.git#idf-master)

100% correct

Or specify just the new toolchain (platform_packages = espressif/toolchain-xtensa32 @ ~1.80400.5678)

100% correct too

Merge the changes in both platform-arduinoespressif32 and arduino-esp32 at the same time

Only needed when changes in toolchain or needed tools referenced in platform.json

@Jason2866 does that sound any good?

Yes, sounds good. If you try one time you will see it is easier than then whole explanations :-)

If we can specify URL for version, then could we say platform_packages = espressif/toolchain-xtensa32 @ https://github.com/espressif/crosstool-NG/releases/download/esp-2021r1/riscv32-esp-elf-gcc8_4_0-esp-2021r1-linux-amd64.tar.gz?

Yes, this is possible (and that what i suggested before) WHEN in this tarball is a valid package.json for linux-amd64.
As Ivan wrote this toolchain is only working for linux-amd64 For CI it is totally okay since it is always Linux.
A windows user needs to reference the correct windows variant. So thats why it is the best solution to upload all variants to platformio registry. By doing this it is OS independend. The entry would look like this platform_packages = espressif/toolchain-xtensa32 @ ~1.80400.210630
This is the way to provide the toolchains for everyone to use with platformio

Jason2866 · 2021-06-30T09:22:54Z

@Jason2866 what would b the default toolchain if not specified?

The default toolchain is the referenced one in platfom.json

ivankravets · 2021-07-01T07:38:02Z

@me-no-dev you don't need to fork the original ESP32 dev-platform for PlatformIO. The steps are very simple:

Publish a new toolchain using espressif organization to the PlatformIO Registry.
Patch platform.json in https://github.com/espressif/arduino-esp32/blob/master/.github/scripts/install-platformio-esp32.sh with your custom toolchain version (we will help here)

P.S: We are in progress with creating espressif organization in PlatformIO Registry. The only we finish it, we will back here.

Jason2866 · 2021-07-14T12:05:10Z

@ivankravets @me-no-dev Any news? We would like to use the new toolchains with Platformio.
Thx!

Jason2866 · 2021-07-15T11:07:42Z

Thx @me-no-dev @ivankravets @valeros
Already using :-)

…5404) Until this commit, Kconfig options (e.g. CONFIG_TINYUSB_ENABLED) were used in conditions preceding idf_component_register to determine which components need to be added to `arduino` component requirements. However the Kconfig options aren't known at the early expansion stage, when the component CMakeLists.txt files are expanded the first time and requirements are evaluated. So all the conditions evaluated as if the options were not set. This commit changes the logic to only add these components as dependencies when the Kconfig options are known. Dependencies become "weak", which means that if one of the components isn't included into the build for some reason, it is not added as a dependency. This may happen, for example, if the component is not present in the `components` directory or is excluded by setting `COMPONENTS` variable in the project CMakeLists.txt file. This also ensures that if the component is not present, it will not be added as a dependency, and this will allow the build to proceed. Follow-up to #5391. Closes #5319.

* Update PlatformIO CI build script - Switch to the latest toolchains 8.4.0 for ESP32, ESP32S2, ESP32C3 - Use PlatformIO from master branch for better robustness * Update package.json for PlatformIO

me-no-dev and others added 2 commits June 28, 2021 06:16

IDF master 8e3e65a47

b9ded1f

Update toolchain

f36c386

me-no-dev mentioned this pull request Jun 28, 2021

Add support for TinyUSB Arduino Library #5333

Closed

remove unnecessary include

2c1ffad

Update package_esp32_index.template.json

c61b1a3

VojtechBartoska added this to the 2.0.0 milestone Jul 14, 2021

Merge branch 'master' into idf-master

400b987

me-no-dev added 2 commits July 16, 2021 12:27

Merge branch 'master' into idf-master

a819990

Merge branch 'master' into idf-master

4f7fb13

me-no-dev and others added 10 commits July 16, 2021 17:25

Revert change to test lib-builder error

ce32e09

Revert change to CMakeLists

8573aab

Merge branch 'master' into idf-master

21cefed

Add rainmaker dependency

6624503

Fix tinyusb poisoned callback

113a043

Fix build error because strncpy tries to write to the same size buffer

8e4a2d2

IDF master d93887f9f

e0504f4

PlatformIO updates for CI (#5387)

cd94b04

* Update PlatformIO CI build script - Switch to the latest toolchains 8.4.0 for ESP32, ESP32S2, ESP32C3 - Use PlatformIO from master branch for better robustness * Update package.json for PlatformIO

Merge branch 'master' into idf-master

3ce7680

me-no-dev merged commit 16f4b0f into master Jul 16, 2021

me-no-dev deleted the idf-master branch July 16, 2021 22:57

VojtechBartoska added the Type: Internal workflow label Jul 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IDF master #5336

IDF master #5336

me-no-dev commented Jun 28, 2021

me-no-dev commented Jun 28, 2021

Jason2866 commented Jun 28, 2021

me-no-dev commented Jun 29, 2021

Jason2866 commented Jun 29, 2021 •

edited

Loading

me-no-dev commented Jun 29, 2021

ivankravets commented Jun 29, 2021

Jason2866 commented Jun 29, 2021

Jason2866 commented Jun 29, 2021

igrr commented Jun 29, 2021

me-no-dev commented Jun 29, 2021

me-no-dev commented Jun 29, 2021

ivankravets commented Jun 29, 2021

me-no-dev commented Jun 29, 2021

Jason2866 commented Jun 29, 2021 •

edited

Loading

me-no-dev commented Jun 29, 2021

me-no-dev commented Jun 30, 2021

Jason2866 commented Jun 30, 2021 •

edited

Loading

Jason2866 commented Jun 30, 2021

ivankravets commented Jul 1, 2021

Jason2866 commented Jul 14, 2021

Jason2866 commented Jul 15, 2021

IDF master #5336

IDF master #5336

Conversation

me-no-dev commented Jun 28, 2021

me-no-dev commented Jun 28, 2021

Jason2866 commented Jun 28, 2021

me-no-dev commented Jun 29, 2021

Jason2866 commented Jun 29, 2021 • edited Loading

me-no-dev commented Jun 29, 2021

ivankravets commented Jun 29, 2021

Jason2866 commented Jun 29, 2021

Jason2866 commented Jun 29, 2021

igrr commented Jun 29, 2021

me-no-dev commented Jun 29, 2021

me-no-dev commented Jun 29, 2021

ivankravets commented Jun 29, 2021

me-no-dev commented Jun 29, 2021

Jason2866 commented Jun 29, 2021 • edited Loading

me-no-dev commented Jun 29, 2021

me-no-dev commented Jun 30, 2021

Jason2866 commented Jun 30, 2021 • edited Loading

Jason2866 commented Jun 30, 2021

ivankravets commented Jul 1, 2021

Jason2866 commented Jul 14, 2021

Jason2866 commented Jul 15, 2021

Jason2866 commented Jun 29, 2021 •

edited

Loading

Jason2866 commented Jun 29, 2021 •

edited

Loading

Jason2866 commented Jun 30, 2021 •

edited

Loading