Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pre-built firmware? #32

Open
tdamsma opened this issue Jun 28, 2019 · 34 comments
Open

Pre-built firmware? #32

tdamsma opened this issue Jun 28, 2019 · 34 comments

Comments

@tdamsma
Copy link

tdamsma commented Jun 28, 2019 •
edited
Loading

As there is no pre-built firmware available, and there are quite a few build dependencies, I think it would be nice to include that and lower the barrier to entry. If there is any interest, I would be willing to see if I can setup a CI pipeline to arrange that.

In the meantime, I have made a dockerfile that builds all dependencies and a firmware for the ESP32 based on this repo here. Building a formware is now as simple as

docker build -t pycopy https://gist.githubusercontent.com/tdamsma/b49359448924d7aa816d50be2170a610/raw/c12ffbf690fb56a5e9f327cdd2cb81b631db3d5d/Dockerfile
docker create --name pycopy pycopy
docker cp pycopy:/esp/micropython/ports/esp32/build-GENERIC/firmware.bin firmware.bin
docker rm pycopy

The only requirement is that you have docker installed (and a bit of patience)

Update:

The method above does not work anymore, a working dockerfile (as of 2020-07-16) with instructions can be found here: https://github.com/flusflas/micropython-builder
@tdamsma tdamsma changed the title Add instructions for building firmware Pre-built firmware? Jun 28, 2019
@pfalcon
Copy link
Owner

pfalcon commented Jun 30, 2019

Thanks for the post.

I think it would be nice to include that and lower the barrier to entry.

Note that I consider the target audience of my work for MicroPython, and this fork in particular, to be proponents and adherents of Open Source, i.e. folks to whom building from source is a natural thing, something they do all the time, or take as a useful and cherishing to learn. I definitely would like to provide binaries and "lower the barrier to entry", but not at the expense of being flooded and needing to provide support to too-beginners which aren't interested to learn things themselves. Finding a good balance isn't easy.

if I can setup a CI pipeline to arrange that.

If you mean to modify existing Travis CI pipeline so it published built binaries via Github Releases or via some other viable mechanism, that would be helpful/useful (and is in my todo list either, though with more things in current priority on top of it). One concern is the security implications of such setup, as I believe github permissions aren't detailed enough, and publishing also implies various other permissions. So, if you'd go for it, please be prepared to point to docs/elaborate on these points.

Thanks!

@tdamsma
Copy link
Author

tdamsma commented Jul 3, 2019

Understand you point on this being not for noobs, but looking at myself there are people interested in programming microchips with the latest new and shiny asyncio micropython features and libraries (and contributing to the libraries) but have no interest in actually developing micropython itself. I don't know C and could care less for compiler toolchains etc. I hate how setting up my computer (pre-docker at least) for one off builds based on lengthy readme messes up everything.

Thanks for your (and others) hard work for making it possible that a python programmer like myself can hobby with actual microprocessors without any C knowledge!

Will have a look at the CI stuff, just noticed something like this was set up upstream the other day

@pfalcon
Copy link
Owner

pfalcon commented Jul 5, 2019

Understand you point on this being not for noobs

I would not formulate it like that. MicroPython in general, and my subproject in particular, is for everyone who want to embrace and learn new things, being them a "pro" or a "noob". (Obviously, a "noob" would enjoy learning much more new things than a rusty "pro".)

@iBobik
Copy link

iBobik commented Aug 10, 2019

Btw I did something similar there: https://gitlab.com/janpoboril/micropython-docker-build

@iBobik
Copy link

iBobik commented Aug 13, 2019

I've updated it to support Pycopy: https://gitlab.com/janpoboril/micropython-docker-build/container_registry

@tdamsma
Copy link
Author

tdamsma commented Aug 14, 2019

great! Guess that everyone compiling their own with custom required modules installed is probably better than having just a prebuilt image anyways it would maybe suffice if the link to https://gitlab.com/janpoboril/micropython-docker-build/tree/master was made somewhat discoverable from the pycopy docs

@iBobik
Copy link

iBobik commented Aug 14, 2019 via email

@iBobik
Copy link

iBobik commented Aug 14, 2019

I've filed issue about optimization: https://gitlab.com/janpoboril/micropython-docker-build/issues/1

@BradenM
Copy link
Contributor

BradenM commented Nov 26, 2019

Thought you guys might like something I have been working on for a bit here. Its a collection of different micropython branches (official/pycopy/pycom) for a couple of different ports (unix/esp32/esp8266) all built automatically via a github actions workflow (seen here) which makes use of 3 different Dockerfiles (one for each port). After each version is built, its auto published to github packages for anyone to pull. See the list of packages. Documentation is pretty sparse right now, but looking at the workflow should provide a general idea. Wish I had seen micropython-docker-build before I started working on this though, definitely would have helped!

@maho
Copy link

maho commented Dec 30, 2019

@BradenM I would love to give it a try, but I have no idea, how to build esp32 firmware of pycopy. You could give some oneliners in README.md

@pfalcon
Copy link
Owner

pfalcon commented Jan 7, 2020

Since the release v2.12.0.3, binaries for unix port (Linux x86, x86-64) are provided: https://github.com/pfalcon/pycopy/releases

@maho
Copy link

maho commented Jan 7, 2020 •
edited
Loading

Unix port is relatively easy to build. The problem is with ESP32 firmware (and probably - esp8266 too).

There should be links either to prebuilt firmwares, or to less or more blessed solutions like dockerfiles

@pfalcon
Copy link
Owner

pfalcon commented Jan 7, 2020

Unix port is relatively easy to build.

That's already great news. And trust me, building any other port is no much complex than unix one ;-).

There should be links either to prebuilt firmwares, or to less or more blessed solutions like dockerfiles

I'm unable to provide support for (or even review) things outside the project. Contributions to the project are welcome. This kind of contribution would be based on existing parts of the project, like https://github.com/pfalcon/pycopy/blob/master/.travis.yml, which already builds esp8266, esp32, and bunch of other ports. Contribution guidelines, which are there to ensure that the contribution is actually helpful, and timely processed, are also available: https://github.com/pfalcon/pycopy/blob/master/CONTRIBUTING.md .

@psy0rz
Copy link

psy0rz commented Feb 3, 2020

Thanks for the post.

I think it would be nice to include that and lower the barrier to entry.

Note that I consider the target audience of my work for MicroPython, and this fork in particular, to be proponents and adherents of Open Source, i.e. folks to whom building from source is a natural thing, something they do all the time, or take as a useful and cherishing to learn. I definitely would like to provide binaries and "lower the barrier to entry", but not at the expense of being flooded and needing to provide support to too-beginners which aren't interested to learn things themselves. Finding a good balance isn't easy.

I see where you're coming from, but i'm a developer myself. I just want to get on with my project, and now i've spended a few hours with finding the "perfect" micropython for my ESP32 project.

  • "Vanilla" micropython: too incomplete, missing asyncio which i need for picoweb
  • Loboris micropython: Seems dead since 2018, but trying it now.
  • Pycopy: Seems very promising, but requires building myself. (im compiling it via provided docker image above right now. But i wont be surprised if its broken because something somewhere has changed in the meantime :)

I think setting up a good travis CI that build firmware can speed up development and actually reduce support requests: It helps if everyone uses "the standard" binary, so that we eliminate a bunch of factors that come into existence when everyone is compiling their own firmware.

I also helps verifying pull requests, because you can create an automatic "build/test" for every pull request. (I used this to streamline ESPEasy development) Now I understand if you dont have time to set it up yourself offcourse.

Project i'm working on: https://github.com/psy0rz/meowton/wiki . I want to provide a link to a page with a ESP32 binary and simple instructions, so people can build their own cat-scale. The project is already complicated enough, without adding the huge burden of compiling your own ESP32 firmware. :)

Edwin

@psy0rz
Copy link

psy0rz commented Feb 3, 2020 •
edited
Loading

@tdamsma , thanks, but broken by now:

Step 18/28 : RUN cd esp-idf && cat /tmp/ESP32-Makefile | grep "^ESPIDF_SUPHASH" | awk '{print $NF}' | xargs git checkout
 ---> Running in 3339400c214a
error: pathspec '310beae373446ceb9a4ad9b36b5428d7fdf2705f' did not match any file(s) known to git.

The other link you provided is esp8266 only. I would need ESP32 with PSRAM support.

@tdamsma
Copy link
Author

tdamsma commented Feb 3, 2020 •
edited
Loading

@psy0rz, check the builds @BradenM made: https://github.com/BradenM/micropy-build/actions
Direct link to the currently latest (6 days old) pycopy/esp32 build availabe here: https://github.com/BradenM/micropy-build/suites/425651893/artifacts/1408160

@psy0rz
Copy link

psy0rz commented Feb 3, 2020

oh cool! I looked around in @BradenM 's github, but I didnt know about a github Actions tab. :) thanks will check it out.

@tdamsma
Copy link
Author

tdamsma commented Feb 3, 2020

@psy0rz, I also updated the dockerfile so it works again.

@psy0rz
Copy link

psy0rz commented Feb 4, 2020

awesome, thanks!

@rvalle
Copy link

rvalle commented Mar 14, 2020 •
edited
Loading

@pfalcon New to this project, and micropython in general. Got lost a couple of days until finding out about the politics involved. I respect your points, but I think they are somehow detached from reality. I don't think there are many Linux kernel contributors whose first step was to compile the Linux Kernel apart of Linus himself. We all evaluate before deciding to contribute.

The reality is that we found out this project because Micropython falling behind and I am guessing they won't deliver much if they keep kicking out core contributors like you. It is obvious that pycopy is superior.

Giving the origing of Pycopy it is only fair to ask how is Pycopy dealing with its community. And I agree that barrier of entry is a key element.

I don't mean to imply that you should do it, or that you should support newcomers personally, I am only referring about pycopy success as an opensource project. Those elements need to be cooked into the project somehow. Entry barrier should be low and yes, there needs to be some kind of support for newbies thou most projects use a forum and newbies help each other.

luckily people like @tdamsma is contributing in this direction a docker builder image is a great step in this direction.

does anybody have a docker for ESP8266 port?

@pfalcon
Copy link
Owner

pfalcon commented Mar 14, 2020

I respect your points, but I think they are somehow detached from reality.

Nice to meet you too ;-). Which points exactly are "detached from reality" (and which reality do you mean)?

For reference, reality under which I work on Pycopy is described in this interview: https://www.blog.pythonlibrary.org/2020/02/10/pydev-of-the-week-paul-sokolovsky/ (the keyword is "human-scale computing").

It is obvious that pycopy is superior.

Quite a u-turn from the previous statement. In what sense do you think Pycopy is superior? For reference, Pycopy just tries to take the principle of the minimalism and layeredness more seriously than MicroPython.

Giving the origing of Pycopy it is only fair to ask how is Pycopy dealing with its community.

I keep the project open-source and invite interested parties to use it and/or contribute it, for as long our aims and approaches coincide.

most projects use a forum

I created a reddit, https://old.reddit.com/r/Pycopy/ , some time ago just in case. I never posted to it before, but now posted a couple of messages so it doesn't look empty.

@iBobik
Copy link

iBobik commented Mar 14, 2020 via email

@iBobik
Copy link

iBobik commented Mar 29, 2020

I've updated my builder to support v3:

https://gitlab.com/janpoboril/micropython-docker-build

@psy0rz
Copy link

psy0rz commented Mar 31, 2020

@BradenM should it build a new version on every push? It seems a few months behind now?

@BradenM
Copy link
Contributor

BradenM commented Mar 31, 2020

@psy0rz at the time I couldn't find a solid method to 'hook into' pushes from a different repository without the owner adding an integration or something... I probably could just set up some sort of polling subscription... maybe at some point haha.

Ill try to find some time to update it today and add a cron job to run it daily ;)

@psy0rz
Copy link

psy0rz commented Mar 31, 2020

ahhh i see :)

@iBobik
Copy link

iBobik commented Mar 31, 2020 via email

@psy0rz
Copy link

psy0rz commented Mar 31, 2020

@iBobik where to find the resulting files? I looked at https://gitlab.com/janpoboril/micropython-docker-build/container_registry/eyJuYW1lIjoiamFucG9ib3JpbC9taWNyb3B5dGhvbi1kb2NrZXItYnVpbGQvcHljb3B5L2VzcDgyNjYiLCJ0YWdzX3BhdGgiOiIvamFucG9ib3JpbC9taWNyb3B5dGhvbi1kb2NrZXItYnVpbGQvcmVnaXN0cnkvcmVwb3NpdG9yeS82Njk0NjEvdGFncz9mb3JtYXQ9anNvbiIsImlkIjo2Njk0NjF9

@iBobik
Copy link

iBobik commented Mar 31, 2020 via email

@psy0rz
Copy link

psy0rz commented Mar 31, 2020

"My builder on GitLab runs every week and builds for all tags on GitHub." (i hoped for us, online :))

@flusflas
Copy link

flusflas commented Jun 15, 2020 •
edited
Loading

As I didn't managed to build the last version of Pycopy for ESP32 using any of the above solutions (maybe I did something wrong), I created (another) repository for building it using Docker. Maybe it'll be helpful for someone.

https://github.com/flusflas/micropython-builder

@tdamsma
Copy link
Author

tdamsma commented Jun 15, 2020 via email

@tdamsma
Copy link
Author

tdamsma commented Jul 16, 2020

So I just had a look at https://github.com/flusflas/micropython-builder and it works flawlessly, nice work @flusflas. I'll edit that into the top comment. It would be nice to have an easily configurable way to include external libraries into the firmware. I managed for now by adjusting the dockerfile, but perhaps there is a nicer to make this configurable.

@iBobik
Copy link

iBobik commented Jul 16, 2020

It would be nice to have an easily configurable way to include external libraries into the firmware.

I do it my project's Makefile like this:

MICROPYTHON_IMAGE=registry.gitlab.com/janpoboril/micropython-docker-build/pycopy/esp8266:v3.0.6
FLASHING_PORT=cu.SLAB_USBtoUART

run-docker-build:
	docker run -it --rm -v ${PWD}:/app $(MICROPYTHON_IMAGE) make build-firmware

build-firmware:
	cp -R lib/* src/modules/* /micropython/ports/esp8266/modules
	cd /micropython/ports/esp8266 && make
	mv /micropython/ports/esp8266/build-GENERIC/firmware-combined.bin /app

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@rvalle @pfalcon @iBobik @psy0rz @maho @flusflas @BradenM @tdamsma