Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[python] secure python package software supply chain to comply with partner org requirements #19401

Closed
7 tasks done
timothytrippel opened this issue Aug 9, 2023 · 0 comments · Fixed by #19531
Closed
7 tasks done

[python] secure python package software supply chain to comply with partner org requirements #19401

timothytrippel opened this issue Aug 9, 2023 · 0 comments · Fixed by #19531
Assignees
@timothytrippel
Labels
Component:CI Continuous Integration (Azure Pipelines & Co.) Component:Software Issue related to Software
Milestone
EarlGrey ES: M2.5.5

Comments

@timothytrippel
Copy link
Contributor

timothytrippel commented Aug 9, 2023 •
edited
Loading

Some partner organizations have security requirements for how python packages are installed. Namely:

  • all python packages (including transitive dependencies) should be specified in the python-requirements.txt file
  • python packages in python-requirements.txt should be pinned to specific versions (already done)
  • python packages in python-requirements.txt should have associated expected hashes
  • when downloading/installing python packages with pip install -r python-requirements.txt the --require-hashes flag should always be passed.

To comply with these requirements (needed to continue running regressions), we need to update how our python packages are declared and maintained in our repo. Specifically, we need to:

Addressed in lowRISC/fusesoc#4:

  • update the lowRISC fork of fusesoc to add the fallback_version to the use_scm_version field in setup.py (so pip-compile can generate the hash for this dep)

Addressed in #19531:

  • add the pip-tools package as a new dep, so we can run pip-compile --generate-hashes python-requirements.in to auto-generate the python-requirements.txt file we check-in to this repo so it contains:
    • all pinned versions,
    • all expected hashes,
    • for all deps (including transitive deps)
  • change the git VCS link references to fusesoc, edalize, and chipwhisperer packages to use plain HTTPS URLs to github hosted zip archives (so pip-compile can generate the hashes)
  • move the current python-requirements.txt file to python-requirements.in
  • autogenerate a python-requirements.txt file with pip-compile --generate-hashes python-requirements.in and check it into the repo
  • add a CI check to ensure the auto-generated python-requirements.txt file checked-in does not get stale
  • update website documentation on how to add python deps, i.e.,
    1. update the python-requirements.in file to add your dep (pinned to a specific version)
    2. run pip-compile --generate-hashes python-requirements.in to autogenerate the new python-requirements.txt file
    3. check both into the repo
    4. to install python packages securely, run pip install --require-hashes -r python-requirements.txt
@timothytrippel timothytrippel added Component:Software Issue related to Software Component:CI Continuous Integration (Azure Pipelines & Co.) labels Aug 9, 2023
@timothytrippel timothytrippel added this to the Discrete: M2.5.5 milestone Aug 9, 2023
@timothytrippel timothytrippel self-assigned this Aug 9, 2023
@timothytrippel timothytrippel mentioned this issue Aug 9, 2023
Merged
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[python] add python package hashes
a2a1db4 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
4. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
2d25ef4 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
83b4cda 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
This was referenced Aug 25, 2023
Closed
Merged
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[python] add python package hashes
92308ba 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
4. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
d3f0a0b 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
77a0505 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[python] add python package hashes
b79ad49 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
4. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
4b1ab80 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Aug 25, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
8fee847 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add python package hashes
89562bf 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
4. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
5956644 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
da27f83 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add python package hashes
d53e708 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. add `importlib-resources` dependency as this does not seem to be
   pinned by the `jsonschema` package and causes CI errors when pinning hashes,
4. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
5. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
b230614 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
a7b2697 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
7b15e0c 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
8ca3cae 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add python package hashes
9c93e44 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. add `importlib-resources` and `pkgutil_resolve_name` dependencies as these do
   not seem to be pinned by the `jsonschema` package and causes CI errors when
   pinning hashes,
4. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
5. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
2a7d8a7 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
c44c75e 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add python package hashes
12a227c 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. add `importlib-resources` and `pkgutil_resolve_name` dependencies as these do
   not seem to be pinned by the `jsonschema` package and causes CI errors when
   pinning hashes,
4. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
5. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
6744868 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 6, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
2dab0f2 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 7, 2023
@timothytrippel
[python] add python package hashes
a56769d 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. add `importlib-resources` and `pkgutil_resolve_name` dependencies as these do
   not seem to be pinned by the `jsonschema` package and causes CI errors when
   pinning hashes,
4. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
5. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 7, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
425ff73 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 7, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
24083a4 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 7, 2023
@timothytrippel
[python] add python package hashes
a089753 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. add `importlib-resources` and `pkgutil_resolve_name` dependencies as these do
   not seem to be pinned by the `jsonschema` package and causes CI errors when
   pinning hashes,
4. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
5. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 7, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
4a86631 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 7, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
e18ef86 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 13, 2023
@timothytrippel
[python] add python package hashes
b519f47 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. add `importlib-resources` and `pkgutil_resolve_name` dependencies as these do
   not seem to be pinned by the `jsonschema` package and causes CI errors when
   pinning hashes,
4. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
5. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 13, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
d75dd01 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses lowRISC#19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit to timothytrippel/opentitan that referenced this issue Sep 13, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
5082568 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit that referenced this issue Sep 14, 2023
@timothytrippel
[python] add python package hashes
8b17d1f 
To comply with partner organization python package supply chain
requirements we add hashes for all python packages (including all
transitive dependencies). To do so we:
1. move the existing `python-requirements.txt` file to
   `python-requirements.in`, as this will become the input to the tool
   (i.e., `pip-compile`) that generates the `python-requirements.txt` file
   we check in,
2. add `pip-tools` as a project dependency, it contains the
   `pip-compile` tool,
3. add `importlib-resources` and `pkgutil_resolve_name` dependencies as these do
   not seem to be pinned by the `jsonschema` package and causes CI errors when
   pinning hashes,
4. change the git VCS link references to fusesoc, edalize, and chipwhisperer
   packages to use plain HTTPS URLs to github hosted zip archives (so
   `pip-compile` can generate the hashes), and
5. autogenerate a `python-requirements.txt` file with `pip-compile
   --generate-hashes python-requirements.in` and check it into the repo.

This partially addresses #19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit that referenced this issue Sep 14, 2023
@timothytrippel
[python] add CI check for python-requirements.txt
4a8e217 
This adds a CI check to ensure the auto-generated `python-requirements.txt`
file checked-in does not get stale. This partially addresses #19401.

Signed-off-by: Tim Trippel <ttrippel@google.com>
timothytrippel added a commit that referenced this issue Sep 14, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
7cc4161 
To fix #19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
mundaym pushed a commit that referenced this issue Nov 13, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
6d1b22c 
To fix #19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
msfschaffner pushed a commit to msfschaffner/opentitan that referenced this issue Nov 13, 2023
@timothytrippel
[docs] add documentation for how to add a python package to the project
772bc1e 
To fix lowRISC#19401, we have a new process for adding Python packages to the
project.

Signed-off-by: Tim Trippel <ttrippel@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@timothytrippel timothytrippel

Labels
Component:CI Continuous Integration (Azure Pipelines & Co.) Component:Software Issue related to Software
Projects
None yet
Milestone
EarlGrey ES: M2.5.5
Development

Successfully merging a pull request may close this issue.

[python] add hashes to python-requirements.txt timothytrippel/opentitan
1 participant
@timothytrippel