Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix #12436 - Migrate to pyproject.toml #14025

Merged
merged 13 commits into from
Nov 22, 2023
Merged

Fix #12436 - Migrate to pyproject.toml #14025

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
7e08113
test
pmbrull Nov 17, 2023
d802294
Use pyproject.toml
pmbrull Nov 19, 2023
c7f8059
Fix pylint
pmbrull Nov 19, 2023
209cf25
makefile
pmbrull Nov 19, 2023
b995023
makefile
pmbrull Nov 19, 2023
ee406af
Fix pylint
pmbrull Nov 19, 2023
e774f2d
isort
pmbrull Nov 19, 2023
6caeb1e
pyproject
pmbrull Nov 19, 2023
8624002
Airflow apis pyproject
pmbrull Nov 19, 2023
e0f79c1
Remove ingestion core
pmbrull Nov 19, 2023
0060e34
isort
pmbrull Nov 19, 2023
8beb7ac
Fix makefile help
pmbrull Nov 20, 2023
513edfd
Merge branch 'main' into issue-12436
pmbrull Nov 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/openmetadata-airflow-apis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,6 @@ jobs:
run: |
make install_dev install_apis
cd openmetadata-airflow-apis; \
python setup.py build sdist bdist_wheel; \
python -m build; \
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the main piece that was going to be deprecated

twine check dist/*; \
twine upload dist/* --verbose
49 changes: 0 additions & 49 deletions .github/workflows/py-ingestion-core-publish.yml
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/py-ingestion-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,6 @@ jobs:
sudo make install_antlr_cli
make install_dev generate install
cd ingestion; \
python setup.py build sdist bdist_wheel; \
python -m build; \
twine check dist/*; \
twine upload dist/* --verbose
48 changes: 7 additions & 41 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ include ingestion/Makefile

.PHONY: help
help:
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[35m%-30s\033[0m %s\n", $$1, $$2}'
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":"}; {printf "\033[35m%-35s\033[0m %s\n", $$2, $$3}'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to fix this. The changes from #13677 left the help command sending the wrong info


.PHONY: install_e2e_tests
install_e2e_tests: ## Install the ingestion module with e2e test dependencies (playwright)
Expand All @@ -24,40 +24,6 @@ yarn_install_cache: ## Use Yarn to install UI dependencies
yarn_start_dev_ui: ## Run the UI locally with Yarn
cd openmetadata-ui/src/main/resources/ui && yarn start

## Ingestion Core
.PHONY: core_install_dev
core_install_dev: ## Prepare a venv for the ingestion-core module
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleaning up the ingestion-core since we're not really making any use of it. We can recover the files if we have the need in the future

cd ingestion-core; \
rm -rf venv; \
python3 -m venv venv; \
. venv/bin/activate; \
python3 -m pip install ".[dev]"

.PHONY: core_clean
core_clean: ## Clean the ingestion-core generated files
rm -rf ingestion-core/src/metadata/generated
rm -rf ingestion-core/build
rm -rf ingestion-core/dist

.PHONY: core_generate
core_generate: ## Generate the pydantic models from the JSON Schemas to the ingestion-core module
$(MAKE) core_install_dev
mkdir -p ingestion-core/src/metadata/generated; \
. ingestion-core/venv/bin/activate; \
datamodel-codegen --input openmetadata-spec/src/main/resources/json/schema --input-file-type jsonschema --output ingestion-core/src/metadata/generated/schema
$(MAKE) core_py_antlr

.PHONY: core_bump_version_dev
core_bump_version_dev: ## Bump a `dev` version to the ingestion-core module. To be used when schemas are updated
$(MAKE) core_install_dev
cd ingestion-core; \
. venv/bin/activate; \
python -m incremental.update metadata --dev

.PHONY: core_py_antlr
core_py_antlr: ## Generate the Python core code for parsing FQNs under ingestion-core
antlr4 -Dlanguage=Python3 -o ingestion-core/src/metadata/generated/antlr ${PWD}/openmetadata-spec/src/main/antlr4/org/openmetadata/schema/*.g4

.PHONY: py_antlr
py_antlr: ## Generate the Python code for parsing FQNs
antlr4 -Dlanguage=Python3 -o ingestion/src/metadata/generated/antlr ${PWD}/openmetadata-spec/src/main/antlr4/org/openmetadata/schema/*.g4
Expand Down Expand Up @@ -171,7 +137,7 @@ generate-schema-docs: ## Generates markdown files for documenting the JSON Sche

#Upgrade release automation scripts below
.PHONY: update_all
update_all: ## To update all the release related files run make update_all RELEASE_VERSION=2.2.2 PY_RELEASE_VERSION=2.2.2.2
update_all: ## To update all the release related files run make update_all RELEASE_VERSION=2.2.2 PY_RELEASE_VERSION=2.2.2.2
@echo "The release version is: $(RELEASE_VERSION)" ; \
echo "The python metadata release version: $(PY_RELEASE_VERSION)" ; \
$(MAKE) update_maven ; \
Expand All @@ -184,15 +150,15 @@ update_all: ## To update all the release related files run make update_all RELEA
#make update_all RELEASE_VERSION=2.2.2 PY_RELEASE_VERSION=2.2.2.2

.PHONY: update_maven
update_maven: ## To update the common and pom.xml maven version
update_maven: ## To update the common and pom.xml maven version
@echo "Updating Maven projects to version $(RELEASE_VERSION)..."; \
mvn versions:set -DnewVersion=$(RELEASE_VERSION)
#remove comment and use the below section when want to use this sub module "update_maven" independently to update github actions
#make update_maven RELEASE_VERSION=2.2.2


.PHONY: update_github_action_paths
update_github_action_paths: ## To update the github action ci docker files
update_github_action_paths: ## To update the github action ci docker files
@echo "Updating docker github action release version to $(RELEASE_VERSION)... "; \
file_paths="docker/docker-compose-quickstart/Dockerfile \
.github/workflows/docker-openmetadata-db.yml \
Expand All @@ -212,7 +178,7 @@ update_github_action_paths: ## To update the github action ci docker files
#make update_github_action_paths RELEASE_VERSION=2.2.2

.PHONY: update_python_release_paths
update_python_release_paths: ## To update the setup.py files
update_python_release_paths: ## To update the setup.py files
file_paths="ingestion/setup.py \
openmetadata-airflow-apis/setup.py"; \
echo "Updating Python setup file versions to $(PY_RELEASE_VERSION)... "; \
Expand All @@ -223,7 +189,7 @@ update_python_release_paths: ## To update the setup.py files
#make update_python_release_paths PY_RELEASE_VERSION=2.2.2.2

.PHONY: update_dockerfile_version
update_dockerfile_version: ## To update the dockerfiles version
update_dockerfile_version: ## To update the dockerfiles version
@file_paths="docker/docker-compose-ingestion/docker-compose-ingestion.yml \
docker/docker-compose-openmetadata/docker-compose-openmetadata.yml \
docker/docker-compose-quickstart/docker-compose-postgres.yml \
Expand All @@ -236,7 +202,7 @@ update_dockerfile_version: ## To update the dockerfiles version
#make update_dockerfile_version RELEASE_VERSION=2.2.2

.PHONY: update_ingestion_dockerfile_version
update_ingestion_dockerfile_version: ## To update the ingestion dockerfiles version
update_ingestion_dockerfile_version: ## To update the ingestion dockerfiles version
@file_paths="ingestion/Dockerfile \
ingestion/operators/docker/Dockerfile"; \
echo "Updating ingestion dockerfile release version to $(PY_RELEASE_VERSION)... "; \
Expand Down
31 changes: 0 additions & 31 deletions ingestion-core/README.md
View file
Open in desktop

This file was deleted.

56 changes: 0 additions & 56 deletions ingestion-core/setup.py
View file
Open in desktop

This file was deleted.

11 changes: 0 additions & 11 deletions ingestion-core/src/metadata/_version.py
View file
Open in desktop

This file was deleted.

11 changes: 0 additions & 11 deletions ingestion/.coveragerc
View file
Open in desktop

This file was deleted.

35 changes: 16 additions & 19 deletions ingestion/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ install_apis: ## Install the REST APIs module to the current environment
python -m pip install $(ROOT_DIR)/openmetadata-airflow-apis/

.PHONY: lint
lint: ## Run pylint on the Python sources to analyze the codebase
PYTHONPATH="${PYTHONPATH}:$(INGESTION_DIR)/plugins" find $(PY_SOURCE) -path $(PY_SOURCE)/metadata/generated -prune -false -o -type f -name "*.py" | xargs pylint
lint: ## Run pylint on the Python sources to analyze the codebase
PYTHONPATH="${PYTHONPATH}:$(INGESTION_DIR)/plugins" find $(PY_SOURCE) -path $(PY_SOURCE)/metadata/generated -prune -false -o -type f -name "*.py" | xargs pylint --rcfile=$(INGESTION_DIR)/pyproject.toml

.PHONY: precommit_install
precommit_install: ## Install the project's precommit hooks from .pre-commit-config.yaml
Expand All @@ -41,32 +41,32 @@ precommit_install: ## Install the project's precommit hooks from .pre-commit-co

.PHONY: py_format
py_format: ## Run black and isort to format the Python codebase
pycln $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --extend-exclude $(PY_SOURCE)/metadata/generated --all
isort $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --skip $(PY_SOURCE)/metadata/generated --skip $(INGESTION_DIR)/env --skip $(INGESTION_DIR)/build --skip $(ROOT_DIR)/openmetadata-airflow-apis/build --profile black --multi-line 3
black $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --extend-exclude $(PY_SOURCE)/metadata/generated
pycln $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --config $(INGESTION_DIR)/pyproject.toml
isort $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --settings-file $(INGESTION_DIR)/pyproject.toml
black $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --config $(INGESTION_DIR)/pyproject.toml

.PHONY: py_format_check
py_format_check: ## Check if Python sources are correctly formatted
pycln $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --diff --extend-exclude $(PY_SOURCE)/metadata/generated --all
isort --check-only $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --skip $(PY_SOURCE)/metadata/generated --skip $(INGESTION_DIR)/build --skip $(ROOT_DIR)/openmetadata-airflow-apis/build --profile black --multi-line 3
black --check --diff $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --extend-exclude $(PY_SOURCE)/metadata/generated
PYTHONPATH="${PYTHONPATH}:$(INGESTION_DIR)/plugins" pylint --fail-under=10 $(PY_SOURCE)/metadata --ignore-paths $(PY_SOURCE)/metadata/generated || (echo "PyLint error code $$?"; exit 1)
pycln $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --diff --config $(INGESTION_DIR)/pyproject.toml
isort --check-only $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --settings-file $(INGESTION_DIR)/pyproject.toml
black --check --diff $(INGESTION_DIR)/ $(ROOT_DIR)/openmetadata-airflow-apis/ --config $(INGESTION_DIR)/pyproject.toml
PYTHONPATH="${PYTHONPATH}:$(INGESTION_DIR)/plugins" pylint --rcfile=$(INGESTION_DIR)/pyproject.toml --fail-under=10 $(PY_SOURCE)/metadata --ignore-paths $(PY_SOURCE)/metadata/generated || (echo "PyLint error code $$?"; exit 1)

.PHONY: unit_ingestion
unit_ingestion: ## Run Python unit tests
coverage run --rcfile $(INGESTION_DIR)/.coveragerc -a --branch -m pytest -c $(INGESTION_DIR)/setup.cfg --junitxml=$(INGESTION_DIR)/junit/test-results-unit.xml --ignore=$(INGESTION_DIR)/tests/unit/source $(INGESTION_DIR)/tests/unit
coverage run --rcfile $(INGESTION_DIR)/pyproject.toml -a --branch -m pytest -c $(INGESTION_DIR)/pyproject.toml --junitxml=$(INGESTION_DIR)/junit/test-results-unit.xml --ignore=$(INGESTION_DIR)/tests/unit/source $(INGESTION_DIR)/tests/unit

## Ingestion tests & QA
.PHONY: run_ometa_integration_tests
run_ometa_integration_tests: ## Run Python integration tests
coverage run --rcfile $(INGESTION_DIR)/.coveragerc -a --branch -m pytest -c $(INGESTION_DIR)/setup.cfg --junitxml=$(INGESTION_DIR)/junit/test-results-integration.xml $(INGESTION_DIR)/tests/integration/ometa $(INGESTION_DIR)/tests/integration/orm_profiler $(INGESTION_DIR)/tests/integration/test_suite $(INGESTION_DIR)/tests/integration/data_insight $(INGESTION_DIR)/tests/integration/lineage
coverage run --rcfile $(INGESTION_DIR)/pyproject.toml -a --branch -m pytest -c $(INGESTION_DIR)/pyproject.toml --junitxml=$(INGESTION_DIR)/junit/test-results-integration.xml $(INGESTION_DIR)/tests/integration/ometa $(INGESTION_DIR)/tests/integration/orm_profiler $(INGESTION_DIR)/tests/integration/test_suite $(INGESTION_DIR)/tests/integration/data_insight $(INGESTION_DIR)/tests/integration/lineage

.PHONY: run_python_tests
run_python_tests: ## Run all Python tests with coverage
coverage erase
$(MAKE) unit_ingestion
$(MAKE) run_ometa_integration_tests
coverage report --rcfile $(INGESTION_DIR)/.coveragerc || true
coverage report --rcfile $(INGESTION_DIR)/pyproject.toml || true

.PHONY: sonar_ingestion
sonar_ingestion: ## Run the Sonar analysis based on the tests results and push it to SonarCloud
Expand All @@ -82,20 +82,17 @@ sonar_ingestion: ## Run the Sonar analysis based on the tests results and push
.PHONY: run_apis_tests
run_apis_tests: ## Run the openmetadata airflow apis tests
coverage erase
coverage run --rcfile $(ROOT_DIR)/openmetadata-airflow-apis/.coveragerc -a --branch -m pytest --junitxml=$(ROOT_DIR)/openmetadata-airflow-apis/junit/test-results.xml $(ROOT_DIR)/openmetadata-airflow-apis/tests
coverage report --rcfile $(ROOT_DIR)/openmetadata-airflow-apis/.coveragerc

coverage run --rcfile $(ROOT_DIR)/openmetadata-airflow-apis/pyproject.toml -a --branch -m pytest -c $(INGESTION_DIR)/pyproject.toml --junitxml=$(ROOT_DIR)/openmetadata-airflow-apis/junit/test-results.xml $(ROOT_DIR)/openmetadata-airflow-apis/tests
coverage report --rcfile $(ROOT_DIR)/openmetadata-airflow-apis/pyproject.toml

.PHONY: coverage_apis
coverage_apis: ## Run the python tests on openmetadata-airflow-apis
$(MAKE) run_apis_tests
coverage xml --rcfile $(ROOT_DIR)/openmetadata-airflow-apis/.coveragerc -o $(ROOT_DIR)/openmetadata-airflow-apis/coverage.xml
coverage xml --rcfile $(ROOT_DIR)/openmetadata-airflow-apis/pyproject.toml -o $(ROOT_DIR)/openmetadata-airflow-apis/coverage.xml
sed -e "s/$(shell python -c "import site; import os; from pathlib import Path; print(os.path.relpath(site.getsitepackages()[0], str(Path.cwd())).replace('/','\/'))")\///g" $(ROOT_DIR)/openmetadata-airflow-apis/coverage.xml >> $(ROOT_DIR)/openmetadata-airflow-apis/ci-coverage.xml



.PHONY: coverage
coverage: ## Run all Python tests and generate the coverage XML report
$(MAKE) run_python_tests
coverage xml --rcfile $(INGESTION_DIR)/.coveragerc -o $(INGESTION_DIR)/coverage.xml || true
coverage xml --rcfile $(INGESTION_DIR)/pyproject.toml -o $(INGESTION_DIR)/coverage.xml || true
sed -e "s/$(shell python -c "import site; import os; from pathlib import Path; print(os.path.relpath(site.getsitepackages()[0], str(Path.cwd())).replace('/','\/'))")/src/g" $(INGESTION_DIR)/coverage.xml >> $(INGESTION_DIR)/ci-coverage.xml
Loading
Loading