Development is using Visual Studio Code with the Devcontainer plugin. Once the code is cloned reopen using devcontainers and then run:
setup-devcontainer.shSetup the demo / test environment
task demo-startGenerate dummy data
task build-test-data-generator
task generate-test-dataThis demo masks customers data first and last name fields. By default it does not mask the customer Jane Smith to prove the masking is working on the other records. As the generate-test-data is random it could by chance not create a record for Jane Smith, if so then please check test-data/demoEvent.json (the data source) and update the Environment variable (UNMASKEDCUSTOMERS) at the top of _Taskfile.yml to not mask a person from the file of your choice.
Build and Deploy the transform
task deploy-demoBuild the data loader
task build-test-data-loaderIn a second terminal run one of the following:
rpk transform logs demo -fThen in the first terminal run one of the following:
task load-td-demoEventAt this point you should check the Redpanda console and look in the demo and output-demo topics. You should see that the source topic demo does not have its data masked whereas all records (except the Jane Smith records) are masked in the output-demo topic.
To clean up the demo
task demo-stopTo debug failing integration tests you can use the rpk tool to interegate the test Redpanda testcontainer instance when running. In order to do this do the following:
task create-rpk-test-profileYou may need to edit the profile and ports for each test run as these change dynamically. Do this by checking the test output for the IP/Port for each of the services then edit the test profile using the command:
rpk profile editOnce the profile is saved the rpk commands will work against the local testcontainer cluster.
The rpk tool uses profiles to identify which cluster to operate against, in order to switch profiles run:
rpk profile list
rpk profile use *PROFILE_NAME*