ps-stream is a Python utility that collects and and parses PeopleSoft rowset-based messages generated by sync and fullsync services into Kafka messages and topics. PeopleSoft sync processes are normally used to sync data between PeopleSoft applications. However, they can also be used as a way to generate an externalized stream of PeopleSoft objects in streaming data pipelines.
There are two major commands in ps-stream.
collect accepts PeopleSoft rowset-based messages over http or https, and produces a Kafka message for each transaction in the PeopleSoft message, stored in one or more Kafka topics.
parse consumes transaction messages stored by collect and produces Kafka messages in topics with KTable semantics. Each record in the resulting stream is oriented to reflect records whose record key is the primary key or identifier of the transaction message.
Collect PeopleSoft sync messages.
$ docker run -p 8000:8000 -d ucalgary/ps-stream collect
You can quickly deploy ps-stream on Play with Docker, a community-run Docker playground, by clicking the following button.
After starting the demo, try POSTing a PeopleSoft rowset-based message to the collector on port 8000, then using kafka-topics-ui on port 8001 to see the message in a Kafka topic.
This repository and image are currently maintained by the Research Management Systems project at the University of Calgary.