Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhanced statistics, added hana driver, added query element based commit scope #7

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

bhuesemann
Copy link

  1. Statistics: added support for
  2. filtered executions count
  3. nested executions count
  4. processing start time (e.g., when query execution delivers first row and the row callback is called the first time)
  5. Commit semantics: added support define a commit sphere around a query element such that when the query element execution ends, all connection batches are flushed and committed. This is useful to control batched commits. For this change I had to extend the dtd specification and add a commit attribute (true|false) to the query element specification.
  6. Updated hsqldb library to current version (with enhanced functionality)

ToDo: complete regression testing because I've not configured the regression test automation tool yet

- filtered statements count (by if expression)
- nested execution statements count (count output rows of a query)
- processing start time for query and scripting elements (e.g. to
measure the time a long running query execution runs before delivering
the first row to scriptella
- configurable query element based commit for batched connections (e.g.
to set explicite commit after query element execution has ended to flush
containing (eventually batched) scripting elements. Thus following
non-batched connection can read the committed result.
@@ -28,6 +28,7 @@
public abstract class ScriptingElement extends XmlConfigurableBase {
private String connectionId;
private String ifExpr;
private String commit;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make it boolean. Update getter and setter

@ejboy
Copy link
Member

ejboy commented May 7, 2015

Thank you for your contribution! I apologize for the very late response. Unfortunately I have no time to work on Scriptella these days :(

Are you still interested in merging these changes? If so, I would like the above comments to be addressed. Also, the fact that unit tests are failing is quite concerning:
Tests in error:
scriptella.jdbc.NestedElementsPerfTest#testNoStat EtlExecutorException /etl/sc...
scriptella.execution.ExecutionStatisticsTest#test EtlExecutorException Locatio...

@bhuesemann
Copy link
Author

Many thanks for taking the time to look through my changes. Yes, I'll address your comments and will try to fix all points. Unfortunately it may take some time as I'm currently quite busy, too. But I'm really impressed by the crisp and clear design of the framework. I've successfully used it for a fully automated cross database replication framework that is selfgenerating the templates all 100% in scriptella logic. Next I'll add delta loading and bitemporal historisation and everything looks quite declarative and easy to maintain. I've also added a simple parallel execution scriptella driver (based on a NotifyingBlockingThreadPoolExecutor) that I may contribute at a later time. I know some of the existing ETL Tools (eg. Informatica) very well and Scriptella is not the silverbullet to replace those. But I see its value for the generation of standard ETL processes. I even began to POC it as a automated data comparision/testing framework but have no time to finish.

@bhuesemann bhuesemann mentioned this pull request May 24, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants