Second Tutorial Part 8: Object Processors

Previous	Main	Next
<-- Part 7	Main	Part 9 -->

All lookupWhere methods have 2 implementations. The first one which we have covered so far simply returns a list. The method can specify a start/size to enable looping and processing larger datasets. For data that is expected to be small (thousands of records), you can simply load all:

ListResults<TestQuestionAnswer_Data> L = lookupWhereForm(C, "Xyz", 0, -1);

For larger datasets, expecting hundreds of thousands of records maybe, you can then loop:

int size = 10_000;
int start = 0;
ListResults<TestQuestionAnswer_Data> L = TestQuestionAnswer_Factory.lookupWhereForm(C, "Xyz", start, size);
while (L.isEmpty() == false)
  {
    // ...
    // do stuff with L;
    // ...
    
    if (L.hasMore() == false)
     break;
    start+=size;
    L = TestQuestionAnswer_Factory.lookupWhereForm(C, "Xyz", start, size);
  }

But it will query the database and load all objects in a list, here up to 10,000 items long. If you have truly large datasets that need processing, that strategy can come short:

• You have to go back to the database every 10,000 records and reissue a new query every time, with a cursor to skip ahead etc...
• If you have millions of records to process, this can get problematic.

Of course, you can increase your count based on your needs, but even so, let's say you increase it to 50,000, my guess is that you have larger data needs anyways. The solution is to use an Object Processor in a "Visitor"-like pattern to process records one by one and likely never keep more than one record in memory at any given time.

private static class MyProcessor implements ObjectProcessor<TestQuestionAnswer_Data>
  {
    @Override
    public boolean process(int Index, TestQuestionAnswer_Data Obj)
    throws Exception
      {
        // ...
        // Do something with Data Object
        // ...
        return true; // or false if failure and need to abort.
      }
  };

private static void Test3(Connection C)
throws Exception
  {
    MyProcessor P = new MyProcessor ();
    TestQuestionAnswer_Factory.lookupWhereForm(C, P, "Xyz", 0, -1);
  }

In our experience, if the work you want to do doesn't fit in the first scenario (simply loading all records in memory), the third approach (a custom ObjectProcessor) only adds a few extra lines of java boilerplate code vs the loop approach, is less prone to bugs, provides a simpler API overall, and packages object-level processing logic in a nice reusable class etc... We have rarely used the second approach (looping).

🎈 NOTE: This approach extends also to your own custom queries. The runSelect method has a version that takes in an ObjectProcessor and ou can/should create a version of you custom lookup logic that takes in such a processor as well.

public static void lookupWhereComplexQuery
                   (Connection C, ObjectProcessor<TestQuestionAnswer_Data> P
                                , String typeSearchStr, int start, int size)
                   throws Exception
  {
    SelectQuery Q = newSelectQuery(C);
    if (TextUtil.isNullOrEmpty(typeSearchStr) == false)
      {
        String[] tokens = typeSearchStr.split("\\s+");
        List<String> L = new ArrayList<String>();
        for (String t : tokens)
          if (TextUtil.isNullOrEmpty(t) == false)
            L.add("%" + t + "%");
        Q.like(COLS.TYPE, CollectionUtil.toStringArray(L), false, true);
      }
    Q.and()
    .isNull(COLS.DELETED);
    runSelect(C, Q, P, start, size);
  }

Of course, you will want to factor out the repetitive code for maintainability.

protected static SelectQuery getSearchQuery(Connection C, String typeSearchStr)
throws Exception
  {
    SelectQuery Q = newSelectQuery(C);
    if (TextUtil.isNullOrEmpty(typeSearchStr) == false)
      {
        String[] tokens = typeSearchStr.split("\\s+");
        List<String> L = new ArrayList<String>();
        for (String t : tokens)
          if (TextUtil.isNullOrEmpty(t) == false)
            L.add("%" + t + "%");
        Q.like(COLS.TYPE, CollectionUtil.toStringArray(L), false, true);
      }
    Q.and()
    .isNull(COLS.DELETED);
    return Q;
  }

public static ListResults<TestQuestionAnswer_Data> lookupWhereComplexQuery(Connection C, String typeSearchStr, int start, int size)
throws Exception
  {
    return runSelect(C, getSearchQuery(C, typeSearchStr), start, size);
  }

public static void lookupWhereComplexQuery(Connection C, ObjectProcessor<TestQuestionAnswer_Data> P, String typeSearchStr, int start, int size)
throws Exception
  {
    runSelect(C, getSearchQuery(C, typeSearchStr), P, start, size);
  }

Previous	Main	Next
<-- Part 7	Main	Part 9 -->