Merge branch 'develop' v0.3.0

README.md

@@ -1,100 +1,27 @@
 #### asyncTest
- is an extension of SDK `ceylon.test` with following capabilities:
- * testing asynchronous multithread code
- * common initialization for a set of test functions
- * reporting several failures for a one particular test execution (test function)
- * marking each failure with `String` title
- * instantiating private classes and invoking private functions of tested module without additional dependecies in
+is an extension to SDK `ceylon.test` module with following capabilities:
+* testing asynchronous multithread code
+* common initialization for a set of test functions
+* storing initialized values on test context and retrieving them during test execution
+* executing tests concurrently or sequentially
+* parameterized testing
+* conditional test execution
+* multi-reporting, i.e. several failures or successes can be reported for a one particular test execution (test function)
+* reporting test results using charts (or graphs)
 
- The extension is based on:
- * `AsyncTestContext` interface which test function has to operate with (basically, reports on fails to).
- * `AsyncTestExecutor` class which satisfies `ceylon.test.engine.spi::TestExecutor` and used by `ceylon.test` module
-   to execute test functions.
- * `TestInitContext` interface and `init` annotation, which can be used for common initialization of test set.
+The module is available on [CeylonHerd](https://herd.ceylon-lang.org/modules/herd.asynctest)
 
 
-#### test procedure:
-1. Declare test function, which accepts `AsyncTestContext` as the first argument:
-
-         test void doTesting(AsyncTestContext context) {...}
-       
-  The other arguments have to be in accordance with `ceylon.test::parameters` annotation
-  or other satisfied `ceylon.test.engine.spi::ArgumentProvider`.  
-  Mark test function with `ceylon.test::test` annotation.
-2. Code test function according to `AsyncTestContext` specification:
-  * call `AsyncTestContext.start` before start testing
-  * perform testing and report on fails via `AsyncTestContext`
-  * call `AsyncTestContext.complete` to complete the testing
-3. Apply `ceylon.test::testExecutor` annotation:
-	* at module level to execute using `AsyncTestExecutor` every functions / classes
-	  marked with `test` annotation in the given module
-
-            testExecutor(`class AsyncTestExecutor`)   
-            native("jvm")   
-            module mymodule "1.0.0"
-	* at function level to execute using `AsyncTestExecutor` the given function only
-
-            test testExecutor(`class AsyncTestExecutor`)  
-            void doTesting(AsyncTestContext context) {...}  
-
-4. Run test in IDE or command line.
-
-#### initialization
-Common asynchronous initialization for a set of test functions can be performed
-by marking these functions with `init` annotation. Argument of this annotation is initializer function,
-which is called just once for all tests.    
-Initializer has to take first argument of `TestInitContext` type
-and other arguments as specified by `ceylon.test::parameters` annotation if marked with.    
-When initialization is completed `TestInitContext.proceed` has to be called.  
-If some error has occured and test has to be aborted, `TestInitContext.abort` can be called.  
-Initializer can store some values on context using `TestInitContext.put` method. These values can be retrieved lately
-by test functions using `AsyncTestContext.get` or `AsyncTestContext.getAll`.  
-Alternatively `init` annotation can be used at `class`, `package` or `module` level to apply initializer to all
-test functions of corresponding container.  
-> Initializer may not be marked with `init` annotation! Test function, package or module should be marked.
+#### Usage and documentation
 
-Example:
-
- 		// initialization parameters
- 		[String, Integer] serverParameters => ["host", 123]; 
-
- 		// initializer - binds to server specified by host:port,
- 		// if successful proceeds with test or aborted if some error occured
- 		parameters(`value serverParameters`)
- 		void setupServer(TestInitContext context, String host, Integer port) {
- 			Server server = Server();
- 			server.bind(host, port).onComplete (
- 				(Server server) {
- 					// storing server on context and notifying to continue with testing
- 					context.put("``host``:``port``", server, server.close); 
- 					context.proceed();
- 				},
- 				(Throwable err) {
- 					// abort initialization since server binding errored
- 					context.abort(err, "server ``host``:``port`` binding error");
- 				}
- 			);
- 		}
-
- 		// test functions, setupServer is called just once - nevertheless the actual number of test functions 
- 		test init(`function setupServer`) void firstTest AsyncTestContext context) {
- 			String serverName = serverParameters.host + ":``port``";
- 			assert ( exists server = context.get<Server>("serverName") );
- 			...
- 		}
- 		test init(`function setupServer`) void secondTest(AsyncTestContext context) {
- 			String serverName = serverParameters.host + ":``port``";
- 			assert ( exists server = context.get<Server>("serverName") );
- 			...
- 		}
-
-
-#### instantiating private classes
-`loadAndInstantiate` function is doing that.
+The extension is aimed to be run using Ceylon test tool.  
+See usage details in [API documentation](https://modules.ceylon-lang.org/repo/1/herd/asynctest/0.3.0/module-doc/api/index.html)
 
-#### invoking private functions
-`loadTopLevelFunction` function is doing that.
-
-#### see also
-[examples](examples/herd/examples/asynctest)  
-[Ceylon API docs](https://modules.ceylon-lang.org/repo/1/herd/asynctest/0.1.0/module-doc/api/index.html)  
+
+#### Examples
+
+* Test of [Fibonnachi numbers calculation](examples/herd/examples/asynctest/fibonnachi).
+  Calculation function is executed on separated thread and returns results using `ceylon.promise`.
+* [Time scheduler](examples/herd/examples/asynctest/scheduler) testing.
+* [Microbenchmark](examples/herd/examples/asynctest/mapperformance) -
+  comparative performance test of Ceylon / Java HashMap and TreeMap.

examples/herd/examples/asynctest/fibonnachi/package.ceylon

@@ -1,5 +1,5 @@
 "
- Testing of asynchronous calculation of Fibonnschi numbers.  
+ Testing of asynchronous calculation of Fibonnachi numbers.  
  Test is performed on [[asyncPositiveFibonnachiNumber]] using [[runFibonnachiTest]] test function.
  >Function [[positiveFibonnachiNumber]] returns incorrect results in order to demonstrate test framework output.
  "

examples/herd/examples/asynctest/mapperformance/MapWrapper.ceylon

@@ -0,0 +1,41 @@
+import ceylon.collection {
+
+	MutableMap
+}
+import java.util {
+
+	AbstractMap
+}
+
+
+"Interface used to wrap Ceylon and Java map to test both by a one code"
+interface MapWrapper<Key, Item> given Key satisfies Object {
+	shared formal void put( Key key, Item item );
+	shared formal Item? get( Key key );
+	shared formal void remove( Key key );
+	shared formal void clear();
+}
+
+
+"Ceylon map wrapper"
+class CeylonMapWrapper<Key, Item>( MutableMap<Key, Item> map )
+		satisfies MapWrapper<Key, Item>
+		given Key satisfies Object
+{
+	shared actual Item? get( Key key ) => map.get( key );
+	shared actual void put( Key key, Item item ) => map.put( key, item );
+	shared actual void remove( Key key ) => map.remove( key );
+	shared actual void clear() => map.clear();
+}
+
+
+"Java map wrapper"
+class JavaMapWrapper<Key, Item>( AbstractMap<Key, Item> map )
+		satisfies MapWrapper<Key, Item>
+		given Key satisfies Object
+{
+	shared actual Item? get( Key key ) => map.get( key );
+	shared actual void put( Key key, Item item ) => map.put( key, item );
+	shared actual void remove( Key key ) => map.remove( key );
+	shared actual void clear() => map.clear();
+}

examples/herd/examples/asynctest/mapperformance/chartMapTest.ceylon

@@ -0,0 +1,96 @@
+import herd.asynctest {
+
+	AsyncTestContext
+}
+import herd.asynctest.chart {
+
+	Plotter
+}
+
+
+"Runs map test and stores results in charts corresponding to [[mapTitle]].
+ The test is repeated [[repeats]] times and reported values are averaged by the number of repeats.  
+ After each repeat initial map is cleared and reused.
+ "
+Float[3] | Throwable chartMapTest (
+	"Context the test is run on." AsyncTestContext context,
+	"Total items in the map." Integer totalItems,
+	"Total number of test repeats." Integer repeats,
+	"Percent from total items -> items used in get / remove tests." Float removePercent,
+	"Title of map used to retrieve charts from context." String mapTitle,
+	"Map test is performed on." MapWrapper<String, Integer> map
+) {
+
+	"total number of items to be > 0"
+	assert( totalItems > 0 );
+	"attempts to be > 0"
+	assert( repeats > 0 );
+
+	String putPlotterName = titles.put + titles.titlesDelimiter + mapTitle;
+	String getPlotterName = titles.get + titles.titlesDelimiter + mapTitle;
+	String removePlotterName = titles.remove + titles.titlesDelimiter + mapTitle;
+
+	if ( exists putPlotter = context.get<Plotter>( putPlotterName ),
+		exists getPlotter = context.get<Plotter>( getPlotterName ),
+		exists removePlotter = context.get<Plotter>( removePlotterName )
+	) {
+		// test map
+
+		variable Integer count = -1;
+		variable Float sumPut = 0.0;
+		variable Float sumGet = 0.0;
+		variable Float sumRemove = 0.0;
+		variable Integer start = 0;
+		String prefix = "value";
+
+		// warming up
+		for ( upper in 0 : 100 ) {
+			for ( lower in 0 : 10000 ) {}
+		}
+
+		while ( count < repeats ) {
+			map.clear();
+
+			// test put
+			start = system.nanoseconds;
+			variable Integer putCount = 0; 
+			while ( putCount < totalItems ) {
+				map.put( prefix + putCount.string, putCount );
+				putCount ++;
+			}
+			if ( count > -1 ) { sumPut += ( system.nanoseconds - start ) / 1000000.0; }
+
+			// test get
+			value indexies = keyList( prefix, totalItems, removePercent );
+			start = system.nanoseconds;
+			for ( item in indexies ) {
+				map.get( item );
+			}
+			if ( count > -1 ) { sumGet += ( system.nanoseconds - start ) / 1000000.0; }
+
+			// test remove
+			start = system.nanoseconds;
+			for ( item in indexies ) {
+				map.remove( item );
+			}
+			if ( count > -1 ) { sumRemove += ( system.nanoseconds - start ) / 1000000.0; }
+
+			count ++;
+		}
+
+		// mean times
+		sumPut = sumPut / repeats;
+		sumGet = sumGet / repeats;
+		sumRemove = sumRemove / repeats;
+
+		// store tested data
+		putPlotter.addPoint( totalItems.float, sumPut );
+		getPlotter.addPoint( totalItems.float, sumGet );
+		removePlotter.addPoint( totalItems.float, sumRemove );
+
+		return [sumPut, sumGet, sumRemove];
+	}
+	else {
+		return Exception( "plotter with name 'XXX-``mapTitle``' doesn't exists" );
+	}
+}

examples/herd/examples/asynctest/mapperformance/fillTestResult.ceylon

@@ -0,0 +1,73 @@
+import herd.asynctest.chart {
+
+	Plotter
+}
+import herd.asynctest {
+
+	AsyncTestContext
+}
+
+
+"Fills results of testing to context.  
+ Returns `String` to complete test with."
+String fillTestResult (
+	"Context results to be filled in." AsyncTestContext context,
+	"Results of Ceylon test." Float[3] | Throwable ceylonResult,
+	"Results of Java test." Float[3] | Throwable javaResult,
+	"Target ratio Ceylon to Java." Float targetRatio,
+	"Total number of items in the map." Integer totalItems,
+	"Title of ratio chart." String ratioChartTitle
+) {
+
+	if ( is Throwable ceylonResult ) {
+		context.abort( ceylonResult );
+		return "";
+	}
+	else if ( is Throwable javaResult ) {
+		context.abort( javaResult );
+		return "";
+	}
+	else {
+		// Ceylon / Java ratios
+		Float putRatio = ceylonResult[0] / javaResult[0];
+		Float getRatio = ceylonResult[1] / javaResult[1];
+		Float removeRatio = ceylonResult[2] / javaResult[2];
+
+		// Formated string representation of ratios
+		String putRatioStr = formatFloat( putRatio, 2, 2 );
+		String getRatioStr = formatFloat( getRatio, 2, 2 );
+		String removeRatioStr = formatFloat( removeRatio, 2, 2 );
+
+		// fail the test if ratio is greater then target
+		String putMessage = "'put' Ceylon / Java ratio of ``putRatioStr`` is greater than target ``targetRatio``";
+		String putSucceed = "'put' Ceylon / Java ratio of ``putRatioStr`` is not greater than target ``targetRatio``";
+		context.assertFalse( putRatio > targetRatio, putMessage, putMessage, putSucceed );
+		String getMessage = "'get' Ceylon / Java ratio of ``getRatioStr`` is greater than target ``targetRatio``";
+		String getSucceed = "'get' Ceylon / Java ratio of ``getRatioStr`` is not greater than target ``targetRatio``";
+		context.assertFalse( getRatio > targetRatio, getMessage, getMessage, getSucceed );
+		String removeMessage = "'remove' Ceylon / Java ratio of ``removeRatioStr`` is greater than target ``targetRatio``";
+		String removeSucceed = "'remove' Ceylon / Java ratio of ``removeRatioStr`` is not greater than target ``targetRatio``";
+		context.assertFalse( removeRatio > targetRatio, removeMessage, removeMessage, removeSucceed );
+
+		String putPlotterName = ratioChartTitle + titles.titlesDelimiter + titles.put;
+		String getPlotterName = ratioChartTitle + titles.titlesDelimiter + titles.get;
+		String removePlotterName = ratioChartTitle + titles.titlesDelimiter + titles.remove;
+
+		if ( exists putPlotter = context.get<Plotter>( putPlotterName ),
+			exists getPlotter = context.get<Plotter>( getPlotterName ),
+			exists removePlotter = context.get<Plotter>( removePlotterName )
+		) {
+			putPlotter.addPoint( totalItems.float, putRatio );
+			getPlotter.addPoint( totalItems.float, getRatio );
+			removePlotter.addPoint( totalItems.float, removeRatio );
+		}
+		else {
+			context.abort( Exception( "plotter with name '``ratioChartTitle``-XXX' doesn't exists" ) );
+		}
+
+		// completing the test, argument will be used only if test is succeeded
+		return "all 'put' of ``putRatioStr``, 'get' of ``getRatioStr`` and 'remove' of ``removeRatioStr`` "
+				+ "Ceylon / Java ratios are less then target ``targetRatio``";
+	}
+
+}

examples/herd/examples/asynctest/mapperformance/keyList.ceylon

@@ -0,0 +1,7 @@
+
+
+"Returns a list of keys with total number of keys as percentage from max index."
+String[] keyList( String prefix, Integer max, Float percent ) {
+	Integer p = ( percent * max ).integer;
+	return [for ( index in 0 : p ) prefix + ( index / p * max ).string];
+}

examples/herd/examples/asynctest/mapperformance/package.ceylon

@@ -0,0 +1,25 @@
+"
+ Comparative Ceylon vs Java performance test on HashMap and TreeMap.  
+ 
+ #### Test functions:
+ * [[runHashMapTest]] - Ceylon vs Java HashMap test
+ * [[runTreeMapTest]] - Ceylon vs Java TreeMap test
+ 
+ #### Test strategy
+ 1. `String` keys and `Integer` items are used.
+ 2. Operations to be tested: `put`, `get`, `remove`.
+ 3. Given: total number of items in the map, percent of get / removed items.
+ 4. For the total number of items `put` operation is performed using `\"value\"+index.string` key and 'index' item.
+    Operation time is measured.
+ 5. For the percented number of items `get` operation is performed. Operation time is measured.
+ 6. For the percented number of items `remove` operation is performed. Operation time is measured.
+ 7. 4 - 6 are repeated for a given number of test repeats. Initial map is cleared and reused for the next repeat.
+ 8. 3 - 7 are repeated with another total number of items.
+ 
+ #### Output
+ Results of the test are reported using charts 'Spent Time vs Total Number Of Items'
+ and 'Ceylon to Java Time Ratio vs Total Number Of Items'.
+ See [[package herd.asynctest.chart]] with charts description.
+ 
+ "
+shared package herd.examples.asynctest.mapperformance;