Move Over from the fbsimctl Needle to the simctl Face

Runtime Dump in Application Test Mode without Booting Simulator

Now Emcee allows you to speed up application test-like runtime dump. Previously, Emcee would boot the simulator and then launch your xctest bundle with fake test. While this mechanism is flexible, it is slow.

Now Emcee allows to perform runtime dump but without booting the simulator.

• Use new runtimeExecutableLaunch test discovery mode when specifying xcTestBundle build artifact in test arg file. You also must specify appBundle.
• Emcee will start start your app without booting simulator (this is magic!). main() will be invoked as usual.
• Inside your main() you should check for the environments:
  • if EMCEE_XCTEST_BUNDLE_PATH is provided, the value will contain a path to xctest bundle on local file system.
  • EMCEE_RUNTIME_TESTS_EXPORT_PATH will contain a path where to write out the resulting JSON with runtime dump, this is as usual
  • Either both envs will be provided, or both will be missing.
• If you detect EMCEE_XCTEST_BUNDLE_PATH, you can then load it using let bundle = Bundle(path: xctestBundlePath); bundle.load(). Then you'll be able to access your bundle's principal class, and perform the runtime dump as usual, and export it to EMCEE_RUNTIME_TESTS_EXPORT_PATH.

To use new way of discovering tests, pass testDiscoveryMode: "runtimeExecutableLaunch".

Xcode 11.4 Support

Emcee is now buildable in Xcode 11.4. This required some dancing around Ruby-lang based technologies like brewery@home, and some bugs in SPM. More in this thread.

Define the Location where Simulators should be Created

Emcee now allows to specify where to create simulators: either in its own private folder like it has been doing up to this point (insideEmceeTempFolder) or in default shared location ~/Library/Developer/CoreSimulator/Devices (insideUserLibrary). The latter allows to use the simulators with tools like xcodebuild and Xcode.app. This is required in order to start running tests using xcodebuild (which is still wasn't implmeneted though).

{
    "simulatorControlTool": {
        "location": "insideEmceeTempFolder",
        "tool": {
            "toolType": "fbsimctl",
            "location": "http://example.com/fbsimctl.zip"
        }
    }
}

As of now, you can safely use insideEmceeTempFolder to keep things unchanged.

simctl

Emcee had support for simctl for a while now, but now it has been tested and enabled for all builds in Avito. No problems so far, and you can switch to it as well. One less dependency to run the tests! In your arg file:

{
    "simulatorControlTool": {
        "location": "insideEmceeTempFolder",
        "tool": {
            "toolType": "simctl"
        }
    }
}

Speed for Need

Metrics

• Queue server now reports notRegistered worker state for those workers that were expected to register with queue but haven't done so.

Job Grouping

• --run-id argument has been renamed to --job-id

• Emcee allows to group jobs. runTestsOnRemoteQueue accepts new optional arguments: --job-group-id and --job-group-priority. Test queue sorts job groups by: job group priority, job group creation time (for job groups with same priority); jobs are sorted by priority and by job creation time (for jobs with same priority) within their groups.

Test Discovery

Runtime dump feature is now part (and implementation detail) of Test Discovery feature. Test arg file has been updated: runtimeDumpMode became testDiscoveryMode, logicTest and appTest were renamed to runtimeLogicTest and runtimeAppTest correspondingly, allProvidedByRuntimeDump became allDiscoveredTests.

Quicker but Unsafer Test Discovery

Emcee now has a new way of discovering tests. When you pass parseFunctionSymbols as a value of testDiscoveryMode in test arg file, Emcee will run nm -j -U on provided .xctest bundle, demangle all symbols using libswiftDemangle.dylib, and then find all @objc .test...() methods (throwable and not, with () return types).

This approach is much quicker than performing a runtime dump. This discovery method does not require any test source code modification.

There are some risks though. Emcee may find excess tests, or may miss some tests. Although in Avito we checked this for our 50 unit test bundles, we still might miss something, so use with care. Also, the lack of runtime support makes this test discovery method less flexible: Emcee won't be able to locate tests that you might be generating dynamically in runtime; Emcee won't export any data, e.g. tags, if you attach them to your tests. For those kind of things you must still use logic runtime dump (runtimeLogicTest) which is quite fast, or runtime with a host app (runtimeAppTest) and by defining NSPrincipalClass in your test bundle plists.

If you experience any inconsistency between parseFunctionSymbols and runtime dump based test discovery, please file an issue.

Totally Makes Sense

This release adds new features and tunes existing behavior. The changes totally make sense.

Automatic Simulator Deletion

This new feature allows to enable the following behavior: when simulators is automatically shutdown after being idle, and stays idle for a given period of time, Emcee will delete it to free up disk space. [commit]

Removal of Blocking Functionality

• Previously, Emcee workers reported to the queue their state with buckets they are executing. Now Emcee worker implemented REST, and the queue polls its workers instead. [commit]

• Emcee had a feature: if worker would switch from executing a bucket of tests to another bucket without sending back results in time, Emcee would consider these buckets as 'stuck' and re-schedule tests back to the queue; but then, if Emcee worker would send bucket results back to the queue, Emcee queue would decide that worker behaves incorrectly and block it since it sends back unexpected results. This blocking functionality has been removed. [commit]

Bug Fix & Clean Up

• Removed some models from Swift codebase

• Fixed a bug: queue has been starting workers twice, once for each deployed file. Now queue starts workers only once, when all files has been deployed to the worker machines. [commit]

• Emcee now expects a --version argument to be passed. This is rather a controversial way of versioning Emcee binary. If you update Emcee, you should update the value of this argument as well. [commit]

• Emcee does not create new simulators when you update fbxctest. [commit]

• Fixed a bug that would sometimes result to a corrupted cache state. When unzipping the downloaded artifacts, Emcee now will use temporary folder, and then move extracted data to a final location. [commit]

• SimulatorVideoRecorder class has been updated to work with Xcode 11. [commit]

• Emcee now passes TMPDIR environment to simulator control tool (e.g. fbsimctl) [commit]

Runtime Dump Cache, Simulator Timeout & Automatic Shutdown Configuration

Runtime Dump Cache

Emcee can now store results of runtime dump in a shared writable cache and re-use them to speed up runtime dumps on the same input. There are some requirements for this to work correctly:

• You must pass an additional config to support shared cache via --remote-cache-config arg. You can read more about its contents here.

• URL is used as a key to runtime dump cache entries. Thus, if you change the contents of xctest bundle behind some URL and attempt to dump it again, Emcee will re-use cached entry if it is available. This may lead to wrong or unexpected runtime dump results.

• Runtime dump should output exactly the same results regardless of execution environment or kind. iOS runtime, runtime dump kind, time of the day, or any environment variable values should not affect the result of runtime dump.

Simulator Timeout Configuration & Automatic Shutdown

Test arg file now expects to have simulatorOperationTimeouts object provided. This object allows you to control timeouts for various simulator operations like creating, booting, shutting down and deleting, as well as automatic shutdown. The latter allows you to specify the time after which idle simulators will be shutdown.

Shutdown simulators allow them to be booted faster than booting a newly created simulator, but reduces RAM, swap, and max_proc/max_files system resources utilization. This is useful for CI where you might be running tests on various iOS versions. Updating Emcee is also safer now because the amount of running simulators now decrease when simulators are idle.

Models Target has been Refactored

Models target previously contained a lot of models used by the Emcee. Now most of them moved to corresponding modules, like SimulatorPoolModels, RunnerModels, TestArgFile, and so on. Emcee exposes all model targets as EmceeInterfaces product for convenience.

Stability Increase

Emcee workers are much stable and do not crash when fetching contents of URLs (fix FYI)

simctl, Plugins Lifecycle

simctl

Emcee now correctly uses xcrun simctl to control simulators, making it unnecessary to provide a location of fbsimctl tool.

To use xcrun simctl, pass the following values via test arg file:

    "toolResources": {
        "simulatorControlTool": {
            "toolType": "simctl"
        },
    }

To switch between Xcode versions, provide developerDir value using toolchainConfiguration field in test arg file.

You can still use fbsimctl if you prefer it more. fbxctest tool is still required to run tests, support for xcrun xcoebuild is still in WIP status.

Plugins

Implemented plugin lifecycle. Previously, Emcee would start plugins on both server and workers, and you defined plugins as part of queue server run configuration. Now this has changed:

• You define plugins for a job using test arg file. This allows you to update plugins easily.
• Plugins are started before executing tests, and terminated afterwards.
• Plugins are expected to start within 30 seconds. Previously the timeout was 170 seconds (for unknown reason)

Bug fix

Fixed a nasty bug. Sometimes, when workers fetch a bucket for execution and then fail to execute it, they won't return it back to the queue. This would look like some worker took a bucket and keep executing it infinitely.

Now, if failure happens during bucket execution, worker will report back a test failure with a description of what happened.

Feature

Junit reports now have hostname and timestamp fields set for tests.

1.1.0

Simulator settings are now passed as data structures via test arg file instead of URLs to some JSON file with not very obvious contents.

These are the new models:

• SimulatorLocalizationSettings.swift

• WatchdogSettings.swift