Merge pull request #29 from yanj-github/main

Release v1.0.1

README.md

@@ -82,7 +82,7 @@ The camera's requirements are:
 ### Recording environment set up
 The set up needs to be in a light-controlled environment and the camera configured to record high quality footage to allow consistent QR code detection. **It is highly unlikely that simply setting up the equipment on a desk in a standard office environment will produce satisfactory results!**
 
-**More detailed guidance, and example videos are contained in "how_to_take_clear_recordings.pptx", available to download from https://dash-large-files.akamaized.net/WAVE/assets/How-to-take-clear-recordings-v2.pptx.zip .**
+**More detailed guidance, and example videos are contained in "how_to_take_clear_recordings.pptx", available to download from https://dash-large-files.akamaized.net/WAVE/assets/YanJiang-how_to_take_clear_recordings.pptx.zip .**
 
 For the initial set up, in Test Runner select and run the "*/avc/sequential-track-playback__stream__.html*" test. (See Test Runner documentation for how to run tests: https://github.com/cta-wave/dpctf-test-runner and https://web-platform-tests.org/running-tests/ ).
 
@@ -95,6 +95,9 @@ For the camera/device set up:
 * The device's screen brightness needs to be adjusted to be neither too bright nor too dim. Too dim and the QR code cannot be discerned. But too bright and the white will "bleed" and prevent the QR code being recognised. See below for some examples.
 * Depending on the device and sofware being used to run the tests, some device/software specific configuration may be required. For e.g. by default some browsers may add menu footers or headers that could partially obscure the QR codes. These will need to be set into e.g. a "full screen" mode. If any part of a QR code is obscured then the Observation Framework cannot operate.
 
+Note: Minimizing time between the start of recording and when the pre-test QR code shows up 
+helps Device Observation Framework to process faster and give test results quicker.
+
 ### **Clear capture example:**
 
 ![image](images/good_capture_example.png)
@@ -115,13 +118,20 @@ To run DPCTF Device Observation Framework enter:
 
 ```shell
 cd device-observation-framework
-python observation_framework.py --input <file>
+
+python observation_framework.py --input <file> --log <info|debug> --scan <general|intensive>
+
 OR
-python3 observation_framework.py --input <file>
+
+python3 observation_framework.py --input <file> --log <info|debug> --scan <general|intensive>
 
 (n.b. Python version must be 3.6 or greater)
 ```
 
+where **log** (optional) specifies log level. Default value is "info" when not specified. See "Additional Options" section below for more details.
+
+where **scan** (optional) specifies scan method to be used. Default value is "general" when not specified. See "Additional Options" section below for more details.
+
 where **file** specifies the path to the recording to be analysed:
 
 **If the session is recorded to a single file**
@@ -136,12 +146,21 @@ alphabetically sort the filenames and use this as the recording order. If for a
   If both these approaches fail then the user will need to rename the files such that when alphabetically
   sorted they are in the correct order.
 
-
 The Observation Framework will analyse the recording and post the results to the Test Runner for viewing on Test Runner's results pages. Note that the observation processing may take considerable time, depending on the duration of the session recording.
 
+When a selected test includes an observation *"The presented sample matches the one reported by the currentTime value within the tolerance of the sample duration."*, a CSV file contains observation data will be generated at ```logs/<session-id>/<test-path>_time_diff.csv```. Where is contains current times and calculated time differences between current time and media time.
+
 At the end of the process the Observation Framework will rename the file recording to:
 ```<file_name>_dpctf_<session-id>.<extension>```
 
+**Additional Options:**
+* When ```--log debug``` is selected, full QR code detection will be extracted to a CSV file at ```logs/<session-id>/qr_code_list.csv```, and the system displays more information to the terminal as well as to a log file ```logs/<session-id>/session.log```.
+This includes information such as decoding of QR codes:
+    * Content ID, Media Time, Frame Number, Frame Rate
+    * Status, Action, Current Time, Delay
+
+* ```--scan intensive``` makes the QR code recognition more robust by allowing an additional adpativethresholded scan, however this will increase prossing time. This option is to be used where it is difficult to take clear repcordings, such as testing on a small screen devices like mobile phones.
+
 ## Troubleshooting
 
 ### Http connection exception raised:

config.ini

@@ -35,18 +35,30 @@ session_log_threshold = 100
 # If the number of missing frames on an individual test is greater than this
 # then posts an error result and terminates the session.
 # Set to 0 to disable the feature.
-missing_frame_threshold = 50
+missing_frame_threshold = 0
+# If number of consecutive camera frames has no qr code
+# then posts an error result and terminates the session.
+# This value is in number of mezzanine frames
+# corespondent camera framecan be calculated by
+# camera_frame_rate/mezzanine_frame_rate * consecutive_no_qr_threshold
+# Set to 0 to disable the feature.
+consecutive_no_qr_threshold = 4
 # session timeout from receiving the status ended QR code
 end_of_session_timeout = 10
 # check for no QR code is detected
 # if timeout is exceeded then session is ended and observation framework terminates
 no_qr_code_timeout = 5
+# serach qr area in seconds, where the search end to
+# 0 to disable search
+search_qr_area_to = 60
+# margin to add around the detected area to crop
+qr_area_margin = 50
 # system mode is for development purposes only
 #system_mode = Debug
 
 [TOLERANCES]
-start_frame_num_tolerance = 1
+start_frame_num_tolerance = 0
 end_frame_num_tolerance = 0
-mid_frame_num_tolerance = 0
+mid_frame_num_tolerance = 10
 splice_start_frame_num_tolerance = 0
 splice_end_frame_num_tolerance = 0

configuration_parser.py

@@ -25,12 +25,14 @@
 Licensor: Consumer Technology Association
 Contributor: Eurofins Digital Product Testing UK Limited
 """
+from exceptions import ConfigError
 import logging
 from typing import Dict
 
 import requests
 import isodate
 import json
+import sys
 from global_configurations import GlobalConfigurations
 
 
@@ -73,18 +75,56 @@ def parse_tests_json(self, test_id: str):
 
             return test_path, test_code
         except KeyError as e:
-            raise Exception(e)
+            raise ConfigError(
+                f"Unrecognised test id is detected. "
+                f"Detected test id({test_id}) is not defined in \"tests.json\". "
+            )
+
 
     def parse_tests_json_content_config(self, parameters: list, test_path: str) -> dict:
         """parse content related config parameters for current test"""
         parameters_dict = {}
 
         for parameter in parameters:
             if parameter == "period_duration":
-                # TODO: hardcode this for now until the configuration is avalible on TR
+                # TODO: hardcode this for now until the configuration is available on TR
                 # assume the period_duration 2 or 3 if not raise exception here
                 parameters_dict[parameter] = [20000, 20000, 20000]
                 # parameters_dict[parameter] = [20000, 20000]
+            elif parameter == "fragment_duration":
+                # fragment_duration_list is used for test when only one representation is used
+                # for example: sequential playback
+                for representation in self.test_config["representations"].values():
+                    if representation["type"] == "video":
+                        try:
+                            if representation["fragment_duration"] == None:
+                                raise TypeError
+                            # the multiplication happens so that we get the fragment duration in ms
+                            parameters_dict[parameter] = representation["fragment_duration"] * 1000
+                            # we are interested just about the first video representation's fragment duration
+                            break
+                        except (TypeError, KeyError) as e:
+                            raise ConfigError(
+                                f"Failed to get a parameter:{e} for the test '{test_path}'"
+                            )
+            elif parameter == "fragment_duration_list":
+                # fragment_duration_list is used for tests when more then one representation is used
+                # for example: switching set
+                # this list is needed to identify the switching points, and to calculate durations
+                parameters_dict[parameter] = {}
+                counter = 1
+                for representation in self.test_config["representations"].values():
+                    if representation["type"] == "video":
+                        try:
+                            if representation["fragment_duration"] == None:
+                                raise TypeError
+                            # the multiplication happens so that we get the fragment duration in ms
+                            parameters_dict[parameter][counter] = representation["fragment_duration"] * 1000
+                            counter += 1
+                        except (TypeError, KeyError) as e:
+                            raise ConfigError(
+                                f"Failed to get a parameter:{e} for the test '{test_path}'"
+                            )
             else:
                 try:
                     config_value = isodate.parse_duration(self.test_config[parameter])
@@ -93,7 +133,7 @@ def parse_tests_json_content_config(self, parameters: list, test_path: str) -> d
                     value = ms + s_to_ms
                     parameters_dict[parameter] = value
                 except KeyError as e:
-                    raise Exception(
+                    raise ConfigError(
                         f"Failed to get a parameter:{e} for the test '{test_path}'"
                     )
 
@@ -118,7 +158,7 @@ def parse_test_config_json(
                         value = self.test_config_json["all"][parameter]
                         parameters_dict[parameter] = value
                     except KeyError as e:
-                        raise Exception(
+                        raise ConfigError(
                             f"Failed to get a parameter:{e} for the test '{test_path}'"
                         )
 
@@ -132,11 +172,11 @@ def _get_json_from_tr(self, json_name: str) -> Dict[str, Dict[str, Dict[str, str
                 config_data = r.json()
                 return config_data
             else:
-                raise Exception(
+                raise ConfigError(
                     f"Error: Failed to get configuration file from test runner {r.status_code}"
                 )
         except requests.exceptions.RequestException as e:
-            raise Exception(e)
+            raise ConfigError(e)
 
     def _get_json_from_local(
         self, json_name: str
@@ -150,4 +190,4 @@ def _get_json_from_local(
                 config_data = json.load(json_file)
                 return config_data
         except requests.exceptions.RequestException as e:
-            raise Exception(e)
+            raise ConfigError(e)

dpctf_qr_decoder.py

@@ -41,6 +41,8 @@
 class MezzanineDecodedQr(DecodedQr):
     data: str
     """qr code string"""
+    location: list
+    """qr code location"""
 
     """A decoded QR code from Mezzanine content
     ID;HH:MM:SS.MMM;<frame #>;<frame-rate>
@@ -62,14 +64,16 @@ class MezzanineDecodedQr(DecodedQr):
     def __init__(
         self,
         data: str,
+        location: list,
         content_id: str,
         media_time: float,
         frame_number: int,
         frame_rate: float,
         camera_frame_num: int,
     ):
-        super().__init__(data)
+        super().__init__(data, location)
         self.data = data
+        self.location = location
         self.content_id = content_id
         self.media_time = media_time
         self.frame_number = frame_number
@@ -81,6 +85,8 @@ def __init__(
 class TestStatusDecodedQr(DecodedQr):
     data: str
     """ qr code string"""
+    location: list
+    """qr code location"""
 
     """A decoded QR code for Test Runner status
     QR code in json format contain following info
@@ -96,14 +102,16 @@ class TestStatusDecodedQr(DecodedQr):
     def __init__(
         self,
         data: str,
+        location: list,
         status: str,
         last_action: str,
         current_time: float,
         delay: int,
         camera_frame_num: int,
     ):
-        super().__init__(data)
+        super().__init__(data, location)
         self.data = data
+        self.location = location
         self.status = status
         self.last_action = last_action
         self.current_time = current_time
@@ -114,6 +122,8 @@ def __init__(
 class PreTestDecodedQr(DecodedQr):
     data: str
     """ qr code string"""
+    location: list
+    """qr code location"""
 
     """A decoded QR code for pre test
     QR code in json format contain following info
@@ -125,23 +135,25 @@ class PreTestDecodedQr(DecodedQr):
     """test id encoded in the test runner QR code.
     """
 
-    def __init__(self, data: str, session_token: str, test_id: str):
-        super().__init__(data)
+    def __init__(self, data: str, location: list, session_token: str, test_id: str):
+        super().__init__(data, location)
         self.data = data
+        self.location = location
         self.session_token = session_token
         self.test_id = test_id
 
 
 class DPCTFQrDecoder(QrDecoder):
     def _translate_qr_test_runner(
-        self, data: str, json_data, camera_frame_num: int
+        self, data: str, location: list, json_data, camera_frame_num: int
     ) -> DecodedQr:
         """translate different type of test runner qr code"""
-        code = DecodedQr("")
+        code = DecodedQr("", [])
 
         try:
             code = TestStatusDecodedQr(
                 data,
+                location,
                 json_data["s"],
                 json_data["a"],
                 float(json_data["ct"]),
@@ -152,6 +164,7 @@ def _translate_qr_test_runner(
             try:
                 code = TestStatusDecodedQr(
                     data,
+                    location,
                     json_data["s"],
                     json_data["a"],
                     0,
@@ -162,16 +175,20 @@ def _translate_qr_test_runner(
                 try:
                     code = TestStatusDecodedQr(
                         data,
+                        location,
                         json_data["s"],
                         json_data["a"],
                         0,
                         0,
-                        camera_frame_num
+                        camera_frame_num,
                     )
                 except Exception:
                     try:
                         code = PreTestDecodedQr(
-                            data, json_data["session_token"], json_data["test_id"]
+                            data,
+                            location,
+                            json_data["session_token"],
+                            json_data["test_id"],
                         )
                     except Exception:
                         logger.error(f"Unrecognised QR code detected: {data}")
@@ -196,22 +213,25 @@ def _media_time_str_to_ms(self, media_time_str: str) -> float:
 
         return media_time
 
-    def translate_qr(self, data: str, camera_frame_num: int) -> DecodedQr:
+    def translate_qr(
+        self, data: str, location: list, camera_frame_num: int
+    ) -> DecodedQr:
         """Given a QR code as reported by pyzbar, parse the data and convert it to
         the format we use.
 
         Returns the translated QR code, or None if it's not a valid QR code.
 
         Mezzanine QR code is higher priority and test status than the start test QR code.
         """
-        code = DecodedQr("")
+        code = DecodedQr("", [])
 
         match = _mezzanine_qr_data_re.match(data)
         if match:
             # matches a mezzanine signature so decode it as such
             media_time = self._media_time_str_to_ms(match.group(2))
             code = MezzanineDecodedQr(
                 data,
+                location,
                 match.group(1),
                 media_time,
                 int(match.group(3)),
@@ -221,8 +241,12 @@ def translate_qr(self, data: str, camera_frame_num: int) -> DecodedQr:
         else:
             try:
                 json_data = json.loads(data)
-                code = self._translate_qr_test_runner(data, json_data, camera_frame_num)
+                code = self._translate_qr_test_runner(
+                    data, location, json_data, camera_frame_num
+                )
             except json.decoder.JSONDecodeError as e:
-                logger.error(f"Unrecognised QR code JSON detected in '{data}'. JSON err: {e}")
+                logger.error(
+                    f"Unrecognised QR code JSON detected in '{data}'. JSON err: {e}"
+                )
 
         return code

exceptions.py

@@ -23,7 +23,6 @@
 Contributor: Eurofins Digital Product Testing UK Limited
 """
 
-
 class ObsFrameError(Exception):
     """Base for specific exceptions thrown by Observation Framework.
 
@@ -45,3 +44,8 @@ class ObsFrameTerminate(ObsFrameError):
     when this is raised the Observation Framework will be terminated.
     """
     pass
+
+
+class ConfigError(ObsFrameError):
+    """Specific exception to indicate config errors"""
+    pass