Non-existing attribute is returned and causes float conversion error

[FWuellhorst]

Describe the bug

For this entity:

{
    _id: {
        id: 'conCenFan_oveAct_u',
        type: 'Actuator',
        servicePath: '/ALPaperStage3'
    },
    attrNames: [
        'conCenFan_oveAct_u_status',
        'conCenFan_oveAct_u_info',
        'TimeInstant'
    ],
    attrs: {
        conCenFan_oveAct_u_status: {
            type: 'commandStatus',
            creDate: 1692798690.8602214,
            modDate: 1693224142.4930723,
            value: 'OK',
            mdNames: [
                'TimeInstant'
            ],
            md: {
                TimeInstant: {
                    type: 'DateTime',
                    value: 1693224142.4899998
                }
            }
        },
        conCenFan_oveAct_u_info: {
            type: 'commandResult',
            creDate: 1692798690.8602214,
            modDate: 1693224142.4930823,
            value: 9.99729039,
            mdNames: [
                'TimeInstant'
            ],
            md: {
                TimeInstant: {
                    type: 'DateTime',
                    value: 1693224142.4899998
                }
            }
        },
        TimeInstant: {
            type: 'DateTime',
            creDate: 1692798690.8602214,
            modDate: 1696261274.039428,
            value: 1696261274.0340004,
            mdNames: []
        }
    },
    creDate: 1692798690.8602214,
    modDate: 1696261274.0394897,
    lastCorrelator: '10a74dbf-2355-4470-be58-a1638b7868b1'
}

I get the following attributes when calling get_attributes in ngsi_v2:

{
  'TimeInstant': {
    'type': 'DateTime',
    'value': '2023-10-02T15:41:14.034Z',
    'metadata': {}
  },
  'conCenFan_oveAct_u_info': {
    'type': 'commandResult',
    'value': 9.99729039,
    'metadata': {
      'TimeInstant': {
        'type': 'DateTime',
        'value': '2023-08-28T12:02:22.490Z'
      }
    }
  },
  'conCenFan_oveAct_u_status': {
    'type': 'commandStatus',
    'value': 'OK',
    'metadata': {
      'TimeInstant': {
        'type': 'DateTime',
        'value': '2023-08-28T12:02:22.490Z'
      }
    }
  },
  'conCenFan_oveAct_u': {
    'type': 'Number',
    'value': '',
    'metadata': {}
  }
}

The attribute conCenFan_oveAct_u does not exist, and as the value is an empty string but the type a Number, I get a float conversion error.

[djs0109]

It seems to me that conCenFan_oveAct_u is a command in IoTAent, and the entity attribute is also created by IoTAent.

If that is the case, I will suggest to use command as the type. For example:

  {
    "protocol": "IoTA-JSON",
    "transport": "MQTT",
    "explicitAttrs": false,
    "device_id":  ???,
    "entity_name": "conCenFan_oveAct_u",
    "entity_type": "Actuator",
    "commands": [
      {
        "name": "conCenFan_oveAct_u",
        "type": "command"
      }
    ],
    "attributes": [
      ...
    ],
    "ngsiVersion": "v2"
  }

[FWuellhorst]

That is correct. But the type of the command is a Number, not "command". And we never changed this implementation, it was working some months ago.

[FWuellhorst]

It works specifying the type="command", but I don't get why this only occurs now. We use the development fiware cluster, did you update the iotagent / orion?

[djs0109]

As far as I know, IoTagent will always create such an attribute in Orion. So I think the creation of conCenFan_oveAct_u is not a bug.

And still, I recommend using command as the data type, because it will be then easier to recognize the command in context broker, and this kind of attribute will not have any meaningful value anyway. The value is stored in conCenFan_oveAct_u_info

Apart from that, are you expecting filip to ignore such a data type error?

{
  "conCenFan_oveAct_u": {
    "type": "Number",
    "value": ""
  }
}

[FWuellhorst]

ok, thanks! Yes, filip should avoid this error. The attr is not even listed in mongo-express. As the error only occurs due to the conversion from "" to float, this should be catched using some sort of float(value) if value else None

[djs0109]

I agree. I will have a look

[github-actions]

Branch 203-Non-existing-attribute-is-returned-and-causes-float-conversion-error created!

[tstorek]

@FWuellhorst @djs0109 I just faced the exact same issue. I looked into it and the trouble occurs in the ContextAttribute creation and validation.

Let's start at the beginning:

1. You cannot see the attribute in the database because does not live in the ContextBroker database. It comes from a registraion that the IoTA-Agent creates for it. For more details check on the registration docs
2. When calling the entity the contextbroker fowards the request to the iotagent and receives the data. Maybe they changed the default value, if the command is empty.
3. ToFix: We need to adjust filip.models.ngsi_v2.base.BaseValueAttribute.validate_value_type to handle EmptyStrings. Otherwise we need to use the the command type so far, which I probably would recommend anyways. If you want to add additional info about the type maybe the metadata field works by now for commands. The latter would require to adjust the command model which would be a minor issue filip.models.ngsi_v2.iot.DeviceCommand. Let's check on the docs! Here we go: https://iotagent-node-lib.readthedocs.io/en/latest/api.html
4. From the docs commands should always have the type command. Probably I followed that in the original implementation but forgot to fix the type in the CommandModel.

I'm happy to help out here :) It is true that this is annoying. However, I will not be free until November. Maybe @FWuellhorst @djs0109 you can already try to realize the second option.

[tstorek]

Oh, btw @FWuellhorst. If you use command you can access them directly using the api of the ContextEntity ;) same for relationships :)

[djs0109]

@tstorek Thanks for your suggestions! I saw that if the data type is command, the value will not be validated in filip.models.ngsi_v2.base.BaseValueAttribute.validate_value_type. That is to say, if using command as type, empty string will not cause any problem.

For adding the metadata in DeviceModel, I check the docs. It is actually contradictory... some part suggests that only attribute can have metadata, and the other part says that command can also have metadata.
So I test it with LibVersion 3.3, JSON-version 2.3:

Metadata in command will cause syntax error

    "commands": [
      {
        "name": "com1",
        "type": "command",
        "metadata": {
            "unitCode": { "type": "Text", "value": "CAL" }
            }
      }
    ]

But metadata in attributes can work

    "attributes": [
      {
        "name": "attr1",
        "type": "Integer",
        "metadata": {
            "unitCode": { "type": "Text", "value": "CAL" }
            }
      }  
    ],

[tstorek]

@djs0109 @FWuellhorst According to the previous test, we should do the two things:

1. Raise an issue in IoTA-Json for metadata
2. Agree on a conversation-convention in filip

What Do you think? Shall we allow any types favoring the missing metadata object? Mostly likely it would be useful also for the other conversions, right?

[FWuellhorst]

Okay, thanks for checking! I would suggest to fix the validate_value_type function for this issue, the rest would be another issue, or?

[djs0109]

I think I would add an if statement to every data type, i.e. to explicitly allow missing values in the form of None, and "". float/int/str(value) if value else None will be too risky, because it will ignore for example 0