Add support for NetBox objects in the custom fields

[markkuleinio]

NetBox 3.2 will support objects in custom fields: netbox-community/netbox#7006

Some changes are most probably needed in pynetbox to support that.

[jeremystretch]

You don't waste any time, do you? 🙂

[markkuleinio]

You know, app development ship takes sometimes a long time to turn, so it's good to start teasing potential developers as soon as possible 😁 And, I may find myself at some point needing that feature 😂

[markkuleinio]

Let's test how these NetBox objects look like in the raw data. I have (in NetBox 3.2.0-dev) defined two custom fields, asn (which is an ASN object) and json (which is a JSON field), and assigned them to devices. I have also created an ASN object, and then created a device with the custom fields assigned. In REST API results it looks like this:

$ curl http://netbox-future.lein.io/api/dcim/devices/1/ -H "Authorization: Token $TOKEN" -H "Accept: application/json" -s | jq .
{
...
  "custom_fields": {
    "asn": {
      "id": 1,
      "url": "http://netbox-future.lein.io/api/ipam/asns/1/",
      "display": "AS65510",
      "asn": 65510
    },
    "json": {
      "here": 123,
      "some": "values"
    }
  },
...
}

The obvious question from pynetbox point of view is: How do we know that a custom field is a NetBox object instead of something else (like JSON in this example)?

I can currently figure out three ways to know it:

1. Call the /extras/custom-fields/ API to find out the custom field type when parsing the results. This would cause pynetbox to call the API every time an object is parsed. There could be an internal cache to reduce the number of API calls though, maybe even with configurable TTL.
2. Let the pynetbox user to tell it if the application wants to take advantage of the custom object fields. For example, pynetbox could contain a pynetbox.Api.resolve_custom_object_fields() method that would call the /extras/custom-fields/ API to get all custom object fields and set up pynetbox internal rules to parse them whenever they are found in API results. The user would need to call that method again if some of the custom fields are reconfigured (this is an obvious concern in long-running applications only).
3. Just parse custom_fields and guess when the custom field is a custom object field, for example when the field is a dict and contains id, url and display attributes (would that be a valid test @jeremystretch?), and figure out the correct pynetbox model from the url path (using Record model if corresponding custom model is not found in pynetbox).

Of course, there is always an option to not do anything in pynetbox: in the example above there is the AS number already contained in the custom field data, and if the user needs the full NetBox object (for example to check the object tags) she can get the object manually with a simple call like device_asn = nb.ipam.asns.get(device.custom_fields["asn"]["id"]). (She could write her own custom field resolving function anyway, using the logic described above, taking the custom field dict as input and returning the corresponding pynetbox object based on whatever the object type is.)

Performance reminder: If the full object data is to be parsed (again for example to get the tags for the object) it would require an extra NetBox API call for each parent object, so I'm positive that not everyone would like that to happen automatically.

[markkuleinio]

Personally I'm leaning towards not doing anything in pynetbox. If I will later use NetBox object in custom fields, I will just resolve the data myself. That will give me the best performance because I can avoid unnecessary NetBox API calls.

I will leave this open for now if some new insight comes up, but @zachmoody feel free to close this if desired.

[jcollie]

Personally, I'd like to see custom objects turned into Records. I'd be OK if that was behind a feature flag if performance suffers. There's also the issue of creating/modifying custom fields through the API.

[jcollie]

A thought that I just had... If NetBox was to return a list of the fields that were custom objects or JSON fields with every API call that would save a round-trip.

{
…
  "custom_field_types":{
    "json": "json",
    "asn": "object"
  },
  "custom_fields":{
    "asn": {
      "id": 1,
      "url": "http://netbox-future.lein.io/api/ipam/asns/1/",
      "display": "AS65510",
      "asn": 65510
    },
    "json": {
      "here": 123,
      "some": "values"
    }
  },
…
}

[zachmoody]

I think opt 3 seems reasonable. Can you think of a reason why it wouldn't be a good idea to just build the check into the Record initializer itself?

[jcollie]

Personally I'm leaning towards not doing anything in pynetbox

I don't think that's an option because pynetbox cannot currently update objects that contain JSON or object custom fields.

[markkuleinio]

For the reference here, #457 tracks the JSON save issue.

[amk1969]

Personally I'm leaning towards not doing anything in pynetbox

I don't think that's an option because pynetbox cannot currently update objects that contain JSON or object custom fields.

I was unable to figure out how to remove the object from a custom field. It is possible to set it once. Trying to assign None, received a traceback. Then figured out that any update of the record is problematic.

Let's hope the support can be added soon. There is also custom_field type of multiobject that can bring additional challenges. Ideally it would be turned into a RecordSet.

Attached a sample script I was planning to add as reproduction into a new issue, but it would be a duplicate of this one.
customflds.txt

[dduransseau]

To manage objects in custom field I use a logic to compare <custom_field_url>.startwith(nb.base_url) after I figure out the pynetbox model like you describe below. I guess this logic can be enable/disabled at pynetbox initialization to decide to query or not related objects.

Just parse custom_fields and guess when the custom field is a custom object field, for example when the field is a dict and contains id, url and display attributes (would that be a valid test @jeremystretch?), and figure out the correct pynetbox model from the url path (using Record model if corresponding custom model is not found in pynetbox).

However I figure out an issue of using netbox object in custom field: after set, it's not possible to update other custom field of main object. This is due to the netbox api return a dict of the related object but accept only the id on update.
Any guidance on how should I consider the issue, if this behavior should be managed by pynetbox or on netbox serializer ?

[oyacky]

However I figure out an issue of using netbox object in custom field: after set, it's not possible to update other custom field of main object. This is due to the netbox api return a dict of the related object but accept only the id on update. Any guidance on how should I consider the issue, if this behavior should be managed by pynetbox or on netbox serializer ?

This bug from pynetbox 7.0.1(PR #518). Normal JSON data is destructed. Function flatten_custom is wrong modifying.