Unable to list all extensions installed on an Arc-enabled AKS cluster

[johnbilliris]

Describe the bug

Running the following command in an Azure Cloud Shell produces a runtime error:

Command: az k8s-extension list --cluster-name --resource-group --cluster-type managedClusters

Error: ModuleNotFoundError: No module named 'azure.mgmt.relay'

Related command

az k8s-extension list --cluster-name --resource-group --cluster-type managedClusters

Errors

The command failed with an unexpected error. Here is the traceback:
No module named 'azure.mgmt.relay'
Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 562, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/init.py", line 521, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 319, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib64/az/lib/python3.9/site-packages/knack/commands.py", line 104, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
op = self.get_op_handler(self.op_path)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
handler = import_module(mod_to_import)
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/john/.azure/cliextensions/k8s-extension/azext_k8s_extension/custom.py", line 30, in
from .partner_extensions.AzureMLKubernetes import AzureMLKubernetes
File "/home/john/.azure/cliextensions/k8s-extension/azext_k8s_extension/partner_extensions/AzureMLKubernetes.py", line 15, in
import azure.mgmt.relay
ModuleNotFoundError: No module named 'azure.mgmt.relay'

Issue script & Debug output

cli.knack.cli: Command arguments: ['k8s-extension', 'list', '--cluster-name', 'hcibox-aks', '--resource-group', 'aks-stackhci-rg', '--cluster-type', 'managedClusters', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f3a9698a160>, <function OutputProducer.on_global_arguments at 0x7f3a968a3d30>, <function CLIQuery.on_global_arguments at 0x7f3a9683a310>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'k8s-extension': ['azext_k8s_extension']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: ai-examples 0.121 1 1 /usr/lib/python3.9/site-packages/azure-cli-extensions/ai-examples
cli.azure.cli.core: k8s-extension 0.003 2 9 /home/john/.azure/cliextensions/k8s-extension
cli.azure.cli.core: Total (2) 0.123 3 10
cli.azure.cli.core: Loaded 3 groups, 10 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : k8s-extension list
cli.azure.cli.core: Command table: k8s-extension list
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f3a95cea820>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/john/.azure/commands/2024-08-07.04-56-59.k8s-extension_list.1317.log'.
az_command_data_logger: command args: k8s-extension list --cluster-name {} --resource-group {} --cluster-type {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f3a95c93430>]
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 562, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/init.py", line 521, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 319, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib64/az/lib/python3.9/site-packages/knack/commands.py", line 104, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
op = self.get_op_handler(self.op_path)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
handler = import_module(mod_to_import)
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/john/.azure/cliextensions/k8s-extension/azext_k8s_extension/custom.py", line 30, in
from .partner_extensions.AzureMLKubernetes import AzureMLKubernetes
File "/home/john/.azure/cliextensions/k8s-extension/azext_k8s_extension/partner_extensions/AzureMLKubernetes.py", line 15, in
import azure.mgmt.relay
ModuleNotFoundError: No module named 'azure.mgmt.relay'

cli.azure.cli.core.azclierror: The command failed with an unexpected error. Here is the traceback:
az_command_data_logger: The command failed with an unexpected error. Here is the traceback:
cli.azure.cli.core.azclierror: No module named 'azure.mgmt.relay'
Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 562, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/init.py", line 521, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 319, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib64/az/lib/python3.9/site-packages/knack/commands.py", line 104, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
op = self.get_op_handler(self.op_path)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
handler = import_module(mod_to_import)
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/john/.azure/cliextensions/k8s-extension/azext_k8s_extension/custom.py", line 30, in
from .partner_extensions.AzureMLKubernetes import AzureMLKubernetes
File "/home/john/.azure/cliextensions/k8s-extension/azext_k8s_extension/partner_extensions/AzureMLKubernetes.py", line 15, in
import azure.mgmt.relay
ModuleNotFoundError: No module named 'azure.mgmt.relay'
az_command_data_logger: No module named 'azure.mgmt.relay'
Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 562, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/init.py", line 521, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 319, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib64/az/lib/python3.9/site-packages/knack/commands.py", line 104, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
op = self.get_op_handler(self.op_path)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
handler = import_module(mod_to_import)
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/john/.azure/cliextensions/k8s-extension/azext_k8s_extension/custom.py", line 30, in
from .partner_extensions.AzureMLKubernetes import AzureMLKubernetes
File "/home/john/.azure/cliextensions/k8s-extension/azext_k8s_extension/partner_extensions/AzureMLKubernetes.py", line 15, in
import azure.mgmt.relay
ModuleNotFoundError: No module named 'azure.mgmt.relay'
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f3a95ceaa60>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 0.382 seconds (init: 0.120, invoke: 0.262)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 7876 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/bin/python3.9 /usr/lib/az/lib/python3.9/site-packages/azure/cli/telemetry/init.py /home/john/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

Expected behaviour is no runtime error, and a list all extensions installed on the cluster to be returned.

Environment Summary

azure-cli 2.62.0 *

core 2.62.0 *
telemetry 1.1.0

Extensions:
acrquery 1.0.1
ai-examples 0.2.5
appservice-kube 0.1.8
connectedk8s 1.9.0
customlocation 0.1.3
k8s-configuration 1.7.0
k8s-extension 1.4.2
logic 0.1.7
ml 2.28.0
ssh 2.0.5

Dependencies:
msal 1.28.1
azure-mgmt-resource 23.1.1

Python location '/usr/bin/python3.9'
Extensions directory '/home/john/.azure/cliextensions'
Extensions system directory '/usr/lib/python3.9/site-packages/azure-cli-extensions'

Python (Linux) 3.9.19 (main, Mar 28 2024, 18:56:59)
[GCC 11.2.0]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

No response

[azure-client-tools-bot-prd]

Hi @johnbilliris,

2.62.0 is not the latest Azure CLI(2.63.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dyu1208, @FumingZhang, @andyliuliming.