Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

az ml model download doesn't download any artifacts #27984

Closed
kmr0877 opened this issue Dec 8, 2023 · 11 comments
Closed

az ml model download doesn't download any artifacts #27984

kmr0877 opened this issue Dec 8, 2023 · 11 comments
Assignees
@banibrata-de
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning az ml Service Attention This issue is responsible by Azure service team.

Comments

@kmr0877
Copy link

kmr0877 commented Dec 8, 2023 •
edited
Loading

Describe the bug

CLI command works fine but the actual model is not downloaded

Related command

az ml model download --name ${AML_MODEL_NAME} --version ${AML_MODEL_VERSION} --download-path models --resource-group ${AML_RESOURCE_GROUP} --workspace-name ${AML_WORKSPACE_NAME} --subscription ${AML_WORKSPACE_SUBSCRIPTION} --debug

Errors

2023-12-07T21:50:45.5995574Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): nmlauecadiapcasa01.blob.core.windows.net:443
2023-12-07T21:50:45.7135854Z DEBUG: urllib3.connectionpool: https://<>.blob.core.windows.net:443 "HEAD /azureml/azureml/quiet_iron_dbx6z8qc3x/mlflow_model_artifacts/ HTTP/1.1" 404 0
2023-12-07T21:50:45.7210683Z DEBUG: urllib3.connectionpool: https://<>.blob.core.windows.net:443 "GET /azureml?restype=container&comp=list&prefix=azureml%2Fquiet_iron_dbx6z8qc3x%2Fmlflow_model_artifacts%2F&delimiter=%2F HTTP/1.1" 200 None
2023-12-07T21:50:45.7217163Z Downloading the model azureml/quiet_iron_dbx6z8qc3x/mlflow_model_artifacts/ at ./models/
2023-12-07T21:50:45.7217643Z
2023-12-07T21:50:45.7284012Z DEBUG: urllib3.connectionpool: https://<>.blob.core.windows.net:443 "GET /azureml?restype=container&comp=list&prefix=azureml%2Fquiet_iron_dbx6z8qc3x%2Fmlflow_model_artifacts%2F&include=metadata HTTP/1.1" 200 None
2023-12-07T21:50:45.7293915Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7f62691b5760>, <function _x509_from_base64_to_hex_transform at 0x7f62691b5800>]
2023-12-07T21:50:45.7295628Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
2023-12-07T21:50:45.7296603Z DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
2023-12-07T21:50:45.7297647Z DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f62693c45e0>]
2023-12-07T21:50:45.7298245Z INFO: az_command_data_logger: exit code: 0
2023-12-07T21:50:45.7298786Z INFO: cli.main: Command ran in 3.163 seconds (init: 0.182, invoke: 2.980)
2023-12-07T21:50:45.7992445Z INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
2023-12-07T21:50:45.7993386Z INFO: telemetry.client: Accumulated 0 events. Flush the clients.
2023-12-07T21:50:45.7994445Z INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
2023-12-07T21:50:45.7995181Z INFO: telemetry.save: Save telemetry record of length 3512 in cache
2023-12-07T21:50:45.7998175Z INFO: telemetry.main: Begin creating telemetry upload process.
2023-12-07T21:50:45.8000452Z INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/init.py /home/vsts/work/_temp/.azclitask"
2023-12-07T21:50:45.8006496Z INFO: telemetry.process: Return from creating process
2023-12-07T21:50:45.8007334Z INFO: telemetry.main: Finish creating telemetry upload process.

Issue script & Debug output

2023-12-07T21:50:45.5995574Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): nmlauecadiapcasa01.blob.core.windows.net:443
2023-12-07T21:50:45.7135854Z DEBUG: urllib3.connectionpool: https://.blob.core.windows.net:443 "HEAD /azureml/azureml/quiet_iron_dbx6z8qc3x/mlflow_model_artifacts/ HTTP/1.1" 404 0
2023-12-07T21:50:45.7210683Z DEBUG: urllib3.connectionpool: https://.blob.core.windows.net:443 "GET /azureml?restype=container&comp=list&prefix=azureml%2Fquiet_iron_dbx6z8qc3x%2Fmlflow_model_artifacts%2F&delimiter=%2F HTTP/1.1" 200 None
2023-12-07T21:50:45.7217163Z Downloading the model azureml/quiet_iron_dbx6z8qc3x/mlflow_model_artifacts/ at ./models/
2023-12-07T21:50:45.7217643Z
2023-12-07T21:50:45.7284012Z DEBUG: urllib3.connectionpool: https://***.blob.core.windows.net:443 "GET /azureml?restype=container&comp=list&prefix=azureml%2Fquiet_iron_dbx6z8qc3x%2Fmlflow_model_artifacts%2F&include=metadata HTTP/1.1" 200 None
2023-12-07T21:50:45.7293915Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7f62691b5760>, <function _x509_from_base64_to_hex_transform at 0x7f62691b5800>]
2023-12-07T21:50:45.7295628Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
2023-12-07T21:50:45.7296603Z DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
2023-12-07T21:50:45.7297647Z DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f62693c45e0>]
2023-12-07T21:50:45.7298245Z INFO: az_command_data_logger: exit code: 0
2023-12-07T21:50:45.7298786Z INFO: cli.main: Command ran in 3.163 seconds (init: 0.182, invoke: 2.980)
2023-12-07T21:50:45.7992445Z INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
2023-12-07T21:50:45.7993386Z INFO: telemetry.client: Accumulated 0 events. Flush the clients.
2023-12-07T21:50:45.7994445Z INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
2023-12-07T21:50:45.7995181Z INFO: telemetry.save: Save telemetry record of length 3512 in cache
2023-12-07T21:50:45.7998175Z INFO: telemetry.main: Begin creating telemetry upload process.
2023-12-07T21:50:45.8000452Z INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/init.py /home/vsts/work/_temp/.azclitask"
2023-12-07T21:50:45.8006496Z INFO: telemetry.process: Return from creating process
2023-12-07T21:50:45.8007334Z INFO: telemetry.main: Finish creating telemetry upload process.

Expected behavior

Model should be downloaded to the local filesystem

Environment Summary

image

Additional context

No response
@kmr0877 kmr0877 added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Dec 8, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented Dec 8, 2023

Thank you for opening this issue, we will look into it.

@kmr0877
Copy link
Author

kmr0877 commented Dec 8, 2023

Linking relevant issue #25601

@microsoft-github-policy-service microsoft-github-policy-service bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. Auto-Assign Auto assign by bot Service Attention This issue is responsible by Azure service team. Machine Learning az ml labels Dec 8, 2023
@banibrata-de banibrata-de self-assigned this Jan 31, 2024
@banibrata-de
Copy link
Member

Closed based on relevant issue #25601

@ArPharazon
Copy link

@banibrata-de

Why is the issue closed? It has not been fixed, and there is no solution provided in #25601

@nemanjarajic
Copy link
Member

Hi @ArPharazon, we are in the process of a new release for the cli (2.24.0) and were not able to repro this issue. The new release should available sometime next week. Can you test it then to see if you are still facing the same issue? Thank you.

@ArPharazon
Copy link

Can you let me know exactly when it will be available next week?

Microsoft Support has given me a workaround, but i'd rather not make those changes in our CICD process if there is a definite fix coming.

@nemanjarajic
Copy link
Member

Hi @ArPharazon, release should be going out later today (3/11/2024).

@ArPharazon
Copy link

Hi @nemanjarajic

I have been checking for updates regularly, but still cannot see any new version of the 'ml' extension yet. Currently still showing v2.23.0 as the max compatible version, so just wondering when v2.24.0 will be available?

Please let me know.

@ArPharazon
Copy link

@nemanjarajic

I have tested the latest version of the 'ml' extension (version 2.24.0) and it still does not work.

Please advise when we can expect this serious bug to be fixed.

@pedrojrv
Copy link

I am facing the exact same issue, something change and I am now unable to donwload the models. Any fixes?

@ArPharazon
Copy link

The issue was closed with no fix, but here is some workarounds.

If your model is registered as CUSTOM type, then you can download model artifacts from the training job or shared registry, like this:

# training job
az ml job download --name "my-job" --workspace-name "my-workspace" --resource-group "my-resource-group" --subscription "my-subscription" --debug

# shared registry
az ml model download --name "model-name" --version "1" --registry-name "my-registry" --resource-group "my-resource-group" --subscription "my-subscription" --debug

If the model is registered as MLFLOW type, then the "az ml model download" command works correctly.

@ArPharazon ArPharazon mentioned this issue Aug 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@banibrata-de banibrata-de

Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning az ml Service Attention This issue is responsible by Azure service team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ArPharazon @kmr0877 @pedrojrv @yonzhan @nemanjarajic @banibrata-de