az storage blob upload-batch flag --if-unmodified-since being ignored due to ModuleNotFoundError under storagev2

[heschmidt04]

Describe the bug

Following https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-macos
For M1 macintosh az is not installed by brew in /opt/homebrew/bin/az per web instructions

I have Python 3.10.10 and az version 2.55 under
$HOMEDIR/miniconda3/bin as the base install before anything is defined.

I expected when I ran my command to upload-batch files from a local directory that the flag --if-unmodified-since would not go through the list of previously loaded files based on a timestamp that is one day past the last file transfer.
It appears that it is being ignored because it can't find the path to the models in Python 3.10 or 3.11 and I get variations on ModuleNotFoundError: No module named
For Python 3.10 I get ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.file'

For env
CONDA_PYTHON_EXE=$HOME/miniconda3/bin/python
PYTHONPATH=/opt/homebrew/bin/python3.10:$HOME/miniconda3/bin/python

I do have the directory under blob as $HOME/miniconda3/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02 -- see env and version info below.

Thanks,
Heidi

Related command

AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1; az storage blob upload-batch --destination $DESTDIR --account-name $STORAGE_BUCKET --source $SOURCE_DIR --sas-token "$SAS_TOKEN" --if-unmodified-since 2023-12-31T12:59:59Z --debug

Errors

ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.file'

and

The specified blob already exists.
RequestId:f992c4ad-b01e-0015-455a-427223000000
Time:2024-01-08T17:47:34.3068817Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.

Issue script & Debug output

I obscured the values for the flags, the hostname in the url, sas token and left the rest of the debug information.

az storage blob upload-batch --destination $DESTDIR --account-name $STORAGE_BUCKET --source $SOURCE_DIR --sas-token "$SAS_TOKEN" --if-unmodified-since 2023-12-31T12:59:59Z --debug

cli.knack.cli: Command arguments: ['storage', 'blob', 'upload-batch', '--destination', '$DESTDIR', '--account-name', '$STORAGE_BUCKET', '--source', '$SOURCEDIR', '--sas-token', '$SAS_TOKEN', '--if-unmodified-since', '2023-12-31T12:59:59Z', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x105320e50>, <function OutputProducer.on_global_arguments at 0x105490280>, <function CLIQuery.on_global_arguments at 0x1054cd510>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'storage': ['azext_storage_preview', 'azure.cli.command_modules.storage']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: storage                   0.077        59       273
cli.azure.cli.core: Total (1)                 0.077        59       273
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.knack.cli: Event: CommandLoader.OnLoadCommandTable []
cli.azure.cli.core: storage-preview           0.007        16        52  $HOME/.azure/cliextensions/storage-preview
cli.azure.cli.core: Total (1)                 0.007        16        52
cli.azure.cli.core: Loaded 65 groups, 295 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : storage blob upload-batch
cli.azure.cli.core: Command table: storage blob upload
cli.azure.cli.core: remaining    :                     batch
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x106325a20>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '$HOME/.azure/commands/2024-01-08.15-43-24.storage_blob_upload-batch.73143.log'.
az_command_data_logger: command args: storage blob upload-batch --destination {} --account-name {} --source {} --sas-token {} --if-unmodified-since {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x10633e7a0>]
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'ListKeyExpand'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'CorsRuleAllowedMethodsItem'

cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x10638c790>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x10638c8b0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x105490310>, <function CLIQuery.handle_query_parameter at 0x1054cd5a0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x10638c820>]
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
cli.azure.cli.command_modules.storage._validators: Try to get storage auth_mode value from environment variables or config file.
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 651, in _get_attr
    op = import_module(full_mod_path)
  File "$HOMEDIR/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.blob'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 651, in _get_attr
    op = import_module(full_mod_path)
  File "$HOMEDIR/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.file'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 651, in _get_attr
    op = import_module(full_mod_path)
  File "$HOMEDIR/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.blob'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 651, in _get_attr
    op = import_module(full_mod_path)
  File "$HOMEDIR/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.file'

cli.azure.cli.core._debug: Connection verification disabled by environment variable AZURE_CLI_DISABLE_CONNECTION_VERIFICATION
azure.multiapi.storagev2.blob.v2022_11_02._generated._serialization: Datetime with no tzinfo will be considered UTC.
urllib3.connectionpool: Starting new HTTPS connection (1): proxy:443
$HOMEDIR/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
urllib3.connectionpool: https://proxy:443 "PUT $PATH-TO-FILE file1.name?$SAS_TOKEN HTTP/1.1" 412 252
cli.azure.cli.command_modules.storage.operations.blob: The specified blob already exists.
RequestId:bf86f837-a01e-0026-2673-422d88000000
Time:2024-01-08T20:43:24.9579532Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.

Expected behavior

The behavior I was hoping for was the command to return with no new files to be loaded and the previous loaded files would not be pre-processed again with ErrorCode:BlobAlreadyExists
Is there a test I can run to see if the erroring modules are imported properly?

Thanks! Heidi

Environment Summary

azure-cli                         2.55.0

core                              2.55.0
telemetry                          1.1.0

Extensions:
storage-preview                  1.0.0b1

Dependencies:
msal                            1.24.0b2
azure-mgmt-resource             23.1.0b2

Python location '/opt/homebrew/Cellar/azure-cli/2.55.0/libexec/bin/python'
Extensions directory '/Users/heidischmidt/.azure/cliextensions'

Python (Darwin) 3.11.6 (main, Oct  2 2023, 13:45:54) [Clang 15.0.0 (clang-1500.0.40.1)]

Legal docs and information: aka.ms/AzureCliLegal

and in the miniconda base the pip list includes the following azure modules

azure-appconfiguration                  1.1.1
azure-batch                             14.0.0
azure-cli                               2.55.0
azure-cli-core                          2.55.0
azure-cli-telemetry                     1.1.0
azure-common                            1.1.28
azure-core                              1.29.6
azure-cosmos                            3.2.0
azure-data-tables                       12.4.0
azure-datalake-store                    0.0.53
azure-graphrbac                         0.60.0
azure-identity                          1.15.0
azure-keyvault-administration           4.4.0b2
azure-keyvault-certificates             4.7.0
azure-keyvault-keys                     4.9.0b3
azure-keyvault-secrets                  4.7.0
azure-loganalytics                      0.1.1
azure-mgmt-advisor                      9.0.0
azure-mgmt-apimanagement                4.0.0
azure-mgmt-appconfiguration             3.0.0
azure-mgmt-appcontainers                2.0.0
azure-mgmt-applicationinsights          1.0.0
azure-mgmt-authorization                4.0.0
azure-mgmt-batch                        17.0.0
azure-mgmt-batchai                      7.0.0b1
azure-mgmt-billing                      6.0.0
azure-mgmt-botservice                   2.0.0
azure-mgmt-cdn                          12.0.0
azure-mgmt-cognitiveservices            13.5.0
azure-mgmt-compute                      30.3.0
azure-mgmt-containerinstance            10.1.0
azure-mgmt-containerregistry            10.1.0
azure-mgmt-containerservice             28.0.0
azure-mgmt-core                         1.4.0
azure-mgmt-cosmosdb                     9.3.0
azure-mgmt-databoxedge                  1.0.0
azure-mgmt-datalake-nspkg               3.0.1
azure-mgmt-datalake-store               0.5.0
azure-mgmt-datamigration                10.0.0
azure-mgmt-devtestlabs                  4.0.0
azure-mgmt-dns                          8.0.0
azure-mgmt-eventgrid                    10.2.0b2
azure-mgmt-eventhub                     10.1.0
azure-mgmt-extendedlocation             1.0.0b2
azure-mgmt-hdinsight                    9.0.0
azure-mgmt-imagebuilder                 1.2.0
azure-mgmt-iotcentral                   10.0.0b2
azure-mgmt-iothub                       3.0.0
azure-mgmt-iothubprovisioningservices   1.1.0
azure-mgmt-keyvault                     10.3.0
azure-mgmt-kusto                        0.3.0
azure-mgmt-loganalytics                 13.0.0b4
azure-mgmt-managedservices              1.0.0
azure-mgmt-managementgroups             1.0.0
azure-mgmt-maps                         2.0.0
azure-mgmt-marketplaceordering          1.1.0
azure-mgmt-media                        9.0.0
azure-mgmt-monitor                      5.0.1
azure-mgmt-msi                          7.0.0
azure-mgmt-netapp                       10.1.0
azure-mgmt-nspkg                        3.0.2
azure-mgmt-policyinsights               1.1.0b4
azure-mgmt-privatedns                   1.0.0
azure-mgmt-rdbms                        10.2.0b13
azure-mgmt-recoveryservices             2.5.0
azure-mgmt-recoveryservicesbackup       7.0.0
azure-mgmt-redhatopenshift              1.4.0
azure-mgmt-redis                        14.1.0
azure-mgmt-resource                     23.1.0b2
azure-mgmt-search                       9.1.0
azure-mgmt-security                     5.0.0
azure-mgmt-servicebus                   8.2.0
azure-mgmt-servicefabric                1.0.0
azure-mgmt-servicefabricmanagedclusters 1.0.0
azure-mgmt-servicelinker                1.2.0b1
azure-mgmt-signalr                      2.0.0b1
azure-mgmt-sql                          4.0.0b13
azure-mgmt-sqlvirtualmachine            1.0.0b5
azure-mgmt-storage                      21.1.0
azure-mgmt-synapse                      2.1.0b5
azure-mgmt-trafficmanager               1.0.0
azure-mgmt-web                          7.0.0
azure-multiapi-storage                  1.2.0
azure-nspkg                             3.0.2
azure-storage-blob                      12.19.0
azure-storage-common                    1.4.2
azure-storage-file-share                12.15.0
azure-synapse-accesscontrol             0.5.0
azure-synapse-artifacts                 0.17.0
azure-synapse-managedprivateendpoints   0.4.0
azure-synapse-spark                     0.2.0
msrestazure                             0.6.4

and in the Python 3.10 site-packages under storagev2/blob/v2022_11_02 directory

ls -l $HOME/miniconda3//lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/                      []
total 1240
-rw-r--r--   1 hes  staff    9915 Jan  8 11:31 __init__.py
drwxr-xr-x  17 hes  staff     544 Jan  8 11:31 __pycache__
-rw-r--r--   1 hes  staff  233131 Jan  8 11:31 _blob_client.py
-rw-r--r--   1 hes  staff   39496 Jan  8 11:31 _blob_service_client.py
-rw-r--r--   1 hes  staff   92743 Jan  8 11:31 _container_client.py
-rw-r--r--   1 hes  staff    8744 Jan  8 11:31 _deserialize.py
-rw-r--r--   1 hes  staff   32099 Jan  8 11:31 _download.py
-rw-r--r--   1 hes  staff   41014 Jan  8 11:31 _encryption.py
drwxr-xr-x  12 hes  staff     384 Jan  8 11:31 _generated
-rw-r--r--   1 hes  staff   18749 Jan  8 11:31 _lease.py
-rw-r--r--   1 hes  staff   14741 Jan  8 11:31 _list_blobs_helper.py
-rw-r--r--   1 hes  staff   57319 Jan  8 11:31 _models.py
-rw-r--r--   1 hes  staff    6270 Jan  8 11:31 _quick_query_helper.py
-rw-r--r--   1 hes  staff    7918 Jan  8 11:31 _serialize.py
drwxr-xr-x  18 hes  staff     576 Jan  8 11:31 _shared
-rw-r--r--   1 hes  staff   32372 Jan  8 11:31 _shared_access_signature.py
-rw-r--r--   1 hes  staff   13955 Jan  8 11:31 _upload_helpers.py
-rw-r--r--   1 hes  staff     331 Jan  8 11:31 _version.py
drwxr-xr-x  12 hes  staff     384 Jan  8 11:31 aio

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[calvinhzy]

Hi @heschmidt04, the actual reason the call is failing is because there is an existing blob. Please use --overwrite for an existing blob. Thanks.

[heschmidt04]

My use case will be that I need the azure cli to ignore the previous files that were transferred, not overwrite, as more are transferred from the same source directory over time. What flag will work so that it will ignore files already transferred?

…

On Tue, Feb 20, 2024, 02:32 Zhiyi Huang ***@***.***> wrote: Hi @heschmidt04 <https://github.com/heschmidt04>, the actual reason the call is failing is because there is an existing blob. Please use --overwrite for an existing blob. Thanks. image.png (view on web) <https://github.com/Azure/azure-cli/assets/17182306/1b1f90d8-8279-44a4-81ff-a14a468c0363> — Reply to this email directly, view it on GitHub <#28153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYR4GNFZXMFJCBBX6RR5RLYURGRHAVCNFSM6AAAAABBSCUDX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJTGYZDONZQHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[calvinhzy]

Hi @heschmidt04, to upload only new files you just need to run az storage blob upload-batch --destination $DESTDIR --account-name $STORAGE_BUCKET --source $SOURCE_DIR --sas-token "$SAS_TOKEN" without either --if-unmodified-since or --overwrite. It would show the number of new files uploaded and number of files failed. This would, however, throw these error messages that might be distracting for your use case. We are considering adding an option to suppress these errors for az storage blob upload-batch