Merge branch 'main' into ashruti/setupBlobfuse

Azure · Nov 22, 2024 · 7436c93 · 7436c93
2 parents d90d5ad + 0ab1a64
commit 7436c93
Show file tree

Hide file tree

Showing 81 changed files with 2,407 additions and 4,089 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -1 +1 @@
-*       @vibhansa-msft @souravgupta-msft @ashruti-msft @syeleti-msft @jainakanksha-msft
+*       @vibhansa-msft @souravgupta-msft @ashruti-msft @syeleti-msft @jainakanksha-msft 
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -0,0 +1,18 @@
+## ✅ What
+
+<!-- A brief description of the changes in this PR. -->
+
+## 🤔 Why
+
+<!-- A brief description of the reason for these changes. -->
+
+## 👩‍🔬 How to validate if applicable
+
+<!-- Step-by-step instructions for how reviewers can verify these changes work as expected. -->
+
+## 🔖 Related links
+
+- [Issues](<link>)
+- [Team thread](<link>)
+- [Documents](<link>)
+- [Email Subject]
diff --git a/.gitignore b/.gitignore
@@ -10,7 +10,7 @@ blobfuse2
 adlsgen1fuse
 venv/
 *.backup
-__debug_bin
+__debug_bin*
 .env
 *.prof
 cpplite/
@@ -21,3 +21,4 @@ lint.log
 azure-storage-fuse
 bfusemon
 test/scripts/dirIterate.go
+component/azstorage/logfile.txt
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,22 @@
-## 2.3.3 (Unreleased)
+## 2.4.0 (Unreleased)
+**Features**
+- Entry cache to hold directory listing results in cache for a given timeout. This will reduce REST calls going to storage while listing the blobs in parallel.
+
 **Bug Fixes**
+- [#1426](https://github.com/Azure/azure-storage-fuse/issues/1426) Read panic in block-cache due to boundary conditions.
+- Do not allow mount path and temp-cache path to be same when using block-cache.
+- Do not allow to mount with non-empty directory provided for disk persistence in block-cache.
+- Rename file was calling an additional getProperties call.
+- Delete empty directories from local cache on rmdir operation.
+- [#1547](https://github.com/Azure/azure-storage-fuse/issues/1547) Truncate logic of file cache is modified to prevent downloading and uploading the entire file.
+
+**Features**
+- Added 'gen-config' command to auto generate blobfuse2 config file.
+
+**Other Changes**
+- Stream config will be converted to block-cache config implicitly and 'stream' component is no longer used from this release onwards.
+- MSI login with object-id will not rely on azcli anymore, rather it will be supported by 'azidentity' SDK.
+- Version check is now moved to a static website hosted on a public container.
 
 ## 2.3.2 (2024-09-03)
 **Bug Fixes**

diff --git a/MIGRATION.md b/MIGRATION.md
@@ -99,7 +99,7 @@ Note: Blobfuse2 accepts all CLI parameters that Blobfuse does, but may ignore pa
 | --log-level=LOG_WARNING                 | --log-level=LOG_WARNING                       | logging.level                         |                                                           |
 | --use-attr-cache=true                   | --use-attr-cache=true                         | attr_cache                            | Add attr_cache to the components list                     |
 | --use-adls=false                        | --use-adls=false                              | azstorage.type                        | Specify either 'block' or 'adls'                          |
-| --no-symlinks=false                     | --no-symlinks=false                           | attr_cache.no-symlinks                |                                                           |
+| --no-symlinks=false                     | --no-symlinks=true                            | attr_cache.no-symlinks                |                                                           |
 | --cache-on-list=true                    | --cache-on-list=true                          | attr_cache.no-cache-on-list           | This parameter has the opposite boolean semantics         |
 | --upload-modified-only=true             | --upload-modified-only=true                   |                                       | Always on in blobfuse2                                    |
 | --max-concurrency=12                    | --max-concurrency=12                          | azstorage.max-concurrency             |                                                           |

diff --git a/NOTICE b/NOTICE
@@ -3899,4 +3899,198 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 THE SOFTWARE.
 
 
+
+
+
+
+****************************************************************************
+
+============================================================================
+>>> github.com/AzureAD/microsoft-authentication-extensions-for-go/cache
+==============================================================================
+
+    MIT License
+
+    Copyright (c) Microsoft Corporation.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE
+
+
+
+
+****************************************************************************
+
+============================================================================
+>>> github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache
+==============================================================================
+
+The MIT License (MIT)
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+
+
+****************************************************************************
+
+============================================================================
+>>> github.com/cespare/xxhash/v2
+==============================================================================
+
+Copyright (c) 2016 Caleb Spare
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+
+
+****************************************************************************
+
+============================================================================
+>>> github.com/dgryski/go-rendezvous
+==============================================================================
+
+The MIT License (MIT)
+
+Copyright (c) 2017-2020 Damian Gryski <damian@gryski.com>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+
+
+
+****************************************************************************
+
+============================================================================
+>>> github.com/keybase/go-keychain
+==============================================================================
+
+The MIT License (MIT)
+
+Copyright (c) 2015 Keybase
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+
+
+
+****************************************************************************
+
+============================================================================
+>>> github.com/redis/go-redis/v9
+==============================================================================
+
+Copyright (c) 2013 The github.com/redis/go-redis Authors.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
 --------------------- END OF THIRD PARTY NOTICE --------------------------------
diff --git a/README.md b/README.md
@@ -13,7 +13,7 @@ Please submit an issue [here](https://github.com/azure/azure-storage-fuse/issues
 ##  NOTICE
 - Due to known data consistency issues when using Blobfuse2 in `block-cache` mode,  it is strongly recommended that all Blobfuse2 installations be upgraded to version 2.3.2. For more information, see [this](https://github.com/Azure/azure-storage-fuse/wiki/Blobfuse2-Known-issues).
 - As of version 2.3.0, blobfuse has updated its authentication methods. For Managed Identity, Object-ID based OAuth is solely accessible via CLI-based login, requiring Azure CLI on the system. For a dependency-free option, users may utilize Application/Client-ID or Resource ID based authentication.
-- `streaming` mode is being deprecated.
+- `streaming` mode is deprecated. Blobfuse2 will implicitly convert your streaming config to block-cache.
 
 ## Limitations in Block Cache
 - Concurrent write operations on the same file using multiple handles is not checked for data consistency and may lead to incorrect data being written.
@@ -38,7 +38,7 @@ Visit [this](https://github.com/Azure/azure-storage-fuse/wiki/Blobfuse2-Supporte
 - Basic file system operations such as mkdir, opendir, readdir, rmdir, open, 
    read, create, write, close, unlink, truncate, stat, rename
 - Local caching to improve subsequent access times
-- Streaming/Block-Cache to support reading AND writing large files 
+- Block-Cache to support reading AND writing large files 
 - Parallel downloads and uploads to improve access time for large files
 - Multiple mounts to the same container for read-only workloads
 
@@ -65,7 +65,7 @@ One of the biggest BlobFuse2 features is our brand new health monitor. It allows
 - CLI to check or update a parameter in the encrypted config
 - Set MD5 sum of a blob while uploading
 - Validate MD5 sum on download and fail file open on mismatch
-- Large file writing through write streaming/Block-Cache
+- Large file writing through write Block-Cache
 
  ## Blobfuse2 performance compared to blobfuse(v1.x.x)
 - 'git clone' operation is 25% faster (tested with vscode repo cloning)
@@ -154,8 +154,6 @@ To learn about a specific command, just include the name of the command (For exa
     * `--high-disk-threshold=<PERCENTAGE>`: If local cache usage exceeds this, start early eviction of files from cache.
     * `--low-disk-threshold=<PERCENTAGE>`: If local cache usage comes below this threshold then stop early eviction.
     * `--sync-to-flush=false` : Sync call will force upload a file to storage container if this is set to true, otherwise it just evicts file from local cache.
-- Stream options
-    * `--block-size-mb=<SIZE IN MB>`: Size of a block to be downloaded during streaming.
 - Block-Cache options
     * `--block-cache-block-size=<SIZE IN MB>`: Size of a block to be downloaded as a unit.
     * `--block-cache-pool-size=<SIZE IN MB>`: Size of pool to be used for caching. This limits total memory used by block-cache. Default - 80% of free memory available.
@@ -203,6 +201,8 @@ To learn about a specific command, just include the name of the command (For exa
 - CPK options: 
     * `AZURE_STORAGE_CPK_ENCRYPTION_KEY`: Customer provided base64-encoded AES-256 encryption key value.
     * `AZURE_STORAGE_CPK_ENCRYPTION_KEY_SHA256`: Base64-encoded SHA256 of the cpk encryption key.
+- Custom component options:
+    * `BLOBFUSE_PLUGIN_PATH`: Specifies plugin file path as a colon-separated list of `.so` files. Example BLOBFUSE_PLUGIN_PATH="/path/to/plugin1.so:/path/to/plugin2.so".
 
 
 ## Config Guide
@@ -230,7 +230,6 @@ Below diagrams guide you to choose right configuration for your workloads.
 <br/><br/>
 - [Sample File Cache Config](./sampleFileCacheConfig.yaml)
 - [Sample Block-Cache Config](./sampleBlockCacheConfig.yaml)
-- [Sample Stream Config](./sampleStreamingConfig.yaml)
 - [All Config options](./setup/baseConfig.yaml) 
 
 

diff --git a/azure-pipeline-templates/e2e-tests-block-cache-data-integrity.yml b/azure-pipeline-templates/e2e-tests-block-cache-data-integrity.yml
@@ -149,6 +149,10 @@ steps:
       VERBOSE_LOG: ${{ parameters.verbose_log }}
     continueOnError: false
 
+  - script: |
+     if [ -d "block_cache" ]; then ls -l block_cache; rm -rf block_cache; mkdir block_cache  ; fi
+    displayName: 'Clear Temp Cache for Block Cache before mounting'
+
   - template: 'mount.yml'
     parameters:
       working_dir: $(WORK_DIR)

diff --git a/azure-pipeline-templates/e2e-tests-block-cache.yml b/azure-pipeline-templates/e2e-tests-block-cache.yml
@@ -76,7 +76,12 @@ steps:
     displayName: 'Unmount RW mount'
 
   - script: |
-      $(WORK_DIR)/blobfuse2 gen-test-config --config-file=$(WORK_DIR)/testdata/config/azure_key_bc.yaml --container-name=${{ parameters.container }} --temp-path=${{ parameters.temp_dir }} --output-file=${{ parameters.config_file }}
+      if [ "${{ parameters.idstring }}" = "Stream" ]; then
+      CONFIG_FILE=$(WORK_DIR)/testdata/config/azure_stream.yaml
+      else
+      CONFIG_FILE=$(WORK_DIR)/testdata/config/azure_key_bc.yaml
+      fi
+      $(WORK_DIR)/blobfuse2 gen-test-config --config-file=$CONFIG_FILE --container-name=${{ parameters.container }} --temp-path=${{ parameters.temp_dir }} --output-file=${{ parameters.config_file }}
     displayName: 'Create Config File for RO mount'
     env:
       NIGHTLY_STO_ACC_NAME: ${{ parameters.account_name }}

diff --git a/blobfuse2-code-coverage.yaml b/blobfuse2-code-coverage.yaml
@@ -592,7 +592,7 @@ stages:
             - script: |
                 echo 'mode: count' > ./blobfuse2_coverage_raw.rpt
                 tail -q -n +2 ./*.cov >> ./blobfuse2_coverage_raw.rpt
-                cat ./blobfuse2_coverage_raw.rpt | grep -v mock_component | grep -v base_component | grep -v loopback | grep -v tools | grep -v "common/log" | grep -v "common/exectime" | grep -v "common/types.go" | grep -v "internal/stats_manager" | grep -v "main.go" | grep -v "component/azstorage/azauthmsi.go" | grep -v "component/azstorage/azauthspn.go" | grep -v "component/stream" | grep -v "component/azstorage/azauthcli.go" > ./blobfuse2_coverage.rpt 
+                cat ./blobfuse2_coverage_raw.rpt | grep -v mock_component | grep -v base_component | grep -v loopback | grep -v tools | grep -v "common/log" | grep -v "common/exectime" | grep -v "common/types.go" | grep -v "internal/stats_manager" | grep -v "main.go" | grep -v "component/azstorage/azauthmsi.go" | grep -v "component/azstorage/azauthspn.go" | grep -v "component/stream" | grep -v "component/custom" | grep -v "component/azstorage/azauthcli.go" | grep -v "exported/exported.go" |  grep -v "component/block_cache/stream.go" > ./blobfuse2_coverage.rpt 
                 go tool cover -func blobfuse2_coverage.rpt  > ./blobfuse2_func_cover.rpt
                 go tool cover -html=./blobfuse2_coverage.rpt -o ./blobfuse2_coverage.html
                 go tool cover -html=./blobfuse2_ut.cov -o ./blobfuse2_ut.html