2.8.2.3.rc0

    Release notes - hops - Version 2.8.2.3

Bug

• [HOPS-109] - Quota management of Small Files
• [HOPS-136] - Container logging does handle lost containers
• [HOPS-137] - Quota is not recorded appropriately for small files which result in negative quota when deleting files/directories
• [HOPS-140] - garbage container logs and checkpoint remaining int the database
• [HOPS-143] - Quota multiplicator fixed period does not behave according to specification
• [HOPS-150] - Test code was commited by accident and block from having more than one gpu
• [HOPS-152] - app get stuck in NEW_SAVING when user name too long
• [HOPS-155] - Store small files on the datanodes if the database disk tables are full.
• [HOPS-157] - Nullpointer and Index out of bound since HOPS-152

New Feature

• [HOPS-141] - Monitor and control CertificateLocalizationService via JMX
• [HOPS-147] - Reloadable key store manager

Improvement

• [HOPS-131] - Base price should be different for GPU resource and for General resources
• [HOPS-133] - Remove deploy-server.sh from hops-metadata-dal-impl-ndb and hops-gpu-management-impl-nvidia
• [HOPS-138] - Increase detail of log output for processed block report
• [HOPS-139] - Add another method for ContentSummary to return the results directly from inodes_attributes in quote enabled directories
• [HOPS-144] - Disabled uber pricing
• [HOPS-149] - Add app IDs in project daily cost
• [HOPS-151] - App's GPU usage in Yarn UI
• [HOPS-153] - [hops-hadoop-chef] Secure webhdfs REST API
• [HOPS-158] - Refactor HopsSSLSocketFactory and add check for Environment Variable set with crypto material

2.8.2.2

    Release notes - hops - Version 2.8.2.2

Bug

• [HOPS-95] - Differentiate certificate localization directory for RM and NMs
• [HOPS-96] - Import HDFS-6570 to fix checkAccess issues with Hive
• [HOPS-99] - Add makeQualified method to HdfsFileStatus and HdfsLocatedFileStatus
• [HOPS-100] - Fix ssl-server.xml path in HopsSSLSocketFactory
• [HOPS-102] - Erasure Coding Tests are failing because of excessive memory use
• [HOPS-104] - Missing check-refresh causes class-cast exception on block report of UC blocks.
• [HOPS-120] - TestErasureCodingManager regularly run out of memory
• [HOPS-123] - Guard for no data in phantom blocks
• [HOPS-127] - pom <repositories> and <distributionManagement> badly configured
• [HOPS-128] - libhopsyarn pointing to wrong table
• [HOPS-129] - The update schema script from 2.8.2.1 to 2.8.2.2 should update yarn_price_multiplicator
• [HOPS-130] - Container log remaining in DB for containers stoped just after creation
• [HOPS-136] - Container logging does handle lost containers
• [HOPS-140] - garbage container logs and checkpoint remaining int the database
• [HOPS-150] - Test code was commited by accident and block from having more than one gpu

New Feature

• [HOPS-98] - Quota for GPU hours
• [HOPS-124] - libhopsyarn and libhopsnvml shouldn't be included in the pom files.

Improvement

• [HOPS-94] - Improve error handling for HopsSSLSocketFactory
• [HOPS-97] - Use logicaltime per inode to order logs on the same file
• [HOPS-105] - Parallellize add-stored block after calculating report diff to improve reporting performance
• [HOPS-119] - Add certificate password as container local resource
• [HOPS-122] - Recovering ssl certificate info
• [HOPS-125] - Cleanup hops git repo from hs_err*.log files
• [HOPS-138] - Increase detail of log output for processed block report

2.8.2.1

    Release notes - hops - Version 2.8.2.1

Bug

• [HOPS-14] - remove thread in NamenodeSelector
• [HOPS-16] - Transactions deadlock exception is thrown when hundreds of clients simultaneously read the same file.
• [HOPS-20] - Lease manager dies on Transient exception.
• [HOPS-21] - Poor Synchronous implementation of Namenode Selector in HopsFS 2.7.3 causes 80-90% drop in throughput
• [HOPS-45] - Slow LS operation on directories with large number of immediate descendants
• [HOPS-47] - TestUsersGroups fail due to wrong order of list elements
• [HOPS-48] - FATAL UniqueKeyConstraintViolationException During Namenode Creation
• [HOPS-51] - hdfs native_tests failing
• [HOPS-56] - setOwner fails with ForeignKeyConstraintViolationException due to outdated cache
• [HOPS-59] - HopsSSLSocketFactory assumes there is a single proxy superuser
• [HOPS-67] - Incoming data are not processed in Server
• [HOPS-69] - Delete directoy does not work non leader namenode when quota is enabled
• [HOPS-71] - Fix permission issue when writing to Cgroup device subsystem
• [HOPS-75] - Fix Log Aggregation in Yarn
• [HOPS-76] - ProcfsBasedProcessTree cannot parse PID process tree info
• [HOPS-81] - Missing version specifier in hadoop-common dependency causes warnings
• [HOPS-82] - Hops-GPU should handle cases when a subset of GPUs are not working correctly
• [HOPS-84] - hadoop-common pom.xml must include hops-gpu-management as dependency
• [HOPS-85] - Change collate for all the remaining yarn_* tables
• [HOPS-89] - Owner not correctly set in Cgroup hierarchy
• [HOPS-90] - Cgroups broken due to bug in container-executor
• [HOPS-91] - DFSClient update NN list as superuser
• [HOPS-92] - Symbol lookup error crashes NodeManager
• [HOPS-93] - Refactor hops-gpu-manage and hops-gpu-management-impl-nvidia
• [HOPS-103] - Concurrency bug in blockreporting causes iterator infinite loop
• [HOPS-107] - Small file's read fails when a decommissioned node is selected as a phanton datanode
• [HOPS-108] - Get HDFS username from environment variable in libhdfs
• [HOPS-110] - Change column type of small files tables from varchar to varbinary
• [HOPS-111] - Support set replication api for small files
• [HOPS-112] - Implement seek api for small files
• [HOPS-114] - Can't build without -Pndb since [HOPS-103]
• [HOPS-116] - Spark applications fail when small fiels are read using read(ByteBuffer) api
• [HOPS-123] - Guard for no data in phantom blocks
• [HOPS-127] - pom <repositories> and <distributionManagement> badly configured
• [HOPS-128] - libhopsyarn pointing to wrong table
• [HOPS-130] - Container log remaining in DB for containers stoped just after creation
• [HOPS-136] - Container logging does handle lost containers
• [HOPS-140] - garbage container logs and checkpoint remaining int the database
• [HOPS-150] - Test code was commited by accident and block from having more than one gpu

Sub-task

• [HOPS-41] - NPE in Rename for metaEnabled dirs and subdirs
• [HOPS-42] - Optimize logging of rename in the same meta enabled directory
• [HOPS-43] - Delete after rename of a metaEnabled directory fail
• [HOPS-55] - Refactoring block report processing
• [HOPS-58] - Correcting code quality issues in FSNamesystem
• [HOPS-60] - Remove remains of Federation from HopsFS
• [HOPS-62] - Replace Block Report RPC with Bucketized RPC
• [HOPS-63] - Create a HashBucket entity
• [HOPS-64] - Replace the block reporting mechanism with the hash-based one

New Feature

• [HOPS-2] - Improve Small File Performance in HopsFS
• [HOPS-4] - Uber style resource price
• [HOPS-7] - TLS support for Hadoop RPC/IPC
• [HOPS-11] - GPU support for HopsYARN (Umbrella)
• [HOPS-12] - Create an abstraction layer for GPU devices
• [HOPS-13] - Create an implementation layer for GPU devices
• [HOPS-19] - Truncate support in HDFS
• [HOPS-23] - avoid running app to be able to access other app data
• [HOPS-124] - libhopsyarn and libhopsnvml shouldn't be included in the pom files.

Improvement

• [HOPS-17] - Optimizing addBlock and incrementalBlockReport RPC for large files
• [HOPS-22] - StorageCallPrevented Exception must not be swallowed
• [HOPS-26] - Clean up old MetadataLog code in TransactionalRequestHandler
• [HOPS-29] - ByteBuffer Mem Leak Problem
• [HOPS-30] - Expose Parameters For TX Retry
• [HOPS-31] - Add support to GPUs for hops-metadata-dal
• [HOPS-32] - Add support to GPUs for hops-metadata-dal-impl-ndb
• [HOPS-34] - HdfsFileStatus should return the fileId
• [HOPS-35] - Optimizing logging of MetaEnabled directories
• [

2.8.2.2.rc0

    Release notes - hops - Version 2.8.2.2

Bug

• [HOPS-95] - Differentiate certificate localization directory for RM and NMs
• [HOPS-96] - Import HDFS-6570 to fix checkAccess issues with Hive
• [HOPS-99] - Add makeQualified method to HdfsFileStatus and HdfsLocatedFileStatus
• [HOPS-100] - Fix ssl-server.xml path in HopsSSLSocketFactory
• [HOPS-102] - Erasure Coding Tests are failing because of excessive memory use
• [HOPS-104] - Missing check-refresh causes class-cast exception on block report of UC blocks.
• [HOPS-120] - TestErasureCodingManager regularly run out of memory
• [HOPS-123] - Guard for no data in phantom blocks

New Feature

• [HOPS-98] - Quota for GPU hours
• [HOPS-124] - libhopsyarn and libhopsnvml shouldn't be included in the pom files.

Improvement

• [HOPS-94] - Improve error handling for HopsSSLSocketFactory
• [HOPS-97] - Use logicaltime per inode to order logs on the same file
• [HOPS-105] - Parallellize add-stored block after calculating report diff to improve reporting performance
• [HOPS-119] - Add certificate password as container local resource
• [HOPS-122] - Recovering ssl certificate info
• [HOPS-125] - Cleanup hops git repo from hs_err*.log files

2.8.2.1.rc0

   Release notes - hops - Version 2.8.2.1

Bug

• [HOPS-14] - remove thread in NamenodeSelector
• [HOPS-16] - Transactions deadlock exception is thrown when hundreds of clients simultaneously read the same file.
• [HOPS-20] - Lease manager dies on Transient exception.
• [HOPS-21] - Poor Synchronous implementation of Namenode Selector in HopsFS 2.7.3 causes 80-90% drop in throughput
• [HOPS-45] - Slow LS operation on directories with large number of immediate descendants
• [HOPS-47] - TestUsersGroups fail due to wrong order of list elements
• [HOPS-48] - FATAL UniqueKeyConstraintViolationException During Namenode Creation
• [HOPS-51] - hdfs native_tests failing
• [HOPS-56] - setOwner fails with ForeignKeyConstraintViolationException due to outdated cache
• [HOPS-59] - HopsSSLSocketFactory assumes there is a single proxy superuser
• [HOPS-67] - Incoming data are not processed in Server
• [HOPS-69] - Delete directoy does not work non leader namenode when quota is enabled
• [HOPS-71] - Fix permission issue when writing to Cgroup device subsystem
• [HOPS-75] - Fix Log Aggregation in Yarn
• [HOPS-76] - ProcfsBasedProcessTree cannot parse PID process tree info
• [HOPS-78] - Fix TestFsck test
• [HOPS-81] - Missing version specifier in hadoop-common dependency causes warnings
• [HOPS-82] - Hops-GPU should handle cases when a subset of GPUs are not working correctly
• [HOPS-84] - hadoop-common pom.xml must include hops-gpu-management as dependency
• [HOPS-85] - Change collate for all the remaining yarn_* tables
• [HOPS-89] - Owner not correctly set in Cgroup hierarchy

Sub-task

• [HOPS-41] - NPE in Rename for metaEnabled dirs and subdirs
• [HOPS-42] - Optimize logging of rename in the same meta enabled directory
• [HOPS-43] - Delete after rename of a metaEnabled directory fail
• [HOPS-55] - Refactoring block report processing
• [HOPS-58] - Correcting code quality issues in FSNamesystem
• [HOPS-60] - Remove remains of Federation from HopsFS
• [HOPS-62] - Replace Block Report RPC with Bucketized RPC
• [HOPS-63] - Create a HashBucket entity
• [HOPS-64] - Replace the block reporting mechanism with the hash-based one

New Feature

• [HOPS-2] - Improve Small File Performance in HopsFS
• [HOPS-4] - Uber style resource price
• [HOPS-7] - TLS support for Hadoop RPC/IPC
• [HOPS-11] - GPU support for HopsYARN (Umbrella)
• [HOPS-12] - Create an abstraction layer for GPU devices
• [HOPS-13] - Create an implementation layer for GPU devices
• [HOPS-19] - Truncate support in HDFS
• [HOPS-23] - avoid running app to be able to access other app data

Improvement

• [HOPS-17] - Optimizing addBlock and incrementalBlockReport RPC for large files
• [HOPS-22] - StorageCallPrevented Exception must not be swallowed
• [HOPS-26] - Clean up old MetadataLog code in TransactionalRequestHandler
• [HOPS-29] - ByteBuffer Mem Leak Problem
• [HOPS-30] - Expose Parameters For TX Retry
• [HOPS-31] - Add support to GPUs for hops-metadata-dal
• [HOPS-32] - Add support to GPUs for hops-metadata-dal-impl-ndb
• [HOPS-34] - HdfsFileStatus should return the fileId
• [HOPS-35] - Optimizing logging of MetaEnabled directories
• [HOPS-37] - Append To file using HDFS clients (version >= v2.4) fails due to NPE on the HDFS clinet side
• [HOPS-49] - Making HopsFS and the DAL Java 8 syntax compliant
• [HOPS-54] - Remove unnecessary debug printouts from dal-impl-ndb
• [HOPS-57] - Hash based block-reporting for improved block reporting speed
• [HOPS-61] - Block Management Refactoring
• [HOPS-65] - update to follow hadoop 2.8.2 updates
• [HOPS-66] - Add a script to upgrade Clusterj with hopsfs fixes and Clusterj Native in Kompics repo
• [HOPS-68] - Pass the keystore and truststore password along with the material
• [HOPS-79] - Apply patch YARN-7083 from upstream fixing Log Aggregation
• [HOPS-83] - Cgroups hierarchy should be created automatically