Release 2.8.2.8.rc1 · hopshadoop/hops-gpu-management-impl-nvidia

    Release notes - hops - Version 2.8.2.8

Bug

[HOPS-28] - hdfs dfs -put doesnt work with multiple NNs (but -copyFromLocal does)
[HOPS-920] - BPOfferService.blockReport should call blockReportCompleted in a finnaly
[HOPS-985] - apply HDFS-7394
[HOPS-986] - apply HDFS-7399
[HOPS-990] - apply HDFS-7406
[HOPS-991] - apply HDFS-7225
[HOPS-992] - apply HDFS-7374
[HOPS-1002] - apply HDFS-4882
[HOPS-1005] - apply HDFS-7303
[HOPS-1007] - apply HDFS-7444
[HOPS-1018] - apply HDFS-7472
[HOPS-1025] - apply HDFS-7481
[HOPS-1026] - apply HDFS-5578
[HOPS-1028] - apply HDFS-7515
[HOPS-1029] - apply HDFS-7497
[HOPS-1030] - apply HDFS-7517
[HOPS-1034] - apply HDFS-6425
[HOPS-1038] - NPE During NN restart.
[HOPS-1039] - A bug in FSNameSystem.addSafeBlocks() is overloading the database
[HOPS-1040] - the CacheReplicationMonitor and ReplicationMonitor for block should not kill the namenode in case of StorageException.
[HOPS-1043] - apply HDFS-7494
[HOPS-1045] - Change log level for LockUpgradeException
[HOPS-1046] - JVM optimisation remove error stack trace
[HOPS-1047] - update ndb to 7.6.9
[HOPS-1052] - apply HDFS-7431
[HOPS-1053] - apply HDFS-7552
[HOPS-1054] - apply HDFS-7443
[HOPS-1061] - apply HDFS-7596
[HOPS-1064] - apply HDFS-7533
[HOPS-1065] - apply HDFS-5445
[HOPS-1069] - apply HDFS-7457
[HOPS-1074] - apply HDFS-7632
[HOPS-1076] - apply HDFS-7637
[HOPS-1077] - apply HDFS-7641
[HOPS-1078] - apply HDFS-7610
[HOPS-1079] - apply HDFS-3443
[HOPS-1080] - apply HDFS-7496
[HOPS-1081] - apply HDFS-7548
[HOPS-1083] - apply HDFS-7575
[HOPS-1084] - apply HDFS-7644
[HOPS-1088] - apply HDFS-7659
[HOPS-1089] - apply HDFS-7224
[HOPS-1090] - apply HDFS-49
[HOPS-1092] - apply HDFS-7566
[HOPS-1093] - Fixing NPE in Lease Manager caused by HOPS-1031
[HOPS-1094] - apply HDFS-7677
[HOPS-1098] - apply HDFS-7696
[HOPS-1099] - Change inode id to long in the scripts that create and drop disk data tables
[HOPS-1100] - BRTrackingService.assignWork should be one transaction
[HOPS-1101] - NN started after the cluster get out of safemode never initializeReplQueues
[HOPS-1102] - Tuple did not exist in RertryCacheCleaner
[HOPS-1106] - HDFS admin ui broken behind proxy
[HOPS-1112] - Fix firstBlockReport check in processReport fn in BlockManager for Multi NN
[HOPS-1117] - apply HDFS-7108
[HOPS-1118] - Database query fails for large number of buckets per datanode
[HOPS-1121] - Remove DB connect information from the log
[HOPS-1122] - apply HDFS-7734
[HOPS-1123] - apply HDFS-7719
[HOPS-1124] - apply HDFS-7709
[HOPS-1127] - apply HDFS-7698
[HOPS-1128] - apply HDFS-7741
[HOPS-1132] - apply HDFS-7647
[HOPS-1133] - NormalUserCertLocServiceHopsSSLCheck should not throw an exception if AppId is null
[HOPS-1134] - apply HDFS-7756
[HOPS-1135] - apply HDFS-7744
[HOPS-1136] - apply HDFS-7714
[HOPS-1141] - apply HDFS-7736
[HOPS-1145] - Use user's username when generating JWT instead of project specific username
[HOPS-1149] - Acquire exclusive lock on fd when updating crypto material and remove IO from NM heartbeat
[HOPS-1150] - Problems in Users&Groups cache invalidation leads to permission denied exception when a user belongs to multiple groups
[HOPS-1151] - Reading all replicas for a bucket is a FTS.
[HOPS-1155] - ClientCache invalidates running Client
[HOPS-1159] - QuotaUpdateManager should not sleep when there is parent updates to apply
[HOPS-1165] - apply HDFS-7721
[HOPS-1166] - apply HDFS-7686
[HOPS-1167] - apply HDFS-7778
[HOPS-1170] - apply HDFS-7798
[HOPS-1174] - apply HDFS-6662
[HOPS-1178] - apply HDFS-7788
[HOPS-1184] - apply HDFS-7807
[HOPS-1185] - apply HDFS-7805
[HOPS-1187] - apply HADOOP-10508
[HOPS-1189] - apply HDFS-7008
[HOPS-1191] - apply HDFS-7495
[HOPS-1195] - apply HDFS-7819
[HOPS-1196] - apply HDFS-7774
[HOPS-1197] - apply HDFS-6753
[HOPS-1201] - BlockManager.processBucketInternal compute bucket's hash with wrong block
[HOPS-1205] - apply HDFS-7879
[HOPS-1207] - ApplicationAttemptState and ApplicationState do not fit in NDB column
[HOPS-1209] - apply HDFS-1522
[HOPS-1211] - apply HDFS-7885
[HOPS-1212] - apply HDFS-7818
[HOPS-1213] - merge HADOOP-9478
[HOPS-1214] - set the limit of block reporting per NN
[HOPS-1215] - Set storage type for Small Files stored in the database
[HOPS-1216] - remove foreign key on yarn_applicationattemptstate
[HOPS-1219] - apply HDFS-7830
[HOPS-1221] - apply HDFS-7880
[HOPS-1222] - apply HDFS-6833
[HOPS-1223] - apply HDFS-7722
[HOPS-1224] - apply HDFS-7926
[HOPS-1226] - apply HDFS-2605
[HOPS-1227] - apply HDFS-7915
[HOPS-1228] - apply HDFS-7886
[HOPS-1232] - apply HDFS-5356
[HOPS-1234] - apply HDFS-7953
[HOPS-1235] - apply HDFS-7948
[HOPS-1236] - apply HDFS-7950
[HOPS-1237] - apply HDFS-7951
[HOPS-1238] - apply HDFS-7945
[HOPS-1240] - apply HDFS-7943
[HOPS-1241] - apply HDFS-7587
[HOPS-1242] - apply HDFS-7816
[HOPS-1243] - apply HDFS-7932
[HOPS-1244] - apply HDFS-7930
[HOPS-1248] - apply HDFS-7957
[HOPS-1249] - apply HDFS-7881
[HOPS-1251] - apply HDFS-7884
[HOPS-1252] - apply HDFS-7960
[HOPS-1254] - TestFileTruncate regularly fail
[HOPS-1255] - apply HDFS-7956
[HOPS-1256] - apply HDFS-3325
[HOPS-1257] - apply HDFS-7961
[HOPS-1259] - Tupple does not exist in blockmanager.checkMisReplicatedRangeQueue
[HOPS-1265] - NameNode binds to wrong IP address in MiniDFS
[HOPS-1266] - MiniDFSCluster should not overwrite DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY
[HOPS-1268] - apply HDFS-7990
[HOPS-1271] - apply HDFS-7742
[HOPS-1272] - apply HDFS-8002
[HOPS-1273] - apply HDFS-7261
[HOPS-1274] - apply HDFS-7748
[HOPS-1277] - apply HDFS-6945
[HOPS-1280] - apply HDFS-8026
[HOPS-1281] - apply HDFS-7922
[HOPS-1283] - apply HDFS-7916
[HOPS-1289] - apply HDFS-8034
[HOPS-1290] - apply HDFS-8039
[HOPS-1291] - apply HDFS-7996
[HOPS-1292] - apply HDFS-8051
[HOPS-1293] - apply HDFS-7999
[HOPS-1300] - apply HDFS-8072
[HOPS-1301] - apply HDFS-8085
[HOPS-1302] - apply HDFS-8046
[HOPS-1304] - apply HDFS-7725
[HOPS-1305] - apply HDFS-8025
[HOPS-1308] - apply HDFS-8096
[HOPS-1310] - apply HDFS-8099
[HOPS-1313] - Suppress transient exceptions
[HOPS-1314] - NullPointerException in QuotaUpdateManager
[HOPS-1315] - NullPointerException in PROCESS_TIMEDOUT_PENDING_BLOCK
[HOPS-1318] - fixing compatibility with hdfs-mount
[HOPS-1319] - Reduce number of retries for faster failover
[HOPS-1320] - Fix TestBalancer
[HOPS-1321] - Read me point to wrong maven repositor
[HOPS-1322] - Fix TestNameNodeMetrics
[HOPS-1323] - Fix removing leases
[HOPS-1324] - Fix StorageCallPreventedException in NameNodeRpcServer.commitBlockSynchronization
[HOPS-1325] - Retry cache return success on transient exception
[HOPS-1326] - Batches size in BlockManager.processReport should be configurable
[HOPS-1327] - RPC server gets stuck in TLS handshake protocol and becomes unresponsive
[HOPS-1328] - fix compilation of hops-gpu-management-impl-amd on MACOS
[HOPS-1329] - Fix HTTP address in hdfs_le_descriptor table.
[HOPS-1330] - TestBlockScanner.testMultipleBlockPoolScanning regularly fail
[HOPS-1331] - DFSClient.setQuotaByStorageType should do the call on the leader NN
[HOPS-1333] - Fix NN web UI
[HOPS-1335] - Fall-back to application user if we can't extract Hopsworks user when generating application JWT
[HOPS-1337] - NullPointer in Clusterj
[HOPS-1338] - RetryCacheCleaner throw OutOfMemoryError
[HOPS-1339] - vulnerability in com.fasterxml.jackson.core:jackson-databind 2.9.5
[HOPS-1341] - Rename, Concat, Delete and Create Files operations do not properly update the number of immediate children for a directory.
[HOPS-1342] - Remove hostname check from SuperUserHopsSSLCheck
[HOPS-1343] - Recover inode if reset subtree lock fails
[HOPS-1344] - Add more information about failed transaction
[HOPS-1345] - Add more information in on_going_subtree_ops table
[HOPS-1352] - failed block report should not be counted as if a block report had been received for the storage
[HOPS-1353] - Lock exception in addStoredBlockImmediateTx
[HOPS-1354] - NullPointerException in getLeastLoadedNode after NN die
[HOPS-1355] - BRTrackingService.canProcessMoreBR should also remove block reports runnin on dead NN
[HOPS-1356] - big dfs.namenode.processReport.batchsize can kill database
[HOPS-1357] - Bump Gson dependency version
[HOPS-1358] - NPE in pushing updated crypto material when RM is recovering
[HOPS-1359] - removeStoredBlocksTx taking default lock when it should take write lock
[HOPS-1360] - Tuple does not exist exception in removeSafeBlock
[HOPS-1361] - NullPointerException in blockReportCompleted
[HOPS-1362] - When the cluster is fully restarted the SafeBlocks table and the Active block report table should be emptied
[HOPS-1363] - Fix FS cache issues when multiple users use the same Configuration object
[HOPS-1364] - Time in RMAppSecurity handlers does not account for timezones
[HOPS-1365] - Balancer slow and killing the database
[HOPS-1367] - Locking error when datanode repport added blocks
[HOPS-1370] - NPE when RM is recovering with Yarn JWT enabled and app is running
[HOPS-1376] - deadlock in BPServiceActor.offerService
[HOPS-1389] - NullPointerException when running FSCK
[HOPS-1390] - HDFS concat should keep srcs order Export
[HOPS-1516] - The Namenode and Reousemanager should stop when the database is dead
[HOPS-1522] - bug in DBRMStateStore.checkAvailableResources overload the database

Improvement

[HOPS-38] - Filesystem checkPath failed while using a different NN ip in the path
[HOPS-826] - Support for HDFS mount
[HOPS-984] - apply HDFS-7386
[HOPS-987] - apply HDFS-7279
[HOPS-989] - apply HDFS-7404
[HOPS-993] - apply HDFS-7409
[HOPS-994] - apply HDFS-7413
[HOPS-996] - apply HDFS-7415
[HOPS-999] - apply HDFS-7420
[HOPS-1000] - apply HDFS-7331
[HOPS-1001] - apply HDFS-7403
[HOPS-1003] - apply HDFS-7419
[HOPS-1004] - apply HDFS-7436
[HOPS-1006] - apply HDFS-7310
[HOPS-1008] - apply HDFS-7210
[HOPS-1009] - apply HDFS-4750
[HOPS-1014] - apply HDFS-7438
[HOPS-1015] - apply HDFS-7462
[HOPS-1016] - apply HDFS-6735
[HOPS-1017] - apply HDFS-7468
[HOPS-1019] - apply HDFS-7474
[HOPS-1020] - apply HDFS-7459
[HOPS-1021] - apply HDFS-7476
[HOPS-1022] - apply HDFS-7384
[HOPS-1024] - apply HDFS-7498
[HOPS-1027] - apply HDFS-7463
[HOPS-1031] - apply HDFS-7509
[HOPS-1032] - apply HDFS-7426
[HOPS-1033] - apply HDFS-7506
[HOPS-1044] - apply HDFS-7528
[HOPS-1050] - apply HDFS-7531
[HOPS-1051] - apply HDFS-7543
[HOPS-1055] - apply HDFS-7555
[HOPS-1056] - apply HDFS-7557
[HOPS-1057] - apply HDFS-7484
[HOPS-1059] - apply HDFS-7579
[HOPS-1060] - apply HDFS-7182
[HOPS-1062] - apply HDFS-7323
[HOPS-1063] - apply HDFS-7326
[HOPS-1066] - apply HDFS-7600
[HOPS-1067] - apply HDFS-7585
[HOPS-1068] - apply HDFS-2219
[HOPS-1073] - apply HDFS-7591
[HOPS-1075] - apply HDFS-7573
[HOPS-1087] - apply HDFS-7638
[HOPS-1091] - apply HDFS-7683
[HOPS-1095] - apply HDFS-7706
[HOPS-1096] - apply HDFS-5631
[HOPS-1097] - apply HDFS-7681
[HOPS-1108] - Change the plus operator to XOR for combining hashes in the block reporting system
[HOPS-1110] - apply HDFS-7712
[HOPS-1115] - Change FTIS to PPIS operation in Excess Replica Map
[HOPS-1125] - apply HDFS-7270
[HOPS-1129] - apply HDFS-7732
[HOPS-1131] - apply HDFS-7743
[HOPS-1142] - apply HDFS-7771
[HOPS-1143] - apply HDFS-6133
[HOPS-1144] - apply HDFS-7761
[HOPS-1148] - apply HDFS-7694
[HOPS-1154] - Clean code for block hash undo in file append and update pipe line RPCs
[HOPS-1156] - fix br not stale variable
[HOPS-1157] - Do not send the blocks in the buckets whose hash matches in order to reduce the size of the block reports
[HOPS-1160] - apply HDFS-7684
[HOPS-1161] - apply HDFS-7790
[HOPS-1162] - apply HDFS-316
[HOPS-1164] - apply HDFS-7430
[HOPS-1169] - apply HDFS-7604
[HOPS-1171] - apply HDFS-7703
[HOPS-1172] - apply HDFS-7797
[HOPS-1173] - apply HDFS-7795
[HOPS-1175] - apply HDFS-7780
[HOPS-1176] - apply HDFS-7772
[HOPS-1180] - apply HDFS-7773
[HOPS-1186] - apply HADOOP-10285
[HOPS-1188] - apply HADOOP-10376
[HOPS-1190] - apply HDFS-7668
[HOPS-1192] - apply HDFS-7467
[HOPS-1193] - apply HDFS-7537
[HOPS-1194] - apply HDFS-7832
[HOPS-1198] - apply HDFS-7308
[HOPS-1199] - apply HDFS-7685
[HOPS-1200] - apply HDFS-5853
[HOPS-1202] - apply HDFS-7439
[HOPS-1203] - apply HDFS-7789
[HOPS-1204] - apply HDFS-6565
[HOPS-1206] - apply HDFS-7434
[HOPS-1210] - apply HDFS-7855
[HOPS-1218] - apply HDFS-7411
[HOPS-1220] - apply HDFS-7491
[HOPS-1225] - apply HDFS-7435
[HOPS-1230] - apply HDFS-2360
[HOPS-1245] - apply HDFS-7962
[HOPS-1246] - apply HDFS-7835
[HOPS-1247] - apply HDFS-7829
[HOPS-1250] - apply HDFS-7917
[HOPS-1258] - apply HDFS-7875
[HOPS-1260] - apply HDFS-7854
[HOPS-1261] - apply HDFS-7713
[HOPS-1262] - apply HDFS-7928
[HOPS-1263] - apply HDFS-7918
[HOPS-1267] - apply HDFS-7410
[HOPS-1270] - apply HDFS-7890
[HOPS-1275] - apply HDFS-7944
[HOPS-1276] - apply HDFS-7671
[HOPS-1279] - apply HDFS-7978
[HOPS-1282] - apply HDFS-8008
[HOPS-1284] - apply HDFS-7888
[HOPS-1285] - apply HDFS-7954
[HOPS-1286] - apply HDFS-8035
[HOPS-1287] - apply HDFS-7893
[HOPS-1294] - apply HDFS-8071
[HOPS-1295] - apply HDFS-8049
[HOPS-1296] - apply HDFS-5215
[HOPS-1297] - apply HDFS-8079
[HOPS-1298] - apply HDFS-8073
[HOPS-1299] - apply HDFS-8080
[HOPS-1303] - apply HDFS-8076
[HOPS-1306] - apply HDFS-8089
[HOPS-1307] - apply HDFS-7979
[HOPS-1309] - apply HDFS-8101
[HOPS-1311] - Adapt protocol for renewing service JWT to Hopsworks changes
[HOPS-1332] - Mvoing to NDB 7.6.10
[HOPS-1334] - Perform TLS handshake in Reader threads
[HOPS-1336] - Fix rename operation for under replicated files
[HOPS-1340] - Fix the logs for deleted blocks
[HOPS-1348] - XAttr value size uses the max size of a ndb table row - no flexibility for future possible add column changes
[HOPS-1369] - Need for more info when a transaction fail during the commit phase
[HOPS-1392] - FS operations on inodes belonging to deleted users/groups are slow

Task

[HOPS-1317] - Include instructions for building hops-gpu-management-impl-amd in README

New Feature

[HOPS-6] - Availability Zone Awareness support for HopsFS
[HOPS-695] - [Umbrella] JWT support
[HOPS-988] - apply HDFS-6982
[HOPS-1012] - Extend ssl-server.xml with passwordfile location
[HOPS-1058] - apply HDFS-7589
[HOPS-1070] - apply HDFS-7189
[HOPS-1086] - apply HDFS-3107
[HOPS-1111] - Support AMD GPUs as a schedulable and isolable resource
[HOPS-1126] - apply HDFS-7655
[HOPS-1130] - apply HDFS-7738
[HOPS-1137] - apply HDFS-7058
[HOPS-1138] - apply HDFS-7760
[HOPS-1139] - apply HDFS-7720
[HOPS-1140] - apply HDFS-7751
[HOPS-1146] - apply HDFS-7056
[HOPS-1147] - apply HDFS-3689
[HOPS-1152] - Allow disabling Users/Groups Cache on the NN side
[HOPS-1158] - apply HDFS-7723
[HOPS-1163] - apply HDFS-7776
[HOPS-1168] - apply HDFS-7775
[HOPS-1177] - apply HDFS-7656
[HOPS-1179] - apply HDFS-7814
[HOPS-1181] - apply HDFS-7740
[HOPS-1182] - apply HDFS-7806
[HOPS-1231] - apply HDFS-7838
[HOPS-1233] - apply HDFS-7940
[HOPS-1239] - apply HDFS-7054
[HOPS-1264] - apply HDFS-7824
[HOPS-1269] - apply HDFS-7700
[HOPS-1278] - apply HDFS-8009
[HOPS-1288] - apply HDFS-7811
[HOPS-1312] - Support extended attributes per file based on HDFS-2006

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.8.2.8.rc1

Bug

Improvement

Task

New Feature