org.apache.shardingsphere.sharding.route.strategy.type.standard.StandardShardingStrategy.doSharding, Is it possible to not check whether the table name returned by the sharding algorithm is in availableTableNames?

[huyu-tom]

The judgment logic is marked in the picture

reason:

1. If there is no judgment, it will be more free. For some special businesses, actual-data-nodes cannot be specified. As long as the user is sure that the database or table exists, it will be fine.
2. There is no judgment logic in ComplexShardingStrategy and HintShardingStrategy! , and the collection returned in the algorithm is directly returned, so canceling the judgment in StandardShardingStrategy will not cause other effects

[linghengqian]

Please do not click on the URL of the strange user @KrissV2 above, it is a typical binary virus. Maybe there are more direct means on the ASF infra side to punish such Github users.

[linghengqian]

@huyu-tom I don't see the Git Diff of your changes, you can try to make your changes in the master branch and execute ./mvnw clean install -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e to see if such changes break the unit tests.

[huyu-tom]

@linghengqian

The thrown exception (ShardingRouteAlgorithmException) was treated as a test expectation, causing one test to fail.

[linghengqian]

• You can simply use @org.junit.jupiter.api.Disabled on the function body of the unit test to prevent the execution of this unit test. I am still not sure how big the impact of this change is.

• If the change is particularly large, you may even consider creating a draft PR to explain the change, whether the relevant PR is opened on your public fork git or on the shardingsphere git.

[huyu-tom]

### success

[linghengqian]

• Well, this seems to be a minor change actually, if you are interested in becoming a contributor, you may consider opening a PR to activate E2E, CDC, transactional, integration tests including Docker Image builds, and further modify the unit tests of this @Disabled.

• But I think that if this change enters the master branch of shardingsphere, this behavior should not be turned on by default, which is a real incompatibility. This should be handled by adding custom shardingsphere properties and documented at https://shardingsphere.apache.org/document/current/cn/user-manual/common-config/props/ .

[huyu-tom]

Well, this seems to be a minor change actually, if you are interested in becoming a contributor, you may consider opening a PR to activate E2E, CDC, transactional, integration tests including Docker Image builds, and further modify the unit tests of this @disabled.

• Don't know the process of becoming a contributor

But I think that if this change enters the master branch of shardingsphere, this behavior should not be turned on by default, which is a real incompatibility. This should be handled by adding custom shardingsphere properties and documented at https://shardingsphere.apache.org/document/current/cn/user-manual/common-config/props/ .

• I think this should be a default, because other sharding strategies do not add similar judgments such as existence. I wonder if the standard sharding strategy should add it?

[linghengqian]

• I think this should be a default, because other sharding strategies do not add similar judgments such as existence. I wonder if the standard sharding strategy should add it?

• In this case, this doesn’t even need to be documented, you can just open a PR and modify the corresponding logic and unit test assertions.

Don't know the process of becoming a contributor

• At least you already know how to compile the project, so it seems that you just don't know how to use git? But how to use git is actually not one of the contents of shardingsphere's documentation.

• Becoming a contributor to an ASF project is no different from other external projects, and ASF does not require contributors to sign a CLA.

• You only need to 1. make your changes into commits, 2. check out a new local branch outside the master local branch, 3. fork a public git of shardingsphere, 4. push the new local branch to the new branch of this forked git, 5. open a new pull request on the shardingsphere git side.

• Refer to https://shardingsphere.apache.org/community/cn/involved/contribute/contributor/ .

[caorong]

#32724

I also meet this problem.
in my case, I simply change the code for internal use.

1. add a filter for routeTables to filter unnessary table
2. remove throw ShardingRouteAlgorithmException.

[huyu-tom]

It seems that this cannot be done. I think it can be done through the SPI mechanism. The ShardingStrategy interface should add a method, getType, to allow us to customize it. The ShardingStrategyFactory factory uses getType to determine

[linghengqian]

• You might consider opening a new issue explaining the new SPI and then submitting a PR. This is mainly because the failing unit tests are on your side. I don't know how much configurability this SPI needs to expose.
• The process for SPI to be public is similar to Make actualDataNodes expose SPI that can define expression with custom rules and add GraalVM Truffle implementation #22899, which is how the previous Row Value Expression can get rid of Groovy. The Row Value Expression functionality was exposed as SPI because there was an existing failing nativeTest under GraalVM Native Image that actually explained the vulnerability. The issue and PR descriptions do not need to be the same, as long as the document can be described normally.

[huyu-tom]

Forget it, use the composite sharding algorithm instead of the standard sharding algorithm. The composite sharding algorithm has no judgment limit. Although I only have one field, it may consume some extra performance. I see that there will be several more Maps or I need to judge whether the field is in the Map.

[huyu-tom]

Conclusion: According to the current logic design of this framework, actual-data-nodes must exist. If there is a binding table (order, order_item), order_item must keep the same order as the actual-data-node of the order table.

sql: select order.*,order_item.* from order inner join order_item where order.id = 1000

Execution logic: The order table will execute the sharding algorithm strategy to get an actual table name (such as order_1). This table name will find the index (findIndex) in the corresponding actual-data-nodes. Assuming that the index is found (index=0), get this index directly in the actual-data-nodes.get(index) of the order_item table, which is order_item_1, so according to this logic, actual-data-nodes seem to be necessary.

Based on the above logic, you can only load all the table names of all tenants, and at the same time, if you add tenants, you can modify and reload the configuration in ContextManager, based on your article https://www.yuque.com/linghengqian/meve2v/cgi5en

public class DynamicUpdatesRule {


  /**
   * 反射获取
   *
   * @param dataSource
   * @return
   */
  private static ContextManager getContextManager(final ShardingSphereDataSource dataSource) {
    try {
      Field field = ShardingSphereDataSource.class.getDeclaredField("contextManager");
      field.setAccessible(true);
      return (ContextManager) field.get(dataSource);
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

  public static void updActualDataNodesByTable(@NotNull ShardingSphereDataSource dataSource,
      @NotNull @NotBlank String schemaName,
      @NotNull @NotEmpty Map<String, String> logicTableNameActualDataNodeMap) {
    ContextManager contextManager = getContextManager(dataSource);

    Collection<RuleConfiguration> oldRuleConfigs = getContextManager(
        dataSource).getMetaDataContexts().getMetaData().getDatabases().get(schemaName)
        .getRuleMetaData().getConfigurations();

    Collection<RuleConfiguration> newRuleConfigList = new LinkedList<>();

    oldRuleConfigs.stream()
        .filter(oldRuleConfig -> oldRuleConfig instanceof ShardingRuleConfiguration)
        .map(oldRuleConfig -> (ShardingRuleConfiguration) oldRuleConfig)
        .forEach(oldAlgorithmConfig -> {
          ShardingRuleConfiguration newConfig = new ShardingRuleConfiguration();

          Collection<ShardingTableRuleConfiguration> newTableRuleConfigList = new LinkedList<>();
          Collection<ShardingTableRuleConfiguration> oldTableRuleConfigList = oldAlgorithmConfig.getTables();

          oldTableRuleConfigList.forEach(oldTableRuleConfig -> {
            final String logicTable = oldTableRuleConfig.getLogicTable();
            final String newActualDatNodes = logicTableNameActualDataNodeMap.get(logicTable);
            if (StringUtils.isNotBlank(newActualDatNodes)) {
              final var newTableRuleConfig = new ShardingTableRuleConfiguration(
                  oldTableRuleConfig.getLogicTable(), newActualDatNodes);
              newTableRuleConfig.setTableShardingStrategy(
                  oldTableRuleConfig.getTableShardingStrategy());
              newTableRuleConfig.setDatabaseShardingStrategy(
                  oldTableRuleConfig.getDatabaseShardingStrategy());
              newTableRuleConfig.setKeyGenerateStrategy(
                  oldTableRuleConfig.getKeyGenerateStrategy());
              newTableRuleConfigList.add(newTableRuleConfig);
            } else {
              newTableRuleConfigList.add(oldTableRuleConfig);
            }
          });
          newConfig.setTables(newTableRuleConfigList);
          newConfig.setAutoTables(oldAlgorithmConfig.getAutoTables());
          newConfig.setBindingTableGroups(oldAlgorithmConfig.getBindingTableGroups());
          newConfig.setDefaultDatabaseShardingStrategy(
              oldAlgorithmConfig.getDefaultDatabaseShardingStrategy());
          newConfig.setDefaultTableShardingStrategy(
              oldAlgorithmConfig.getDefaultTableShardingStrategy());
          newConfig.setDefaultKeyGenerateStrategy(
              oldAlgorithmConfig.getDefaultKeyGenerateStrategy());
          newConfig.setDefaultShardingColumn(oldAlgorithmConfig.getDefaultShardingColumn());
          newConfig.setShardingAlgorithms(oldAlgorithmConfig.getShardingAlgorithms());
          newConfig.setKeyGenerators(oldAlgorithmConfig.getKeyGenerators());
          newRuleConfigList.add(newConfig);
        });

    newRuleConfigList.stream().forEach(((rule) -> {
      contextManager.getConfigurationContextManager().alterRuleConfiguration(schemaName, rule);
    }));

    logicTableNameActualDataNodeMap.keySet().stream().forEach((tableName) -> {
      contextManager.reloadTable(contextManager.getDatabase(schemaName), schemaName, tableName);
    });
  }
}

[linghengqian]

• I never actually got around to refactoring that article. Updating it involves tracking a lot of git commits and unit tests, and that kind of work isn’t really easy to document in markdown.
• If you are interested in how to use the internal API of ShardingSphere's master branch, you can refer to the unit test of ShardingSphere Agent.
• Or you could consider taking over the work at [Event] ShardingSphere is going to be a mentor community for GSoC 2024 #30252 (comment) to support using DistSQL directly in ShardingSpehre JDBC. Currently, this GSOC task has been lost. DistSQL is something that is really intuitive and easy to document.

[linghengqian]

• I did open Fixes the outdated description of Key Generation and Hadoop in HiveServer2 integration doc #33739 to try to avoid implicit configuration of actualDataNodes. This is not intuitive.