add mm, sqrt, div operators for complex tensor to support OneScience #10269
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r_and_complex_attr
3 tasks
levi131
pushed a commit
that referenced
this pull request
Jun 5, 2023
) ### Original requirements 现有的 conj kernel, real kernel, imag kernel 使用 KernelUtil 来进行组织，在 kernel 层面复数数据类型和实数数据类型的调用 conj 无法实现统一。故移除原有 KernelUtil 的实现，使用 ElementwiseUnary 的 primitive 进行实现 - [x] 新增 `kConj` 这一 UnaryOp，使用 ElementwiseUnary 的 Primitive 实现 conj，并移除原有 conj 的 KernelUtil - [x] 新增 `kReal` 和 `kRealGrad` UnaryOp，使用 ElementwiseUnary 的 Primitive 实现 real 和 real_grad，并移除原有 real 和 real_grad 的 KernelUtil - [x] 新增 `kImag` 和 `kImagGrad` UnaryOp，使用 ElementwiseUnary 的 Primitive 实现 imag 和 imag_grad，并移除原有 imag 和 imag_grad 的 KernelUtil ## 注意： 复数基础设施建设系列 pr: 1. 使用 primitive 来实现 conj, real 等常见复数算子： #10281 2. 将现有支持复数数据类型的算子测例迁移到 autotest 模块中，以使复数算子复用实数算子的测试用例：#10284 3. 继续拓展支持复数数据类型的算子，比如 matmul, sqrt, div 等：#10269 依赖关系： 本 pr 基于：[pr2](#10284) 和 [pr3](#10269) 的基础，请优先 merge 此 pr
L-Xiafeng
pushed a commit
that referenced
this pull request
Jun 26, 2023
### Original requirements **Autotest**: We found that the previous testing of operators supporting complex tensor was not complete. We decided to reuse the real tensor operator tests to ensure completeness. Since complex tensor tests are supported in the `autotest` module from pr (#10027) , in this pr we applied the autotest module to the tests of complex tensor operators already available in Oneflow **Fix**: In addition, the autograd rules for some previous operators of complex numbers might not conform to the convention of ["Conjugate Wirtinger Derivative"](https://en.wikipedia.org/wiki/Wirtinger_derivatives). We have fixed these bugs in this pr at the same time. #### Main Works **Applying `autotest` module on existing operators that have already support complex tensor:** `Complex and Real Behave the Same Way`: means we don't need to add conjugate operation in op grad. Because regardless of whether the input data type involved in the operation is real or complex, the gradient result using the winterger derivative is the same as the real derivative rule, `Grad Not Supported in OF`: means the grad of this op is not supported in oneflow - broadcast_elementwise_binary | Op | complex type | Backend | Using autotest | conjugate Wirtinger derivative | |:-----:|:-------------:|:------:|:------:|:------:| | Add | cp64, cp128 | CPU, CUDA | DONE | Complex and Real Behave the Same Way | | Mul | cp64, cp128 | CPU, CUDA | DONE | DONE | | Sub | cp64, cp128 | CPU, CUDA | DONE | Complex and Real Behave the Same Way | | Equal | cp64, cp128 | CPU, CUDA | DONE | Complex and Real Behave the Same Way | | NotEqual | cp64, cp128 | CPU, CUDA | DONE | Grad Not Supported in OF | - broadcast_elementwise_unary | Op | complex type | Backend | Using autotest | conjugate Wirtinger derivative | |:----------:|:-------------:|:------:|:------:|:------:| | Cast | cp64, cp128 | CPU, CUDA | DONE | Complex and Real Behave the Same Way | - other exisiting operations | Op | complex type | Backend | Using autotest | conjugate Wirtinger derivative | |:----------:|:-------------:|:------:|:------:|:------:| | constant_pad | cp64, cp128 | CPU, CUDA | Done | Complex and Real Behave the Same Way | | reduce_sum | cp64, cp128 | CPU, CUDA | TO-DO | Complex and Real Behave the Same Way | ## 注意： 复数基础设施建设系列 pr: 1. 使用 primitive 来实现 conj, real 等常见复数算子： #10281 2. 将现有支持复数数据类型的算子测例迁移到 autotest 模块中，以使复数算子复用实数算子的测试用例：#10284 3. 继续拓展支持复数数据类型的算子，比如 matmul, sqrt, div 等：#10269 依赖关系： 本 pr 基于：[pr1](https://github.com/Oneflow-Inc/oneflow/pull/10281)，需要在 merge [pr1](#10281) 后，再 Merge 本 pr
format
format
format
format
fix compile error
rm useless import
L-Xiafeng
requested review from
oneflow-ci-bot
and removed request for
oneflow-ci-bot
set test_complex
complex test
test complex
test complex
skip test out of memory
L-Xiafeng
approved these changes
Jun 30, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Original requirements
AI for Science models FNO use a series of operators such as sqrt, mm, div, etc. for complex tensor. So we need to support complex tensor for these ops.
Main Works
Support existing ops for complex tensor in this pr as below:
broadcast_elementwise_binary
broadcast_elementwise_unary
other exisiting operations
注意：
复数基础设施建设系列 pr:
依赖关系：
本 pr 基于：pr1 和 pr2 ，需要在 merge pr1 和 pr2 后，再 Merge 本 pr