Skip to content

Commit 7add7d9

Browse files
AlexandreEichenbergerAlexandreEichenberger
authored
Merge branch 'main' into simd-framwork-v1
2 parents 3fbd9fe + 37b8393 commit 7add7d9

File tree

14 files changed

+419
-140
lines changed

14 files changed

+419
-140
lines changed

‎docs/Dialects/onnx.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3589,6 +3589,63 @@ where the mean and variance are computed per instance per group of channels, and
35893589
groups `num_groups` should be divisible by the number of channels so that there are
35903590
an equal number of channels per group.
35913591

3592+
The overall computation has two stages: the first stage normalizes the elements to
3593+
have zero mean and unit variance for each instance in each group, and the second
3594+
stage scales and shifts the results of the first stage. The floating-point precision
3595+
used in the first stage is determined by the `stash_type` attribute. For example,
3596+
if `stash_type` is 1, the operator casts all input variables to 32-bit float,
3597+
performs the computation, and finally casts the normalized results back to the
3598+
original type of `X`. The second stage does not depend on `stash_type`.
3599+
3600+
When the number of groups is the same as the number of channels, this operator is
3601+
equivalent to InstanceNormalization. When there is only one group, this operator
3602+
is equivalent to LayerNormalization.
3603+
3604+
Traits: `AlwaysSpeculatableImplTrait`
3605+
3606+
Interfaces: `ConditionallySpeculatable`, `NoMemoryEffect (MemoryEffectOpInterface)`, `ShapeHelperOpInterface`, `ShapeInferenceOpInterface`
3607+
3608+
Effects: `MemoryEffects::Effect{}`
3609+
3610+
#### Attributes:
3611+
3612+
<table>
3613+
<tr><th>Attribute</th><th>MLIR Type</th><th>Description</th></tr>
3614+
<tr><td><code>epsilon</code></td><td>::mlir::FloatAttr</td><td>32-bit float attribute</td></tr>
3615+
<tr><td><code>num_groups</code></td><td>::mlir::IntegerAttr</td><td>64-bit signed integer attribute</td></tr>
3616+
<tr><td><code>stash_type</code></td><td>::mlir::IntegerAttr</td><td>64-bit signed integer attribute</td></tr>
3617+
</table>
3618+
3619+
#### Operands:
3620+
3621+
| Operand | Description |
3622+
| :-----: | ----------- |
3623+
| `X` | tensor of bfloat16 type values or tensor of 16-bit float values or tensor of 32-bit float values or tensor of 64-bit float values
3624+
| `scale` | tensor of bfloat16 type values or tensor of 16-bit float values or tensor of 32-bit float values or tensor of 64-bit float values
3625+
| `bias` | tensor of bfloat16 type values or tensor of 16-bit float values or tensor of 32-bit float values or tensor of 64-bit float values
3626+
3627+
#### Results:
3628+
3629+
| Result | Description |
3630+
| :----: | ----------- |
3631+
| `Y` | tensor of bfloat16 type values or tensor of 16-bit float values or tensor of 32-bit float values or tensor of 64-bit float values
3632+
3633+
### `onnx.GroupNormalizationV18` (ONNXGroupNormalizationV18Op)
3634+
3635+
_ONNX GroupNormalization operation_
3636+
3637+
A GroupNormalization function. Carries out group normalization as described in
3638+
the paper https://arxiv.org/abs/1803.08494
3639+
3640+
This operator transforms input according to
3641+
```
3642+
y = scale * (x - mean) / sqrt(variance + epsilon) + bias,
3643+
```
3644+
where the mean and variance are computed per instance per group of channels, and
3645+
`scale` and `bias` should be specified for each group of channels. The number of
3646+
groups `num_groups` should be divisible by the number of channels so that there are
3647+
an equal number of channels per group.
3648+
35923649
When the number of groups is the same as the number of channels, this operator is
35933650
equivalent to InstanceNormalization. When there is only one group, this operator
35943651
is equivalent to LayerNormalization.

‎docs/SupportedONNXOps-NNPA.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@
33

44
# Supported ONNX Operation for Target *NNPA*.
55

6-
Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitations are listed when applicable. This documentation highlights the minimum and maximum opset versions that are fully supported by onnx-mlir and not the version changes.
6+
Onnx-mlir currently supports ONNX operations targeting up to opset 21. Limitations are listed when applicable. This documentation highlights the minimum and maximum opset versions that are fully supported by onnx-mlir and not the version changes.
77

88
* Operations are defined by the [ONNX Standard](https://github.com/onnx/onnx/blob/main/docs/Operators.md).
99
* **Supported Opsets** indicates the lowest and highest opset a model may have for onnx-mlir to support compiling a model with the operator.
10-
* A * indicates onnx-mlir is compatible with the latest version of that operator available as of opset 20.
10+
* A * indicates onnx-mlir is compatible with the latest version of that operator available as of opset 21.
1111

1212

1313
NNPA has hardware limitations in dimension index size and tensor size, which are described in [NNPALimit.hpp](../src/Accelerators/NNPA/Support/NNPALimit.hpp). They are large enough for normal use cases, but if your model exceeds the limitations, CPU is used instead of NNPA.

‎docs/SupportedONNXOps-cpu.md

Lines changed: 20 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@
33

44
# Supported ONNX Operation for Target *cpu*.
55

6-
Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitations are listed when applicable. This documentation highlights the minimum and maximum opset versions that are fully supported by onnx-mlir and not the version changes.
6+
Onnx-mlir currently supports ONNX operations targeting up to opset 21. Limitations are listed when applicable. This documentation highlights the minimum and maximum opset versions that are fully supported by onnx-mlir and not the version changes.
77

88
* Operations are defined by the [ONNX Standard](https://github.com/onnx/onnx/blob/main/docs/Operators.md).
99
* **Supported Opsets** indicates the lowest and highest opset a model may have for onnx-mlir to support compiling a model with the operator.
10-
* A * indicates onnx-mlir is compatible with the latest version of that operator available as of opset 20.
10+
* A * indicates onnx-mlir is compatible with the latest version of that operator available as of opset 21.
1111

1212

1313
| Op |Supported Opsets (inclusive) |Limitations |Notes |
@@ -36,8 +36,8 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
3636
| **BitwiseOr** |18 - * | | |
3737
| **BitwiseXor** |18 - * | | |
3838
| **BlackmanWindow** |none | | | |
39-
| **Cast** |6 - * |Cast only between float and double types. Only ppc64le and MacOS platforms support float16. | |
40-
| **CastLike** |19 - * |CastLike only between float and double types. Only ppc64le and MacOS platforms support float16. | |
39+
| **Cast** |6 - * |Cast only between float and double types. Only ppc64le and MacOS platforms support float16. Does not support int4 and uint4. | |
40+
| **CastLike** |19 - * |CastLike only between float and double types. Only ppc64le and MacOS platforms support float16. Does not support int4 and uint4. | |
4141
| **CastMap** |none | | | |
4242
| **CategoryMapper** |none | | | |
4343
| **Ceil** |6 - * | | |
@@ -48,8 +48,8 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
4848
| **Compress** |9 - * | | |
4949
| **Concat** |6 - * | | |
5050
| **ConcatFromSequence** |none | | | |
51-
| **Constant** |6 - * | | |
52-
| **ConstantOfShape** |9 - * | | |
51+
| **Constant** |6 - * |Does not support int4 and uint4. | |
52+
| **ConstantOfShape** |9 - * |Does not support int4 and uint4. | |
5353
| **Conv** |6 - * | | |
5454
| **ConvInteger** |none | | | |
5555
| **ConvTranspose** |6 - * |Spatial dimensions (H and W in input `X`, and kH and kW in input `W`) must be static dimension. | |
@@ -59,7 +59,7 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
5959
| **DFT** |17 - * | | |
6060
| **DeformConv** |none | | | |
6161
| **DepthToSpace** |13 - * | | |
62-
| **DequantizeLinear** |10 - * |Only support for per-tensor or layer dequantization. No support for per-axis dequantization. | |
62+
| **DequantizeLinear** |10 - * |Only support for per-tensor or layer dequantization. No support for per-axis dequantization. Does not support int4 and uint4. | |
6363
| **Det** |none | | | |
6464
| **DictVectorizer** |none | | | |
6565
| **Div** |6 - * |No support for short integers. | |
@@ -73,7 +73,7 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
7373
| **Expand** |8 - * |Input `shape` must have static shape. | |
7474
| **EyeLike** |none | | | |
7575
| **FeatureVectorizer** |none | | | |
76-
| **Flatten** |6 - * | | |
76+
| **Flatten** |6 - * |Does not support int4 and uint4. | |
7777
| **Floor** |6 - * | | |
7878
| **GRU** |7 - * |W, B and R must be constants. | |
7979
| **Gather** |6 - * | | |
@@ -94,8 +94,8 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
9494
| **HardSigmoid** |6 - * | | |
9595
| **HardSwish** |none | | | |
9696
| **Hardmax** |6 - * | | |
97-
| **Identity** |16 - * |Sequence identity not supported. | |
98-
| **If** |16 - * |Sequence and Optional outputs are not supported. | |
97+
| **Identity** |16 - * |Sequence identity not supported. Does not support int4 and uint4. | |
98+
| **If** |16 - * |Sequence and Optional outputs are not supported. Does not support int4 and uint4. | |
9999
| **Imputer** |none | | | |
100100
| **InstanceNormalization** |6 - * | | |
101101
| **IsInf** |20 - * |Currently no support for float16 infinity value. Only for float32 and float64. | |
@@ -111,7 +111,7 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
111111
| **LinearRegressor** |none | | | |
112112
| **Log** |6 - * | | |
113113
| **LogSoftmax** |13 - * |Axis 0, 1, and default currently disabled due to changes in ONNX 1.8.1/Opset 13. |Temporally removed due to changes in onnx 1.8.1. |
114-
| **Loop** |6 - * |Input must have static shape. | |
114+
| **Loop** |6 - * |Input must have static shape. Does not support int4 and uint4. | |
115115
| **LpNormalization** |none | | | |
116116
| **LpPool** |none | | | |
117117
| **MatMul** |6 - * | | |
@@ -142,11 +142,11 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
142142
| **OptionalHasElement** |none | | | |
143143
| **Or** |7 - * | | |
144144
| **PRelu** |6 - * | | |
145-
| **Pad** |6 - * |axes input not supported. | |
145+
| **Pad** |6 - * |axes input not supported. Does not support int4 and uint4. | |
146146
| **Pow** |7 - * |No support for power with integer types. | |
147147
| **QLinearConv** |none | | | |
148148
| **QLinearMatMul** |none | | | |
149-
| **QuantizeLinear** |10 - * |Do not support per-axis and i8 quantization. | |
149+
| **QuantizeLinear** |10 - * |Does not support per-axis and i8 quantization. Does not support int4 and uint4. | |
150150
| **RNN** |7 - * |W, B and R must be constants. | |
151151
| **RandomNormal** |none | | | |
152152
| **RandomNormalLike** |none | | | |
@@ -165,7 +165,7 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
165165
| **ReduceSum** |6 - * |Default axis and do_not_keep_dim not supported. |Default axis and do_not_keep_dim temporarily removed due to changes in onnx 1.8.1. |
166166
| **ReduceSumSquare** |13 - * |Default axis and do_not_keep_dim not supported. | |
167167
| **Relu** |6 - * | | |
168-
| **Reshape** |6 - * |allowzero not supported. Input `shape` must have static dimension. | |
168+
| **Reshape** |6 - * |allowzero not supported. Input `shape` must have static dimension. Does not support int4 and uint4. | |
169169
| **Resize** |10 - * |Missing support for linear, cubic, crop, pytorch_half_pixel, and floor. Attributes antialias, axes and keep_aspect_ratio_policy are not supported. `scales` and `sizes` must have static dimension. | |
170170
| **ReverseSequence** |10 - * | | |
171171
| **RoiAlign** |none | | | |
@@ -174,7 +174,7 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
174174
| **SVMClassifier** |none | | | |
175175
| **SVMRegressor** |none | | | |
176176
| **Scaler** |none | | | |
177-
| **Scan** |8 - * |Does not support dynamic shapes. |Precision issue with newer opset, maybe just unsupported. Dynamic shape?. |
177+
| **Scan** |8 - * |Does not support dynamic shapes. Does not support int4 and uint4. |Precision issue with newer opset, maybe just unsupported. Dynamic shape?. |
178178
| **Scatter** |none | | | |
179179
| **ScatterElements** |11 - * |Does not support duplicate indices. | |
180180
| **ScatterND** |11 - * |Does not support scatternd add/multiply. | |
@@ -186,13 +186,13 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
186186
| **SequenceInsert** |11 - * |Does not support unranked sequence element. | |
187187
| **SequenceLength** |none | | | |
188188
| **SequenceMap** |none | | | |
189-
| **Shape** |15 - * |Does not support start and end attributes. | |
189+
| **Shape** |15 - * |Does not support start and end attributes. Does not support int4 and uint4. | |
190190
| **Shrink** |none | | | |
191191
| **Sigmoid** |6 - * | | |
192192
| **Sign** |9 - * | | |
193193
| **Sin** |7 - * | | |
194194
| **Sinh** |9 - * | | |
195-
| **Size** |13 - * | | |
195+
| **Size** |13 - * |Does not support int4 and uint4. | |
196196
| **Slice** |13 - * |Axis must be a constant argument. |Add tests to slices, currently have none. |
197197
| **Softmax** |6 - * | | |
198198
| **SoftmaxCrossEntropyLoss** |none | | | |
@@ -202,7 +202,7 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
202202
| **Split** |6 - * |Does not support static and dynamic shape, zero size splits. |Temporally removed due to changes in onnx 1.8.1. |
203203
| **SplitToSequence** |none | | | |
204204
| **Sqrt** |6 - * | | |
205-
| **Squeeze** |6 - * |Does not support static and dynamic shape. |Temporally removed due to changes in onnx 1.8.1. |
205+
| **Squeeze** |6 - * |Does not support static and dynamic shape. Does not support int4 and uint4. |Temporally removed due to changes in onnx 1.8.1. |
206206
| **StringNormalizer** |none | | | |
207207
| **Sub** |6 - * |Does not support short integers. | |
208208
| **Sum** |6 - * | | |
@@ -212,12 +212,12 @@ Onnx-mlir currently supports ONNX operations targeting up to opset 20. Limitatio
212212
| **ThresholdedRelu** |none | | | |
213213
| **Tile** |6 - * | | |
214214
| **TopK** |10 - * |`K`, the number of top elements to retrieve, must have static shape. | |
215-
| **Transpose** |6 - * | | |
215+
| **Transpose** |6 - * |Does not support int4 and uint4. | |
216216
| **TreeEnsembleClassifier** |none | | | |
217217
| **TreeEnsembleRegressor** |none | | | |
218218
| **Trilu** |14 - * | | |
219219
| **Unique** |11 - * | | |
220-
| **Unsqueeze** |6 - * |Does not support static and dynamic shape. |Temporally removed due to changes in onnx 1.8.1. |
220+
| **Unsqueeze** |6 - * |Does not support static and dynamic shape. Does not support int4 and uint4. |Temporally removed due to changes in onnx 1.8.1. |
221221
| **Upsample** |7 - * |Input `X` and `Y` must have static shape. | |
222222
| **Where** |9 - * | | |
223223
| **Xor** |7 - * | | |

‎requirements.txt

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
lit~=15.0
22
# numpy 1.24 deprecates np.object, np.bool, np.float, np.complex, np.str,
33
# and np.int which are used heavily in onnx-mlir.
4-
numpy~=1.22.2, <=1.23.5
4+
numpy==2.0.1
5+
onnx==1.16.2
56
protobuf==4.21.12
6-
pytest~=7.2
7-
pytest-xdist~=3.0
7+
pytest==8.3.2
8+
pytest-xdist==3.6.1

‎src/Builder/OpBuildTable.inc

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ op_dialect_version_map_["Gradient"] = {1};
8080
op_dialect_version_map_["Greater"] = {13};
8181
op_dialect_version_map_["GreaterOrEqual"] = {16};
8282
op_dialect_version_map_["GridSample"] = {16};
83-
op_dialect_version_map_["GroupNormalization"] = {18};
83+
op_dialect_version_map_["GroupNormalization"] = {21, 18};
8484
op_dialect_version_map_["HammingWindow"] = {17};
8585
op_dialect_version_map_["HannWindow"] = {17};
8686
op_dialect_version_map_["HardSigmoid"] = {6};
@@ -358,6 +358,8 @@ import_handler_map_["GridSample"] =
358358
&onnx_mlir::detail::FrontendGenImpl::buildOperation<mlir::ONNXGridSampleOp>;
359359
import_handler_map_["GroupNormalization"] =
360360
&onnx_mlir::detail::FrontendGenImpl::buildOperation<mlir::ONNXGroupNormalizationOp>;
361+
import_handler_map_["GroupNormalizationV18"] =
362+
&onnx_mlir::detail::FrontendGenImpl::buildOperation<mlir::ONNXGroupNormalizationV18Op>;
361363
import_handler_map_["HammingWindow"] =
362364
&onnx_mlir::detail::FrontendGenImpl::buildOperation<mlir::ONNXHammingWindowOp>;
363365
import_handler_map_["HannWindow"] =

‎src/Dialect/ONNX/DialectBuilder.cpp

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
//===----- DialectBuilder.cpp - Helper functions for ONNX dialects -------===//
66
//
7-
// Copyright 2019-2023 The IBM Research Authors.
7+
// Copyright 2019-2024 The IBM Research Authors.
88
//
99
// =============================================================================
1010
//
@@ -164,6 +164,19 @@ Value OnnxBuilder::layerNorm(Type outputType, Value input, Value scale,
164164
toTensor(bias), axisAttr, epsilon, stashTypeAttr);
165165
return layerNormOp.getY();
166166
}
167+
// In the case of GroupNormalization when stashType can be specified
168+
Value OnnxBuilder::layerNorm(Type outputType, Value input, Value scale,
169+
Value bias, int64_t axis, FloatAttr epsilon, IntegerAttr stashType) const {
170+
IntegerAttr axisAttr = getSignedInt64Attr(axis);
171+
Value noneVal = none();
172+
Type noneType = noneVal.getType();
173+
ONNXLayerNormalizationOp layerNormOp =
174+
createOpAndInferShapes<ONNXLayerNormalizationOp>(
175+
/*Y type*/ toTensor(outputType), /*mean type*/ noneType,
176+
/*std dev Type*/ noneType, toTensor(input), toTensor(scale),
177+
toTensor(bias), axisAttr, epsilon, stashType);
178+
return layerNormOp.getY();
179+
}
167180

168181
Value OnnxBuilder::qlinearMatMul(Type outputType, Value a, Value aScale,
169182
Value aZeroPoint, Value b, Value bScale, Value bZeroPoint, Value yScale,

‎src/Dialect/ONNX/DialectBuilder.hpp

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,10 @@ struct OnnxBuilder : DialectBuilder {
9191
mlir::Value layerNorm(mlir::Type outputType, mlir::Value input,
9292
mlir::Value scale, mlir::Value bias, int64_t axis,
9393
mlir::FloatAttr epsilon) const;
94+
// In the case of GroupNormalization when stashType can be specified
95+
mlir::Value layerNorm(mlir::Type outputType, mlir::Value input,
96+
mlir::Value scale, mlir::Value bias, int64_t axis,
97+
mlir::FloatAttr epsilon, mlir::IntegerAttr stashType) const;
9498

9599
// ONNXQLinearMatMulOp
96100
mlir::Value qlinearMatMul(mlir::Type outputType, mlir::Value a,

0 commit comments

Comments
 (0)