Difference in implementation and paper

[germayneng]

Good day,

Thank you for the paper. I noticed that there are some difference in implementation described in the paper as well as the source code,

1. For example, the spectral sequential cell defined in the paper uses a convo 1D operation but i cant seem to see it in the code (unless i am missing something )

2. In the training loop, the backcast are not used in the loss function

3. Another question i have is that i dont understand how the following:

igfted = torch.matmul(gconv_input, self.weight)
igfted = torch.sum(igfted, dim=1)

is actually the IGFT operation. I understand that as opposed to the paper, the Chebyshev Polynomial is used to directly compute the GFT operation without the need for the eigenvectors of L.

Will be good to explain! thanks!

[WooheonHong]

I totally agree with you..
It's nothing like a paper.

[lsrock1]

I agree, is there any reason for these differences?

[lsrock1]

I think the authors use Chebyshev Polynomials to reduce time complexity (laplacian(n^3)).
(review 3 of neurips)
ref https://proceedings.neurips.cc/paper/2020/file/cdf6581cb7aca4b7e19ef136c6e601a5-Review.html
ref https://towardsdatascience.com/an-introduction-to-graph-neural-networks-e23dc7bdfba5

[WooheonHong]

Output layers using GLU and fully connected layers are also not implemented.

Is there anyone who reproduced the paper's performance?
I can't reproduce metr-la and ECG data performance in any way.
I also read the author's commited code, but it is not reproduced.
https://github.com/microsoft/StemGNN/commits?author=t-decao

[germayneng]

Is there anyone who reproduced the paper's performance?

I think the current pytorch implementation is not following the paper's defined architecture. Perhaps we can try to replicate the exact architecture to see if the results will be closer?

[lsrock1]

I tried but this paper lacks detail about implementation.
The overall concept is clear but the core detail is dismissed.
It is really weird that even ECG cannot get closer to the reported performance.

[razvanc92]

I've also tried to reproduce the results on PEMS 3,4 and 7 but without success.

[Einsturing]

Hi, I'm reading the paper and code rencently and also find the difference like the loss F you've described.Have you found a solution to the problem?

[zezhishao]

Yes...Some concepts of StemGNN are really attractive, but they seem to be missing in the code...

[lixus7]

why not do some ablation experiment about single Y1 and single Y2?
Anyway, the lack of the ablation experiment lead me doubt it is the contribution of your ensemble architecture.

[superarthurlx]

it seems the author made a few mistakes on the task they performed. For example pems07 is a traffic speed data but the author use it as a traffic flow task, meanwhile the result of pems04 experiment is definitely a traffic flow task but the author run it as a traffic speed task.

[razvanc92]

@superarthurlx I think there are 2 different pems07. For example the one released by STSGCN(AAAI-2020) is traffic flow.

[zezhishao]

it seems the author made a few mistakes on the task they performed. For example pems07 is a traffic speed data but the author use it as a traffic flow task, meanwhile the result of pems04 experiment is definitely a traffic flow task but the author run it as a traffic speed task.

@superarthurlx I think these concepts are getting blurred, maybe because they are the same to the model. For example the STGNN[1] also use METR-LA and PEMS-BAY as traffic flow prediction datasets, however they record traffic speed.

[1] Wang X, Ma Y, Wang Y, et al. Traffic flow prediction via spatial temporal graph neural network[C]//Proceedings of The Web Conference 2020. 2020: 1082-1092.

[superarthurlx]

@superarthurlx I think there are 2 different pems07. For example the one released by STSGCN(AAAI-2020) is traffic flow.

look at Table 2, the author used the same result presented in ST-GCN[38] which is actually called STGCN(Cheb), so the datasets should be same as well which is a speed data.

[lixus7]

i had loaded the dataset pems07 from the github of STSGCN，and proved that it was traffic speed.

…

------------------ Original ------------------ From: superarthurlx ***@***.***> Date: Wed,Oct 27,2021 11:04 PM To: microsoft/StemGNN ***@***.***> Cc: lixus7 ***@***.***>, Comment ***@***.***> Subject: Re: [microsoft/StemGNN] Difference in implementation and paper (#12) @superarthurlx I think there are 2 different pems07. For example the one released by STSGCN(AAAI-2020) is traffic flow. look at Table 2, the author used the same result presented in ST-GCN[38] which is actually called STGCN(Cheb), so the datasets should be same as well which is a speed data. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[superarthurlx]

it seems the author made a few mistakes on the task they performed. For example pems07 is a traffic speed data but the author use it as a traffic flow task, meanwhile the result of pems04 experiment is definitely a traffic flow task but the author run it as a traffic speed task.

@superarthurlx I think these concepts are getting blurred, maybe because they are the same to the model. For example the STGNN[1] also use METR-LA and PEMS-BAY as traffic flow prediction datasets, however they record traffic speed.

[1] Wang X, Ma Y, Wang Y, et al. Traffic flow prediction via spatial temporal graph neural network[C]//Proceedings of The Web Conference 2020. 2020: 1082-1092.

Check table2 or pems07.csv file you can find the different. Flow data should be all int number while speed data are float numbers and usually less than 70 which is the speed limit in pems area.

here are some of my reproductions, there are still many differences like mape values. ==!

[zezhishao]

it seems the author made a few mistakes on the task they performed. For example pems07 is a traffic speed data but the author use it as a traffic flow task, meanwhile the result of pems04 experiment is definitely a traffic flow task but the author run it as a traffic speed task.

@superarthurlx I think these concepts are getting blurred, maybe because they are the same to the model. For example the STGNN[1] also use METR-LA and PEMS-BAY as traffic flow prediction datasets, however they record traffic speed.
[1] Wang X, Ma Y, Wang Y, et al. Traffic flow prediction via spatial temporal graph neural network[C]//Proceedings of The Web Conference 2020. 2020: 1082-1092.

Check table2 or pems07.csv file you can find the different. Flow data should be all int number while speed data are float numbers and usually less than 70 which is the speed limit in pems area.

here are some of my reproductions, there are still many differences like mape values. ==!

Thanks for explaining the difference.
The reproductions are ... amazing... lol

[catcatwang]

看来作者在他们执行的任务上犯了一些错误。比如pems07是一个交通速度数据，作者把它当成一个交通流任务，同时pems04实验的结果肯定是一个交通流任务，但是作者把它作为一个交通速度任务来运行。

@superarthurlx我认为这些概念越来越模糊，可能是因为它们与模型相同。例如，STGNN[1] 也使用 METR-LA 和 PEMS-BAY 作为交通流预测数据集，但是它们记录交通速度。
[1] 王 X，马 Y，王 Y，等。基于时空图神经网络的交通流预测[C]//Proceedings of The Web Conference 2020. 2020: 1082-1​​092.

检查 table2 或 pems07.csv 文件你可以找到不同的。流量数据应该是整数，而速度数据是浮点数，通常小于 70，这是 pem 区域的速度限制。

这是我的一些复制品，仍然存在许多差异，例如 mape 值。==！
Hello, can you tell me the parameters you are replicating on the ECG dataset? I can't get the results you show.

[semink]

Hello,

So, no one has succeeded in reproducing the results on their paper?

[stemill]

Also interested in whether or not anyone has successfully reproduced output? Sounds like there are some outstanding discrepancies between the paper and the code?

[zezhishao]

I found that the calculation of metrics in StemGNN seems to be unfair and inconsistent with baselines, such as the calculation in GWNet.