LSTMAggregation
#4731
LSTMAggregation
#4731
Conversation
Codecov Report
@@ Coverage Diff @@
## master #4731 +/- ##
==========================================
- Coverage 84.32% 82.43% -1.89%
==========================================
Files 321 322 +1
Lines 17215 17237 +22
==========================================
- Hits 14516 14209 -307
- Misses 2699 3028 +329
Continue to review full report at Codecov.
|
| f"aggregation in first dimension (got '{dim}')") | ||
|
|
||
| x, _ = to_dense_batch(x, index, batch_size=dim_size) | ||
| return self.lstm(x)[0][:, -1] |
There was a problem hiding this comment.
Sorry, I am not familiar with LSTM. Some comments: I don't know whether using the output of the last time step is a good idea or not. 1. many padded zeros and fed into LSTM for nodes with smaller degrees. 2. the forgetting issue for long sequences.
How about something like this to remove the effects of padded zeros and average over all the time steps (not sure how it performs):
x, mask = to_dense_batch(x, index, batch_size=dim_size)
return self.reduce(self.lstm(x)[0][mask], index, ptr, dim_size, dim, reduce='mean')
There was a problem hiding this comment.
I think this is a valid point. AFAIK, GraphSAGE (and any other LSTM-style aggregation procedure, e.g., Jumping Knowledge), just read out the embeddings after the last element of the sequence has been processed, so I opted to go for this solution. It might be very interesting to explore which reduction performs better here.
There was a problem hiding this comment.
Sure, it is worth trying it out later.
No description provided.