Add Basic RWKV Block to GPT-NeoX #1167
Description
We want to add RWKV to gpt-neox:
- Add basic RWKV block, without kernels, from https://github.com/BlinkDL/RWKV-LM to https://github.com/EleutherAI/gpt-neox/tree/main/megatron/model
- Add rwkv kernels
- Add config options for rwkv
- Add assertions to gpt-neox so that parallelism schemes and other architectures are disabled when rwkv is enabled in config