Pre-training

IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation

We present IT5, the first family of encoder-decoder transformer models pretrained specifically on Italian on more than 40 billion words, reaching state-of-the-art performance for most Italian conditional language generation tasks.