(There's also the mysterious multi-modal 1.5trillion+ 'Wu Dao' MOE model but little known about it). Microsoft trains a 530billion parameter GPT3-style language model. Other studies, like one published by Intel, MIT, and Canadian AI initiative CIFAR researchers in April, have found high levels of stereotypical bias from some of the most popular open source models, including Google’s BERT, XLNet, and Facebook’s RoBERTa. A group at Georgetown University has used GPT-3 to generate misinformation, including stories around a false narrative, articles altered to push a bogus perspective, and tweets riffing on particular points of disinformation. In a paper, the Middlebury Institute of International Studies’ Center on Terrorism, Extremism, and Counterterrorism claim that GPT-3 and similar models can generate “informational” and “influential” text that might radicalize people into far-right extremist ideologies and behaviors. It’s well-established that models like MT-NLG can amplify the biases in data on which they were trained, and indeed, Microsoft and Nvidia acknowledge that the model “picks up stereotypes and biases from the data.” That’s likely because a portion of the dataset was sourced from communities with pervasive gender, race, physical, and religious prejudices, which curation can’t completely address. When benchmarked, Microsoft says that MT-NLG can infer basic mathematical operations even when the symbols are “badly obfuscated.” While not extremely accurate, the model seems to go beyond memorization for arithmetic and manages to complete tasks containing questions that prompt it for an answer, a major challenge in NLP. Training took place across 560 Nvidia DGX A100 servers, each containing 8 Nvidia A100 80GB GPUs.
#MICROSOFT DAVID VOICE GENERATOR CODE#
The Pile spans academic sources (e.g., Arxiv, PubMed), communities (StackExchange, Wikipedia), code repositories (Github), and more, which Microsoft and Nvidia say they curated and combined with filtered snapshots of the Common Crawl, a large collection of webpages including news stories and social media posts. The dataset largely came from The Pile, an 835GB collection of 22 smaller datasets created by the open source AI research effort EleutherAI. Like all AI models, MT-NLG had to “train” by ingesting a set of examples to learn patterns among data points, like grammatical and syntactical rules. Tokens, a way of separating pieces of text into smaller units in natural language, can either be words, characters, or parts of words.
To train MT-NLG, Microsoft and Nvidia say that they created a training dataset with 270 billion tokens from English-language websites. Language models with large numbers of parameters, more data, and more training time have been shown to acquire a richer, more nuanced understanding of language, for example gaining the ability to summarize books and even complete programming code. Generally speaking, in the language domain, the correlation between the number of parameters and sophistication has held up remarkably well.
In machine learning, parameters are the part of the model that’s learned from historical training data. The journey is long and far from complete, but we are excited by what is possible and what lies ahead.” “We look forward to how MT-NLG will shape tomorrow’s products and motivate the community to push the boundaries of natural language processing (NLP) even further. The innovations of DeepSpeed and Megatron-LM will benefit existing and future AI model development and make large AI models cheaper and faster to train,” Nvidia’s senior director of product management and marketing for accelerated computing, Paresh Kharya, and group program manager for the Microsoft Turing team, Ali Alvi wrote in a blog post.
#MICROSOFT DAVID VOICE GENERATOR FULL#
“The quality and results that we have obtained today are a big step forward in the journey towards unlocking the full promise of AI in natural language. The successor to the companies’ Turing NLG 17B and Megatron-LM models, MT-NLG contains 530 billion parameters and achieves “unmatched” accuracy in a broad set of natural language tasks, Microsoft and Nvidia say - including reading comprehension, commonsense reasoning, and natural language inferences.
Microsoft and Nvidia today announced that they trained what they claim is the largest and most capable AI-powered language model to date: Megatron-Turing Natural Language Generation (MT-NLG). Join AI and data leaders for insightful talks and exciting networking opportunities. We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 - 28.
0 kommentar(er)
