I opened the LETR website this summer, and I think it was not long ago that I started a blog, but after all, I'm posting the last content of the year. Well, last week '2021 Top Artificial Intelligence and Natural Language Processing News (1)Following”, I will introduce the main news for the second half of the year.
July
Joshua Bengio, Jeffrey Hinton, and Jan Rechun present joint paper 'Deep Learning for AI (Deep Learning for AI) '
When they won the Turing Award in 2018, Jan Lecun, Jeffrey Hinton, and Joshua Benzio, sources
Deep learning pioneers Joshua Benzio, Jeffrey Hinton, and Jan LeCoon presented a paper called “Deep Learning for AI (Deep Learning for AI).” Through this paper, the three presented the problems of existing deep learning learning methods, solutions to them, and a progressive future. While pointing out current problems with deep learning, such as large amounts of human labor and the need for large-scale computing resources, he presented a vision for the future of deep learning by referring to examples such as “Transformers (Transformers)” and “Self-Supervised Learning (Self-Supervised Learning)” created without labeling.
GitHub + OpenAI launches' Copilot (Copilot) ', an artificial intelligence with automatic code completion
GitHub Copilot, sources
GitHub and OpenAI have launched “Copilot (Copilot)”, a code writing tool based on artificial intelligence. GitHub described Co-Pilot as an “artificial intelligence pair programmer” who writes code together rather than a tool. Of course, Copilot cannot replace developers yet, but it is expected that it will greatly help developers learn and work more efficiently in the future. It also makes me imagine the advent of an AI-driven development era of more automated and faster development that may soon become a reality.
August
Stanford Human-Centered Research Institute presents a paper on the relationship between large-scale language models and AI ethics
A chart on the problem of bias in the paper, sources
Researchers at Stanford University, in which masters such as PayPay Lee and Percy Liang participated,On the Opportunities and Risks of Foundation Models (On the Opportunities and Risks of Foundation Models)I published a paper called”. This paper raised questions about large-scale language models that have been appearing one after another in recent years and are in the limelight. While pointing out issues such as potential biases in language models and the negative impact of large-scale computing on the environment, it was proposed to develop and deploy in an ethical and socially responsible manner.
ACL-IJCNLP 2021 Confirms Potential Development of NLP in Asia
ACL-IJCNLP 2021sources
The 11th International Conference on Natural Language Processing (ACL-IJCNLP2021) was held online in Bangkok, Thailand. This event is a place for Asian natural language processing (NLP) AI model developers to present and share research results. Various workshops and sessions were held at this week-long event on the metaverse platform, and many leading companies such as Baidu and Naver participated.
Tesla AI Day Presents Technologies and Visions for Autonomous Driving, Supercomputers, and Robots
Tesla AI DAY, sources
Tesla held an AI Day event and revealed its future vision centered on the semiconductor “D1,” the supercomputer “Dojo (Dojo),” and “Tesla Bot (Tesla Bot).” In particular, the differentiation strategy, which proposed a unique technology approach rather than mergers and acquisitions or strategic partnerships, stood out. CEO Elon Musk introduced the humanoid “Tesla Bot” that it will dramatically change the global economy by reducing tedious and repetitive labor.
Researchers from Korea University and Princeton University in the US develop an AI model for real-time question-and-answer
Densprays demo page, sources
A research team from Korea University and Princeton University developed an AI model called DensePlays (DensePlays) through joint research. The research team announced that they would find answers to users' everyday language queries on Wikipedia and process them around 100( 0.1 seconds). In particular, while providing performance similar to existing highest-performing AI models, the processing speed has been improved by more than 20 times, and it is said that there is no need for expensive GPUs required to train and run existing deep learning models.
October
Facebook announces first-person AI development project 'EGO4D'
EGO4D data collection, sources
EGO4D announced by Facebook is a project to enable artificial intelligence to see and interact with the world from a human perspective. To this end, 13 universities and 2 research institutes from 9 countries formed a consortium, and it is reported that over 700 participants collected more than 3,000 hours of first-person videos about their daily lives. Although there are concerns about infringement of privacy, it is expected that AI will be useful when people lose the location of an item or when recalling stories they've heard from others, such as helping humans find objects and help them remember stories.
DeepMind develops AI to predict precipitation after 90 minutes
DGMR concept diagram, sources
DeepMind collaborated with the UK Meteorological Service to develop 'DGMR', a deep learning tool that accurately predicts the probability of precipitation within 90 minutes. To this end, it is said that the AI learned British radar data taken from 2016 to 2018, and then began forecasting experiments in 2019. Compared to existing weather prediction systems that use complex mathematical equations using numerical forecasting, DGMR is said to be able to make much more accurate predictions using a 'generative modeling' method where AI analyzes the distribution of data and generates similar new data.
November
Kakao Unveils Korean Supergiant AI Language Model 'KoGPT'
An introduction to 'KogPT' in 'if (kakao) 2021', sources
Following Naver Hyper Clova, Kakao also unveiled the Korean-specific AI language model “KogPT.” KoGPT announced that it is a Korean-specific version of GPT-3, built based on 6 billion parameters and 200 billion tokens (tokens) of Korean data. In particular, Kakao plans to increase the largest language model by more than 100 times in the future by introducing blockchain technology.
December
DeepMind Unveils Supergiant AI Language Model Gopher and Search-based Giant Model RETRO
Gopher and GPT-3, comparison of human expert abilities, sources
Gopher is a super-large AI language model developed in-house by DeepMind. It consists of 280 billion parameters, which is larger than Open AI's GPT-3. Furthermore, it is said that it shows performance that surpasses existing super-large language models.
RETRO concept map, sources
RETRO (Retrieval-Enhanced Transformer), which was later released together, is a giant model based on search. It's a new language model that uses external memory, unlike traditional models that require all information to be memorized. It is said that this use of external memory not only shows results comparable to existing models that are about 25 times larger, but also drastically reduces the time and cost required to train very large models.
DeepMind's six ethical categories, sources
DeepMind also published a paper on ethical issues related to the use of super-large language models. It is said that they predicted ethical and social risks that may occur in language models and created a comprehensive taxonomy of these risks by referring to existing studies. As a result, 6 subject areas and 21 risk factors were presented, as shown in the image above.
Kakao unveils super-giant AI multimodal (multi-modal) 'Mindall-e'
Images created by AI understanding instructions, sources
Following 'KogPT', a super-large Korean language model, Kakao has developed a multimodal that simultaneously understands text and images. Open AI (Open AI)'s “DALL-E” is a small-sized model that is easy for anyone to access, and is called “MindAll-E.” Furthermore, it is expected that the service will be expanded in the future, such as releasing various models in the future.
References
[1] Deep Learning for AI (ACM Journal) https://cacm.acm.org/magazines/2021/7/253464-deep-learning-for-ai/fulltext
[2] Deep Learning for AI (discussion video by the authors) https://vimeo.com/554817366
[3] Do you play the code alone? Now let's play with AI! https://maily.so/1step/posts/0f41a5
[4] Stanford University researchers, “Large-scale language models reinforce bias and cause serious environmental pollution” http://www.aitimes.com/news/articleView.html?idxno=140210
[5] On the Opportunities and Risks of Foundation Models https://arxiv.org/abs/2108.07258
[6] 'Tesla AI Day'...” Make semiconductors, supercomputers, and robots by yourself”https://www.etnews.com/20210820000176
[7] Tesla AI Day video https://youtu.be/j0z4FweCy4M
[8] ACL-IJCNLP 2021 just came to an end... Confirming the potential for development of NLP development in Asia http://www.aitimes.com/news/articleView.html?idxno=140049
[9] Dr. Lee Jin-hyuk, Korea University, develops artificial intelligence for natural language processing with Princeton University... Real-time question and answer (Q&A) model 'AI Denspray' http://www.aitimes.kr/news/articleView.html?idxno=22015
[10] Learning Dense representations of Expression at Scale https://arxiv.org/abs/2012.12624
[11] https://github.com/princeton-nlp/DensePhrases
[12] Facebook announces first-person AI development project 'EGO4D'...” AI has subjectivity” http://www.aitimes.com/news/articleView.html?idxno=141051
[13] Ego4D: Around the World in 3,000 Hours of Egocentric Video https://arxiv.org/abs/2110.07058
[14] Ego4D: Teaching AiTo Presenting the World Through Your Eyes https://youtu.be/taC2ZKl9IsE
[15] DeepMind's AI greatly improved the accuracy of predicting weather within 90 minutes https://www.technologyreview.kr/deepminds-ai-predicts-almost-exactly-when-and-where-its-going-to-rain/
[16] Skillful repetition nowcasting using deep generative models of radar https://www.nature.com/articles/s41586-021-03854-z
[17] KakaoBrain Unveils Korean Supergiant AI Language Model 'KoGPT' https://www.kakaocorp.com/page/detail/9600
[18] Language Modelling at Scale: Gopher, Ethical Discipline, and Retrieval https://www.deepmind.com/blog/article/language-modelling-at-scale
[19] Scaling Language Models: Methods, Analysis & Insights from Training Gopher https://storage.googleapis.com/deepmind-media/research/language-research/Training%20Gopher.pdf
[20] Challenges in Detoxifying Language Models https://deepmind.com/research/publications/2021/Challenges-in-Detoxifying-Language-Models
