Distributed and Democratized Learning: Philosophy and Research Challenges

Note: This blog is a short discussion of our recent work accepted in IEEE Computational Intelligence Magazine. The preprint version of this work is available here.

News: Our technical paper on Dem-AI implementation, “Self-organizing Democratized Learning: Toward Large-scale Distributed Learning Systems”, got accepted in IEEE Transactions on Neural Networks and Learning Systems (TNNLS). The preprint version of which is available here.

Update: Implementation code of Dem-AI on MNIST and Fashion-MNIST dataset here.

TLDR: We develop a novel design philosophy called democratized learning (Dem-AI) whose goal is to build large-scale distributed learning systems that rely on the hierarchical self-organization of distributed learning agents that are well-connected, but limited in learning capabilities. Dem-AI learning system can evolve and regulate itself based on the underlying duality of two processes that we call specialized and generalized processes.

Artificial Intelligence (AI) is moving towards edge devices with the availability of massively distributed data sources and the increase in computing power for handheld and wireless devices such as smartphones or self-driving cars. This has generated growing interest to develop large-scale distributed machine learning paradigms.

Democracy in Learning

Democracy in learning features a unique characterization of participation in the learning process, and consequently develops the notion of democracy in learning whose principles include the following:

• According to the differences in characteristics of learning agents, they are divided into appropriate groups that can be specialized for the learning tasks. These specialized groups are self-organized in a hierarchical structure to mediate voluntary contributions from all members in the collaborative learning for solving multiple complex tasks.
• The shared generalized learning knowledge supports specialized groups and learning agents to improve their learning performance by reducing individual biases during participation. In particular, the learning system allows new group members to: a) speed up their learning process with the existing group knowledge and b) incorporate their new learning knowledge in expanding the generalization capability of the whole group.

To that end, these characteristics motivate us to develop a novel design philosophy for future large-scale distributed learning systems.

“Dem-AI learning system can evolve and regulate itself based on the underlying duality of two processes that we call specialized and generalized processes.”

Towards a Large-scale Distributed Learning System

The distributed learning scenario is a new paradigm in machine learning in which learning agents collectively perform machine learning tasks. In this regard, voluntary contributions from the learning agents in a distributed learning system-like humans interactions (when they do) in a social group- will empower in solving various complex tasks. However, a large number of biased learning agents (those having unbalanced and small personalized data) makes it challenging to guarantee high learning performance of learning agents after such collaboration.

“Individuals contribute to society in resolving multiple complex tasks in a way similar to the learning agents in large-scale distributed learning systems.”

Q1. This raises an important, fundamental research question: How can one resolve the discrepancies between global and personalized accuracy, regarding the heterogeneity in the characteristics of agents?

To better understand this research question, let’s go through the following observation about the role of individual capabilities in the social structure development process.

Individuals contribute to society in resolving multiple complex tasks in a way similar to the learning agents in large-scale distributed learning systems. An individual exists as a unit entity with some basic survival objectives and functions/skills in the social hierarchy while interacting, contributing, and forming smaller groups such as a family. The conglomeration of families and relatives characterizes a society that behaves as a bigger group to resolve complex life issues and resolve/create conflicts (as illustrated in Fig. 1). Furthermore, as the highest socialized structure, the states form a global world organization, such as the United Nations, to maintain global harmony and solve overly complex global issues. Thus, many small groups unite to form a hierarchical structure for knowledge sharing and solving complex tasks.

“In the future, a transition from traditional centralized learning systems towards large-scale distributed AI systems is imperative.”

Interestingly, we can derive a similar understanding in the collective behavior of an interactive crowd, often characterized by swarm intelligence, which is well-observed in numerous biological systems.

Fig. 1: Analogy of a hierarchical distributed learning system.

In another observation in any form of biological intelligence, we observe a continual developmental process: from stem cells to complex structures with multiple functionalities, such as the human brain. For example, in humans, the learning process consists of many stages such as newly born, childhood, and grown-up. The transition from the newly born stage to the childhood stage is characterized by the individual’s pursuit to have a specialization capability over a set of already known basic skills and to further explore the world with a hierarchical structure of knowledge-involving new knowledge to tackle the complex tasks. While, at an early age, the generalization capability with a high level of neurosynaptic plasticity facilitates to learn diverse, yet basic knowledge and skills. Basically, the synaptic plasticity level is intrinsically involved to consolidate knowledge for learning and adapting to the dynamic environment.

These observations provide sufficient hints about the underlying duality of the generalized and specialized processes in the entire development process of biological intelligence or systems.

Q2. This also raises an important question: How can one understand and formalize the duality of the generalized and specialized processes regarding the plasticity and stability of a distributed learning system?

In summary

Existing machine learning designs (such as meta-learning, multi-task learning, reinforcement learning, federated learning) face critical challenges to scale up the current centralized AI systems into the distributed AI systems that can perform multiple complex learning tasks. Therefore, in the future, a transition from traditional centralized learning systems towards large-scale distributed AI systems is imperative.

“Dem-AI learning system can evolve and regulate itself based on the underlying duality of two processes that we call specialized and generalized processes.”

What we really need?

1. How can large-scale distributed AI systems be self-organized in a suitable hierarchical structure to perform knowledge sharing?
2. How can learning knowledge be shared among learning agents and tasks?
3. How can learning agents integrate generalized knowledge to enhance their learning performance?

Fig. 2: Anatomy of Democratized Learning.

Our contributions

1. First, we propose a novel design philosophy [Fig. 3] called democratized learning (Dem-AI) whose goal is to build large-scale distributed learning systems that rely on the self-organization of distributed learning agents that are well-connected, but limited in learning capabilities. Dem-AI learning system can evolve and regulate itself based on the underlying duality of two processes that we call specialized and generalized processes.
2. Second, we present a reference design as a guideline to realize future Dem-AI systems, inspired by various interdisciplinary fields. Accordingly, we introduce four underlying mechanisms in the design such as plasticity-stability transition mechanism, self-organizing hierarchical structuring, specialized learning, and generalization.
3. Finally, we establish possible extensions and new challenges for the existing learning approaches to provide better scalable, flexible, and more powerful learning systems with the new setting of Dem-AI.

Fig. 3: Dem-AI philosophy.

Toy Example with Dem-AI: A Multi-language Handwriting Recognition

A typical handwriting language recognition application has an embedded virtual assistant to improve the capability for understanding human-written texts in various languages. However, to realize such systems, we need separate recognition models for each language (e.g., English and Korean [see Fig. 4]). Using our Dem-AI reference design, agents undergo self-organization to form appropriate hierarchical regional/social groups so as to share the similarity in the characteristics of their languages. By exploiting such structures, the learning system can collectively incorporate the personalized experiences of users that improve the generalization learning model. Subsequently, it empowers the recognition capability at each agent along with increasing the importance of the specialized process in the system. This kind of application can scale up to a large number of agents and support multiple languages. Thus, it has the potential to integrate different voice recognition models for developing a fully supporting virtual assistant at each client.

Fig. 4: An example of Dem-AI systems: Multi-language handwriting recognition.

Takeaway

We realize that Dem-AI philosophy is relatable in our development process, for instance, in dealing with representation and diversity when doing collaborative tasks, and further, reducing biases through open discussion. Most importantly, Dem-AI philosophy enriches our understanding of a distributed learning system with a broader perspective, and in practice, it opens up unlimited applicabilities for future distributed learning systems.

— The authors.