# Knowledge Distillation from Neural Networks to Differentiable Logic Networks

DevFeed: [Knowledge Distillation from Neural Networks to Differentiable Logic Networks](<https://devfeed.tech/articles/knowledge-distillation-from-neural-networks-to-differentiable-logic-networks-123342.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09024>)

Author: Hee Yun Suh, Niraj K. Jha

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [mlp](<https://devfeed.tech/topics/mlp.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [inference](<https://devfeed.tech/tags/inference.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [knowledge-distillation](<https://devfeed.tech/tags/knowledge-distillation.md>), [logic](<https://devfeed.tech/tags/logic.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [training](<https://devfeed.tech/tags/training.md>), [weight](<https://devfeed.tech/tags/weight.md>)

## AI overview

Knowledge distillation can improve differentiable logic network (DLN) accuracy while preserving the student model's interpretability, sparsity, and low inference cost. The method trains a DLN student from a trained multi-layer perceptron teacher using a distillation objective in a two-phase loop; the teacher is discarded afterward, and neither the DLN architecture nor inference procedure changes. The study compares four variants: joint search over architecture and distillation hyperparameters, grid search over distillation weight and temperature with architecture fixed, and distillation in only the neuron-function or connection phase. Across 20 public tabular classification tasks, balanced-class accuracy improves by as much as 15%, sometimes exceeding the teacher, with negligible changes to gate-level operation counts and model size. Gains vary by dataset.

## Source excerpt

arXiv:2611.09024v1 Announce Type: new Abstract: Differentiable logic networks (DLNs) are interpretable, edge-friendly classifiers that learn both their logic operators and their connections through gradient descent, achieving accuracy competitive with multi-layer perceptrons at orders-of-magnitude lower inference cost. Their accuracy, however, is bounded by the problems of vanishing gradients, scalability, and lack of an ecosystem, leaving a gap relative to conventional neural networks on several datasets. We introduce knowledge distillation for DLNs to narrow this gap. We distill from a trained multi-layer perceptron teacher into a DLN student, integrating the distillation objective into a two-phase training loop without altering the architecture or the inference procedure. The teacher is discarded after training. Hence, the distilled model retains the interpretability, sparsity, and inference cost of a conventionally trained DLN. We investigate this approach through four variants: a joint hyperparameter search that optimizes the distillation weight and temperature alongside the architecture, a grid search over these two hyperparameters with the architecture fixed, and two single-phase variants that employ distillation during only the neuron-function or only the connection phase. On 20 public tabular classification tasks, distillation improves balanced-class accuracy at negligible cost to gate-level operation counts and model size, though the benefit varies by dataset. Performance improvements of up to 15% are observed, in some cases surpassing the teacher itself.