# MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference

DevFeed: [MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference](<https://devfeed.tech/articles/moai-module-optimizing-architecture-for-non-interactive-secure-transformer-inference-128337.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13641>)

Author: Linru Zhang, Xiangning Wang, Jun Jie Sim, Zhicong Huang, Jiahao Zhong, Huaxiong Wang, Pu Duan, Kwok-Yan Lam

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [homomorphic encryption](<https://devfeed.tech/topics/homomorphic-encryption.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [batching](<https://devfeed.tech/tags/batching.md>), [bert](<https://devfeed.tech/tags/bert.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [encryption](<https://devfeed.tech/tags/encryption.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [flow](<https://devfeed.tech/tags/flow.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [homomorphic-encryption](<https://devfeed.tech/tags/homomorphic-encryption.md>), [inference](<https://devfeed.tech/tags/inference.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [module](<https://devfeed.tech/tags/module.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

## AI overview

MOAI is a non-interactive homomorphic-encryption framework for transformer inference that reduces costly HE rotations through a consistent column-and-diagonal packing flow, rotation-free Softmax and LayerNorm algorithms, and interleaved batching. For BERT-base, it removes 2,448 rotations in inference and uses at least 1.7x fewer rotations across matrix multiplications than prior work. The authors report 52.8% lower evaluation time than THOR and, when applying MOAI to Powerformer's framework, 55.7% lower time than Powerformer. They report 2.36 minutes amortized per input on a single GPU and demonstrate extension to LLaMA-3-8B. These are abstract-reported results; no further experimental details are supplied here.

## Source excerpt

arXiv:2611.13641v1 Announce Type: cross Abstract: Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from: (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, removing 2,448 HE rotations in BERT-base inference. (3) Column packing removes rotations in plaintext-ciphertext matrix multiplications and interleaved batching further reduces the rotations in ciphertext-ciphertext matrix multiplications. MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix multiplications of BERT-base. As a result, we achieve a $52.8 \%$ reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS 25). We then apply MOAI on the Powerformer's framework and achieve a $55.7 \%$ reduction in evaluation time compared to Powerformer (Park et al., ACL 25), which approximates Softmax and LayerNorm with simpler functions in transformer and proposes HE-based non-interactive transformer inference. We report an amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI in LLaMA-3-8B. Our implementation is publicly available as open source.