The release of the dots3-note preview marks a significant expansion in the available open-weight multimodal model landscape. Developed by dots studio, this model architecture utilizes a Mixture-of-Experts (MoE) design with 280B total parameters, though it activates only 16B parameters per forward pass. This efficiency strategy is paired with support for a 512K token context window, positioning the model for long-context tasks that require processing extensive documents, video streams, or multi-step agentic workflows. The model is currently available under the Apache License 2.0, providing a permissive framework for researchers and developers to integrate it into their own pipelines.### What Changed The primary shift with the dots3-note preview is the combination of massive parameter scale with the operational efficiency of MoE. While many recent open-weight models have focused on dense architectures or smaller MoE configurations, dots3-note targets the upper end of the parameter spectrum. The model is natively multimodal, accepting text, image, video, and audio inputs. This is not merely a text-based model with an adapter; the architecture includes a dedicated MoE Vision Encoder (7B total, 1.2B activated) and a dense Audio Encoder (800M). This integration allows for native video understanding, where the model processes video inputs alongside their audio tracks, a capability that is increasingly critical for agentic workflows involving real-world interaction.### Technical Details The architecture of dots3-note is complex, designed to balance raw reasoning power with inference latency. It features 1 dense layer and 45 MoE layers. The MoE structure is particularly dense, utilizing 256 routed experts and 1 shared expert, with a top-8 routing mechanism. This high number of experts allows for a more granular specialization of weights, which theoretically improves the model's ability to handle diverse tasks without increasing the compute cost linearly. The attention mechanism is a hybrid, utilizing 13 Distributed Sparse Attention (DSA) layers and 33 Sliding Window Attention (SWA) layers, maintaining a ratio of approximately 1:3. This configuration is specifically tuned to handle the 512K context length, ensuring that the KV cache management remains efficient during long-sequence generation. The model supports both BF16 and FP8 precision, with the FP8 checkpoint being the primary recommendation for deployment on 8-GPU nodes. The hidden size is 5120, with an FFN hidden size of 13824 for the dense layer and 1536 per expert. This architecture is designed to be served using high-throughput frameworks like vLLM and SGLang, which are required to manage the memory footprint of a 280B model, even when quantized.### Developer Implications Deploying a 280B parameter model is a significant undertaking that requires careful infrastructure planning. The recommended deployment strategy involves a single 8-GPU node, specifically targeting the FP8 checkpoint to fit the model into memory. Developers should note that native support is currently available via vLLM and SGLang, though the integration is still evolving. For those using vLLM, the model requires a recent nightly build, as the support is being upstreamed. The configuration for vLLM involves setting the tensor-parallel-size to 8 and enabling expert parallelism with the deep_gemm backend. SGLang offers a more granular control set, including speculative decoding options like NEXTN, which can reduce Time Per Output Token (TPOT) by more than 50%. The model also supports OpenAI-compatible tool calling, which simplifies the integration into existing agentic frameworks. Developers should be aware that the Transformers implementation is currently tied to specific pull requests (#47844), meaning that a standard pip install of the main branch may not yet support the model's specific architecture. It is highly recommended to use the provided Docker images or build from the specific PR branches to ensure compatibility with the model's custom layers.### Bottom Line The dots3-note preview represents a high-water mark for open-weight multimodal models in terms of total parameter count and context capability. By leveraging a 280B MoE architecture, dots studio has created a model that bridges the gap between massive proprietary models and the current generation of open-weight alternatives. However, the requirement for 8-GPU nodes and specialized inference backends limits its accessibility to teams with significant compute resources. For organizations building agentic systems that require deep multimodal understanding and long-context memory, dots3-note offers a powerful, albeit resource-intensive, solution.