Right-sized AI is about using the smallest model that meets your application's needs, not just for sustainability but also for performance. Smaller models consume less energy, and if they fit, they can run entirely on the user's device. This eliminates network latency and keeps user data on-device, which can be a significant product advantage.

Prototype big, deploy small

The strategy for finding the right model is to focus on proving your use case first, then optimizing for size. This can be broken down into a repeatable process:

  1. Validate the task's feasibility. Before optimizing, you must confirm that your core task is possible with AI at all. Use the largest and most capable model available—for example, a large foundation model—to establish a baseline for what an acceptable output looks like.
  2. Define success criteria. You need a test set. This is a collection of inputs paired with their ideal outputs. In a translation app, these inputs would be phrases in English with the correct Spanish translations as the outputs. These examples are the benchmark that every model you test will be measured against.
  3. Test from smallest to largest. Start with the most compact model you can find and compare its outputs against success criteria you've defined. Now proceed to larger models and continue until you find one that passes your tests. Prompt engineering and even recruiting a larger model to help rewrite prompts can produce better results from smaller models.
  4. Select the smallest passing model. The winner is the smallest model you tested that still produces acceptable outputs based on your success criteria. This choice holds whether the model is deployed on a server or, if small enough, on the user's device. Using a smaller model is always the more efficient path.

Model categories and use cases

Models are generally categorized by the type of data they process. Here are some examples of small, efficient models grouped into visual, audio, and text processing categories.

Visual processing

These models handle still images or video streams. Common use cases fall into the following groups:

  • Image classification: Determines the contents of an image. Use this for generating alt-text, content screening, or any time you need to understand an image without human review. Models available include MobileNet, ResNeXt, and ConvNeXt.
  • Object detection: Identifies and locates specific objects within an image. You could use this to build inventory systems that count items or interactive AR experiences. Examples include YOLOv8 and DETR.
  • Body pose detection: Tracks a person's body in an image or video. Common use cases are gesture-controlled interfaces, virtual try-on experiences, and monitoring movement for telehealth platforms. You can use models like MoveNet and BlazePose.
  • Face keypoint detection: Detects features on a person's face. This is for use cases like secure authentication, emotion detection, and real-time filters. Examples include MediaPipe FaceMesh and OpenPose.
  • Hand pose detection: Models a person's hands, enabling touch-free navigation via gestures or sign-language translation. Consider these for interfaces where touching a screen is impractical, like in medical settings or large presentations. Models include MediaPipe Hands.
  • Handwriting recognition: Converts stylus or pen input into digital text. This is useful for digitizing forms or building note-taking applications. Example technologies for OCR include MiniCPM-o, H2OVL-Mississippi, and Surya.
  • Image segmentation: Analyzes each pixel to identify distinct image regions, rather than tagging the content of an object in a bounded box. This is used for tasks that require precise attention to areas of an image, such as removing backgrounds or isolating specific elements in photo editing. Try models like Segment Anything (SAM) or Mask R-CNN.
  • Image generation: Creates new images based on a text prompt without licensing concerns. This can power personalized avatars or custom marketing visuals. Diffusion models such as Nano Banana, Flux, and Qwen Image are examples.

Audio processing

For use with audio files, there are models for generation, and for classifying audio without needing to transcribe it:

  • Audio classification: Labels and identifies sounds, from speech to environmental noises to music. This applies to automatic audio content tagging that doesn't need human review. Models like Wav2Vec2 and AudioMAE fit this role.
  • Audio generation: This model type generates sound effects or other audio on demand without licensing. These are often custom and hyper-specific, so there are no standard models to reference.
  • Text-to-speech (TTS): Turns written content into natural-sounding speech. Useful for providing accessibility options, creating voiceover narration, or building voice-enabled multilingual interfaces. Orpheus and Sesame CSM are models you can use.
  • Speech-to-text (STT): Transcribes spoken audio. This can be used to provide real-time transcription in meetings, enable voice controls, or caption video content. Try models such as Whisper Web Turbo, NVIDIA Canary, and Kyutai.

Text processing

  • Natural language processing (NLP): You can categorize user messages or support tickets into themes for sentiment analysis or moderation. This works well for sorting and tagging large amounts of text. Many older models like BERT, DistilBERT, and RoBERTa are tuned for this type of task.
  • Conversational AI: For chat interfaces and assistants, you can now use language models small enough to fit on a device without needing the energy footprint of an LLM hosted server-side. For example, consider Gemma 2 27B, Llama 3.1, and Qwen2.5.
  • Translation models: Small Language Models (SLMs) handle efficient translation tasks and support multiple languages. These smaller models can keep sensitive documents private because they can process on-device. Geni Nano, Granite 1.5B, GSmolLM3, and Qwen 3.4B are solid options in this category.

The need for model transparency

To properly compare small and efficient models, you need standardized resource usage figures, much like a nutrition label. Different models consume varying amounts of energy and other resources depending on the hardware and where it is hosted. There's no standard baseline measurement for this yet, but there is a movement underway to create one.

Model cards are a standardized approach for reporting a model's intended use, limitations, ethical considerations, and performance. IBM's AI Factsheets cover an AI model's lifecycle, accountability, and governance, making them common in enterprise. Newer AI regulations, such as the EU AI Act, NIST AI Risk Management Framework, and ISO 42001, are also pushing for more self-reporting.

Google has also called for inference cost transparency across the industry. These figures could be added to existing model factsheets.

Avoiding hidden costs

Right-sized AI is a pragmatic choice for your customers and your business. The benefits are not just sustainability metrics, but also lower latency, a more performant user experience, and reduced operating costs.

You can push the industry forward by making this type of info available for all users.

Asking your hosted model providers to disclose the baseline energy and resources their models require for training and inference will encourage transparency for you and other developers who are trying to compare options. If enough customers demand these details, model providers will make actual measurements available to enhance their fact cards - and make smarter choices possible for everyone.