Summary of Large Language Models on Small Resource-constrained Systems: Performance Characterization, Analysis and Trade-offs, by Liam Seymour et al.
Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs
by Liam Seymour, Basar Kutukcu, Sabur Baidya
First submitted to arxiv on: 19 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computational Complexity (cs.CC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract discusses the recent advancements in generative AI, specifically Large Language Models (LLMs), which are now publicly available services like ChatGPT that can be accessed through cloud servers. However, running LLMs locally on edge devices is necessary due to privacy and security concerns, as well as application requirements. Researchers have optimized transformer-based models for resource-constrained devices, but these efforts typically focus on older hardware. This study aims to provide a baseline characterization of recent embedded hardware for LLMs and develop a utility for batch testing on Jetson Orin devices using publicly available LLMs (Pythia) with varying parameters. The research presents experimental evaluations with different software and hardware settings, showcasing trade-off spaces and optimization choices. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary LLMs like ChatGPT are now easily accessible through cloud servers. But what if you need to run these models locally? That’s where this study comes in! It looks at how to make Large Language Models work on newer embedded devices called Jetson Orin, using a type of model called Pythia. The researchers tested different settings and showed what works best. |
Keywords
» Artificial intelligence » Optimization » Transformer
