Close Menu
Ztoog
    What's Hot
    AI

    With AI, researchers predict the location of virtually any protein within a human cell | Ztoog

    Technology

    What’s Next for Clearpath Robotics?

    Technology

    New features and security patch incoming

    Important Pages:
    • About Us
    • Contact us
    • Privacy Policy
    • Terms & Conditions
    Facebook X (Twitter) Instagram Pinterest
    Facebook X (Twitter) Instagram Pinterest
    Ztoog
    • Home
    • The Future

      Any wall can be turned into a camera to see around corners

      JD Vance and President Trump’s Sons Hype Bitcoin at Las Vegas Conference

      AI may already be shrinking entry-level jobs in tech, new research suggests

      Today’s NYT Strands Hints, Answer and Help for May 26 #449

      LiberNovo Omni: The World’s First Dynamic Ergonomic Chair

    • Technology

      A Replit employee details a critical security flaw in web apps created using AI-powered app builder Lovable that exposes API keys and personal info of app users (Reed Albergotti/Semafor)

      Gemini in Google Drive can now help you skip watching that painfully long Zoom meeting

      Apple iPhone exports from China to the US fall 76% as India output surges

      Today’s NYT Wordle Hints, Answer and Help for May 26, #1437

      5 Skills Kids (and Adults) Need in an AI World – O’Reilly

    • Gadgets

      8 Best Vegan Meal Delivery Services and Kits (2025), Tested and Reviewed

      Google Home is getting deeper Gemini integration and a new widget

      Google Announces AI Ultra Subscription Plan With Premium Features

      Google shows off Android XR-based glasses, announces Warby Parker team-up

      The market’s down, but this OpenAI for the stock market can help you trade up

    • Mobile

      Microsoft is done being subtle – this new tool screams “upgrade now”

      Wallpaper Wednesday: Android wallpapers 2025-05-28

      Google can make smart glasses accessible with Warby Parker, Gentle Monster deals

      vivo T4 Ultra specs leak

      Forget screens: more details emerge on the mysterious Jony Ive + OpenAI device

    • Science

      Analysts Say Trump Trade Wars Would Harm the Entire US Energy Sector, From Oil to Solar

      Do we have free will? Quantum experiments may soon reveal the answer

      Was Planet Nine exiled from the solar system as a baby?

      How farmers can help rescue water-loving birds

      A trip to the farm where loofahs grow on vines

    • AI

      Rationale engineering generates a compact new tool for gene therapy | Ztoog

      The AI Hype Index: College students are hooked on ChatGPT

      Learning how to predict rare kinds of failures | Ztoog

      Anthropic’s new hybrid AI model can work on tasks autonomously for hours at a time

      AI learns how vision and sound are connected, without human intervention | Ztoog

    • Crypto

      GameStop bought $500 million of bitcoin

      CoinW Teams Up with Superteam Europe to Conclude Solana Hackathon and Accelerate Web3 Innovation in Europe

      Ethereum Net Flows Turn Negative As Bulls Push For $3,500

      Bitcoin’s Power Compared To Nuclear Reactor By Brazilian Business Leader

      Senate advances GENIUS Act after cloture vote passes

    Ztoog
    Home » How Large Language Models are Redefining Data Compression and Providing Unique Insights into Machine Learning Scalability? Researchers from DeepMind Introduce a Novel Compression Paradigm
    AI

    How Large Language Models are Redefining Data Compression and Providing Unique Insights into Machine Learning Scalability? Researchers from DeepMind Introduce a Novel Compression Paradigm

    Facebook Twitter Pinterest WhatsApp
    How Large Language Models are Redefining Data Compression and Providing Unique Insights into Machine Learning Scalability? Researchers from DeepMind Introduce a Novel Compression Paradigm
    Share
    Facebook Twitter LinkedIn Pinterest WhatsApp

    Was this response higher or worse?BetterWorseSame

    It has been stated that info principle and machine studying are “two sides of the same coin” due to their shut relationship. One beautiful relationship is the basic similarity between probabilistic information fashions and lossless compression. The important principle defining this idea is the supply coding theorem, which states that the expected message size in bits of a great entropy encoder equals the adverse log2 chance of the statistical mannequin. In different phrases, lowering the quantity of bits wanted for every message is similar to growing the log2 -likelihood. Different methods to attain lossless compression with a probabilistic mannequin embrace Huffman coding, arithmetic coding, and uneven numeral programs. 

    Figure 1 | Arithmetic encoding of the sequence ‘AIXI’ with a probabilistic (language) mannequin P (each in blue) yields the binary code ‘0101001’ (in inexperienced). Data is compressed through arithmetic coding by giving symbols sure intervals relying on the chance given by P. It progressively smoothes out these pauses to supply compressed bits that stand in for the unique message. Based on the incoming compressed bits, arithmetic coding initializes an interval throughout decoding. To rebuild the unique message, it iteratively matches intervals with symbols utilizing the chances offered by P.

    The complete compression effectivity depends on the capabilities of the probabilistic mannequin since arithmetic coding is understood to be optimum by way of coding size (Fig. 1). Furthermore, enormous pre-trained Transformers, often known as basis fashions, have lately demonstrated wonderful efficiency throughout a number of prediction duties and are thus enticing candidates to be used with arithmetic coding. Transformer-based compression with arithmetic coding has generated cutting-edge ends in on-line and offline environments. The offline possibility they take into account of their work includes coaching the mannequin on an exterior dataset earlier than utilizing it to compress a (maybe completely different) information stream. In the net context, a pseudo-randomly initialized mannequin is straight away educated on the stream of knowledge that’s to be compressed. As a consequence, offline compression makes use of a mounted set of mannequin parameters and is completed in context. 

    Transformers are completely suited to offline discount since they’ve proven excellent in-context studying capabilities. Transformers are taught to compress successfully, as they’ll describe on this activity. Therefore, they should have robust contextual studying abilities. The context size, a essential offline compression limiting issue, determines the utmost variety of bytes a mannequin can squeeze concurrently. Transformers are computationally intensive and can solely compress a small quantity of knowledge (a “token” is programmed with 2 or 3 bytes). Since many tough predicting duties (akin to algorithmic reasoning or long-term reminiscence) want prolonged contexts, extending the context lengths of those fashions is a vital subject that’s receiving extra consideration. The in-context compression view sheds mild on how the current basis fashions fail. Researchers from Google DeepMind and Meta AI & Inria promote utilizing compression to discover the prediction downside and assess how effectively large (basis) fashions compress information. 

    They make the next contributions: 

    • They do empirical analysis on the inspiration fashions’ capability for lossless compression. To that objective, they discover arithmetic coding’s position in predictive mannequin compression and draw consideration to the connection between the 2 fields of examine. 

    • They exhibit that basis fashions with in-context studying capabilities, educated totally on textual content, are general-purpose compressors. For occasion, Chinchilla 70B outperforms domain-specific compressors like PNG (58.5%) or FLAC (30.3%), attaining compression charges of 43.4% on ImageNet patches and 16.4% on LibriSpeech samples. 

    • They current a recent perspective on scaling legal guidelines by demonstrating that scaling is just not a magic repair and that the scale of the dataset units a strict higher restrict on mannequin measurement by way of compression efficiency. 

    • They use compressors as generative fashions and use the compression-prediction equivalence to characterize the underlying compressor’s efficiency graphically.

    • They present that tokenization, which might be regarded as a pre-compression, doesn’t, on common, enhance compression efficiency. Instead, it permits fashions to extend the knowledge content material of their atmosphere and is often used to reinforce prediction efficiency.


    Check out the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t neglect to hitch our 30k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, the place we share the most recent AI analysis information, cool AI initiatives, and extra.

    If you want our work, you’ll love our publication..


    Aneesh Tickoo is a consulting intern at MarktechPost. He is at present pursuing his undergraduate diploma in Data Science and Artificial Intelligence from the Indian Institute of Technology(IIT), Bhilai. He spends most of his time engaged on initiatives geared toward harnessing the facility of machine studying. His analysis curiosity is picture processing and is obsessed with constructing options round it. He loves to attach with individuals and collaborate on fascinating initiatives.


    🚀 The finish of mission administration by people (Sponsored)

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp

    Related Posts

    AI

    Rationale engineering generates a compact new tool for gene therapy | Ztoog

    AI

    The AI Hype Index: College students are hooked on ChatGPT

    AI

    Learning how to predict rare kinds of failures | Ztoog

    AI

    Anthropic’s new hybrid AI model can work on tasks autonomously for hours at a time

    AI

    AI learns how vision and sound are connected, without human intervention | Ztoog

    AI

    How AI is introducing errors into courtrooms

    AI

    With AI, researchers predict the location of virtually any protein within a human cell | Ztoog

    AI

    Google DeepMind’s new AI agent cracks real-world problems better than humans can

    Leave A Reply Cancel Reply

    Follow Us
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    Top Posts
    The Future

    Part Two is Still a Hit, Passes First Dune’s Box Office

    Image: Warner Bros.Two weeks in the past, Dune: Part Two got here out in theaters,…

    Crypto

    Ukraine and Binance Boast Transformed Lives with Tech Education Project

    Share this text Overcoming job loss attributable to the continued battle, greater than 2,000 Ukrainians…

    Gadgets

    Apple Maps vs Google Maps: How their offline downloadable maps compare

    Enlarge / With offline maps, Apple has achieved parity with Google Maps on this one…

    Mobile

    A glaring Android TV security flaw might put your Gmail at risk

    What it’s worthwhile to knowA loophole in Android TV might permit unauthorized entry to Gmail…

    Technology

    Ham Radio Inspired This Scranton University Student to Pursue Engineering

    Many school college students take part in sports activities, pay attention to music, or play…

    Our Picks
    Mobile

    Wired earbuds are making a comeback, and I’m excited

    AI

    Meet MetaGPT: The Open-Source AI Framework That Transforms GPTs into Engineers, Architects, and Managers

    Science

    Spinach and Sunlight, the Ingredients for Converting Carbon Dioxide into Fertilizer

    Categories
    • AI (1,493)
    • Crypto (1,753)
    • Gadgets (1,804)
    • Mobile (1,850)
    • Science (1,866)
    • Technology (1,802)
    • The Future (1,648)
    Most Popular
    Technology

    Model Collapse: An Experiment – O’Reilly

    Gadgets

    The LifeStraw Peak Solo is a tiny water filter for camping and emergencies

    AI

    Symbol tuning improves in-context learning in language models – Google Research Blog

    Ztoog
    Facebook X (Twitter) Instagram Pinterest
    • Home
    • About Us
    • Contact us
    • Privacy Policy
    • Terms & Conditions
    © 2025 Ztoog.

    Type above and press Enter to search. Press Esc to cancel.