Close Menu
Ztoog
    What's Hot
    AI

    Meet LMDrive: A Unique AI Framework For Language-Guided, End-To-End, Closed-Loop Autonomous Driving

    The Future

    How CFOs can reduce SaaS spend by 30% in these tough times

    AI

    Researchers from Meta and UNC-Chapel Hill Introduce Branch-Solve-Merge: A Revolutionary Program Enhancing Large Language Models’ Performance in Complex Language Tasks

    Important Pages:
    • About Us
    • Contact us
    • Privacy Policy
    • Terms & Conditions
    Facebook X (Twitter) Instagram Pinterest
    Facebook X (Twitter) Instagram Pinterest
    Ztoog
    • Home
    • The Future

      JD Vance and President Trump’s Sons Hype Bitcoin at Las Vegas Conference

      AI may already be shrinking entry-level jobs in tech, new research suggests

      Today’s NYT Strands Hints, Answer and Help for May 26 #449

      LiberNovo Omni: The World’s First Dynamic Ergonomic Chair

      Common Security Mistakes Made By Businesses and How to Avoid Them

    • Technology

      Gemini in Google Drive can now help you skip watching that painfully long Zoom meeting

      Apple iPhone exports from China to the US fall 76% as India output surges

      Today’s NYT Wordle Hints, Answer and Help for May 26, #1437

      5 Skills Kids (and Adults) Need in an AI World – O’Reilly

      How To Come Back After A Layoff

    • Gadgets

      8 Best Vegan Meal Delivery Services and Kits (2025), Tested and Reviewed

      Google Home is getting deeper Gemini integration and a new widget

      Google Announces AI Ultra Subscription Plan With Premium Features

      Google shows off Android XR-based glasses, announces Warby Parker team-up

      The market’s down, but this OpenAI for the stock market can help you trade up

    • Mobile

      Microsoft is done being subtle – this new tool screams “upgrade now”

      Wallpaper Wednesday: Android wallpapers 2025-05-28

      Google can make smart glasses accessible with Warby Parker, Gentle Monster deals

      vivo T4 Ultra specs leak

      Forget screens: more details emerge on the mysterious Jony Ive + OpenAI device

    • Science

      Do we have free will? Quantum experiments may soon reveal the answer

      Was Planet Nine exiled from the solar system as a baby?

      How farmers can help rescue water-loving birds

      A trip to the farm where loofahs grow on vines

      AI Is Eating Data Center Power Demand—and It’s Only Getting Worse

    • AI

      The AI Hype Index: College students are hooked on ChatGPT

      Learning how to predict rare kinds of failures | Ztoog

      Anthropic’s new hybrid AI model can work on tasks autonomously for hours at a time

      AI learns how vision and sound are connected, without human intervention | Ztoog

      How AI is introducing errors into courtrooms

    • Crypto

      GameStop bought $500 million of bitcoin

      CoinW Teams Up with Superteam Europe to Conclude Solana Hackathon and Accelerate Web3 Innovation in Europe

      Ethereum Net Flows Turn Negative As Bulls Push For $3,500

      Bitcoin’s Power Compared To Nuclear Reactor By Brazilian Business Leader

      Senate advances GENIUS Act after cloture vote passes

    Ztoog
    Home » Learning the importance of training data under concept drift – Google Research Blog
    AI

    Learning the importance of training data under concept drift – Google Research Blog

    Facebook Twitter Pinterest WhatsApp
    Learning the importance of training data under concept drift – Google Research Blog
    Share
    Facebook Twitter LinkedIn Pinterest WhatsApp

    Posted by Nishant Jain, Pre-doctoral Researcher, and Pradeep Shenoy, Research Scientist, Google Research

    The continually altering nature of the world round us poses a major problem for the growth of AI fashions. Often, fashions are skilled on longitudinal data with the hope that the training data used will precisely characterize inputs the mannequin might obtain in the future. More typically, the default assumption that each one training data are equally related usually breaks in apply. For instance, the determine beneath reveals pictures from the CLEAR nonstationary studying benchmark, and it illustrates how visible options of objects evolve considerably over a ten 12 months span (a phenomenon we consult with as sluggish concept drift), posing a problem for object categorization fashions.

    Sample pictures from the CLEAR benchmark. (Adapted from Lin et al.)

    Alternative approaches, comparable to on-line and continuous studying, repeatedly replace a mannequin with small quantities of latest data with a purpose to maintain it present. This implicitly prioritizes latest data, as the learnings from previous data are progressively erased by subsequent updates. However in the actual world, totally different varieties of data lose relevance at totally different charges, so there are two key points: 1) By design they focus solely on the most up-to-date data and lose any sign from older data that’s erased. 2) Contributions from data situations decay uniformly over time irrespective of the contents of the data.

    In our latest work, “Instance-Conditional Timescales of Decay for Non-Stationary Learning”, we suggest to assign every occasion an importance rating throughout training with a purpose to maximize mannequin efficiency on future data. To accomplish this, we make use of an auxiliary mannequin that produces these scores utilizing the training occasion in addition to its age. This mannequin is collectively discovered with the main mannequin. We handle each the above challenges and obtain vital good points over different sturdy studying strategies on a variety of benchmark datasets for nonstationary studying. For occasion, on a latest large-scale benchmark for nonstationary studying (~39M photographs over a ten 12 months interval), we present as much as 15% relative accuracy good points by means of discovered reweighting of training data.

    The problem of concept drift for supervised studying

    To achieve quantitative perception into sluggish concept drift, we constructed classifiers on a latest picture categorization activity, comprising roughly 39M images sourced from social media web sites over a ten 12 months interval. We in contrast offline training, which iterated over all the training data a number of occasions in random order, and continuous training, which iterated a number of occasions over every month of data in sequential (temporal) order. We measured mannequin accuracy each throughout the training interval and through a subsequent interval the place each fashions had been frozen, i.e., not up to date additional on new data (proven beneath). At the finish of the training interval (left panel, x-axis = 0), each approaches have seen the similar quantity of data, however present a big efficiency hole. This is because of catastrophic forgetting, an issue in continuous studying the place a mannequin’s information of data from early on in the training sequence is diminished in an uncontrolled method. On the different hand, forgetting has its benefits — over the take a look at interval (proven on the proper), the continuous skilled mannequin degrades a lot much less quickly than the offline mannequin as a result of it’s much less depending on older data. The decay of each fashions’ accuracy in the take a look at interval is affirmation that the data is certainly evolving over time, and each fashions develop into more and more much less related.

    Comparing offline and regularly skilled fashions on the picture classification activity.

    Time-sensitive reweighting of training data

    We design a technique combining the advantages of offline studying (the flexibility of successfully reusing all out there data) and continuous studying (the means to downplay older data) to deal with sluggish concept drift. We construct upon offline studying, then add cautious management over the affect of previous data and an optimization goal, each designed to cut back mannequin decay in the future.

    Suppose we want to practice a mannequin, M, given some training data collected over time. We suggest to additionally practice a helper mannequin that assigns a weight to every level primarily based on its contents and age. This weight scales the contribution from that data level in the training goal for M. The goal of the weights is to enhance the efficiency of M on future data.

    In our work, we describe how the helper mannequin will be meta-learned, i.e., discovered alongside M in a way that helps the studying of the mannequin M itself. A key design selection of the helper mannequin is that we separated out instance- and age-related contributions in a factored method. Specifically, we set the weight by combining contributions from a number of totally different fastened timescales of decay, and be taught an approximate “assignment” of a given occasion to its most suited timescales. We discover in our experiments that this way of the helper mannequin outperforms many different options we thought-about, starting from unconstrained joint capabilities to a single timescale of decay (exponential or linear), because of its mixture of simplicity and expressivity. Full particulars could also be present in the paper.

    Instance weight scoring

    The high determine beneath reveals that our discovered helper mannequin certainly up-weights extra modern-looking objects in the CLEAR object recognition problem; older-looking objects are correspondingly down-weighted. On nearer examination (backside determine beneath, gradient-based characteristic importance evaluation), we see that the helper mannequin focuses on the main object inside the picture, versus, e.g., background options that will spuriously be correlated with occasion age.

    Sample pictures from the CLEAR benchmark (digital camera & laptop classes) assigned the highest and lowest weights respectively by our helper mannequin.

    Feature importance evaluation of our helper mannequin on pattern pictures from the CLEAR benchmark.

    Results

    Gains on large-scale data

    We first research the large-scale picture categorization activity (PCAT) on the YFCC100M dataset mentioned earlier, utilizing the first 5 years of data for training and the subsequent 5 years as take a look at data. Our methodology (proven in pink beneath) improves considerably over the no-reweighting baseline (black) in addition to many different sturdy studying methods. Interestingly, our methodology intentionally trades off accuracy on the distant previous (training data unlikely to reoccur in the future) in alternate for marked enhancements in the take a look at interval. Also, as desired, our methodology degrades lower than different baselines in the take a look at interval.

    Comparison of our methodology and related baselines on the PCAT dataset.

    Broad applicability

    We validated our findings on a variety of nonstationary studying problem datasets sourced from the educational literature (see 1, 2, 3, 4 for particulars) that spans data sources and modalities (photographs, satellite tv for pc pictures, social media textual content, medical information, sensor readings, tabular data) and sizes (starting from 10k to 39M situations). We report vital good points in the take a look at interval when in comparison with the nearest printed benchmark methodology for every dataset (proven beneath). Note that the earlier best-known methodology could also be totally different for every dataset. These outcomes showcase the broad applicability of our method.

    Performance achieve of our methodology on a spread of duties finding out pure concept drift. Our reported good points are over the earlier best-known methodology for every dataset.

    Extensions to continuous studying

    Finally, we contemplate an attention-grabbing extension of our work. The work above described how offline studying will be prolonged to deal with concept drift utilizing concepts impressed by continuous studying. However, typically offline studying is infeasible — for instance, if the quantity of training data out there is simply too giant to keep up or course of. We tailored our method to continuous studying in an easy method by making use of temporal reweighting inside the context of every bucket of data getting used to sequentially replace the mannequin. This proposal nonetheless retains some limitations of continuous studying, e.g., mannequin updates are carried out solely on most-recent data, and all optimization selections (together with our reweighting) are solely remodeled that data. Nevertheless, our method persistently beats common continuous studying in addition to a variety of different continuous studying algorithms on the picture categorization benchmark (see beneath). Since our method is complementary to the concepts in lots of baselines in contrast right here, we anticipate even bigger good points when mixed with them.

    Results of our methodology tailored to continuous studying, in comparison with the newest baselines.

    Conclusion

    We addressed the problem of data drift in studying by combining the strengths of earlier approaches — offline studying with its efficient reuse of data, and continuous studying with its emphasis on more moderen data. We hope that our work helps enhance mannequin robustness to concept drift in apply, and generates elevated curiosity and new concepts in addressing the ubiquitous drawback of sluggish concept drift.

    Acknowledgements

    We thank Mike Mozer for a lot of attention-grabbing discussions in the early part of this work, in addition to very useful recommendation and suggestions throughout its growth.

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp

    Related Posts

    AI

    The AI Hype Index: College students are hooked on ChatGPT

    AI

    Learning how to predict rare kinds of failures | Ztoog

    AI

    Anthropic’s new hybrid AI model can work on tasks autonomously for hours at a time

    AI

    AI learns how vision and sound are connected, without human intervention | Ztoog

    AI

    How AI is introducing errors into courtrooms

    AI

    With AI, researchers predict the location of virtually any protein within a human cell | Ztoog

    AI

    Google DeepMind’s new AI agent cracks real-world problems better than humans can

    AI

    Study shows vision-language models can’t handle queries with negation words | Ztoog

    Leave A Reply Cancel Reply

    Follow Us
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    Top Posts
    Mobile

    Xiaomi phones with unlocked bootloader won’t get HyperOS updates

    With the announcement of the brand new Xiaomi 14 sequence, the corporate additionally launched a…

    AI

    Meta’s global affairs president Nick Clegg: AI-generated election misinformation not yet happening at a systemic scale

    As voters will head to polls this 12 months in additional than 50 international locations,…

    The Future

    What is the future of AI? Google and the EU have very different ideas

    Alphabet CEO Sundar Pichai saying Google’s new strategy to AIBloomberg by way of Getty Images…

    Mobile

    Beeper Mini is currently down, possibly shut down by Apple (Update: partially working)

    UPDATE 12/9/23: Beeper has supplied an replace on each the Beeper Cloud and Beeper Mini…

    AI

    Top 50 AI Writing Tools To Try (August 2023)

    Grammarly is a good instrument for enhancing writing. It evaluations grammar, spelling, punctuation, and magnificence…

    Our Picks
    Science

    How to wrap your mind around the real multiverse

    AI

    Google AI Introduces WeatherBench 2: A Machine Learning Framework for Evaluating and Comparing Various Weather Forecasting Models

    Technology

    In Battle Over Health Care Costs, Private Equity Plays Both Sides

    Categories
    • AI (1,492)
    • Crypto (1,753)
    • Gadgets (1,804)
    • Mobile (1,850)
    • Science (1,865)
    • Technology (1,801)
    • The Future (1,647)
    Most Popular
    AI

    Technology, AI, Society and Culture – Ztoog

    Science

    Largest quake ever seen on Mars points to surprising seismic activity

    Crypto

    Massive Ethereum Whale Transfer Threatens To End ETH Rally, Here’s Why

    Ztoog
    Facebook X (Twitter) Instagram Pinterest
    • Home
    • About Us
    • Contact us
    • Privacy Policy
    • Terms & Conditions
    © 2025 Ztoog.

    Type above and press Enter to search. Press Esc to cancel.