Close Menu
Ztoog
    What's Hot
    Technology

    What AMD Learned From Its Big Chiplet Push

    Mobile

    Release date, price, specs, rumors, and more

    Science

    Futuristic adhesive imitates the gecko’s unique sticking skills

    Important Pages:
    • About Us
    • Contact us
    • Privacy Policy
    • Terms & Conditions
    Facebook X (Twitter) Instagram Pinterest
    Facebook X (Twitter) Instagram Pinterest
    Ztoog
    • Home
    • The Future

      How I Turn Unstructured PDFs into Revenue-Ready Spreadsheets

      Is it the best tool for 2025?

      The clocks that helped define time from London’s Royal Observatory

      Summer Movies Are Here, and So Are the New Popcorn Buckets

      India-Pak conflict: Pak appoints ISI chief, appointment comes in backdrop of the Pahalgam attack

    • Technology

      Ensure Hard Work Is Recognized With These 3 Steps

      Cicada map 2025: Where will Brood XIV cicadas emerge this spring?

      Is Duolingo the face of an AI jobs crisis?

      The US DOD transfers its AI-based Open Price Exploration for National Security program to nonprofit Critical Minerals Forum to boost Western supply deals (Ernest Scheyder/Reuters)

      The more Google kills Fitbit, the more I want a Fitbit Sense 3

    • Gadgets

      Maono Caster G1 Neo & PD200X Review: Budget Streaming Gear for Aspiring Creators

      Apple plans to split iPhone 18 launch into two phases in 2026

      Upgrade your desk to Starfleet status with this $95 USB-C hub

      37 Best Graduation Gift Ideas (2025): For College Grads

      Backblaze responds to claims of “sham accounting,” customer backups at risk

    • Mobile

      Samsung Galaxy S25 Edge promo materials leak

      What are people doing with those free T-Mobile lines? Way more than you’d expect

      Samsung doesn’t want budget Galaxy phones to use exclusive AI features

      COROS’s charging adapter is a neat solution to the smartwatch charging cable problem

      Fortnite said to return to the US iOS App Store next week following court verdict

    • Science

      Failed Soviet probe will soon crash to Earth – and we don’t know where

      Trump administration cuts off all future federal funding to Harvard

      Does kissing spread gluten? New research offers a clue.

      Why Balcony Solar Panels Haven’t Taken Off in the US

      ‘Dark photon’ theory of light aims to tear up a century of physics

    • AI

      How to build a better AI benchmark

      Q&A: A roadmap for revolutionizing health care through data-driven innovation | Ztoog

      This data set helps researchers spot harmful stereotypes in LLMs

      Making AI models more trustworthy for high-stakes settings | Ztoog

      The AI Hype Index: AI agent cyberattacks, racing robots, and musical models

    • Crypto

      ‘The Big Short’ Coming For Bitcoin? Why BTC Will Clear $110,000

      Bitcoin Holds Above $95K Despite Weak Blockchain Activity — Analytics Firm Explains Why

      eToro eyes US IPO launch as early as next week amid easing concerns over Trump’s tariffs

      Cardano ‘Looks Dope,’ Analyst Predicts Big Move Soon

      Speak at Ztoog Disrupt 2025: Applications now open

    Ztoog
    Home » Autonomous visual information seeking with large language models – Google Research Blog
    AI

    Autonomous visual information seeking with large language models – Google Research Blog

    Facebook Twitter Pinterest WhatsApp
    Autonomous visual information seeking with large language models – Google Research Blog
    Share
    Facebook Twitter LinkedIn Pinterest WhatsApp

    Posted by Ziniu Hu, Student Researcher, and Alireza Fathi, Research Scientist, Google Research, Perception Team

    There has been nice progress in direction of adapting large language models (LLMs) to accommodate multimodal inputs for duties together with picture captioning, visual query answering (VQA), and open vocabulary recognition. Despite such achievements, present state-of-the-art visual language models (VLMs) carry out inadequately on visual information seeking datasets, akin to Infoseek and OK-VQA, the place exterior data is required to reply the questions.

    Examples of visual information seeking queries the place exterior data is required to reply the query. Images are taken from the OK-VQA dataset.

    In “AVIS: Autonomous Visual Information Seeking with Large Language Models”, we introduce a novel methodology that achieves state-of-the-art outcomes on visual information seeking duties. Our methodology integrates LLMs with three forms of instruments: (i) pc imaginative and prescient instruments for extracting visual information from photos, (ii) an online search software for retrieving open world data and details, and (iii) a picture search software to glean related information from metadata related with visually related photos. AVIS employs an LLM-powered planner to decide on instruments and queries at every step. It additionally makes use of an LLM-powered reasoner to research software outputs and extract key information. A working reminiscence part retains information all through the method.

    An instance of AVIS’s generated workflow for answering a difficult visual information seeking query. The enter picture is taken from the Infoseek dataset.

    Comparison to earlier work

    Recent research (e.g., Chameleon, ViperGPT and MM-ReAct) explored including instruments to LLMs for multimodal inputs. These methods comply with a two-stage course of: planning (breaking down questions into structured packages or directions) and execution (utilizing instruments to collect information). Despite success in fundamental duties, this method usually falters in complicated real-world eventualities.

    There has additionally been a surge of curiosity in making use of LLMs as autonomous brokers (e.g., WebGPT and ReAct). These brokers work together with their surroundings, adapt primarily based on real-time suggestions, and obtain objectives. However, these strategies don’t prohibit the instruments that may be invoked at every stage, resulting in an immense search house. Consequently, even essentially the most superior LLMs as we speak can fall into infinite loops or propagate errors. AVIS tackles this by way of guided LLM use, influenced by human choices from a person research.

    Informing LLM choice making with a person research

    Many of the visual questions in datasets akin to Infoseek and OK-VQA pose a problem even for people, usually requiring the help of numerous instruments and APIs. An instance query from the OK-VQA dataset is proven under. We performed a person research to know human decision-making when utilizing exterior instruments.

    We performed a person research to know human decision-making when utilizing exterior instruments. Image is taken from the OK-VQA dataset.

    The customers had been geared up with an an identical set of instruments as our methodology, together with PALI, PaLM, and internet search. They acquired enter photos, questions, detected object crops, and buttons linked to picture search outcomes. These buttons provided various information concerning the detected object crops, akin to data graph entities, related picture captions, associated product titles, and an identical picture captions.

    We document person actions and outputs and use it as a information for our system in two key methods. First, we assemble a transition graph (proven under) by analyzing the sequence of choices made by customers. This graph defines distinct states and restricts the out there set of actions at every state. For instance, initially state, the system can take solely certainly one of these three actions: PALI caption, PALI VQA, or object detection. Second, we use the examples of human decision-making to information our planner and reasoner with related contextual situations to reinforce the efficiency and effectiveness of our system.

    AVIS transition graph.

    General framework

    Our method employs a dynamic decision-making technique designed to reply to visual information-seeking queries. Our system has three major elements. First, we’ve a planner to find out the next motion, together with the suitable API name and the question it must course of. Second, we’ve a working reminiscence that retains information concerning the outcomes obtained from API executions. Last, we’ve a reasoner, whose position is to course of the outputs from the API calls. It determines whether or not the obtained information is ample to supply the ultimate response, or if further knowledge retrieval is required.

    The planner undertakes a sequence of steps every time a call is required relating to which software to make use of and what question to ship to it. Based on the current state, the planner supplies a variety of potential subsequent actions. The potential motion house could also be so large that it makes the search house intractable. To tackle this problem, the planner refers back to the transition graph to remove irrelevant actions. The planner additionally excludes the actions which have already been taken earlier than and are saved within the working reminiscence.

    Next, the planner collects a set of related in-context examples which can be assembled from the selections beforehand made by people throughout the person research. With these examples and the working reminiscence that holds knowledge collected from previous software interactions, the planner formulates a immediate. The immediate is then despatched to the LLM, which returns a structured reply, figuring out the subsequent software to be activated and the question to be dispatched to it. This design permits the planner to be invoked a number of instances all through the method, thereby facilitating dynamic decision-making that steadily results in answering the enter question.

    We make use of a reasoner to research the output of the software execution, extract the helpful information and determine into which class the software output falls: informative, uninformative, or last reply. Our methodology makes use of the LLM with applicable prompting and in-context examples to carry out the reasoning. If the reasoner concludes that it’s prepared to supply a solution, it should output the ultimate response, thus concluding the duty. If it determines that the software output is uninformative, it should revert again to the planner to pick out one other motion primarily based on the present state. If it finds the software output to be helpful, it should modify the state and switch management again to the planner to make a brand new choice on the new state.

    AVIS employs a dynamic decision-making technique to reply to visual information-seeking queries.

    Results

    We consider AVIS on Infoseek and OK-VQA datasets. As proven under, even sturdy visual-language models, akin to OFA and PaLI, fail to yield excessive accuracy when fine-tuned on Infoseek. Our method (AVIS), with out fine-tuning, achieves 50.7% accuracy on the unseen entity cut up of this dataset.

    AVIS visual query answering outcomes on Infoseek dataset. AVIS achieves larger accuracy compared to earlier baselines primarily based on PaLI, PaLM and OFA.

    Our outcomes on the OK-VQA dataset are proven under. AVIS with few-shot in-context examples achieves an accuracy of 60.2%, larger than a lot of the earlier works. AVIS achieves decrease however comparable accuracy compared to the PALI mannequin fine-tuned on OK-VQA. This distinction, in comparison with Infoseek the place AVIS outperforms fine-tuned PALI, is because of the truth that most question-answer examples in OK-VQA depend on frequent sense data somewhat than on fine-grained data. Therefore, PaLI is ready to encode such generic data within the mannequin parameters and doesn’t require exterior data.

    Visual query answering outcomes on A-OKVQA. AVIS achieves larger accuracy compared to earlier works that use few-shot or zero-shot studying, together with Flamingo, PaLI and ViperGPT. AVIS additionally achieves larger accuracy than a lot of the earlier works which can be fine-tuned on OK-VQA dataset, together with REVEAL, ReVIVE, KAT and KRISP, and achieves outcomes which can be near the fine-tuned PaLI mannequin.

    Conclusion

    We current a novel method that equips LLMs with the flexibility to make use of quite a lot of instruments for answering knowledge-intensive visual questions. Our methodology, anchored in human decision-making knowledge collected from a person research, employs a structured framework that makes use of an LLM-powered planner to dynamically determine on software choice and question formation. An LLM-powered reasoner is tasked with processing and extracting key information from the output of the chosen software. Our methodology iteratively employs the planner and reasoner to leverage totally different instruments till all vital information required to reply the visual query is amassed.

    Acknowledgements

    This analysis was performed by Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A. Ross, Cordelia Schmid and Alireza Fathi.

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp

    Related Posts

    AI

    How to build a better AI benchmark

    AI

    Q&A: A roadmap for revolutionizing health care through data-driven innovation | Ztoog

    AI

    This data set helps researchers spot harmful stereotypes in LLMs

    AI

    Making AI models more trustworthy for high-stakes settings | Ztoog

    AI

    The AI Hype Index: AI agent cyberattacks, racing robots, and musical models

    AI

    Novel method detects microbial contamination in cell cultures | Ztoog

    AI

    Seeing AI as a collaborator, not a creator

    AI

    “Periodic table of machine learning” could fuel AI discovery | Ztoog

    Leave A Reply Cancel Reply

    Follow Us
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    Top Posts
    Technology

    Back at OpenAI, Sam Altman Outlines the Company’s Priorities

    OpenAI mentioned on Wednesday that it had accomplished the first section of a brand new…

    Crypto

    Cryptocurrency Seizure Bill Successfully Passes In The UK

    A cryptocurrency seizure invoice to fight financial crime within the UK is within the pipeline,…

    Science

    Pooping on the Moon Is a Messy Business

    In addition to elevating these authorized and moral quandaries, the Apollo waste luggage have additionally…

    The Future

    Musk-Branson space race heats up as Virgin Galactic announces commercial space mission

    By clicking “Accept All Cookies”, you conform to the storing of cookies in your machine…

    Mobile

    One of Android 14’s anticipated features is disabled in the beta builds; you can enable it now

    The one huge downside with the Android again gesture is that you by no means…

    Our Picks
    Technology

    Here’s What to Do When You’re Kicked Off Someone Else’s Netflix Account

    AI

    Meet DiffPoseTalk: A New Speech-to-3D Animation Artificial Intelligence Framework

    Crypto

    Ethereum (ETH) Price Drops Due Whale Selling, Key Levels

    Categories
    • AI (1,482)
    • Crypto (1,744)
    • Gadgets (1,796)
    • Mobile (1,839)
    • Science (1,853)
    • Technology (1,789)
    • The Future (1,635)
    Most Popular
    Gadgets

    Gigabyte BIOS update outs next-gen AMD Ryzen APUs with upgraded Radeon GPUs

    Technology

    Alex Murdaugh trial: South Carolina lawyer found guilty of killing his wife and son

    Gadgets

    Snoopy MoonSwatch: Price, Availability, Release Date

    Ztoog
    Facebook X (Twitter) Instagram Pinterest
    • Home
    • About Us
    • Contact us
    • Privacy Policy
    • Terms & Conditions
    © 2025 Ztoog.

    Type above and press Enter to search. Press Esc to cancel.