Evaluating Dynamic Knowledge Encoding: Experimental Setup for Multi-Hop Logical Reasoning
Table of Links
Abstract and 1. Introduction
-
Background
-
Method
-
Experiments
4.1 Multi-hop Reasoning Performance
4.2 Reasoning with Distractors
4.3 Generalization to Real-World knowledge
4.4 Run-time Analysis
4.5 Memorizing Knowledge
-
Related Work
-
Conclusion, Acknowledgements, and References
\ A. Dataset
B. In-context Reasoning with Distractors
C. Implementation Details
D. Adaptive Learning Rate
E. Experiments with Large Language Models
4 Experiments
Setup We conduct our experiments on three datasets focusing on multi-hop logical reasoning over natural language knowledge: ProofWriter [73], which measures the model’s ability to emulate reasoning over facts and rules expressed in natural language; CLUTRR-SG [28], which is generated from the CLUTRR [71] benchmark, a logical reasoning task that involves reasoning over family relationships between entities grounded in first-order logical proofs; and FOLIO [29], a reasoning benchmark with first-order logical reasoning problems written by expert annotators based on real-world knowledge. Each problem in these datasets requires multiple reasoning hops to answer.[1]
\ We compare our method against the following baselines: (1) a fine-tuned model that performs a forward pass on only the question without access to the knowledge (No-Facts), (2) a fine-tuned model that performs a forward pass on only the knowledge without access to the question (No-Question), (3) a model trained using RECKONING with random knowledge that is not relevant to the questions (Random-Facts), and (4) an ICR baseline that concatenates the knowledge K with the question x in a single context and is trained using supervised learning to predict the answer (FT-ICR). Our first three baselines sanity-check whether any surface-level patterns in the questions and facts can be exploited to make accurate predictions. The last baseline compares RECKONING to the conventional way of reasoning with language models. Unless stated otherwise, we use the GPT-2-small [59] model (∼124M parameters) as our initialization and refer by RECKONING to our method trained with the multi-task objective. We compute each score from the average across three different runs. For more details on the implementation, datasets, and examples, see Appendix A and Appendix C.
\
:::info Authors:
(1) Zeming Chen, EPFL (zeming.chen@epfl.ch);
(2) Gail Weiss, EPFL (antoine.bosselut@epfl.ch);
(3) Eric Mitchell, Stanford University (eric.mitchell@cs.stanford.edu)';
(4) Asli Celikyilmaz, Meta AI Research (aslic@meta.com);
(5) Antoine Bosselut, EPFL (antoine.bosselut@epfl.ch).
:::
:::info This paper is available on arxiv under CC BY 4.0 DEED license.
:::
[1] In ProofWriter, the number of reasoning hops is called the proof depth. To unify the presentation of the results, we use the term “hop” to describe the number of reasoning steps for both datasets.
You May Also Like
Why Is Pi Network Token (PI) Up 22% on Oct. 29?
German VDMA cautions U.S. tariff expansion will impact over 50% of machine exports
BNB Price Drops 2% as the Dex Volume Tumbles Cautioning Further Downside
Highlights: The BNB price is down 2% to $1111.46, despite the trading volume spiking 26%. The BNB on-chain demand has slipped, with the open interest plummeting 3% showing a drop in demand. The technical outlook shows a tight tug-of-war, with the bulls attempting to overcome resistance zones. The BNB price is down 2% today, to trade at $1111.46. Despite the plunge, the daily trading volume has soared 26% showing increased market activity among traders. However, BNB Chain has seen declining network activity, with the open interest plummeting, signaling a drop in demand. On Chain Demand on BNB Cools Off The BNB Chain is in a state of cooldown of network activity, which indicates low on-chain demand. In most instances, when a network fails to ensure large volumes or revenues, it means that there is low demand or outflows to other networks. BNB DeFi Data: DeFiLlama According to DeFiLlama data, the volume of the Decentralized Exchanges (DEXs) is down to at least $2.12 billion in comparison to the high of $6.313 billion on October 8, which also means low on-chain liquidity. On the other hand, Coinglass data shows that the volume of BNB has grown by 3.97% to reach $4.95 billion. However, the open interest in BNB futures has dropped by 3.36% to reach $1.74 billion. This reduction in open interest is an indication of a conservative stance by investors since the number of new positions being opened is low. This could be an indication that investors are not so sure about the short-term price outlook. BNB Derivatives Data: CoinGlass Meanwhile, the long-to-short ratio is sitting at 0.9091. This shows that the traders are undecided on BNB price’s next move, as it sits below 1. BNB Price Moves Into Consolidation The chart displays the BNB/USD price action on a 4-hour timeframe, with the token currently hovering around $1111.46. The 50-day Simple Moving Average (SMA) is at $1113, while the 200-day SMA sits at $1129, cushioning the bulls against upside movement. The price has mostly been trending below both SMAs, indicating that the bears are having the upper hand. The BNB trading volume is up, soaring 26%, signaling the momentum is real. On the 4-hour chart, BNB is trading within a consolidation channel. In such a case, this pattern may act as an accumulation period, giving the bulls hind wings to break above resistance zones. BNB/USD 4-hour chart: TradingView Zooming in, the Relative Strength Index (RSI) sits at 44.15, below the 50 level. This shows weakening momentum in the BNB market, and might lead to the RSI plunging to the oversold region if the bulls don’t regain control. In the short term, the BNB price could move up to $1113 resistance and flip it into support. A close above this zone will see the bulls target $1126 resistance, giving the bulls strength to reclaim the $1230 mark. Conversely, if the resistance zones prove too strong, a dip towards $1012 could be plausible. In such a case, this could be a prime buy zone for the risk-takers. In the long term, if the token keeps the hype alive, the bulls may reclaim the $1375 high or higher. eToro Platform Best Crypto Exchange Over 90 top cryptos to trade Regulated by top-tier entities User-friendly trading app 30+ million users 9.9 Visit eToro eToro is a multi-asset investment platform. The value of your investments may go up or down. Your capital is at risk. Don’t invest unless you’re prepared to lose all the money you invest. This is a high-risk investment, and you should not expect to be protected if something goes wrong.