While rules and exemplars are usually viewed as opposites, this paper argues that they form end points of the same distribution. By representing both rules and exemplars as (partial) trees, we can take into account the fluid middle ground between the two extremes. This insight is the starting point for a new theory of language learning that is based on the following idea: If a language learner does not know which phrase‐structure trees should be assigned to initial sentences, s/he allows (...) (implicitly) for all possible trees and lets linguistic experience decide which is the “best” tree for each sentence. The best tree is obtained by maximizing “structural analogy” between a sentence and previous sentences, which is formalized by the most probable shortest combination of subtrees from all trees of previous sentences. Corpus‐based experiments with this model on the Penn Treebank and the Childes database indicate that it can learn both exemplar‐based and rule‐based aspects of language, ranging from phrasal verbs to auxiliary fronting. By having learned the syntactic structures of sentences, we have also learned the grammar implicit in these structures, which can in turn be used to produce new sentences. We show that our model mimicks children’s language development from item‐based constructions to abstract constructions, and that the model can simulate some of the errors made by children in producing complex questions. (shrink)

In the Bayesian framework, a language learner should seek a grammar that explains observed data well and is also a priori probable. This paper proposes such a measure of prior probability. Indeed it develops a full statistical framework for lexicalized syntax. The learner's job is to discover the system of probabilistic transformations (often called lexical redundancy rules) that underlies the patterns of regular and irregular syntactic constructions listed in the lexicon. Specifically, the learner discovers what transformations apply in (...) the language, how often they apply, and in what contexts. It considers simpler systems of transformations to be more probable a priori. Experiments show that the learned transformations are more effective than previous statistical models at predicting the probabilities of lexical entries, especially those for which the learner had no direct evidence. (shrink)

Machine learning and statistical methods have yielded impressive results in a wide variety of natural language processing tasks. These advances have generally been regarded as engineering achievements. In fact it is possible to argue that the success of machine learning methods is significant for our understanding of the cognitive basis of language acquisition and processing. Recent work in unsupervised grammar induction is particularly relevant to this issue. It suggests that knowledge of language can be achieved through general (...) learning procedures, and that a richly articulated language faculty is not required to explain its acquisition. (shrink)

An inductive logic is a system of inference that describes the relation between propositions on data, and propositions that extend beyond the data, such as predictions over future data, and general conclusions on all possible data. Statistics, on the other hand, is a mathematical discipline that describes procedures for deriving results about a population from sample data. These results include predictions on future samples, decisions on rejecting or accepting a hypothesis about the population, the determination of probability assignments over such (...) hypotheses, the selection of a statistical model for studying the population, and so on. Both inductive logic and statistics are calculi for getting from the given data to propositions or results that transcend the data. (shrink)

A method is described for inducing a type-logical grammar from a sample of bare sentence trees which are annotated by lambda terms, called term-labelled trees . Any type logic from a permitted class of multimodal logics may be specified for use with the procedure, which induces the lexicon of the grammar including the grammatical categories. A first stage of semantic bootstrapping is performed, which induces a general form lexicon from the sample of term-labelled trees using Fulop’s (J Log (...) Lang Inf 14(1):49–86, 2005) procedure. Next we present a two-stage procedure for performing distributional learning by unifying the lexical types that are initially discovered. The first structural unification algorithm in essence unifies the initial family of sets of types so that the resulting grammar will generate all term-labelled trees that follow the usage patterns evident from the learning sample. Further altering the lexical categories to generate a recursively extended language can be accomplished by a second unification. The combined unification algorithm is shown to yield a new type-logical lexicon that extends the learning sample to a possibly infinite (and possibly context-sensitive) language in a principled fashion. Finally, the complete learning strategy is analyzed from the perspective of algorithmic learning theory; the range of the procedure is shown to be a class of term-labelled tree languages which is finitely learnable from good examples (Lange et al in Algorithmic learning theory, Vol 872 of lecture notes in artificial intelligence, Springer, Berlin, pp 423–437), and so is identifiable in the limit as a corollary. (shrink)

Inductive logic generalizes the idea of logical entailment and provides standards for the evaluation of non-conclusive arguments. A main application of inductive logic is the generalization of observational data to theoretical models. In the empirical sciences, the mathematical theory of statistics addresses the same problem. This paper argues that there is no separable purely logical aspect of statistical inference in a variety of complex problems. Instead, statistical practice is often motivated by decision-theoretic considerations and resembles empirical science.

1. Deductive and Inductive Inference. Within the traditional treatments of scientific method, e.g., in and, it was customary to divide scientific inference into two parts: deductive and inductive. Deductive inference was taken to mean the activity of deducing theorems from postulates and definitions, whereas inductive inference represented the activity of constructing a general statement from a set of particular “facts.” Deductive inference was relegated to the mathematical sciences, and inductive inference to the empirical sciences. As a consequence, the whole of (...) science was split into two quite distinct parts: deductive science was conceived to start with precise axioms, postulates, and definitions, and its method consisted in drawing inferences from these “givens” in such a way that if the original assumptions are taken as true, then the theorems must be taken as true also. The obligation behind this “must” of the mathematical sciences was an obligation based on strict rules of deductive inference; in the earlier analyses, it was supposed that these rules were all aspects of the general theory of the syllogism, as it was described in an almost complete form in but in later work the rules have been formalized in more general terms. Inductive inference, on the other hand, was supposed to start with observed facts; these facts, or “data,” were presumably given by the senses, and described particular aspects of the external world. If the facts were given in a certain way, then the empirical scientist could “infer” a generalized description; this generalization had the characteristic that if it were granted as true, then all the facts could be deduced as consequences. The inductive process admittedly does not lead to unique generalizations, and the philosopher of science searched for other defining characteristics of good inference, such as “simplicity” and “convenience.” These empirical generalizations were taken to be “descriptions of the phenomenal world of the scientist”; a further inductive step might be taken by constructing a set of very general principles out of the more specialized generalizations within specific research problems. These “higher” inductive inferences were regarded as “theories,” which could be studied by themselves within the method of deductive inference. The scientist was then described as “checking” theory by determining whether or not the predicted consequences deduced from theory actually hold in the phenomenal world. (shrink)

This evenhanded treatment addresses the decades-old dispute among probability theorists, asserting that both statistical and inductive probabilities may be treated as sentence-theoretic measurements, and that the latter qualify as estimates of the former. Beginning with a survey of the essentials of sentence theory and of set theory, the author examines statistical probabilities, showing that statistical probabilities may be passed on to sentences, and thereby qualify as truth-values. An exploration of inductive probabilities follows, demonstrating their reinterpretation as estimates (...) of truth-values. Each chapter is preceded by a summary of its contents. Illustrations and footnotes elucidate definitions, theorems, and technicalities. 1962 ed. (shrink)

A careful presentation of the foundations of probability theory, containing many valuable innovations. Two accounts of probability are adduced: probability as a measure on the subsets of a probability set, and as a measure on the sentences of a formal language. The book stresses connections between these two accounts; of particular interest is its thesis that statistical probabilities may be regarded as estimates of inductive probabilities.—R. H. T.

This paper presents a novel approach to the unsupervised learning of syntactic analyses of natural language text. Most previous work has focused on maximizing likelihood according to generative PCFG models. In contrast, we employ a simpler probabilistic model over trees based directly on constituent identity and linear context, and use an EM-like iterative procedure to induce structure. This method produces much higher quality analyses, giving the best published results on the ATIS dataset.

In this chapter we consider unsupervised learning from two perspectives. First, we briefly look at its advantages and disadvantages as an engineering technique applied to large corpora in natural language processing. While supervised learning generally achieves greater accuracy with less data, unsupervised learning offers significant savings in the intensive labour required for annotating text. Second, we discuss the possible relevance of unsupervised learning to debates on the cognitive basis of human language acquisition. In this context we explore the implications of (...) recent work on grammar induction for poverty of stimulus arguments that purport to motivate a strong bias model of language learning, commonly formulated as a theory of Universal Grammar (UG). We examine the second issue both as a problem in computational learning theory, and with reference to empirical work on unsupervised Machine Learning (ML) of syntactic structure. We compare two models of learning theory and the place of unsupervised learning within each of them. Looking at recent work on part of speech tagging and the recognition of syntactic structure, we see how far unsupervised ML methods have come in acquiring different kinds of grammatical knowledge from raw text. (shrink)

We present a generative distributional model for the unsupervised induction of natural language syntax which explicitly models constituent yields and contexts. Parameter search with EM produces higher quality analyses than previously exhibited by unsupervised systems, giving the best published unsupervised parsing results on the ATIS corpus. Experiments on Penn treebank sentences of comparable length show an even higher F1 of 71% on nontrivial brackets. We compare distributionally induced and actual part-of-speech tags as input data, and examine extensions to the (...) basic model. We discuss errors made by the system, compare the system to previous models, and discuss upper bounds, lower bounds, and stability for this task. (shrink)

A key motivation for Universal Grammar (UG) is developmental: UG can help children acquire the linguistic knowledge that they do as quickly as they do from the data that's available to them. Some of the most fruitful recent work in language acquisition has combined ideas about different hypothesis space building blocks with domain‐general statistical learning. Statistical learning can then provide a way to help navigate the hypothesis space in order to converge on the correct hypothesis. Reinforcement learning (...) is a principled way to update the probability of a categorical option that's in competition with other categorical options. There are two prominent options for linking theory theoretical representations: the Uniformity of Theta Assignment Hypothesis (UTAH) and the relativized form of that theory, relativized UTAH. The learning strategy of Pearl and Sprouse shares the intuition with subjacency that there's a local structural anomaly when syntactic islands occur. (shrink)

While the traditional conception of inductive logic is Carnapian, I develop a Peircean alternative and use it to unify formal learning theory, statistics, and a significant part of machine learning: supervised learning. Some crucial standards for evaluating non-deductive inferences have been assumed separately in those areas, but can actually be justified by a unifying principle.

With his new riddle of induction, Goodman raised a problem for enumerative induction which many have taken to show that only some ‘natural’ properties can be used for making inductive inferences. Arguably, however, (i) enumerative induction is not a method that scientists use for making inductive inferences in the first place. Moreover, it seems at first sight that (ii) Goodman’s problem does not affect the method that scientists actually use for making such inferences—namely, classical statistics. Taken together, (...) this would indicate that (iii) ‘naturalness’ is not such a relevant concept for the problem of induction, after all. I have no objections against (i) and (iii). But, contrary to (ii), I argue that classical statistics does face a version of Goodman’s problem, which I call the statistical riddle of induction. Given that his problem merely is an instance of the more general problem of underdetermination, this is hardly surprising. Perhaps more importantly, I also show how my riddle can be used for exposing a common fallacy in probabilistic reasoning, without relying on Bayesian assumptions. (shrink)

Selectional preferences have a long history in both generative and computational linguistics. However, since the publication of Resnik's dissertation in 1993, a new approach has surfaced in the computational linguistics community. This new line of research combines knowledge represented in a pre‐defined semantic class hierarchy with statistical tools including information theory, statistical modeling, and Bayesian inference. These tools are used to learn selectional preferences from examples in a corpus. Instead of simple sets of semantic classes, selectional preferences are (...) viewed as probability distributions over various entities. We survey research that extends Resnik's initial work, discuss the strengths and weaknesses of each approach, and show how they together form a cohesive line of research. (shrink)

Statistical Learning Theory (e.g., Hastie et al., 2001; Vapnik, 1998, 2000, 2006) is the basic theory behind contemporary machine learning and data-mining. We suggest that the theory provides an excellent framework for philosophical thinking about inductive inference.

At its strongest, Hume's problem of induction denies the existence of any well justified assumptionless inductive inference rule. At the weakest, it challenges our ability to articulate and apply good inductive inference rules. This paper examines an analysis that is closer to the latter camp. It reviews one answer to this problem drawn from the VC theorem in statistical learning theory and argues for its inadequacy. In particular, I show that it cannot be computed, in general, whether we (...) are in a situation where the VC theorem can be applied for the purpose we want it to. (shrink)

According to the Bayesian view, scientific hypotheses must be appraised in terms of their posterior probabilities relative to the available experimental data. Such posterior probabilities are derived from the prior probabilities of the hypotheses by applying Bayes'theorem. One of the most important problems arising within the Bayesian approach to scientific methodology is the choice of prior probabilities. Here this problem is considered in detail w.r.t. two applications of the Bayesian approach: (1) the theory of inductive probabilities (TIP) developed by Rudolf (...) Carnap and other epistomologists and (2) the analysis of the multinational inferences provided by Bayesian statstics (BS). ... Zie: Summary. (shrink)

A significant part of everyday learning occurs incidentally — a process typically described as implicit learning. A central issue in this domain and others, such as language acquisition, is the extent to which performance depends on the acquisition and deployment of abstract rules. Shanks and colleagues [22], [11] have suggested (1) that discrimination between grammatical and ungrammatical instances of a biconditional grammar requires the acquisition and use of abstract rules, and (2) that training conditions — in particular whether instructions (...) orient participants to identify the relevant rules or not — strongly influence the extent to which such rules will be learned. In this paper, we show (1) that a Simple Recurrent Network can in fact, under some conditions, learn a biconditional grammar, (2) that training conditions indeed influence learning in simple auto-associators networks and (3) that such networks can likewise learn about biconditional grammars, albeit to a lesser extent than human participants. These findings suggest that mastering biconditional grammars does not require the acquisition of abstract rules to the extent implied by Shanks and colleagues, and that performance on such material may in fact be based, at least in part, on simple associative learning mechanisms. (shrink)

After some general remarks about the interrelation between philosophical and statistical thinking, the discussion centres largely on significance tests. These are defined as the calculation of p-values rather than as formal procedures for ‘acceptance‘ and ‘rejection‘. A number of types of null hypothesis are described and a principle for evidential interpretation set out governing the implications of p- values in the specific circumstances of each application, as contrasted with a long-run interpretation. A number of more complicated situ- ations are (...) discussed in which modification of the simple p-value may be essential. (shrink)

In this study, information about inductive and deductive methods and the differences between these methods was given. Afterwards, grammar teaching in Turkey was evaluated in terms of method. Then, the grammar teaching methods of the Arabic grammar books called al-Naḥwu'l-Wâḍıḥ and al-Qawâʻid al-ʻArabiyya al-Muyassara, which were prepared by following the inductive method, were examined under the headings of preface, handling of the subjects and exercises. In the continuation of the study a qualitative research was conducted using the (...) semi-structured interview technique with 16 randomly selected students, 8 of whom who have been studying Arabic for a long time and 8 of whom have started their Arabic education at university in order to learn the approaches of the students to the grammar teaching methods who took the course which the work called al-Qavâʻid al-ʻArabiyya al-Muyassara was used as a source book in the 2020-2021 academic year, in the first year of the Department of Arabic Language and Literature at Selcuk University in addition to a lesson which the grammar book prepared with the deductive method. While it was seen that the majority of the students in both groups participated in the interview preferred the deductive method, it was understood that the students had different approaches with their method preferences in terms of explaining the grammar issue in detail and giving more space to terms. In the study, it was concluded that it would be more beneficial to follow a mixed method in which the the positive aspects are considered, since it was seen that there were aspects that both educators and students found positive and negative in both methods. (shrink)

Chow ably defends classical significance testing by relating this method to venerable principles for inductive reasoning. Chow's success does not preclude the use of other approaches to statistical reasoning, which is fortunate not only for Bayesian rivals, but even for some fellow classicists.

In _Reliable Reasoning_, Gilbert Harman and Sanjeev Kulkarni -- a philosopher and an engineer -- argue that philosophy and cognitive science can benefit from statistical learning theory, the theory that lies behind recent advances in machine learning. The philosophical problem of induction, for example, is in part about the reliability of inductive reasoning, where the reliability of a method is measured by its statistically expected percentage of errors -- a central topic in SLT. After discussing philosophical attempts to (...) evade the problem of induction, Harman and Kulkarni provide an admirably clear account of the basic framework of SLT and its implications for inductive reasoning. They explain the Vapnik-Chervonenkis dimension of a set of hypotheses and distinguish two kinds of inductive reasoning. The authors discuss various topics in machine learning, including nearest-neighbor methods, neural networks, and support vector machines. Finally, they describe transductive reasoning and suggest possible new models of human reasoning suggested by developments in SLT. (shrink)

Although written in Japanese, what the statistical inference is philosophically investigated.

In this paper I want to discuss some basic problems of inductive logic, i.e. of the attempt to solve the problem of induction by means of a calculus of logical probability. I shall try to throw some light upon these problems by contrasting inductive logic, based on logical probability, and working with undefined samples of observations, with mathematical statistics, based on statistical probability, and working with representative random samples.