Connectionist and Statistical Language Processing

Leitung: Prof. Dr. Matthew Crocker & Dr. Frank Keller

Format: Vorlesung mit Übung, in Englisch, zweiter Studienabschnitt

Leistungpunkte: 4 Punkte

Beginn: Montag, 27 Oktober 2003

Vorlesung: Mo 11-13 Uhr, Konferenzraum (Keller, 17.1)

Übung: Mi 11-13 Uhr, CIP-Pool, Coli

Inhalt:

This course aims at introducing current connectionist models and statistical/machine learning techniques for both language learning and language processing. The goal of the course is to give students both an understanding of the general issues behind the innateness versus emergence debate and also a solid grasp of how empirically-oriented techniques can be applied to language. The course will motivate the use connectionist and statistical/machine learning techniques for both cognitive modelling and practical NLP applications, emphasising their general properties (advantages and disadvantages). Topics will include:

Part I: Connectionist Approaches (Crocker)

30 October - 11 December

Part II: Statistical and Machine Learning Approaches (Keller)

17 December - 12 February

Tutorials will be practical sessions which involve implementations of connectionist and statistical/machine learning models. Based on these implementations, students will complete practical assignments (this will usually be possible during the tutorial sessions).

Voraussetzungen/Bemerkungen:

No programming is required, and all exercises will be done using the Tlearn connectionist simulator or the Weka machine learning environment. No background in connectionist or statistical/machine learning methods is required, but students should have reasonable mathematical background (MG III: Statistische Methoden, ideal).

Prüfungsleistungen:

The exam will take place on 12.02.2002, beginning at 14 Uhr s.t. In order to be eligible to sit the exam, students must satisfactorily complete all exercises (these will typically be completed during the practical session, and evaluated on a pass/fail basis), and can miss no more than 1 tutorial session.

The exam consists of 2 parts, worth 50 points each. Part 1 will consist of a set of obligatory short answer questions. Part 2 will contain a set of 4-5 discussion questions, from which you must select 2 to answer (worth equal points, e.g 25 points each). You may answer these questions in English or German.

Stellung im Studienplan:

Zählt 4 Leistungspunkte, Kategorie 1: Syntax und Morphologie.

Kursmaterialien:

The files for use in tutorials can be found here.

Week 1Vorbesprechung
Lecture 1: Introduction to Connectionist Networks
Tutorial 1: Activation Propagation and Specifying Networks
Week 2Lecture 2: Learning and Training
Week 3Lecture 3: Multi-layer Networks
Tutorial 2: Training Two-layer Networks
Week 4Lecture 4: Pattern Associators and Competetive Networks
Tutorial 3: Networks with Hidden Layers
Week 5Lecture 5: Learning Phonology and Morphology
Tutorial 4: Learning the English Past Tense
Week 6Lecture 6: Simple Recurrent Networks
Tutorial 5: Autoassociation and Cluster Analysis
Week 7Lecture 7: Learning Linguistic Structure in Simple Recurrent Networks
Tutorial 6: Simple Recurrent Networks
Week 8Lecture 8: Introduction to Machine Learning
Tutorial 7: Introduction to Machine Learning
Week 9Lecture 9: Decision Trees
Tutorial 8: Decision Trees
Week 10Lecture 10: Naive Bayes Classifiers
Tutorial 9: Naive Bayes Classifiers
Lecture 11: Evaluation
Week 11Lecture 12: Linear Models
Tutorial 10: Linear Models
Week 12Lecture 13: Clustering
Tutorial 11: Clustering
Week 13Lecture 14: Summary Part I; Lecture 14: Summary Part II
Week 14Klausur

Software:

The Tlearn connectionist simulator can be found here.

The Weka machine learning enviornment can be found here.

Literatur:

N. Chater and M. Christiansen (1999). Connectionism and natural language processing. Chapter 8 of Garrod and Pickering (eds.): Language Processing. Psychology Press.

J. Elman et al. (1996). Chapter 2: Why Connectionism? In: Rethinking Innateness. MIT Press.

C. D. Manning and H. Schütze (1999). Foundations of Statistical Natural Language Processing. MIT Press.

P. McLeod, K. Plunkett and E. T. Rolls (1998). Introduction to Connectionist Modelling of Cognitive Processes. Oxford University Press. Chapters: 1-5, 7, 9.

T. M. Mitchell (1997). Machine Learning. McGraw-Hill.

K. Plunkett and J. Elman (1997). Exercises in rethinking innateness: A Handbook for Connectionist Simulations. MIT Press. Chapters: 1-8, 11, 12.

I. H. Witten and E. Frank (1999). Data Mining: Practical Machine Learning Tools. Morgan Kaufmann.

Weitere Literatur:

M. Christiansen and N. Chater (1999). Connectionist Natural Language Processing: The State of the Art. Cognitive Science, 23(4): 417-437.

J. Elman (1990). Finding Structure in Time. Cognitive Science, 14: 179-211.

J. Elman (1991). Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7: 195-225.

J. Elman (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48: 71-99.

M. Seidenberg (1997). Language Acquisition and Use: Learning and Applying Probabilistic Constraints. Science, 275: 1599-1603.

M. Seidenberg and M. MacDonald (1999). A Probabilistic Constraints Approach to Language Acquisition and Processing. Cognitive Science, 23(4): 569-588.

M. Steedman (1999). Connectionist Sentence Processing in Perspective. Cognitive Science, 23(4): 615-634.