Marco Z ocramz

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

You'll generally want to look at:

heap
stack
gc profiles

Do not get bogged down in microoptimizations before you've assessed any macro optimizations that are available. IO and the choice of algorithm dominate any low level changes you may make. In the end you have to think hard about your code!

Topos> For example, if i see that a particular pure function is taking a  long time relative to the rest of the code, and that it's Text, and I'm seeing ARR_WORDS rise linearly in the heap, I probably have a thunk-based memory leak. This is knowledge you build up over time.

Profiling in Haskell

Do not get bogged down in microoptimizations before you've assessed any macro optimizations that are available. IO and the choice of algorithm dominate any low level changes you may make. In the end you have to think hard about your code!

Before starting to optimize:

Is the -O2 flag on ?
Profile: which part of the code is the slow one.
Use the best algorithm in that part.
Optimize: implement it in the most efficient way.

Optics By Example Cheat Sheets

The following are appendices from Optics By Example, a comprehensive guide to optics from beginner to advanced! If you like the content below, there's plenty more where that came from; pick up the book!

Operators
Composition Guide
Substitution Guide
Optical Constarints and Types

Operator Cheat Sheet

Exploring possibilities with simple indexed fields

Database records and formlets and optionally populated records can be neatly all represented with the same data type when the fields are all indexed.

class Indexed i a where
  type Index i (a :: *)

	module Example (getWaiApp) where

	import BasePrelude -- Eh, I had BasePrelude and MTLPrelude at hand.
	import MTLPrelude -- Ignore them :)

	import Network.Wai qualified as Wai
	import Wai.CryptoCookie qualified as WC
	import Wai.CryptoCookie.Encryption qualified as WC
	import Web.Scotty.Trans qualified as S

	{-# LANGUAGE DeriveFunctor, DeriveFoldable, DeriveTraversable, TypeFamilies, TemplateHaskell #-}

	import Data.Functor.Foldable
	import Data.Functor.Foldable.TH
	import Data.Functor.Compose

	-- Example type
	data Exp
	= IntValue Int
	\| Sum Exp Exp

	# Haskell stack project Github Actions template
	# https://gist.github.com/mstksg/11f753d891cee5980326a8ea8c865233
	#
	# To use, mainly change the list in 'plans' and modify 'include' for
	# any OS package manager deps.
	#
	# Currently not working for cabal-install >= 3
	#
	# Based on https://raw.githubusercontent.com/commercialhaskell/stack/stable/doc/travis-complex.yml
	#

	{-# LANGUAGE AllowAmbiguousTypes #-}
	{-# LANGUAGE DataKinds #-}
	{-# LANGUAGE DeriveGeneric #-}
	{-# LANGUAGE FlexibleContexts #-}
	{-# LANGUAGE FlexibleInstances #-}
	{-# LANGUAGE FunctionalDependencies #-}
	{-# LANGUAGE PolyKinds #-}
	{-# LANGUAGE ScopedTypeVariables #-}
	{-# LANGUAGE TypeApplications #-}
	{-# LANGUAGE TypeFamilies #-}

	-- in response to https://www.reddit.com/r/haskell/comments/a50xpr/datahaskell_solve_this_small_problem_to_fill_some/
	{-# LANGUAGE BangPatterns, OverloadedStrings, RecordWildCards, ScopedTypeVariables #-}
	module Main where

	import Control.Category ((>>>))
	import Data.Function ((&))
	import Data.Map.Strict (Map, (!))
	import Data.Set (Set)
	import Test.DocTest (doctest)