The history limit is a pane attribute that is fixed at the time of pane creation and cannot be changed for existing panes. The value is taken from the history-limit session option (the default value is 2000).
To create a pane with a different value you will need to set the appropriate history-limit option before creating the pane.
To establish a different default, you can put a line like the following in your .tmux.conf file:
set-option -g history-limit 3000
Note: Be careful setting a very large default value, it can easily consume lots of RAM if you create many panes.
For a new pane (or the initial pane in a new window) in an existing session, you can set that session’s history-limit. You might use a command like this (from a shell): tmux set-option history-limit 5000 \; new-window
For (the initial pane of the initial window in) a new session you will need to set the “global” history-limit before creating the session: tmux set-option -g history-limit 5000 \; new-session
For those looking for a simple answer, just use prefix + :, then type in capture-pane -S -3000 + return (Replace 3000 with however many lines you'd like to save.) This copies those lines into a buffer.
Then, to save the buffer to a file, just use prefix + : again, and type in save-buffer filename.txt + return, replacing filename with whatever you'd like.
With tmux 1.5, the capture-pane command accepts -S and -E to specify the start and end lines of the capture; negative values can be used to specify lines from the history. Once you have the data in a buffer, you can save it with save-buffer.
Here is an example binding (suitable for .tmux.conf) that wraps it all up with a prompt for the filename:
bind-key P command-prompt -p 'save history to filename:' -I '~/tmux.history' 'capture-pane -S -32768 ; save-buffer %1 ; delete-buffer'
This captures (up to) 32768 lines of history plus the currently displayed lines. Starting with tmux 1.6, you can use numbers down to INT_MIN if your pane has a history that is deeper than 32Ki lines (usually up to 2Gi lines). Starting in tmux 2.0, you can use capture-pane -S - to mean “start at the beginning of history” (i.e. no large, hard-coded negative number).