Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1530910 > unrolled thread

[PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles Manuel Schölling <manuel.schoelling@gmx.de> - 2016-11-27 18:00 +0100
    Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Andrey Utkin <andrey_utkin@fastmail.com> - 2016-11-27 22:40 +0100
      Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Adam Borowski <kilobyte@angband.pl> - 2016-11-28 00:20 +0100
        Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Jakub Wilk <jwilk@jwilk.net> - 2016-11-28 00:40 +0100
        Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Andrey Utkin <andrey_utkin@fastmail.com> - 2016-11-28 01:10 +0100
      Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Manuel Schölling <manuel.schoelling@gmx.de> - 2016-11-28 22:30 +0100
        Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Andrey Utkin <andrey_utkin@fastmail.com> - 2016-11-29 11:10 +0100
          Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Adam Borowski <kilobyte@angband.pl> - 2016-11-29 11:50 +0100
          Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Manuel Schölling <manuel.schoelling@gmx.de> - 2016-11-29 17:40 +0100
          Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Manuel Schölling <manuel.schoelling@gmx.de> - 2016-12-01 22:10 +0100
            Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Andrey Utkin <andrey_utkin@fastmail.com> - 2016-12-01 22:40 +0100
    Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Adam Borowski <kilobyte@angband.pl> - 2016-11-28 01:00 +0100
      Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for  all VGA consoles Manuel Schölling <manuel.schoelling@gmx.de> - 2016-11-28 22:30 +0100

#1530910 — [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

Changes in v7:
  - Add new callback to consw struct for flushing video console driver's
    scrollback buffer. Fixes issues with escape sequence '\e[3J' reported
    by Adam Borowski (kilobyte@angband.pl).
  - Fix style issues
Changes in v6:
  - Change of check if feature is enabled in 
    vgacon_scrollback_switch()
Changes in v5:
  - Clearify documentation
  - Skip superfluous array initialization
  - Disable scrollback if buffer allocation fails
  - Refactor vgacon_switch_scrollback()
  - Rename vgacon_switch_scrollback() to vgacon_scrollback_switch()
  - Add check for fg_console in vgacon_scrollback_update
Changes in v4.1:
  - Fix compiler error
Changes in v4:
  - Rename from VGACON_SOFT_SCROLLBACK_FOR_EACH_CONSOLE to
    VGACON_SOFT_SCROLLBACK_PERSISTENT
  - Split into two patches
  - Rework documentation
  - Remove cosmetic changes in comments (postponed)
Changes in v3:
  - Add config option for this feature
  - Fallback to old scrollback buffer if kcalloc() fails
  - Remove ioctl() call again and add documentation about existing
    escape sequence to flush the scrollback buffer
Changes in v2:
  - Add ioctl() call to flush scrollback buffer
  - (Patch v2 was not labeled as such, sorry)

Manuel Schölling (3):
  console: Move scrollback data into its own struct
  console: Add callback to flush scrollback buffer to consw struct
  console: Add persistent scrollback buffers for all VGA consoles

 drivers/tty/vt/vt.c            |   9 +++
 drivers/video/console/Kconfig  |  25 ++++++-
 drivers/video/console/vgacon.c | 165 ++++++++++++++++++++++++++++-------------
 include/linux/console.h        |   4 +
 4 files changed, 148 insertions(+), 55 deletions(-)

-- 
2.1.4

[toc] | [next] | [standalone]

#1530939 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

Hi Manuel,

I've just patched next-20161125 with this set and given it a run.

Scrollback persistence works fine, just as in earlier versions.

This time I didn't forget to test clear operation.

The only important concern is that after logout, the scrollback is not
wiped by /bin/login or /sbin/agetty (not sure who of them is responsible
for that). What do you see on your workstations in this case?

I guess we need to do something of the following:
 - catch some control character sequences to wipe the scrollback
 - indicate (by some flag) some feature capability for this
 - request update in terminfo database or whatever, to let ncurses know
   that it is capable of scrollback wiping by some control charater
   sequences


Some useless notes follow.

I see the user experience is subpar to what I'm accustomed to (I use
Konsole and "Clear Scrollback and Reset" action, default shortcut is
Ctrl+Shift+K). The strange behaviour moments have nothing to do with
current patchset but are properties of vgacon, though. (I compared it
with another PC which runs without this patchset, and it looks like it
runs vgacon, too, however, I'm not sure how to ensure this at runtime.)

clear(1) doesn't wipe the scrollback at all, it is still reachable, all
of it.

echo -e "\e[3J" seems to wipe the scrollback, but if you do it several
times in a row, every time you (or at last I do) get your prompt a bit
lower, so after many times you end up with blank screen and the prompt
at the bottom of the screen.

Have you encountered this, or is it something specific to my setup (I
use bash prompt spanning to multiple lines, and calling "stty sane" from
inside every PS1 evaluation. I can share the config if you request it).

[toc] | [prev] | [next] | [standalone]

#1530960 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

On Sun, Nov 27, 2016 at 09:37:30PM +0000, Andrey Utkin wrote:
> I've just patched next-20161125 with this set and given it a run.
> 
> Scrollback persistence works fine, just as in earlier versions.
> 
> This time I didn't forget to test clear operation.
> 
> The only important concern is that after logout, the scrollback is not
> wiped by /bin/login or /sbin/agetty (not sure who of them is responsible
> for that). What do you see on your workstations in this case?

If you're on Debian or a derivative, that's clear_console.  It uses a
switch-vt-then-back hack which obviously doesn't work with scrollback
persistence.  Reported as https://bugs.debian.org/845177 -- I'll molest the
maintainer if the patch doesn't get applied soon, so we can have the fix in
time for stretch (then Ubuntu zesty).

Because of a sad lack of a time machine, old systems will use clear_console
with that hack until they die, that's why
CONFIG_VGACON_SOFT_SCROLLBACK_PERSISTENT defaults to n; in a few years it'll
be ok to flip it.

> I guess we need to do something of the following:
>  - catch some control character sequences to wipe the scrollback

\e[3J

>  - indicate (by some flag) some feature capability for this

Terminfo calls this flag "E3".

>  - request update in terminfo database or whatever, to let ncurses know
>    that it is capable of scrollback wiping by some control charater
>    sequences

Already there for quite a while.

> clear(1) doesn't wipe the scrollback at all, it is still reachable, all
> of it.

It does for me on the console.  The man page says:

# clear clears your screen if this is possible, including its scrollback
# buffer (if the extended "E3" capability is defined).  clear looks in the
# environment for the terminal type and then in the terminfo database to
# determine how to clear the screen.

Because of its reliance on terminfo, you need to have TERM=linux in your
environment; also, screen/tmux obviously breaks this.

> echo -e "\e[3J" seems to wipe the scrollback, but if you do it several
> times in a row, every time you (or at last I do) get your prompt a bit
> lower, so after many times you end up with blank screen and the prompt
> at the bottom of the screen.

Yeah, none of \e[J subcommands move the cursor at all.  As you use echo
without -n, you move two lines lower, and even with -n the command you typed
takes a line.  You want to move the cursor explicitly, add "\e[H".


Meow!
-- 
The bill declaring Jesus as the King of Poland fails to specify whether
the addition is at the top or end of the list of kings.  What should the
historians do?

[toc] | [prev] | [next] | [standalone]

#1530962 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

* Adam Borowski <kilobyte@angband.pl>, 2016-11-28, 00:15:
>>clear(1) doesn't wipe the scrollback at all, it is still reachable, all of 
>>it.
>
>It does for me on the console.  The man page says:
>
># clear clears your screen if this is possible, including its scrollback
># buffer (if the extended "E3" capability is defined).  clear looks in the
># environment for the terminal type and then in the terminfo database to
># determine how to clear the screen.
>
>Because of its reliance on terminfo, you need to have TERM=linux in your 
>environment; also, screen/tmux obviously breaks this.

The "linux" terminfo entry didn't have E3 until very recently.
You will need ncurses >= 20160514.

-- 
Jakub Wilk

[toc] | [prev] | [next] | [standalone]

#1530968 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

On Mon, Nov 28, 2016 at 12:15:48AM +0100, Adam Borowski wrote:
> On Sun, Nov 27, 2016 at 09:37:30PM +0000, Andrey Utkin wrote:
> > I've just patched next-20161125 with this set and given it a run.
> > 
> > Scrollback persistence works fine, just as in earlier versions.
> > 
> > This time I didn't forget to test clear operation.
> > 
> > The only important concern is that after logout, the scrollback is not
> > wiped by /bin/login or /sbin/agetty (not sure who of them is responsible
> > for that). What do you see on your workstations in this case?
> 
> If you're on Debian or a derivative, that's clear_console.  It uses a
> switch-vt-then-back hack which obviously doesn't work with scrollback
> persistence.  Reported as https://bugs.debian.org/845177 -- I'll molest the
> maintainer if the patch doesn't get applied soon, so we can have the fix in
> time for stretch (then Ubuntu zesty).

I'm on Gentoo.

> Because of a sad lack of a time machine, old systems will use clear_console
> with that hack until they die, that's why
> CONFIG_VGACON_SOFT_SCROLLBACK_PERSISTENT defaults to n; in a few years it'll
> be ok to flip it.
> 
> > I guess we need to do something of the following:
> >  - catch some control character sequences to wipe the scrollback
> 
> \e[3J
> 
> >  - indicate (by some flag) some feature capability for this
> 
> Terminfo calls this flag "E3".
> 
> >  - request update in terminfo database or whatever, to let ncurses know
> >    that it is capable of scrollback wiping by some control charater
> >    sequences
> 
> Already there for quite a while.
> 
> > clear(1) doesn't wipe the scrollback at all, it is still reachable, all
> > of it.
> 
> It does for me on the console.  The man page says:
> 
> # clear clears your screen if this is possible, including its scrollback
> # buffer (if the extended "E3" capability is defined).  clear looks in the
> # environment for the terminal type and then in the terminfo database to
> # determine how to clear the screen.
> 
> Because of its reliance on terminfo, you need to have TERM=linux in your
> environment; also, screen/tmux obviously breaks this.

I wonder whether my ncurses is not bleeding-edge enough, or I have some
non-standard config. Anyway, thanks for explanation.

> > echo -e "\e[3J" seems to wipe the scrollback, but if you do it several
> > times in a row, every time you (or at last I do) get your prompt a bit
> > lower, so after many times you end up with blank screen and the prompt
> > at the bottom of the screen.
> 
> Yeah, none of \e[J subcommands move the cursor at all.  As you use echo
> without -n, you move two lines lower, and even with -n the command you typed
> takes a line.  You want to move the cursor explicitly, add "\e[H".

Thanks for explanation.

[toc] | [prev] | [next] | [standalone]

#1531679 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

Hi Andrey,

Adam already discussed some of your notes, but I want to catch up one
this one:

On So, 2016-11-27 at 21:37 +0000, Andrey Utkin wrote:
> I see the user experience is subpar to what I'm accustomed to (I use
> Konsole and "Clear Scrollback and Reset" action, default shortcut is
> Ctrl+Shift+K). The strange behaviour moments have nothing to do with
> current patchset but are properties of vgacon, though. (I compared it
> with another PC which runs without this patchset, and it looks like it
> runs vgacon, too, however, I'm not sure how to ensure this at runtime.)
I'm not sure what you mean with 'subpar'. Ctrl+Shift+K would probably be
nice - but it might interfere with some shortcuts of programs.
Are you missing any other features?
(I am working on persistent scrollback for framebuffer consoles in the
mean time...)

Bye,

Manuel

[toc] | [prev] | [next] | [standalone]

#1532086 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

On Mon, Nov 28, 2016 at 10:28:19PM +0100, Manuel Schölling wrote:
> Hi Andrey,
> 
> Adam already discussed some of your notes, but I want to catch up one
> this one:
> 
> On So, 2016-11-27 at 21:37 +0000, Andrey Utkin wrote:
> > I see the user experience is subpar to what I'm accustomed to (I use
> > Konsole and "Clear Scrollback and Reset" action, default shortcut is
> > Ctrl+Shift+K). The strange behaviour moments have nothing to do with
> > current patchset but are properties of vgacon, though. (I compared it
> > with another PC which runs without this patchset, and it looks like it
> > runs vgacon, too, however, I'm not sure how to ensure this at runtime.)
> I'm not sure what you mean with 'subpar'. Ctrl+Shift+K would probably be
> nice - but it might interfere with some shortcuts of programs.
> Are you missing any other features?
> (I am working on persistent scrollback for framebuffer consoles in the
> mean time...)

I meant that in my terminal of choice, I have single action which does
both of these things at same time:
 - scrollback cleaning (also not leaving screen worth of emptiness in
   scrollback)
 - prompt positioning to top line

That's all. I don't appeal to bring Ctrl+Shift+K hotkey or whatever
else.

Since these cosmetic matters are inherent to vgacon I'm fine with it.

Regarding logout scrollback clearing not working for me. ncurses-6.0-rc1
which I tested it with is the latest available in Gentoo portage, please
confirm whether I need any newer version, or should I tune something
else. I'd appreciate if you also tested your patch with gentoo setup.

[toc] | [prev] | [next] | [standalone]

#1532116 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

On Tue, Nov 29, 2016 at 10:01:15AM +0000, Andrey Utkin wrote:
> Regarding logout scrollback clearing not working for me. ncurses-6.0-rc1
> which I tested it with is the latest available in Gentoo portage, please
> confirm whether I need any newer version, or should I tune something
> else. I'd appreciate if you also tested your patch with gentoo setup.

Could you please check whether its terminfo carries the required
definitions?  In ncurses sources, that's misc/terminfo.src

The relevant parts are:
.--====
# The 3.0 kernel adds support for clearing scrollback buffer (capability E3).
# It is the same as xterm's erase-saved-lines feature.
linux3.0|linux 3.0 kernels,
        E3=\E[3J, use=linux2.6,

# This is Linux console for ncurses.
linux|linux console,
        use=linux3.0,
`----

I believe the first part was added first; if that's true it's possible this
will work for you:
    TERM=linux3.0 clear


I'm not sure what Gentoo does to clear the console during logout: it might
just invoke "clear" (or its underlying ncurses implementation), it might
carry a copy of Debian's "clear_console"[1], it might do something else
entirely.


Meow!

[1]. It originally came from Ubuntu, forked there from "clear".
-- 
The bill declaring Jesus as the King of Poland fails to specify whether
the addition is at the top or end of the list of kings.  What should the
historians do?

[toc] | [prev] | [next] | [standalone]

#1532484 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

Hi Andrey,

On Di, 2016-11-29 at 10:01 +0000, Andrey Utkin wrote:
> Regarding logout scrollback clearing not working for me. ncurses-6.0-rc1
> which I tested it with is the latest available in Gentoo portage, please
> confirm whether I need any newer version, or should I tune something
> else. I'd appreciate if you also tested your patch with gentoo setup.
Are you sure ncurses is involved at all?
My Debian agetty(8) manpage says:

       -J,--noclear
              Do not clear the screen before prompting for the login name (the
              screen is normally cleared).

And digging into the source code of agetty shows these lines [1]:

static void termio_clear(int fd)
{
	/*
	 * Do not write a full reset (ESC c) because this destroys
	 * the unicode mode again if the terminal was in unicode
	 * mode.  Also it clears the CONSOLE_MAGIC features which
	 * are required for some languages/console-fonts.
	 * Just put the cursor to the home position (ESC [ H),
	 * erase everything below the cursor (ESC [ J), and set the
	 * scrolling region to the full window (ESC [ r)
	 */
	write_all(fd, "\033[r\033[H\033[J", 9);
}

So I guess that agetty relies on on switching the console for flushing
the scrollback buffer and we'd had to add the \E[3J sequence here.

Note that up until now I just had a look at the theory (manpage and
source code). I'd need some days to find time to show at runtime that
this really is the reason why the buffer is not flushed.

Bye,

Manuel

[1] https://github.com/karelzak/util-linux/blob/master/term-utils/agetty.c#L1175

[toc] | [prev] | [next] | [standalone]

#1534439 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

Hi Andrey,

On Di, 2016-11-29 at 10:01 +0000, Andrey Utkin wrote:
> On Mon, Nov 28, 2016 at 10:28:19PM +0100, Manuel Schölling wrote:
> Regarding logout scrollback clearing not working for me. ncurses-6.0-rc1
> which I tested it with is the latest available in Gentoo portage, please
> confirm whether I need any newer version, or should I tune something
> else. I'd appreciate if you also tested your patch with gentoo setup.

I finally setup gentoo running agetty (util-linux-2.26.2) and patching
the file term-utils/agetty.c with

static void termio_clear(int fd)
{
	/*
	 * Do not write a full reset (ESC c) because this destroys
	 * the unicode mode again if the terminal was in unicode
	 * mode.  Also it clears the CONSOLE_MAGIC features which
	 * are required for some languages/console-fonts.
	 * Just put the cursor to the home position (ESC [ H),
	 * erase everything below the cursor (ESC [ J), and set the
	 * scrolling region to the full window (ESC [ r)
	 */
-	write_all(fd, "\033[r\033[H\033[J", 9);
+	write_all(fd, "\033[3J\033[r\033[H\033[J", 13);
}

solves the issue with the scrollback buffer after log out.
Let me know if you agree that this is the right way to go and I will
send a patch to the maintainer of util-linux.

Thanks again for spending all this time to test the patch!

Have a good weekend!

Manuel

[toc] | [prev] | [next] | [standalone]

#1534469 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

On Thu, Dec 01, 2016 at 10:03:23PM +0100, Manuel Schölling wrote:
> Hi Andrey,
> 
> On Di, 2016-11-29 at 10:01 +0000, Andrey Utkin wrote:
> > On Mon, Nov 28, 2016 at 10:28:19PM +0100, Manuel Schölling wrote:
> > Regarding logout scrollback clearing not working for me. ncurses-6.0-rc1
> > which I tested it with is the latest available in Gentoo portage, please
> > confirm whether I need any newer version, or should I tune something
> > else. I'd appreciate if you also tested your patch with gentoo setup.
> 
> I finally setup gentoo

Wow, what a big undertaking.

> running agetty (util-linux-2.26.2) and patching
> the file term-utils/agetty.c with
> 
> static void termio_clear(int fd)
> {
> 	/*
> 	 * Do not write a full reset (ESC c) because this destroys
> 	 * the unicode mode again if the terminal was in unicode
> 	 * mode.  Also it clears the CONSOLE_MAGIC features which
> 	 * are required for some languages/console-fonts.
> 	 * Just put the cursor to the home position (ESC [ H),
> 	 * erase everything below the cursor (ESC [ J), and set the
> 	 * scrolling region to the full window (ESC [ r)
> 	 */
> -	write_all(fd, "\033[r\033[H\033[J", 9);
> +	write_all(fd, "\033[3J\033[r\033[H\033[J", 13);
> }
> 
> solves the issue with the scrollback buffer after log out.
> Let me know if you agree that this is the right way to go and I will
> send a patch to the maintainer of util-linux.

I believe you that this works and you must know better than me whether
this solution is correct. Besides that, I'd suggest updating that large
block comment before the updated write_all() call to describe the new
action you're doing. Please CC me in your discussion with util-linux
maintainers.

Also I'd suggest coming back in a while, to set this feature enabled by
default. I wonder how many years to wait gracefully until "stable"
distros update util-linux to ensure secure scrollback wiping. 3? 5?

> Thanks again for spending all this time to test the patch!

Thank you very much for your time and effort in bringing this useful
feature!

I give all sorts of my approval on this patchset:
Reviewed-by: Andrey Utkin <andrey_utkin@fastmail.com>
Tested-by: Andrey Utkin <andrey_utkin@fastmail.com>

[toc] | [prev] | [next] | [standalone]

#1530965 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

\e[3J works well now, thanks!

I haven't found any more problems; your changes also appear to make no
regressions in at least nouveau fb (which obviously doesn't have this goodie
yet).

Patch 2 doesn't apply cleanly on current Linus' tree but it's just a matter
of more fuzz than "git am" allows.

Tested-by: Adam Borowski <kilobyte@angband.pl>


Meow!
-- 
The bill declaring Jesus as the King of Poland fails to specify whether
the addition is at the top or end of the list of kings.  What should the
historians do?

[toc] | [prev] | [next] | [standalone]

#1531680 — Re: [PATCH v7 0/3] console: Add persistent scrollback buffers for all VGA consoles

On Mo, 2016-11-28 at 00:53 +0100, Adam Borowski wrote:
> \e[3J works well now, thanks!
Great to hear that!

> Tested-by: Adam Borowski <kilobyte@angband.pl>
Thanks, Adam, for spending all this time testing the patches!

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web