pasting long block from clipboard inserts wrong content for line breaks

[newbie-02]

hi @ all, I'm new here, sorry for all I might do wrong,

short description: pasting a long block from a web-browser inserts one special block of 'foreign content' after / instead of multiple line breaks, making result unusable.

I'm aware it's a weird problem, did all I can do, persistent, thus ask for help, if not solvable already info if 'clean or bad on other systems' or hints for further track down can be helpful.

After I first noticed I tried lots of things, none helped, just now installed mc from scratch, issue persists.

System: actual Kali linux ( Debian ) on Intel ( Xeon ) hardware which works quite stable.

Reproducibility: copy some text from a webpage, as reference I marked the main text of ​https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ from 'Comparing Floating Point Numbers, 2012 Edition' to 'with all of its flaws' in actual version of Brave and copied by Ctrl-C.

If I paste into a text application, e.g. 'Mousepad', by Ctrl-V I get a meaningful display and can save the text.

Opening an empty text file with mc ( F4 ) and pasting there with Shift-Ctrl-V the pasting takes some time, with cursor flipping around on the screen, and then I get the start of the page as wanted, but starting at some point a special block of nonsense is inserted instead of most ( all? ) linebreaks. The point where the nonsense starts varies from try to try.

I'll try to attach the head of a copy from pasting into Mousepad and alternately mc.

The problem is persistent, the block is always the same, and behavior didn't change with fresh installation of mc ( ver. 4.8.31 from today - git clone ​https://github.com/MidnightCommander/mc.git - ./autogen.sh - sudo apt install build-dep mc ( S-Lang was missing ) - ./configure - make -j12 - sudo make install -j12 ).

As the unwanted block is part of another text on my system I think me or my system bad, as it's always the same I think must be configured / referenced somewhere, but couldn't find that block in all my files ( searched all files inc. configuration and hidden ) except in the origin and where I saved pasting tries with mc.

Even tried to work on another disk ( home instead of data ), same problem, same block.

Have been involved in two other pasting bugs, LO Calc sometimes not pasting content from other apps, was a windows 'occupied, wait and retry' problem, and GNOME Gnumeric not pasting webbrowser content, was a 'browser exports bad html' problem, IMHO none of them fits here.

Trying a workaround which there often helped, paste into pure text application, copy from there and then paste into mc doesn't work here, produces same / similar wrong content.

I'm clueless, can anybody give a hint?

To make it a good bug report, info as requested:

'search before create new ticket' - I searched 'paste' and 'wrong', 2 pages of reports, none matching my experience,

LC_MESSAGES=C mc -V:

GNU Midnight Commander 4.8.31
Built with GLib 2.78.4
Built with S-Lang 2.3.3 with terminfo database
Built with libssh2 1.11.0
With builtin Editor and Aspell support
With subshell support as default
With support for background operations
With mouse support on xterm and Linux console
With support for X11 events
With internationalization support
With multiple codepages support
With ext2fs attributes support
Virtual File Systems:

cpiofs, tarfs, sfs, extfs, ext2undelfs, ftpfs, sftpfs, shell

Data types:

char: 8; int: 32; long: 64; void *: 64; size_t: 64; off_t: 64;

LC_MESSAGES=C mc -F:

Home directory: /home/kali
Profile root directory: /home/kali

[System data]

Config directory: /etc/mc/
Data directory: /usr/share/mc/
File extension handlers: /usr/lib/mc/ext.d/
VFS plugins and scripts: /usr/lib/mc/

extfs.d: /usr/lib/mc/extfs.d/
shell: /usr/lib/mc/shell/

[User data]

Config directory: /home/kali/.config/mc/
Data directory: /home/kali/.local/share/mc/

skins: /home/kali/.local/share/mc/skins/
extfs.d: /home/kali/.local/share/mc/extfs.d/
shell: /home/kali/.local/share/mc/shell/
mcedit macros: /home/kali/.local/share/mc/mc.macros
mcedit external macros: /home/kali/.local/share/mc/mcedit/macros.d/macro.*

Cache directory: /home/kali/.cache/mc/

mc --configure-options:

'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' 'AWK=awk' 'X11_WWW=x-www-browser' '--libexecdir=/usr/lib' '--with-x' '--with-screen=slang' '--disable-rpath' '--disable-static' '--disable-silent-rules' '--enable-aspell' '--enable-vfs-sftp' '--enable-vfs-undelfs' '--enable-tests' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -Werror=implicit-function-declaration -ffile-prefix-map=/build/reproducible-path/mc-4.8.31=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now -Wl,--as-needed' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'

search: did, see above,

distribution first: did fresh compile from sources,

steps: see head of this report,

expected: file like attached 'mc_paste_should.txt', note that that's only the first few paragraphs!

see instead: file like attached 'mc_paste_happening.txt', note that that's only the first few paragraphs! The issue strikes in long pasting only!

[newbie-02]

pasting as it should be:

[newbie-02]

paste as it happens with inserted nonsense:

[zaytsev]

I can't reproduce your problem on macOS with Apple Terminal.

In as far as I'm concerned, it looks like it's the terminal emulation application you are using that is cutting the text that is being given to mc, whereas mc itself is able to deal with any large blocks of text that are given to it.

Try a different terminal application (xterm, urxvt, ...).

[newbie-02]

[ solved ]

got it tracked down, I had a ~27kB

mc.macros

file in

~/.local/share/mc

with content like:

[editor]
kpenter=Enter:-1;InsertChar:40;InsertChar:120;InsertChar:110;InsertChar:41;InsertChar:32;InsertChar:226;InsertChar:136;InsertChar:146;Enter:-1;InsertChar:102;InsertChar:40;InsertChar:40;InsertChar:120;InsertChar:41; ... 

after removing it mc / mcedit works as expected.

No clue how this file creeped in, sorry for wasting your time, hope this topic can help others if they stumble in similar oddities. The structured work trough for this topic and a cross check on another machine helped me to find the source of problem.

Pls. close topic if / after someone added a comment about correct use of mc.macros .

[newbie-02]

Replying to zaytsev:

hello @ zaytsev,

thanks for caring, looks like we had 'mid-air collision' in commenting, pls. change status if necessary ( I'm not used to it ).

As you see in my comment comment:2 it's somehow a 'mc internal issue', haven't yet investigated about 'mc.macros', would appreciate if someone with knowledge about could add info here.

best regards,

newbie-02

[zaytsev]

You probably inadvertently triggered the "record macro" feature and then pasted something that got recorded as a macro.

[newbie-02]

hmmmmmmmmm ...

yes and no, I somtimes use Ctrl-r to re-read directory contents, when in editor that triggers macro recording, can imagine having made a mistake there.

But don't understand how that steps into pasting. IMHO macros shall be executed by Ctrl-a and a letter assigned to them. Is there some intention to run them in the process of pasting? And if so why doesn't it hit with short texts, why does it hit / start at different positions?

_Assume_ there is something weak in code, can't say what and why, but consider it inconvenient for users, 'not what they want' ... will try to reopen.

[zaytsev]

Please don't reopen unless you have a reproducer. As I said, my working assumption is that you accidentally pressed CTRL+R in the editor at some point and the pasted your stuff which got recorded as a huge macro, so there is nothing to fix here.

[newbie-02]

hello @ zaytsev,

reproducer: tried to boil it down.
Naturally I can't say how the macro creeped in or which start character it's assigned to.
I tried the following:

1. starting mc without a

   mc.macro
   

   file in

   ~/.local/share/mc
   

2. copying the website from 'Comparing Floating Point Numbers, 2012 Edition' up to 'What do you mean 'correct, or 2a. use the text I'll try to attach as 'mc_paste_test01.txt',
3. make a new document,
4. open it with F4,
5. paste by Shift-Ctrl-V,

produces a copy of the file as expected.

6. place the attached shortened macro file

   mc.macros
   

   in

   ~/.local/share/mc
   

7. repeat steps 2. to 5. from above,

produces a file like the attached 'mc_paste_test02.txt', which has some injected parts

(xn) −
f((x))
f

think that isn't wanted?

[newbie-02]

Testfile which changes in pasting,

[newbie-02]

mc.macros file which triggers the change,

[newbie-02]

File changed in pasting

[zaytsev]

Naturally I can't say how the macro creeped in or which start character it's assigned to.

This is the only thing that matters though.

think that isn't wanted?

With the macro that's to be expected.

[ossi]

With the macro that's to be expected.

no, it's not. notably, the editor already contains code that suppresses auto-indentation on pasted text. it obviously should suppress macro expansion as well.

[newbie-02]

Maybe I'm thinking too simple there, which is always an option, but if I want to paste something, I usually want to paste >that<, not changes.
If there is an option for combined pasting and 'prettifying' or similar, I would expect it to be documented and that I would be asked before the action.

@ zaytsev:

Naturally I can't say how the macro creeped in or which start character it's assigned to.

This is the only thing that matters though.

don't think so, consider it an accident in typing, but should somehow be communicated and / or accessible to the user that he now has a macro and in which situations it will strike.

think that isn't wanted?

With the macro that's to be expected.

can I find somewhere which type / content of macro would take which action in which situation? All I found yet was how to record, assign a key and run. Nothing about any automated action and how to design for that.

@ ossi:
agree that pasting shouldn't be changing, if it's implemented / good / wanted for special purposes it should be documented and optional.

[zaytsev]

Well, detecting pasting is not possible reliably in a general way. We might try to use paste mode, if the terminal supports it, but I'm not sure whether we do have any code in the editor right now for that. If you're saying there is something there to suppress autoindent, then it can be extended to support macros, if it doesn't do this already. If you want to make a patch, we might try to take it in.

[newbie-02]

Am I right in understanding that there is something like 'automatic macro expansion' which in mc's internal editor automatically executes a replacement of the character assigned as macro start character by the steps defined for the macro?
E.g. having

[editor]
u=InsertChar:88;
x=InsertChar:85;

in mc.macros will replace every typed 'u' by a 'X' and 'x' by 'U'?
And same for contents of pasting?
Without option to switch off?

That's quite different to starting macros by Ctrl-a [assigned character].
Think that renders either macros or the internal editor useless /
only useable for special purposes.

Workaround tip for others affected: avoid macros or use another editor.

[ossi]

yes, the macro functionality is extremely aggressive. while it can be very convenient to assign shortcuts like ctrl-a (though the really interesting ones like ctrl-f1 don't seem to work), it seems a tad over the top that printable characters can be mapped as well.

note that the text from your webpage testcase actually contains markup; you can verify that by pasting it into libreoffice. apparently, your terminal interprets this by inserting kpenter (whatever that might be) into the stream.

[newbie-02]

short:

• make automatic replacement optional by one or two switches / options for typing and pasting.
• evaluate external changes to mc.macros on the fly.
• split in macros vs. replacement patterns.
• implement Ctrl-C and Ctrl-V in mc-editor for active push / pull clipboard?

IMHO defining macros as something the user can activate by a special typed! key sequence ( Ctrl-A and the assigned start character ) and then evaluating it at totally different situations ( pasting ) and in a totally different way ( replace the assigned character on its own ) is 'questionable'.

Agreeing that it's convenient and powerful for some use case to have 'automatic macro expansion' or 'automatic replacement' without needing manual activation, as well for typing as for pasting, but it's unwanted in other - most? - cases.

Thus I'd think it should have some switch to toggle between 'automatic macro expansion' or 'pure typing / pasting'. Two switches for replace on typing and replace on paste? While not implemented users may toggle by removing or renaming? 'mc.macros'.

Where it's inconvenient / inconsistent that adding macros is accounted on the fly, while changing or deleting them or even removing the mc.macros file needs restart of mc to take effect.

Evtl. it could also be meaningful to have different definitions for macros vs. automatic replacements?

AFAIK the clipboard contains different versions ( e.g. STRING, UTF8_STRING, TEXT, text/html ... ) of the copied text. Versions can be checked and partly steered by 'xclip', get: 'sudo apt install xclip', show targets: 'xclip -selection clipboard -o -t TARGETS', clear except UTF8_STRING: 'xclip -selection c -o | xclip -selection c'.

Don't know which one is selected or how it can be steered which to paste by Ctrl-Shift-V. But simple character replacement in mc hits on any version.

Would say the effect that pasting short texts with my obscure macro works clean, and after some length / content replacing steps in, replacing also in positions prior to the length / content needed to activate, is ... hard to understand.

Don't know if shell ( zsh in my case ) or terminal ( debian / XFCE Terminal Emulator in my case ) handles Shift-Ctrl-V, just learned at some point that that's the easy way to get content into mc editor as it doesn't accept Ctrl-V.

Don't know anything about 'kpenter', but think mc knows as it's evaluating it in some way?

Would say the general assigning of shortcuts / hotkeys on different system layers and in different programs becomes questionable at least when Ctrl-C is sometimes 'break' and sometimes 'copy'. But as widespread as Ctrl-C and Ctrl-V are in use for copy and paste I wouldn't bother if mc-editor would adopt that to push / pull to from clipboard instead of waiting for terminal to push something in it ... but that's for experienced people to decide what's possible / meaningful.

[zaytsev]

kpenter is "Keypad Enter"

[ossi]

yes. from my side the question was what the actual meaning/code is in a terminal, as i couldn't produce it in konsole without doing research first.

xclip can be used by mc; it's just a mess and requires configuration (see #4458 & #3083).

[newbie-02]

hmmmmm ...

so similar had been thought in the past.

I didn't want to propose using xclip if that's any difficult, just mentioned as user tool to check the clipboard.

But having usual reaction on Ctrl-C and Ctrl-V which users intuitively expect acc. 'common user interface' would be good. Is mc restricted to terminal functions to communicate with the clipboard? Or are there other paths's available?

As some level is capable of working with 'Shift-Ctrl-C' and 'Shift-Ctrl-V' it could be possible to use Ctrl-C and Ctrl-V as hotkeys and activate / trigger the code users would start by Shift-Ctrl...?

Similar it could be 'nice' to work on 'normal selection' like Ctrl-F does, rather than to have to mark text by 'Shift-mousing'.

If someone reworks that area: placing '<------>' which mc shows in editor instead of double-TAB or eight spaces is normally unwanted by users, as well as filling rows with spaces to full screen width when marking with shift-mouse and then copying with 'Shift-Ctrl-C'. Can't tell if terminal or mc issue, just in case ...

'kpenter' if that's something special and distinguished from 'Enter' it probably shouldn't trigger in pasting?

BTW, I tried to use another editor, unsetting 'Options - Configuration - use internal edit' fell back on vim, none of 'select-editor' or manually changing ~/.selected_editor with or without new start of mc could switch to nano, only exporting 'export EDITOR=nano' before starting mc changes. 'export EDITOR=mousepad' works too, could become my favorite.

[andrew_b]

src/editor/editwidget.c

@@ -959 +959 @@
             cb_ret_t ret = MSG_NOT_HANDLED;
 
             /* The user may override the access-keys for the menu bar. */
-            if (macro_index == -1 && edit_execute_macro (e, parm))
+            if (macro_index == -1 && !bracketed_pasting_in_progress && edit_execute_macro (e, parm))
             {
                 edit_update_screen (e);
                 ret = MSG_HANDLED;

[zaytsev]

Do you want to commit? I approve.

[andrew_b]

Branch: 4562_mcedit_macros_paste
changeset:e9258fcc6cd81e38c556732d8d8e954615ade04e

[andrew_b]

Merged to master: [45c28d9d11509fcf34d54743cadad7bebd2d9cd3].