Using TYPE with non-English text

[Joe Caverly]

     _x64: 1
   _admin: 1
_elevated: 1

TCC  27.00.18 x64   Windows 10 [Version 10.0.19042.746]

In the clipboard, I have an Arabic word.

TYPE does not return the proper characters, but @LINE does.

Why the difference?

Joe

 

[rconn]

Is it Unicode?

 

[Joe Caverly]

Joe

 

[Charles Dye]

The contents of the clipboard. Which I guess means, "How are you putting this text on the clipboard?"

 

[Joe Caverly]

Google Translate

I click on the lower-right icon to copy to the clipboard.

Joe

 

[Charles Dye]

Google Translate

I click on the lower-right icon to copy to the clipboard.

If it's the browser doing it, then I'm gonna guess CF_UNICODE.

 

[rconn]

The text snippet isn't long enough for TYPE to determine whether it's ASCII, UTF8, or UTF16. (And it doesn't have a BOM to help.)

 

[vefatica]

The text snippet isn't long enough for TYPE to determine whether it's ASCII, UTF8, or UTF16. (And it doesn't have a BOM to help.)

How do @clip[0] and @line[clip:,0] manage to get it right?

 

[Charles Dye]

The text snippet isn't long enough for TYPE to determine whether it's ASCII, UTF8, or UTF16. (And it doesn't have a BOM to help.)

Is TYPE reading from the clipboard directly? Or is it copying the data out to a file, and then dumping that?

 

[rconn]

How do @clip[0] and @line[clip:,0] manage to get it right?

They always assume it's Unicode.

 

[vefatica]

Hmmm! In Windows Terminal I see a rendering of it that looks more like Google's (than TCMD's rendering; consoles just give the unprintable symbol). Why does TCMD show it differently? And I'd like to know how WT/TCCTCMD/ConHost (???) knows it's right-to-left text. In the third line below, I pasted it on the command line and pressed backspace once; the cursor was at the righthand end but the leftmost character was deleted.

 

[vefatica]

Here they are, side by side, Windows Terminal on the left, TCMD on the right, pasted from the same clipboard, same font (Consolas). Why do they look different? And when I look at Consolas with CharMap, I don't see ANY Arabic characters.

 

[rconn]

Hmmm! In Windows Terminal I see a rendering of it that looks more like Google's (than TCMD's rendering; consoles just give the unprintable symbol). Why does TCMD show it differently? And I'd like to know how WT/TCCTCMD/ConHost (???) knows it's right-to-left text. In the third line below, I pasted it on the command line and pressed backspace once; the cursor was at the righthand end but the leftmost character was deleted.

View attachment 3246

Take Command and TCC do not support RTL character sets.

 

[Joe Caverly]

Here they are, side by side, Windows Terminal on the left, TCMD on the right, pasted from the same clipboard, same font (Consolas). Why do they look different? And when I look at Consolas with CharMap, I don't see ANY Arabic characters.

View attachment 3248

Hey @vefatica do you get the same results using the @UNICODE function on Windows Terminal as TCMD?

EDIT: Yep, same numbers as TCMD here on Windows Terminal;

Joe

 

[vefatica]

Hey @vefatica do you get the same results using the @UNICODE function on Windows Terminal as TCMD?

EDIT: Yep, same numbers as TCMD here on Windows Terminal;

View attachment 3249

Joe

They're the same for me too.

 

[Charles Dye]

The command line always uses UTF-16.

 

[Joe Caverly]

This works;

Joe

 

[vefatica]

So does this (in TCMD or Terminal)

 

[Joe Caverly]

VIEW works as it should with Arabic text in a file;

v.exe e:\utils\test.txt

This does not work;

e:\utils>echo %@line[test.txt,0]
مرحبا

...but this works;

e:\utils>echo %@line[test.txt,0] | *view

Why does echo @line work in my OP, that is, using the clip: as source, but echo @line does not work when the source is a text file?

Here's the hex VIEW for the file;

Yet, echo @line works when piped through to VIEW.

Not sure why I did not have problems a few years back when working with Hebrew characters, but Arabic is not playing nice.

Joe

 

[Charles Dye]

My psychic powers tell me that TEST.TXT is UTF-8 with no BOM, and OPTION UTF8 returns a No.

 

[Joe Caverly]

My psychic powers tell me that TEST.TXT is UTF-8 with no BOM, and OPTION UTF8 returns a No.

Well, psychic one, you are correct!

I've enabled Unicode output and UTF8, and it seems to be working now.

This works now;

Thanks @Charles Dye

Joe

 

[Charles Dye]

My chance to channel Raymond Chen....

 

[Joe Caverly]

Don't forget to #define UNICODE if you want Unicode