Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.clipper.visual-objects > #8049 > unrolled thread
| Started by | Frans Wauters <info@wautersit.com> |
|---|---|
| First post | 2015-05-06 02:02 -0700 |
| Last post | 2015-05-09 09:06 +0200 |
| Articles | 13 — 5 participants |
Back to article view | Back to comp.lang.clipper.visual-objects
CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Frans Wauters <info@wautersit.com> - 2015-05-06 02:02 -0700
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Frans Wauters <info@wautersit.com> - 2015-05-06 02:04 -0700
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Frans Wauters <info@wautersit.com> - 2015-05-06 07:17 -0700
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Karl Faller <mail_withoutthat_@_drei-f_andthat_.de> - 2015-05-06 11:16 +0200
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Frans Wauters <info@wautersit.com> - 2015-05-06 02:47 -0700
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) "Fred Killet" <info@killetsoft.de> - 2015-05-06 12:13 +0200
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Frans Wauters <info@wautersit.com> - 2015-05-06 04:26 -0700
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Karl Faller <mail_withoutthat_@_drei-f_andthat_.de> - 2015-05-06 15:43 +0200
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) "Fred Killet" <info@killetsoft.de> - 2015-05-08 17:28 +0200
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Karl Faller <mail_withoutthat_@_drei-f_andthat_.de> - 2015-05-10 21:04 +0200
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) "Philippe Bayard (Belgium)" <philbayard@gmail.com> - 2015-05-06 03:58 -0700
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) Frans Wauters <info@wautersit.com> - 2015-05-06 04:19 -0700
Re: CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) "Arne Ortlinghaus" <Arne.Ortlinghaus@acs.it> - 2015-05-09 09:06 +0200
| From | Frans Wauters <info@wautersit.com> |
|---|---|
| Date | 2015-05-06 02:02 -0700 |
| Subject | CSV text files in several formats ( ANSI, Unicode, UTF-8, ... ) |
| Message-ID | <23619998-fe76-40de-9402-0b873aa6b4be@googlegroups.com> |
Greetings all,
Lately I receive (from third parties) very big CSV text files in several formats and I look for a way to detect in which format I can read the content correctly and/or convert them to/from ANSI or UNICODE etc...
When I use notepad, I can read and see all data correctly. Notepad and Notepad++ is capable to read all these different text file formats. However when I perform a type in and CMD box, I can't read some files correctly. In notepad, and when I using "save as", I can see that some files are in ANSI where other are in UNICODE and even UTF-8.
Example: Use notepad and copy/past the following 20 chars:
àäâãéèëêïîöôõùüû EURO$£¥
In notepad, use the "save as" option and save it 3 times, as an ANSI, as UNICODE and as UTP-8 file.
Quick test code ( console application ) or use TYPE <filename> in a DOS box or even an hexeditor to see the difference in txt format.
FUNCTION Start()
LOCAL cTxt AS STRING
LOCAL i AS DWORD
? cTxt := MemoRead( "test-ansi.csv" )
?
?
FOR i := 1 TO Len( cTxt )
?? StrZero( Asc( SubStr( cTxt, i , 1 ) ), 3 ) + " " + SubStr( cTxt, i , 1 ) + " "
NEXT
?
WAIT
RETURN NIL
Kind regards,
Frans
[toc] | [next] | [standalone]
| From | Frans Wauters <info@wautersit.com> |
|---|---|
| Date | 2015-05-06 02:04 -0700 |
| Message-ID | <18482dfa-bb61-4022-a168-e2f00c390748@googlegroups.com> |
| In reply to | #8049 |
It seems that Google change the "EURO" symbol to EURO. Even not shure is Google change it again. HTML code for the Euro sign = €
[toc] | [prev] | [next] | [standalone]
| From | Frans Wauters <info@wautersit.com> |
|---|---|
| Date | 2015-05-06 07:17 -0700 |
| Message-ID | <acb61070-cd38-4c31-b4ca-91abbf42fc46@googlegroups.com> |
| In reply to | #8050 |
UTF-8 Unicode and explained in 10 minits. Characters, Symbols and the Unicode Miracle - Computerphile https://www.youtube.com/watch?v=MijmeoH9LT4 Some YouTube movies are very instructive.
[toc] | [prev] | [next] | [standalone]
| From | Karl Faller <mail_withoutthat_@_drei-f_andthat_.de> |
|---|---|
| Date | 2015-05-06 11:16 +0200 |
| Message-ID | <upmjkatg741jctqbndd5i9ol7g8hrerh66@4ax.com> |
| In reply to | #8049 |
Frans, if "...Notepad and Notepad++ is capable to read all these different text file formats..." why not use them? Start them with filename from VO, let them save to your preferred encoding and then read the result with your code. Probably easier than trying to detect encoding markers in the source files... Karl
[toc] | [prev] | [next] | [standalone]
| From | Frans Wauters <info@wautersit.com> |
|---|---|
| Date | 2015-05-06 02:47 -0700 |
| Message-ID | <92a48fec-cae8-42e8-8d44-43dd1f23c7b5@googlegroups.com> |
| In reply to | #8051 |
Hi Karl, Some of these files are big, realy big, 600 MB till 1300 MB depending the file format. That's real live horror using notapad to convert them. I'm testing something using a batch file: FOR %%A in (*.csv) DO cmd /a /c type %%A > ANSI\%%A FOR %%A in (*.csv) DO cmd /u /c type %%A > UNICODE\%%A But even then, these results are hard to check and UTF-8 is a pain in the a** as not supported -> /a for ansi and /u for unicode. Someting as ConvertTo <ansi|unicode|utf-8> <inputfile> <outputfile> would be ideal to use this in a batchfile so I can script it to run in a windows sheduler. Regards, Frans
[toc] | [prev] | [next] | [standalone]
| From | "Fred Killet" <info@killetsoft.de> |
|---|---|
| Date | 2015-05-06 12:13 +0200 |
| Message-ID | <micph4$fia$1@news.albasani.net> |
| In reply to | #8049 |
Hi Frans,
use the program CONVERT. It handles input file, output file and a parameter
file from the command line.
"CONVERT data converter freeware converts databases and files between the
file formats dBase, CSV and SDF and the character sets ANSI, ASCII, UTF8 and
Unicode and allows in the same time to sort the data and to do selections."
It is written completely in VO and it is fast.
Download from http://www.killetsoft.de/p_cona_e.htm. If you use it not
commercialy it is free.
Fred Killet
killet-nospam@killetsoft.de
"Frans Wauters" <info@wautersit.com> schrieb im Newsbeitrag
news:23619998-fe76-40de-9402-0b873aa6b4be@googlegroups.com...
Greetings all,
Lately I receive (from third parties) very big CSV text files in several
formats and I look for a way to detect in which format I can read the
content correctly and/or convert them to/from ANSI or UNICODE etc...
When I use notepad, I can read and see all data correctly. Notepad and
Notepad++ is capable to read all these different text file formats. However
when I perform a type in and CMD box, I can't read some files correctly. In
notepad, and when I using "save as", I can see that some files are in ANSI
where other are in UNICODE and even UTF-8.
Example: Use notepad and copy/past the following 20 chars:
àäâãéèëêïîöôõùüû EURO$£¥
In notepad, use the "save as" option and save it 3 times, as an ANSI, as
UNICODE and as UTP-8 file.
Quick test code ( console application ) or use TYPE <filename> in a DOS box
or even an hexeditor to see the difference in txt format.
FUNCTION Start()
LOCAL cTxt AS STRING
LOCAL i AS DWORD
? cTxt := MemoRead( "test-ansi.csv" )
?
?
FOR i := 1 TO Len( cTxt )
?? StrZero( Asc( SubStr( cTxt, i , 1 ) ), 3 ) + " " + SubStr( cTxt, i
, 1 ) + " "
NEXT
?
WAIT
RETURN NIL
Kind regards,
Frans
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
http://www.avast.com
[toc] | [prev] | [next] | [standalone]
| From | Frans Wauters <info@wautersit.com> |
|---|---|
| Date | 2015-05-06 04:26 -0700 |
| Message-ID | <91891001-b89a-4922-a573-c7cd407c6a2f@googlegroups.com> |
| In reply to | #8053 |
Hi Fred, Thanks, I downloaded it and I will test it and let you know if the "stupid and dirty" txt files I get can be fixed :) I also have contacted the provider of these files and ask them to check there output engine and produced csv files. Kind regards, Frans
[toc] | [prev] | [next] | [standalone]
| From | Karl Faller <mail_withoutthat_@_drei-f_andthat_.de> |
|---|---|
| Date | 2015-05-06 15:43 +0200 |
| Message-ID | <0k6kkat8tpssba4h99to9189his5pgjv7i@4ax.com> |
| In reply to | #8053 |
Fred, >use the program CONVERT. It handles input file, output file and a parameter >file from the command line. > >"CONVERT data converter freeware converts databases and files between the >file formats dBase, CSV and SDF and the character sets ANSI, ASCII, UTF8 and >Unicode and allows in the same time to sort the data and to do selections." what about FPT files? Karl
[toc] | [prev] | [next] | [standalone]
| From | "Fred Killet" <info@killetsoft.de> |
|---|---|
| Date | 2015-05-08 17:28 +0200 |
| Message-ID | <miikm7$i7d$1@news.albasani.net> |
| In reply to | #8057 |
Hi Karl, CONVERT only supports the famos dBase, CSV and SDF file formats. Fred -- Fred Killet killet-nospam@killetsoft.de "Karl Faller" <mail_withoutthat_@_drei-f_andthat_.de> schrieb im Newsbeitrag news:0k6kkat8tpssba4h99to9189his5pgjv7i@4ax.com... > Fred, >>use the program CONVERT. It handles input file, output file and a >>parameter >>file from the command line. >> >>"CONVERT data converter freeware converts databases and files between the >>file formats dBase, CSV and SDF and the character sets ANSI, ASCII, UTF8 >>and >>Unicode and allows in the same time to sort the data and to do >>selections." > what about FPT files? > > Karl --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. http://www.avast.com
[toc] | [prev] | [next] | [standalone]
| From | Karl Faller <mail_withoutthat_@_drei-f_andthat_.de> |
|---|---|
| Date | 2015-05-10 21:04 +0200 |
| Message-ID | <5tavkahtsn142r694lj2ihm931kbcubbtu@4ax.com> |
| In reply to | #8065 |
Fred, >Hi Karl, > >CONVERT only supports the famos dBase, CSV and SDF file formats. what a pity... Karl
[toc] | [prev] | [next] | [standalone]
| From | "Philippe Bayard (Belgium)" <philbayard@gmail.com> |
|---|---|
| Date | 2015-05-06 03:58 -0700 |
| Message-ID | <d955dda1-45ac-497b-90cc-0335f92c43b0@googlegroups.com> |
| In reply to | #8049 |
Hi Frans,
for UTF-8 ( or at least UTF-8 with BOM ) you can check the 4 first characters of the file.
Something like this :
Function File_encoding( FilePath ) as STRING PASCAL
LOCAL UTF_16_BE_BOM := "\xFE\xFF" as STRING
LOCAL UTF_16_LE_BOM := "\xFF\xFE" as STRING
LOCAL UTF_8_BOM := "\xEF\xBB\xBF" as STRING
LOCAL UTF_32_BE_BOM := "\x00\x00\xFE\xFF" as STRING
LOCAL UTF_32_LE_BOM := "\xFF\xFE\x00\x00" as STRING
LOCAL ptrHandle as ptr
LOCAL cBuffer as psz
LOCAL dwSize as DWORD
ptrHandle := FOpen(FilePath, FO_READ)
IF ptrHandle = F_ERROR
return ""
ELSE
dwSize := 4
cBuffer := MemAlloc( dwSize )
IF FRead(ptrHandle, @cBuffer, 4) <> 4
ELSE
IF cBuffer[1] == 0xEF .and. cBuffer[2] == 0xBB .and. cBuffer[3] == 0xBF
//UTF-8
MemFree(cBuffer)
FClose(ptrHandle)
return "UTF-8"
ELSEIF ( cBuffer[1] == 0xFE .and. cBuffer[2] == 0xFF) .or. ( cBuffer[2] == 0xFE .and. cBuffer[1] == 0xFF)
//UTF-16 (BE)
MemFree(cBuffer)
FClose(ptrHandle)
return "UTF-16"
ELSEIF ( cBuffer[1] == 0x00 .and. cBuffer[2] == 0x00 .and. cBuffer[3] == 0xFE .and. cBuffer[4] == 0xFF) .or. ( cBuffer[1] == 0xFF .and. cBuffer[2] == 0xFE .and. cBuffer[3] == 0x00 .and. cBuffer[4] == 0x00)
//UTF-16 (BE)
MemFree(cBuffer)
FClose(ptrHandle)
return "UTF-32"
endif
ENDIF
ENDIF
MemFree(cBuffer)
FClose(ptrHandle)
return ""
Best regards,
Philippe
[toc] | [prev] | [next] | [standalone]
| From | Frans Wauters <info@wautersit.com> |
|---|---|
| Date | 2015-05-06 04:19 -0700 |
| Message-ID | <8dcaaac5-cc66-4147-ab69-4806ed3735c7@googlegroups.com> |
| In reply to | #8054 |
Hi Philippe, For the most of UTF-8 files, this code is usable. I just checked one of the files in UTF-8 format ( according to notepad ) with an hexviewer. The file start with: 22 43 61 74 65 67 6F 72 2C 22 2C 22 43 6F 64 65 22 ... Corresponding with : " C a t e g o r y " , " C o d e " ... I find some parts in the file as : C3 A9 that needs to be translated to one leter: é Thanks for the effort. Best regards, Frans
[toc] | [prev] | [next] | [standalone]
| From | "Arne Ortlinghaus" <Arne.Ortlinghaus@acs.it> |
|---|---|
| Date | 2015-05-09 09:06 +0200 |
| Message-ID | <mikbk1$2jo$1@dont-email.me> |
| In reply to | #8055 |
Unfortunately not all files have the "hidden" charachters at the beginning of the file. So it is not always possible to detect the right format. It is only possible to try to have a rate as good as possible limiting the algorithms to what will probably arrive. Even Notepad ++ sometimes does not find the correct encoding. I have the same task but until now I have not inserted an automatic detection in our programs because of the many possible cases. A WinAPI function exists for trying to detect the format, but I do not believe that it works well. Currently I have forgot the name of the function. It was hard to find comments about it so it seems not to be used often. Here are some advices. http://en.wikipedia.org/wiki/Charset_detection Arne Ortlinghaus ACS Data Systems
[toc] | [prev] | [standalone]
Back to top | Article view | comp.lang.clipper.visual-objects
csiph-web