Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.misc > #11878
| From | ram@zedat.fu-berlin.de (Stefan Ram) |
|---|---|
| Newsgroups | comp.lang.misc |
| Subject | Re: Resources to learn common lisp? |
| Date | 2026-09-18 15:18 +0000 |
| Organization | Stefan Ram |
| Message-ID | <words-20260918160015@ram.dialup.fu-berlin.de> (permalink) |
| References | (23 earlier) <expressiveness-20260827113139@ram.dialup.fu-berlin.de> <gU1rS.86252$Pq3.1276@fx14.ams4> <regexp-20260918112658@ram.dialup.fu-berlin.de> <n4brS.155253$NMu2.131377@fx15.ams4> <counting-20260918151542@ram.dialup.fu-berlin.de> |
ram@zedat.fu-berlin.de (Stefan Ram) wrote or quoted:
>Johann 'Myrkraverk' Oskarsson <johann@myrkraverk.invalid> wrote or quoted:
>>I use the word /try/ here because I'm not offhand sure a good word
>>counting program can be done with strictly a regular expression,
>During this thread I did not think about it as a real
>application, but just as a example for programming. But if
>you ask me about it, the most important thing for a really
>good word-counting program is one's definition of "word".
For one example, I wrote a program in 2020 to count the fre-
quencies of Japanese "words", where I had my program figure
out which characters go together by itself using something
that vaguely resembles byte-pair encoding (BPE), but I came
up with it myself before I ever heard of BPE.
Japanese words are usually not separated by spaces. So, my
program just observes which combinations of characters appear
most frequently in my collection of Japanese texts, but also
can deal with words that appear both separated from other words
and also as parts of other words. For example, below both "ああ"
and "あ" are listed, and the 2*32232 occurences of "あ" in "ああ"
are not counted again to get the 22929 occurences of "あ".
These are "words" by the definition of my program - which might
not agree with other definitions.
So, now you know which "words" you should learn first because
they are most common - from the output of my program (start):
It says, "Hai" ("yes") is the most common Japanese word! Readers
might be surprised by "ああ" - this appears often in the spoken
language I was interested in, but rarely in formal texts.
And we see that the few kanji I know ("何", "私", and "今") are
indeed very common! Again the 15591 hits of "何" in isolation
are counted only when its not part of the "何か", "何で", and
"何だ", which are listed and counted separately. And this also
applies to longer sequences correspondingly.
"はい" ×67739 ↔2 – HA I
"あっ" ×51279 ↔2 – A SMALL-TU
"うん" ×38420 ↔2 – U N
"えっ" ×37268 ↔2 – E SMALL-TU
"いや" ×32336 ↔2 – I YA
"ああ" ×32232 ↔2 – A A
"でも" ×29729 ↔2 – DE MO
"もう" ×23096 ↔2 – MO U
"あ" ×22929 ↔1 – A
"じゃあ" ×20123 ↔3 – ZI SMALL-YA A
"ちょっと" ×19575 ↔4 – TI SMALL-YO SMALL-TU TO
"あの" ×18596 ↔2 – A NO
"え" ×16755 ↔1 – E
"ん" ×16391 ↔1 – N
"これ" ×16078 ↔2 – KO RE
"何" ×15591 ↔1 – X-4F55
"私" ×14642 ↔1 – X-79C1
"だから" ×14125 ↔3 – DA KA RA
"おい" ×13855 ↔2 – O I
"まあ" ×13307 ↔2 – MA A
"今" ×12493 ↔1 – X-4ECA
"すいません" ×11940 ↔5 – SU I MA SE N
"私は" ×11247 ↔2 – X-79C1 HA
"ええ" ×11237 ↔2 – E E
"何か" ×10606 ↔2 – X-4F55 KA
"お前" ×10569 ↔2 – O X-524D
"それは" ×10475 ↔3 – SO RE HA
"ありがとう" ×10469 ↔5 – A RI GA TO U
"そう" ×9965 ↔2 – SO U
"ねえ" ×9752 ↔2 – NE E
"あれ" ×9180 ↔2 – A RE
"どうぞ" ×8971 ↔3 – DO U ZO
"ほら" ×8948 ↔2 – HO RA
"何で" ×8822 ↔2 – X-4F55 DE
"ありがとうございます" ×8571 ↔10 – A RI GA TO U GO ZA I MA SU
"あぁ" ×8396 ↔2 – A SMALL-A
"お願いします" ×7552 ↔6 – O X-9858 I SI MA SU
"俺" ×7515 ↔1 – X-4FFA
"また" ×7467 ↔2 – MA TA
"で" ×7283 ↔1 – DE
"よし" ×7214 ↔2 – YO SI
"これは" ×7143 ↔3 – KO RE HA
"いえ" ×7052 ↔2 – I E
"はぁ" ×7008 ↔2 – HA SMALL-A
"どうして" ×6943 ↔4 – DO U SI TE
"まだ" ×6851 ↔2 – MA DA
"大丈夫" ×6838 ↔3 – X-5927 X-4E08 X-592B
"それ" ×6710 ↔2 – SO RE
"俺は" ×6630 ↔2 – X-4FFA HA
"って" ×6593 ↔2 – SMALL-TU TE
"一同" ×6477 ↔2 – X-4E00 X-540C
"だって" ×6413 ↔3 – DA SMALL-TU TE
"ごめん" ×6407 ↔3 – GO ME N
"やっぱり" ×6314 ↔4 – YA SMALL-TU PA RI
"みんな" ×6033 ↔3 – MI N NA
"先生" ×5932 ↔2 – X-5148 X-751F
"えッ" ×5932 ↔2 – E SMALL-TU
"しかし" ×5825 ↔3 – SI KA SI
"い" ×5813 ↔1 – I
"それで" ×5798 ↔3 – SO RE DE
"そして" ×5790 ↔3 – SO SI TE
"う" ×5629 ↔1 – U
"はあ" ×5624 ↔2 – HA A
"どうも" ×5535 ↔3 – DO U MO
"は" ×5511 ↔1 – HA
"では" ×5275 ↔2 – DE HA
"あッ" ×5256 ↔2 – A SMALL-TU
"さあ" ×5123 ↔2 – SA A
"ごめんなさい" ×4997 ↔6 – GO ME N NA SA I
"今日は" ×4907 ↔3 – X-4ECA X-65E5 HA
"そんな" ×4819 ↔3 – SO N NA
"はっ" ×4805 ↔2 – HA SMALL-TU
"それが" ×4802 ↔3 – SO RE GA
"すみません" ×4755 ↔5 – SU MI MA SE N
"まさか" ×4616 ↔3 – MA SA KA
"失礼します" ×4494 ↔5 – X-5931 X-793C SI MA SU
"そうか" ×4428 ↔3 – SO U KA
"何だ" ×4237 ↔2 – X-4F55 DA
. . .
Newsgroups: comp.lang.misc
Back to comp.lang.misc | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Re: Resources to learn common lisp? Johann 'Myrkraverk' Oskarsson <johann@myrkraverk.invalid> - 2026-08-24 19:38 +0800
Re: Resources to learn common lisp? tfb <tfb@work.it.out> - 2026-08-24 12:22 +0000
Re: Resources to learn common lisp? Johann 'Myrkraverk' Oskarsson <johann@myrkraverk.invalid> - 2026-08-24 21:04 +0800
Re: Resources to learn common lisp? tfb <tfb@work.it.out> - 2026-08-24 13:18 +0000
Re: Resources to learn common lisp? Johann 'Myrkraverk' Oskarsson <johann@myrkraverk.invalid> - 2026-08-24 22:16 +0800
Re: Resources to learn common lisp? tfb <tfb@work.it.out> - 2026-08-24 20:10 +0000
Re: Resources to learn common lisp? Johann 'Myrkraverk' Oskarsson <johann@myrkraverk.invalid> - 2026-08-25 10:34 +0800
Re: Resources to learn common lisp? Paul Rubin <no.email@nospam.invalid> - 2026-08-24 17:10 -0700
Re: Resources to learn common lisp? Johann 'Myrkraverk' Oskarsson <johann@myrkraverk.invalid> - 2026-08-25 10:41 +0800
Re: Resources to learn common lisp? tfb <tfb@work.it.out> - 2026-08-25 09:22 +0000
Re: Resources to learn common lisp? Paul Rubin <no.email@nospam.invalid> - 2026-08-25 22:13 -0700
Re: Resources to learn common lisp? Anton Antimo <anton@safunu.org> - 2026-08-26 15:43 -0300
Re: Resources to learn common lisp? Paul Rubin <no.email@nospam.invalid> - 2026-08-26 18:23 -0700
Re: Resources to learn common lisp? Anton Antimo <anton@safunu.org> - 2026-08-28 17:29 -0300
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-08-27 10:17 +0000
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-08-27 10:47 +0000
Re: Resources to learn common lisp? "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> - 2026-09-18 10:49 +0800
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-18 10:25 +0000
Re: Resources to learn common lisp? "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> - 2026-09-18 11:05 +0800
Re: Resources to learn common lisp? red floyd <no.spam.here@its.invalid> - 2026-09-17 21:43 -0700
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-18 10:28 +0000
Re: Resources to learn common lisp? Johann 'Myrkraverk' Oskarsson <johann@myrkraverk.invalid> - 2026-09-18 21:33 +0800
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-18 13:51 +0000
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-18 14:23 +0000
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-18 15:18 +0000
Re: Resources to learn common lisp? "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> - 2026-09-19 00:07 +0800
Resources to learn Japanese (was: Resources to learn common lisp?) ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-18 16:52 +0000
Re: Resources to learn common lisp? boltar@caprica.universe - 2026-09-18 15:11 +0000
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-18 15:25 +0000
Re: Resources to learn common lisp? boltar@caprica.universe - 2026-09-18 15:49 +0000
Re: Resources to learn common lisp? ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-18 16:05 +0000
Re: Resources to learn common lisp? boltar@caprica.universe - 2026-09-19 08:29 +0000
Re: Resources to learn common lisp? "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> - 2026-09-19 16:50 +0800
Re: Resources to learn common lisp? David Brown <david.brown@hesbynett.no> - 2026-09-18 18:18 +0200
Re: Resources to learn common lisp? "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> - 2026-09-19 00:35 +0800
Why no string switch? (was: Resources to learn common lisp?) ram@zedat.fu-berlin.de (Stefan Ram) - 2026-09-19 12:46 +0000
Re: Why no string switch? (was: Resources to learn common lisp?) boltar@caprica.universe - 2026-09-19 14:47 +0000
Re: Resources to learn common lisp? "Johann \"Myrkraverk\" Oskarsson" <johann@myrkraverk.invalid> - 2026-09-18 23:39 +0800
csiph-web