Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #110702
| From | Michael Selik <michael.selik@gmail.com> |
|---|---|
| Newsgroups | comp.lang.python |
| Subject | Re: fastest way to read a text file in to a numpy array |
| Date | 2016-06-28 14:29 +0000 |
| Message-ID | <mailman.83.1467124225.2358.python-list@python.org> (permalink) |
| References | <37b46ad8-8318-4d67-a65c-7dd7a50a3848@googlegroups.com> <CAGgTfkOJC3Ws6aQtVq=e2g7rb8xqChMieJ9Sok93A65bL2dMNg@mail.gmail.com> <CA+1uRFxz-kTa8uCHNZLkZ5Xj2XkafNPLXSyGADYawedcqeL+wQ@mail.gmail.com> <CAGgTfkPTRhG_pc5q2QzbV8WqwZbhJNy+LYe6TAJg-jwmtcNp-w@mail.gmail.com> |
On Tue, Jun 28, 2016 at 10:08 AM Hedieh Ebrahimi <hemla21@gmail.com> wrote: > File 1 has : > x1,y1,z1 > x2,y2,z2 > .... > > and file2 has : > x1,y1,z1,value1 > x2,y2,z2,value2 > x3,y3,z3,value3 > ... > > I need to read the coordinates from file 1 and then interpolate a value > for these coordinates on file 2 to the closest coordinate possible. The > problem is file 2 is has around 5M lines. So I was wondering what would be > the fastest approach? > Is this a one-time task, or something you'll need to repeat frequently? How many points need to be interpolated? How do you define distance? Euclidean 3d distance? K-nearest? 5 million can probably fit into memory, so it's not so bad. NumPy is a good option for broadcasting the distance function across all 5 million labeled points for each unlabeled point. Given that file format, NumPy can probably read from file directly into an array. http://stackoverflow.com/questions/3518778/how-to-read-csv-into-record-array-in-numpy
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
fastest way to read a text file in to a numpy array Heli <hemla21@gmail.com> - 2016-06-28 06:45 -0700
Re: fastest way to read a text file in to a numpy array Michael Selik <michael.selik@gmail.com> - 2016-06-28 14:00 +0000
Re: fastest way to read a text file in to a numpy array Michael Selik <michael.selik@gmail.com> - 2016-06-28 14:29 +0000
Re: fastest way to read a text file in to a numpy array Cody Piersall <cody.piersall@gmail.com> - 2016-06-28 09:37 -0500
Re: fastest way to read a text file in to a numpy array Heli <hemla21@gmail.com> - 2016-06-30 08:49 -0700
Re: fastest way to read a text file in to a numpy array Christian Gollwitzer <auriocus@gmx.de> - 2016-06-30 23:02 +0200
csiph-web