Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.arch > #111126 > unrolled thread
| Started by | Robert Finch <robfi680@gmail.com> |
|---|---|
| First post | 2025-03-13 00:50 -0400 |
| Last post | 2025-03-17 17:33 -0400 |
| Articles | 20 on this page of 211 — 12 participants |
Back to article view | Back to comp.arch
MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-13 00:50 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-13 16:34 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-13 13:49 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-13 18:42 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-13 15:59 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-13 20:46 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-13 21:39 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-13 17:00 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-13 23:27 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-13 21:27 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-13 19:12 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-13 18:34 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-13 19:28 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-13 21:14 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-13 21:34 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-13 23:34 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-13 23:52 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-14 14:52 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-14 15:34 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-14 17:35 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-14 18:12 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-14 18:42 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-14 22:10 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-15 17:46 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-15 16:46 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-15 22:28 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-16 17:35 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-16 22:00 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-17 13:38 +0000
Re: MSI interrupts Michael S <already5chosen@yahoo.com> - 2025-03-17 16:11 +0200
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-17 11:23 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-17 15:44 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-17 14:33 -0400
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-17 17:14 -0400
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-18 14:51 -0400
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-18 17:59 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-18 23:00 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-19 13:49 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-19 15:54 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-19 16:14 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-19 19:03 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-19 14:03 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-19 15:59 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-20 12:44 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-20 19:41 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-20 20:25 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-20 21:55 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 12:37 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-24 13:59 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-24 17:47 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 18:49 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-24 12:12 -0700
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-24 19:46 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 20:14 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-24 19:02 -0700
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-24 19:05 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 11:17 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 11:14 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-25 13:32 -0700
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-24 17:36 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-24 14:02 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-24 19:09 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 20:28 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-24 22:29 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 23:27 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-25 13:43 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 15:08 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-25 11:48 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-25 16:29 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-25 13:28 -0400
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-25 13:53 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-25 20:33 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 17:47 +0000
Re: MSI interrupts Terje Mathisen <terje.mathisen@tmsw.no> - 2025-03-27 10:55 +0100
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-27 20:31 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-24 19:08 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 11:20 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-25 13:38 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 22:21 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-25 16:23 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 23:29 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-25 16:53 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 18:55 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-24 13:46 -0400
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-24 12:09 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 20:11 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-24 22:50 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 23:45 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-25 01:53 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-25 01:26 -0400
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-25 00:07 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 12:23 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-25 13:45 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-25 15:01 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 15:10 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-25 16:40 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 17:54 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-25 20:40 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 22:19 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-25 18:49 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 23:33 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-25 22:58 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-26 04:40 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-26 19:42 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-27 09:17 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-27 13:42 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-25 17:09 -0700
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-26 01:00 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-26 00:56 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-26 04:08 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-26 09:55 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-26 14:57 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-26 13:22 -0700
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-04-03 22:19 -0700
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-26 19:21 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-26 15:41 -0400
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-26 15:50 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-26 23:16 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-26 21:35 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-26 23:22 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-26 23:48 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-27 00:18 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-27 12:46 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-27 17:19 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-27 14:27 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-27 18:34 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-27 19:18 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-28 01:01 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-28 01:35 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-28 01:47 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-27 22:02 -0400
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-27 19:14 -0700
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-29 10:31 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-28 02:56 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-27 21:52 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-28 02:58 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-27 19:09 -0700
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-27 19:10 -0700
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-27 21:47 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-28 02:53 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-28 15:24 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-28 11:41 -0400
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-28 09:51 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-29 14:28 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-29 15:23 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-29 20:58 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-29 22:04 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-25 10:54 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-25 16:31 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-25 17:44 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-28 14:40 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-29 20:55 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-30 16:34 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-31 03:33 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-31 01:30 -0400
Software synchronization (was: Re: MSI interrupts) cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-31 14:42 +0000
Threads (was Re: MSI interrupts) scott@slp53.sl.home (Scott Lurndal) - 2025-03-31 15:28 +0000
Re: Threads (was Re: MSI interrupts) Terje Mathisen <terje.mathisen@tmsw.no> - 2025-04-01 16:44 +0200
Re: Threads (was Re: MSI interrupts) scott@slp53.sl.home (Scott Lurndal) - 2025-04-01 15:31 +0000
Re: Threads (was Re: MSI interrupts) "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-04-01 12:49 -0700
Re: Threads (was Re: MSI interrupts) scott@slp53.sl.home (Scott Lurndal) - 2025-04-01 20:01 +0000
Re: Threads (was Re: MSI interrupts) "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-04-01 13:17 -0700
Re: Threads (was Re: MSI interrupts) scott@slp53.sl.home (Scott Lurndal) - 2025-04-01 22:04 +0000
Re: Threads (was Re: MSI interrupts) anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2025-04-02 06:25 +0000
Re: Threads (was Re: MSI interrupts) cross@spitfire.i.gajendra.net (Dan Cross) - 2025-04-02 13:04 +0000
Re: Threads (was Re: MSI interrupts) Terje Mathisen <terje.mathisen@tmsw.no> - 2025-04-03 16:45 +0200
Re: Threads (was Re: MSI interrupts) scott@slp53.sl.home (Scott Lurndal) - 2025-04-03 15:00 +0000
Re: Threads (was Re: MSI interrupts) cross@spitfire.i.gajendra.net (Dan Cross) - 2025-04-03 17:16 +0000
Re: Threads (was Re: MSI interrupts) scott@slp53.sl.home (Scott Lurndal) - 2025-04-03 17:24 +0000
Re: Threads (was Re: MSI interrupts) cross@spitfire.i.gajendra.net (Dan Cross) - 2025-04-04 01:44 +0000
Centronics (was Re: Threads (was Re: MSI interrupts)) George Neuner <gneuner2@comcast.net> - 2025-04-03 22:22 -0400
Re: Centronics (was Re: Threads (was Re: MSI interrupts)) scott@slp53.sl.home (Scott Lurndal) - 2025-04-04 02:29 +0000
Re: Threads (was Re: MSI interrupts) George Neuner <gneuner2@comcast.net> - 2025-04-03 01:38 -0400
Re: Threads (was Re: MSI interrupts) cross@spitfire.i.gajendra.net (Dan Cross) - 2025-04-03 12:15 +0000
Re: Threads (was Re: MSI interrupts) Robert Swindells <rjs@fdy2.co.uk> - 2025-04-04 10:41 +0000
Re: Threads (was Re: MSI interrupts) George Neuner <gneuner2@comcast.net> - 2025-04-04 23:10 -0400
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-04-01 10:07 -0400
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-04-01 17:39 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-19 23:28 +0000
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-20 12:50 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-20 22:28 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-23 12:16 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-23 16:48 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-24 11:55 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-24 18:56 +0000
Re: MSI interrupts "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-03-23 17:48 -0700
Re: MSI interrupts cross@spitfire.i.gajendra.net (Dan Cross) - 2025-03-24 18:39 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-20 12:03 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-20 19:52 +0000
Re: MSI interrupts EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-19 13:07 -0400
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-19 15:47 -0400
Re: MSI interrupts - Instructions Robert Finch <robfi680@gmail.com> - 2025-03-26 06:56 -0400
Re: MSI interrupts - Instructions EricP <ThatWouldBeTelling@thevillage.com> - 2025-03-26 10:33 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-18 22:49 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-18 20:39 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-19 01:22 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-18 23:22 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-19 13:54 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-19 15:56 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-17 21:49 +0000
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-18 01:34 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-18 18:51 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-18 19:19 +0000
Re: MSI interrupts Stefan Monnier <monnier@iro.umontreal.ca> - 2025-03-19 11:20 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-19 16:00 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-19 13:27 -0400
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-19 19:17 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-17 10:12 -0400
Re: MSI interrupts scott@slp53.sl.home (Scott Lurndal) - 2025-03-17 15:53 +0000
Re: MSI interrupts mitchalsup@aol.com (MitchAlsup1) - 2025-03-17 19:06 +0000
Re: MSI interrupts Robert Finch <robfi680@gmail.com> - 2025-03-17 17:33 -0400
Page 1 of 11 [1] 2 3 … 11 Next page →
| From | Robert Finch <robfi680@gmail.com> |
|---|---|
| Date | 2025-03-13 00:50 -0400 |
| Subject | MSI interrupts |
| Message-ID | <vqto79$335c6$1@dont-email.me> |
Planning on having the Q+ CPU respond directly to MSI interrupts. My thought is to integrate an interrupt controller directly into the CPU which will have several queues to prioritize interrupts. The controller will take care of detecting the highest priority interrupt to process, performing priority inversion occasionally, and detect stuck interrupts. Setting up an MSI interrupt protocol for Q+. The controller will sit on the CPU's main response collect bus. Interrupt messages contain the interrupter's bus/device/function + a cause code. They may also contain a target core number. Planning on distributing the interrupt control to peripheral devices instead of having a system level interrupt controller.
[toc] | [next] | [standalone]
| From | mitchalsup@aol.com (MitchAlsup1) |
|---|---|
| Date | 2025-03-13 16:34 +0000 |
| Message-ID | <3d5200797dd507ae051195e0b2d8ff56@www.novabbs.org> |
| In reply to | #111126 |
On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote:
> Planning on having the Q+ CPU respond directly to MSI interrupts. My
> thought is to integrate an interrupt controller directly into the CPU
> which will have several queues to prioritize interrupts. The controller
> will take care of detecting the highest priority interrupt to process,
> performing priority inversion occasionally, and detect stuck interrupts.
My 66000 went all the way to MSI-X interrupts as its base.
There are mapping methods to map INTx and INTk into MSI-X messages.
MSI->MSI-X mapping is "just zero extend the message".
The Interrupt port lives in the memory controller and takes an
aperture of addresses and maps them to various interrupt tables.
There is a broadcast mechanism that notifies cores of new higher
priority level interrupts pending, and a protocol for getting
an interrupt and notifying the IC that it will be handled by a
core.
Along with the 32-bit message, the architecture provides for
6-bits of priority, 2-bits of SW-stack indexing, and an 8-bit
ISR handler index number.
Interrupt tables remain active even when the virtual machine
it inactive. When next scheduled, priority ISRs will be serviced
prior to low priority thread instructions.
{And some things I can't talk about}
> Setting up an MSI interrupt protocol for Q+.
>
> The controller will sit on the CPU's main response collect bus.
> Interrupt messages contain the interrupter's bus/device/function + a
> cause code. They may also contain a target core number.
>
> Planning on distributing the interrupt control to peripheral devices
> instead of having a system level interrupt controller.
[toc] | [prev] | [next] | [standalone]
| From | Robert Finch <robfi680@gmail.com> |
|---|---|
| Date | 2025-03-13 13:49 -0400 |
| Message-ID | <vqv5rj$3kaa7$1@dont-email.me> |
| In reply to | #111136 |
On 2025-03-13 12:34 p.m., MitchAlsup1 wrote:
> On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote:
>
>> Planning on having the Q+ CPU respond directly to MSI interrupts. My
>> thought is to integrate an interrupt controller directly into the CPU
>> which will have several queues to prioritize interrupts. The controller
>> will take care of detecting the highest priority interrupt to process,
>> performing priority inversion occasionally, and detect stuck interrupts.
>
> My 66000 went all the way to MSI-X interrupts as its base.
>
> There are mapping methods to map INTx and INTk into MSI-X messages.
> MSI->MSI-X mapping is "just zero extend the message".
MSI interrupts are the way to go when there are transistors available.
Just having discovered them not to long ago, I still think they are great.
I have the luxury of not needing to support legacy interrupt systems
(yet), all the peripheral cores needing interrupts will have to support
MSI responses.
>
> The Interrupt port lives in the memory controller and takes an
> aperture of addresses and maps them to various interrupt tables.
>
I have read that MSI devices target particular I/O addresses to send the
messages to. Not sure what having a dedicated I/O addresses buys. In my
architecture, rather than target an I/O address a specific processing
core number is specified.
> There is a broadcast mechanism that notifies cores of new higher
> priority level interrupts pending, and a protocol for getting
> an interrupt and notifying the IC that it will be handled by a
> core.
>
> Along with the 32-bit message, the architecture provides for
> 6-bits of priority, 2-bits of SW-stack indexing, and an 8-bit
> ISR handler index number.
My architecture goes with a 3-bit priority field, and an 8-bit field to
help resolve the cause of the interrupt. The bus/device/function is also
part of the interrupt response. A 6-bit target core number field is
included in the response too. A target core number of all ones is
treated as a broadcast to all cores. The entire interrupt response
message is available to the CPU so software can decide how to handle the
interrupt. There is some potential for data to be passed back. The
contents of the data field passed back on the bus is unspecified. The
other info is passed back in the address field of the bus.
>
> Interrupt tables remain active even when the virtual machine
> it inactive. When next scheduled, priority ISRs will be serviced
> prior to low priority thread instructions.
>
> {And some things I can't talk about}
Shucks.
>
>> Setting up an MSI interrupt protocol for Q+.
>>
>> The controller will sit on the CPU's main response collect bus.
>> Interrupt messages contain the interrupter's bus/device/function + a
>> cause code. They may also contain a target core number.
>>
>> Planning on distributing the interrupt control to peripheral devices
>> instead of having a system level interrupt controller.
I sketched up an interrupt controller in SystemVerilog and it was only
about 700 LUTs although it does use seven BRAMs for priority queues.
The priority queues can hold hundreds of interrupt messages which is
likely far more than would be needed. So if a queue overflows it would
indicate an issue with the interrupt system. The controller also keeps a
message history with timestamps. If the same message is present at the
start and end of the history along with a short difference in the
timestamp, it is considered a stuck interrupt.
The controller inverts interrupt priorities for one cycle every 64 clock
cycles. The priority inversion is with respect to which queue is read
from. The idea is to try and prevent a low-priority task that is waiting
on an interrupt from being permanently disabled.
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-13 18:42 +0000 |
| Message-ID | <pGFAP.729660$eNx6.592144@fx14.iad> |
| In reply to | #111138 |
Robert Finch <robfi680@gmail.com> writes:
>On 2025-03-13 12:34 p.m., MitchAlsup1 wrote:
>> On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote:
>>
>>> Planning on having the Q+ CPU respond directly to MSI interrupts. My
>>> thought is to integrate an interrupt controller directly into the CPU
>>> which will have several queues to prioritize interrupts. The controller
>>> will take care of detecting the highest priority interrupt to process,
>>> performing priority inversion occasionally, and detect stuck interrupts.
>>
>> My 66000 went all the way to MSI-X interrupts as its base.
>>
>> There are mapping methods to map INTx and INTk into MSI-X messages.
>> MSI->MSI-X mapping is "just zero extend the message".
>
>MSI interrupts are the way to go when there are transistors available.
>Just having discovered them not to long ago, I still think they are great.
>I have the luxury of not needing to support legacy interrupt systems
>(yet), all the peripheral cores needing interrupts will have to support
>MSI responses.
MSI has been deprecated for years. MSI-X is preferred and used by
most modern devices (and supported by major operating software).
>>
>> The Interrupt port lives in the memory controller and takes an
>> aperture of addresses and maps them to various interrupt tables.
>>
>I have read that MSI devices target particular I/O addresses to send the
>messages to. Not sure what having a dedicated I/O addresses buys. In my
>architecture, rather than target an I/O address a specific processing
>core number is specified.
There are many reasons that MSI-X supports vector specific target addresses.
1) It is possible to target DRAM (so software can poll for the interrupt)
2) It is possible to support level-sensitive semantics by using two vectors;
one to assert the interrupt and the second to deassert it. Each needs
to target a different interrupt controller register (address).
3) A system may have multiple interrupt controllers for different
security states (e.g. monitor/secure vs. hypervisor vs. direct virtual
to a guest).
>
>My architecture goes with a 3-bit priority field, and an 8-bit field to
>help resolve the cause of the interrupt. The bus/device/function is also
>part of the interrupt response.
That's what ARM64 GIC does, each interrupt is accompanied by a 20+ bit
'device id', which is composed of the segment, bus, dev/func values
for the sending device. Since each PCIe controller can consume the
entire bus space potentially, the segment widens the device id space
to include a large number of devices. The 'device id' is used to
select the in-memory table used to map the interrupt to the target
guest and the target guest vector number.
I think you really want an interrupt controller as an intermediary
between the device and the CPU interrupt signal, allows a great deal
of implementation flexibility (e.g. changing the target CPU in the
interrupt controller tables rather than requiring the device driver
to change the msix-vector address field).
[toc] | [prev] | [next] | [standalone]
| From | Robert Finch <robfi680@gmail.com> |
|---|---|
| Date | 2025-03-13 15:59 -0400 |
| Message-ID | <vqvde9$3qe81$1@dont-email.me> |
| In reply to | #111140 |
On 2025-03-13 2:42 p.m., Scott Lurndal wrote: > Robert Finch <robfi680@gmail.com> writes: >> On 2025-03-13 12:34 p.m., MitchAlsup1 wrote: >>> On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote: >>> >>>> Planning on having the Q+ CPU respond directly to MSI interrupts. My >>>> thought is to integrate an interrupt controller directly into the CPU >>>> which will have several queues to prioritize interrupts. The controller >>>> will take care of detecting the highest priority interrupt to process, >>>> performing priority inversion occasionally, and detect stuck interrupts. >>> >>> My 66000 went all the way to MSI-X interrupts as its base. >>> >>> There are mapping methods to map INTx and INTk into MSI-X messages. >>> MSI->MSI-X mapping is "just zero extend the message". >> >> MSI interrupts are the way to go when there are transistors available. >> Just having discovered them not to long ago, I still think they are great. >> I have the luxury of not needing to support legacy interrupt systems >> (yet), all the peripheral cores needing interrupts will have to support >> MSI responses. > > MSI has been deprecated for years. MSI-X is preferred and used by > most modern devices (and supported by major operating software). > >>> >>> The Interrupt port lives in the memory controller and takes an >>> aperture of addresses and maps them to various interrupt tables. >>> >> I have read that MSI devices target particular I/O addresses to send the >> messages to. Not sure what having a dedicated I/O addresses buys. In my >> architecture, rather than target an I/O address a specific processing >> core number is specified. > > There are many reasons that MSI-X supports vector specific target addresses. > 1) It is possible to target DRAM (so software can poll for the interrupt) This can be done using a dummy core in the system that echos interrupts messages back to the DRAM. A current issue with the way interrupts in my system are working, is they always spit out the interrupt on the response bus, which means they cannot act like the master. To drive messages onto the bus as requests instead of responses every peripheral device would need to be able to be a bus master. It would add a lot of logic to the system. Extra cross-bar ports etc. > 2) It is possible to support level-sensitive semantics by using two vectors; > one to assert the interrupt and the second to deassert it. Each needs > to target a different interrupt controller register (address). I think this can be handled with the cause code being returned as part of the response. > 3) A system may have multiple interrupt controllers for different > security states (e.g. monitor/secure vs. hypervisor vs. direct virtual > to a guest). Is this just to get more target vectors for each interrupt? Could there be one interrupt controller with multiple target vectors depending on the security state? Q+ CPU always switches to machine mode to process interrupts. It can then redirect to other operating modes. A security state code could be added to the response so that different vector tables may be used. > > >> >> My architecture goes with a 3-bit priority field, and an 8-bit field to >> help resolve the cause of the interrupt. The bus/device/function is also >> part of the interrupt response. > > That's what ARM64 GIC does, each interrupt is accompanied by a 20+ bit > 'device id', which is composed of the segment, bus, dev/func values > for the sending device. Since each PCIe controller can consume the > entire bus space potentially, the segment widens the device id space > to include a large number of devices. The 'device id' is used to > select the in-memory table used to map the interrupt to the target > guest and the target guest vector number. Seems like going out to memory tables to fetch the targets of an interrupt would be costly, perhaps slow. In the SoC there are already a half dozen devices fighting for access to the DRAM. > > I think you really want an interrupt controller as an intermediary > between the device and the CPU interrupt signal, allows a great deal > of implementation flexibility (e.g. changing the target CPU in the Absolutely. Be crazy not to. There is an irq controller component separate from the CPU. (I decided to move the controller outside of the CPU). > interrupt controller tables rather than requiring the device driver > to change the msix-vector address field). > Updating I/O is faster than updating DRAM tables. I need to study more on the use of interrupt tables.
[toc] | [prev] | [next] | [standalone]
| From | mitchalsup@aol.com (MitchAlsup1) |
|---|---|
| Date | 2025-03-13 20:46 +0000 |
| Message-ID | <2543154ec4cab8f4bc791e3796492e76@www.novabbs.org> |
| In reply to | #111146 |
On Thu, 13 Mar 2025 19:59:02 +0000, Robert Finch wrote:
> On 2025-03-13 2:42 p.m., Scott Lurndal wrote:
>> Robert Finch <robfi680@gmail.com> writes:
>>> On 2025-03-13 12:34 p.m., MitchAlsup1 wrote:
>>>> On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote:
>>>>
>>>>> Planning on having the Q+ CPU respond directly to MSI interrupts. My
>>>>> thought is to integrate an interrupt controller directly into the CPU
>>>>> which will have several queues to prioritize interrupts. The controller
>>>>> will take care of detecting the highest priority interrupt to process,
>>>>> performing priority inversion occasionally, and detect stuck interrupts.
>>>>
>>>> My 66000 went all the way to MSI-X interrupts as its base.
>>>>
>>>> There are mapping methods to map INTx and INTk into MSI-X messages.
>>>> MSI->MSI-X mapping is "just zero extend the message".
>>>
>>> MSI interrupts are the way to go when there are transistors available.
>>> Just having discovered them not to long ago, I still think they are
>>> great.
>>> I have the luxury of not needing to support legacy interrupt systems
>>> (yet), all the peripheral cores needing interrupts will have to support
>>> MSI responses.
>>
>> MSI has been deprecated for years. MSI-X is preferred and used by
>> most modern devices (and supported by major operating software).
>>
>>>>
>>>> The Interrupt port lives in the memory controller and takes an
>>>> aperture of addresses and maps them to various interrupt tables.
>>>>
>>> I have read that MSI devices target particular I/O addresses to send the
>>> messages to. Not sure what having a dedicated I/O addresses buys. In my
>>> architecture, rather than target an I/O address a specific processing
>>> core number is specified.
>>
>> There are many reasons that MSI-X supports vector specific target
>> addresses.
>> 1) It is possible to target DRAM (so software can poll for the
>> interrupt)
>
> This can be done using a dummy core in the system that echos interrupts
> messages back to the DRAM. A current issue with the way interrupts in my
> system are working, is they always spit out the interrupt on the
> response bus, which means they cannot act like the master.
>
> To drive messages onto the bus as requests instead of responses every
> peripheral device would need to be able to be a bus master. It would add
> a lot of logic to the system. Extra cross-bar ports etc.
>
>
>> 2) It is possible to support level-sensitive semantics by using two
>> vectors;
>> one to assert the interrupt and the second to deassert it. Each
>> needs
>> to target a different interrupt controller register (address).
>
> I think this can be handled with the cause code being returned as part
> of the response.
>
>> 3) A system may have multiple interrupt controllers for different
>> security states (e.g. monitor/secure vs. hypervisor vs. direct
>> virtual
>> to a guest).
>
> Is this just to get more target vectors for each interrupt? Could there
> be one interrupt controller with multiple target vectors depending on
> the security state?
Take a AHCI STAT drive, that has multiple GuestOSs and several
HyperVisors using various functions therein.
While ALL ISRs from this device can use the same code, they must
not be in a position where any state is shared across system-
states. It is often easier to not share the code, even though it
is exactly the same set of bit patterns because where the "other
state" resides is necessarily different in GuestOS as it is in
HyperVisor[k] as it is in Secure Monitor[j].
> Q+ CPU always switches to machine mode to process interrupts. It can
> then redirect to other operating modes. A security state code could be
> added to the response so that different vector tables may be used.
Interrupts in My 66000 can be directed at any interrupt table, at
any privilege level, at any priority. So, in practice, one could
allow for a user to handle his own SATA interrupts without ever
taking an excursion through any GuestOS or Hypervisor code/state;
nor is it architecturally necessary to disable interrupts or
exceptions.
>>
>>
>>>
>>> My architecture goes with a 3-bit priority field, and an 8-bit field to
>>> help resolve the cause of the interrupt. The bus/device/function is also
>>> part of the interrupt response.
>>
>> That's what ARM64 GIC does, each interrupt is accompanied by a 20+ bit
>> 'device id', which is composed of the segment, bus, dev/func values
>> for the sending device. Since each PCIe controller can consume the
>> entire bus space potentially, the segment widens the device id space
>> to include a large number of devices. The 'device id' is used to
>> select the in-memory table used to map the interrupt to the target
>> guest and the target guest vector number.
>
> Seems like going out to memory tables to fetch the targets of an
> interrupt would be costly, perhaps slow. In the SoC there are already a
> half dozen devices fighting for access to the DRAM.
Cache it and pay the cost once (or once per context switch).
It is not like the ISR for the SATA drive for GuestOS[7]
changes very often !! And it is (I S) used often.
>> I think you really want an interrupt controller as an intermediary
>> between the device and the CPU interrupt signal, allows a great deal
>> of implementation flexibility (e.g. changing the target CPU in the
Wrong abstraction::
You want an interrupt service device that services interrupts
(logs them into tables) even when no core (Virtual Machine or
Virtual Machine Monitor) knows anything about said interrupt
table. That is an interrupt to a table is serviced even when
there is no recipient at this instant in time. This prevents
loss of interrupts the table is supposed to accumulate.
Only when at least 1 core is processing VM[k] or VMM[j]
associated with that interrupt table are cores running
VM[k] or VMM[j] concerned with any random interrupt from
that table.
----------------
Connecting the missing dots can be done several ways, My
66000 does::
My 66000 has a service port (in the memory controller) which
services the MMI/O requests to the interrupt aperture. I (currently)
call this the interrupt service port (rather than interrupt
controller) because it just services messages to the interrupt
aperture with almost no regard to any consumer. Said consumer
regard is via a broadcast MMI/O message using the DRAM address
of the interrupt table with a 64-bit message of raised interrupts.
Should the Interrupt table CR match broadcast address, core
captures the raised vector...and decided to deal with interrupt
or not.
A core can IPI a VM or VMM by performing a MMI/O write using
the same bit patterns the I/O device would have used. Since
all interrupts carry privilege (SW stack index) and priority
along with 8-bit ISR and 32-bit message, SW has a "big Enough"
interrupt name-space to do anything it likes.
Finally, there is interrupt negotiation where a service port
on the core sees the interrupt table broadcast and decides
to try-to process said interrupt.
Thus, the Interrupt Service Philosophy
a) is not close to any core
b) cores are notified by MMI/O broadcast
c) cores continue to run code while negotiating for interrupts
d) HW prefetches thread state and RF before interrupt acceptance
e) anybody with VAS access can perform IPIs via MMU-PTEs
Thus, the local Pic is a 6-rail-road-track state machine,
with a single DW of state (&interrupt_table).
> Absolutely. Be crazy not to. There is an irq controller component
> separate from the CPU. (I decided to move the controller outside of the
> CPU).
You need 2 concepts:
a) the deposition of interrupts in the table
b) the delivery of interrupts to consuming (cores)
(a) must happen even when no core is looking at the table.
(b) can happen by having SW perform read requests to interrupt
aperture or by the normal HW broadcast mechanism.
>> interrupt controller tables rather than requiring the device driver
>> to change the msix-vector address field).
The MSI-X address is (nominally) set once just after GuestOS device
discovery. And becomes a virtual constant pointer afterwards.
> Updating I/O is faster than updating DRAM tables.
Expand:: is this the distinction between flip-flops and a RAM
buried inside a pre-defined pin-boundary ??
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-13 21:39 +0000 |
| Message-ID | <LgIAP.61127$Xq5f.20985@fx38.iad> |
| In reply to | #111147 |
mitchalsup@aol.com (MitchAlsup1) writes: >On Thu, 13 Mar 2025 19:59:02 +0000, Robert Finch wrote: > >> >> Is this just to get more target vectors for each interrupt? Could there >> be one interrupt controller with multiple target vectors depending on >> the security state? > >Take a AHCI STAT drive, that has multiple GuestOSs and several >HyperVisors using various functions therein. Well, use NVMe as an example. AHCI only supports one function and cannot be virtualized. One can, however, have one AHCI controller per guest, but that would be both expensive and quite limiting such that a single guest would always need to use the same AHCI controller due to the actual disk connectivity. NVMe does support lots of guest OS accessing the device and apportions resources based on the control function (physical function 0, allocates virtual functions to the various guests with appropriate configuration to restrict a guest to a particular virtual SSD). >>> I think you really want an interrupt controller as an intermediary >>> between the device and the CPU interrupt signal, allows a great deal >>> of implementation flexibility (e.g. changing the target CPU in the > >Wrong abstraction:: > >You want an interrupt service device that services interrupts >(logs them into tables) even when no core (Virtual Machine or >Virtual Machine Monitor) knows anything about said interrupt >table. That is an interrupt to a table is serviced even when >there is no recipient at this instant in time. This prevents >loss of interrupts the table is supposed to accumulate. That's exactly what an interrupt controller does. It routes interrupts to active guests and records pending interrupts for non-active guests (and interrupts to the hypervisor to schedule the guest). For recording pending, there are tables in DRAM for each guest describing the characteristics and routing (i.e. to which physical CPU(s) the guest is running on) for each interrupt. Give a 32 (or cut-down 24) bit interrupt space, that's definitely stored in DRAM, not flops or on-chip SRAM. > >Only when at least 1 core is processing VM[k] or VMM[j] >associated with that interrupt table are cores running >VM[k] or VMM[j] concerned with any random interrupt from >that table. >---------------- >Connecting the missing dots can be done several ways, My >66000 does:: > >My 66000 has a service port (in the memory controller) which >services the MMI/O requests to the interrupt aperture. I (currently) >call this the interrupt service port (rather than interrupt >controller) because it just services messages to the interrupt >aperture with almost no regard to any consumer. Said consumer >regard is via a broadcast MMI/O message using the DRAM address >of the interrupt table with a 64-bit message of raised interrupts. And if a device has 1024 interrupt vectors per function? It's not uncommon. > >>> interrupt controller tables rather than requiring the device driver >>> to change the msix-vector address field). > >The MSI-X address is (nominally) set once just after GuestOS device >discovery. And becomes a virtual constant pointer afterwards. > >> Updating I/O is faster than updating DRAM tables. > >Expand:: is this the distinction between flip-flops and a RAM >buried inside a pre-defined pin-boundary ?? The tables should be in the main DRAM, accessible to the operating system (actually the operating system (or hypervisor) programs the base address of the tables into the interrupt controller). The cost for on-chip SRAM or worse, flops, to track all this state is both area and cost prohibitive.
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2025-03-13 17:00 -0400 |
| Message-ID | <jwvecz01wba.fsf-monnier+comp.arch@gnu.org> |
| In reply to | #111146 |
> Updating I/O is faster than updating DRAM tables.
What do you mean by DRAM tables?
You mean (IO)MMU-style pagetables, or the tables mapping
physical addresses to specific DRAM modules, or ...?
Stefan
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-13 23:27 +0000 |
| Message-ID | <KRJAP.403745$zz8b.344993@fx09.iad> |
| In reply to | #111149 |
Stefan Monnier <monnier@iro.umontreal.ca> writes: >> Updating I/O is faster than updating DRAM tables. > >What do you mean by DRAM tables? Please don't snip attributions.
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-13 21:27 +0000 |
| Message-ID | <d5IAP.61126$Xq5f.51927@fx38.iad> |
| In reply to | #111146 |
Robert Finch <robfi680@gmail.com> writes: >On 2025-03-13 2:42 p.m., Scott Lurndal wrote: >> Robert Finch <robfi680@gmail.com> writes: >>> On 2025-03-13 12:34 p.m., MitchAlsup1 wrote: >>>> On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote: >>>> >>>>> Planning on having the Q+ CPU respond directly to MSI interrupts. My >>>>> thought is to integrate an interrupt controller directly into the CPU >>>>> which will have several queues to prioritize interrupts. The controller >>>>> will take care of detecting the highest priority interrupt to process, >>>>> performing priority inversion occasionally, and detect stuck interrupts. >>>> >>>> My 66000 went all the way to MSI-X interrupts as its base. >>>> >>>> There are mapping methods to map INTx and INTk into MSI-X messages. >>>> MSI->MSI-X mapping is "just zero extend the message". >>> >>> MSI interrupts are the way to go when there are transistors available. >>> Just having discovered them not to long ago, I still think they are great. >>> I have the luxury of not needing to support legacy interrupt systems >>> (yet), all the peripheral cores needing interrupts will have to support >>> MSI responses. >> >> MSI has been deprecated for years. MSI-X is preferred and used by >> most modern devices (and supported by major operating software). >> >>>> >>>> The Interrupt port lives in the memory controller and takes an >>>> aperture of addresses and maps them to various interrupt tables. >>>> >>> I have read that MSI devices target particular I/O addresses to send the >>> messages to. Not sure what having a dedicated I/O addresses buys. In my >>> architecture, rather than target an I/O address a specific processing >>> core number is specified. >> >> There are many reasons that MSI-X supports vector specific target addresses. >> 1) It is possible to target DRAM (so software can poll for the interrupt) > >This can be done using a dummy core in the system that echos interrupts >messages back to the DRAM. A current issue with the way interrupts in my >system are working, is they always spit out the interrupt on the >response bus, which means they cannot act like the master. > >To drive messages onto the bus as requests instead of responses every >peripheral device would need to be able to be a bus master. It would add >a lot of logic to the system. Extra cross-bar ports etc. > By its very nature, a device that generates an MSI-X message _must_ have bus mastering capability (and in PCI/PCIe, the Master Enable bit needs to be set in the COMMAND register to allow the message to be transmitted to the root port). If one uses (and exposes to the OS) MSI-X, then all the semantics need to be followed, including the bus master enable bit (and the enable and mask bits in the MSI-X capability). Software will expect it. > >> 2) It is possible to support level-sensitive semantics by using two vectors; >> one to assert the interrupt and the second to deassert it. Each needs >> to target a different interrupt controller register (address). > >I think this can be handled with the cause code being returned as part >of the response. > >> 3) A system may have multiple interrupt controllers for different >> security states (e.g. monitor/secure vs. hypervisor vs. direct virtual >> to a guest). > >Is this just to get more target vectors for each interrupt? Could there >be one interrupt controller with multiple target vectors depending on >the security state? Not really. The data payload allows specifying 2^32 unique interrupt numbers. Consider a system that has two security domains - one for firmware and perhaps video decoders that can interface directly with the graphics controller and the other to run standard operating software. Both need interrupts, but they need to be completely separate; the non-secure side and secure side of the interrupt controller are mostly independent (other than when looking at interrupt priority; half the priority range is reserved for the secure space, the other half for the non-secure space). This is how ARM's aarch64 works. In the ARM case, a wired, per-processor or inter-processor interrupt can be assigned to a security group by the secure software, such that secure interrupts would be delivered to the secure monitor and non-secure would be delivered to the hypervisor or bare-metal OS. MSI-X interrupts are always non-secure. >> That's what ARM64 GIC does, each interrupt is accompanied by a 20+ bit >> 'device id', which is composed of the segment, bus, dev/func values >> for the sending device. Since each PCIe controller can consume the >> entire bus space potentially, the segment widens the device id space >> to include a large number of devices. The 'device id' is used to >> select the in-memory table used to map the interrupt to the target >> guest and the target guest vector number. > >Seems like going out to memory tables to fetch the targets of an >interrupt would be costly, perhaps slow. In the SoC there are already a >half dozen devices fighting for access to the DRAM. Cache. The ARM GIC caches the memory tables (and provides the appropriate flush and invalidate operations for configuration changes). When you need to support an interrupt space 24-bits wide, you can't do it with wires and flops. > >> >> I think you really want an interrupt controller as an intermediary >> between the device and the CPU interrupt signal, allows a great deal >> of implementation flexibility (e.g. changing the target CPU in the > >Absolutely. Be crazy not to. There is an irq controller component >separate from the CPU. (I decided to move the controller outside of the >CPU). > >> interrupt controller tables rather than requiring the device driver >> to change the msix-vector address field). >> >Updating I/O is faster than updating DRAM tables. > >I need to study more on the use of interrupt tables. They don't change often; generally only when a new guest is created or an existing guest is destroyed. The ARM GIC also stores the interrupt priority in memory for the 2^24 interrupt (LPI) space; one byte per supported interrupt (upper limit programmable by the OS). A single enable bit and 7 priority bit per interrupt, indexed by the interrupt number. These properties are also cached in the GIC, so for a network device running at full speed at 200GB, they won't need to go to memory for every interrupt (although a network controller at that speed will use interrupt moderation and receive side scaling techniques to improve throughput and reduce interrupt overhead).
[toc] | [prev] | [next] | [standalone]
| From | mitchalsup@aol.com (MitchAlsup1) |
|---|---|
| Date | 2025-03-13 19:12 +0000 |
| Message-ID | <c953aaddb5c22bc15f9af786add41a15@www.novabbs.org> |
| In reply to | #111138 |
On Thu, 13 Mar 2025 17:49:35 +0000, Robert Finch wrote:
> On 2025-03-13 12:34 p.m., MitchAlsup1 wrote:
>> On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote:
>>
>>> Planning on having the Q+ CPU respond directly to MSI interrupts. My
>>> thought is to integrate an interrupt controller directly into the CPU
>>> which will have several queues to prioritize interrupts. The controller
>>> will take care of detecting the highest priority interrupt to process,
>>> performing priority inversion occasionally, and detect stuck interrupts.
>>
>> My 66000 went all the way to MSI-X interrupts as its base.
>>
>> There are mapping methods to map INTx and INTk into MSI-X messages.
>> MSI->MSI-X mapping is "just zero extend the message".
>
> MSI interrupts are the way to go when there are transistors available.
> Just having discovered them not to long ago, I still think they are
> great.
> I have the luxury of not needing to support legacy interrupt systems
> (yet), all the peripheral cores needing interrupts will have to support
> MSI responses.
>>
>> The Interrupt port lives in the memory controller and takes an
>> aperture of addresses and maps them to various interrupt tables.
>>
> I have read that MSI devices target particular I/O addresses to send the
> messages to. Not sure what having a dedicated I/O addresses buys. In my
> architecture, rather than target an I/O address a specific processing
> core number is specified.
MXI and MSI-X target an MMI/O address aperture (i.e., range), the
position and the meaning of the address bits and the message bits
are system dependent.
It is expected that some address bits will indicate what interrupt
table to access.
It is expected that either (or both) bits from the address and message
will indicate priority of the interrupt.
In a classical Operating System, each GuestOS gets its own interrupt
table, and any core running that GuestOS will be watching that inter-
rupt table. Hypervisors get their own interrupt tables, as do Secure
monitors. {{To hand waving accuracy}}
>
>> There is a broadcast mechanism that notifies cores of new higher
>> priority level interrupts pending, and a protocol for getting
>> an interrupt and notifying the IC that it will be handled by a
>> core.
>>
>> Along with the 32-bit message, the architecture provides for
>> 6-bits of priority, 2-bits of SW-stack indexing, and an 8-bit
>> ISR handler index number.
>
> My architecture goes with a 3-bit priority field, and an 8-bit field to
> help resolve the cause of the interrupt.
Having an advocate of such a position (EricP BTW) that 8-priority
levels are sufficient; made me give great through to this space.
I ended up with 64 priority levels, mainly, because I know of no
general purpose way of comparing priority levels found in JoeOS1
versus those in JoeOS2 or JoeOS3. 8-levles is arguably sufficient
for JoeOS[k]. But what if one wants some of JoeOS2's priorities
lower than all JoeOS1's priorities, but the others above them ?!?
So, 64-levels, occupying 1-bit each fit exactly the typical HW
storage container (64-bits) allows for 8-JoeOSs to have their
8-levels each to be fit into one concept we call system-priority
without any constraint on any JoeOS-chosen at random.
> The bus/device/function is also
> part of the interrupt response. A 6-bit target core number field is
> included in the response too. A target core number of all ones is
> treated as a broadcast to all cores. The entire interrupt response
> message is available to the CPU so software can decide how to handle the
> interrupt. There is some potential for data to be passed back. The
> contents of the data field passed back on the bus is
PCIe
> unspecified. The
> other info is passed back in the address field of the bus.
>>
>> Interrupt tables remain active even when the virtual machine
>> it inactive. When next scheduled, priority ISRs will be serviced
>> prior to low priority thread instructions.
>>
>> {And some things I can't talk about}
> Shucks.
>>
>>> Setting up an MSI interrupt protocol for Q+.
>>>
>>> The controller will sit on the CPU's main response collect bus.
>>> Interrupt messages contain the interrupter's bus/device/function + a
>>> cause code. They may also contain a target core number.
>>>
>>> Planning on distributing the interrupt control to peripheral devices
>>> instead of having a system level interrupt controller.
>
> I sketched up an interrupt controller in SystemVerilog and it was only
> about 700 LUTs although it does use seven BRAMs for priority queues.
> The priority queues can hold hundreds of interrupt messages which is
> likely far more than would be needed. So if a queue overflows it would
> indicate an issue with the interrupt system. The controller also keeps a
> message history with timestamps. If the same message is present at the
> start and end of the history along with a short difference in the
> timestamp, it is considered a stuck interrupt.
>
> The controller inverts interrupt priorities for one cycle every 64 clock
> cycles. The priority inversion is with respect to which queue is read
> from. The idea is to try and prevent a low-priority task that is waiting
> on an interrupt from being permanently disabled.
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-13 18:34 +0000 |
| Message-ID | <YyFAP.729659$eNx6.235106@fx14.iad> |
| In reply to | #111136 |
mitchalsup@aol.com (MitchAlsup1) writes: >On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote: > >> Planning on having the Q+ CPU respond directly to MSI interrupts. My >> thought is to integrate an interrupt controller directly into the CPU >> which will have several queues to prioritize interrupts. The controller >> will take care of detecting the highest priority interrupt to process, >> performing priority inversion occasionally, and detect stuck interrupts. > >My 66000 went all the way to MSI-X interrupts as its base. > >There are mapping methods to map INTx and INTk into MSI-X messages. >MSI->MSI-X mapping is "just zero extend the message". In both cases, the a 32-bit data payload (the MSIX Vector #) is written to the MSI-X target address (typically the interrupt controller). The difference between them is MSI MSI-X Supports 1 to 32 Supports an unlimited number of _contiguous_ data vectors, each with an unique non-contiguous payloads (vector vector index (data payload) and target index). address. With MSI, only the base interrupt number (data payload) is defined and only one interrupt controller target address is use used for all vectors. So vector 0 will send the base number, vector 1 will send the base number plus one, etc. With MSI-X each vector has an assigned data payload (vector number) which do not need to be contiguous or monotonically increasing. With MSI-X, each interrupt can have a different target address (i.e. perhaps a secure interrupt controller for some, and a non-secure interrupt controller for others) In the ARM case, there are different interrupt types (SGI, PPI, SPI, LPI) and each one has a different interrupt controller address that the MSI-X vector address should point to. This allows the message-based emulation of level sensitive interrupts by using two vectors, one to assert the interrupt and a second to deassert the interrupt at the interrupt controller. So MSI is useless for modern devices and basically deprecated and cannot be used support level sensitive (e.g. legacy PCI) interrupts. Most modern devices advertise the MSI-X capability instead.
[toc] | [prev] | [next] | [standalone]
| From | mitchalsup@aol.com (MitchAlsup1) |
|---|---|
| Date | 2025-03-13 19:28 +0000 |
| Message-ID | <6731f278e3a9eb70d34250f43b7f15f2@www.novabbs.org> |
| In reply to | #111139 |
On Thu, 13 Mar 2025 18:34:32 +0000, Scott Lurndal wrote:
> mitchalsup@aol.com (MitchAlsup1) writes:
>>On Thu, 13 Mar 2025 4:50:47 +0000, Robert Finch wrote:
>>
>>> Planning on having the Q+ CPU respond directly to MSI interrupts. My
>>> thought is to integrate an interrupt controller directly into the CPU
>>> which will have several queues to prioritize interrupts. The controller
>>> will take care of detecting the highest priority interrupt to process,
>>> performing priority inversion occasionally, and detect stuck interrupts.
>>
>>My 66000 went all the way to MSI-X interrupts as its base.
>>
>>There are mapping methods to map INTx and INTk into MSI-X messages.
>>MSI->MSI-X mapping is "just zero extend the message".
>
> In both cases, the a 32-bit data payload (the MSIX Vector #) is written
> to the MSI-X target address (typically the interrupt controller).
>
> The difference between them is
>
> MSI MSI-X
> Supports 1 to 32 Supports an unlimited number of
> _contiguous_ data vectors, each with an unique non-contiguous
> payloads (vector vector index (data payload) and target
> index). address.
Yes, MSI is a strict and degenerate version of MSI-X
> With MSI, only the base interrupt number (data payload) is
> defined and only one interrupt controller target address is
> use used for all vectors. So vector 0 will send the base number, vector
> 1 will send the base number plus one, etc.
>
> With MSI-X each vector has an assigned data payload (vector
> number) which do not need to be contiguous or monotonically
> increasing. With MSI-X, each interrupt can have a different
> target address (i.e. perhaps a secure interrupt controller
> for some, and a non-secure interrupt controller for others)
>
> In the ARM case, there are different interrupt types (SGI,
> PPI, SPI, LPI) and each one has a different interrupt controller
> address that the MSI-X vector address should point to.
In My 66000 case, there is an 8-bit field from the Originating-
Software-Stack (which I/O MMU uses in translation) that prioritize
DMA*, interrupt delivery, page fault delivery, and device error
delivery.
(*) DRAM access in My 66000 uses the priority of the core at the
instant of AGEN to allow higher priority cache miss requests to
percolate through the interconnect faster, and to allow a higher
priority ATOMIC event to actively interfere with a lower priority
ATOMIC event, but not the reverse. The interconnect uses it more
like a virtual channel than a priority.
As to that 8-bit field:: when an interrupt of higher priority than
a core is currently running arrives, core captures the interrupt.
The interrupt message (now 42-bits in size) sets the priority of
the software-stack (2-bits). SW then extracts the 8-bit field
and uses it to dispatch to an interrupt-class service handler
{ISR, Timer, ECC-fail, SVC, SVR, SATA, NVMe, keyboard, mouse, ...}.
At the instant control arrives, the dispatcher is already re-entrant
{able to take a higher priority event even of itself} running at its
priority, under its Root Pointers, and ASIDs, with its pending
exceptions, and register file, with registers R0..4 filled with
exception data union{Instruction, Operand 1..3},...}.
> This
> allows the message-based emulation of level sensitive interrupts
> by using two vectors, one to assert the interrupt and a second
> to deassert the interrupt at the interrupt controller.
I will have to borrow a trick such as this.
> So MSI is useless for modern devices and basically deprecated
> and cannot be used support level sensitive (e.g. legacy PCI)
> interrupts.
That was my view once I ran into MSI-X:: MSI-X is a complete and
more flexible superset of MSI.
>
> Most modern devices advertise the MSI-X capability instead.
And why not:: its just a few more flip-flops and almost no more
sequencing logic.
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-13 21:14 +0000 |
| Message-ID | <AUHAP.61125$Xq5f.14972@fx38.iad> |
| In reply to | #111143 |
mitchalsup@aol.com (MitchAlsup1) writes: >On Thu, 13 Mar 2025 18:34:32 +0000, Scott Lurndal wrote: > >> Most modern devices advertise the MSI-X capability instead. > >And why not:: its just a few more flip-flops and almost no more >sequencing logic. I've seen several devices with more than 200 MSI-X vectors; thats 96 bits per vector to store a full 64-bit address and 32-bit data payload. Keep in mind that a guest operating system may be writing the 64-bit address field in a virtual function assigned to it with guest virtual addresses, so the inbound path for MSI-X needs to traverse the IOMMU before hitting the interrupt controller logic. The guest OS also specifies the data payload, which may be an IRQ number on an intel system, or a virtual IRQ number translated by the IOMMU or Interrupt controller into a physical IRQ number (allowing multiple guest OS to use the same IRQs they would use on real hardware).
[toc] | [prev] | [next] | [standalone]
| From | mitchalsup@aol.com (MitchAlsup1) |
|---|---|
| Date | 2025-03-13 21:34 +0000 |
| Message-ID | <748c0cc0ba18704b4678fd553193573e@www.novabbs.org> |
| In reply to | #111152 |
On Thu, 13 Mar 2025 21:14:08 +0000, Scott Lurndal wrote: > mitchalsup@aol.com (MitchAlsup1) writes: >>On Thu, 13 Mar 2025 18:34:32 +0000, Scott Lurndal wrote: >> > >>> Most modern devices advertise the MSI-X capability instead. >> >>And why not:: its just a few more flip-flops and almost no more >>sequencing logic. > > I've seen several devices with more than 200 MSI-X vectors; > thats 96 bits per vector to store a full 64-bit address and > 32-bit data payload. At this point, with 200+ entries, flip-flops are not recommended, instead these would be placed in a RAM of some sort. Since RAMs come in 1KB and 2KB quanta; we have 1 of 2K and 1 of 1K and we have 256 said message containers, with 1 cycle access (after you get to that corner of some chip). > Keep in mind that a guest operating system may be writing > the 64-bit address field in a virtual function assigned to > it with guest virtual addresses, so the inbound path for > MSI-X needs to traverse the IOMMU before hitting the > interrupt controller logic. An important concept that is hard to find in the PCIe specifications. > The guest OS also specifies > the data payload, which may be an IRQ number on an intel > system, or a virtual IRQ number translated by the IOMMU > or Interrupt controller into a physical IRQ number (allowing > multiple guest OS to use the same IRQs they would use on real > hardware). Just note that GuestOS[k].IRQ[j] uses the same bit patterns at the device, they get delivered to their own unique GuestOS through its interrupt table. But even when GuestOS[k] and GuestOS[j] use the same 96-bit MSI-X pattern, each GuestOS can interpret its message the way it wants, sort its drivers in the way it wants, and operate with blissful ignorance of GuestOS[other].
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-13 23:34 +0000 |
| Message-ID | <2YJAP.403746$zz8b.238811@fx09.iad> |
| In reply to | #111154 |
mitchalsup@aol.com (MitchAlsup1) writes: >On Thu, 13 Mar 2025 21:14:08 +0000, Scott Lurndal wrote: > >> mitchalsup@aol.com (MitchAlsup1) writes: >>>On Thu, 13 Mar 2025 18:34:32 +0000, Scott Lurndal wrote: >>> >> >>>> Most modern devices advertise the MSI-X capability instead. >>> >>>And why not:: its just a few more flip-flops and almost no more >>>sequencing logic. >> >> I've seen several devices with more than 200 MSI-X vectors; >> thats 96 bits per vector to store a full 64-bit address and >> 32-bit data payload. > >At this point, with 200+ entries, flip-flops are not recommended, >instead these would be placed in a RAM of some sort. Since RAMs >come in 1KB and 2KB quanta; we have 1 of 2K and 1 of 1K and we >have 256 said message containers, with 1 cycle access (after >you get to that corner of some chip). That is 200 _per function_. Consider a physical function that supports the SRIOV capability and configures 2048 virtual functions. So that's 2049 * 200 MSI-X vectors just for one device. It's not unusual. These vectors are, of course, stored on the device itself. Mostly in RAMs. > >> Keep in mind that a guest operating system may be writing >> the 64-bit address field in a virtual function assigned to >> it with guest virtual addresses, so the inbound path for >> MSI-X needs to traverse the IOMMU before hitting the >> interrupt controller logic. > >An important concept that is hard to find in the PCIe specifications. It's more of a system-level thing than a PCI thing, so I wouldn't expect to find it the PCI specification. System level standards (like ARM's SBSA (server base system architecture) covers these type of topics). > >> The guest OS also specifies >> the data payload, which may be an IRQ number on an intel >> system, or a virtual IRQ number translated by the IOMMU >> or Interrupt controller into a physical IRQ number (allowing >> multiple guest OS to use the same IRQs they would use on real >> hardware). > >Just note that GuestOS[k].IRQ[j] uses the same bit patterns >at the device, they get delivered to their own unique >GuestOS through its interrupt table. > >But even when GuestOS[k] and GuestOS[j] use the same 96-bit MSI-X >pattern, each GuestOS can interpret its message the way it wants, >sort its drivers in the way it wants, and operate with blissful >ignorance of GuestOS[other]. Linux, for example, knows that the format of the data payload on an MSI-X message is an unformatted 32-bit integer. Limiting that or defining a hard format is a non-starter, as the kernel software folks will just point and laugh. They really don't like accomodating new architectures that play fast and loose with the standard behavior. DAMHIKT.
[toc] | [prev] | [next] | [standalone]
| From | mitchalsup@aol.com (MitchAlsup1) |
|---|---|
| Date | 2025-03-13 23:52 +0000 |
| Message-ID | <53b8227eba214e0340cad309241af7b5@www.novabbs.org> |
| In reply to | #111165 |
On Thu, 13 Mar 2025 23:34:22 +0000, Scott Lurndal wrote: > mitchalsup@aol.com (MitchAlsup1) writes: >>On Thu, 13 Mar 2025 21:14:08 +0000, Scott Lurndal wrote: >> >>> mitchalsup@aol.com (MitchAlsup1) writes: >>>>On Thu, 13 Mar 2025 18:34:32 +0000, Scott Lurndal wrote: >>>> >>> >>>>> Most modern devices advertise the MSI-X capability instead. >>>> >>>>And why not:: its just a few more flip-flops and almost no more >>>>sequencing logic. >>> >>> I've seen several devices with more than 200 MSI-X vectors; >>> thats 96 bits per vector to store a full 64-bit address and >>> 32-bit data payload. >> >>At this point, with 200+ entries, flip-flops are not recommended, >>instead these would be placed in a RAM of some sort. Since RAMs >>come in 1KB and 2KB quanta; we have 1 of 2K and 1 of 1K and we >>have 256 said message containers, with 1 cycle access (after >>you get to that corner of some chip). > > That is 200 _per function_. Consider a physical function that > supports the SRIOV capability and configures 2048 virtual > functions. So that's 2049 * 200 MSI-X vectors just for one > device. It's not unusual. These vectors are, of course, > stored on the device itself. Mostly in RAMs. In My 66000, said device can place those value-holding containers in actual DRAM should it want to punt the storage. This adds latency and decreases on-die storage. At that point, the device has an unlimited number of "special things". >> >>> Keep in mind that a guest operating system may be writing >>> the 64-bit address field in a virtual function assigned to >>> it with guest virtual addresses, so the inbound path for >>> MSI-X needs to traverse the IOMMU before hitting the >>> interrupt controller logic. >> >>An important concept that is hard to find in the PCIe specifications. > > It's more of a system-level thing than a PCI thing, so I > wouldn't expect to find it the PCI specification. System > level standards (like ARM's SBSA (server base system architecture) > covers these type of topics). > >> >>> The guest OS also specifies >>> the data payload, which may be an IRQ number on an intel >>> system, or a virtual IRQ number translated by the IOMMU >>> or Interrupt controller into a physical IRQ number (allowing >>> multiple guest OS to use the same IRQs they would use on real >>> hardware). >> >>Just note that GuestOS[k].IRQ[j] uses the same bit patterns >>at the device, they get delivered to their own unique >>GuestOS through its interrupt table. >> >>But even when GuestOS[k] and GuestOS[j] use the same 96-bit MSI-X >>pattern, each GuestOS can interpret its message the way it wants, >>sort its drivers in the way it wants, and operate with blissful >>ignorance of GuestOS[other]. > > Linux, for example, knows that the format of the data payload > on an MSI-X message is an unformatted 32-bit integer. Limiting > that or defining a hard format is a non-starter, as the kernel > software folks will just point and laugh. They really don't > like accomodating new architectures that play fast and loose > with the standard behavior. DAMHIKT. Not al all (laughing Linux people); I make no specification to the system (at the architectural level) of how the 32-bit value is interpreted, nor which range of addresses in MMI/O space map to interrupt tables. Those ae all system choices-- possibly dynamic (ASLR and all that, or moving once an hour...)
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-14 14:52 +0000 |
| Message-ID | <3pXAP.584096$FVcd.26370@fx10.iad> |
| In reply to | #111166 |
mitchalsup@aol.com (MitchAlsup1) writes: >On Thu, 13 Mar 2025 23:34:22 +0000, Scott Lurndal wrote: > >> mitchalsup@aol.com (MitchAlsup1) writes: >>>On Thu, 13 Mar 2025 21:14:08 +0000, Scott Lurndal wrote: >>> >>>> mitchalsup@aol.com (MitchAlsup1) writes: >>>>>On Thu, 13 Mar 2025 18:34:32 +0000, Scott Lurndal wrote: >>>>> >>>> >>>>>> Most modern devices advertise the MSI-X capability instead. >>>>> >>>>>And why not:: its just a few more flip-flops and almost no more >>>>>sequencing logic. >>>> >>>> I've seen several devices with more than 200 MSI-X vectors; >>>> thats 96 bits per vector to store a full 64-bit address and >>>> 32-bit data payload. >>> >>>At this point, with 200+ entries, flip-flops are not recommended, >>>instead these would be placed in a RAM of some sort. Since RAMs >>>come in 1KB and 2KB quanta; we have 1 of 2K and 1 of 1K and we >>>have 256 said message containers, with 1 cycle access (after >>>you get to that corner of some chip). >> >> That is 200 _per function_. Consider a physical function that >> supports the SRIOV capability and configures 2048 virtual >> functions. So that's 2049 * 200 MSI-X vectors just for one >> device. It's not unusual. These vectors are, of course, >> stored on the device itself. Mostly in RAMs. > >In My 66000, said device can place those value-holding containers >in actual DRAM should it want to punt the storage. Any device can do that today if it is designed to do so. But that requires an additional DMA operation to send an MSI-X interrupt; the device must first read the address and data fields from host dram (as configured by the device driver) before building the inbound memory write TLP that gets sent from the device to the root port. Just adds latency and requires the driver to allocate space and tell the device the base address of the array of vectors. This adds latency >and decreases on-die storage. At that point, the device has an >unlimited number of "special things". Most PCIe devices are third party IP, which you must live with. Your on-chip PCIe-like devices can do as they wish, unless they're standard IP such as a Synopsis SATA controller, or third party network controller PCIe endpoint IP. I will just note that the high-end SoCs with on-chip PCIe-like devices that I'm familiar with all use SRAMs for the MSI-X vectors (unless there is only a couple, in which case flops work). The latency overhead of fetching the vector from DRAM is prohibitive for high-speed devices such as network controllers.
[toc] | [prev] | [next] | [standalone]
| From | mitchalsup@aol.com (MitchAlsup1) |
|---|---|
| Date | 2025-03-14 15:34 +0000 |
| Message-ID | <795b541375e3e0f53e2c76a55ffe3f20@www.novabbs.org> |
| In reply to | #111186 |
On Fri, 14 Mar 2025 14:52:47 +0000, Scott Lurndal wrote:
> mitchalsup@aol.com (MitchAlsup1) writes:
>>On Thu, 13 Mar 2025 23:34:22 +0000, Scott Lurndal wrote:
>>
>>> mitchalsup@aol.com (MitchAlsup1) writes:
>>>>On Thu, 13 Mar 2025 21:14:08 +0000, Scott Lurndal wrote:
>>>>
>>>>> mitchalsup@aol.com (MitchAlsup1) writes:
>>>>>>On Thu, 13 Mar 2025 18:34:32 +0000, Scott Lurndal wrote:
>>>>>>
>>>>>
>>>>>>> Most modern devices advertise the MSI-X capability instead.
>>>>>>
>>>>>>And why not:: its just a few more flip-flops and almost no more
>>>>>>sequencing logic.
>>>>>
>>>>> I've seen several devices with more than 200 MSI-X vectors;
>>>>> thats 96 bits per vector to store a full 64-bit address and
>>>>> 32-bit data payload.
>>>>
>>>>At this point, with 200+ entries, flip-flops are not recommended,
>>>>instead these would be placed in a RAM of some sort. Since RAMs
>>>>come in 1KB and 2KB quanta; we have 1 of 2K and 1 of 1K and we
>>>>have 256 said message containers, with 1 cycle access (after
>>>>you get to that corner of some chip).
>>>
>>> That is 200 _per function_. Consider a physical function that
>>> supports the SRIOV capability and configures 2048 virtual
>>> functions. So that's 2049 * 200 MSI-X vectors just for one
>>> device. It's not unusual. These vectors are, of course,
>>> stored on the device itself. Mostly in RAMs.
We CPU guys deal with dozens of cores, each having 2×64KB L1 caches,
a 256KB-1024KB L2 cache, and have that dozen cores share a 16MB L3
cache. This means the chip contains 26,624 1KB SRAM macros.
In the above you are complaining that the I/O device can only afford
a few of these, whereas we CPU guys count then in the thousands
(and approaching 100's of thousands.)
You can forgive me when the device guys shudder at the thought of
a dozen 1K SRAM macros as "expensive".....and I don't see their
plight immediately.
>>
>>In My 66000, said device can place those value-holding containers
>>in actual DRAM should it want to punt the storage.
>
> Any device can do that today if it is designed to do so. But that
> requires an additional DMA operation to send an MSI-X interrupt;
> the device must first read the address and data fields from
> host dram (as configured by the device driver) before building
> the inbound memory write TLP that gets sent from the device to
> the root port.
Was thinking about this last night::
a) device goes up and reads DRAM via L3::MC and DRC
b) DRAM data is delivered to device 15ns later
c) device uses data to send MSI-X message to interrupt 'controller'
d) interrupt controller in L3 sees interrupt
{to hand waving accuracy}
So, we have dozen ns up the PCIe tree, dozen ns over the interconnect,
50ns in DRAM, dozens ns over the interconnect, dozens of ns down the
PCIe tree, 1ns at device, dozen ns up the PCIe tree, dozens across
interconnect, arriving at interrupt service port after 122 ns or
about the equivalent of 600± clocks to log the interrupt into the
table.
The Priority broadcast is going to take another dozen ns, core
request for interrupt will be another dozen to service controller,
even if the service port request is serviced instantaneously,
the MSI-X message does not arrive at core until 72ns after arriving
at service port--for a best case latency on the order of 200 ns
(or 1000 CPU cycles or ~ 2,000 instructions worth of execution.)
And that is under the assumption that no traffic interference
is encountered up or down the PCIe trees.
whereas::
if the device DRAM read request was known to contain an MSI-X
message, then a trip up and down the PCIe tree could be completely
eliminated (in terms of interrupt latency)--we still have the
4 round trip latency of the interconnect between service port
and core.
>
> Just adds latency and requires the driver to allocate space and
> tell the device the base address of the array of vectors.
>
>
> This adds latency
>>and decreases on-die storage. At that point, the device has an
>>unlimited number of "special things".
>
> Most PCIe devices are third party IP, which you must live with.
Captain Obvious strikes again.
> Your on-chip PCIe-like devices can do as they wish, unless they're
> standard IP such as a Synopsis SATA controller, or third party
> network controller PCIe endpoint IP.
>
> I will just note that the high-end SoCs with on-chip PCIe-like
> devices that I'm familiar with all use SRAMs for the MSI-X vectors
> (unless there is only a couple, in which case flops work).
That was my originating assumption that you spent the middle of this
thread dissuading ...
> The latency overhead of fetching the vector from DRAM is
> prohibitive for high-speed devices such as network controllers.
Here we have the situation where one can context switch in a lower
number of clock cycles than one can deliver an interrupt from
a device to a servicing core.
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-03-14 17:35 +0000 |
| Message-ID | <vNZAP.37553$D_V4.18229@fx39.iad> |
| In reply to | #111187 |
mitchalsup@aol.com (MitchAlsup1) writes:
>On Fri, 14 Mar 2025 14:52:47 +0000, Scott Lurndal wrote:
>
>
>We CPU guys deal with dozens of cores, each having 2×64KB L1 caches,
>a 256KB-1024KB L2 cache, and have that dozen cores share a 16MB L3
>cache. This means the chip contains 26,624 1KB SRAM macros.
You've a lot more area to work with, and generally a more
recent process node.
>
>Was thinking about this last night::
>a) device goes up and reads DRAM via L3::MC and DRC
>b) DRAM data is delivered to device 15ns later
15ns? That's optimistic and presumes a cache hit, right?
Don't forget to factor in PCIe latency (bus to RC and RC to endpoint).
>c) device uses data to send MSI-X message to interrupt 'controller'
>d) interrupt controller in L3 sees interrupt
>
>{to hand waving accuracy}
>So, we have dozen ns up the PCIe tree, dozen ns over the interconnect,
>50ns in DRAM, dozens ns over the interconnect, dozens of ns down the
>PCIe tree, 1ns at device, dozen ns up the PCIe tree, dozens across
>interconnect, arriving at interrupt service port after 122 ns or
>about the equivalent of 600± clocks to log the interrupt into the
>table.
>
>The Priority broadcast is going to take another dozen ns, core
>request for interrupt will be another dozen to service controller,
>even if the service port request is serviced instantaneously,
>the MSI-X message does not arrive at core until 72ns after arriving
>at service port--for a best case latency on the order of 200 ns
>(or 1000 CPU cycles or ~ 2,000 instructions worth of execution.)
>
>And that is under the assumption that no traffic interference
>is encountered up or down the PCIe trees.
>
>whereas::
>
>if the device DRAM read request was known to contain an MSI-X
>message,
You can't know that a priori, it's just another memory write
(or read if you need to fetch the address and data from DRAM)
TLP as part of the inbound DMA. Which needs to hit the IOMMU
first to translate the PCI memory space address to the host
physical address space address.
If the MSI-X tables were kept in DRAM, you also need to include
the IOMMU translation latency in the inbound path that fetches
the vector address and vector data (96-bits, so that's two
round trips from the device to memory). For a virtual function,
the MSI-X table is owned and managed by the guest, and all
transaction addresses from the device must be translated from
guest physical addresses to host physical addresses.
A miss in the IOMMU adds a _lot_ of latency to the request.
So, that's three round trips from the device to the
Uncore/RoC just to send a single interrupt from the device.
>> The latency overhead of fetching the vector from DRAM is
>> prohibitive for high-speed devices such as network controllers.
>
>Here we have the situation where one can context switch in a lower
>number of clock cycles than one can deliver an interrupt from
>a device to a servicing core.
Device needs to send an interrupt when vectors stored in host DRAM
instead of internal SRAM or flops:
- send non-posted MRD TLP to RC to fetch MSI-X address
- receiver (pcie controller (RC), for example) passes
MRD address to IOMMU for translation (assuming
the device and host don't implement ATS),
IOMMU translates (table walk latency) the
address from the TLP to a host physical
address (which could involve two levels of
translation, so up to 22 DRAM accesses (intel/amd/aarch64)
on IOMMU TLB miss). The latency is dependent
up on the IOMMU table format - Intel has EPT
while ARM and AMD use the same format as the CPU
page tables for the IOMMU tables.
(this leaves out any further latency hit when
using the PCI Page Request Interface (PRI) to make
the target page resident).
- LLC/DRAM satisfies the MRD and returns data to
PCIe controller, which sends a completion TLP
to device. LLC (minimum), DRAM (maximum) latency added.
- RC/host sends response with address to device
- Device sends non-posted MRD TLP to RC to fetch MSI-X Data
(32-bit). Again with the IOMMU, but will likely
hit TLB. Lesser latency than a miss, but nonzero.
- RC returns completion TLP to device.
- Device sends MWR TLP (data payload) with the translated
address to the root complex, which passes it to the
internal bus structure for routing to the final address
(interrupt controller).
- Then add the latency from the interrupt controller to
the target core (which may include making the target guest
resident).
That's a whole pile of latency to send an interrupt.
[toc] | [prev] | [next] | [standalone]
Page 1 of 11 [1] 2 3 … 11 Next page →
Back to top | Article view | comp.arch
csiph-web