Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1600020 > unrolled thread

Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-14 04:50 +0100
    Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Hans Verkuil <hverkuil@xs4all.nl> - 2017-03-14 09:00 +0100
      Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-14 11:30 +0100
        media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4  14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline) Pavel Machek <pavel@ucw.cz> - 2017-03-14 23:40 +0100
          Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-15 02:00 +0100
            Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH         v4 14/36] [media] v4l2-mc: add a function to inherit controls from         a pipeline) Philippe De Muyter <phdm@macq.eu> - 2017-03-15 12:10 +0100
              Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Nicolas Dufresne <nicolas@ndufresne.ca> - 2017-03-15 20:00 +0100
                Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Philipp Zabel <p.zabel@pengutronix.de> - 2017-03-16 10:30 +0100
                  Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH         v4 14/36] [media] v4l2-mc: add a function to inherit controls from         a pipeline) Philippe De Muyter <phdm@macq.eu> - 2017-03-16 11:00 +0100
                    Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Philipp Zabel <p.zabel@pengutronix.de> - 2017-03-16 11:10 +0100
                      Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH         v4 14/36] [media] v4l2-mc: add a function to inherit controls from         a pipeline) Philippe De Muyter <phdm@macq.eu> - 2017-03-16 11:30 +0100
            Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4  14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline) Pavel Machek <pavel@ucw.cz> - 2017-03-15 19:10 +0100
              Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-15 21:30 +0100
                Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4  14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline) Pavel Machek <pavel@ucw.cz> - 2017-03-16 23:20 +0100
        Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Hans Verkuil <hverkuil@xs4all.nl> - 2017-03-20 14:30 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-20 16:50 +0100
            Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-03-20 17:20 +0100
              Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-20 18:40 +0100
      Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-03-17 12:50 +0100
        Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Sakari Ailus <sakari.ailus@linux.intel.com> - 2017-03-17 13:00 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Philipp Zabel <p.zabel@pengutronix.de> - 2017-03-17 15:00 +0100
            Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-03-17 15:50 +0100
              Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Hans Verkuil <hverkuil@xs4all.nl> - 2017-03-20 14:20 +0100
                Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-20 16:20 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-17 15:10 +0100
            Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Pavel Machek <pavel@ucw.cz> - 2017-03-21 12:20 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Hans Verkuil <hverkuil@xs4all.nl> - 2017-03-20 12:20 +0100
        Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Philipp Zabel <p.zabel@pengutronix.de> - 2017-03-17 13:10 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-03-17 13:30 +0100
            Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-17 19:00 +0100
        Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Pavel Machek <pavel@ucw.cz> - 2017-03-19 14:30 +0100
      Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline Laurent Pinchart <laurent.pinchart@ideasonboard.com> - 2017-03-26 18:50 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1603338

On Fri, 2017-03-17 at 10:24 -0300, Mauro Carvalho Chehab wrote:
[...]
> The big question, waiting for an answer on the last 8 years is
> who would do that? Such person would need to have several different
> hardware from different vendors, in order to ensure that it has
> a generic solution.
> 
> It is a way more feasible that the Kernel developers that already 
> have a certain hardware on their hands to add support inside the
> driver to forward the controls through the pipeline and to setup
> a "default" pipeline that would cover the common use cases at
> driver's probe.

Actually, would setting pipeline via libv4l2 plugin and letting drivers
provide a sane enabled default pipeline configuration be mutually
exclusive? Not sure about the control forwarding, but at least a simple
link setup and format forwarding would also be possible in the kernel
without hindering userspace from doing it themselves later.

regards
Philipp

[toc] | [prev] | [next] | [standalone]

#1603369

On Fri, Mar 17, 2017 at 02:51:10PM +0100, Philipp Zabel wrote:
> On Fri, 2017-03-17 at 10:24 -0300, Mauro Carvalho Chehab wrote:
> [...]
> > The big question, waiting for an answer on the last 8 years is
> > who would do that? Such person would need to have several different
> > hardware from different vendors, in order to ensure that it has
> > a generic solution.
> > 
> > It is a way more feasible that the Kernel developers that already 
> > have a certain hardware on their hands to add support inside the
> > driver to forward the controls through the pipeline and to setup
> > a "default" pipeline that would cover the common use cases at
> > driver's probe.
> 
> Actually, would setting pipeline via libv4l2 plugin and letting drivers
> provide a sane enabled default pipeline configuration be mutually
> exclusive? Not sure about the control forwarding, but at least a simple
> link setup and format forwarding would also be possible in the kernel
> without hindering userspace from doing it themselves later.

I think this is the exact same problem as controls in ALSA.

When ALSA started off in life, the requirement was that all controls
shall default to minimum, and the user is expected to adjust controls
after the system is running.

After OSS, this gave quite a marked change in system behaviour, and
led to a lot of "why doesn't my sound work anymore" problems, because
people then had to figure out which combination of controls had to be
set to get sound out of their systems.

Now it seems to be much better, where install Linux on a platform, and
you have a working sound system (assuming that the drivers are all there
which is generally the case for x86.)

However, it's still possible to adjust all the controls from userspace.
All that's changed is the defaults.

Why am I mentioning this - because from what I understand Mauro saying,
it's no different from this situation.  Userspace will still have the
power to disable all links and setup its own.  The difference is that
there will be a default configuration that the kernel sets up at boot
time that will be functional, rather than the current default
configuration where the system is completely non-functional until
manually configured.

However, at the end of the day, I don't care _where_ the usability
problems are solved, only that there is some kind of solution.  It's not
the _where_ that's the real issue here, but the _how_, and discussion of
the _how_ is completely missing.

So, let's try kicking off a discussion about _how_ to do things.

_How_ do we setup a media controller system so that we end up with a
usable configuration - let's start with the obvious bit... which links
should be enabled.

I think the first pre-requisit is that we stop exposing capture devices
that can never be functional for the hardware that's present on the board,
so that there isn't this plentora of useless /dev/video* nodes and useless
subdevices.

One possible solution to finding a default path may be "find the shortest
path between the capture device and the sensor and enable intervening
links".

Then we need to try configuring that path with format/resolution
information.

However, what if something in the shortest path can't handle the format
that the sensor produces?  I think at that point, we'd need to drop that
subdev out of the path resolution, re-run the "find the shortest path"
algorithm, and try again.

Repeat until success or no path between the capture and sensor exists.

This works fine if you have just one sensor visible from a capture device,
but not if there's more than one (which I suspect is the case with the
Sabrelite board with its two cameras and video receiver.)  That breaks
the "find the shortest path" algorithm.

So, maybe it's a lot better to just let the board people provide via DT
a default setup for the connectivity of the modules somehow - certainly
one big step forward would be to disable in DT parts of the capture
system that can never be used (remembering that boards like the RPi /
Hummingboard may end up using DT overlays to describe this for different
cameras, so the capture setup may change after initial boot.)

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1604552

On 03/17/2017 03:37 PM, Russell King - ARM Linux wrote:
> On Fri, Mar 17, 2017 at 02:51:10PM +0100, Philipp Zabel wrote:
>> On Fri, 2017-03-17 at 10:24 -0300, Mauro Carvalho Chehab wrote:
>> [...]
>>> The big question, waiting for an answer on the last 8 years is
>>> who would do that? Such person would need to have several different
>>> hardware from different vendors, in order to ensure that it has
>>> a generic solution.
>>>
>>> It is a way more feasible that the Kernel developers that already 
>>> have a certain hardware on their hands to add support inside the
>>> driver to forward the controls through the pipeline and to setup
>>> a "default" pipeline that would cover the common use cases at
>>> driver's probe.
>>
>> Actually, would setting pipeline via libv4l2 plugin and letting drivers
>> provide a sane enabled default pipeline configuration be mutually
>> exclusive? Not sure about the control forwarding, but at least a simple
>> link setup and format forwarding would also be possible in the kernel
>> without hindering userspace from doing it themselves later.
> 
> I think this is the exact same problem as controls in ALSA.
> 
> When ALSA started off in life, the requirement was that all controls
> shall default to minimum, and the user is expected to adjust controls
> after the system is running.
> 
> After OSS, this gave quite a marked change in system behaviour, and
> led to a lot of "why doesn't my sound work anymore" problems, because
> people then had to figure out which combination of controls had to be
> set to get sound out of their systems.
> 
> Now it seems to be much better, where install Linux on a platform, and
> you have a working sound system (assuming that the drivers are all there
> which is generally the case for x86.)
> 
> However, it's still possible to adjust all the controls from userspace.
> All that's changed is the defaults.
> 
> Why am I mentioning this - because from what I understand Mauro saying,
> it's no different from this situation.  Userspace will still have the
> power to disable all links and setup its own.  The difference is that
> there will be a default configuration that the kernel sets up at boot
> time that will be functional, rather than the current default
> configuration where the system is completely non-functional until
> manually configured.
> 
> However, at the end of the day, I don't care _where_ the usability
> problems are solved, only that there is some kind of solution.  It's not
> the _where_ that's the real issue here, but the _how_, and discussion of
> the _how_ is completely missing.
> 
> So, let's try kicking off a discussion about _how_ to do things.
> 
> _How_ do we setup a media controller system so that we end up with a
> usable configuration - let's start with the obvious bit... which links
> should be enabled.
> 
> I think the first pre-requisit is that we stop exposing capture devices
> that can never be functional for the hardware that's present on the board,
> so that there isn't this plentora of useless /dev/video* nodes and useless
> subdevices.
> 
> One possible solution to finding a default path may be "find the shortest
> path between the capture device and the sensor and enable intervening
> links".
> 
> Then we need to try configuring that path with format/resolution
> information.
> 
> However, what if something in the shortest path can't handle the format
> that the sensor produces?  I think at that point, we'd need to drop that
> subdev out of the path resolution, re-run the "find the shortest path"
> algorithm, and try again.
> 
> Repeat until success or no path between the capture and sensor exists.
> 
> This works fine if you have just one sensor visible from a capture device,
> but not if there's more than one (which I suspect is the case with the
> Sabrelite board with its two cameras and video receiver.)  That breaks
> the "find the shortest path" algorithm.
> 
> So, maybe it's a lot better to just let the board people provide via DT
> a default setup for the connectivity of the modules somehow - certainly
> one big step forward would be to disable in DT parts of the capture
> system that can never be used (remembering that boards like the RPi /
> Hummingboard may end up using DT overlays to describe this for different
> cameras, so the capture setup may change after initial boot.)

The MC was developed before the device tree came along. But now that the DT
is here, I think this could be a sensible idea to let the DT provide an
initial path.

Sakari, Laurent, Mauro: any opinions?

Regards,

	Hans

[toc] | [prev] | [next] | [standalone]

#1604684

Em Mon, 20 Mar 2017 14:10:30 +0100
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> On 03/17/2017 03:37 PM, Russell King - ARM Linux wrote:
> > On Fri, Mar 17, 2017 at 02:51:10PM +0100, Philipp Zabel wrote:  
> >> On Fri, 2017-03-17 at 10:24 -0300, Mauro Carvalho Chehab wrote:
> >> [...]  
> >>> The big question, waiting for an answer on the last 8 years is
> >>> who would do that? Such person would need to have several different
> >>> hardware from different vendors, in order to ensure that it has
> >>> a generic solution.
> >>>
> >>> It is a way more feasible that the Kernel developers that already 
> >>> have a certain hardware on their hands to add support inside the
> >>> driver to forward the controls through the pipeline and to setup
> >>> a "default" pipeline that would cover the common use cases at
> >>> driver's probe.  
> >>
> >> Actually, would setting pipeline via libv4l2 plugin and letting drivers
> >> provide a sane enabled default pipeline configuration be mutually
> >> exclusive? Not sure about the control forwarding, but at least a simple
> >> link setup and format forwarding would also be possible in the kernel
> >> without hindering userspace from doing it themselves later.  
> > 
> > I think this is the exact same problem as controls in ALSA.
> > 
> > When ALSA started off in life, the requirement was that all controls
> > shall default to minimum, and the user is expected to adjust controls
> > after the system is running.
> > 
> > After OSS, this gave quite a marked change in system behaviour, and
> > led to a lot of "why doesn't my sound work anymore" problems, because
> > people then had to figure out which combination of controls had to be
> > set to get sound out of their systems.
> > 
> > Now it seems to be much better, where install Linux on a platform, and
> > you have a working sound system (assuming that the drivers are all there
> > which is generally the case for x86.)
> > 
> > However, it's still possible to adjust all the controls from userspace.
> > All that's changed is the defaults.
> > 
> > Why am I mentioning this - because from what I understand Mauro saying,
> > it's no different from this situation.  Userspace will still have the
> > power to disable all links and setup its own.  The difference is that
> > there will be a default configuration that the kernel sets up at boot
> > time that will be functional, rather than the current default
> > configuration where the system is completely non-functional until
> > manually configured.
> > 
> > However, at the end of the day, I don't care _where_ the usability
> > problems are solved, only that there is some kind of solution.  It's not
> > the _where_ that's the real issue here, but the _how_, and discussion of
> > the _how_ is completely missing.
> > 
> > So, let's try kicking off a discussion about _how_ to do things.
> > 
> > _How_ do we setup a media controller system so that we end up with a
> > usable configuration - let's start with the obvious bit... which links
> > should be enabled.
> > 
> > I think the first pre-requisit is that we stop exposing capture devices
> > that can never be functional for the hardware that's present on the board,
> > so that there isn't this plentora of useless /dev/video* nodes and useless
> > subdevices.
> > 
> > One possible solution to finding a default path may be "find the shortest
> > path between the capture device and the sensor and enable intervening
> > links".
> > 
> > Then we need to try configuring that path with format/resolution
> > information.
> > 
> > However, what if something in the shortest path can't handle the format
> > that the sensor produces?  I think at that point, we'd need to drop that
> > subdev out of the path resolution, re-run the "find the shortest path"
> > algorithm, and try again.
> > 
> > Repeat until success or no path between the capture and sensor exists.
> > 
> > This works fine if you have just one sensor visible from a capture device,
> > but not if there's more than one (which I suspect is the case with the
> > Sabrelite board with its two cameras and video receiver.)  That breaks
> > the "find the shortest path" algorithm.
> > 
> > So, maybe it's a lot better to just let the board people provide via DT
> > a default setup for the connectivity of the modules somehow - certainly
> > one big step forward would be to disable in DT parts of the capture
> > system that can never be used (remembering that boards like the RPi /
> > Hummingboard may end up using DT overlays to describe this for different
> > cameras, so the capture setup may change after initial boot.)  
> 
> The MC was developed before the device tree came along. But now that the DT
> is here, I think this could be a sensible idea to let the DT provide an
> initial path.
> 
> Sakari, Laurent, Mauro: any opinions?

It makes perfect sense to me.

By setting the pipeline via DT on boards with simple configurations,
e. g. just one CSI physical interface, it can create just one
devnode (e. g. /dev/video0) with would fully control the device,
without enabling subdev API for such hardware, making the hardware
usable with all V4L2 applications.

Regards,
Mauro

[toc] | [prev] | [next] | [standalone]

#1603346

Em Fri, 17 Mar 2017 13:55:33 +0200
Sakari Ailus <sakari.ailus@linux.intel.com> escreveu:

> Hi Russell,
> 
> On 03/17/17 13:42, Russell King - ARM Linux wrote:
> > On Tue, Mar 14, 2017 at 08:55:36AM +0100, Hans Verkuil wrote:  
> >> We're all very driver-development-driven, and userspace gets very little
> >> attention in general. So before just throwing in the towel we should take
> >> a good look at the reasons why there has been little or no development: is
> >> it because of fundamental design defects, or because nobody paid attention
> >> to it?
> >>
> >> I strongly suspect it is the latter.
> >>
> >> In addition, I suspect end-users of these complex devices don't really care
> >> about a plugin: they want full control and won't typically use generic
> >> applications. If they would need support for that, we'd have seen much more
> >> interest. The main reason for having a plugin is to simplify testing and
> >> if this is going to be used on cheap hobbyist devkits.  
> > 
> > I think you're looking at it with a programmers hat on, not a users hat.

I fully agree with you: whatever solution is provided, this should be fully
transparent for the end user, no matter what V4L2 application he's using.

> > 
> > Are you really telling me that requiring users to 'su' to root, and then
> > use media-ctl to manually configure the capture device is what most
> > users "want" ?  

The need of su can easily be fixed with a simple addition at the udev
rules.d, for it to consider media controller as part of the v4l stuff:

--- udev/rules.d/50-udev-default.rules	2017-02-01 19:45:35.000000000 -0200
+++ udev/rules.d/50-udev-default.rules	2015-08-29 07:54:16.033122614 -0300
@@ -30,6 +30,7 @@ SUBSYSTEM=="mem", KERNEL=="mem|kmem|port
 SUBSYSTEM=="input", GROUP="input"
 SUBSYSTEM=="input", KERNEL=="js[0-9]*", MODE="0664"
 
+SUBSYSTEM=="media", GROUP="video"
 SUBSYSTEM=="video4linux", GROUP="video"
 SUBSYSTEM=="graphics", GROUP="video"
 SUBSYSTEM=="drm", GROUP="video"

Ok, someone should base it on upstream and submit this to
udev maintainers[1].

[1] On a side note, it would also be possible to have an udev rule that
    would be automatically setting the pipeline then the media device pops
    up.

   I wrote something like that for remote controllers, with gets
   installed together with v4l-utils package: when a remote controller
   is detected, it checks the driver and the remote controller table
   that the driver wants and load it on userspace.

   It would be possible to do something like that for MC, but someone 
   would need to do such task. Of course, that would require to have
   a way for a generic application to detect the board type and be
   able to automatically setup the pipelines. So, we'll go back to
   the initial problem that nobody was able to do that so far.

> It depends on who the user is. I don't think anyone is suggesting a
> regular end user is the user of all these APIs: it is either an
> application tailored for that given device, a skilled user with his test
> scripts 

Test scripts are just test scripts, meant for development purposes.
We shouldn't even consider this seriously.

> Making use of the full potential of the hardware requires using a more
> expressive interface. 

That's the core of the problem: most users don't need "full potential
of the hardware". It is actually worse than that: several boards
don't allow "full potential" of the SoC capabilities.

Ok, when the user requires "full potential", they may need a complex
tailored application. But, on most cases, all it is needed is to
support a simple application that controls a video stream via
/dev/video0.

> That's what the kernel interface must provide. If
> we decide to limit ourselves to a small sub-set of that potential on the
> level of the kernel interface, we have made a wrong decision. It's as
> simple as that. This is why the functionality (and which requires taking
> a lot of policy decisions) belongs to the user space. We cannot have
> multiple drivers providing multiple kernel interfaces for the same hardware.

I strongly disagree. Looking only at the hardware capabilities without
having a solution to provide what the user wants is *wrong*.

The project decisions should be based on the common use cases, and, if
possible and not to expensive/complex, covering exotic cases.

The V4L2 API was designed to fulfill the user needs. Drivers should
take it in consideration when choosing policy decisions.

In order to give you some examples, before the V4L2 API, the bttv driver
used to have its own set of ioctls, meant to provide functionality based
on its hardware capabilities (like selecting other standards like PAL/M).
The end result is that applications written for bttv were not generic
enough[1]. The V4L2 API was designed to be generic enough to cover the
common use-cases.

[1] As the bttv board were very popular, most userspace apps didn't
    work on other hardware. The big advantage of V4L2 API is that it
    contains a set of functions that it is good enough to control any
    hardware. Ok, some features are missing. 

   For example, in the past, one could use the bttv hardware to "decrypt"
   analog cable TV using custom ioctls. Those ioctls got removed during
   the V4L2 conversion, as they are specific to the way bttv hardware
   works.
 
   Also, the cx88 hardware could be used as a generic fast A/D converter, 
   using a custom set of ioctls - something similar to SDR. In this
   specific case, such patchset was never merged upstream.


To give you a few examples about policy decisions taken by the drivers
in order to fulfill the user needs, the Conexant chipsets (supported by
bttv,  cx88, cx231xx and cx25821 drivers, among others) provide a way 
more possibilities than what the driver supports.

Basically, they all have fast A/D converters, running at around
27-30 MHz clock. The samples of the A/D can be passed as-is to
userspace and/or handled by some IP blocks inside the chips.

Also, even the simplest one (bttv) uses a RISC with a firmware that
is built dynamically at runtime by the Kernel driver. Such firmware
needs to setup several DMA pipelines.

For example, the cx88 driver sets those DMA pipelines
(see drivers/media/pci/cx88/cx88-core.c):

 * FIFO space allocations:
 *    channel  21    (y video)  - 10.0k
 *    channel  22    (u video)  -  2.0k
 *    channel  23    (v video)  -  2.0k
 *    channel  24    (vbi)      -  4.0k
 *    channels 25+26 (audio)    -  4.0k
 *    channel  28    (mpeg)     -  4.0k
 *    channel  27    (audio rds)-  3.0k

In order to provide the generic V4L2 API, the driver need to
take lots of device-specific decisions. 

For example, the DMA from channel 28 is only enabled if the
device has an IP block to do MPEG, and the user wants to
receive mpeg data, instead of YUV.

The DMA from channel 27 has samples taken from the audio IF.
The audio decoding itself are at DMA channels 25 and 26.

The data from DMA channel 27 is used by a software dsp code, 
implemented at drivers/media/pci/cx88/cx88-dsp.c, with detects 
if the IF contains AM or FM modulation and what are the carriers,
in order to identify the audio standard. Once the standard is
detected, it sets the hardware audio decoder to the right
standard, and the DMA from channels 25 and 26 will contain
audio PCM samples.

The net result is that an user of any of the V4L2 drivers

That's a way more complex than deciding if a pipeline would
require an IP block to convert from Bayer format to YUV.

Another case: the cx25821 hardware supports 12 video streams, 
consuming almost all available bandwidth of an ePCI bus. Each video 
stream connector can either be configured to be capture or output, in
runtime. The hardware vendor chose to hardcode the driver to provide
8 inputs and 4 outputs. Their decision was based in the fact that
the driver is already very complex, and it satisfies their customer's 
needs. The cost/efforts of make the driver to be reconfigured in
runtime were too high for almost no benefit.

> That said, I'm not trying to provide an excuse for not having libraries
> available to help the user to configure and control the device more or
> less automatically even in terms of best effort. It's something that
> does require attention, a lot more of it than it has received in recent
> few years.

The big question, waiting for an answer on the last 8 years is
who would do that? Such person would need to have several different
hardware from different vendors, in order to ensure that it has
a generic solution.

It is a way more feasible that the Kernel developers that already 
have a certain hardware on their hands to add support inside the
driver to forward the controls through the pipeline and to setup
a "default" pipeline that would cover the common use cases at
driver's probe.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1605506

[Multipart message — attachments visible in raw view] — view raw

Hi!

> > Making use of the full potential of the hardware requires using a more
> > expressive interface. 
> 
> That's the core of the problem: most users don't need "full potential
> of the hardware". It is actually worse than that: several boards
> don't allow "full potential" of the SoC capabilities.

Well, in kernel we usually try to support "full hardware" potential.

And we are pretty sure users would like to take still photos, even if
common v4l2 applications can not do it.

> > That's what the kernel interface must provide. If
> > we decide to limit ourselves to a small sub-set of that potential on the
> > level of the kernel interface, we have made a wrong decision. It's as
> > simple as that. This is why the functionality (and which requires taking
> > a lot of policy decisions) belongs to the user space. We cannot have
> > multiple drivers providing multiple kernel interfaces for the same hardware.
> 
> I strongly disagree. Looking only at the hardware capabilities without
> having a solution to provide what the user wants is *wrong*.

Hardware manufacturers already did this kind of research for us. They
don't usually include features noone wants...

> Another case: the cx25821 hardware supports 12 video streams, 
> consuming almost all available bandwidth of an ePCI bus. Each video 
> stream connector can either be configured to be capture or output, in
> runtime. The hardware vendor chose to hardcode the driver to provide
> 8 inputs and 4 outputs. Their decision was based in the fact that
> the driver is already very complex, and it satisfies their customer's 
> needs. The cost/efforts of make the driver to be reconfigured in
> runtime were too high for almost no benefit.

Well, it is okay to provide 'limited' driver -- there's possibility to
fix the driver. But IMO it is not okay to provide 'limited' kernel
interface -- because if you try to fix it, you'll suddenly have
regressions.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1604391

On 03/17/2017 12:55 PM, Sakari Ailus wrote:
> Hi Russell,
> 
> On 03/17/17 13:42, Russell King - ARM Linux wrote:
>> On Tue, Mar 14, 2017 at 08:55:36AM +0100, Hans Verkuil wrote:
>>> We're all very driver-development-driven, and userspace gets very little
>>> attention in general. So before just throwing in the towel we should take
>>> a good look at the reasons why there has been little or no development: is
>>> it because of fundamental design defects, or because nobody paid attention
>>> to it?
>>>
>>> I strongly suspect it is the latter.
>>>
>>> In addition, I suspect end-users of these complex devices don't really care
>>> about a plugin: they want full control and won't typically use generic
>>> applications. If they would need support for that, we'd have seen much more
>>> interest. The main reason for having a plugin is to simplify testing and
>>> if this is going to be used on cheap hobbyist devkits.
>>
>> I think you're looking at it with a programmers hat on, not a users hat.
>>
>> Are you really telling me that requiring users to 'su' to root, and then
>> use media-ctl to manually configure the capture device is what most
>> users "want" ?
> 
> It depends on who the user is. I don't think anyone is suggesting a
> regular end user is the user of all these APIs: it is either an
> application tailored for that given device, a skilled user with his test
> scripts or as suggested previously, a libv4l plugin knowing the device
> or a generic library geared towards providing best effort service. The
> last one of this list does not exist yet and the second last item
> requires help.
> 
> Typically this class of devices is simply not up to provide the level of
> service you're requesting without additional user space control library
> which is responsible for automatic white balance, exposure and focus.
> 
> Making use of the full potential of the hardware requires using a more
> expressive interface. That's what the kernel interface must provide. If
> we decide to limit ourselves to a small sub-set of that potential on the
> level of the kernel interface, we have made a wrong decision. It's as
> simple as that. This is why the functionality (and which requires taking
> a lot of policy decisions) belongs to the user space. We cannot have
> multiple drivers providing multiple kernel interfaces for the same hardware.

Right. With my Cisco hat on I can tell you that Cisco would want full low-level
control. If the driver would limit us we would not be able to use it.

Same with anyone who wants to put Android CameraHAL on top of a V4L2 driver:
they would need full control. Some simplified interface would be unacceptable.

> 
> That said, I'm not trying to provide an excuse for not having libraries
> available to help the user to configure and control the device more or
> less automatically even in terms of best effort. It's something that
> does require attention, a lot more of it than it has received in recent
> few years.

Right.

Regards,

	Hans

[toc] | [prev] | [next] | [standalone]

#1603241

On Fri, 2017-03-17 at 11:42 +0000, Russell King - ARM Linux wrote:
> On Tue, Mar 14, 2017 at 08:55:36AM +0100, Hans Verkuil wrote:
> > We're all very driver-development-driven, and userspace gets very little
> > attention in general. So before just throwing in the towel we should take
> > a good look at the reasons why there has been little or no development: is
> > it because of fundamental design defects, or because nobody paid attention
> > to it?
> > 
> > I strongly suspect it is the latter.
> > 
> > In addition, I suspect end-users of these complex devices don't really care
> > about a plugin: they want full control and won't typically use generic
> > applications. If they would need support for that, we'd have seen much more
> > interest. The main reason for having a plugin is to simplify testing and
> > if this is going to be used on cheap hobbyist devkits.
> 
> I think you're looking at it with a programmers hat on, not a users hat.
> 
> Are you really telling me that requiring users to 'su' to root, and then
> use media-ctl to manually configure the capture device is what most
> users "want" ?
> 
> Hasn't the way technology has moved towards graphical interfaces,
> particularly smart phones, taught us that the vast majority of users
> want is intuitive, easy to use interfaces, and not the command line
> with reams of documentation?
> 
> Why are smart phones soo popular - it's partly because they're flashy,
> but also because of the wealth of apps, and apps which follow the
> philosophy of "do one job, do it well" (otherwise they get bad reviews.)

> > An additional complication is simply that it is hard to find fully supported
> > MC hardware. omap3 boards are hard to find these days, renesas boards are not
> > easy to get, freescale isn't the most popular either. Allwinner, mediatek,
> > amlogic, broadcom and qualcomm all have closed source implementations or no
> > implementation at all.
> 
> Right, and that in itself tells us something - the problem that we're
> trying to solve is not one that commonly exists in the real world.
> 
> Yes, the hardware we have in front of us may be very complex, but if
> there's very few systems out there which are capable of making use of
> all that complexity, then we're trying to solve a problem that isn't
> the common case - and if it's going to take years to solve it (it
> already has taken years) then it's the wrong problem to be solved.
> 
> I bet most of the problem can be eliminated if, rather than exposing
> all this complexity, we instead expose a simpler capture system where
> the board designer gets to "wire up" the capture system.
> 
> I'll go back to my Bayer example, because that's the simplest.  As
> I've already said many times in these threads, there is only one
> possible path through the iMX6 device that such a source can be used
> with - it's a fixed path.  The actual path depends on the CSI2
> virtual channel that the camera has been _configured_ to use, but
> apart from that, it's effectively a well known set of blocks.  Such
> a configuration could be placed in DT.
> 
> For RGB connected to a single parallel CSI, things get a little more
> complex - capture through the CSI or through two other capture devices
> for de-interlacing or other features.  However, I'm not convinced that
> exposing multiple /dev/video* devices for different features for the
> same video source is a sane approach - I think that's a huge usability
> problem.  (The user is expected to select the capture device on iMX6
> depending on the features they want, and if they want to change features,
> they're expected to shut down their application and start it up on a
> different capture device.)  For the most part on iMX6, there's one
> path down to the CSI block, and then there's optional routing through
> the rest of the IPU depending on what features you want (such as
> de-interlacing.)
>
> The complex case is a CSI2 connected camera which produces multiple
> streams through differing virtual channels - and that's IMHO the only
> case where we need multiple different /dev/video* capture devices to
> be present.

I wanted to have the IC PRP outputs separate because the IC PRP should
support running both the VF and ENC tasks with different parameters from
the same input. That would allow to capture two different resolutions
(up to 1024x1024) at the same time.

I think most of the simple, fixed pipeline use cases could be handled by
libv4l2, by allowing to pass a v4l2 subdevice path to v4l2_open. If that
function internally would set up the media links to the
nearest /dev/video interface, propagate format, resolution and frame
intervals if necessary, and return an fd to the video device, there'd be
no additional complexity for the users beyond selecting the v4l2_subdev
instead of the video device.

regards
Philipp

[toc] | [prev] | [next] | [standalone]

#1603265

On Fri, Mar 17, 2017 at 01:02:07PM +0100, Philipp Zabel wrote:
> I think most of the simple, fixed pipeline use cases could be handled by
> libv4l2, by allowing to pass a v4l2 subdevice path to v4l2_open. If that
> function internally would set up the media links to the
> nearest /dev/video interface, propagate format, resolution and frame
> intervals if necessary, and return an fd to the video device, there'd be
> no additional complexity for the users beyond selecting the v4l2_subdev
> instead of the video device.

... which would then require gstreamer to be modified too. The gstreamer
v4l2 plugin looks for /dev/video* or /dev/v4l2/video* devices and monitors
these for changes, so gstreamer applications know which capture devices
are available:

  const gchar *paths[] = { "/dev", "/dev/v4l2", NULL };
  const gchar *names[] = { "video", NULL };

  /* Add some depedency, so the dynamic features get updated upon changes in
   * /dev/video* */
  gst_plugin_add_dependency (plugin,
      NULL, paths, names, GST_PLUGIN_DEPENDENCY_FLAG_FILE_NAME_IS_PREFIX);

I haven't checked yet whether sys/v4l2/gstv4l2deviceprovider.c knows
anything about the v4l2 subdevs.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1603511

Em Fri, 17 Mar 2017 12:16:08 +0000
Russell King - ARM Linux <linux@armlinux.org.uk> escreveu:

> On Fri, Mar 17, 2017 at 01:02:07PM +0100, Philipp Zabel wrote:
> > I think most of the simple, fixed pipeline use cases could be handled by
> > libv4l2, by allowing to pass a v4l2 subdevice path to v4l2_open. If that
> > function internally would set up the media links to the
> > nearest /dev/video interface, propagate format, resolution and frame
> > intervals if necessary, and return an fd to the video device, there'd be
> > no additional complexity for the users beyond selecting the v4l2_subdev
> > instead of the video device.  
> 
> ... which would then require gstreamer to be modified too. The gstreamer
> v4l2 plugin looks for /dev/video* or /dev/v4l2/video* devices and monitors
> these for changes, so gstreamer applications know which capture devices
> are available:
> 
>   const gchar *paths[] = { "/dev", "/dev/v4l2", NULL };
>   const gchar *names[] = { "video", NULL };
> 
>   /* Add some depedency, so the dynamic features get updated upon changes in
>    * /dev/video* */
>   gst_plugin_add_dependency (plugin,
>       NULL, paths, names, GST_PLUGIN_DEPENDENCY_FLAG_FILE_NAME_IS_PREFIX);
> 
> I haven't checked yet whether sys/v4l2/gstv4l2deviceprovider.c knows
> anything about the v4l2 subdevs.

Not only gstreamer do that, but all simple V4L2 applications, although
on most of them, you can either pass a command line argument or setup
the patch via GUI.

Btw, I've no idea from where gstreamer took /dev/v4l2 :-)
I'm yet to find a distribution using it.

On the other hand, /dev/v4l/by-patch and /dev/v4l/by-id are usual directories
where V4L2 devices can be found, and should provide persistent names. So, IMHO,
gst should prefer those names, when they exist:

$ tree /dev/v4l
/dev/v4l
├── by-id
│   ├── usb-046d_HD_Pro_Webcam_C920_55DA1CCF-video-index0 -> ../../video1
│   └── usb-Sunplus_mMobile_Inc_USB_Web-CAM-video-index0 -> ../../video0
└── by-path
    ├── platform-3f980000.usb-usb-0:1.2:1.0-video-index0 -> ../../video1
    └── platform-3f980000.usb-usb-0:1.5:1.0-video-index0 -> ../../video0




Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1603975

[Multipart message — attachments visible in raw view] — view raw

On Fri 2017-03-17 11:42:03, Russell King - ARM Linux wrote:
> On Tue, Mar 14, 2017 at 08:55:36AM +0100, Hans Verkuil wrote:
> > We're all very driver-development-driven, and userspace gets very little
> > attention in general. So before just throwing in the towel we should take
> > a good look at the reasons why there has been little or no development: is
> > it because of fundamental design defects, or because nobody paid attention
> > to it?
> > 
> > I strongly suspect it is the latter.
> > 
> > In addition, I suspect end-users of these complex devices don't really care
> > about a plugin: they want full control and won't typically use generic
> > applications. If they would need support for that, we'd have seen much more
> > interest. The main reason for having a plugin is to simplify testing and
> > if this is going to be used on cheap hobbyist devkits.
> 
> I think you're looking at it with a programmers hat on, not a users hat.
> 
> Are you really telling me that requiring users to 'su' to root, and then
> use media-ctl to manually configure the capture device is what most
> users "want" ?

If you want to help users, right way is to improve userland support. 

> Hasn't the way technology has moved towards graphical interfaces,
> particularly smart phones, taught us that the vast majority of users
> want is intuitive, easy to use interfaces, and not the command line
> with reams of documentation?

How is it relevant to _kernel_ interfaces?
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1609353 — Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline

Hi Hans,

On Tuesday 14 Mar 2017 08:55:36 Hans Verkuil wrote:
> On 03/14/2017 04:45 AM, Mauro Carvalho Chehab wrote:
> > Hi Sakari,
> > 
> > I started preparing a long argument about it, but gave up in favor of a
> > simpler one.
> > 
> > Em Mon, 13 Mar 2017 14:46:22 +0200 Sakari Ailus escreveu:
> >> Drivers are written to support hardware, not particular use case.
> > 
> > No, it is just the reverse: drivers and hardware are developed to
> > support use cases.
> > 
> > Btw, you should remember that the hardware is the full board, not just the
> > SoC. In practice, the board do limit the use cases: several provide a
> > single physical CSI connector, allowing just one sensor.
> > 
> >>> This situation is there since 2009. If I remember well, you tried to
> >>> write such generic plugin in the past, but never finished it, apparently
> >>> because it is too complex. Others tried too over the years.
> >> 
> >> I'd argue I know better what happened with that attempt than you do. I
> >> had a prototype of a generic pipeline configuration library but due to
> >> various reasons I haven't been able to continue working on that since
> >> around 2012.
> > ...
> > 
> >>> The last trial was done by Jacek, trying to cover just the exynos4
> >>> driver. Yet, even such limited scope plugin was not good enough, as it
> >>> was never merged upstream. Currently, there's no such plugins upstream.
> >>> 
> >>> If we can't even merge a plugin that solves it for just *one* driver,
> >>> I have no hope that we'll be able to do it for the generic case.
> >> 
> >> I believe Jacek ceased to work on that plugin in his day job; other than
> >> that, there are some matters left to be addressed in his latest patchset.
> > 
> > The two above basically summaries the issue: the task of doing a generic
> > plugin on userspace, even for a single driver is complex enough to
> > not cover within a reasonable timeline.
> > 
> > From 2009 to 2012, you were working on it, but didn't finish it.
> > 
> > Apparently, nobody worked on it between 2013-2014 (but I may be wrong, as
> > I didn't check when the generic plugin interface was added to libv4l).
> > 
> > In the case of Jacek's work, the first patch I was able to find was
> > 
> > written in Oct, 2014:
> > 	https://patchwork.kernel.org/patch/5098111/
> > 	(not sure what happened with the version 1).
> > 
> > The last e-mail about this subject was issued in Dec, 2016.
> > 
> > In summary, you had this on your task for 3 years for an OMAP3
> > plugin (where you have a good expertise), and Jacek for 2 years,
> > for Exynos 4, where he should also have a good knowledge.
> > 
> > Yet, with all that efforts, no concrete results were achieved, as none
> > of the plugins got merged.
> > 
> > Even if they were merged, if we keep the same mean time to develop a
> > libv4l plugin, that would mean that a plugin for i.MX6 could take 2-3
> > years to be developed.
> > 
> > There's a clear message on it:
> > 	- we shouldn't keep pushing for a solution via libv4l.
> 
> Or:
> 	- userspace plugin development had a very a low priority and
> 	  never got the attention it needed.
>
> I know that's *my* reason. I rarely if ever looked at it. I always assumed
> Sakari and/or Laurent would look at it. If this reason is also valid for
> Sakari and Laurent, then it is no wonder nothing has happened in all that
> time.

The reason is also valid for me. I'd really love to work on the userspace 
side, but I just can't find time at the moment.

> We're all very driver-development-driven, and userspace gets very little
> attention in general. So before just throwing in the towel we should take
> a good look at the reasons why there has been little or no development: is
> it because of fundamental design defects, or because nobody paid attention
> to it?
> 
> I strongly suspect it is the latter.
> 
> In addition, I suspect end-users of these complex devices don't really care
> about a plugin: they want full control and won't typically use generic
> applications. If they would need support for that, we'd have seen much more
> interest. The main reason for having a plugin is to simplify testing and
> if this is going to be used on cheap hobbyist devkits.
> 
> An additional complication is simply that it is hard to find fully supported
> MC hardware. omap3 boards are hard to find these days, renesas boards are
> not easy to get, freescale isn't the most popular either. Allwinner,
> mediatek, amlogic, broadcom and qualcomm all have closed source
> implementations or no implementation at all.
> 
> I know it took me a very long time before I had a working omap3.
> 
> So I am not at all surprised that little progress has been made.

-- 
Regards,

Laurent Pinchart

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web