Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1600020 > unrolled thread

Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-14 04:50 +0100
    Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Hans Verkuil <hverkuil@xs4all.nl> - 2017-03-14 09:00 +0100
      Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-14 11:30 +0100
        media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4  14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline) Pavel Machek <pavel@ucw.cz> - 2017-03-14 23:40 +0100
          Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-15 02:00 +0100
            Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH         v4 14/36] [media] v4l2-mc: add a function to inherit controls from         a pipeline) Philippe De Muyter <phdm@macq.eu> - 2017-03-15 12:10 +0100
              Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Nicolas Dufresne <nicolas@ndufresne.ca> - 2017-03-15 20:00 +0100
                Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Philipp Zabel <p.zabel@pengutronix.de> - 2017-03-16 10:30 +0100
                  Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH         v4 14/36] [media] v4l2-mc: add a function to inherit controls from         a pipeline) Philippe De Muyter <phdm@macq.eu> - 2017-03-16 11:00 +0100
                    Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Philipp Zabel <p.zabel@pengutronix.de> - 2017-03-16 11:10 +0100
                      Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH         v4 14/36] [media] v4l2-mc: add a function to inherit controls from         a pipeline) Philippe De Muyter <phdm@macq.eu> - 2017-03-16 11:30 +0100
            Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4  14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline) Pavel Machek <pavel@ucw.cz> - 2017-03-15 19:10 +0100
              Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH  v4 14/36] [media] v4l2-mc: add a function to inherit controls from a  pipeline) Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-15 21:30 +0100
                Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4  14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline) Pavel Machek <pavel@ucw.cz> - 2017-03-16 23:20 +0100
        Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Hans Verkuil <hverkuil@xs4all.nl> - 2017-03-20 14:30 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-20 16:50 +0100
            Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-03-20 17:20 +0100
              Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-20 18:40 +0100
      Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-03-17 12:50 +0100
        Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Sakari Ailus <sakari.ailus@linux.intel.com> - 2017-03-17 13:00 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Philipp Zabel <p.zabel@pengutronix.de> - 2017-03-17 15:00 +0100
            Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-03-17 15:50 +0100
              Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Hans Verkuil <hverkuil@xs4all.nl> - 2017-03-20 14:20 +0100
                Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-20 16:20 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-17 15:10 +0100
            Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Pavel Machek <pavel@ucw.cz> - 2017-03-21 12:20 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Hans Verkuil <hverkuil@xs4all.nl> - 2017-03-20 12:20 +0100
        Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Philipp Zabel <p.zabel@pengutronix.de> - 2017-03-17 13:10 +0100
          Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-03-17 13:30 +0100
            Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-03-17 19:00 +0100
        Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit  controls from a pipeline Pavel Machek <pavel@ucw.cz> - 2017-03-19 14:30 +0100
      Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline Laurent Pinchart <laurent.pinchart@ideasonboard.com> - 2017-03-26 18:50 +0200

Page 1 of 2  [1] 2  Next page →

#1600020 — Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline

Hi Sakari,

I started preparing a long argument about it, but gave up in favor of a
simpler one.

Em Mon, 13 Mar 2017 14:46:22 +0200
Sakari Ailus <sakari.ailus@iki.fi> escreveu:

> Drivers are written to support hardware, not particular use case.  

No, it is just the reverse: drivers and hardware are developed to
support use cases.

Btw, you should remember that the hardware is the full board, not just the
SoC. In practice, the board do limit the use cases: several provide a
single physical CSI connector, allowing just one sensor.

> > This situation is there since 2009. If I remember well, you tried to write
> > such generic plugin in the past, but never finished it, apparently because
> > it is too complex. Others tried too over the years.   
> 
> I'd argue I know better what happened with that attempt than you do. I had a
> prototype of a generic pipeline configuration library but due to various
> reasons I haven't been able to continue working on that since around 2012.

...

> > The last trial was done by Jacek, trying to cover just the exynos4 driver. 
> > Yet, even such limited scope plugin was not good enough, as it was never
> > merged upstream. Currently, there's no such plugins upstream.
> > 
> > If we can't even merge a plugin that solves it for just *one* driver,
> > I have no hope that we'll be able to do it for the generic case.  
> 
> I believe Jacek ceased to work on that plugin in his day job; other than
> that, there are some matters left to be addressed in his latest patchset.

The two above basically summaries the issue: the task of doing a generic
plugin on userspace, even for a single driver is complex enough to
not cover within a reasonable timeline.

From 2009 to 2012, you were working on it, but didn't finish it.

Apparently, nobody worked on it between 2013-2014 (but I may be wrong, as
I didn't check when the generic plugin interface was added to libv4l).

In the case of Jacek's work, the first patch I was able to find was
written in Oct, 2014:
	https://patchwork.kernel.org/patch/5098111/
	(not sure what happened with the version 1).

The last e-mail about this subject was issued in Dec, 2016.

In summary, you had this on your task for 3 years for an OMAP3
plugin (where you have a good expertise), and Jacek for 2 years, 
for Exynos 4, where he should also have a good knowledge.

Yet, with all that efforts, no concrete results were achieved, as none
of the plugins got merged.

Even if they were merged, if we keep the same mean time to develop a
libv4l plugin, that would mean that a plugin for i.MX6 could take 2-3
years to be developed.

There's a clear message on it:
	- we shouldn't keep pushing for a solution via libv4l.

Thanks,
Mauro

[toc] | [next] | [standalone]

#1600091

On 03/14/2017 04:45 AM, Mauro Carvalho Chehab wrote:
> Hi Sakari,
> 
> I started preparing a long argument about it, but gave up in favor of a
> simpler one.
> 
> Em Mon, 13 Mar 2017 14:46:22 +0200
> Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> 
>> Drivers are written to support hardware, not particular use case.  
> 
> No, it is just the reverse: drivers and hardware are developed to
> support use cases.
> 
> Btw, you should remember that the hardware is the full board, not just the
> SoC. In practice, the board do limit the use cases: several provide a
> single physical CSI connector, allowing just one sensor.
> 
>>> This situation is there since 2009. If I remember well, you tried to write
>>> such generic plugin in the past, but never finished it, apparently because
>>> it is too complex. Others tried too over the years.   
>>
>> I'd argue I know better what happened with that attempt than you do. I had a
>> prototype of a generic pipeline configuration library but due to various
>> reasons I haven't been able to continue working on that since around 2012.
> 
> ...
> 
>>> The last trial was done by Jacek, trying to cover just the exynos4 driver. 
>>> Yet, even such limited scope plugin was not good enough, as it was never
>>> merged upstream. Currently, there's no such plugins upstream.
>>>
>>> If we can't even merge a plugin that solves it for just *one* driver,
>>> I have no hope that we'll be able to do it for the generic case.  
>>
>> I believe Jacek ceased to work on that plugin in his day job; other than
>> that, there are some matters left to be addressed in his latest patchset.
> 
> The two above basically summaries the issue: the task of doing a generic
> plugin on userspace, even for a single driver is complex enough to
> not cover within a reasonable timeline.
> 
> From 2009 to 2012, you were working on it, but didn't finish it.
> 
> Apparently, nobody worked on it between 2013-2014 (but I may be wrong, as
> I didn't check when the generic plugin interface was added to libv4l).
> 
> In the case of Jacek's work, the first patch I was able to find was
> written in Oct, 2014:
> 	https://patchwork.kernel.org/patch/5098111/
> 	(not sure what happened with the version 1).
> 
> The last e-mail about this subject was issued in Dec, 2016.
> 
> In summary, you had this on your task for 3 years for an OMAP3
> plugin (where you have a good expertise), and Jacek for 2 years, 
> for Exynos 4, where he should also have a good knowledge.
> 
> Yet, with all that efforts, no concrete results were achieved, as none
> of the plugins got merged.
> 
> Even if they were merged, if we keep the same mean time to develop a
> libv4l plugin, that would mean that a plugin for i.MX6 could take 2-3
> years to be developed.
> 
> There's a clear message on it:
> 	- we shouldn't keep pushing for a solution via libv4l.

Or:

	- userspace plugin development had a very a low priority and
	  never got the attention it needed.

I know that's *my* reason. I rarely if ever looked at it. I always assumed
Sakari and/or Laurent would look at it. If this reason is also valid for
Sakari and Laurent, then it is no wonder nothing has happened in all that
time.

We're all very driver-development-driven, and userspace gets very little
attention in general. So before just throwing in the towel we should take
a good look at the reasons why there has been little or no development: is
it because of fundamental design defects, or because nobody paid attention
to it?

I strongly suspect it is the latter.

In addition, I suspect end-users of these complex devices don't really care
about a plugin: they want full control and won't typically use generic
applications. If they would need support for that, we'd have seen much more
interest. The main reason for having a plugin is to simplify testing and
if this is going to be used on cheap hobbyist devkits.

An additional complication is simply that it is hard to find fully supported
MC hardware. omap3 boards are hard to find these days, renesas boards are not
easy to get, freescale isn't the most popular either. Allwinner, mediatek,
amlogic, broadcom and qualcomm all have closed source implementations or no
implementation at all.

I know it took me a very long time before I had a working omap3.

So I am not at all surprised that little progress has been made.

Regards,

	Hans

[toc] | [prev] | [next] | [standalone]

#1600195

Em Tue, 14 Mar 2017 08:55:36 +0100
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> On 03/14/2017 04:45 AM, Mauro Carvalho Chehab wrote:
> > Hi Sakari,
> > 
> > I started preparing a long argument about it, but gave up in favor of a
> > simpler one.
> > 
> > Em Mon, 13 Mar 2017 14:46:22 +0200
> > Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> >   
> >> Drivers are written to support hardware, not particular use case.    
> > 
> > No, it is just the reverse: drivers and hardware are developed to
> > support use cases.
> > 
> > Btw, you should remember that the hardware is the full board, not just the
> > SoC. In practice, the board do limit the use cases: several provide a
> > single physical CSI connector, allowing just one sensor.
> >   
> >>> This situation is there since 2009. If I remember well, you tried to write
> >>> such generic plugin in the past, but never finished it, apparently because
> >>> it is too complex. Others tried too over the years.     
> >>
> >> I'd argue I know better what happened with that attempt than you do. I had a
> >> prototype of a generic pipeline configuration library but due to various
> >> reasons I haven't been able to continue working on that since around 2012.  
> > 
> > ...
> >   
> >>> The last trial was done by Jacek, trying to cover just the exynos4 driver. 
> >>> Yet, even such limited scope plugin was not good enough, as it was never
> >>> merged upstream. Currently, there's no such plugins upstream.
> >>>
> >>> If we can't even merge a plugin that solves it for just *one* driver,
> >>> I have no hope that we'll be able to do it for the generic case.    
> >>
> >> I believe Jacek ceased to work on that plugin in his day job; other than
> >> that, there are some matters left to be addressed in his latest patchset.  
> > 
> > The two above basically summaries the issue: the task of doing a generic
> > plugin on userspace, even for a single driver is complex enough to
> > not cover within a reasonable timeline.
> > 
> > From 2009 to 2012, you were working on it, but didn't finish it.
> > 
> > Apparently, nobody worked on it between 2013-2014 (but I may be wrong, as
> > I didn't check when the generic plugin interface was added to libv4l).
> > 
> > In the case of Jacek's work, the first patch I was able to find was
> > written in Oct, 2014:
> > 	https://patchwork.kernel.org/patch/5098111/
> > 	(not sure what happened with the version 1).
> > 
> > The last e-mail about this subject was issued in Dec, 2016.
> > 
> > In summary, you had this on your task for 3 years for an OMAP3
> > plugin (where you have a good expertise), and Jacek for 2 years, 
> > for Exynos 4, where he should also have a good knowledge.
> > 
> > Yet, with all that efforts, no concrete results were achieved, as none
> > of the plugins got merged.
> > 
> > Even if they were merged, if we keep the same mean time to develop a
> > libv4l plugin, that would mean that a plugin for i.MX6 could take 2-3
> > years to be developed.
> > 
> > There's a clear message on it:
> > 	- we shouldn't keep pushing for a solution via libv4l.  
> 
> Or:
> 
> 	- userspace plugin development had a very a low priority and
> 	  never got the attention it needed.

The end result is the same: we can't count on it.

> 
> I know that's *my* reason. I rarely if ever looked at it. I always assumed
> Sakari and/or Laurent would look at it. If this reason is also valid for
> Sakari and Laurent, then it is no wonder nothing has happened in all that
> time.
> 
> We're all very driver-development-driven, and userspace gets very little
> attention in general. So before just throwing in the towel we should take
> a good look at the reasons why there has been little or no development: is
> it because of fundamental design defects, or because nobody paid attention
> to it?

No. We should look it the other way: basically, there are patches
for i.MX6 driver that sends control from videonode to subdevs. 

If we nack apply it, who will write the userspace plugin? When
such change will be merged upstream?

If we don't have answers to any of the above questions, we should not
nack it.

That's said, that doesn't prevent merging a libv4l plugin if/when
someone can find time/interest to develop it.

> I strongly suspect it is the latter.
> 
> In addition, I suspect end-users of these complex devices don't really care
> about a plugin: they want full control and won't typically use generic
> applications. If they would need support for that, we'd have seen much more
> interest. The main reason for having a plugin is to simplify testing and
> if this is going to be used on cheap hobbyist devkits.

What are the needs for a cheap hobbyist devkit owner? Do we currently
satisfy those needs? I'd say that having a functional driver when
compiled without the subdev API, that implements the ioctl's/controls
that a generic application like camorama/google talk/skype/zbar...
would work should be enough to make them happy, even if they need to
add some udev rule and/or run some "prep" application that would setup 
the pipelines via MC and eventually rename the device with a working
pipeline to /dev/video0.

> 
> An additional complication is simply that it is hard to find fully supported
> MC hardware. omap3 boards are hard to find these days, renesas boards are not
> easy to get, freescale isn't the most popular either. Allwinner, mediatek,
> amlogic, broadcom and qualcomm all have closed source implementations or no
> implementation at all.

I'd say that we should not care anymore on providing a solution for
generic applications to run on boards like OMAP3[1]. For hardware that
are currently available that have Kernel driver and boards developed
to be used as "cheap hobbyist devkit", I'd say we should implement
a Kernel solution that would allow them to be used without subdev
API, e. g. having all ioctls needed by generic applications to work
functional, after some external application sets the pipeline.

[1] Yet, I might eventually do that for fun, an OMAP3 board with tvp5150
just arrived here last week. It would be nice to have xawtv3 running on it :-)
So, if I have a lot of spare time (with is very unlikely), I might eventually 
do something for it to work.

> I know it took me a very long time before I had a working omap3.

My first OMAP3 board with working V4L2 source just arrived last week :-)

> So I am not at all surprised that little progress has been made.

I'm not surprised, but I'm disappointed, as I tried to push toward a
solution for this problem since when we had our initial meetings about
it.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1600939 — media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

[Multipart message — attachments visible in raw view] — view raw

Hi!

> > > Even if they were merged, if we keep the same mean time to develop a
> > > libv4l plugin, that would mean that a plugin for i.MX6 could take 2-3
> > > years to be developed.
> > > 
> > > There's a clear message on it:
> > > 	- we shouldn't keep pushing for a solution via libv4l.  
> > 
> > Or:
> > 
> > 	- userspace plugin development had a very a low priority and
> > 	  never got the attention it needed.
> 
> The end result is the same: we can't count on it.
> 
> > 
> > I know that's *my* reason. I rarely if ever looked at it. I always assumed
> > Sakari and/or Laurent would look at it. If this reason is also valid for
> > Sakari and Laurent, then it is no wonder nothing has happened in all that
> > time.
> > 
> > We're all very driver-development-driven, and userspace gets very little
> > attention in general. So before just throwing in the towel we should take
> > a good look at the reasons why there has been little or no development: is
> > it because of fundamental design defects, or because nobody paid attention
> > to it?
> 
> No. We should look it the other way: basically, there are patches
> for i.MX6 driver that sends control from videonode to subdevs. 
> 
> If we nack apply it, who will write the userspace plugin? When
> such change will be merged upstream?

Well, I believe first question is: what applications would we want to
run on complex devices? Will sending control from video to subdevs
actually help?

mplayer is useful for testing... but that one already works (after you
setup the pipeline, and configure exposure/gain).

But thats useful for testing, not really for production. Image will be
out of focus and with wrong white balance.

What I would really like is an application to get still photos. For
taking pictures with manual settings we need

a) units for controls: user wants to focus on 1m, and take picture
with ISO200, 1/125 sec. We should also tell him that lens is f/5.6 and
focal length is 20mm with 5mm chip.

But... autofocus/autogain would really be good to have. Thus we need:

b) for each frame, we need exposure settings and focus position at
time frame was taken. Otherwise autofocus/autogain will be too
slow. At least focus position is going to be tricky -- either kernel
would have to compute focus position for us (not trivial) or we'd need
enough information to compute it in userspace.

There are more problems: hardware-accelerated preview is not trivial
to set up (and I'm unsure if it can be done in generic way). Still
photos application needs to switch resolutions between preview and
photo capture. Probably hardware-accelerated histograms are needed for
white balance, auto gain and auto focus, ....

It seems like there's a _lot_ of stuff to be done before we have
useful support for complex cameras...

(And I'm not sure... when application such as skype is running, is
there some way to run autogain/autofocus/autowhitebalance? Is that
something we want to support?)

> If we don't have answers to any of the above questions, we should not
> nack it.
> 
> That's said, that doesn't prevent merging a libv4l plugin if/when
> someone can find time/interest to develop it.

I believe other question is: will not having same control on main
video device and subdevs be confusing? Does it actually help userspace
in any way? Yes, we can make controls accessible to old application,
but does it make them more useful? 

> > In addition, I suspect end-users of these complex devices don't really care
> > about a plugin: they want full control and won't typically use generic
> > applications. If they would need support for that, we'd have seen much more
> > interest. The main reason for having a plugin is to simplify testing and
> > if this is going to be used on cheap hobbyist devkits.
> 
> What are the needs for a cheap hobbyist devkit owner? Do we currently
> satisfy those needs? I'd say that having a functional driver when
> compiled without the subdev API, that implements the ioctl's/controls

Having different interface based on config options... is just
weird. What about poor people (like me) trying to develop complex
applications?

> [1] Yet, I might eventually do that for fun, an OMAP3 board with tvp5150
> just arrived here last week. It would be nice to have xawtv3 running on it :-)
> So, if I have a lot of spare time (with is very unlikely), I might eventually 
> do something for it to work.
> 
> > I know it took me a very long time before I had a working omap3.
> 
> My first OMAP3 board with working V4L2 source just arrived last week
> :-)

You can get Nokia N900 on aliexpress. If not, they are still available
between people :-).

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1600971 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

Em Tue, 14 Mar 2017 23:32:54 +0100
Pavel Machek <pavel@ucw.cz> escreveu:

> Hi!
> 
> > > > Even if they were merged, if we keep the same mean time to develop a
> > > > libv4l plugin, that would mean that a plugin for i.MX6 could take 2-3
> > > > years to be developed.
> > > > 
> > > > There's a clear message on it:
> > > > 	- we shouldn't keep pushing for a solution via libv4l.    
> > > 
> > > Or:
> > > 
> > > 	- userspace plugin development had a very a low priority and
> > > 	  never got the attention it needed.  
> > 
> > The end result is the same: we can't count on it.
> >   
> > > 
> > > I know that's *my* reason. I rarely if ever looked at it. I always assumed
> > > Sakari and/or Laurent would look at it. If this reason is also valid for
> > > Sakari and Laurent, then it is no wonder nothing has happened in all that
> > > time.
> > > 
> > > We're all very driver-development-driven, and userspace gets very little
> > > attention in general. So before just throwing in the towel we should take
> > > a good look at the reasons why there has been little or no development: is
> > > it because of fundamental design defects, or because nobody paid attention
> > > to it?  
> > 
> > No. We should look it the other way: basically, there are patches
> > for i.MX6 driver that sends control from videonode to subdevs. 
> > 
> > If we nack apply it, who will write the userspace plugin? When
> > such change will be merged upstream?  
> 
> Well, I believe first question is: what applications would we want to
> run on complex devices? Will sending control from video to subdevs
> actually help?

I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
with those, it will likely run with any other application.

> mplayer is useful for testing... but that one already works (after you
> setup the pipeline, and configure exposure/gain).
> 
> But thats useful for testing, not really for production. Image will be
> out of focus and with wrong white balance.
> 
> What I would really like is an application to get still photos. For
> taking pictures with manual settings we need
> 
> a) units for controls: user wants to focus on 1m, and take picture
> with ISO200, 1/125 sec. We should also tell him that lens is f/5.6 and
> focal length is 20mm with 5mm chip.
> 
> But... autofocus/autogain would really be good to have. Thus we need:
> 
> b) for each frame, we need exposure settings and focus position at
> time frame was taken. Otherwise autofocus/autogain will be too
> slow. At least focus position is going to be tricky -- either kernel
> would have to compute focus position for us (not trivial) or we'd need
> enough information to compute it in userspace.
> 
> There are more problems: hardware-accelerated preview is not trivial
> to set up (and I'm unsure if it can be done in generic way). Still
> photos application needs to switch resolutions between preview and
> photo capture. Probably hardware-accelerated histograms are needed for
> white balance, auto gain and auto focus, ....
> 
> It seems like there's a _lot_ of stuff to be done before we have
> useful support for complex cameras...

Taking still pictures using a hardware-accelerated preview is
a sophisticated use case. I don't know any userspace application
that does that. Ok, several allow taking snapshots, by simply
storing the image of the current frame.

> (And I'm not sure... when application such as skype is running, is
> there some way to run autogain/autofocus/autowhitebalance? Is that
> something we want to support?)

Autofocus no. Autogain/Autowhite can be done via libv4l, provided that
it can access the device's controls via /dev/video devnode. Other
applications may be using some other similar algorithms.

Ok, they don't use histograms provided by the SoC. So, they do it in
software, with is slower. Still, it works fine when the light
conditions don't change too fast.

> > If we don't have answers to any of the above questions, we should not
> > nack it.
> > 
> > That's said, that doesn't prevent merging a libv4l plugin if/when
> > someone can find time/interest to develop it.  
> 
> I believe other question is: will not having same control on main
> video device and subdevs be confusing? Does it actually help userspace
> in any way? Yes, we can make controls accessible to old application,
> but does it make them more useful? 

Yes. As I said, libv4l (and some apps) have logic inside to adjust
the image via bright, contrast and white balance controls, using the
video devnode. They don't talk subdev API. So, if those controls
aren't exported, they won't be able to provide a good quality image.

> > > In addition, I suspect end-users of these complex devices don't really care
> > > about a plugin: they want full control and won't typically use generic
> > > applications. If they would need support for that, we'd have seen much more
> > > interest. The main reason for having a plugin is to simplify testing and
> > > if this is going to be used on cheap hobbyist devkits.  
> > 
> > What are the needs for a cheap hobbyist devkit owner? Do we currently
> > satisfy those needs? I'd say that having a functional driver when
> > compiled without the subdev API, that implements the ioctl's/controls  
> 
> Having different interface based on config options... is just
> weird. What about poor people (like me) trying to develop complex
> applications?

Well, that could be done using other mechanisms, like a modprobe
parameter or by switching the behaviour if a subdev interface is
opened. I don't see much trouble on allowing accessing a control via
both interfaces.

> 
> > [1] Yet, I might eventually do that for fun, an OMAP3 board with tvp5150
> > just arrived here last week. It would be nice to have xawtv3 running on it :-)
> > So, if I have a lot of spare time (with is very unlikely), I might eventually 
> > do something for it to work.
> >   
> > > I know it took me a very long time before I had a working omap3.  
> > 
> > My first OMAP3 board with working V4L2 source just arrived last week
> > :-)  
> 
> You can get Nokia N900 on aliexpress. If not, they are still available
> between people :-)

I have one. Unfortunately, I never had a chance to use it, as the display
stopped working one week after I get it.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1601278 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

On Tue, Mar 14, 2017 at 09:54:31PM -0300, Mauro Carvalho Chehab wrote:
> Em Tue, 14 Mar 2017 23:32:54 +0100
> Pavel Machek <pavel@ucw.cz> escreveu:
> 
> > 
> > Well, I believe first question is: what applications would we want to
> > run on complex devices? Will sending control from video to subdevs
> > actually help?
> 
> I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
> with those, it will likely run with any other application.
> 

I would like to add the 'v4l2src' plugin of gstreamer, and on the imx6 its
imx-specific counterpart 'imxv4l2videosrc' from the gstreamer-imx package
at https://github.com/Freescale/gstreamer-imx, and 'v4l2-ctl'.

Philippe

-- 
Philippe De Muyter +32 2 6101532 Macq SA rue de l'Aeronef 2 B-1140 Bruxelles

[toc] | [prev] | [next] | [standalone]

#1601663 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

[Multipart message — attachments visible in raw view] — view raw

Le mercredi 15 mars 2017 à 11:50 +0100, Philippe De Muyter a écrit :
> > I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
> > with those, it will likely run with any other application.
> > 
> 
> I would like to add the 'v4l2src' plugin of gstreamer, and on the
> imx6 its

While it would be nice if somehow you would get v4l2src to work (in
some legacy/emulation mode through libv4l2), the longer plan is to
implement smart bin that handle several v4l2src, that can do the
required interactions so we can expose similar level of controls as
found in Android Camera HAL3, and maybe even further assuming userspace
can change the media tree at run-time. We might be a long way from
there, specially that some of the features depends on how much the
hardware can do. Just being able to figure-out how to build the MC tree
dynamically seems really hard when thinking of generic mechanism. Also,
Request API will be needed.

I think for this one, we'll need some userspace driver that enable the
features (not hide them), and that's what I'd be looking for from
libv4l2 in this regard.

> imx-specific counterpart 'imxv4l2videosrc' from the gstreamer-imx
> package
> at https://github.com/Freescale/gstreamer-imx, and 'v4l2-ctl'.

This one is specific to IMX hardware using vendor driver. You can
probably ignore that.

Nicolas

[toc] | [prev] | [next] | [standalone]

#1602122 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

On Wed, 2017-03-15 at 14:55 -0400, Nicolas Dufresne wrote:
> Le mercredi 15 mars 2017 à 11:50 +0100, Philippe De Muyter a écrit :
> > > I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
> > > with those, it will likely run with any other application.
> > > 
> > 
> > I would like to add the 'v4l2src' plugin of gstreamer, and on the
> > imx6 its
> 
> While it would be nice if somehow you would get v4l2src to work (in
> some legacy/emulation mode through libv4l2),

v4l2src works just fine, provided the pipeline is configured manually in
advance via media-ctl.

>  the longer plan is to
> implement smart bin that handle several v4l2src, that can do the
> required interactions so we can expose similar level of controls as
> found in Android Camera HAL3, and maybe even further assuming userspace
> can change the media tree at run-time. We might be a long way from
> there, specially that some of the features depends on how much the
> hardware can do. Just being able to figure-out how to build the MC tree
> dynamically seems really hard when thinking of generic mechanism. Also,
> Request API will be needed.
> 
> I think for this one, we'll need some userspace driver that enable the
> features (not hide them), and that's what I'd be looking for from
> libv4l2 in this regard.

regards
Philipp

[toc] | [prev] | [next] | [standalone]

#1602145 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

On Thu, Mar 16, 2017 at 10:26:00AM +0100, Philipp Zabel wrote:
> On Wed, 2017-03-15 at 14:55 -0400, Nicolas Dufresne wrote:
> > Le mercredi 15 mars 2017 à 11:50 +0100, Philippe De Muyter a écrit :
> > > > I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
> > > > with those, it will likely run with any other application.
> > > > 
> > > 
> > > I would like to add the 'v4l2src' plugin of gstreamer, and on the
> > > imx6 its
> > 
> > While it would be nice if somehow you would get v4l2src to work (in
> > some legacy/emulation mode through libv4l2),
> 
> v4l2src works just fine, provided the pipeline is configured manually in
> advance via media-ctl.

Including choosing the framerate ?  Sorry, I have no time these days
to test it myself.

And I cited imxv4l2videosrc for its ability to provide the physical address
of the image buffers for further processing by other (not necessarily next
in gstreamer pipeline, or for all frames) hardware-accelerated plugins likes
the h.264 video encoder.  As I am stuck with fsl/nxp kernel and driver on that
matter, I don't know how the interfaces have evolved in current linux kernels.

BR

Philippe

-- 
Philippe De Muyter +32 2 6101532 Macq SA rue de l'Aeronef 2 B-1140 Bruxelles

[toc] | [prev] | [next] | [standalone]

#1602151 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

On Thu, 2017-03-16 at 10:47 +0100, Philippe De Muyter wrote:
> On Thu, Mar 16, 2017 at 10:26:00AM +0100, Philipp Zabel wrote:
> > On Wed, 2017-03-15 at 14:55 -0400, Nicolas Dufresne wrote:
> > > Le mercredi 15 mars 2017 à 11:50 +0100, Philippe De Muyter a écrit :
> > > > > I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
> > > > > with those, it will likely run with any other application.
> > > > > 
> > > > 
> > > > I would like to add the 'v4l2src' plugin of gstreamer, and on the
> > > > imx6 its
> > > 
> > > While it would be nice if somehow you would get v4l2src to work (in
> > > some legacy/emulation mode through libv4l2),
> > 
> > v4l2src works just fine, provided the pipeline is configured manually in
> > advance via media-ctl.
> 
> Including choosing the framerate ?  Sorry, I have no time these days
> to test it myself.

No, the framerate is set with media-ctl on the CSI output pad. To really
choose the framerate, the element would indeed need a deeper
understanding of the pipeline, as the resulting framerate depends on at
least the source v4l2_subdevice (sensor) framerate and the CSI frame
skipping.

> And I cited imxv4l2videosrc for its ability to provide the physical address
> of the image buffers for further processing by other (not necessarily next
> in gstreamer pipeline, or for all frames) hardware-accelerated plugins likes
> the h.264 video encoder.  As I am stuck with fsl/nxp kernel and driver on that
> matter, I don't know how the interfaces have evolved in current linux kernels.

The physical address of the image buffers is hidden from userspace by
dma-buf objects, but those can be passed around to the next driver
without copying the image data.

regards
Philipp

[toc] | [prev] | [next] | [standalone]

#1602162 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

On Thu, Mar 16, 2017 at 11:01:56AM +0100, Philipp Zabel wrote:
> On Thu, 2017-03-16 at 10:47 +0100, Philippe De Muyter wrote:
> > On Thu, Mar 16, 2017 at 10:26:00AM +0100, Philipp Zabel wrote:
> > > On Wed, 2017-03-15 at 14:55 -0400, Nicolas Dufresne wrote:
> > > > Le mercredi 15 mars 2017 à 11:50 +0100, Philippe De Muyter a écrit :
> > > > > > I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
> > > > > > with those, it will likely run with any other application.
> > > > > > 
> > > > > 
> > > > > I would like to add the 'v4l2src' plugin of gstreamer, and on the
> > > > > imx6 its
> > > > 
> > > > While it would be nice if somehow you would get v4l2src to work (in
> > > > some legacy/emulation mode through libv4l2),
> > > 
> > > v4l2src works just fine, provided the pipeline is configured manually in
> > > advance via media-ctl.
> > 
> > Including choosing the framerate ?  Sorry, I have no time these days
> > to test it myself.
> 
> No, the framerate is set with media-ctl on the CSI output pad. To really
> choose the framerate, the element would indeed need a deeper
> understanding of the pipeline, as the resulting framerate depends on at
> least the source v4l2_subdevice (sensor) framerate and the CSI frame
> skipping.

Count me in than as a supporter of Steve's "v4l2-mc: add a function to
inherit controls from a pipeline" patch

> > of the image buffers for further processing by other (not necessarily next
> > in gstreamer pipeline, or for all frames) hardware-accelerated plugins likes
> > the h.264 video encoder.  As I am stuck with fsl/nxp kernel and driver on that
> > matter, I don't know how the interfaces have evolved in current linux kernels.
> 
> The physical address of the image buffers is hidden from userspace by
> dma-buf objects, but those can be passed around to the next driver
> without copying the image data.

OK

thanks

Philippe

[toc] | [prev] | [next] | [standalone]

#1601608 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

[Multipart message — attachments visible in raw view] — view raw

Hi!

> > Well, I believe first question is: what applications would we want to
> > run on complex devices? Will sending control from video to subdevs
> > actually help?
> 
> I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
> with those, it will likely run with any other application.

I'll take a look when I'm at better internet access.

> > mplayer is useful for testing... but that one already works (after you
> > setup the pipeline, and configure exposure/gain).
> > 
> > But thats useful for testing, not really for production. Image will be
> > out of focus and with wrong white balance.
> > 
> > What I would really like is an application to get still photos. For
> > taking pictures with manual settings we need
> > 
> > a) units for controls: user wants to focus on 1m, and take picture
> > with ISO200, 1/125 sec. We should also tell him that lens is f/5.6 and
> > focal length is 20mm with 5mm chip.
> > 
> > But... autofocus/autogain would really be good to have. Thus we need:
> > 
> > b) for each frame, we need exposure settings and focus position at
> > time frame was taken. Otherwise autofocus/autogain will be too
> > slow. At least focus position is going to be tricky -- either kernel
> > would have to compute focus position for us (not trivial) or we'd need
> > enough information to compute it in userspace.
> > 
> > There are more problems: hardware-accelerated preview is not trivial
> > to set up (and I'm unsure if it can be done in generic way). Still
> > photos application needs to switch resolutions between preview and
> > photo capture. Probably hardware-accelerated histograms are needed for
> > white balance, auto gain and auto focus, ....
> > 
> > It seems like there's a _lot_ of stuff to be done before we have
> > useful support for complex cameras...
> 
> Taking still pictures using a hardware-accelerated preview is
> a sophisticated use case. I don't know any userspace application
> that does that. Ok, several allow taking snapshots, by simply
> storing the image of the current frame.

Well, there are applications that take still pictures. Android has
one. Maemo has another. Then there's fcam-dev. Its open source; with
modified kernel it is fully usable. I have version that runs on recent
nearly-mainline on N900. 

So yes, I'd like solution for problems a) and b).

> > (And I'm not sure... when application such as skype is running, is
> > there some way to run autogain/autofocus/autowhitebalance? Is that
> > something we want to support?)
> 
> Autofocus no. Autogain/Autowhite can be done via libv4l, provided that
> it can access the device's controls via /dev/video devnode. Other
> applications may be using some other similar algorithms.
> 
> Ok, they don't use histograms provided by the SoC. So, they do it in
> software, with is slower. Still, it works fine when the light
> conditions don't change too fast.

I guess it is going to work well enough with higher CPU
usage. Question is if camera without autofocus is usable. I'd say "not
really".

> > I believe other question is: will not having same control on main
> > video device and subdevs be confusing? Does it actually help userspace
> > in any way? Yes, we can make controls accessible to old application,
> > but does it make them more useful? 
> 
> Yes. As I said, libv4l (and some apps) have logic inside to adjust
> the image via bright, contrast and white balance controls, using the
> video devnode. They don't talk subdev API. So, if those controls
> aren't exported, they won't be able to provide a good quality image.

Next question is if the libv4l will do the right thing if we just put
all controls to one node. For example on N900 you have exposure/gain
and brightness. But the brightness is applied at preview phase, so it
is "basically useless". You really need to adjust the image using the
exposure/gain.

> > > > In addition, I suspect end-users of these complex devices don't really care
> > > > about a plugin: they want full control and won't typically use generic
> > > > applications. If they would need support for that, we'd have seen much more
> > > > interest. The main reason for having a plugin is to simplify testing and
> > > > if this is going to be used on cheap hobbyist devkits.  
> > > 
> > > What are the needs for a cheap hobbyist devkit owner? Do we currently
> > > satisfy those needs? I'd say that having a functional driver when
> > > compiled without the subdev API, that implements the ioctl's/controls  
> > 
> > Having different interface based on config options... is just
> > weird. What about poor people (like me) trying to develop complex
> > applications?
> 
> Well, that could be done using other mechanisms, like a modprobe
> parameter or by switching the behaviour if a subdev interface is
> opened. I don't see much trouble on allowing accessing a control via
> both interfaces.

If we really want to go that way (is not modifying library to access
the right files quite easy?), I believe non-confusing option would be
to have '/dev/video0 -- omap3 camera for legacy applications' which
would include all the controls.

> > You can get Nokia N900 on aliexpress. If not, they are still available
> > between people :-)
> 
> I have one. Unfortunately, I never had a chance to use it, as the display
> stopped working one week after I get it.

Well, I guess the easiest option is to just get another one :-).

But otoh -- N900 is quite usable without the screen. 0xffff tool can
be used to boot the kernel, then you can use nfsroot and usb
networking. It also has serial port (over strange
connector). Connected over ssh over usb network is actually how I do
most of the v4l work.

Best regards,

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1601724 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

Em Wed, 15 Mar 2017 19:04:21 +0100
Pavel Machek <pavel@ucw.cz> escreveu:

> Hi!
> 
> > > Well, I believe first question is: what applications would we want to
> > > run on complex devices? Will sending control from video to subdevs
> > > actually help?  
> > 
> > I would say: camorama, xawtv3, zbar, google talk, skype. If it runs
> > with those, it will likely run with any other application.  
> 
> I'll take a look when I'm at better internet access.

Ok.

> > > mplayer is useful for testing... but that one already works (after you
> > > setup the pipeline, and configure exposure/gain).
> > > 
> > > But thats useful for testing, not really for production. Image will be
> > > out of focus and with wrong white balance.
> > > 
> > > What I would really like is an application to get still photos. For
> > > taking pictures with manual settings we need
> > > 
> > > a) units for controls: user wants to focus on 1m, and take picture
> > > with ISO200, 1/125 sec. We should also tell him that lens is f/5.6 and
> > > focal length is 20mm with 5mm chip.
> > > 
> > > But... autofocus/autogain would really be good to have. Thus we need:
> > > 
> > > b) for each frame, we need exposure settings and focus position at
> > > time frame was taken. Otherwise autofocus/autogain will be too
> > > slow. At least focus position is going to be tricky -- either kernel
> > > would have to compute focus position for us (not trivial) or we'd need
> > > enough information to compute it in userspace.
> > > 
> > > There are more problems: hardware-accelerated preview is not trivial
> > > to set up (and I'm unsure if it can be done in generic way). Still
> > > photos application needs to switch resolutions between preview and
> > > photo capture. Probably hardware-accelerated histograms are needed for
> > > white balance, auto gain and auto focus, ....
> > > 
> > > It seems like there's a _lot_ of stuff to be done before we have
> > > useful support for complex cameras...  
> > 
> > Taking still pictures using a hardware-accelerated preview is
> > a sophisticated use case. I don't know any userspace application
> > that does that. Ok, several allow taking snapshots, by simply
> > storing the image of the current frame.  
> 
> Well, there are applications that take still pictures. Android has
> one. Maemo has another. Then there's fcam-dev. Its open source; with
> modified kernel it is fully usable. I have version that runs on recent
> nearly-mainline on N900. 

Hmm... it seems that FCam is specific for N900:
	http://fcam.garage.maemo.org/

If so, then we have here just the opposite problem, if want it to be
used as a generic application, as very likely it requires OMAP3-specific
graph/subdevs.

> So yes, I'd like solution for problems a) and b).
> 
> > > (And I'm not sure... when application such as skype is running, is
> > > there some way to run autogain/autofocus/autowhitebalance? Is that
> > > something we want to support?)  
> > 
> > Autofocus no. Autogain/Autowhite can be done via libv4l, provided that
> > it can access the device's controls via /dev/video devnode. Other
> > applications may be using some other similar algorithms.
> > 
> > Ok, they don't use histograms provided by the SoC. So, they do it in
> > software, with is slower. Still, it works fine when the light
> > conditions don't change too fast.  
> 
> I guess it is going to work well enough with higher CPU
> usage.

Yes.

> Question is if camera without autofocus is usable. I'd say "not
> really".qv4l2

That actually depends on the sensor and how focus is adjusted.

I'm testing right now this camera module for RPi:
   https://www.raspberrypi.org/products/camera-module-v2/

I might be wrong, but this sensor doesn't seem to have auto-focus.
Instead, it seems to use a wide-angle lens. So, except when the
object is too close, the focus look OK.

> > > I believe other question is: will not having same control on main
> > > video device and subdevs be confusing? Does it actually help userspace
> > > in any way? Yes, we can make controls accessible to old application,
> > > but does it make them more useful?   
> > 
> > Yes. As I said, libv4l (and some apps) have logic inside to adjust
> > the image via bright, contrast and white balance controls, using the
> > video devnode. They don't talk subdev API. So, if those controls
> > aren't exported, they won't be able to provide a good quality image.  
> 
> Next question is if the libv4l will do the right thing if we just put
> all controls to one node. For example on N900 you have exposure/gain
> and brightness. But the brightness is applied at preview phase, so it
> is "basically useless". You really need to adjust the image using the
> exposure/gain.

I've no idea, but I suspect it shouldn't be hard to teach libv4l to
prefer use an exposure/gain instead of brightness when available.

> > > > > In addition, I suspect end-users of these complex devices don't really care
> > > > > about a plugin: they want full control and won't typically use generic
> > > > > applications. If they would need support for that, we'd have seen much more
> > > > > interest. The main reason for having a plugin is to simplify testing and
> > > > > if this is going to be used on cheap hobbyist devkits.    
> > > > 
> > > > What are the needs for a cheap hobbyist devkit owner? Do we currently
> > > > satisfy those needs? I'd say that having a functional driver when
> > > > compiled without the subdev API, that implements the ioctl's/controls    
> > > 
> > > Having different interface based on config options... is just
> > > weird. What about poor people (like me) trying to develop complex
> > > applications?  
> > 
> > Well, that could be done using other mechanisms, like a modprobe
> > parameter or by switching the behaviour if a subdev interface is
> > opened. I don't see much trouble on allowing accessing a control via
> > both interfaces.  
> 
> If we really want to go that way (is not modifying library to access
> the right files quite easy?), I believe non-confusing option would be
> to have '/dev/video0 -- omap3 camera for legacy applications' which
> would include all the controls.

Yeah, keeping /dev/video0 reserved for generic applications is something
that could work. Not sure how easy would be to implement it.

> 
> > > You can get Nokia N900 on aliexpress. If not, they are still available
> > > between people :-)  
> > 
> > I have one. Unfortunately, I never had a chance to use it, as the display
> > stopped working one week after I get it.  
> 
> Well, I guess the easiest option is to just get another one :-).

:-)  Well, I guess very few units of N900 was sold in Brazil. Importing
one is too expensive, due to taxes.

> But otoh -- N900 is quite usable without the screen. 0xffff tool can
> be used to boot the kernel, then you can use nfsroot and usb
> networking. It also has serial port (over strange
> connector). Connected over ssh over usb network is actually how I do
> most of the v4l work.

If you pass me the pointers, I can try it when I have some time.

Anyway, I got myself an ISEE IGEPv2, with the expansion board:
	https://www.isee.biz/products/igep-processor-boards/igepv2-dm3730
	https://www.isee.biz/products/igep-expansion-boards/igepv2-expansion

The expansion board comes with a tvp5150 analog TV demod. So, with
this device, I can simply connect it to a composite input signal.
I have some sources here that I can use to test it.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1602873 — Re: media / v4l2-mc: wishlist for complex cameras (was Re: [PATCH v4 14/36] [media] v4l2-mc: add a function to inherit controls from a pipeline)

[Multipart message — attachments visible in raw view] — view raw

Hi!

> > > > mplayer is useful for testing... but that one already works (after you
> > > > setup the pipeline, and configure exposure/gain).
> > > > 
> > > > But thats useful for testing, not really for production. Image will be
> > > > out of focus and with wrong white balance.
> > > > 
> > > > What I would really like is an application to get still photos. For
> > > > taking pictures with manual settings we need
> > > > 
> > > > a) units for controls: user wants to focus on 1m, and take picture
> > > > with ISO200, 1/125 sec. We should also tell him that lens is f/5.6 and
> > > > focal length is 20mm with 5mm chip.
> > > > 
> > > > But... autofocus/autogain would really be good to have. Thus we need:
> > > > 
> > > > b) for each frame, we need exposure settings and focus position at
> > > > time frame was taken. Otherwise autofocus/autogain will be too
> > > > slow. At least focus position is going to be tricky -- either kernel
> > > > would have to compute focus position for us (not trivial) or we'd need
> > > > enough information to compute it in userspace.
> > > > 
> > > > There are more problems: hardware-accelerated preview is not trivial
> > > > to set up (and I'm unsure if it can be done in generic way). Still
> > > > photos application needs to switch resolutions between preview and
> > > > photo capture. Probably hardware-accelerated histograms are needed for
> > > > white balance, auto gain and auto focus, ....
> > > > 
> > > > It seems like there's a _lot_ of stuff to be done before we have
> > > > useful support for complex cameras...  
> > > 
> > > Taking still pictures using a hardware-accelerated preview is
> > > a sophisticated use case. I don't know any userspace application
> > > that does that. Ok, several allow taking snapshots, by simply
> > > storing the image of the current frame.  
> > 
> > Well, there are applications that take still pictures. Android has
> > one. Maemo has another. Then there's fcam-dev. Its open source; with
> > modified kernel it is fully usable. I have version that runs on recent
> > nearly-mainline on N900. 
> 
> Hmm... it seems that FCam is specific for N900:
> 	http://fcam.garage.maemo.org/
> 
> If so, then we have here just the opposite problem, if want it to be
> used as a generic application, as very likely it requires OMAP3-specific
> graph/subdevs.

Well... there's quick and great version on maemo.org. I do have local
version (still somehow N900-specific), but it no longer uses hardware
histogram/sharpness support. Should be almost generic.

> > So yes, I'd like solution for problems a) and b).

...but it has camera parameters hardcoded (problem a) and slow
(problem b). 

> > Question is if camera without autofocus is usable. I'd say "not
> > really".qv4l2
> 
> That actually depends on the sensor and how focus is adjusted.
> 
> I'm testing right now this camera module for RPi:
>    https://www.raspberrypi.org/products/camera-module-v2/
> 
> I might be wrong, but this sensor doesn't seem to have auto-focus.
> Instead, it seems to use a wide-angle lens. So, except when the
> object is too close, the focus look OK.

Well, cameras without autofocus are somehow usable without
autofocus. But cameras with autofocus don't work too well without one.

> > If we really want to go that way (is not modifying library to access
> > the right files quite easy?), I believe non-confusing option would be
> > to have '/dev/video0 -- omap3 camera for legacy applications' which
> > would include all the controls.
> 
> Yeah, keeping /dev/video0 reserved for generic applications is something
> that could work. Not sure how easy would be to implement it.

Plus advanced applications would just ignore /dev/video0.. and not be confused.

> > > > > > You can get Nokia N900 on aliexpress. If not, they are still available
> > > > between people :-)  
> > > 
> > > I have one. Unfortunately, I never had a chance to use it, as the display
> > > stopped working one week after I get it.  
> > 
> > Well, I guess the easiest option is to just get another one :-).
> 
> :-)  Well, I guess very few units of N900 was sold in Brazil. Importing
> one is too expensive, due to taxes.

Try to ask at local mailing list. Those machines were quite common.

> > But otoh -- N900 is quite usable without the screen. 0xffff tool can
> > be used to boot the kernel, then you can use nfsroot and usb
> > networking. It also has serial port (over strange
> > connector). Connected over ssh over usb network is actually how I do
> > most of the v4l work.
> 
> If you pass me the pointers, I can try it when I have some time.

Ok, I guess I'll do that in private email.

> Anyway, I got myself an ISEE IGEPv2, with the expansion board:
> 	https://www.isee.biz/products/igep-processor-boards/igepv2-dm3730
> 	https://www.isee.biz/products/igep-expansion-boards/igepv2-expansion
> 
> The expansion board comes with a tvp5150 analog TV demod. So, with
> this device, I can simply connect it to a composite input signal.
> I have some sources here that I can use to test it.

Well... it looks like TV capture is a "solved" problem. Taking useful
photos is what is hard...


									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1604568

On 03/14/2017 11:21 AM, Mauro Carvalho Chehab wrote:
> Em Tue, 14 Mar 2017 08:55:36 +0100
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> 
>> On 03/14/2017 04:45 AM, Mauro Carvalho Chehab wrote:
>>> Hi Sakari,
>>>
>>> I started preparing a long argument about it, but gave up in favor of a
>>> simpler one.
>>>
>>> Em Mon, 13 Mar 2017 14:46:22 +0200
>>> Sakari Ailus <sakari.ailus@iki.fi> escreveu:
>>>   
>>>> Drivers are written to support hardware, not particular use case.    
>>>
>>> No, it is just the reverse: drivers and hardware are developed to
>>> support use cases.
>>>
>>> Btw, you should remember that the hardware is the full board, not just the
>>> SoC. In practice, the board do limit the use cases: several provide a
>>> single physical CSI connector, allowing just one sensor.
>>>   
>>>>> This situation is there since 2009. If I remember well, you tried to write
>>>>> such generic plugin in the past, but never finished it, apparently because
>>>>> it is too complex. Others tried too over the years.     
>>>>
>>>> I'd argue I know better what happened with that attempt than you do. I had a
>>>> prototype of a generic pipeline configuration library but due to various
>>>> reasons I haven't been able to continue working on that since around 2012.  
>>>
>>> ...
>>>   
>>>>> The last trial was done by Jacek, trying to cover just the exynos4 driver. 
>>>>> Yet, even such limited scope plugin was not good enough, as it was never
>>>>> merged upstream. Currently, there's no such plugins upstream.
>>>>>
>>>>> If we can't even merge a plugin that solves it for just *one* driver,
>>>>> I have no hope that we'll be able to do it for the generic case.    
>>>>
>>>> I believe Jacek ceased to work on that plugin in his day job; other than
>>>> that, there are some matters left to be addressed in his latest patchset.  
>>>
>>> The two above basically summaries the issue: the task of doing a generic
>>> plugin on userspace, even for a single driver is complex enough to
>>> not cover within a reasonable timeline.
>>>
>>> From 2009 to 2012, you were working on it, but didn't finish it.
>>>
>>> Apparently, nobody worked on it between 2013-2014 (but I may be wrong, as
>>> I didn't check when the generic plugin interface was added to libv4l).
>>>
>>> In the case of Jacek's work, the first patch I was able to find was
>>> written in Oct, 2014:
>>> 	https://patchwork.kernel.org/patch/5098111/
>>> 	(not sure what happened with the version 1).
>>>
>>> The last e-mail about this subject was issued in Dec, 2016.
>>>
>>> In summary, you had this on your task for 3 years for an OMAP3
>>> plugin (where you have a good expertise), and Jacek for 2 years, 
>>> for Exynos 4, where he should also have a good knowledge.
>>>
>>> Yet, with all that efforts, no concrete results were achieved, as none
>>> of the plugins got merged.
>>>
>>> Even if they were merged, if we keep the same mean time to develop a
>>> libv4l plugin, that would mean that a plugin for i.MX6 could take 2-3
>>> years to be developed.
>>>
>>> There's a clear message on it:
>>> 	- we shouldn't keep pushing for a solution via libv4l.  
>>
>> Or:
>>
>> 	- userspace plugin development had a very a low priority and
>> 	  never got the attention it needed.
> 
> The end result is the same: we can't count on it.
> 
>>
>> I know that's *my* reason. I rarely if ever looked at it. I always assumed
>> Sakari and/or Laurent would look at it. If this reason is also valid for
>> Sakari and Laurent, then it is no wonder nothing has happened in all that
>> time.
>>
>> We're all very driver-development-driven, and userspace gets very little
>> attention in general. So before just throwing in the towel we should take
>> a good look at the reasons why there has been little or no development: is
>> it because of fundamental design defects, or because nobody paid attention
>> to it?
> 
> No. We should look it the other way: basically, there are patches
> for i.MX6 driver that sends control from videonode to subdevs. 
> 
> If we nack apply it, who will write the userspace plugin? When
> such change will be merged upstream?
> 
> If we don't have answers to any of the above questions, we should not
> nack it.
> 
> That's said, that doesn't prevent merging a libv4l plugin if/when
> someone can find time/interest to develop it.

I don't think this control inheritance patch will magically prevent you
from needed a plugin.

> 
>> I strongly suspect it is the latter.
>>
>> In addition, I suspect end-users of these complex devices don't really care
>> about a plugin: they want full control and won't typically use generic
>> applications. If they would need support for that, we'd have seen much more
>> interest. The main reason for having a plugin is to simplify testing and
>> if this is going to be used on cheap hobbyist devkits.
> 
> What are the needs for a cheap hobbyist devkit owner? Do we currently
> satisfy those needs? I'd say that having a functional driver when
> compiled without the subdev API, that implements the ioctl's/controls
> that a generic application like camorama/google talk/skype/zbar...
> would work should be enough to make them happy, even if they need to
> add some udev rule and/or run some "prep" application that would setup 
> the pipelines via MC and eventually rename the device with a working
> pipeline to /dev/video0.

This is literally the first time we have to cater to a cheap devkit. We
were always aware of this issue, but nobody really needed it.

> 
>>
>> An additional complication is simply that it is hard to find fully supported
>> MC hardware. omap3 boards are hard to find these days, renesas boards are not
>> easy to get, freescale isn't the most popular either. Allwinner, mediatek,
>> amlogic, broadcom and qualcomm all have closed source implementations or no
>> implementation at all.
> 
> I'd say that we should not care anymore on providing a solution for
> generic applications to run on boards like OMAP3[1]. For hardware that
> are currently available that have Kernel driver and boards developed
> to be used as "cheap hobbyist devkit", I'd say we should implement
> a Kernel solution that would allow them to be used without subdev
> API, e. g. having all ioctls needed by generic applications to work
> functional, after some external application sets the pipeline.

I liked Russell's idea of having the DT set up an initial video path.
This would (probably) make it much easier to provide a generic plugin since
there is already an initial valid path when the driver is loaded, and it
doesn't require custom code in the driver since this is left to the DT
which really knows about the HW.

> 
> [1] Yet, I might eventually do that for fun, an OMAP3 board with tvp5150
> just arrived here last week. It would be nice to have xawtv3 running on it :-)
> So, if I have a lot of spare time (with is very unlikely), I might eventually 
> do something for it to work.
> 
>> I know it took me a very long time before I had a working omap3.
> 
> My first OMAP3 board with working V4L2 source just arrived last week :-)
> 
>> So I am not at all surprised that little progress has been made.
> 
> I'm not surprised, but I'm disappointed, as I tried to push toward a
> solution for this problem since when we had our initial meetings about
> it.

So many things to do, so little time. Sounds corny, but really, that's what
this is all about. There were always (and frankly, still are) more important
things that needed to be done.

Regards,

	Hans

[toc] | [prev] | [next] | [standalone]

#1604702

Em Mon, 20 Mar 2017 14:24:25 +0100
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> On 03/14/2017 11:21 AM, Mauro Carvalho Chehab wrote:
> > Em Tue, 14 Mar 2017 08:55:36 +0100
> > Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> >   
> >> On 03/14/2017 04:45 AM, Mauro Carvalho Chehab wrote:  
> >>> Hi Sakari,
> >>>

> >> We're all very driver-development-driven, and userspace gets very little
> >> attention in general. So before just throwing in the towel we should take
> >> a good look at the reasons why there has been little or no development: is
> >> it because of fundamental design defects, or because nobody paid attention
> >> to it?  
> > 
> > No. We should look it the other way: basically, there are patches
> > for i.MX6 driver that sends control from videonode to subdevs. 
> > 
> > If we nack apply it, who will write the userspace plugin? When
> > such change will be merged upstream?
> > 
> > If we don't have answers to any of the above questions, we should not
> > nack it.
> > 
> > That's said, that doesn't prevent merging a libv4l plugin if/when
> > someone can find time/interest to develop it.  
> 
> I don't think this control inheritance patch will magically prevent you
> from needed a plugin.

Yeah, it is not just control inheritance. The driver needs to work
without subdev API, e. g. mbus settings should also be done via the
video devnode.

Btw, Sakari made a good point on IRC: what happens if some app 
try to change the pipeline or subdev settings while another
application is using the device? The driver should block such 
changes, maybe using the V4L2 priority mechanism.

> This is literally the first time we have to cater to a cheap devkit. We
> were always aware of this issue, but nobody really needed it.

That's true. Now that we have a real need for that, we need to
provide a solution.

> > I'd say that we should not care anymore on providing a solution for
> > generic applications to run on boards like OMAP3[1]. For hardware that
> > are currently available that have Kernel driver and boards developed
> > to be used as "cheap hobbyist devkit", I'd say we should implement
> > a Kernel solution that would allow them to be used without subdev
> > API, e. g. having all ioctls needed by generic applications to work
> > functional, after some external application sets the pipeline.  
> 
> I liked Russell's idea of having the DT set up an initial video path.
> This would (probably) make it much easier to provide a generic plugin since
> there is already an initial valid path when the driver is loaded, and it
> doesn't require custom code in the driver since this is left to the DT
> which really knows about the HW.

Setting the device via DT indeed makes easier (either for a kernel
or userspace solution), but things like resolution changes should
be possible without needing to change DT.

Also, as MC and subdev changes should be blocked while a V4L2 app
is using the device, using a mechanism like calling VIDIOC_S_PRIORITY
ioctl via the V4l2 interface, Kernel will require changes, anyway.

My suggestion is to touch on one driver to make it work with a
generic application. As we currently have efforts and needs for
the i.MX6 to do it, I'd say that the best would be to make it
work on such driver. Once the work is done, we can see if the
approach taken there would work at libv4l or not.

In parallel, someone has to fix libv4l for it to be default on
applications like gstreamer, e. g. adding support for DMABUF
and fixing other issues that are preventing it to be used by
default.

Nicolas,

Why libv4l is currently disabled at gstreamer's default settings?

> > [1] Yet, I might eventually do that for fun, an OMAP3 board with tvp5150
> > just arrived here last week. It would be nice to have xawtv3 running on it :-)
> > So, if I have a lot of spare time (with is very unlikely), I might eventually 
> > do something for it to work.
> >   
> >> I know it took me a very long time before I had a working omap3.  
> > 
> > My first OMAP3 board with working V4L2 source just arrived last week :-)
> >   
> >> So I am not at all surprised that little progress has been made.  
> > 
> > I'm not surprised, but I'm disappointed, as I tried to push toward a
> > solution for this problem since when we had our initial meetings about
> > it.  
> 
> So many things to do, so little time. Sounds corny, but really, that's what
> this is all about. There were always (and frankly, still are) more important
> things that needed to be done.

What's most important for some developer may not be so important for
another developer.

My understanding here is that there are developers wanting/needing
to have standard V4L2 apps support for (some) i.MX6 devices. Those are
the ones that may/will allocate some time for it to happen.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1604735

On Mon, Mar 20, 2017 at 12:39:38PM -0300, Mauro Carvalho Chehab wrote:
> Em Mon, 20 Mar 2017 14:24:25 +0100
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> > I don't think this control inheritance patch will magically prevent you
> > from needed a plugin.
> 
> Yeah, it is not just control inheritance. The driver needs to work
> without subdev API, e. g. mbus settings should also be done via the
> video devnode.
> 
> Btw, Sakari made a good point on IRC: what happens if some app 
> try to change the pipeline or subdev settings while another
> application is using the device? The driver should block such 
> changes, maybe using the V4L2 priority mechanism.

My understanding is that there are already mechanisms in place to
prevent that, but it's driver dependent - certainly several of the
imx driver subdevs check whether they have an active stream, and
refuse (eg) all set_fmt calls with -EBUSY if that is so.

(That statement raises another question in my mind: if the subdev is
streaming, should it refuse all set_fmt, even for the TRY stuff?)

> In parallel, someone has to fix libv4l for it to be default on
> applications like gstreamer, e. g. adding support for DMABUF
> and fixing other issues that are preventing it to be used by
> default.

Hmm, not sure what you mean there - I've used dmabuf with gstreamer's
v4l2src linked to libv4l2, importing the buffers into etnaviv using
a custom plugin.  There are distros around (ubuntu) where the v4l2
plugin is built against libv4l2.

> My understanding here is that there are developers wanting/needing
> to have standard V4L2 apps support for (some) i.MX6 devices. Those are
> the ones that may/will allocate some time for it to happen.

Quite - but we need to first know what is acceptable to the v4l2
community before we waste a lot of effort coding something up that
may not be suitable.  Like everyone else, there's only a limited
amount of effort available, so using it wisely is a very good idea.

Up until recently, it seemed that the only solution was to solve the
userspace side of the media controller API via v4l2 plugins and the
like.

Much of my time that I have available to look at the imx6 capture
stuff at the moment is taken up by triping over UAPI issues with the
current code (such as the ones about CSI scaling, colorimetry, etc)
and trying to get concensus on what the right solution to fix those
issues actually is, and at the moment I don't have spare time to
start addressing any kind of v4l2 plugin for user controls nor any
other solution.

Eg, I spent much of this last weekend sorting out my IMX219 camera
sensor driver for my new understanding about how scaling is supposed
to work, the frame enumeration issue (which I've posted patches for)
and the CSI scaling issue (which I've some half-baked patch for at the
moment, but probably by the time I've finished sorting that, Philipp
or Steve will already have a solution.)

That said, my new understanding of the scaling impacts the four patches
I posted, and probably makes the frame size enumeration in CSI (due to
its scaling) rather obsolete.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1604838

Em Mon, 20 Mar 2017 16:10:03 +0000
Russell King - ARM Linux <linux@armlinux.org.uk> escreveu:

> On Mon, Mar 20, 2017 at 12:39:38PM -0300, Mauro Carvalho Chehab wrote:
> > Em Mon, 20 Mar 2017 14:24:25 +0100
> > Hans Verkuil <hverkuil@xs4all.nl> escreveu:  
> > > I don't think this control inheritance patch will magically prevent you
> > > from needed a plugin.  
> > 
> > Yeah, it is not just control inheritance. The driver needs to work
> > without subdev API, e. g. mbus settings should also be done via the
> > video devnode.
> > 
> > Btw, Sakari made a good point on IRC: what happens if some app 
> > try to change the pipeline or subdev settings while another
> > application is using the device? The driver should block such 
> > changes, maybe using the V4L2 priority mechanism.  
> 
> My understanding is that there are already mechanisms in place to
> prevent that, but it's driver dependent - certainly several of the
> imx driver subdevs check whether they have an active stream, and
> refuse (eg) all set_fmt calls with -EBUSY if that is so.
> 
> (That statement raises another question in my mind: if the subdev is
> streaming, should it refuse all set_fmt, even for the TRY stuff?)

Returning -EBUSY only when streaming is too late, as ioctl's
may be changing the pipeline configuration and/or buffer allocation,
while the application is sending other ioctls in order to prepare
for streaming.

V4L2 has a mechanism of blocking other apps to change such
parameters via VIDIOC_S_PRIORITY[1]. If an application sets
priority to V4L2_PRIORITY_RECORD, any other application attempting
to change the device via some other file descriptor will fail.
So, it is a sort of "exclusive write access".

On a quick look at V4L2 core, currently, sending a 
VIDIOC_S_PRIORITY ioctl to a /dev/video device doesn't seem to have
any effect at either MC or V4L2 subdev API for the subdevs connected
to it. We'll likely need to add some code at v4l2_prio_change()
for it to notify the subdevs for them to return -EBUSY if one
would try to change something there, while the device is priorized.

[1] https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/vidioc-g-priority.html

> > In parallel, someone has to fix libv4l for it to be default on
> > applications like gstreamer, e. g. adding support for DMABUF
> > and fixing other issues that are preventing it to be used by
> > default.  
> 
> Hmm, not sure what you mean there - I've used dmabuf with gstreamer's
> v4l2src linked to libv4l2, importing the buffers into etnaviv using
> a custom plugin.  There are distros around (ubuntu) where the v4l2
> plugin is built against libv4l2.

Hmm... I guess some gst developer mentioned that there are/where
some restrictions at libv4l2 with regards to DMABUF. I may be
wrong.

> 
> > My understanding here is that there are developers wanting/needing
> > to have standard V4L2 apps support for (some) i.MX6 devices. Those are
> > the ones that may/will allocate some time for it to happen.  
> 
> Quite - but we need to first know what is acceptable to the v4l2
> community before we waste a lot of effort coding something up that
> may not be suitable.  Like everyone else, there's only a limited
> amount of effort available, so using it wisely is a very good idea.

Sure. That's why we're discussing here :-)

> Up until recently, it seemed that the only solution was to solve the
> userspace side of the media controller API via v4l2 plugins and the
> like.
> 
> Much of my time that I have available to look at the imx6 capture
> stuff at the moment is taken up by triping over UAPI issues with the
> current code (such as the ones about CSI scaling, colorimetry, etc)
> and trying to get concensus on what the right solution to fix those
> issues actually is, and at the moment I don't have spare time to
> start addressing any kind of v4l2 plugin for user controls nor any
> other solution.

I hear you. A solution via libv4l could be more elegant, but it
doesn't seem simple, as nobody did it before, and depends on the
libv4l plugin mechanism, with is currently unused.

Also, I think it is easier to provide a solution using DT and some
driver and/or core support for it, specially since, AFAICT,
currently there's no way request exclusive access to MC and subdevs.

It is probably not possible do to that exclusively in userspace.

> Eg, I spent much of this last weekend sorting out my IMX219 camera
> sensor driver for my new understanding about how scaling is supposed
> to work, the frame enumeration issue (which I've posted patches for)
> and the CSI scaling issue (which I've some half-baked patch for at the
> moment, but probably by the time I've finished sorting that, Philipp
> or Steve will already have a solution.)
> 
> That said, my new understanding of the scaling impacts the four patches
> I posted, and probably makes the frame size enumeration in CSI (due to
> its scaling) rather obsolete.

Yeah, when there's no scaler, it should report just the resolution(s)
supported by the sensor (actually, at the CSI) via
V4L2_FRMSIZE_TYPE_DISCRETE.

However, when there's a scaler at the pipeline, it should report the
range supported by the scaler, e. g.:

- V4L2_FRMSIZE_TYPE_CONTINUOUS - when an entire range of resolutions
  is supported with step = 1 for both H and V .

- V4L2_FRMSIZE_TYPE_STEPWISE - when there's either a H or V step at
  the possible values for resolutions. This is actually more common
  in practice, as several encodings take a 2x2 pixel block. So, the
  step should be at least 2.

There is something to be considered by the logic that forwards the
resolution to the CSI: a lower resolution there means a higher number of
frames per second.

So, if the driver is setting the resolution via a V4L2 device, it
will provide a higher number of fps if it selects the lowest 
resolution at CSI that it is greater or equal to the resolution
set at the scaler. On the other hand, the image quality could be 
better if it doesn't scale at CSI.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1603234

On Tue, Mar 14, 2017 at 08:55:36AM +0100, Hans Verkuil wrote:
> We're all very driver-development-driven, and userspace gets very little
> attention in general. So before just throwing in the towel we should take
> a good look at the reasons why there has been little or no development: is
> it because of fundamental design defects, or because nobody paid attention
> to it?
> 
> I strongly suspect it is the latter.
> 
> In addition, I suspect end-users of these complex devices don't really care
> about a plugin: they want full control and won't typically use generic
> applications. If they would need support for that, we'd have seen much more
> interest. The main reason for having a plugin is to simplify testing and
> if this is going to be used on cheap hobbyist devkits.

I think you're looking at it with a programmers hat on, not a users hat.

Are you really telling me that requiring users to 'su' to root, and then
use media-ctl to manually configure the capture device is what most
users "want" ?

Hasn't the way technology has moved towards graphical interfaces,
particularly smart phones, taught us that the vast majority of users
want is intuitive, easy to use interfaces, and not the command line
with reams of documentation?

Why are smart phones soo popular - it's partly because they're flashy,
but also because of the wealth of apps, and apps which follow the
philosophy of "do one job, do it well" (otherwise they get bad reviews.)

> An additional complication is simply that it is hard to find fully supported
> MC hardware. omap3 boards are hard to find these days, renesas boards are not
> easy to get, freescale isn't the most popular either. Allwinner, mediatek,
> amlogic, broadcom and qualcomm all have closed source implementations or no
> implementation at all.

Right, and that in itself tells us something - the problem that we're
trying to solve is not one that commonly exists in the real world.

Yes, the hardware we have in front of us may be very complex, but if
there's very few systems out there which are capable of making use of
all that complexity, then we're trying to solve a problem that isn't
the common case - and if it's going to take years to solve it (it
already has taken years) then it's the wrong problem to be solved.

I bet most of the problem can be eliminated if, rather than exposing
all this complexity, we instead expose a simpler capture system where
the board designer gets to "wire up" the capture system.

I'll go back to my Bayer example, because that's the simplest.  As
I've already said many times in these threads, there is only one
possible path through the iMX6 device that such a source can be used
with - it's a fixed path.  The actual path depends on the CSI2
virtual channel that the camera has been _configured_ to use, but
apart from that, it's effectively a well known set of blocks.  Such
a configuration could be placed in DT.

For RGB connected to a single parallel CSI, things get a little more
complex - capture through the CSI or through two other capture devices
for de-interlacing or other features.  However, I'm not convinced that
exposing multiple /dev/video* devices for different features for the
same video source is a sane approach - I think that's a huge usability
problem.  (The user is expected to select the capture device on iMX6
depending on the features they want, and if they want to change features,
they're expected to shut down their application and start it up on a
different capture device.)  For the most part on iMX6, there's one
path down to the CSI block, and then there's optional routing through
the rest of the IPU depending on what features you want (such as
de-interlacing.)

The complex case is a CSI2 connected camera which produces multiple
streams through differing virtual channels - and that's IMHO the only
case where we need multiple different /dev/video* capture devices to
be present.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1603239

Hi Russell,

On 03/17/17 13:42, Russell King - ARM Linux wrote:
> On Tue, Mar 14, 2017 at 08:55:36AM +0100, Hans Verkuil wrote:
>> We're all very driver-development-driven, and userspace gets very little
>> attention in general. So before just throwing in the towel we should take
>> a good look at the reasons why there has been little or no development: is
>> it because of fundamental design defects, or because nobody paid attention
>> to it?
>>
>> I strongly suspect it is the latter.
>>
>> In addition, I suspect end-users of these complex devices don't really care
>> about a plugin: they want full control and won't typically use generic
>> applications. If they would need support for that, we'd have seen much more
>> interest. The main reason for having a plugin is to simplify testing and
>> if this is going to be used on cheap hobbyist devkits.
> 
> I think you're looking at it with a programmers hat on, not a users hat.
> 
> Are you really telling me that requiring users to 'su' to root, and then
> use media-ctl to manually configure the capture device is what most
> users "want" ?

It depends on who the user is. I don't think anyone is suggesting a
regular end user is the user of all these APIs: it is either an
application tailored for that given device, a skilled user with his test
scripts or as suggested previously, a libv4l plugin knowing the device
or a generic library geared towards providing best effort service. The
last one of this list does not exist yet and the second last item
requires help.

Typically this class of devices is simply not up to provide the level of
service you're requesting without additional user space control library
which is responsible for automatic white balance, exposure and focus.

Making use of the full potential of the hardware requires using a more
expressive interface. That's what the kernel interface must provide. If
we decide to limit ourselves to a small sub-set of that potential on the
level of the kernel interface, we have made a wrong decision. It's as
simple as that. This is why the functionality (and which requires taking
a lot of policy decisions) belongs to the user space. We cannot have
multiple drivers providing multiple kernel interfaces for the same hardware.

That said, I'm not trying to provide an excuse for not having libraries
available to help the user to configure and control the device more or
less automatically even in terms of best effort. It's something that
does require attention, a lot more of it than it has received in recent
few years.

-- 
Kind regards,

Sakari Ailus
sakari.ailus@linux.intel.com

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web