Groups > linux.kernel > #1741368 > unrolled thread

[RFC PATCH 0/9] V4L2 Jobs API WIP

Started by	Alexandre Courbot <acourbot@chromium.org>
First post	2017-09-28 12:00 +0200
Last post	2017-09-28 12:00 +0200
Articles	6 — 1 participant

Back to article view | Back to linux.kernel

  [RFC PATCH 0/9] V4L2 Jobs API WIP Alexandre Courbot <acourbot@chromium.org> - 2017-09-28 12:00 +0200
    [RFC PATCH 2/9] [media] v4l2-core: add core jobs API support Alexandre Courbot <acourbot@chromium.org> - 2017-09-28 12:00 +0200
    [RFC PATCH 5/9] [media] v4l2-job: add generic jobs ops Alexandre Courbot <acourbot@chromium.org> - 2017-09-28 12:00 +0200
    [RFC PATCH 3/9] [media] videobuf2: add support for jobs API Alexandre Courbot <acourbot@chromium.org> - 2017-09-28 12:00 +0200
    [RFC PATCH 4/9] [media] v4l2-ctrls: add support for jobs API Alexandre Courbot <acourbot@chromium.org> - 2017-09-28 12:00 +0200
    [RFC PATCH 7/9] [media] vim2m: add jobs API support Alexandre Courbot <acourbot@chromium.org> - 2017-09-28 12:00 +0200

#1741368 — [RFC PATCH 0/9] V4L2 Jobs API WIP

From	Alexandre Courbot <acourbot@chromium.org>
Date	2017-09-28 12:00 +0200
Subject	[RFC PATCH 0/9] V4L2 Jobs API WIP
Message-ID	<uuDLQ-7o2-25@gated-at.bofh.it>

Hi everyone,

Here is a new attempt at the "request" (which I propose to rename "jobs") API
for V4L2, hopefully in a manner that can converge to something that will be
merged. The core ideas should be easy to grasp for those familiar with the
previous attemps, yet there are a few important differences.

Most notably, user-space does not need to explicitly allocate and manage
requests/jobs (but still can if this makes sense). We noticed that only specific
use-cases require such an explicit management, and opted for a jobs queue that
controls the flow of work over a set of opened devices. This should simplify
user-space code quite a bit, while still retaining the ability to manage states
explicitly like the previous request API proposals allowed to do.

The jobs API defines a few new concepts that user-space can use to control the
workflow on a set of opened V4L2 devices:

A JOB QUEUE can be created from a set of opened FDs that are part of a pipeline
and need to cooperate (be it capture, m2m, or media controller devices).

A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
then submitted to the job queue. Once the job queue schedules the job, its
parameters (controls, etc) are applied to the devices of the queue, and itsd
buffers are processed. Immediately after a job is submitted, the next job is
ready to be set up without further user action.

Once a job completes, it must be dequeued and user-space can then read back its
properties (notably controls) at completion time.

Internally, the state of jobs is managed through STATE HANDLERS. Each driver
supporting the jobs API needs to specify an implementation of a state handler.
Fortunately, most drivers can rely on the generic state handler implementation
that simply records and replays a job's parameter using standard V4L2 functions.
Thanks to this, adding jobs API support to a driver relying on the control
framework and vb2 only requires a dozen lines of codes.

Drivers with specific needs or opportunities for optimization can however
provide their own implementation of a state handler. This may in particular be
beneficial for hardware that supports configuration or command buffers (thinking
about VSP1 here).

This is still very early work, and focus has been on the following points:

* Provide something that anybody can test (currently using vim2m and vivid),
* Reuse the current V4L2 APIs as much as possible,
* Remain flexible enough to accomodate the inevitable changes that will be
  requested,
* Keep line count low, even if functionality is missing at the moment.

Please keep this in mind while going through the patches. In particular, at the
moment the parameters of a job are limited to integer controls. I know that much
more is expected, but V4L2 has quite a learning curve and I preferred to focus
on the general concepts for now. More is coming though! :)

I have written two small example programs that demonstrate the use of this API:

- With a codec device (vim2m): https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42

- With a capture device (vivid): https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04

Considering the history with the request API, I don't expect everything proposed
here to be welcome or understood immediately. In particular I apologize for not
reusing any of the previous attempts - I was just more comfortable laying down
my ideas from scratch.

If this proposal is not dismissed as complete garbage I will also be happy to
discuss it in-person at the mini-summit in Prague. :)

Cheers,
Alex.

Alexandre Courbot (9):
  [media] v4l2-core: add v4l2_is_v4l2_file function
  [media] v4l2-core: add core jobs API support
  [media] videobuf2: add support for jobs API
  [media] v4l2-ctrls: add support for jobs API
  [media] v4l2-job: add generic jobs ops
  [media] m2m: add generic support for jobs API
  [media] vim2m: add jobs API support
  [media] vivid: add jobs API support for capture device
  [media] document jobs API

 Documentation/media/intro.rst                      |   2 +
 Documentation/media/media_uapi.rst                 |   1 +
 Documentation/media/uapi/jobs/jobs-api.rst         |  23 +
 Documentation/media/uapi/jobs/jobs-example.rst     |  69 ++
 Documentation/media/uapi/jobs/jobs-intro.rst       |  61 ++
 Documentation/media/uapi/jobs/jobs-queue.rst       |  73 ++
 Documentation/media/uapi/jobs/jobs-queue.svg       | 192 ++++++
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst          |   6 +
 drivers/media/platform/vim2m.c                     |  24 +
 drivers/media/platform/vivid/vivid-core.c          |  16 +
 drivers/media/platform/vivid/vivid-core.h          |   2 +
 drivers/media/platform/vivid/vivid-kthread-cap.c   |   5 +
 drivers/media/v4l2-core/Makefile                   |   4 +-
 drivers/media/v4l2-core/v4l2-ctrls.c               |  50 +-
 drivers/media/v4l2-core/v4l2-dev.c                 |  12 +
 drivers/media/v4l2-core/v4l2-job-generic.c         | 394 +++++++++++
 drivers/media/v4l2-core/v4l2-jobqueue-dev.c        | 173 +++++
 drivers/media/v4l2-core/v4l2-jobqueue.c            | 764 +++++++++++++++++++++
 drivers/media/v4l2-core/v4l2-mem2mem.c             |  19 +
 drivers/media/v4l2-core/videobuf2-core.c           |  33 +-
 include/media/v4l2-ctrls.h                         |   6 +
 include/media/v4l2-dev.h                           |  13 +
 include/media/v4l2-fh.h                            |   4 +
 include/media/v4l2-job-generic.h                   |  47 ++
 include/media/v4l2-job-state.h                     |  75 ++
 include/media/v4l2-jobqueue-dev.h                  |  24 +
 include/media/v4l2-jobqueue.h                      |  54 ++
 include/media/v4l2-mem2mem.h                       |  11 +
 include/media/videobuf2-core.h                     |  16 +
 include/uapi/linux/v4l2-jobs.h                     |  40 ++
 include/uapi/linux/videodev2.h                     |   2 +
 31 files changed, 2205 insertions(+), 10 deletions(-)
 create mode 100644 Documentation/media/uapi/jobs/jobs-api.rst
 create mode 100644 Documentation/media/uapi/jobs/jobs-example.rst
 create mode 100644 Documentation/media/uapi/jobs/jobs-intro.rst
 create mode 100644 Documentation/media/uapi/jobs/jobs-queue.rst
 create mode 100644 Documentation/media/uapi/jobs/jobs-queue.svg
 create mode 100644 drivers/media/v4l2-core/v4l2-job-generic.c
 create mode 100644 drivers/media/v4l2-core/v4l2-jobqueue-dev.c
 create mode 100644 drivers/media/v4l2-core/v4l2-jobqueue.c
 create mode 100644 include/media/v4l2-job-generic.h
 create mode 100644 include/media/v4l2-job-state.h
 create mode 100644 include/media/v4l2-jobqueue-dev.h
 create mode 100644 include/media/v4l2-jobqueue.h
 create mode 100644 include/uapi/linux/v4l2-jobs.h

-- 
2.14.2.822.g60be5d43e6-goog

[toc] | [next] | [standalone]

#1741369 — [RFC PATCH 2/9] [media] v4l2-core: add core jobs API support

From	Alexandre Courbot <acourbot@chromium.org>
Date	2017-09-28 12:00 +0200
Subject	[RFC PATCH 2/9] [media] v4l2-core: add core jobs API support
Message-ID	<uuDLR-7o2-53@gated-at.bofh.it>
In reply to	#1741368

Add core support code for jobs API. This manages the life cycle of jobs
and creation of a jobs queue, as well as the interface for job states.

It also exposes the user-space jobs API.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/Makefile            |   3 +-
 drivers/media/v4l2-core/v4l2-dev.c          |   6 +
 drivers/media/v4l2-core/v4l2-jobqueue-dev.c | 173 +++++++
 drivers/media/v4l2-core/v4l2-jobqueue.c     | 764 ++++++++++++++++++++++++++++
 include/media/v4l2-dev.h                    |   4 +
 include/media/v4l2-fh.h                     |   4 +
 include/media/v4l2-job-state.h              |  75 +++
 include/media/v4l2-jobqueue-dev.h           |  24 +
 include/media/v4l2-jobqueue.h               |  54 ++
 include/uapi/linux/v4l2-jobs.h              |  40 ++
 include/uapi/linux/videodev2.h              |   2 +
 11 files changed, 1148 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-jobqueue-dev.c
 create mode 100644 drivers/media/v4l2-core/v4l2-jobqueue.c
 create mode 100644 include/media/v4l2-job-state.h
 create mode 100644 include/media/v4l2-jobqueue-dev.h
 create mode 100644 include/media/v4l2-jobqueue.h
 create mode 100644 include/uapi/linux/v4l2-jobs.h

diff --git a/drivers/media/v4l2-core/Makefile b/drivers/media/v4l2-core/Makefile
index 098ad5fd5231..a717bb8f1a25 100644
--- a/drivers/media/v4l2-core/Makefile
+++ b/drivers/media/v4l2-core/Makefile
@@ -6,7 +6,8 @@ tuner-objs	:=	tuner-core.o
 
 videodev-objs	:=	v4l2-dev.o v4l2-ioctl.o v4l2-device.o v4l2-fh.o \
 			v4l2-event.o v4l2-ctrls.o v4l2-subdev.o v4l2-clk.o \
-			v4l2-async.o
+			v4l2-async.o v4l2-jobqueue.o v4l2-jobqueue-dev.o
+
 ifeq ($(CONFIG_COMPAT),y)
   videodev-objs += v4l2-compat-ioctl32.o
 endif
diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 5a7063886c93..fb229b671b9d 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -30,6 +30,7 @@
 #include <media/v4l2-common.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-ioctl.h>
+#include <media/v4l2-jobqueue-dev.h>
 
 #define VIDEO_NUM_DEVICES	256
 #define VIDEO_NAME              "video4linux"
@@ -1058,6 +1059,10 @@ static int __init videodev_init(void)
 		return -EIO;
 	}
 
+	ret = v4l2_jobqueue_device_init();
+	if (ret < 0)
+		printk(KERN_WARNING "video_dev: channel initialization failed\n");
+
 	return 0;
 }
 
@@ -1065,6 +1070,7 @@ static void __exit videodev_exit(void)
 {
 	dev_t dev = MKDEV(VIDEO_MAJOR, 0);
 
+	v4l2_jobqueue_device_exit();
 	class_unregister(&video_class);
 	unregister_chrdev_region(dev, VIDEO_NUM_DEVICES);
 }
diff --git a/drivers/media/v4l2-core/v4l2-jobqueue-dev.c b/drivers/media/v4l2-core/v4l2-jobqueue-dev.c
new file mode 100644
index 000000000000..688c4ba275a6
--- /dev/null
+++ b/drivers/media/v4l2-core/v4l2-jobqueue-dev.c
@@ -0,0 +1,173 @@
+/*
+    V4L2 job queue device
+
+    Copyright (C) 2017  The Chromium project
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+ */
+
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/fs.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-jobqueue.h>
+#include <uapi/linux/v4l2-jobs.h>
+
+#define CLASS_NAME "v4l2_jobqueue"
+#define DEVICE_NAME "v4l2_jobqueue"
+
+static int major;
+static struct class *jobqueue_class;
+static struct device *jobqueue_device;
+
+static int v4l2_jobqueue_device_open(struct inode *inode, struct file *filp)
+{
+	struct v4l2_jobqueue *jq;
+
+	jq = v4l2_jobqueue_new();
+	if (IS_ERR(jq))
+		return PTR_ERR(jq);
+
+	filp->private_data = jq;
+
+	return 0;
+}
+
+static int v4l2_jobqueue_device_release(struct inode *inode, struct file *filp)
+{
+	struct v4l2_jobqueue *jq = filp->private_data;
+
+	return v4l2_jobqueue_del(jq);
+}
+
+static long v4l2_jobqueue_ioctl_init(struct file *filp, void *arg)
+{
+	struct v4l2_jobqueue *jq = filp->private_data;
+	struct v4l2_jobqueue_init *cinit = arg;
+
+	return v4l2_jobqueue_init(jq, cinit);
+}
+
+static long v4l2_jobqueue_device_ioctl_qjob(struct file *filp, void *arg)
+{
+	struct v4l2_jobqueue *jq = filp->private_data;
+
+	return v4l2_jobqueue_qjob(jq);
+}
+
+static long v4l2_jobqueue_device_ioctl_dqjob(struct file *filp, void *arg)
+{
+	struct v4l2_jobqueue *jq = filp->private_data;
+
+	return v4l2_jobqueue_dqjob(jq);
+}
+
+static long v4l2_jobqueue_device_ioctl_export_job(struct file *filp, void *arg)
+{
+	struct v4l2_jobqueue *jq = filp->private_data;
+	struct v4l2_jobqueue_job *job = arg;
+
+	return v4l2_jobqueue_export_job(jq, job);
+}
+
+static long v4l2_jobqueue_device_ioctl_import_job(struct file *filp, void *arg)
+{
+	struct v4l2_jobqueue *jq = filp->private_data;
+	struct v4l2_jobqueue_job *job = arg;
+
+	return v4l2_jobqueue_import_job(jq, job);
+}
+
+static long v4l2_jobqueue_device_do_ioctl(struct file *filp, unsigned int cmd,
+					  void *arg)
+{
+	switch (cmd) {
+		case VIDIOC_JOBQUEUE_INIT:
+			return v4l2_jobqueue_ioctl_init(filp, arg);
+
+		case VIDIOC_JOBQUEUE_QJOB:
+			return v4l2_jobqueue_device_ioctl_qjob(filp, arg);
+
+		case VIDIOC_JOBQUEUE_DQJOB:
+			return v4l2_jobqueue_device_ioctl_dqjob(filp, arg);
+
+		case VIDIOC_JOBQUEUE_EXPORT_JOB:
+			return v4l2_jobqueue_device_ioctl_export_job(filp, arg);
+
+		case VIDIOC_JOBQUEUE_IMPORT_JOB:
+			return v4l2_jobqueue_device_ioctl_import_job(filp, arg);
+
+		default:
+			pr_err("Invalid ioctl!\n");
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static long v4l2_jobqueue_device_ioctl(struct file *filp, unsigned int cmd,
+				       unsigned long arg)
+{
+	return video_usercopy(filp, cmd, arg, v4l2_jobqueue_device_do_ioctl);
+}
+
+static const struct file_operations v4l2_jobqueue_devnode_fops = {
+	.owner = THIS_MODULE,
+	.open = v4l2_jobqueue_device_open,
+	.unlocked_ioctl = v4l2_jobqueue_device_ioctl,
+#ifdef CONFIG_COMPAT
+	/* TODO */
+	/* .compat_ioctl = jobqueue_compat_ioctl, */
+#endif
+	.release = v4l2_jobqueue_device_release,
+};
+
+int __init v4l2_jobqueue_device_init(void)
+{
+	/* Set to error value so v4l2_jobqueue_device_exit does nothing if we
+	 * don't initialize properly */
+	jobqueue_device = ERR_PTR(-EINVAL);
+
+	major = register_chrdev(0, DEVICE_NAME, &v4l2_jobqueue_devnode_fops);
+	if (major < 0) {
+		pr_err("unable to allocate major\n");
+		return major;
+	}
+
+	jobqueue_class = class_create(THIS_MODULE, CLASS_NAME);
+	if (IS_ERR(jobqueue_class)) {
+		pr_err("cannot create class\n");
+		unregister_chrdev(major, DEVICE_NAME);
+		return PTR_ERR(jobqueue_class);
+	}
+
+	jobqueue_device = device_create(jobqueue_class, NULL, MKDEV(major, 0),
+					NULL, DEVICE_NAME);
+	if (IS_ERR(jobqueue_device)) {
+		pr_err("cannot create device\n");
+		class_destroy(jobqueue_class);
+		unregister_chrdev(major, DEVICE_NAME);
+		return PTR_ERR(jobqueue_device);
+	}
+
+	return 0;
+}
+
+void __exit v4l2_jobqueue_device_exit(void)
+{
+	if (IS_ERR(jobqueue_device))
+		return;
+
+	device_destroy(jobqueue_class, MKDEV(major, 0));
+	class_destroy(jobqueue_class);
+	unregister_chrdev(major, DEVICE_NAME);
+}
diff --git a/drivers/media/v4l2-core/v4l2-jobqueue.c b/drivers/media/v4l2-core/v4l2-jobqueue.c
new file mode 100644
index 000000000000..36d2dd48b086
--- /dev/null
+++ b/drivers/media/v4l2-core/v4l2-jobqueue.c
@@ -0,0 +1,764 @@
+/*
+    V4L2 job queue implementation
+
+    Copyright (C) 2017  The Chromium project
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+ */
+
+#include <linux/compat.h>
+#include <linux/export.h>
+#include <linux/string.h>
+#include <linux/file.h>
+#include <linux/list.h>
+#include <linux/kref.h>
+#include <linux/anon_inodes.h>
+#include <linux/slab.h>
+#include <linux/mutex.h>
+#include <linux/workqueue.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-fh.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-jobqueue.h>
+#include <media/v4l2-job-state.h>
+#include <uapi/linux/v4l2-jobs.h>
+
+/* Limited by the size of atomic_t to track devices that completed a job */
+#define V4L2_JOBQUEUE_MAX_DEVICES sizeof(atomic_t)
+
+/*
+ * State of all managed devices for a given job
+ */
+struct v4l2_job {
+	struct kref refcount;
+	struct v4l2_jobqueue *jq;
+	/* node in v4l2_jobqueue's queued_jobs or completed_jobs */
+	struct list_head node;
+	/* global list of existing jobs for this queue */
+	struct list_head jobs_list;
+	/* mask of devices that completed this job */
+	atomic_t completed;
+	/* fd exported to user-space */
+	int fd;
+	enum v4l2_job_status status;
+
+	/* per-device states */
+	struct v4l2_job_state *state[0];
+};
+
+/*
+ * A job queue manages the job flow for a given set of devices, applies their
+ * state, and activates them in lockstep.
+ *
+ * A job goes through the following stages through its life:
+ *
+ * * current_job: the job has been created and is waiting to be queued. S_CTRL
+ *   will apply to it. Once queued, it is pushed into
+ * * queued_jobs: a queue of jobs to be processed in sequential order. The head
+ *   of this list becomes the
+ * * active_job: the job currently being processed by the hardware. Once
+ *   completed, the next job in queued_job becomes active, and the previous
+ *   active job goes into
+ * * completed_jobs: a list of completed jobs waiting to be dequeued by
+ *   user-space. As user-space called the DQJOB ioctl, the head becomes the
+ * * dequeued_job: the job on which G_CTRL will be performed on. A job stays
+ *   in this state until another one is dequeued, at which point it is deleted.
+ */
+struct v4l2_jobqueue {
+	/* List of all jobs created for this queue, regardless of state */
+	struct list_head jobs_list;
+	/*
+	 * Job that user-space is currently preparing, to be added to
+	 * queued_jobs upon QJOB ioctl.
+	 */
+	struct v4l2_job *current_job;
+
+	/* List of jobs that are ready to be processed */
+	struct list_head queued_jobs;
+
+	/* Job that is currently processed by the devices */
+	struct v4l2_job *active_job;
+
+	/* List of completed jobs, ready to be dequeued */
+	struct list_head completed_jobs;
+
+	/* Job that has last been dequeued and can be queried by user-space */
+	struct v4l2_job *dequeued_job;
+
+	/* Projects the *_job[s] lists/pointers above */
+	struct mutex lock;
+	struct work_struct job_complete_work;
+
+	wait_queue_head_t done_wq;
+
+	unsigned int nb_devs;
+	struct {
+		struct file *f;
+		struct v4l2_job_state_handler *state_handler;
+	} *devs;
+};
+
+static bool v4l2_jobqueue_is_locked(struct v4l2_jobqueue *jq)
+{
+	return mutex_is_locked(&jq->lock);
+}
+
+static void v4l2_jobqueue_free_job(struct v4l2_job *job)
+{
+	struct v4l2_jobqueue *jq = job->jq;
+	int i;
+
+	for (i = 0; i < jq->nb_devs; i++) {
+		struct v4l2_job_state_handler *hdl = jq->devs[i].state_handler;
+		if (job->state[i])
+			hdl->ops->job_free(hdl, job->state[i]);
+	}
+	kfree(job);
+}
+
+/*
+ * Must be called with jobqueue lock held
+ */
+static void v4l2_jobqueue_delete_job(struct kref *ref)
+{
+	struct v4l2_job *job = container_of(ref, struct v4l2_job, refcount);
+
+	list_del(&job->jobs_list);
+
+	   v4l2_jobqueue_free_job(job);
+}
+
+/*
+ * Must be called with the jobqueue lock acquired
+ */
+static void job_get(struct v4l2_job *job)
+{
+	kref_get(&job->refcount);
+}
+
+/*
+ * Must be called with the jobqueue lock acquired
+ */
+static int job_put(struct v4l2_job *job)
+{
+	return kref_put(&job->refcount, v4l2_jobqueue_delete_job);
+}
+
+/*
+ * Move a job from one state to another in the jobqueue state machine, making
+ * extensive sanity tests.
+ *
+ * jobqueue lock must be held when this function is called.
+ */
+static void jobqueue_set_job_state(struct v4l2_job *job,
+				   enum v4l2_job_status status)
+{
+	struct v4l2_jobqueue *jq = job->jq;
+	int i;
+
+	BUG_ON(!v4l2_jobqueue_is_locked(jq));
+
+	/* Sanity checks & jobqueue state update */
+	switch (status) {
+	case CURRENT:
+		BUG_ON(job->status != OUT_OF_QUEUE);
+		BUG_ON(jq->current_job != NULL);
+		jq->current_job = job;
+		break;
+	case QUEUED:
+		BUG_ON(job->status != CURRENT);
+		BUG_ON(jq->current_job != job);
+		jq->current_job = NULL;
+		list_add_tail(&job->node, &jq->queued_jobs);
+		break;
+	case ACTIVE:
+		BUG_ON(job->status != QUEUED);
+		BUG_ON(jq->active_job != NULL);
+		BUG_ON(list_first_entry_or_null(&jq->queued_jobs,
+						struct v4l2_job, node) != job);
+		list_del(&job->node);
+		jq->active_job = job;
+		break;
+	case COMPLETED:
+		BUG_ON(job->status != ACTIVE);
+		BUG_ON(jq->active_job != job);
+		jq->active_job = NULL;
+		list_add_tail(&job->node, &jq->completed_jobs);
+		break;
+	case DEQUEUED:
+		BUG_ON(job->status != COMPLETED);
+		BUG_ON(jq->dequeued_job != NULL);
+		BUG_ON(list_first_entry_or_null(&jq->completed_jobs,
+						struct v4l2_job, node) != job);
+		list_del(&job->node);
+		jq->dequeued_job = job;
+		break;
+	case OUT_OF_QUEUE:
+		BUG_ON(job->status != DEQUEUED);
+		BUG_ON(jq->dequeued_job != job);
+		jq->dequeued_job = NULL;
+		break;
+	};
+
+	job->status = status;
+
+	for (i = 0; i < jq->nb_devs; i++) {
+		struct v4l2_job_state_handler *hdl = jq->devs[i].state_handler;
+		struct v4l2_job_state *state = job->state[i];
+
+		switch (status) {
+		case CURRENT:
+			hdl->current_state = state;
+			break;
+		case ACTIVE:
+			hdl->active_state = state;
+			break;
+		case DEQUEUED:
+			hdl->dequeued_state = state;
+			break;
+		default:
+			break;
+		}
+
+		if (hdl->ops->state_changed)
+			hdl->ops->state_changed(hdl, state, status);
+	}
+}
+
+/*
+ * jobqueue lock must be held by caller
+ */
+static int v4l2_jobqueue_new_job(struct v4l2_jobqueue *jq)
+{
+	struct v4l2_job *job;
+	int i;
+
+	BUG_ON(!v4l2_jobqueue_is_locked(jq));
+
+	if (jq->current_job)
+		return -EBUSY;
+
+	job = kzalloc(sizeof(*job) + sizeof(job->state[0]) * jq->nb_devs,
+		      GFP_KERNEL);
+	if (job == NULL)
+		return -ENOMEM;
+
+	list_add(&job->jobs_list, &jq->jobs_list);
+	kref_init(&job->refcount);
+	job->jq = jq;
+	job->status = OUT_OF_QUEUE;
+	job->fd = -1;
+	atomic_set(&job->completed, 0);
+	for (i = 0; i < jq->nb_devs; i++) {
+		struct v4l2_job_state_handler *hdl = jq->devs[i].state_handler;
+		struct v4l2_job_state *state;
+
+		state = hdl->ops->job_new(hdl);
+		job->state[i] = state;
+		if (IS_ERR(state)) {
+			         v4l2_jobqueue_free_job(job);
+			return PTR_ERR(state);
+		}
+		state->job = job;
+	}
+
+	jobqueue_set_job_state(job, CURRENT);
+
+	return 0;
+}
+
+/*
+ * Prepare the next queued job for streaming.
+ *
+ * jobqueue lock must be held by the caller.
+ */
+static void v4l2_jobqueue_prepare_streaming(struct v4l2_jobqueue *jq)
+{
+	struct v4l2_job *job;
+
+	BUG_ON(!v4l2_jobqueue_is_locked(jq));
+
+	/* Already streaming */
+	if (jq->active_job)
+		return;
+
+	job = list_first_entry_or_null(&jq->queued_jobs, struct v4l2_job, node);
+	/* No job ready to be streamed */
+	if (!job)
+		return;
+	jobqueue_set_job_state(job, ACTIVE);
+}
+
+/*
+ * Asks all devices to start processing the prepared job
+ *
+ */
+static void v4l2_jobqueue_start_streaming(struct v4l2_jobqueue *jq)
+{
+	int ret;
+	int i;
+
+	/* No job ready to be streamed */
+	if (!jq->active_job)
+		return;
+
+	/*
+	 * TODO move what follows into a worker?
+	 */
+
+	/* Set the desired state on all devices */
+	for (i = 0; i < jq->nb_devs; i++) {
+		struct v4l2_job_state_handler *handler;
+
+		handler = jq->devs[i].state_handler;
+		ret = handler->ops->job_apply(handler);
+		/* TODO proper cleanup and reporting after error */
+		if (ret < 0) {
+			pr_err("error while setting job queue parameters!\n");
+			return;
+		}
+	}
+
+	/* Start streaming on all devices */
+	for (i = 0; i < jq->nb_devs; i++) {
+		struct v4l2_job_state_handler *handler;
+
+		handler = jq->devs[i].state_handler;
+
+		if (handler->process_active_job)
+			handler->process_active_job(jq->devs[i].state_handler);
+	}
+}
+
+static void v4l2_jobqueue_job_complete(struct work_struct *work)
+{
+	struct v4l2_jobqueue *jq = container_of(work, struct v4l2_jobqueue,
+						job_complete_work);
+
+	v4l2_jobqueue_lock(jq);
+
+	jobqueue_set_job_state(jq->active_job, COMPLETED);
+	wake_up(&jq->done_wq);
+
+	v4l2_jobqueue_prepare_streaming(jq);
+
+	v4l2_jobqueue_unlock(jq);
+
+	/* see if we can perform the next job */
+	v4l2_jobqueue_start_streaming(jq);
+}
+
+struct v4l2_jobqueue *v4l2_jobqueue_new(void)
+{
+	struct v4l2_jobqueue *jq;
+
+	jq = kzalloc(sizeof(*jq), GFP_KERNEL);
+	if (jq == NULL)
+		return ERR_PTR(-ENOMEM);
+
+	INIT_LIST_HEAD(&jq->jobs_list);
+	INIT_LIST_HEAD(&jq->queued_jobs);
+	INIT_LIST_HEAD(&jq->completed_jobs);
+	mutex_init(&jq->lock);
+	init_waitqueue_head(&jq->done_wq);
+	INIT_WORK(&jq->job_complete_work, v4l2_jobqueue_job_complete);
+
+	return jq;
+}
+EXPORT_SYMBOL(v4l2_jobqueue_new);
+
+int v4l2_jobqueue_del(struct v4l2_jobqueue *jq)
+{
+	struct v4l2_job *job, *_job_t;
+	int i;
+
+	v4l2_jobqueue_lock(jq);
+
+	if (jq->current_job != NULL) {
+		job_put(jq->current_job);
+		jq->current_job = NULL;
+	}
+
+	/* Clean all pending jobs */
+	list_for_each_entry_safe(job, _job_t, &jq->queued_jobs, node) {
+		pr_warn("Deleting pending queued job\n");
+		list_del(&job->node);
+		job_put(job);
+	}
+
+	/* Wait for active job to complete, if any */
+	while (jq->active_job != NULL) {
+		v4l2_jobqueue_unlock(jq);
+		wait_event(jq->done_wq, jq->active_job == NULL);
+		cancel_work_sync(&jq->job_complete_work);
+		v4l2_jobqueue_lock(jq);
+	}
+
+	/* Clean all completed jobs */
+	list_for_each_entry_safe(job, _job_t, &jq->completed_jobs, node) {
+		pr_warn("Deleting pending completed job\n");
+		list_del(&job->node);
+		job_put(job);
+	}
+
+	/* Delete currently dequeued job, if any */
+	if (jq->dequeued_job != NULL) {
+		job = jq->dequeued_job;
+		jobqueue_set_job_state(job, OUT_OF_QUEUE);
+		job_put(job);
+	}
+
+	/* No job should exist anymore */
+	WARN_ON(jq->dequeued_job);
+	WARN_ON(!list_empty(&jq->completed_jobs));
+	WARN_ON(jq->active_job);
+	WARN_ON(!list_empty(&jq->queued_jobs));
+	WARN_ON(jq->current_job);
+
+	/*
+	 * must invalidate all fds exported to user-space, otherwise users
+	 * may do funny things with them, or they will be automatically closed
+	 * when the process exits
+	 *
+	 * TODO improve this. We can still enter a race if jobqueue_release_job
+	 * if called right before we set private_data to NULL. Unfortunately
+	 * we also cannot use fput() here the call to release is asynchronous.
+	 */
+	if (!list_empty(&jq->jobs_list)) {
+		pr_warn("Exported jobs still exist, removing them\n");
+		list_for_each_entry_safe(job, _job_t, &jq->jobs_list, jobs_list)
+			v4l2_jobqueue_free_job(job);
+	}
+
+	v4l2_jobqueue_unlock(jq);
+
+	for (i = 0; i < jq->nb_devs; i++) {
+		jq->devs[i].state_handler->jobqueue = NULL;
+		fput(jq->devs[i].f);
+	}
+	kfree(jq->devs);
+	kfree(jq);
+
+	return 0;
+}
+EXPORT_SYMBOL(v4l2_jobqueue_del);
+
+int v4l2_jobqueue_init(struct v4l2_jobqueue *jq,
+		       struct v4l2_jobqueue_init *cinit)
+{
+	int ret;
+	int i;
+
+	if (cinit->nb_devs > V4L2_JOBQUEUE_MAX_DEVICES)
+		return -ENOSPC;
+
+	jq->devs = kzalloc(sizeof(jq->devs[0]) * cinit->nb_devs, GFP_KERNEL);
+	if (!jq->devs) {
+		pr_err("error: out of memory\n");
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < cinit->nb_devs; i++) {
+		struct file *f;
+		struct video_device *vdev;
+		struct v4l2_fh *fh;
+		struct v4l2_job_state_handler *handler;
+		struct v4l2_ctrl_handler *ctrl_handler;
+
+		f = fget(cinit->fd[i]);
+		jq->devs[i].f = f;
+		if (!v4l2_is_v4l2_file(f)) {
+			pr_err("error: passed fd is not v4l2 device!\n");
+			ret = -EINVAL;
+			goto error;
+		}
+
+		fh = f->private_data;
+		vdev = video_devdata(f);
+
+		ctrl_handler = fh ? fh->ctrl_handler : vdev->ctrl_handler;
+		if (!ctrl_handler) {
+			pr_err("error: no control handler in device!\n");
+			ret = -EINVAL;
+			goto error;
+		}
+
+		if (fh)
+			handler = fh->state_handler;
+
+		if (!handler && vdev)
+			handler = vdev->state_handler;
+
+		if (!handler) {
+			pr_err("error: no state handler in device!\n");
+			ret = -EINVAL;
+			goto error;
+		}
+		handler->jobqueue = jq;
+		jq->devs[0].state_handler = handler;
+	}
+
+	jq->nb_devs = cinit->nb_devs;
+
+	/* Create first job */
+	v4l2_jobqueue_lock(jq);
+	ret = v4l2_jobqueue_new_job(jq);
+	v4l2_jobqueue_unlock(jq);
+	if (ret < 0)
+		goto error;
+
+	return 0;
+
+error:
+	jq->nb_devs = 0;
+	for (i = 0; i < cinit->nb_devs; i++)
+		if (jq->devs[i].f)
+			fput(jq->devs[i].f);
+	kfree(jq->devs);
+	jq->devs = NULL;
+	return ret;
+}
+EXPORT_SYMBOL(v4l2_jobqueue_init);
+
+void v4l2_jobqueue_lock(struct v4l2_jobqueue *jq)
+{
+	mutex_lock(&jq->lock);
+}
+EXPORT_SYMBOL(v4l2_jobqueue_lock);
+
+void v4l2_jobqueue_unlock(struct v4l2_jobqueue *jq)
+{
+	mutex_unlock(&jq->lock);
+}
+EXPORT_SYMBOL(v4l2_jobqueue_unlock);
+
+void v4l2_jobqueue_job_finish(struct v4l2_job_state_handler *handler)
+{
+	struct v4l2_job_state *state = handler->active_state;
+	struct v4l2_job *job;
+	struct v4l2_jobqueue *jq;
+	unsigned int finished_mask;
+	int pos;
+
+	BUG_ON(!state);
+
+	job = state->job;
+	jq = job->jq;
+	pos = state - job->state[0];
+	finished_mask = BIT(jq->nb_devs) - 1;
+
+	handler->ops->job_complete(handler);
+	handler->active_state = NULL;
+
+	/* have all devices completed? */
+	if (!atomic_add_return(BIT(pos), &job->completed) != finished_mask)
+		schedule_work(&jq->job_complete_work);
+}
+EXPORT_SYMBOL(v4l2_jobqueue_job_finish);
+
+int v4l2_jobqueue_qjob(struct v4l2_jobqueue *jq)
+{
+	bool start_streaming;
+	int ret = 0;
+
+	v4l2_jobqueue_lock(jq);
+
+	if (jq->current_job == NULL) {
+		/* This should never happen */
+		pr_err("no current job at the moment!\n");
+		ret = -EINVAL;
+		goto out;
+	}
+
+	jobqueue_set_job_state(jq->current_job, QUEUED);
+
+	start_streaming = (jq->active_job == NULL);
+
+	v4l2_jobqueue_prepare_streaming(jq);
+
+	v4l2_jobqueue_unlock(jq);
+
+	v4l2_jobqueue_lock(jq);
+
+	ret = v4l2_jobqueue_new_job(jq);
+	if (ret < 0)
+		goto out;
+
+out:
+	v4l2_jobqueue_unlock(jq);
+
+	if (ret == 0 && start_streaming)
+		v4l2_jobqueue_start_streaming(jq);
+
+	return ret;
+}
+EXPORT_SYMBOL(v4l2_jobqueue_qjob);
+
+int v4l2_jobqueue_dqjob(struct v4l2_jobqueue *jq)
+{
+	struct v4l2_job *job = NULL;
+	struct v4l2_job *old_job;
+	int ret;
+
+	v4l2_jobqueue_lock(jq);
+
+	while (true) {
+		job = list_first_entry_or_null(&jq->completed_jobs,
+					       struct v4l2_job, node);
+		if (job)
+			break;
+
+		v4l2_jobqueue_unlock(jq);
+		ret = wait_event_interruptible(jq->done_wq,
+					      !list_empty(&jq->completed_jobs));
+		if (ret)
+			return ret;
+		v4l2_jobqueue_lock(jq);
+	}
+
+	old_job = jq->dequeued_job;
+	if (old_job)
+		jobqueue_set_job_state(old_job, OUT_OF_QUEUE);
+
+	jobqueue_set_job_state(job, DEQUEUED);
+
+	v4l2_jobqueue_unlock(jq);
+
+	/* dispose of reference to previous job */
+	if (old_job)
+		job_put(old_job);
+
+	return 0;
+}
+EXPORT_SYMBOL(v4l2_jobqueue_dqjob);
+
+static int v4l2_jobqueue_release_job(struct inode *inode, struct file *filp)
+{
+	struct v4l2_job *job = filp->private_data;
+	struct v4l2_jobqueue *jq = job->jq;
+
+	/* This can happen if a job queue is closed before all its exported
+	 * jobs. In that case the job becomes inactive until its fd finally
+	 * gets closed */
+	if (job == NULL)
+		return 0;
+
+	v4l2_jobqueue_lock(jq);
+
+	job->fd = -1;
+	job_put(job);
+
+	v4l2_jobqueue_unlock(jq);
+
+	return 0;
+}
+
+static const struct file_operations v4l2_jobqueue_job_ops = {
+	.owner = THIS_MODULE,
+	.release = v4l2_jobqueue_release_job,
+};
+
+int v4l2_jobqueue_export_job(struct v4l2_jobqueue *jq,
+			     struct v4l2_jobqueue_job *export_job)
+{
+	struct v4l2_job *job = jq->current_job;
+	int fd;
+	int i;
+
+	v4l2_jobqueue_lock(jq);
+
+	if (job->fd >= 0) {
+		v4l2_jobqueue_unlock(jq);
+		pr_warn("job already exported!\n");
+		return -EBUSY;
+	}
+
+	/* Sanity check that all devices support export */
+	for (i = 0; i < jq->nb_devs; i++) {
+		if (!jq->devs[i].state_handler->ops->job_export) {
+			v4l2_jobqueue_unlock(jq);
+			pr_warn("some devices do not support job export\n");
+			return -ENOTSUPP;
+		}
+	}
+
+	for (i = 0; i < jq->nb_devs; i++) {
+		struct v4l2_job_state_handler *hdl = jq->devs[i].state_handler;
+		int ret;
+
+		ret = hdl->ops->job_export(hdl);
+		if (ret < 0) {
+			v4l2_jobqueue_unlock(jq);
+			pr_warn("job export failed\n");
+			return ret;
+		}
+	}
+
+	fd = anon_inode_getfd("v4l2", &v4l2_jobqueue_job_ops, job, O_CLOEXEC);
+	if (fd < 0) {
+		v4l2_jobqueue_unlock(jq);
+		return fd;
+	}
+
+	export_job->fd = job->fd = fd;
+	job_get(job);
+
+	v4l2_jobqueue_unlock(jq);
+
+	return 0;
+}
+EXPORT_SYMBOL(v4l2_jobqueue_export_job);
+
+int v4l2_jobqueue_import_job(struct v4l2_jobqueue *jq,
+			     struct v4l2_jobqueue_job *import_job)
+{
+	struct v4l2_job *current_job = NULL;
+	struct v4l2_job *job;
+	struct file *f = fget(import_job->fd);
+
+	if (!f)
+		return -EINVAL;
+
+	/* Is this fd legit? */
+	if (f->f_op != &v4l2_jobqueue_job_ops)
+		return -EINVAL;
+
+	job = f->private_data;
+	fput(f);
+
+	/* Does the job actually belong to us? */
+	if (job->jq != jq) {
+		pr_err("job belongs to a different queue!\n");
+		return -EINVAL;
+	}
+
+	v4l2_jobqueue_lock(jq);
+
+	/* Cannot import a job that is not out of queue */
+	if (job->status != OUT_OF_QUEUE) {
+		pr_err("job is already in the queue!\n");
+		v4l2_jobqueue_unlock(jq);
+		return -EBUSY;
+	}
+
+	job_get(job);
+	current_job = jq->current_job;
+	jq->current_job = NULL;
+	jobqueue_set_job_state(job, CURRENT);
+
+	v4l2_jobqueue_unlock(jq);
+	if (current_job)
+		job_put(current_job);
+
+	return 0;
+}
+EXPORT_SYMBOL(v4l2_jobqueue_import_job);
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index b73d646980da..a1558d9bf309 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -38,6 +38,7 @@ struct v4l2_ioctl_callbacks;
 struct video_device;
 struct v4l2_device;
 struct v4l2_ctrl_handler;
+struct v4l2_job_state_handler;
 
 /* Flag to mark the video_device struct as registered.
    Drivers can clear this flag if they want to block all future
@@ -184,6 +185,8 @@ struct v4l2_file_operations {
  * @dev_parent: pointer to &struct device parent
  * @ctrl_handler: Control handler associated with this device node.
  *	 May be NULL.
+ * @state_handler: State handler associated with this device node.
+ *	 May be NULL.
  * @queue: &struct vb2_queue associated with this device node. May be NULL.
  * @prio: pointer to &struct v4l2_prio_state with device's Priority state.
  *	 If NULL, then v4l2_dev->prio will be used.
@@ -229,6 +232,7 @@ struct video_device
 	struct device *dev_parent;
 
 	struct v4l2_ctrl_handler *ctrl_handler;
+	struct v4l2_job_state_handler *state_handler;
 
 	struct vb2_queue *queue;
 
diff --git a/include/media/v4l2-fh.h b/include/media/v4l2-fh.h
index 0b0757090c04..ea86d713a59a 100644
--- a/include/media/v4l2-fh.h
+++ b/include/media/v4l2-fh.h
@@ -28,6 +28,7 @@
 
 struct video_device;
 struct v4l2_ctrl_handler;
+struct v4l2_job_state_handler;
 
 /**
  * struct v4l2_fh - Describes a V4L2 file handler
@@ -35,6 +36,8 @@ struct v4l2_ctrl_handler;
  * @list: list of file handlers
  * @vdev: pointer to &struct video_device
  * @ctrl_handler: pointer to &struct v4l2_ctrl_handler
+ * @state_handler: pointer to &struct v4l2_job_state_handler, may be NULL if
+ *                 jobs API not supported by this device.
  * @prio: priority of the file handler, as defined by &enum v4l2_priority
  *
  * @wait: event' s wait queue
@@ -48,6 +51,7 @@ struct v4l2_fh {
 	struct list_head	list;
 	struct video_device	*vdev;
 	struct v4l2_ctrl_handler *ctrl_handler;
+	struct v4l2_job_state_handler *state_handler;
 	enum v4l2_priority	prio;
 
 	/* Events */
diff --git a/include/media/v4l2-job-state.h b/include/media/v4l2-job-state.h
new file mode 100644
index 000000000000..42048e8d11a8
--- /dev/null
+++ b/include/media/v4l2-job-state.h
@@ -0,0 +1,75 @@
+/*
+    V4L2 job states interface
+
+    Copyright (C) 2017  The Chromium project
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+ */
+
+#ifndef _V4L2_JOB_STATE_H
+#define _V4L2_JOB_STATE_H
+
+struct v4l2_jobqueue;
+struct v4l2_job_state_handler;
+struct v4l2_ext_control;
+struct v4l2_job;
+struct v4l2_ctrl;
+
+enum v4l2_job_status;
+
+struct v4l2_job_state {
+	struct v4l2_job *job;
+};
+
+struct v4l2_job_state_handler_ops {
+	/* Allocate a new job state for this device */
+	struct v4l2_job_state *(*job_new)(struct v4l2_job_state_handler *hdl);
+	/* Free a previously allocated job state */
+	void (*job_free)(struct v4l2_job_state_handler *hdl,
+			 struct v4l2_job_state *state);
+	/* Apply current job state */
+	int (*job_apply)(struct v4l2_job_state_handler *hdl);
+	/* Signal that the device has completed its part of the job */
+	void (*job_complete)(struct v4l2_job_state_handler *hdl);
+	/* Prepare the current job for being exported */
+	int (*job_export)(struct v4l2_job_state_handler *hdl);
+
+	/* Set control the the current state */
+	int (*s_ctrl)(struct v4l2_job_state_handler *hdl,
+		      struct v4l2_ext_control *c);
+	/* Get control from the dequeued state */
+	int (*g_ctrl)(struct v4l2_job_state_handler *hdl, u32 ctrl_id,
+		      struct v4l2_ext_control *c, u32 which);
+
+	/* Called whenever the HW value of a non-volatile control is changed */
+	void (*ctrl_changed)(struct v4l2_job_state_handler *hdl,
+			     struct v4l2_ctrl *ctrl);
+	/* Called whenever a job has moved in the job queue */
+	void (*state_changed)(struct v4l2_job_state_handler *hdl,
+			      struct v4l2_job_state *state,
+		       enum v4l2_job_status status);
+};
+
+/*
+ * Manages the state of one particular device. Contains pointers to the
+ * current, active and dequeued state that are updated by the jobs framework
+ */
+struct v4l2_job_state_handler {
+	struct v4l2_jobqueue *jobqueue;
+	const struct v4l2_job_state_handler_ops *ops;
+	void (*process_active_job)(struct v4l2_job_state_handler *hdl);
+	struct v4l2_job_state *current_state;
+	struct v4l2_job_state *active_state;
+	struct v4l2_job_state *dequeued_state;
+};
+
+#endif
diff --git a/include/media/v4l2-jobqueue-dev.h b/include/media/v4l2-jobqueue-dev.h
new file mode 100644
index 000000000000..9c3dcff72563
--- /dev/null
+++ b/include/media/v4l2-jobqueue-dev.h
@@ -0,0 +1,24 @@
+/*
+    V4L2 jobqueue device
+
+    Copyright (C) 2017  The Chromium project
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+ */
+
+#ifndef _V4L2_JOBQUEUE_DEV_H_
+#define _V4L2_JOBQUEUE_DEV_H
+
+int v4l2_jobqueue_device_init(void);
+void v4l2_jobqueue_device_exit(void);
+
+#endif
diff --git a/include/media/v4l2-jobqueue.h b/include/media/v4l2-jobqueue.h
new file mode 100644
index 000000000000..a294042b7c13
--- /dev/null
+++ b/include/media/v4l2-jobqueue.h
@@ -0,0 +1,54 @@
+/*
+    V4L2 jobqueue support header.
+
+    Copyright (C) 2017  The Chromium project
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+ */
+
+#ifndef _V4L2_JOBQUEUE_H
+#define _V4L2_JOBQUEUE_H
+
+struct v4l2_jobqueue;
+struct v4l2_job_state_handler;
+
+#include <uapi/linux/v4l2-jobs.h>
+
+enum v4l2_job_status {
+	CURRENT,
+	QUEUED,
+	ACTIVE,
+	COMPLETED,
+	DEQUEUED,
+	OUT_OF_QUEUE,
+};
+
+/* Jobqueue device interface */
+struct v4l2_jobqueue *v4l2_jobqueue_new(void);
+int v4l2_jobqueue_del(struct v4l2_jobqueue *jq);
+
+/* Ioctls support */
+int v4l2_jobqueue_init(struct v4l2_jobqueue *jq,
+		       struct v4l2_jobqueue_init *init);
+int v4l2_jobqueue_qjob(struct v4l2_jobqueue *jq);
+int v4l2_jobqueue_dqjob(struct v4l2_jobqueue *jq);
+int v4l2_jobqueue_export_job(struct v4l2_jobqueue *jq,
+			     struct v4l2_jobqueue_job *job);
+int v4l2_jobqueue_import_job(struct v4l2_jobqueue *jq,
+			     struct v4l2_jobqueue_job *job);
+
+/* Driver/state handler interface */
+void v4l2_jobqueue_job_finish(struct v4l2_job_state_handler *hdl);
+void v4l2_jobqueue_lock(struct v4l2_jobqueue *jq);
+void v4l2_jobqueue_unlock(struct v4l2_jobqueue *jq);
+
+#endif
diff --git a/include/uapi/linux/v4l2-jobs.h b/include/uapi/linux/v4l2-jobs.h
new file mode 100644
index 000000000000..2cba4d20e62f
--- /dev/null
+++ b/include/uapi/linux/v4l2-jobs.h
@@ -0,0 +1,40 @@
+/*
+ * V4L2 jobs API
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __LINUX_V4L2_JOBS_H
+#define __LINUX_V4L2_JOBS_H
+
+#ifndef __KERNEL__
+#include <stdint.h>
+#endif
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+struct v4l2_jobqueue_init {
+	__u32 nb_devs;
+	__s32 *fd;
+};
+
+struct v4l2_jobqueue_job {
+	__s32 fd;
+};
+
+#define VIDIOC_JOBQUEUE_IOCTL_START	0x80
+
+#define VIDIOC_JOBQUEUE_INIT		_IOW('|', VIDIOC_JOBQUEUE_IOCTL_START + 0x00, struct v4l2_jobqueue_init)
+#define VIDIOC_JOBQUEUE_QJOB		_IO('|', VIDIOC_JOBQUEUE_IOCTL_START + 0x01)
+#define VIDIOC_JOBQUEUE_DQJOB		_IO('|', VIDIOC_JOBQUEUE_IOCTL_START + 0x02)
+#define VIDIOC_JOBQUEUE_EXPORT_JOB	_IOR('|', VIDIOC_JOBQUEUE_IOCTL_START + 0x03, struct v4l2_jobqueue_job)
+#define VIDIOC_JOBQUEUE_IMPORT_JOB	_IOW('|', VIDIOC_JOBQUEUE_IOCTL_START + 0x03, struct v4l2_jobqueue_job)
+
+#endif
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 45cf7359822c..7f43e97cf461 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1591,6 +1591,8 @@ struct v4l2_ext_controls {
 #define V4L2_CTRL_MAX_DIMS	  (4)
 #define V4L2_CTRL_WHICH_CUR_VAL   0
 #define V4L2_CTRL_WHICH_DEF_VAL   0x0f000000
+#define V4L2_CTRL_WHICH_CURJOB_VAL   0x0e000000
+#define V4L2_CTRL_WHICH_DEQJOB_VAL   0x0d000000
 
 enum v4l2_ctrl_type {
 	V4L2_CTRL_TYPE_INTEGER	     = 1,
-- 
2.14.2.822.g60be5d43e6-goog

[toc] | [prev] | [next] | [standalone]

#1741370 — [RFC PATCH 5/9] [media] v4l2-job: add generic jobs ops

From	Alexandre Courbot <acourbot@chromium.org>
Date	2017-09-28 12:00 +0200
Subject	[RFC PATCH 5/9] [media] v4l2-job: add generic jobs ops
Message-ID	<uuDLS-7o2-55@gated-at.bofh.it>
In reply to	#1741368

Add a generic state handler that records controls to be set for a given
job and applies them once the job is scheduled, while also allowing said
controls to be read back once the job is dequeued.

This implementation can be used as-is for most drivers, with only drivers
with specific needs needing to provide an alternative implementation.

Note: this is still very early, but should do the job to demonstrate the
jobs API feasibility. Amongst the current limitations:

- We use v4l2_ext_control to store controls, which expects user-space
pointers. As a consequence only integer controls are supported at the
moment.
- No support for try_ctrl yet.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/Makefile           |   3 +-
 drivers/media/v4l2-core/v4l2-job-generic.c | 394 +++++++++++++++++++++++++++++
 include/media/v4l2-job-generic.h           |  47 ++++
 3 files changed, 443 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-job-generic.c
 create mode 100644 include/media/v4l2-job-generic.h

diff --git a/drivers/media/v4l2-core/Makefile b/drivers/media/v4l2-core/Makefile
index a717bb8f1a25..ee09e1f29129 100644
--- a/drivers/media/v4l2-core/Makefile
+++ b/drivers/media/v4l2-core/Makefile
@@ -6,7 +6,8 @@ tuner-objs	:=	tuner-core.o
 
 videodev-objs	:=	v4l2-dev.o v4l2-ioctl.o v4l2-device.o v4l2-fh.o \
 			v4l2-event.o v4l2-ctrls.o v4l2-subdev.o v4l2-clk.o \
-			v4l2-async.o v4l2-jobqueue.o v4l2-jobqueue-dev.o
+			v4l2-async.o v4l2-jobqueue.o v4l2-jobqueue-dev.o \
+			v4l2-job-generic.o
 
 ifeq ($(CONFIG_COMPAT),y)
   videodev-objs += v4l2-compat-ioctl32.o
diff --git a/drivers/media/v4l2-core/v4l2-job-generic.c b/drivers/media/v4l2-core/v4l2-job-generic.c
new file mode 100644
index 000000000000..ded6464e723a
--- /dev/null
+++ b/drivers/media/v4l2-core/v4l2-job-generic.c
@@ -0,0 +1,394 @@
+/*
+    V4L2 generic jobs implementation
+
+    Copyright (C) 2017  The Chromium project
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+ */
+
+#include <linux/slab.h>
+#include <linux/mutex.h>
+#include <linux/list.h>
+
+#include <media/v4l2-job-generic.h>
+#include <media/v4l2-jobqueue.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-fh.h>
+#include <media/v4l2-dev.h>
+#include <linux/videodev2.h>
+
+#define to_generic_state_handler(hdl) \
+	container_of(hdl, struct v4l2_generic_state_handler, base)
+
+struct v4l2_generic_job_state {
+	struct v4l2_job_state base;
+	struct list_head node;
+
+	int nr_ctrls;
+	struct v4l2_ext_control ctrls[0];
+};
+#define to_generic_job_state(job) \
+	container_of(job, struct v4l2_generic_job_state, base)
+
+/* TODO this is O(n). Find a better way to store/lookup controls */
+static struct v4l2_ext_control *
+v4l2_generic_job_find_control(struct v4l2_generic_job_state *job, u32 id)
+{
+	int i;
+
+	for (i = 0; i < job->nr_ctrls; i++)
+		if (job->ctrls[i].id == id)
+			return &job->ctrls[i];
+
+	return NULL;
+}
+
+static struct v4l2_job_state *
+v4l2_job_generic_job_new(struct v4l2_job_state_handler *_hdl)
+{
+	struct v4l2_generic_state_handler *hdl = to_generic_state_handler(_hdl);
+	struct v4l2_generic_job_state *ret;
+
+	ret = kzalloc(sizeof(*ret) + sizeof(ret->ctrls[0]) * hdl->nr_ctrls,
+		      GFP_KERNEL);
+	if (ret == NULL)
+		return ERR_PTR(-ENOMEM);
+
+	return &ret->base;
+}
+
+static void v4l2_job_generic_job_free(struct v4l2_job_state_handler *hdl,
+					struct v4l2_job_state *_job)
+{
+	struct v4l2_generic_job_state *job = to_generic_job_state(_job);
+
+	kfree(job);
+}
+
+static int v4l2_job_generic_job_apply(struct v4l2_job_state_handler *_hdl)
+{
+	struct v4l2_generic_state_handler *hdl = to_generic_state_handler(_hdl);
+	struct v4l2_generic_job_state *job;
+	struct v4l2_ext_controls ctrls;
+	int ret;
+
+	job = to_generic_job_state(_hdl->active_state);
+
+	if (job->nr_ctrls == 0)
+		return 0;
+
+	ctrls.which = V4L2_CTRL_WHICH_CUR_VAL;
+	ctrls.count = job->nr_ctrls;
+	ctrls.controls = job->ctrls;
+
+	ret = v4l2_s_ext_ctrls(hdl->fh, hdl->ctrl_hdl, &ctrls);
+	if (ret) {
+		pr_err("Cannot set job controls: %d\n", ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int v4l2_job_generic_s_ctrl(struct v4l2_job_state_handler *_hdl,
+				   struct v4l2_ext_control *c)
+{
+	struct v4l2_generic_state_handler *hdl = to_generic_state_handler(_hdl);
+	struct v4l2_generic_job_state *job;
+	struct v4l2_ext_control *ctrl = NULL;
+
+	job = to_generic_job_state(_hdl->current_state);
+
+	ctrl = v4l2_generic_job_find_control(job, c->id);
+	if (!ctrl)
+		ctrl = &job->ctrls[job->nr_ctrls++];
+
+	/* This should never happen as we allocate exactly enough space for
+	 * all the controls of our device */
+	BUG_ON(job->nr_ctrls > hdl->nr_ctrls);
+
+	/* TODO manage pointers, do a try, etc */
+	ctrl->id = c->id;
+	ctrl->value = c->value;
+
+	return 0;
+}
+
+/*
+ * Try to read the control value from the passed job, then its parents, then
+ * the hardware if none has defined a state.
+ */
+static int
+v4l2_job_generic_g_ctrl_locked(struct v4l2_generic_state_handler *hdl,
+			       struct v4l2_generic_job_state *job, u32 ctrl_id,
+			       struct v4l2_ext_control *c, bool upward)
+{
+	struct v4l2_ext_control *ctrl;
+	struct list_head *node;
+
+	if (job == NULL || &job->base == hdl->base.active_state) {
+		struct v4l2_control ctrl = {
+			.id = ctrl_id,
+		};
+		int ret;
+
+		/* TODO terrible! */
+		mutex_unlock(hdl->ctrl_hdl->lock);
+		ret = v4l2_g_ctrl(hdl->ctrl_hdl, &ctrl);
+		mutex_lock(hdl->ctrl_hdl->lock);
+		if (ret < 0)
+			return ret;
+
+		c->value = ctrl.value;
+
+		return 0;
+	}
+
+	ctrl = v4l2_generic_job_find_control(job, ctrl_id);
+	if (ctrl) {
+		/* TODO handle pointers, etc. */
+		c->value = ctrl->value;
+		return 0;
+	}
+
+	if (upward)
+		node = job->node.prev;
+	else
+		node = job->node.next;
+
+	/* That was our last job, request hardware state */
+	if (node == &hdl->jobs)
+		job = NULL;
+	else
+		job = container_of(node, struct v4l2_generic_job_state, node);
+	return v4l2_job_generic_g_ctrl_locked(hdl, job, ctrl_id, c, upward);
+}
+
+static int
+v4l2_job_generic_g_ctrl(struct v4l2_job_state_handler *_hdl, u32 ctrl_id,
+			struct v4l2_ext_control *c, u32 which)
+{
+	struct v4l2_jobqueue *jq = _hdl->jobqueue;
+	struct v4l2_generic_state_handler *hdl = to_generic_state_handler(_hdl);
+	struct v4l2_job_state *_job;
+	struct v4l2_generic_job_state *job;
+	bool upward;
+	int ret;
+
+	if (which != V4L2_CTRL_WHICH_CURJOB_VAL &&
+	    which != V4L2_CTRL_WHICH_DEQJOB_VAL)
+		return -EINVAL;
+
+	v4l2_jobqueue_lock(jq);
+
+	if (which == V4L2_CTRL_WHICH_DEQJOB_VAL) {
+		_job = hdl->base.dequeued_state;
+		upward = true;
+	} else {
+		_job = hdl->base.current_state;
+		upward = false;
+	}
+	if (!_job) {
+		pr_err("No state to query controls from!\n");
+		return -EINVAL;
+	}
+
+	job = to_generic_job_state(_job);
+
+	ret = v4l2_job_generic_g_ctrl_locked(hdl, job, ctrl_id, c, upward);
+
+	v4l2_jobqueue_unlock(jq);
+
+	return ret;
+}
+
+static void v4l2_job_generic_job_complete(struct v4l2_job_state_handler *_hdl)
+{
+	struct v4l2_generic_state_handler *hdl = to_generic_state_handler(_hdl);
+	struct v4l2_generic_job_state *job;
+	struct v4l2_ext_controls cs;
+	struct v4l2_ext_control ctrls[hdl->nr_vol_ctrls];
+	int ret;
+	int i;
+
+	/* We need to update all volatile controls, if any */
+	if (hdl->nr_vol_ctrls == 0)
+		return;
+
+	job = to_generic_job_state(_hdl->active_state);
+
+	memset(job->ctrls, 0, sizeof(job->ctrls[0]) * hdl->nr_ctrls);
+
+	for (i = 0; i < hdl->nr_vol_ctrls; i++)
+		job->ctrls[i].id = hdl->ctrls_ids[i];
+
+	cs.which = V4L2_CTRL_WHICH_CUR_VAL;
+	cs.count = hdl->nr_vol_ctrls;
+	cs.controls = ctrls;
+
+	ret = v4l2_g_ext_ctrls(hdl->ctrl_hdl, &cs);
+	if (ret < 0) {
+		pr_err("Cannot read output controls: %d\n", ret);
+		return;
+	}
+
+	for (i = 0; i < cs.count; i++) {
+		ret = v4l2_job_generic_s_ctrl(_hdl, &ctrls[i]);
+		if (ret < 0)
+			pr_err("Cannot update volatile control value!\n");
+	}
+}
+
+static void v4l2_job_generic_state_changed(struct v4l2_job_state_handler *_hdl,
+					   struct v4l2_job_state *_job,
+					   enum v4l2_job_status status)
+{
+	struct v4l2_generic_state_handler *hdl = to_generic_state_handler(_hdl);
+	struct v4l2_generic_job_state *job = to_generic_job_state(_job);
+
+	switch (status) {
+	case CURRENT:
+		list_add(&job->node, &hdl->jobs);
+		break;
+	case COMPLETED:
+		hdl->last_completed = job;
+		break;
+	case OUT_OF_QUEUE:
+		list_del(&job->node);
+		if (hdl->last_completed == job)
+			hdl->last_completed = NULL;
+		break;
+	default:
+		break;
+	}
+}
+
+static void v4l2_job_generic_ctrl_changed(struct v4l2_job_state_handler *_hdl,
+					  struct v4l2_ctrl *ctrl)
+{
+	struct v4l2_jobqueue *jq = _hdl->jobqueue;
+	struct v4l2_generic_state_handler *hdl = to_generic_state_handler(_hdl);
+	struct v4l2_generic_job_state *job;
+	struct v4l2_ext_control *ext_ctrl;
+
+	/* prevent the job queue from changing state */
+	v4l2_jobqueue_lock(jq);
+
+	job = hdl->last_completed;
+
+	if (!job)
+		goto out;
+
+	/* store the current value of the control into the job so it reflects
+	 * the state at the time it completed */
+	ext_ctrl = v4l2_generic_job_find_control(job, ctrl->id);
+	/* we already have a completion value stored, nothing to do */
+	if (ext_ctrl)
+		goto out;
+
+	ext_ctrl = &job->ctrls[job->nr_ctrls++];
+	ext_ctrl->id = ctrl->id;
+	ext_ctrl->value = ctrl->cur.val;
+
+out:
+	v4l2_jobqueue_unlock(jq);
+}
+
+static int v4l2_job_generic_job_export(struct v4l2_job_state_handler *_hdl)
+{
+	struct v4l2_generic_state_handler *hdl = to_generic_state_handler(_hdl);
+	struct v4l2_generic_job_state *job;
+	struct v4l2_ctrl_handler *ctrl_hdl = hdl->ctrl_hdl;
+	struct v4l2_ctrl *ctrl;
+
+	job = to_generic_job_state(_hdl->current_state);
+
+	/*
+	 * Read and store all controls, so the full state can be reapplied
+	 * when we reuse this state
+	 */
+	mutex_lock(ctrl_hdl->lock);
+
+	list_for_each_entry(ctrl, &ctrl_hdl->ctrls, node) {
+		/* dummy */
+		struct v4l2_ext_control c;
+
+		if (ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
+			continue;
+
+		c.id = ctrl->id;
+		/* get the current value either from the HW or a parent state */
+		v4l2_job_generic_g_ctrl_locked(hdl, job, ctrl->id, &c, false);
+		/* ... and store it */
+		v4l2_job_generic_s_ctrl(&hdl->base, &c);
+	}
+
+	mutex_unlock(ctrl_hdl->lock);
+
+	return 0;
+}
+
+static const struct v4l2_job_state_handler_ops v4l2_generic_job_ops = {
+	.job_new = v4l2_job_generic_job_new,
+	.job_free = v4l2_job_generic_job_free,
+	.job_apply = v4l2_job_generic_job_apply,
+	.job_complete = v4l2_job_generic_job_complete,
+	.job_export = v4l2_job_generic_job_export,
+
+	.s_ctrl = v4l2_job_generic_s_ctrl,
+	.g_ctrl = v4l2_job_generic_g_ctrl,
+
+	.state_changed = v4l2_job_generic_state_changed,
+	.ctrl_changed = v4l2_job_generic_ctrl_changed,
+};
+
+int v4l2_job_generic_init(struct v4l2_generic_state_handler *hdl,
+		    void (*process_active_job)(struct v4l2_job_state_handler *),
+		    struct v4l2_fh *fh, struct video_device *vdev)
+{
+	struct v4l2_ctrl *ctrl;
+	struct v4l2_ctrl_handler *ctrl_hdl;
+
+	hdl->base.process_active_job = process_active_job;
+
+	ctrl_hdl = fh ? fh->ctrl_handler : vdev->ctrl_handler;
+	ctrl_hdl->state_handler = &hdl->base;
+
+	if (fh)
+		fh->state_handler = &hdl->base;
+	else
+		vdev->state_handler = &hdl->base;
+
+	hdl->ctrl_hdl = ctrl_hdl;
+	hdl->fh = fh;
+	INIT_LIST_HEAD(&hdl->jobs);
+	hdl->base.ops = &v4l2_generic_job_ops;
+
+	mutex_lock(ctrl_hdl->lock);
+
+	/* Count how many controls we have to manage */
+	list_for_each_entry(ctrl, &ctrl_hdl->ctrls, node) {
+		/* Reserve permanent space for volatile controls */
+		if (ctrl->flags & V4L2_CTRL_FLAG_VOLATILE) {
+			if (hdl->nr_vol_ctrls >= V4L2_GENERIC_JOB_MAX_CTRLS)
+				return -ENOSPC;
+			hdl->ctrls_ids[hdl->nr_vol_ctrls++] = ctrl->id;
+		}
+
+		hdl->nr_ctrls++;
+	}
+
+	mutex_unlock(ctrl_hdl->lock);
+
+	return 0;
+}
+EXPORT_SYMBOL(v4l2_job_generic_init);
diff --git a/include/media/v4l2-job-generic.h b/include/media/v4l2-job-generic.h
new file mode 100644
index 000000000000..5f6ee55d68e4
--- /dev/null
+++ b/include/media/v4l2-job-generic.h
@@ -0,0 +1,47 @@
+/*
+    V4L2 generic jobs support header.
+
+    Copyright (C) 2017  The Chromium project
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+ */
+
+#ifndef _V4L2_JOB_GENERIC_H
+#define _V4L2_JOB_GENERIC_H
+
+#include <media/v4l2-job-state.h>
+#include <linux/videodev2.h>
+#include <linux/list.h>
+
+struct video_device;
+struct v4l2_fh;
+struct v4l2_ctrl;
+struct v4l2_ctrl_handler;
+struct v4l2_generic_job_state;
+
+#define V4L2_GENERIC_JOB_MAX_CTRLS 32
+struct v4l2_generic_state_handler {
+	struct v4l2_job_state_handler base;
+	struct list_head jobs;
+	struct v4l2_ctrl_handler *ctrl_hdl;
+	struct v4l2_fh *fh;
+	struct v4l2_generic_job_state *last_completed;
+	unsigned int nr_ctrls;
+	unsigned int nr_vol_ctrls;
+	u32 ctrls_ids[V4L2_GENERIC_JOB_MAX_CTRLS];
+};
+
+int v4l2_job_generic_init(struct v4l2_generic_state_handler *handler,
+		    void (*process_active_job)(struct v4l2_job_state_handler *),
+		    struct v4l2_fh *fh, struct video_device *vdev);
+
+#endif
-- 
2.14.2.822.g60be5d43e6-goog

[toc] | [prev] | [next] | [standalone]

#1741373 — [RFC PATCH 3/9] [media] videobuf2: add support for jobs API

From	Alexandre Courbot <acourbot@chromium.org>
Date	2017-09-28 12:00 +0200
Subject	[RFC PATCH 3/9] [media] videobuf2: add support for jobs API
Message-ID	<uuDLS-7o2-69@gated-at.bofh.it>
In reply to	#1741368

Add generic support for jobs in videobuf2. When the jobs API is active,
the passing of buffers to the driver is delayed until their job is
submitted. Drivers need to call the vb2_queue_active_job_buffers()
function in order to receive the buffers corresponding to the active
job.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/videobuf2-core.c | 33 ++++++++++++++++++++++++++++----
 include/media/videobuf2-core.h           | 16 ++++++++++++++++
 2 files changed, 45 insertions(+), 4 deletions(-)

diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
index 14f83cecfa92..2ac880ebe192 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -26,6 +26,7 @@
 
 #include <media/videobuf2-core.h>
 #include <media/v4l2-mc.h>
+#include <media/v4l2-job-state.h>
 
 #include <trace/events/vb2.h>
 
@@ -1304,6 +1305,22 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+void vb2_queue_active_job_buffers(struct vb2_queue *q)
+{
+	const struct v4l2_job_state_handler *hdl = q->state_handler;
+	struct vb2_buffer *vb;
+
+	if (!q->start_streaming_called)
+		return;
+
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		if (!hdl || hdl->active_state == vb->job)
+			__enqueue_in_driver(vb);
+	}
+}
+EXPORT_SYMBOL_GPL(vb2_queue_active_job_buffers);
+
+
 /**
  * vb2_start_streaming() - Attempt to start streaming.
  * @q:		videobuf2 queue
@@ -1320,15 +1337,15 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	struct vb2_buffer *vb;
 	int ret;
 
+	q->start_streaming_called = 1;
+
 	/*
 	 * If any buffers were queued before streamon,
 	 * we can now pass them to driver for processing.
 	 */
-	list_for_each_entry(vb, &q->queued_list, queued_entry)
-		__enqueue_in_driver(vb);
+	vb2_queue_active_job_buffers(q);
 
 	/* Tell the driver to start streaming */
-	q->start_streaming_called = 1;
 	ret = call_qop(q, start_streaming, q,
 		       atomic_read(&q->owned_by_drv_count));
 	if (!ret)
@@ -1398,6 +1415,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	q->queued_count++;
 	q->waiting_for_buffers = false;
 	vb->state = VB2_BUF_STATE_QUEUED;
+	vb->job = q->state_handler ? q->state_handler->current_state : NULL;
 
 	if (pb)
 		call_void_bufop(q, copy_timestamp, vb, pb);
@@ -1407,8 +1425,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	/*
 	 * If already streaming, give the buffer to driver for processing.
 	 * If not, the buffer will be given to driver on next streamon.
+	 *
+	 * If using the jobs API, we will give the buffer to the driver when
+	 * its job becomes active.
 	 */
-	if (q->start_streaming_called)
+	if (q->start_streaming_called && !vb->job)
 		__enqueue_in_driver(vb);
 
 	/* Fill buffer information for the userspace */
@@ -1422,6 +1443,8 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	 * then we can finally call start_streaming().
 	 */
 	if (q->streaming && !q->start_streaming_called &&
+	/* TODO potential issue: what if we have less than min_buffers_needed
+	 * in the next job? */
 	    q->queued_count >= q->min_buffers_needed) {
 		ret = vb2_start_streaming(q);
 		if (ret)
@@ -1728,6 +1751,8 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
 	 * Tell driver to start streaming provided sufficient buffers
 	 * are available.
 	 */
+	/* TODO potential issue: what if we have less than min_buffers_needed
+	 * in the next job? */
 	if (q->queued_count >= q->min_buffers_needed) {
 		ret = v4l_vb2q_enable_media_source(q);
 		if (ret)
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index cb97c224be73..9e172168e011 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -246,6 +246,7 @@ struct vb2_buffer {
 	unsigned int		num_planes;
 	struct vb2_plane	planes[VB2_MAX_PLANES];
 	u64			timestamp;
+	struct v4l2_job_state	*job;
 
 	/* private: internal use only
 	 *
@@ -506,6 +507,7 @@ struct vb2_queue {
 	const struct vb2_ops		*ops;
 	const struct vb2_mem_ops	*mem_ops;
 	const struct vb2_buf_ops	*buf_ops;
+	const struct v4l2_job_state_handler *state_handler;
 
 	void				*drv_priv;
 	unsigned int			buf_struct_size;
@@ -625,6 +627,20 @@ void vb2_discard_done(struct vb2_queue *q);
  */
 int vb2_wait_for_all_buffers(struct vb2_queue *q);
 
+/**
+ * vb2_queue_active_job_buffers() - Pass all buffers for the active job to the
+ *                                  driver
+ *
+ * @q:		videobuf2 queue
+ *
+ * When using the jobs API, buffers are not passed to the driver until their
+ * job becomes active. Drivers using the jobs API are thus expected to call
+ * this function whenever a new job becomes active, so all buffers assigned
+ * to this job are passed to them.
+ */
+void vb2_queue_active_job_buffers(struct vb2_queue *q);
+
+
 /**
  * vb2_core_querybuf() - query video buffer information
  * @q:		videobuf queue
-- 
2.14.2.822.g60be5d43e6-goog

[toc] | [prev] | [next] | [standalone]

#1741374 — [RFC PATCH 4/9] [media] v4l2-ctrls: add support for jobs API

From	Alexandre Courbot <acourbot@chromium.org>
Date	2017-09-28 12:00 +0200
Subject	[RFC PATCH 4/9] [media] v4l2-ctrls: add support for jobs API
Message-ID	<uuDLS-7o2-65@gated-at.bofh.it>
In reply to	#1741368

Add generic support for jobs in the control framework by handling the
new V4L2_CTRL_WHICH_*JOB_VAL which values. This calls the state handler
callbacks in such a case and takes care of control management for
drivers with jobs API support.

Note: this is a very simple and naive way to manage controls in jobs.
Doing this properly will probably require more changes to the control
framework, but the current form is enough to demonstrate the general
ideas.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 50 ++++++++++++++++++++++++++++++++----
 include/media/v4l2-ctrls.h           |  6 +++++
 2 files changed, 51 insertions(+), 5 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index dd1db678718c..fcc644a83cf0 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -27,6 +27,7 @@
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-event.h>
 #include <media/v4l2-dev.h>
+#include <media/v4l2-job-state.h>
 
 #define has_op(master, op) \
 	(master->ops && master->ops->op)
@@ -1603,8 +1604,14 @@ static void new_to_cur(struct v4l2_fh *fh, struct v4l2_ctrl *ctrl, u32 ch_flags)
 
 	/* has_changed is set by cluster_changed */
 	changed = ctrl->has_changed;
-	if (changed)
+	if (changed) {
+		struct v4l2_job_state_handler *state = ctrl->handler->state_handler;
+
+		if (state && state->ops->ctrl_changed)
+			state->ops->ctrl_changed(state, ctrl);
+
 		ptr_to_ptr(ctrl, ctrl->p_new, ctrl->p_cur);
+	}
 
 	if (ch_flags & V4L2_EVENT_CTRL_CH_FLAGS) {
 		/* Note: CH_FLAGS is only set for auto clusters. */
@@ -2738,6 +2745,8 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
 
 		if (cs->which &&
 		    cs->which != V4L2_CTRL_WHICH_DEF_VAL &&
+		    cs->which != V4L2_CTRL_WHICH_CURJOB_VAL &&
+		    cs->which != V4L2_CTRL_WHICH_DEQJOB_VAL &&
 		    V4L2_CTRL_ID2WHICH(id) != cs->which)
 			return -EINVAL;
 
@@ -2817,7 +2826,9 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
    whether there are any controls at all. */
 static int class_check(struct v4l2_ctrl_handler *hdl, u32 which)
 {
-	if (which == 0 || which == V4L2_CTRL_WHICH_DEF_VAL)
+	if (which == 0 || which == V4L2_CTRL_WHICH_DEF_VAL ||
+	    which == V4L2_CTRL_WHICH_CURJOB_VAL ||
+	    which == V4L2_CTRL_WHICH_DEQJOB_VAL)
 		return list_empty(&hdl->ctrl_refs) ? -EINVAL : 0;
 	return find_ref_lock(hdl, which | 1) ? 0 : -EINVAL;
 }
@@ -2829,11 +2840,14 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 {
 	struct v4l2_ctrl_helper helper[4];
 	struct v4l2_ctrl_helper *helpers = helper;
+	struct v4l2_job_state_handler *state;
 	int ret;
 	int i, j;
-	bool def_value;
+	bool def_value, job_value;
 
 	def_value = (cs->which == V4L2_CTRL_WHICH_DEF_VAL);
+	job_value = (cs->which == V4L2_CTRL_WHICH_DEQJOB_VAL ||
+		     cs->which == V4L2_CTRL_WHICH_CURJOB_VAL);
 
 	cs->error_idx = cs->count;
 	cs->which = V4L2_CTRL_ID2WHICH(cs->which);
@@ -2841,6 +2855,10 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 	if (hdl == NULL)
 		return -EINVAL;
 
+	state = hdl->state_handler;
+	if (!state && job_value)
+		return -EINVAL;
+
 	if (cs->count == 0)
 		return class_check(hdl, cs->which);
 
@@ -2874,18 +2892,20 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 		v4l2_ctrl_lock(master);
 
 		/* g_volatile_ctrl will update the new control values */
-		if (!def_value &&
+		if (!def_value && !job_value &&
 		    ((master->flags & V4L2_CTRL_FLAG_VOLATILE) ||
 		    (master->has_volatiles && !is_cur_manual(master)))) {
 			for (j = 0; j < master->ncontrols; j++)
 				cur_to_new(master->cluster[j]);
 			ret = call_op(master, g_volatile_ctrl);
 			ctrl_to_user = new_to_user;
+		} else if (job_value) {
+			ret = state->ops->g_ctrl(state, master->id, cs->controls + i, cs->which);
 		}
 		/* If OK, then copy the current (for non-volatile controls)
 		   or the new (for volatile controls) control values to the
 		   caller */
-		if (!ret) {
+		if (!ret && !job_value) {
 			u32 idx = i;
 
 			do {
@@ -3008,6 +3028,7 @@ static int try_or_set_cluster(struct v4l2_fh *fh, struct v4l2_ctrl *master,
 	/* Don't set if there is no change */
 	if (ret || !set || !cluster_changed(master))
 		return ret;
+
 	ret = call_op(master, s_ctrl);
 	if (ret)
 		return ret;
@@ -3082,6 +3103,8 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 {
 	struct v4l2_ctrl_helper helper[4];
 	struct v4l2_ctrl_helper *helpers = helper;
+	struct v4l2_job_state_handler *state;
+	bool job_value;
 	unsigned i, j;
 	int ret;
 
@@ -3096,6 +3119,14 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 	if (hdl == NULL)
 		return -EINVAL;
 
+	if (cs->which == V4L2_CTRL_WHICH_DEQJOB_VAL)
+		return -EINVAL;
+
+	job_value = (cs->which == V4L2_CTRL_WHICH_CURJOB_VAL);
+	state = hdl->state_handler;
+	if (!state && job_value)
+		return -EINVAL;
+
 	if (cs->count == 0)
 		return class_check(hdl, cs->which);
 
@@ -3121,6 +3152,15 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 		master = helpers[i].mref->ctrl;
 		v4l2_ctrl_lock(master);
 
+		if (job_value) {
+			ret = state->ops->s_ctrl(state, cs->controls + idx);
+			v4l2_ctrl_unlock(master);
+			if (ret)
+				return ret;
+			continue;
+		}
+
+
 		/* Reset the 'is_new' flags of the cluster */
 		for (j = 0; j < master->ncontrols; j++)
 			if (master->cluster[j])
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index 2d2aed56922f..da3eb69d1af1 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -21,8 +21,11 @@
 #include <linux/mutex.h>
 #include <linux/videodev2.h>
 
+#include <media/v4l2-jobqueue.h>
+
 /* forward references */
 struct file;
+struct v4l2_job_state_handler;
 struct v4l2_ctrl_handler;
 struct v4l2_ctrl_helper;
 struct v4l2_ctrl;
@@ -72,6 +75,7 @@ struct v4l2_ctrl_ops {
 	int (*s_ctrl)(struct v4l2_ctrl *ctrl);
 };
 
+
 /**
  * struct v4l2_ctrl_type_ops - The control type operations that the driver
  *			       has to provide.
@@ -257,6 +261,7 @@ struct v4l2_ctrl_ref {
  *	controls: both the controls owned by the handler and those inherited
  *	from other handlers.
  *
+ * @state_handler: State handler to use when jobs API is in use.
  * @_lock:	Default for "lock".
  * @lock:	Lock to control access to this handler and its controls.
  *		May be replaced by the user right after init.
@@ -275,6 +280,7 @@ struct v4l2_ctrl_ref {
  * @error:	The error code of the first failed control addition.
  */
 struct v4l2_ctrl_handler {
+	struct v4l2_job_state_handler *state_handler;
 	struct mutex _lock;
 	struct mutex *lock;
 	struct list_head ctrls;
-- 
2.14.2.822.g60be5d43e6-goog

[toc] | [prev] | [next] | [standalone]

#1741375 — [RFC PATCH 7/9] [media] vim2m: add jobs API support

From	Alexandre Courbot <acourbot@chromium.org>
Date	2017-09-28 12:00 +0200
Subject	[RFC PATCH 7/9] [media] vim2m: add jobs API support
Message-ID	<uuDLS-7o2-71@gated-at.bofh.it>
In reply to	#1741368

Add support for jobs in vim2m, using the generic state handler.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/platform/vim2m.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index 970b9b6dab25..aff21d53d898 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -31,6 +31,8 @@
 #include <media/v4l2-event.h>
 #include <media/videobuf2-vmalloc.h>
 
+#include <media/v4l2-job-generic.h>
+
 MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
 MODULE_AUTHOR("Pawel Osciak, <pawel@osciak.com>");
 MODULE_LICENSE("GPL");
@@ -155,6 +157,7 @@ struct vim2m_ctx {
 	struct vim2m_dev	*dev;
 
 	struct v4l2_ctrl_handler hdl;
+	struct v4l2_generic_state_handler state;
 
 	/* Processed buffers in this transaction */
 	u8			num_processed;
@@ -877,6 +880,15 @@ static const struct v4l2_ctrl_config vim2m_ctrl_trans_num_bufs = {
 	.step = 1,
 };
 
+static void vim2m_process_active_job(struct v4l2_job_state_handler *hdl)
+{
+	struct vim2m_ctx *ctx = container_of(hdl, struct vim2m_ctx, state.base);
+
+	vb2_queue_active_job_buffers(&ctx->fh.m2m_ctx->cap_q_ctx.q);
+	vb2_queue_active_job_buffers(&ctx->fh.m2m_ctx->out_q_ctx.q);
+	v4l2_m2m_try_schedule(ctx->fh.m2m_ctx);
+}
+
 /*
  * File operations
  */
@@ -886,6 +898,7 @@ static int vim2m_open(struct file *file)
 	struct vim2m_ctx *ctx = NULL;
 	struct v4l2_ctrl_handler *hdl;
 	int rc = 0;
+	int ret;
 
 	if (mutex_lock_interruptible(&dev->dev_mutex))
 		return -ERESTARTSYS;
@@ -913,6 +926,15 @@ static int vim2m_open(struct file *file)
 	ctx->fh.ctrl_handler = hdl;
 	v4l2_ctrl_handler_setup(hdl);
 
+	ret = v4l2_job_generic_init(&ctx->state, vim2m_process_active_job,
+				    &ctx->fh, NULL);
+	if (ret) {
+		v4l2_ctrl_handler_free(hdl);
+		v4l2_fh_exit(&ctx->fh);
+		kfree(ctx);
+		goto open_unlock;
+	}
+
 	ctx->q_data[V4L2_M2M_SRC].fmt = &formats[0];
 	ctx->q_data[V4L2_M2M_SRC].width = 640;
 	ctx->q_data[V4L2_M2M_SRC].height = 480;
@@ -934,6 +956,8 @@ static int vim2m_open(struct file *file)
 		goto open_unlock;
 	}
 
+	v4l2_mem_ctx_job_init(ctx->fh.m2m_ctx, &ctx->state.base);
+
 	v4l2_fh_add(&ctx->fh);
 	atomic_inc(&dev->num_inst);
 
-- 
2.14.2.822.g60be5d43e6-goog

[toc] | [prev] | [standalone]

csiph-web

[RFC PATCH 0/9] V4L2 Jobs API WIP

Contents

#1741368 — [RFC PATCH 0/9] V4L2 Jobs API WIP

#1741369 — [RFC PATCH 2/9] [media] v4l2-core: add core jobs API support

#1741370 — [RFC PATCH 5/9] [media] v4l2-job: add generic jobs ops

#1741373 — [RFC PATCH 3/9] [media] videobuf2: add support for jobs API

#1741374 — [RFC PATCH 4/9] [media] v4l2-ctrls: add support for jobs API

#1741375 — [RFC PATCH 7/9] [media] vim2m: add jobs API support