Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1390741 > unrolled thread
| Started by | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| First post | 2016-04-29 06:50 +0200 |
| Last post | 2016-04-29 23:20 +0200 |
| Articles | 20 on this page of 30 — 3 participants |
Back to article view | Back to linux.kernel
[PATCH 00/32] 2nd Iteration of Cache QoS Monitoring support. David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
[PATCH 01/32] perf/x86/intel/cqm: temporarily remove MBM from CQM and cleanup David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
Re: [PATCH 01/32] perf/x86/intel/cqm: temporarily remove MBM from CQM and cleanup Vikas Shivappa <vikas.shivappa@linux.intel.com> - 2016-04-29 22:30 +0200
[PATCH 24/32] perf/x86/intel/cqm: use PERF_INACTIVE_*_READ_* flags in CQM David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
[PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
Re: [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook Peter Zijlstra <peterz@infradead.org> - 2016-04-29 11:00 +0200
Re: [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 20:50 +0200
Re: [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook Vikas Shivappa <vikas.shivappa@linux.intel.com> - 2016-04-29 22:30 +0200
Re: [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 23:00 +0200
[PATCH 14/32] perf/x86/intel/cqm: add preallocation of anodes David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
[PATCH 32/32] perf/stat: revamp error handling for snapshot and per_pkg events David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
[PATCH 09/32] perf/x86/intel/cqm: add per-package RMIDs, data and locks David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
Re: [PATCH 09/32] perf/x86/intel/cqm: add per-package RMIDs, data and locks Vikas Shivappa <vikas.shivappa@linux.intel.com> - 2016-04-29 23:00 +0200
[PATCH 29/32] perf,perf/x86,perf/powerpc,perf/arm,perf/*: add int error return to pmu::read David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
[PATCH 06/32] x86/intel,cqm: add CONFIG_INTEL_RDT configuration flag and refactor PQR David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 06:50 +0200
[PATCH 15/32] perf/core: add hooks to expose architecture specific features in perf_cgroup David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 18/32] perf/x86/intel/cqm: use pmu::event_terminate David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 16/32] perf/x86/intel/cqm: add cgroup support David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 17/32] perf/core: adding pmu::event_terminate David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 05/32] perf/core: remove unused pmu->count David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 04/32] perf/x86/intel/cqm: make read of RMIDs per package (Temporal) David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 11/32] perf/x86/intel/cqm: (I)state and limbo prmids David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 12/32] perf/x86/intel/cqm: add per-package RMID rotation David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 08/32] perf/x86/intel/cqm: prepare for next patches David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
Re: [PATCH 08/32] perf/x86/intel/cqm: prepare for next patches Peter Zijlstra <peterz@infradead.org> - 2016-04-29 11:20 +0200
[PATCH 07/32] perf/x86/intel/cqm: separate CQM PMU's attributes from x86 PMU David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 19/32] perf/core: introduce PMU event flag PERF_CGROUP_NO_RECURSION David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
[PATCH 10/32] perf/x86/intel/cqm: basic RMID hierarchy with per package rmids David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 07:00 +0200
Re: [PATCH 00/32] 2nd Iteration of Cache QoS Monitoring support. Vikas Shivappa <vikas.shivappa@linux.intel.com> - 2016-04-29 23:10 +0200
Re: [PATCH 00/32] 2nd Iteration of Cache QoS Monitoring support. David Carrillo-Cisneros <davidcc@google.com> - 2016-04-29 23:20 +0200
Page 1 of 2 [1] 2 Next page →
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 00/32] 2nd Iteration of Cache QoS Monitoring support. |
| Message-ID | <rt8xj-8qy-3@gated-at.bofh.it> |
This series introduces the next iteration of kernel support for the
Cache QoS Monitoring (CQM) technology available in Intel Xeon processors.
One of the main limitations of the previous version is the inability
to simultaneously monitor:
1) cpu event and any other event in that cpu.
2) cgroup events for cgroups in same descendancy line.
3) cgroup events and any thread event of a cgroup in the same
descendancy line.
Another limitation is that monitoring for a cgroup was enabled/disabled by
the existence of a perf event for that cgroup. Since the event
llc_occupancy measures changes in occupancy rather than total occupancy,
in order to read meaningful llc_occupancy values, an event should be
enabled for a long enough period of time. The overhead in context switches
caused by the perf events is undesired in some sensitive scenarios.
This series of patches addresses the shortcomings mentioned above and,
add some other improvements. The main changes are:
- No more potential conflicts between different events. New
version builds a hierarchy of RMIDs that captures the dependency
between monitored cgroups. llc_occupancy for cgroup is the sum of
llc_occupancies for that cgroup RMID and all other RMIDs in the
cgroups subtree (both monitored cgroups and threads).
- A cgroup integration that allows to monitor the a cgroup without
creating a perf event, decreasing the context switch overhead.
Monitoring is controlled by a boolean cgroup subsystem attribute
in each perf cgroup, this is:
echo 1 > cgroup_path/perf_event.cqm_cont_monitoring
starts CQM monitoring whether or not there is a perf_event
attached to the cgroup. Setting the attribute to 0 makes
monitoring dependent on the existence of a perf_event.
A perf_event is always required in order to read llc_occupancy.
This cgroup integration uses Intel's PQR code and is intended to
be used by upcoming versions of Intel's CAT.
- A more stable rotation algorithm: New algorithm uses SLOs that
guarantee:
- A minimum of enabled time for monitored cgroups and
threads.
- A maximum time disabled before error is introduced by
reusing dirty RMIDs.
- A minimum rate at which RMIDs recycling must progress.
- Reduced impact of stealing/rotation of RMIDs: The new algorithm
accounts the residual occupancy held by limbo RMIDs towards the
former owner of the limbo RMID, decreasing the error introduced
by RMID rotation.
It also allows a limbo RMID to be reused by its former owner when
appropriate, decreasing the potential error of reusing dirty RMIDs
and allowing to make progress even if most limbo RMIDs do not
drop occupancy fast enough.
- Elimination of pmu::count: perf generic's perf_event_count()
perform a quick add of atomic types. The introduction of
pmu::count in the previous CQM series to read occupancy for thread
events changed the behavior of perf_event_count() by performing a
potentially slow IPI and write/read to MSR. It also made pmu::read
to have different behaviors depending on whether the event was a
cpu/cgroup event or a thread. This patches serie removes the custom
pmu::count from CQM and provides a consistent behavior for all
calls of perf_event_read .
- Added error return for pmu::read: Reads to CQM events may fail
due to stealing of RMIDs, even after successfully adding an event
to a PMU. This patch series expands pmu::read with an int return
value and propagates the error to callers that can fail
(ie. perf_read).
The ability to fail of pmu::read is consistent with the recent
changes that allow perf_event_read to fail for transactional
reading of event groups.
- Introduces the field pmu_event_flags that contain flags set by
the PMU to signal variations on the default behavior to perf's
generic code. In this series, three flags are introduced:
- PERF_CGROUP_NO_RECURSION : Signals generic code to add
events of the cgroup ancestors of a cgroup.
- PERF_INACTIVE_CPU_READ_PKG: Signals generic coda that
this CPU event can be read in any CPU in its event::cpu's
package, even if the event is not active.
- PERF_INACTIVE_EV_READ_ANY_CPU: Signals generic code that
this event can be read in any CPU in any package in the
system even if the event is not active.
Using the above flags takes advantage of the CQM's hw ability to
read llc_occupancy even when the associated perf event is not
running in a CPU.
This patch series also updates the perf tool to fix error handling and to
better handle the idiosyncrasies of snapshot and per-pkg events.
David Carrillo-Cisneros (31):
perf/x86/intel/cqm: temporarily remove MBM from CQM and cleanup
perf/x86/intel/cqm: remove check for conflicting events
perf/x86/intel/cqm: remove all code for rotation of RMIDs
perf/x86/intel/cqm: make read of RMIDs per package (Temporal)
perf/core: remove unused pmu->count
x86/intel,cqm: add CONFIG_INTEL_RDT configuration flag and refactor
PQR
perf/x86/intel/cqm: separate CQM PMU's attributes from x86 PMU
perf/x86/intel/cqm: prepare for next patches
perf/x86/intel/cqm: add per-package RMIDs, data and locks
perf/x86/intel/cqm: basic RMID hierarchy with per package rmids
perf/x86/intel/cqm: (I)state and limbo prmids
perf/x86/intel/cqm: add per-package RMID rotation
perf/x86/intel/cqm: add polled update of RMID's llc_occupancy
perf/x86/intel/cqm: add preallocation of anodes
perf/core: add hooks to expose architecture specific features in
perf_cgroup
perf/x86/intel/cqm: add cgroup support
perf/core: adding pmu::event_terminate
perf/x86/intel/cqm: use pmu::event_terminate
perf/core: introduce PMU event flag PERF_CGROUP_NO_RECURSION
x86/intel/cqm: use PERF_CGROUP_NO_RECURSION in CQM
perf/x86/intel/cqm: handle inherit event and inherit_stat flag
perf/x86/intel/cqm: introduce read_subtree
perf/core: introduce PERF_INACTIVE_*_READ_* flags
perf/x86/intel/cqm: use PERF_INACTIVE_*_READ_* flags in CQM
sched: introduce the finish_arch_pre_lock_switch() scheduler hook
perf/x86/intel/cqm: integrate CQM cgroups with scheduler
perf/core: add perf_event cgroup hooks for subsystem attributes
perf/x86/intel/cqm: add CQM attributes to perf_event cgroup
perf,perf/x86,perf/powerpc,perf/arm,perf/*: add int error return to
pmu::read
perf,perf/x86: add hook perf_event_arch_exec
perf/stat: revamp error handling for snapshot and per_pkg events
Stephane Eranian (1):
perf/stat: fix bug in handling events in error state
arch/alpha/kernel/perf_event.c | 3 +-
arch/arc/kernel/perf_event.c | 3 +-
arch/arm64/include/asm/hw_breakpoint.h | 2 +-
arch/arm64/kernel/hw_breakpoint.c | 3 +-
arch/metag/kernel/perf/perf_event.c | 5 +-
arch/mips/kernel/perf_event_mipsxx.c | 3 +-
arch/powerpc/include/asm/hw_breakpoint.h | 2 +-
arch/powerpc/kernel/hw_breakpoint.c | 3 +-
arch/powerpc/perf/core-book3s.c | 11 +-
arch/powerpc/perf/core-fsl-emb.c | 5 +-
arch/powerpc/perf/hv-24x7.c | 5 +-
arch/powerpc/perf/hv-gpci.c | 3 +-
arch/s390/kernel/perf_cpum_cf.c | 5 +-
arch/s390/kernel/perf_cpum_sf.c | 3 +-
arch/sh/include/asm/hw_breakpoint.h | 2 +-
arch/sh/kernel/hw_breakpoint.c | 3 +-
arch/sparc/kernel/perf_event.c | 2 +-
arch/tile/kernel/perf_event.c | 3 +-
arch/x86/Kconfig | 6 +
arch/x86/events/amd/ibs.c | 2 +-
arch/x86/events/amd/iommu.c | 5 +-
arch/x86/events/amd/uncore.c | 3 +-
arch/x86/events/core.c | 3 +-
arch/x86/events/intel/Makefile | 3 +-
arch/x86/events/intel/bts.c | 3 +-
arch/x86/events/intel/cqm.c | 3847 +++++++++++++++++++++---------
arch/x86/events/intel/cqm.h | 519 ++++
arch/x86/events/intel/cstate.c | 3 +-
arch/x86/events/intel/pt.c | 3 +-
arch/x86/events/intel/rapl.c | 3 +-
arch/x86/events/intel/uncore.c | 3 +-
arch/x86/events/intel/uncore.h | 2 +-
arch/x86/events/msr.c | 3 +-
arch/x86/include/asm/hw_breakpoint.h | 2 +-
arch/x86/include/asm/perf_event.h | 41 +
arch/x86/include/asm/pqr_common.h | 74 +
arch/x86/include/asm/processor.h | 4 +
arch/x86/kernel/cpu/Makefile | 4 +
arch/x86/kernel/cpu/pqr_common.c | 43 +
arch/x86/kernel/hw_breakpoint.c | 3 +-
arch/x86/kvm/pmu.h | 10 +-
drivers/bus/arm-cci.c | 3 +-
drivers/bus/arm-ccn.c | 3 +-
drivers/perf/arm_pmu.c | 3 +-
include/linux/perf_event.h | 91 +-
kernel/events/core.c | 170 +-
kernel/sched/core.c | 1 +
kernel/sched/sched.h | 3 +
kernel/trace/bpf_trace.c | 5 +-
tools/perf/builtin-stat.c | 43 +-
tools/perf/util/counts.h | 19 +
tools/perf/util/evsel.c | 44 +-
tools/perf/util/evsel.h | 8 +-
tools/perf/util/stat.c | 35 +-
54 files changed, 3746 insertions(+), 1337 deletions(-)
create mode 100644 arch/x86/events/intel/cqm.h
create mode 100644 arch/x86/include/asm/pqr_common.h
create mode 100644 arch/x86/kernel/cpu/pqr_common.c
--
2.8.0.rc3.226.g39d4020
[toc] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 01/32] perf/x86/intel/cqm: temporarily remove MBM from CQM and cleanup |
| Message-ID | <rt8xk-8qy-29@gated-at.bofh.it> |
| In reply to | #1390741 |
Removing MBM code from arch/x86/events/intel/cqm.c. MBM will be added
using the new RMID infrastucture introduced in this patch series.
Also, remove updates to CQM that are superseded by this series.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/x86/events/intel/cqm.c | 486 ++++----------------------------------------
include/linux/perf_event.h | 1 -
2 files changed, 44 insertions(+), 443 deletions(-)
diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index 7b5fd81..1b064c4 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -13,16 +13,8 @@
#define MSR_IA32_QM_CTR 0x0c8e
#define MSR_IA32_QM_EVTSEL 0x0c8d
-#define MBM_CNTR_WIDTH 24
-/*
- * Guaranteed time in ms as per SDM where MBM counters will not overflow.
- */
-#define MBM_CTR_OVERFLOW_TIME 1000
-
static u32 cqm_max_rmid = -1;
static unsigned int cqm_l3_scale; /* supposedly cacheline size */
-static bool cqm_enabled, mbm_enabled;
-unsigned int mbm_socket_max;
/**
* struct intel_pqr_state - State cache for the PQR MSR
@@ -50,37 +42,8 @@ struct intel_pqr_state {
* interrupts disabled, which is sufficient for the protection.
*/
static DEFINE_PER_CPU(struct intel_pqr_state, pqr_state);
-static struct hrtimer *mbm_timers;
-/**
- * struct sample - mbm event's (local or total) data
- * @total_bytes #bytes since we began monitoring
- * @prev_msr previous value of MSR
- */
-struct sample {
- u64 total_bytes;
- u64 prev_msr;
-};
/*
- * samples profiled for total memory bandwidth type events
- */
-static struct sample *mbm_total;
-/*
- * samples profiled for local memory bandwidth type events
- */
-static struct sample *mbm_local;
-
-#define pkg_id topology_physical_package_id(smp_processor_id())
-/*
- * rmid_2_index returns the index for the rmid in mbm_local/mbm_total array.
- * mbm_total[] and mbm_local[] are linearly indexed by socket# * max number of
- * rmids per socket, an example is given below
- * RMID1 of Socket0: vrmid = 1
- * RMID1 of Socket1: vrmid = 1 * (cqm_max_rmid + 1) + 1
- * RMID1 of Socket2: vrmid = 2 * (cqm_max_rmid + 1) + 1
- */
-#define rmid_2_index(rmid) ((pkg_id * (cqm_max_rmid + 1)) + rmid)
-/*
* Protects cache_cgroups and cqm_rmid_free_lru and cqm_rmid_limbo_lru.
* Also protects event->hw.cqm_rmid
*
@@ -102,13 +65,9 @@ static cpumask_t cqm_cpumask;
#define RMID_VAL_ERROR (1ULL << 63)
#define RMID_VAL_UNAVAIL (1ULL << 62)
-/*
- * Event IDs are used to program IA32_QM_EVTSEL before reading event
- * counter from IA32_QM_CTR
- */
-#define QOS_L3_OCCUP_EVENT_ID 0x01
-#define QOS_MBM_TOTAL_EVENT_ID 0x02
-#define QOS_MBM_LOCAL_EVENT_ID 0x03
+#define QOS_L3_OCCUP_EVENT_ID (1 << 0)
+
+#define QOS_EVENT_MASK QOS_L3_OCCUP_EVENT_ID
/*
* This is central to the rotation algorithm in __intel_cqm_rmid_rotate().
@@ -252,21 +211,6 @@ static void __put_rmid(u32 rmid)
list_add_tail(&entry->list, &cqm_rmid_limbo_lru);
}
-static void cqm_cleanup(void)
-{
- int i;
-
- if (!cqm_rmid_ptrs)
- return;
-
- for (i = 0; i < cqm_max_rmid; i++)
- kfree(cqm_rmid_ptrs[i]);
-
- kfree(cqm_rmid_ptrs);
- cqm_rmid_ptrs = NULL;
- cqm_enabled = false;
-}
-
static int intel_cqm_setup_rmid_cache(void)
{
struct cqm_rmid_entry *entry;
@@ -274,7 +218,7 @@ static int intel_cqm_setup_rmid_cache(void)
int r = 0;
nr_rmids = cqm_max_rmid + 1;
- cqm_rmid_ptrs = kzalloc(sizeof(struct cqm_rmid_entry *) *
+ cqm_rmid_ptrs = kmalloc(sizeof(struct cqm_rmid_entry *) *
nr_rmids, GFP_KERNEL);
if (!cqm_rmid_ptrs)
return -ENOMEM;
@@ -305,9 +249,11 @@ static int intel_cqm_setup_rmid_cache(void)
mutex_unlock(&cache_mutex);
return 0;
-
fail:
- cqm_cleanup();
+ while (r--)
+ kfree(cqm_rmid_ptrs[r]);
+
+ kfree(cqm_rmid_ptrs);
return -ENOMEM;
}
@@ -335,13 +281,9 @@ static bool __match_event(struct perf_event *a, struct perf_event *b)
/*
* Events that target same task are placed into the same cache group.
- * Mark it as a multi event group, so that we update ->count
- * for every event rather than just the group leader later.
*/
- if (a->hw.target == b->hw.target) {
- b->hw.is_group_event = true;
+ if (a->hw.target == b->hw.target)
return true;
- }
/*
* Are we an inherited event?
@@ -450,26 +392,10 @@ static bool __conflict_event(struct perf_event *a, struct perf_event *b)
struct rmid_read {
u32 rmid;
- u32 evt_type;
atomic64_t value;
};
static void __intel_cqm_event_count(void *info);
-static void init_mbm_sample(u32 rmid, u32 evt_type);
-static void __intel_mbm_event_count(void *info);
-
-static bool is_mbm_event(int e)
-{
- return (e >= QOS_MBM_TOTAL_EVENT_ID && e <= QOS_MBM_LOCAL_EVENT_ID);
-}
-
-static void cqm_mask_call(struct rmid_read *rr)
-{
- if (is_mbm_event(rr->evt_type))
- on_each_cpu_mask(&cqm_cpumask, __intel_mbm_event_count, rr, 1);
- else
- on_each_cpu_mask(&cqm_cpumask, __intel_cqm_event_count, rr, 1);
-}
/*
* Exchange the RMID of a group of events.
@@ -487,12 +413,12 @@ static u32 intel_cqm_xchg_rmid(struct perf_event *group, u32 rmid)
*/
if (__rmid_valid(old_rmid) && !__rmid_valid(rmid)) {
struct rmid_read rr = {
- .rmid = old_rmid,
- .evt_type = group->attr.config,
.value = ATOMIC64_INIT(0),
+ .rmid = old_rmid,
};
- cqm_mask_call(&rr);
+ on_each_cpu_mask(&cqm_cpumask, __intel_cqm_event_count,
+ &rr, 1);
local64_set(&group->count, atomic64_read(&rr.value));
}
@@ -504,22 +430,6 @@ static u32 intel_cqm_xchg_rmid(struct perf_event *group, u32 rmid)
raw_spin_unlock_irq(&cache_lock);
- /*
- * If the allocation is for mbm, init the mbm stats.
- * Need to check if each event in the group is mbm event
- * because there could be multiple type of events in the same group.
- */
- if (__rmid_valid(rmid)) {
- event = group;
- if (is_mbm_event(event->attr.config))
- init_mbm_sample(rmid, event->attr.config);
-
- list_for_each_entry(event, head, hw.cqm_group_entry) {
- if (is_mbm_event(event->attr.config))
- init_mbm_sample(rmid, event->attr.config);
- }
- }
-
return old_rmid;
}
@@ -927,72 +837,6 @@ static void intel_cqm_rmid_rotate(struct work_struct *work)
schedule_delayed_work(&intel_cqm_rmid_work, delay);
}
-static u64 update_sample(unsigned int rmid, u32 evt_type, int first)
-{
- struct sample *mbm_current;
- u32 vrmid = rmid_2_index(rmid);
- u64 val, bytes, shift;
- u32 eventid;
-
- if (evt_type == QOS_MBM_LOCAL_EVENT_ID) {
- mbm_current = &mbm_local[vrmid];
- eventid = QOS_MBM_LOCAL_EVENT_ID;
- } else {
- mbm_current = &mbm_total[vrmid];
- eventid = QOS_MBM_TOTAL_EVENT_ID;
- }
-
- wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid);
- rdmsrl(MSR_IA32_QM_CTR, val);
- if (val & (RMID_VAL_ERROR | RMID_VAL_UNAVAIL))
- return mbm_current->total_bytes;
-
- if (first) {
- mbm_current->prev_msr = val;
- mbm_current->total_bytes = 0;
- return mbm_current->total_bytes;
- }
-
- /*
- * The h/w guarantees that counters will not overflow
- * so long as we poll them at least once per second.
- */
- shift = 64 - MBM_CNTR_WIDTH;
- bytes = (val << shift) - (mbm_current->prev_msr << shift);
- bytes >>= shift;
-
- bytes *= cqm_l3_scale;
-
- mbm_current->total_bytes += bytes;
- mbm_current->prev_msr = val;
-
- return mbm_current->total_bytes;
-}
-
-static u64 rmid_read_mbm(unsigned int rmid, u32 evt_type)
-{
- return update_sample(rmid, evt_type, 0);
-}
-
-static void __intel_mbm_event_init(void *info)
-{
- struct rmid_read *rr = info;
-
- update_sample(rr->rmid, rr->evt_type, 1);
-}
-
-static void init_mbm_sample(u32 rmid, u32 evt_type)
-{
- struct rmid_read rr = {
- .rmid = rmid,
- .evt_type = evt_type,
- .value = ATOMIC64_INIT(0),
- };
-
- /* on each socket, init sample */
- on_each_cpu_mask(&cqm_cpumask, __intel_mbm_event_init, &rr, 1);
-}
-
/*
* Find a group and setup RMID.
*
@@ -1005,7 +849,6 @@ static void intel_cqm_setup_event(struct perf_event *event,
bool conflict = false;
u32 rmid;
- event->hw.is_group_event = false;
list_for_each_entry(iter, &cache_groups, hw.cqm_groups_entry) {
rmid = iter->hw.cqm_rmid;
@@ -1013,8 +856,6 @@ static void intel_cqm_setup_event(struct perf_event *event,
/* All tasks in a group share an RMID */
event->hw.cqm_rmid = rmid;
*group = iter;
- if (is_mbm_event(event->attr.config) && __rmid_valid(rmid))
- init_mbm_sample(rmid, event->attr.config);
return;
}
@@ -1031,9 +872,6 @@ static void intel_cqm_setup_event(struct perf_event *event,
else
rmid = __get_rmid();
- if (is_mbm_event(event->attr.config) && __rmid_valid(rmid))
- init_mbm_sample(rmid, event->attr.config);
-
event->hw.cqm_rmid = rmid;
}
@@ -1055,10 +893,7 @@ static void intel_cqm_event_read(struct perf_event *event)
if (!__rmid_valid(rmid))
goto out;
- if (is_mbm_event(event->attr.config))
- val = rmid_read_mbm(rmid, event->attr.config);
- else
- val = __rmid_read(rmid);
+ val = __rmid_read(rmid);
/*
* Ignore this reading on error states and do not update the value.
@@ -1089,100 +924,10 @@ static inline bool cqm_group_leader(struct perf_event *event)
return !list_empty(&event->hw.cqm_groups_entry);
}
-static void __intel_mbm_event_count(void *info)
-{
- struct rmid_read *rr = info;
- u64 val;
-
- val = rmid_read_mbm(rr->rmid, rr->evt_type);
- if (val & (RMID_VAL_ERROR | RMID_VAL_UNAVAIL))
- return;
- atomic64_add(val, &rr->value);
-}
-
-static enum hrtimer_restart mbm_hrtimer_handle(struct hrtimer *hrtimer)
-{
- struct perf_event *iter, *iter1;
- int ret = HRTIMER_RESTART;
- struct list_head *head;
- unsigned long flags;
- u32 grp_rmid;
-
- /*
- * Need to cache_lock as the timer Event Select MSR reads
- * can race with the mbm/cqm count() and mbm_init() reads.
- */
- raw_spin_lock_irqsave(&cache_lock, flags);
-
- if (list_empty(&cache_groups)) {
- ret = HRTIMER_NORESTART;
- goto out;
- }
-
- list_for_each_entry(iter, &cache_groups, hw.cqm_groups_entry) {
- grp_rmid = iter->hw.cqm_rmid;
- if (!__rmid_valid(grp_rmid))
- continue;
- if (is_mbm_event(iter->attr.config))
- update_sample(grp_rmid, iter->attr.config, 0);
-
- head = &iter->hw.cqm_group_entry;
- if (list_empty(head))
- continue;
- list_for_each_entry(iter1, head, hw.cqm_group_entry) {
- if (!iter1->hw.is_group_event)
- break;
- if (is_mbm_event(iter1->attr.config))
- update_sample(iter1->hw.cqm_rmid,
- iter1->attr.config, 0);
- }
- }
-
- hrtimer_forward_now(hrtimer, ms_to_ktime(MBM_CTR_OVERFLOW_TIME));
-out:
- raw_spin_unlock_irqrestore(&cache_lock, flags);
-
- return ret;
-}
-
-static void __mbm_start_timer(void *info)
-{
- hrtimer_start(&mbm_timers[pkg_id], ms_to_ktime(MBM_CTR_OVERFLOW_TIME),
- HRTIMER_MODE_REL_PINNED);
-}
-
-static void __mbm_stop_timer(void *info)
-{
- hrtimer_cancel(&mbm_timers[pkg_id]);
-}
-
-static void mbm_start_timers(void)
-{
- on_each_cpu_mask(&cqm_cpumask, __mbm_start_timer, NULL, 1);
-}
-
-static void mbm_stop_timers(void)
-{
- on_each_cpu_mask(&cqm_cpumask, __mbm_stop_timer, NULL, 1);
-}
-
-static void mbm_hrtimer_init(void)
-{
- struct hrtimer *hr;
- int i;
-
- for (i = 0; i < mbm_socket_max; i++) {
- hr = &mbm_timers[i];
- hrtimer_init(hr, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
- hr->function = mbm_hrtimer_handle;
- }
-}
-
static u64 intel_cqm_event_count(struct perf_event *event)
{
unsigned long flags;
struct rmid_read rr = {
- .evt_type = event->attr.config,
.value = ATOMIC64_INIT(0),
};
@@ -1195,9 +940,7 @@ static u64 intel_cqm_event_count(struct perf_event *event)
return __perf_event_count(event);
/*
- * Only the group leader gets to report values except in case of
- * multiple events in the same group, we still need to read the
- * other events.This stops us
+ * Only the group leader gets to report values. This stops us
* reporting duplicate values to userspace, and gives us a clear
* rule for which task gets to report the values.
*
@@ -1205,7 +948,7 @@ static u64 intel_cqm_event_count(struct perf_event *event)
* specific packages - we forfeit that ability when we create
* task events.
*/
- if (!cqm_group_leader(event) && !event->hw.is_group_event)
+ if (!cqm_group_leader(event))
return 0;
/*
@@ -1232,7 +975,7 @@ static u64 intel_cqm_event_count(struct perf_event *event)
if (!__rmid_valid(rr.rmid))
goto out;
- cqm_mask_call(&rr);
+ on_each_cpu_mask(&cqm_cpumask, __intel_cqm_event_count, &rr, 1);
raw_spin_lock_irqsave(&cache_lock, flags);
if (event->hw.cqm_rmid == rr.rmid)
@@ -1303,14 +1046,8 @@ static int intel_cqm_event_add(struct perf_event *event, int mode)
static void intel_cqm_event_destroy(struct perf_event *event)
{
struct perf_event *group_other = NULL;
- unsigned long flags;
mutex_lock(&cache_mutex);
- /*
- * Hold the cache_lock as mbm timer handlers could be
- * scanning the list of events.
- */
- raw_spin_lock_irqsave(&cache_lock, flags);
/*
* If there's another event in this group...
@@ -1342,14 +1079,6 @@ static void intel_cqm_event_destroy(struct perf_event *event)
}
}
- raw_spin_unlock_irqrestore(&cache_lock, flags);
-
- /*
- * Stop the mbm overflow timers when the last event is destroyed.
- */
- if (mbm_enabled && list_empty(&cache_groups))
- mbm_stop_timers();
-
mutex_unlock(&cache_mutex);
}
@@ -1357,13 +1086,11 @@ static int intel_cqm_event_init(struct perf_event *event)
{
struct perf_event *group = NULL;
bool rotate = false;
- unsigned long flags;
if (event->attr.type != intel_cqm_pmu.type)
return -ENOENT;
- if ((event->attr.config < QOS_L3_OCCUP_EVENT_ID) ||
- (event->attr.config > QOS_MBM_LOCAL_EVENT_ID))
+ if (event->attr.config & ~QOS_EVENT_MASK)
return -EINVAL;
/* unsupported modes and filters */
@@ -1383,21 +1110,9 @@ static int intel_cqm_event_init(struct perf_event *event)
mutex_lock(&cache_mutex);
- /*
- * Start the mbm overflow timers when the first event is created.
- */
- if (mbm_enabled && list_empty(&cache_groups))
- mbm_start_timers();
-
/* Will also set rmid */
intel_cqm_setup_event(event, &group);
- /*
- * Hold the cache_lock as mbm timer handlers be
- * scanning the list of events.
- */
- raw_spin_lock_irqsave(&cache_lock, flags);
-
if (group) {
list_add_tail(&event->hw.cqm_group_entry,
&group->hw.cqm_group_entry);
@@ -1416,7 +1131,6 @@ static int intel_cqm_event_init(struct perf_event *event)
rotate = true;
}
- raw_spin_unlock_irqrestore(&cache_lock, flags);
mutex_unlock(&cache_mutex);
if (rotate)
@@ -1431,16 +1145,6 @@ EVENT_ATTR_STR(llc_occupancy.unit, intel_cqm_llc_unit, "Bytes");
EVENT_ATTR_STR(llc_occupancy.scale, intel_cqm_llc_scale, NULL);
EVENT_ATTR_STR(llc_occupancy.snapshot, intel_cqm_llc_snapshot, "1");
-EVENT_ATTR_STR(total_bytes, intel_cqm_total_bytes, "event=0x02");
-EVENT_ATTR_STR(total_bytes.per-pkg, intel_cqm_total_bytes_pkg, "1");
-EVENT_ATTR_STR(total_bytes.unit, intel_cqm_total_bytes_unit, "MB");
-EVENT_ATTR_STR(total_bytes.scale, intel_cqm_total_bytes_scale, "1e-6");
-
-EVENT_ATTR_STR(local_bytes, intel_cqm_local_bytes, "event=0x03");
-EVENT_ATTR_STR(local_bytes.per-pkg, intel_cqm_local_bytes_pkg, "1");
-EVENT_ATTR_STR(local_bytes.unit, intel_cqm_local_bytes_unit, "MB");
-EVENT_ATTR_STR(local_bytes.scale, intel_cqm_local_bytes_scale, "1e-6");
-
static struct attribute *intel_cqm_events_attr[] = {
EVENT_PTR(intel_cqm_llc),
EVENT_PTR(intel_cqm_llc_pkg),
@@ -1450,38 +1154,9 @@ static struct attribute *intel_cqm_events_attr[] = {
NULL,
};
-static struct attribute *intel_mbm_events_attr[] = {
- EVENT_PTR(intel_cqm_total_bytes),
- EVENT_PTR(intel_cqm_local_bytes),
- EVENT_PTR(intel_cqm_total_bytes_pkg),
- EVENT_PTR(intel_cqm_local_bytes_pkg),
- EVENT_PTR(intel_cqm_total_bytes_unit),
- EVENT_PTR(intel_cqm_local_bytes_unit),
- EVENT_PTR(intel_cqm_total_bytes_scale),
- EVENT_PTR(intel_cqm_local_bytes_scale),
- NULL,
-};
-
-static struct attribute *intel_cmt_mbm_events_attr[] = {
- EVENT_PTR(intel_cqm_llc),
- EVENT_PTR(intel_cqm_total_bytes),
- EVENT_PTR(intel_cqm_local_bytes),
- EVENT_PTR(intel_cqm_llc_pkg),
- EVENT_PTR(intel_cqm_total_bytes_pkg),
- EVENT_PTR(intel_cqm_local_bytes_pkg),
- EVENT_PTR(intel_cqm_llc_unit),
- EVENT_PTR(intel_cqm_total_bytes_unit),
- EVENT_PTR(intel_cqm_local_bytes_unit),
- EVENT_PTR(intel_cqm_llc_scale),
- EVENT_PTR(intel_cqm_total_bytes_scale),
- EVENT_PTR(intel_cqm_local_bytes_scale),
- EVENT_PTR(intel_cqm_llc_snapshot),
- NULL,
-};
-
static struct attribute_group intel_cqm_events_group = {
.name = "events",
- .attrs = NULL,
+ .attrs = intel_cqm_events_attr,
};
PMU_FORMAT_ATTR(event, "config:0-7");
@@ -1569,12 +1244,15 @@ static struct pmu intel_cqm_pmu = {
static inline void cqm_pick_event_reader(int cpu)
{
- int reader;
+ int phys_id = topology_physical_package_id(cpu);
+ int i;
+
+ for_each_cpu(i, &cqm_cpumask) {
+ if (phys_id == topology_physical_package_id(i))
+ return; /* already got reader for this socket */
+ }
- /* First online cpu in package becomes the reader */
- reader = cpumask_any_and(&cqm_cpumask, topology_core_cpumask(cpu));
- if (reader >= nr_cpu_ids)
- cpumask_set_cpu(cpu, &cqm_cpumask);
+ cpumask_set_cpu(cpu, &cqm_cpumask);
}
static void intel_cqm_cpu_starting(unsigned int cpu)
@@ -1592,17 +1270,24 @@ static void intel_cqm_cpu_starting(unsigned int cpu)
static void intel_cqm_cpu_exit(unsigned int cpu)
{
- int target;
+ int phys_id = topology_physical_package_id(cpu);
+ int i;
- /* Is @cpu the current cqm reader for this package ? */
+ /*
+ * Is @cpu a designated cqm reader?
+ */
if (!cpumask_test_and_clear_cpu(cpu, &cqm_cpumask))
return;
- /* Find another online reader in this package */
- target = cpumask_any_but(topology_core_cpumask(cpu), cpu);
+ for_each_online_cpu(i) {
+ if (i == cpu)
+ continue;
- if (target < nr_cpu_ids)
- cpumask_set_cpu(target, &cqm_cpumask);
+ if (phys_id == topology_physical_package_id(i)) {
+ cpumask_set_cpu(i, &cqm_cpumask);
+ break;
+ }
+ }
}
static int intel_cqm_cpu_notifier(struct notifier_block *nb,
@@ -1628,70 +1313,12 @@ static const struct x86_cpu_id intel_cqm_match[] = {
{}
};
-static void mbm_cleanup(void)
-{
- if (!mbm_enabled)
- return;
-
- kfree(mbm_local);
- kfree(mbm_total);
- mbm_enabled = false;
-}
-
-static const struct x86_cpu_id intel_mbm_local_match[] = {
- { .vendor = X86_VENDOR_INTEL, .feature = X86_FEATURE_CQM_MBM_LOCAL },
- {}
-};
-
-static const struct x86_cpu_id intel_mbm_total_match[] = {
- { .vendor = X86_VENDOR_INTEL, .feature = X86_FEATURE_CQM_MBM_TOTAL },
- {}
-};
-
-static int intel_mbm_init(void)
-{
- int ret = 0, array_size, maxid = cqm_max_rmid + 1;
-
- mbm_socket_max = topology_max_packages();
- array_size = sizeof(struct sample) * maxid * mbm_socket_max;
- mbm_local = kmalloc(array_size, GFP_KERNEL);
- if (!mbm_local)
- return -ENOMEM;
-
- mbm_total = kmalloc(array_size, GFP_KERNEL);
- if (!mbm_total) {
- ret = -ENOMEM;
- goto out;
- }
-
- array_size = sizeof(struct hrtimer) * mbm_socket_max;
- mbm_timers = kmalloc(array_size, GFP_KERNEL);
- if (!mbm_timers) {
- ret = -ENOMEM;
- goto out;
- }
- mbm_hrtimer_init();
-
-out:
- if (ret)
- mbm_cleanup();
-
- return ret;
-}
-
static int __init intel_cqm_init(void)
{
- char *str = NULL, scale[20];
+ char *str, scale[20];
int i, cpu, ret;
- if (x86_match_cpu(intel_cqm_match))
- cqm_enabled = true;
-
- if (x86_match_cpu(intel_mbm_local_match) &&
- x86_match_cpu(intel_mbm_total_match))
- mbm_enabled = true;
-
- if (!cqm_enabled && !mbm_enabled)
+ if (!x86_match_cpu(intel_cqm_match))
return -ENODEV;
cqm_l3_scale = boot_cpu_data.x86_cache_occ_scale;
@@ -1748,41 +1375,16 @@ static int __init intel_cqm_init(void)
cqm_pick_event_reader(i);
}
- if (mbm_enabled)
- ret = intel_mbm_init();
- if (ret && !cqm_enabled)
- goto out;
-
- if (cqm_enabled && mbm_enabled)
- intel_cqm_events_group.attrs = intel_cmt_mbm_events_attr;
- else if (!cqm_enabled && mbm_enabled)
- intel_cqm_events_group.attrs = intel_mbm_events_attr;
- else if (cqm_enabled && !mbm_enabled)
- intel_cqm_events_group.attrs = intel_cqm_events_attr;
+ __perf_cpu_notifier(intel_cqm_cpu_notifier);
ret = perf_pmu_register(&intel_cqm_pmu, "intel_cqm", -1);
- if (ret) {
+ if (ret)
pr_err("Intel CQM perf registration failed: %d\n", ret);
- goto out;
- }
-
- if (cqm_enabled)
+ else
pr_info("Intel CQM monitoring enabled\n");
- if (mbm_enabled)
- pr_info("Intel MBM enabled\n");
- /*
- * Register the hot cpu notifier once we are sure cqm
- * is enabled to avoid notifier leak.
- */
- __perf_cpu_notifier(intel_cqm_cpu_notifier);
out:
cpu_notifier_register_done();
- if (ret) {
- kfree(str);
- cqm_cleanup();
- mbm_cleanup();
- }
return ret;
}
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 9f64044..00bb6b5 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -121,7 +121,6 @@ struct hw_perf_event {
struct { /* intel_cqm */
int cqm_state;
u32 cqm_rmid;
- int is_group_event;
struct list_head cqm_events_entry;
struct list_head cqm_groups_entry;
struct list_head cqm_group_entry;
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | Vikas Shivappa <vikas.shivappa@linux.intel.com> |
|---|---|
| Date | 2016-04-29 22:30 +0200 |
| Subject | Re: [PATCH 01/32] perf/x86/intel/cqm: temporarily remove MBM from CQM and cleanup |
| Message-ID | <rtncZ-3HL-7@gated-at.bofh.it> |
| In reply to | #1390742 |
On Thu, 28 Apr 2016, David Carrillo-Cisneros wrote: > Removing MBM code from arch/x86/events/intel/cqm.c. MBM will be added > using the new RMID infrastucture introduced in this patch series. I am still working on to rebase mbm on top of the new cqm series (probably with the new quick fixes i sent yesterday).. Thanks, Vikas
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 24/32] perf/x86/intel/cqm: use PERF_INACTIVE_*_READ_* flags in CQM |
| Message-ID | <rt8xk-8qy-33@gated-at.bofh.it> |
| In reply to | #1390741 |
Use newly added pmu_event flags to:
- Allow thread events to be read from any CPU even if not in
ACTIVE state. Since inter-package values are polled, a thread's
occupancy is always:
local occupancy (read from hw) + remote occupancy (polled values)
- Allow cpu/cgroup events to be read from any CPU in the package where
they run. This potentially saves IPIs when the read function runs in the
same package but in a distinct CPU than the event.
Since reading will always return a new value and inherit_stats is not
supported (due to all children events sharing the same RMID), there is no
need to read during sched_out of an event.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/x86/events/intel/cqm.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index c14f1c7..daf9fdf 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -2702,6 +2702,16 @@ static int intel_cqm_event_init(struct perf_event *event)
*/
event->pmu_event_flags |= PERF_CGROUP_NO_RECURSION;
+ /* Events in CQM PMU are per-package and can be read even when
+ * the cpu is not running the event.
+ */
+ if (event->cpu < 0) {
+ WARN_ON_ONCE(!(event->attach_state & PERF_ATTACH_TASK));
+ event->pmu_event_flags |= PERF_INACTIVE_EV_READ_ANY_CPU;
+ } else {
+ event->pmu_event_flags |= PERF_INACTIVE_CPU_READ_PKG;
+ }
+
mutex_lock(&cqm_mutex);
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook |
| Message-ID | <rt8xl-8qy-35@gated-at.bofh.it> |
| In reply to | #1390741 |
This hook allows architecture specific code to be called at the end of
the task switch and after perf_events' context switch but before the
scheduler lock is released.
The specific use case in this series is to avoid multiple writes to a slow
MSR until all functions which modify such register in task switch have
finished.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/x86/include/asm/processor.h | 4 ++++
kernel/sched/core.c | 1 +
kernel/sched/sched.h | 3 +++
3 files changed, 8 insertions(+)
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 9264476..036d94a 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -22,6 +22,7 @@ struct vm86;
#include <asm/nops.h>
#include <asm/special_insns.h>
#include <asm/fpu/types.h>
+#include <asm/pqr_common.h>
#include <linux/personality.h>
#include <linux/cache.h>
@@ -841,4 +842,7 @@ bool xen_set_default_idle(void);
void stop_this_cpu(void *dummy);
void df_debug(struct pt_regs *regs, long error_code);
+
+#define finish_arch_pre_lock_switch pqr_update
+
#endif /* _ASM_X86_PROCESSOR_H */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8b489fc..bcd5473 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2620,6 +2620,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)
prev_state = prev->state;
vtime_task_switch(prev);
perf_event_task_sched_in(prev, current);
+ finish_arch_pre_lock_switch();
finish_lock_switch(rq, prev);
finish_arch_post_lock_switch();
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index ec2e8d2..cb48b5c 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1077,6 +1077,9 @@ static inline int task_on_rq_migrating(struct task_struct *p)
#ifndef prepare_arch_switch
# define prepare_arch_switch(next) do { } while (0)
#endif
+#ifndef finish_arch_pre_lock_switch
+# define finish_arch_pre_lock_switch() do { } while (0)
+#endif
#ifndef finish_arch_post_lock_switch
# define finish_arch_post_lock_switch() do { } while (0)
#endif
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-04-29 11:00 +0200 |
| Subject | Re: [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook |
| Message-ID | <rtcrg-2XJ-1@gated-at.bofh.it> |
| In reply to | #1390744 |
On Thu, Apr 28, 2016 at 09:43:31PM -0700, David Carrillo-Cisneros wrote: > This hook allows architecture specific code to be called at the end of > the task switch and after perf_events' context switch but before the > scheduler lock is released. > > The specific use case in this series is to avoid multiple writes to a slow > MSR until all functions which modify such register in task switch have > finished. Yeah, no. This really need way more justification. Why can't you use the regular perf sched-in stuff for CQM?
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 20:50 +0200 |
| Subject | Re: [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook |
| Message-ID | <rtlEe-2fj-1@gated-at.bofh.it> |
| In reply to | #1390880 |
(Re-sending in plain text) This hook is used in the following patch in the series to write to PQR_ASSOC_MSR, a msr that is utilized both by CQM/CMT and by CAT. Since CAT is not dependent on perf, I created this hook to start CQM monitoring right after other events start while keeping it independent of perf. The idea is to have future versions of CAT to also rely on this hook. On Fri, Apr 29, 2016 at 11:05 AM, David Carrillo-Cisneros <davidcc@google.com> wrote: > This hook is used in the following patch in the series to write to > PQR_ASSOC_MSR, a msr that is utilized both by CQM/CMT and by CAT. Since CAT > is not dependent on perf, I created this hook to start CQM monitoring right > after other events start while keeping it independent of perf. The idea is > to have future versions of CAT to also rely on this hook. > > On Fri, Apr 29, 2016 at 1:52 AM Peter Zijlstra <peterz@infradead.org> wrote: >> >> On Thu, Apr 28, 2016 at 09:43:31PM -0700, David Carrillo-Cisneros wrote: >> > This hook allows architecture specific code to be called at the end of >> > the task switch and after perf_events' context switch but before the >> > scheduler lock is released. >> > >> > The specific use case in this series is to avoid multiple writes to a >> > slow >> > MSR until all functions which modify such register in task switch have >> > finished. >> >> Yeah, no. This really need way more justification. Why can't you use the >> regular perf sched-in stuff for CQM?
[toc] | [prev] | [next] | [standalone]
| From | Vikas Shivappa <vikas.shivappa@linux.intel.com> |
|---|---|
| Date | 2016-04-29 22:30 +0200 |
| Subject | Re: [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook |
| Message-ID | <rtnd0-3HL-21@gated-at.bofh.it> |
| In reply to | #1391326 |
On Fri, 29 Apr 2016, David Carrillo-Cisneros wrote: > (Re-sending in plain text) > > This hook is used in the following patch in the series to write to > PQR_ASSOC_MSR, a msr that is utilized both by CQM/CMT and by CAT. > Since CAT is not dependent on perf, I created this hook to start CQM > monitoring right after other events start while keeping it independent > of perf. The idea is to have future versions of CAT to also rely on > this hook. CAT did the msr write in switch_to as Peter did not want a new hook to be used. Same could be done here. Thanks, Vikas > > On Fri, Apr 29, 2016 at 11:05 AM, David Carrillo-Cisneros > <davidcc@google.com> wrote: >> This hook is used in the following patch in the series to write to >> PQR_ASSOC_MSR, a msr that is utilized both by CQM/CMT and by CAT. Since CAT >> is not dependent on perf, I created this hook to start CQM monitoring right >> after other events start while keeping it independent of perf. The idea is >> to have future versions of CAT to also rely on this hook. >> >> On Fri, Apr 29, 2016 at 1:52 AM Peter Zijlstra <peterz@infradead.org> wrote: >>> >>> On Thu, Apr 28, 2016 at 09:43:31PM -0700, David Carrillo-Cisneros wrote: >>>> This hook allows architecture specific code to be called at the end of >>>> the task switch and after perf_events' context switch but before the >>>> scheduler lock is released. >>>> >>>> The specific use case in this series is to avoid multiple writes to a >>>> slow >>>> MSR until all functions which modify such register in task switch have >>>> finished. >>> >>> Yeah, no. This really need way more justification. Why can't you use the >>> regular perf sched-in stuff for CQM? >
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 23:00 +0200 |
| Subject | Re: [PATCH 25/32] sched: introduce the finish_arch_pre_lock_switch() scheduler hook |
| Message-ID | <rtnG2-3Ts-13@gated-at.bofh.it> |
| In reply to | #1391398 |
That's a possibility, although it will increase the distance between pmu->add for other perf events and the effective time that CQM monitoring starts. On Fri, Apr 29, 2016 at 1:21 PM, Vikas Shivappa <vikas.shivappa@linux.intel.com> wrote: > > > On Fri, 29 Apr 2016, David Carrillo-Cisneros wrote: > >> (Re-sending in plain text) >> >> This hook is used in the following patch in the series to write to >> PQR_ASSOC_MSR, a msr that is utilized both by CQM/CMT and by CAT. >> Since CAT is not dependent on perf, I created this hook to start CQM >> monitoring right after other events start while keeping it independent >> of perf. The idea is to have future versions of CAT to also rely on >> this hook. > > > CAT did the msr write in switch_to as Peter did not want a new hook to be > used. Same could be done here. > > Thanks, > Vikas > > >> >> On Fri, Apr 29, 2016 at 11:05 AM, David Carrillo-Cisneros >> <davidcc@google.com> wrote: >>> >>> This hook is used in the following patch in the series to write to >>> PQR_ASSOC_MSR, a msr that is utilized both by CQM/CMT and by CAT. Since >>> CAT >>> is not dependent on perf, I created this hook to start CQM monitoring >>> right >>> after other events start while keeping it independent of perf. The idea >>> is >>> to have future versions of CAT to also rely on this hook. >>> >>> On Fri, Apr 29, 2016 at 1:52 AM Peter Zijlstra <peterz@infradead.org> >>> wrote: >>>> >>>> >>>> On Thu, Apr 28, 2016 at 09:43:31PM -0700, David Carrillo-Cisneros wrote: >>>>> >>>>> This hook allows architecture specific code to be called at the end of >>>>> the task switch and after perf_events' context switch but before the >>>>> scheduler lock is released. >>>>> >>>>> The specific use case in this series is to avoid multiple writes to a >>>>> slow >>>>> MSR until all functions which modify such register in task switch have >>>>> finished. >>>> >>>> >>>> Yeah, no. This really need way more justification. Why can't you use the >>>> regular perf sched-in stuff for CQM? >> >> >
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 14/32] perf/x86/intel/cqm: add preallocation of anodes |
| Message-ID | <rt8xl-8qy-37@gated-at.bofh.it> |
| In reply to | #1390741 |
Pre-allocate enough anodes to be able to at least hold one set of RMIDs
per package before running out of anodes.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/x86/events/intel/cqm.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index 904f2d3..98a919f 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -194,6 +194,7 @@ static int anode_pool__alloc_one(u16 pkg_id);
/* Init cqm pkg_data for @cpu 's package. */
static int pkg_data_init_cpu(int cpu)
{
+ int i, nr_anodes;
struct pkg_data *pkg_data;
struct cpuinfo_x86 *c = &cpu_data(cpu);
u16 pkg_id = topology_physical_package_id(cpu);
@@ -257,6 +258,15 @@ static int pkg_data_init_cpu(int cpu)
pkg_data->timed_update_cpu = cpu;
cqm_pkgs_data[pkg_id] = pkg_data;
+
+ /* Pre-allocate pool with one anode more than minimum needed to contain
+ * all the RMIDs in the package.
+ */
+ nr_anodes = (pkg_data->max_rmid + NR_RMIDS_PER_NODE) /
+ NR_RMIDS_PER_NODE + 1;
+
+ for (i = 0; i < nr_anodes; i++)
+ anode_pool__alloc_one(pkg_id);
return 0;
}
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 32/32] perf/stat: revamp error handling for snapshot and per_pkg events |
| Message-ID | <rt8xl-8qy-39@gated-at.bofh.it> |
| In reply to | #1390741 |
A package wide event can return a valid read even if it has not run in a
specific cpu, this does not fit well with the assumption that run == 0
is equivalent to a <not counted>.
To fix the problem, this patch defines special error values for val,
run and ena (~0ULL), and use them to signal read errors, allowing run == 0
to be a valid value for package events. A new value, NA, is output on
read error and when event has not been enabled (timed enabled == 0).
Finally, this patch revamps calculation of deltas and scaling for snapshot
events, removing the calculation of deltas for time running and enabled in
snapshot event, as should be.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
tools/perf/builtin-stat.c | 37 ++++++++++++++++++++++++++-----------
tools/perf/util/counts.h | 19 +++++++++++++++++++
tools/perf/util/evsel.c | 44 +++++++++++++++++++++++++++++++++-----------
tools/perf/util/evsel.h | 8 ++++++--
tools/perf/util/stat.c | 35 +++++++++++------------------------
5 files changed, 95 insertions(+), 48 deletions(-)
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index a4e5610..f1c2166 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -63,6 +63,7 @@
#include "util/tool.h"
#include "asm/bug.h"
+#include <math.h>
#include <stdlib.h>
#include <sys/prctl.h>
#include <locale.h>
@@ -290,10 +291,8 @@ static int read_counter(struct perf_evsel *counter)
count = perf_counts(counter->counts, cpu, thread);
if (perf_evsel__read(counter, cpu, thread, count)) {
- counter->counts->scaled = -1;
- perf_counts(counter->counts, cpu, thread)->ena = 0;
- perf_counts(counter->counts, cpu, thread)->run = 0;
- return -1;
+ /* do not write stat for failed reads. */
+ continue;
}
if (STAT_RECORD) {
@@ -668,12 +667,16 @@ static int run_perf_stat(int argc, const char **argv)
static void print_running(u64 run, u64 ena)
{
+ bool is_na = run == PERF_COUNTS_NA || ena == PERF_COUNTS_NA || !ena;
+
if (csv_output) {
- fprintf(stat_config.output, "%s%" PRIu64 "%s%.2f",
- csv_sep,
- run,
- csv_sep,
- ena ? 100.0 * run / ena : 100.0);
+ if (is_na)
+ fprintf(stat_config.output, "%sNA%sNA", csv_sep, csv_sep);
+ else
+ fprintf(stat_config.output, "%s%" PRIu64 "%s%.2f",
+ csv_sep, run, csv_sep, 100.0 * run / ena);
+ } else if (is_na) {
+ fprintf(stat_config.output, " (NA)");
} else if (run != ena) {
fprintf(stat_config.output, " (%.2f%%)", 100.0 * run / ena);
}
@@ -1046,7 +1049,7 @@ static void printout(int id, int nr, struct perf_evsel *counter, double uval,
if (counter->cgrp)
os.nfields++;
}
- if (run == 0 || ena == 0 || counter->counts->scaled == -1) {
+ if (run == PERF_COUNTS_NA || ena == PERF_COUNTS_NA || counter->counts->scaled == -1) {
if (metric_only) {
pm(&os, NULL, "", "", 0);
return;
@@ -1152,12 +1155,17 @@ static void print_aggr(char *prefix)
id = aggr_map->map[s];
first = true;
evlist__for_each(evsel_list, counter) {
+ bool all_nan = true;
val = ena = run = 0;
nr = 0;
for (cpu = 0; cpu < perf_evsel__nr_cpus(counter); cpu++) {
s2 = aggr_get_id(perf_evsel__cpus(counter), cpu);
if (s2 != id)
continue;
+ /* skip NA reads. */
+ if (perf_counts_values__is_na(perf_counts(counter->counts, cpu, 0)))
+ continue;
+ all_nan = false;
val += perf_counts(counter->counts, cpu, 0)->val;
ena += perf_counts(counter->counts, cpu, 0)->ena;
run += perf_counts(counter->counts, cpu, 0)->run;
@@ -1171,6 +1179,10 @@ static void print_aggr(char *prefix)
fprintf(output, "%s", prefix);
uval = val * counter->scale;
+ if (all_nan) {
+ run = PERF_COUNTS_NA;
+ ena = PERF_COUNTS_NA;
+ }
printout(id, nr, counter, uval, prefix, run, ena, 1.0);
if (!metric_only)
fputc('\n', output);
@@ -1249,7 +1261,10 @@ static void print_counter(struct perf_evsel *counter, char *prefix)
if (prefix)
fprintf(output, "%s", prefix);
- uval = val * counter->scale;
+ if (val != PERF_COUNTS_NA)
+ uval = val * counter->scale;
+ else
+ uval = NAN;
printout(cpu, 0, counter, uval, prefix, run, ena, 1.0);
fputc('\n', output);
diff --git a/tools/perf/util/counts.h b/tools/perf/util/counts.h
index 34d8baa..b65e97a 100644
--- a/tools/perf/util/counts.h
+++ b/tools/perf/util/counts.h
@@ -3,6 +3,9 @@
#include "xyarray.h"
+/* Not Available (NA) value. Any operation with a NA equals a NA. */
+#define PERF_COUNTS_NA ((u64)~0ULL)
+
struct perf_counts_values {
union {
struct {
@@ -14,6 +17,22 @@ struct perf_counts_values {
};
};
+static inline void
+perf_counts_values__make_na(struct perf_counts_values *values)
+{
+ values->val = PERF_COUNTS_NA;
+ values->ena = PERF_COUNTS_NA;
+ values->run = PERF_COUNTS_NA;
+}
+
+static inline bool
+perf_counts_values__is_na(struct perf_counts_values *values)
+{
+ return values->val == PERF_COUNTS_NA ||
+ values->ena == PERF_COUNTS_NA ||
+ values->run == PERF_COUNTS_NA;
+}
+
struct perf_counts {
s8 scaled;
struct perf_counts_values aggr;
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 52a0c35..da63a87 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -1109,6 +1109,9 @@ void perf_evsel__compute_deltas(struct perf_evsel *evsel, int cpu, int thread,
if (!evsel->prev_raw_counts)
return;
+ if (perf_counts_values__is_na(count))
+ return;
+
if (cpu == -1) {
tmp = evsel->prev_raw_counts->aggr;
evsel->prev_raw_counts->aggr = *count;
@@ -1117,26 +1120,38 @@ void perf_evsel__compute_deltas(struct perf_evsel *evsel, int cpu, int thread,
*perf_counts(evsel->prev_raw_counts, cpu, thread) = *count;
}
- count->val = count->val - tmp.val;
+ /* Snapshot events do not calculate deltas for count values. */
+ if (!evsel->snapshot)
+ count->val = count->val - tmp.val;
count->ena = count->ena - tmp.ena;
count->run = count->run - tmp.run;
}
void perf_counts_values__scale(struct perf_counts_values *count,
- bool scale, s8 *pscaled)
+ bool scale, bool per_pkg, bool snapshot, s8 *pscaled)
{
s8 scaled = 0;
+ if (perf_counts_values__is_na(count)) {
+ if (pscaled)
+ *pscaled = -1;
+ return;
+ }
+
if (scale) {
- if (count->run == 0) {
+ /* per-pkg events can have run == 0 and be valid. */
+ if (count->run == 0 && !per_pkg) {
scaled = -1;
count->val = 0;
} else if (count->run < count->ena) {
scaled = 1;
- count->val = (u64)((double) count->val * count->ena / count->run + 0.5);
+ /* Snapshot events do not scale counts values. */
+ if (!snapshot && count->run)
+ count->val = (u64)((double) count->val * count->ena /
+ count->run + 0.5);
}
- } else
- count->ena = count->run = 0;
+ }
+ count->run = count->ena;
if (pscaled)
*pscaled = scaled;
@@ -1150,8 +1165,10 @@ int perf_evsel__read(struct perf_evsel *evsel, int cpu, int thread,
if (FD(evsel, cpu, thread) < 0)
return -EINVAL;
- if (readn(FD(evsel, cpu, thread), count, sizeof(*count)) <= 0)
+ if (readn(FD(evsel, cpu, thread), count, sizeof(*count)) <= 0) {
+ perf_counts_values__make_na(count);
return -errno;
+ }
return 0;
}
@@ -1159,6 +1176,7 @@ int perf_evsel__read(struct perf_evsel *evsel, int cpu, int thread,
int __perf_evsel__read_on_cpu(struct perf_evsel *evsel,
int cpu, int thread, bool scale)
{
+ int ret = 0;
struct perf_counts_values count;
size_t nv = scale ? 3 : 1;
@@ -1168,13 +1186,17 @@ int __perf_evsel__read_on_cpu(struct perf_evsel *evsel,
if (evsel->counts == NULL && perf_evsel__alloc_counts(evsel, cpu + 1, thread + 1) < 0)
return -ENOMEM;
- if (readn(FD(evsel, cpu, thread), &count, nv * sizeof(u64)) <= 0)
- return -errno;
+ if (readn(FD(evsel, cpu, thread), &count, nv * sizeof(u64)) <= 0) {
+ perf_counts_values__make_na(&count);
+ ret = -errno;
+ goto exit;
+ }
perf_evsel__compute_deltas(evsel, cpu, thread, &count);
- perf_counts_values__scale(&count, scale, NULL);
+ perf_counts_values__scale(&count, scale, evsel->per_pkg, evsel->snapshot, NULL);
+exit:
*perf_counts(evsel->counts, cpu, thread) = count;
- return 0;
+ return ret;
}
static int get_group_fd(struct perf_evsel *evsel, int cpu, int thread)
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index b993218..e6a5854 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -74,6 +74,10 @@ struct perf_evsel_config_term {
* @is_pos: the position (counting backwards) of the event id (PERF_SAMPLE_ID or
* PERF_SAMPLE_IDENTIFIER) in a non-sample event i.e. if sample_id_all
* is used there is an id sample appended to non-sample events
+ * @snapshot: an event that whose raw value cannot be extrapolated based on
+ * the ratio of running/enabled time.
+ * @per_pkg: an event that runs package wide. All cores in same package will
+ * read the same value, even if running time == 0.
* @priv: And what is in its containing unnamed union are tool specific
*/
struct perf_evsel {
@@ -144,8 +148,8 @@ static inline int perf_evsel__nr_cpus(struct perf_evsel *evsel)
return perf_evsel__cpus(evsel)->nr;
}
-void perf_counts_values__scale(struct perf_counts_values *count,
- bool scale, s8 *pscaled);
+void perf_counts_values__scale(struct perf_counts_values *count, bool scale,
+ bool per_pkg, bool snapshot, s8 *pscaled);
void perf_evsel__compute_deltas(struct perf_evsel *evsel, int cpu, int thread,
struct perf_counts_values *count);
diff --git a/tools/perf/util/stat.c b/tools/perf/util/stat.c
index 4d9b481..b0f0d41 100644
--- a/tools/perf/util/stat.c
+++ b/tools/perf/util/stat.c
@@ -197,7 +197,7 @@ static void zero_per_pkg(struct perf_evsel *counter)
}
static int check_per_pkg(struct perf_evsel *counter,
- struct perf_counts_values *vals, int cpu, bool *skip)
+ int cpu, bool *skip)
{
unsigned long *mask = counter->per_pkg_mask;
struct cpu_map *cpus = perf_evsel__cpus(counter);
@@ -219,17 +219,6 @@ static int check_per_pkg(struct perf_evsel *counter,
counter->per_pkg_mask = mask;
}
- /*
- * we do not consider an event that has not run as a good
- * instance to mark a package as used (skip=1). Otherwise
- * we may run into a situation where the first CPU in a package
- * is not running anything, yet the second is, and this function
- * would mark the package as used after the first CPU and would
- * not read the values from the second CPU.
- */
- if (!(vals->run && vals->ena))
- return 0;
-
s = cpu_map__get_socket(cpus, cpu, NULL);
if (s < 0)
return -1;
@@ -244,30 +233,27 @@ process_counter_values(struct perf_stat_config *config, struct perf_evsel *evsel
struct perf_counts_values *count)
{
struct perf_counts_values *aggr = &evsel->counts->aggr;
- static struct perf_counts_values zero;
bool skip = false;
- if (check_per_pkg(evsel, count, cpu, &skip)) {
+ if (check_per_pkg(evsel, cpu, &skip)) {
pr_err("failed to read per-pkg counter\n");
return -1;
}
- if (skip)
- count = &zero;
-
switch (config->aggr_mode) {
case AGGR_THREAD:
case AGGR_CORE:
case AGGR_SOCKET:
case AGGR_NONE:
- if (!evsel->snapshot)
- perf_evsel__compute_deltas(evsel, cpu, thread, count);
- perf_counts_values__scale(count, config->scale, NULL);
+ perf_evsel__compute_deltas(evsel, cpu, thread, count);
+ perf_counts_values__scale(count, config->scale,
+ evsel->per_pkg, evsel->snapshot, NULL);
if (config->aggr_mode == AGGR_NONE)
perf_stat__update_shadow_stats(evsel, count->values, cpu);
break;
case AGGR_GLOBAL:
- aggr->val += count->val;
+ if (!skip)
+ aggr->val += count->val;
if (config->scale) {
aggr->ena += count->ena;
aggr->run += count->run;
@@ -331,9 +317,10 @@ int perf_stat_process_counter(struct perf_stat_config *config,
if (config->aggr_mode != AGGR_GLOBAL)
return 0;
- if (!counter->snapshot)
- perf_evsel__compute_deltas(counter, -1, -1, aggr);
- perf_counts_values__scale(aggr, config->scale, &counter->counts->scaled);
+ perf_evsel__compute_deltas(counter, -1, -1, aggr);
+ perf_counts_values__scale(aggr, config->scale,
+ counter->per_pkg, counter->snapshot,
+ &counter->counts->scaled);
for (i = 0; i < 3; i++)
update_stats(&ps->res_stats[i], count[i]);
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 09/32] perf/x86/intel/cqm: add per-package RMIDs, data and locks |
| Message-ID | <rt8xk-8qy-31@gated-at.bofh.it> |
| In reply to | #1390741 |
First part of new CQM logic. This patch introduces the struct pkg_data
that contains all per-package CQM data required by the new RMID hierarchy.
The raw RMID value is encapsulated in a Package RMID (prmid) structure
that provides atomic updates and caches recent reads. This caching
throttles the frequency at which (slow) hardware reads are performed and
ameliorates the impact of the worst case scenarios while traversing the
hierarchy of RMIDs (hierarchy and operations are introduced in future
patches within this series).
There is a set of prmids per physical package (socket) in the system. Each
package may have different number of prmids (different hw max_rmid_index).
Each package maintains its own pool of prmids (only a free pool as of this
patch, more pools to add in future patches in this series). Also, each
package has its own mutex and lock to protect the RMID pools and rotation
logic. This per-package separation reduces the contention for each lock
and mutex compared with the previous version (with system-wide mutex
and lock).
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/x86/events/intel/cqm.c | 426 +++++++++++++++++++++-----------------
arch/x86/events/intel/cqm.h | 154 ++++++++++++++
arch/x86/include/asm/pqr_common.h | 2 +
3 files changed, 392 insertions(+), 190 deletions(-)
diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index f678014..541e515 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -12,7 +12,6 @@
#define MSR_IA32_QM_CTR 0x0c8e
#define MSR_IA32_QM_EVTSEL 0x0c8d
-static u32 cqm_max_rmid = -1;
static unsigned int cqm_l3_scale; /* supposedly cacheline size */
#define RMID_VAL_ERROR (1ULL << 63)
@@ -30,39 +29,13 @@ static struct perf_pmu_events_attr event_attr_##v = { \
}
/*
- * Updates caller cpu's cache.
- */
-static inline void __update_pqr_rmid(u32 rmid)
-{
- struct intel_pqr_state *state = this_cpu_ptr(&pqr_state);
- if (state->rmid == rmid)
- return;
- state->rmid = rmid;
- wrmsr(MSR_IA32_PQR_ASSOC, rmid, state->closid);
-}
-
-/*
* Groups of events that have the same target(s), one RMID per group.
* Protected by cqm_mutex.
*/
static LIST_HEAD(cache_groups);
static DEFINE_MUTEX(cqm_mutex);
-static DEFINE_RAW_SPINLOCK(cache_lock);
-/*
- * Mask of CPUs for reading CQM values. We only need one per-socket.
- */
-static cpumask_t cqm_cpumask;
-
-
-/*
- * This is central to the rotation algorithm in __intel_cqm_rmid_rotate().
- *
- * This rmid is always free and is guaranteed to have an associated
- * near-zero occupancy value, i.e. no cachelines are tagged with this
- * RMID, once __intel_cqm_rmid_rotate() returns.
- */
-static u32 intel_cqm_rotation_rmid;
+struct pkg_data *cqm_pkgs_data[PQR_MAX_NR_PKGS];
/*
* Is @rmid valid for programming the hardware?
@@ -82,162 +55,220 @@ static inline bool __rmid_valid(u32 rmid)
static u64 __rmid_read(u32 rmid)
{
+ /* XXX: Placeholder, will be removed in next patch. */
+ return 0;
+}
+
+/*
+ * Update if enough time has passed since last read.
+ *
+ * Must be called in a cpu in the package where prmid belongs.
+ * This function is safe to be called concurrently since it is guaranteed
+ * that entry->last_read_value is updated to a occupancy value obtained
+ * after the time set in entry->last_read_time .
+ * Return 1 if value was updated, 0 if not, negative number if error.
+ */
+static inline int __cqm_prmid_update(struct prmid *prmid,
+ unsigned long jiffies_min_delta)
+{
+ unsigned long now = jiffies;
+ unsigned long last_read_time;
u64 val;
/*
- * Ignore the SDM, this thing is _NOTHING_ like a regular perfcnt,
- * it just says that to increase confusion.
+ * Shortcut the calculation of elapsed time for the
+ * case jiffies_min_delta == 0
*/
- wrmsr(MSR_IA32_QM_EVTSEL, QOS_L3_OCCUP_EVENT_ID, rmid);
+ if (jiffies_min_delta > 0) {
+ last_read_time = atomic64_read(&prmid->last_read_time);
+ if (time_after(last_read_time + jiffies_min_delta, now))
+ return 0;
+ }
+
+ wrmsr(MSR_IA32_QM_EVTSEL, QOS_L3_OCCUP_EVENT_ID, prmid->rmid);
rdmsrl(MSR_IA32_QM_CTR, val);
/*
- * Aside from the ERROR and UNAVAIL bits, assume this thing returns
- * the number of cachelines tagged with @rmid.
+ * Ignore this reading on error states and do not update the value.
*/
- return val;
-}
+ WARN_ON_ONCE(val & (RMID_VAL_ERROR | RMID_VAL_UNAVAIL));
+ if (val & RMID_VAL_ERROR)
+ return -EINVAL;
+ if (val & RMID_VAL_UNAVAIL)
+ return -ENODATA;
-enum rmid_recycle_state {
- RMID_YOUNG = 0,
- RMID_AVAILABLE,
- RMID_DIRTY,
-};
+ atomic64_set(&prmid->last_read_value, val);
+ /*
+ * Protect last_read_time from being updated before last_read_value is.
+ * So reader always receive an updated value even if sometimes values
+ * are updated twice.
+ */
+ smp_wmb();
-struct cqm_rmid_entry {
- u32 rmid;
- enum rmid_recycle_state state;
- struct list_head list;
- unsigned long queue_time;
-};
+ atomic64_set(&prmid->last_read_time, now);
-/*
- * cqm_rmid_free_lru - A least recently used list of RMIDs.
- *
- * Oldest entry at the head, newest (most recently used) entry at the
- * tail. This list is never traversed, it's only used to keep track of
- * the lru order. That is, we only pick entries of the head or insert
- * them on the tail.
- *
- * All entries on the list are 'free', and their RMIDs are not currently
- * in use. To mark an RMID as in use, remove its entry from the lru
- * list.
- *
- *
- * cqm_rmid_limbo_lru - list of currently unused but (potentially) dirty RMIDs.
- *
- * This list is contains RMIDs that no one is currently using but that
- * may have a non-zero occupancy value associated with them. The
- * rotation worker moves RMIDs from the limbo list to the free list once
- * the occupancy value drops below __intel_cqm_threshold.
- *
- * Both lists are protected by cqm_mutex.
- */
-static LIST_HEAD(cqm_rmid_free_lru);
-static LIST_HEAD(cqm_rmid_limbo_lru);
+ return 1;
+}
+
+static inline int cqm_prmid_update(struct prmid *prmid)
+{
+ return __cqm_prmid_update(prmid, __rmid_min_update_time);
+}
/*
- * We use a simple array of pointers so that we can lookup a struct
- * cqm_rmid_entry in O(1). This alleviates the callers of __get_rmid()
- * and __put_rmid() from having to worry about dealing with struct
- * cqm_rmid_entry - they just deal with rmids, i.e. integers.
- *
- * Once this array is initialized it is read-only. No locks are required
- * to access it.
- *
- * All entries for all RMIDs can be looked up in the this array at all
- * times.
+ * Updates caller cpu's cache.
*/
-static struct cqm_rmid_entry **cqm_rmid_ptrs;
-
-static inline struct cqm_rmid_entry *__rmid_entry(u32 rmid)
+static inline void __update_pqr_prmid(struct prmid *prmid)
{
- struct cqm_rmid_entry *entry;
+ struct intel_pqr_state *state = this_cpu_ptr(&pqr_state);
- entry = cqm_rmid_ptrs[rmid];
- WARN_ON(entry->rmid != rmid);
+ if (state->rmid == prmid->rmid)
+ return;
+ state->rmid = prmid->rmid;
+ wrmsr(MSR_IA32_PQR_ASSOC, prmid->rmid, state->closid);
+}
- return entry;
+static inline bool __valid_pkg_id(u16 pkg_id)
+{
+ return pkg_id < PQR_MAX_NR_PKGS;
}
/*
* Returns < 0 on fail.
*
- * We expect to be called with cqm_mutex held.
+ * We expect to be called with cache_mutex held.
*/
static u32 __get_rmid(void)
{
- struct cqm_rmid_entry *entry;
+ /* XXX: Placeholder, will be removed in next patch. */
+ return 0;
+}
+
+static void __put_rmid(u32 rmid)
+{
+ /* XXX: Placeholder, will be removed in next patch. */
+}
+
+/* Init cqm pkg_data for @cpu 's package. */
+static int pkg_data_init_cpu(int cpu)
+{
+ struct pkg_data *pkg_data;
+ struct cpuinfo_x86 *c = &cpu_data(cpu);
+ u16 pkg_id = topology_physical_package_id(cpu);
+
+ if (cqm_pkgs_data[pkg_id])
+ return 0;
- lockdep_assert_held(&cqm_mutex);
- if (list_empty(&cqm_rmid_free_lru))
- return INVALID_RMID;
+ pkg_data = kmalloc_node(sizeof(struct pkg_data),
+ GFP_KERNEL, cpu_to_node(cpu));
+ if (!pkg_data)
+ return -ENOMEM;
+
+ pkg_data->max_rmid = c->x86_cache_max_rmid;
- entry = list_first_entry(&cqm_rmid_free_lru, struct cqm_rmid_entry, list);
- list_del(&entry->list);
+ /* Does hardware has more rmids than this driver can handle? */
+ if (WARN_ON(pkg_data->max_rmid >= INVALID_RMID))
+ pkg_data->max_rmid = INVALID_RMID - 1;
- return entry->rmid;
+ if (c->x86_cache_occ_scale != cqm_l3_scale) {
+ pr_err("Multiple LLC scale values, disabling\n");
+ kfree(pkg_data);
+ return -EINVAL;
+ }
+
+ pkg_data->prmids_by_rmid = kmalloc_node(
+ sizeof(struct prmid *) * (1 + pkg_data->max_rmid),
+ GFP_KERNEL, cpu_to_node(cpu));
+
+ if (!pkg_data) {
+ kfree(pkg_data);
+ return -ENOMEM;
+ }
+
+ INIT_LIST_HEAD(&pkg_data->free_prmids_pool);
+
+ mutex_init(&pkg_data->pkg_data_mutex);
+ raw_spin_lock_init(&pkg_data->pkg_data_lock);
+
+ /* XXX: Chose randomly*/
+ pkg_data->rotation_cpu = cpu;
+
+ cqm_pkgs_data[pkg_id] = pkg_data;
+ return 0;
}
-static void __put_rmid(u32 rmid)
+static inline bool __valid_rmid(u16 pkg_id, u32 rmid)
{
- struct cqm_rmid_entry *entry;
+ return rmid <= cqm_pkgs_data[pkg_id]->max_rmid;
+}
- lockdep_assert_held(&cqm_mutex);
+static inline bool __valid_prmid(u16 pkg_id, struct prmid *prmid)
+{
+ struct pkg_data *pkg_data = cqm_pkgs_data[pkg_id];
+ bool valid = __valid_rmid(pkg_id, prmid->rmid);
- WARN_ON(!__rmid_valid(rmid));
- entry = __rmid_entry(rmid);
+ WARN_ON_ONCE(valid && pkg_data->prmids_by_rmid[
+ prmid->rmid]->rmid != prmid->rmid);
+ return valid;
+}
- entry->queue_time = jiffies;
- entry->state = RMID_YOUNG;
+static inline struct prmid *
+__prmid_from_rmid(u16 pkg_id, u32 rmid)
+{
+ struct prmid *prmid;
- list_add_tail(&entry->list, &cqm_rmid_limbo_lru);
+ if (!__valid_rmid(pkg_id, rmid))
+ return NULL;
+ prmid = cqm_pkgs_data[pkg_id]->prmids_by_rmid[rmid];
+ WARN_ON_ONCE(!__valid_prmid(pkg_id, prmid));
+ return prmid;
}
-static int intel_cqm_setup_rmid_cache(void)
+static int intel_cqm_setup_pkg_prmid_pools(u16 pkg_id)
{
- struct cqm_rmid_entry *entry;
- unsigned int nr_rmids;
- int r = 0;
-
- nr_rmids = cqm_max_rmid + 1;
- cqm_rmid_ptrs = kmalloc(sizeof(struct cqm_rmid_entry *) *
- nr_rmids, GFP_KERNEL);
- if (!cqm_rmid_ptrs)
- return -ENOMEM;
+ int r;
+ unsigned long flags;
+ struct prmid *prmid;
+ struct pkg_data *pkg_data = cqm_pkgs_data[pkg_id];
+
+ if (!__valid_pkg_id(pkg_id))
+ return -EINVAL;
- for (; r <= cqm_max_rmid; r++) {
- struct cqm_rmid_entry *entry;
+ for (r = 0; r <= pkg_data->max_rmid; r++) {
- entry = kmalloc(sizeof(*entry), GFP_KERNEL);
- if (!entry)
+ prmid = kmalloc_node(sizeof(struct prmid), GFP_KERNEL,
+ cpu_to_node(pkg_data->rotation_cpu));
+ if (!prmid)
goto fail;
- INIT_LIST_HEAD(&entry->list);
- entry->rmid = r;
- cqm_rmid_ptrs[r] = entry;
+ atomic64_set(&prmid->last_read_value, 0L);
+ atomic64_set(&prmid->last_read_time, 0L);
+ INIT_LIST_HEAD(&prmid->pool_entry);
+ prmid->rmid = r;
- list_add_tail(&entry->list, &cqm_rmid_free_lru);
- }
+ /* Lock needed if called during CPU hotplug. */
+ raw_spin_lock_irqsave_nested(
+ &pkg_data->pkg_data_lock, flags, pkg_id);
+ pkg_data->prmids_by_rmid[r] = prmid;
- /*
- * RMID 0 is special and is always allocated. It's used for all
- * tasks that are not monitored.
- */
- entry = __rmid_entry(0);
- list_del(&entry->list);
- mutex_lock(&cqm_mutex);
- intel_cqm_rotation_rmid = __get_rmid();
- mutex_unlock(&cqm_mutex);
+ /* RMID 0 is special and makes the root of rmid hierarchy. */
+ if (r != 0)
+ list_add_tail(&prmid->pool_entry,
+ &pkg_data->free_prmids_pool);
+ raw_spin_unlock_irqrestore(&pkg_data->pkg_data_lock, flags);
+ }
return 0;
fail:
- while (r--)
- kfree(cqm_rmid_ptrs[r]);
-
- kfree(cqm_rmid_ptrs);
+ while (!list_empty(&pkg_data->free_prmids_pool)) {
+ prmid = list_first_entry(&pkg_data->free_prmids_pool,
+ struct prmid, pool_entry);
+ list_del(&prmid->pool_entry);
+ kfree(pkg_data->prmids_by_rmid[prmid->rmid]);
+ kfree(prmid);
+ }
return -ENOMEM;
}
@@ -322,8 +353,9 @@ static void intel_cqm_event_read(struct perf_event *event)
unsigned long flags;
u32 rmid;
u64 val;
+ u16 pkg_id = topology_physical_package_id(smp_processor_id());
- raw_spin_lock_irqsave(&cache_lock, flags);
+ raw_spin_lock_irqsave(&cqm_pkgs_data[pkg_id]->pkg_data_lock, flags);
rmid = event->hw.cqm_rmid;
if (!__rmid_valid(rmid))
@@ -339,7 +371,8 @@ static void intel_cqm_event_read(struct perf_event *event)
local64_set(&event->count, val);
out:
- raw_spin_unlock_irqrestore(&cache_lock, flags);
+ raw_spin_unlock_irqrestore(
+ &cqm_pkgs_data[pkg_id]->pkg_data_lock, flags);
}
static inline bool cqm_group_leader(struct perf_event *event)
@@ -349,29 +382,32 @@ static inline bool cqm_group_leader(struct perf_event *event)
static void intel_cqm_event_start(struct perf_event *event, int mode)
{
+ u16 pkg_id = topology_physical_package_id(smp_processor_id());
if (!(event->hw.state & PERF_HES_STOPPED))
return;
event->hw.state &= ~PERF_HES_STOPPED;
- __update_pqr_rmid(event->hw.cqm_rmid);
+ __update_pqr_prmid(__prmid_from_rmid(pkg_id, event->hw.cqm_rmid));
}
static void intel_cqm_event_stop(struct perf_event *event, int mode)
{
+ u16 pkg_id = topology_physical_package_id(smp_processor_id());
if (event->hw.state & PERF_HES_STOPPED)
return;
event->hw.state |= PERF_HES_STOPPED;
intel_cqm_event_read(event);
- __update_pqr_rmid(0);
+ __update_pqr_prmid(__prmid_from_rmid(pkg_id, 0));
}
static int intel_cqm_event_add(struct perf_event *event, int mode)
{
unsigned long flags;
u32 rmid;
+ u16 pkg_id = topology_physical_package_id(smp_processor_id());
- raw_spin_lock_irqsave(&cache_lock, flags);
+ raw_spin_lock_irqsave(&cqm_pkgs_data[pkg_id]->pkg_data_lock, flags);
event->hw.state = PERF_HES_STOPPED;
rmid = event->hw.cqm_rmid;
@@ -379,7 +415,8 @@ static int intel_cqm_event_add(struct perf_event *event, int mode)
if (__rmid_valid(rmid) && (mode & PERF_EF_START))
intel_cqm_event_start(event, mode);
- raw_spin_unlock_irqrestore(&cache_lock, flags);
+ raw_spin_unlock_irqrestore(
+ &cqm_pkgs_data[pkg_id]->pkg_data_lock, flags);
return 0;
}
@@ -503,9 +540,10 @@ max_recycle_threshold_show(
{
ssize_t rv;
- mutex_lock(&cqm_mutex);
- rv = snprintf(page, PAGE_SIZE-1, "%u\n", __intel_cqm_max_threshold);
- mutex_unlock(&cqm_mutex);
+ monr_hrchy_acquire_mutexes();
+ rv = snprintf(page, PAGE_SIZE - 1, "%u\n",
+ __intel_cqm_max_threshold);
+ monr_hrchy_release_mutexes();
return rv;
}
@@ -522,9 +560,12 @@ max_recycle_threshold_store(struct device *dev,
if (ret)
return ret;
- mutex_lock(&cqm_mutex);
+ /* Mutex waits for rotation logic in all packages to complete. */
+ monr_hrchy_acquire_mutexes();
+
__intel_cqm_max_threshold = bytes;
- mutex_unlock(&cqm_mutex);
+
+ monr_hrchy_release_mutexes();
return count;
}
@@ -561,49 +602,42 @@ static struct pmu intel_cqm_pmu = {
static inline void cqm_pick_event_reader(int cpu)
{
- int phys_id = topology_physical_package_id(cpu);
- int i;
-
- for_each_cpu(i, &cqm_cpumask) {
- if (phys_id == topology_physical_package_id(i))
- return; /* already got reader for this socket */
- }
-
- cpumask_set_cpu(cpu, &cqm_cpumask);
+ u16 pkg_id = topology_physical_package_id(cpu);
+ /* XXX: lock, check if rotation cpu is online, maybe */
+ /*
+ * Pick a reader if there isn't one already.
+ */
+ if (cqm_pkgs_data[pkg_id]->rotation_cpu != -1)
+ cqm_pkgs_data[pkg_id]->rotation_cpu = cpu;
}
static void intel_cqm_cpu_starting(unsigned int cpu)
{
struct intel_pqr_state *state = &per_cpu(pqr_state, cpu);
struct cpuinfo_x86 *c = &cpu_data(cpu);
+ u16 pkg_id = topology_physical_package_id(cpu);
state->rmid = 0;
state->closid = 0;
- WARN_ON(c->x86_cache_max_rmid != cqm_max_rmid);
+ /* XXX: lock */
+ /* XXX: Make sure this case is handled when hotplug happens. */
+ WARN_ON(c->x86_cache_max_rmid != cqm_pkgs_data[pkg_id]->max_rmid);
WARN_ON(c->x86_cache_occ_scale != cqm_l3_scale);
}
static void intel_cqm_cpu_exit(unsigned int cpu)
{
- int phys_id = topology_physical_package_id(cpu);
- int i;
-
/*
* Is @cpu a designated cqm reader?
*/
- if (!cpumask_test_and_clear_cpu(cpu, &cqm_cpumask))
- return;
-
- for_each_online_cpu(i) {
- if (i == cpu)
- continue;
+ u16 pkg_id = topology_physical_package_id(cpu);
- if (phys_id == topology_physical_package_id(i)) {
- cpumask_set_cpu(i, &cqm_cpumask);
- break;
- }
- }
+ if (cqm_pkgs_data[pkg_id]->rotation_cpu != cpu)
+ return;
+ /* XXX: do remove unused packages */
+ cqm_pkgs_data[pkg_id]->rotation_cpu = cpumask_any_but(
+ topology_core_cpumask(cpu), cpu);
}
static int intel_cqm_cpu_notifier(struct notifier_block *nb,
@@ -616,6 +650,7 @@ static int intel_cqm_cpu_notifier(struct notifier_block *nb,
intel_cqm_cpu_exit(cpu);
break;
case CPU_STARTING:
+ pkg_data_init_cpu(cpu);
intel_cqm_cpu_starting(cpu);
cqm_pick_event_reader(cpu);
break;
@@ -632,12 +667,17 @@ static const struct x86_cpu_id intel_cqm_match[] = {
static int __init intel_cqm_init(void)
{
char *str, scale[20];
- int i, cpu, ret;
+ int i, cpu, ret = 0, min_max_rmid = 0;
if (!x86_match_cpu(intel_cqm_match))
return -ENODEV;
cqm_l3_scale = boot_cpu_data.x86_cache_occ_scale;
+ if (WARN_ON(cqm_l3_scale == 0))
+ cqm_l3_scale = 1;
+
+ for (i = 0; i < PQR_MAX_NR_PKGS; i++)
+ cqm_pkgs_data[i] = NULL;
/*
* It's possible that not all resources support the same number
@@ -650,17 +690,20 @@ static int __init intel_cqm_init(void)
*/
cpu_notifier_register_begin();
+ /* XXX: assert all cpus in pkg have same nr rmids (they should). */
for_each_online_cpu(cpu) {
- struct cpuinfo_x86 *c = &cpu_data(cpu);
-
- if (c->x86_cache_max_rmid < cqm_max_rmid)
- cqm_max_rmid = c->x86_cache_max_rmid;
+ ret = pkg_data_init_cpu(cpu);
+ if (ret)
+ goto error;
+ }
- if (c->x86_cache_occ_scale != cqm_l3_scale) {
- pr_err("Multiple LLC scale values, disabling\n");
- ret = -EINVAL;
- goto out;
- }
+ /* Select the minimum of the maximum rmids to use as limit for
+ * threshold. XXX: per-package threshold.
+ */
+ cqm_pkg_id_for_each_online(i) {
+ if (min_max_rmid < cqm_pkgs_data[i]->max_rmid)
+ min_max_rmid = cqm_pkgs_data[i]->max_rmid;
+ intel_cqm_setup_pkg_prmid_pools(i);
}
/*
@@ -671,21 +714,17 @@ static int __init intel_cqm_init(void)
* For a 35MB LLC and 56 RMIDs, this is ~1.8% of the LLC.
*/
__intel_cqm_max_threshold =
- boot_cpu_data.x86_cache_size * 1024 / (cqm_max_rmid + 1);
+ boot_cpu_data.x86_cache_size * 1024 / (min_max_rmid + 1);
snprintf(scale, sizeof(scale), "%u", cqm_l3_scale);
str = kstrdup(scale, GFP_KERNEL);
if (!str) {
ret = -ENOMEM;
- goto out;
+ goto error;
}
event_attr_intel_cqm_llc_scale.event_str = str;
- ret = intel_cqm_setup_rmid_cache();
- if (ret)
- goto out;
-
for_each_online_cpu(i) {
intel_cqm_cpu_starting(i);
cqm_pick_event_reader(i);
@@ -695,13 +734,20 @@ static int __init intel_cqm_init(void)
ret = perf_pmu_register(&intel_cqm_pmu, "intel_cqm", -1);
if (ret)
- pr_err("Intel CQM perf registration failed: %d\n", ret);
- else
- pr_info("Intel CQM monitoring enabled\n");
+ goto error;
-out:
+ cpu_notifier_register_done();
+
+ pr_info("Intel CQM monitoring enabled with at least %u rmids per package.\n",
+ min_max_rmid + 1);
+
+ return ret;
+
+error:
+ pr_err("Intel CQM perf registration failed: %d\n", ret);
cpu_notifier_register_done();
return ret;
}
+
device_initcall(intel_cqm_init);
diff --git a/arch/x86/events/intel/cqm.h b/arch/x86/events/intel/cqm.h
index e25d0a1..a25d49b 100644
--- a/arch/x86/events/intel/cqm.h
+++ b/arch/x86/events/intel/cqm.h
@@ -19,14 +19,168 @@
#include <asm/pqr_common.h>
/*
+ * struct prmid: Package RMID. Per-package wrapper for a rmid.
+ * @last_read_value: Least read value.
+ * @last_read_time: Time last read, used when throtling read rate.
+ * @pool_entry: Attaches to a prmid pool in cqm_pkg_data.
+ * @rmid: The rmid value to be programed in hardware.
+ *
+ * Its accesors ensure that CQM events for this rmid are read atomically and
+ * allow to throtle the frequency of reads to up to one each
+ * __rmid_min_update_time ms.
+ */
+struct prmid {
+ atomic64_t last_read_value;
+ atomic64_t last_read_time;
+ struct list_head pool_entry;
+ u32 rmid;
+};
+
+/*
* Minimum time elapsed between reads of occupancy value for an RMID when
* transversing the monr hierarchy.
*/
#define RMID_DEFAULT_MIN_UPDATE_TIME 20 /* ms */
+static unsigned int __rmid_min_update_time = RMID_DEFAULT_MIN_UPDATE_TIME;
+
+static inline int cqm_prmid_update(struct prmid *prmid);
# define INVALID_RMID (-1)
/*
+ * struct pkg_data: Per-package CQM data.
+ * @max_rmid: Max rmid valid for cpus in this package.
+ * @prmids_by_rmid: Utility mapping between rmid values and prmids.
+ * XXX: Make it an array of prmids.
+ * @free_prmid_pool: Free prmids.
+ * @pkg_data_mutex: Hold for stability when modifying pmonrs
+ * hierarchy.
+ * @pkg_data_lock: Hold to protect variables that may be accessed
+ * during process scheduling. The locks for all
+ * packages must be held when modifying the monr
+ * hierarchy.
+ * @rotation_cpu: CPU to run @rotation_work on, it must be in the
+ * package associated to this instance of pkg_data.
+ */
+struct pkg_data {
+ u32 max_rmid;
+ /* Quick map from rmids to prmids. */
+ struct prmid **prmids_by_rmid;
+
+ /*
+ * Pools of prmids used in rotation logic.
+ */
+ struct list_head free_prmids_pool;
+
+ struct mutex pkg_data_mutex;
+ raw_spinlock_t pkg_data_lock;
+
+ int rotation_cpu;
+};
+
+extern struct pkg_data *cqm_pkgs_data[PQR_MAX_NR_PKGS];
+
+static inline u16 __cqm_pkgs_data_next_online(u16 pkg_id)
+{
+ while (!cqm_pkgs_data[++pkg_id] && pkg_id < PQR_MAX_NR_PKGS)
+ ;
+ return pkg_id;
+}
+
+static inline u16 __cqm_pkgs_data_first_online(void)
+{
+ if (cqm_pkgs_data[0])
+ return 0;
+ return __cqm_pkgs_data_next_online(0);
+}
+
+/* Iterate for each online pkgs data */
+#define cqm_pkg_id_for_each_online(pkg_id__) \
+ for (pkg_id__ = __cqm_pkgs_data_first_online(); \
+ pkg_id__ < PQR_MAX_NR_PKGS; \
+ pkg_id__ = __cqm_pkgs_data_next_online(pkg_id__))
+
+#define __pkg_data(pmonr, member) cqm_pkgs_data[pmonr->pkg_id]->member
+
+/*
+ * Utility function and macros to manage per-package locks.
+ * Use macros to keep flags in caller's stace.
+ * Hold lock in all the packages, required to alter the monr hierarchy
+ */
+static inline void monr_hrchy_acquire_mutexes(void)
+{
+ int i;
+
+ cqm_pkg_id_for_each_online(i)
+ mutex_lock_nested(&cqm_pkgs_data[i]->pkg_data_mutex, i);
+}
+
+# define monr_hrchy_acquire_raw_spin_locks_irq_save(flags, i) \
+ do { \
+ raw_local_irq_save(flags); \
+ cqm_pkg_id_for_each_online(i) {\
+ raw_spin_lock_nested( \
+ &cqm_pkgs_data[i]->pkg_data_lock, i); \
+ } \
+ } while (0)
+
+#define monr_hrchy_acquire_locks(flags, i) \
+ do {\
+ monr_hrchy_acquire_mutexes(); \
+ monr_hrchy_acquire_raw_spin_locks_irq_save(flags, i); \
+ } while (0)
+
+static inline void monr_hrchy_release_mutexes(void)
+{
+ int i;
+
+ cqm_pkg_id_for_each_online(i)
+ mutex_unlock(&cqm_pkgs_data[i]->pkg_data_mutex);
+}
+
+# define monr_hrchy_release_raw_spin_locks_irq_restore(flags, i) \
+ do { \
+ cqm_pkg_id_for_each_online(i) {\
+ raw_spin_unlock(&cqm_pkgs_data[i]->pkg_data_lock); \
+ } \
+ raw_local_irq_restore(flags); \
+ } while (0)
+
+#define monr_hrchy_release_locks(flags, i) \
+ do {\
+ monr_hrchy_release_raw_spin_locks_irq_restore(flags, i); \
+ monr_hrchy_release_mutexes(); \
+ } while (0)
+
+static inline void monr_hrchy_assert_held_mutexes(void)
+{
+ int i;
+
+ cqm_pkg_id_for_each_online(i)
+ lockdep_assert_held(&cqm_pkgs_data[i]->pkg_data_mutex);
+}
+
+static inline void monr_hrchy_assert_held_raw_spin_locks(void)
+{
+ int i;
+
+ cqm_pkg_id_for_each_online(i)
+ lockdep_assert_held(&cqm_pkgs_data[i]->pkg_data_lock);
+}
+#ifdef CONFIG_LOCKDEP
+static inline int monr_hrchy_count_held_raw_spin_locks(void)
+{
+ int i, nr_held = 0;
+
+ cqm_pkg_id_for_each_online(i) {
+ if (lockdep_is_held(&cqm_pkgs_data[i]->pkg_data_lock))
+ nr_held++;
+ }
+ return nr_held;
+}
+#endif
+
+/*
* Time between execution of rotation logic. The frequency of execution does
* not affect the rate at which RMIDs are recycled, except by the delay by the
* delay updating the prmid's and their pools.
diff --git a/arch/x86/include/asm/pqr_common.h b/arch/x86/include/asm/pqr_common.h
index 0c2001b..f770637 100644
--- a/arch/x86/include/asm/pqr_common.h
+++ b/arch/x86/include/asm/pqr_common.h
@@ -27,5 +27,7 @@ struct intel_pqr_state {
DECLARE_PER_CPU(struct intel_pqr_state, pqr_state);
+#define PQR_MAX_NR_PKGS 8
+
#endif
#endif
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | Vikas Shivappa <vikas.shivappa@linux.intel.com> |
|---|---|
| Date | 2016-04-29 23:00 +0200 |
| Subject | Re: [PATCH 09/32] perf/x86/intel/cqm: add per-package RMIDs, data and locks |
| Message-ID | <rtnG2-3Ts-1@gated-at.bofh.it> |
| In reply to | #1390748 |
> @@ -27,5 +27,7 @@ struct intel_pqr_state {
>
> DECLARE_PER_CPU(struct intel_pqr_state, pqr_state);
>
> +#define PQR_MAX_NR_PKGS 8
topology_max_packages()
> +
> #endif
> #endif
> --
> 2.8.0.rc3.226.g39d4020
>
>
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 29/32] perf,perf/x86,perf/powerpc,perf/arm,perf/*: add int error return to pmu::read |
| Message-ID | <rt8xl-8qy-43@gated-at.bofh.it> |
| In reply to | #1390741 |
New PMUs, such as CQM's, do not guarantee that a read will succeed even
if pmu::add was successful.
In the generic code, this patch adds an int error return and completes the
error checking path up to perf_read().
In CQM's PMU, it adds proper error handling of hw read failure errors.
In other PMUs, pmu::read() simply returns 0.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/alpha/kernel/perf_event.c | 3 +-
arch/arc/kernel/perf_event.c | 3 +-
arch/arm64/include/asm/hw_breakpoint.h | 2 +-
arch/arm64/kernel/hw_breakpoint.c | 3 +-
arch/metag/kernel/perf/perf_event.c | 5 ++-
arch/mips/kernel/perf_event_mipsxx.c | 3 +-
arch/powerpc/include/asm/hw_breakpoint.h | 2 +-
arch/powerpc/kernel/hw_breakpoint.c | 3 +-
arch/powerpc/perf/core-book3s.c | 11 +++---
arch/powerpc/perf/core-fsl-emb.c | 5 ++-
arch/powerpc/perf/hv-24x7.c | 5 ++-
arch/powerpc/perf/hv-gpci.c | 3 +-
arch/s390/kernel/perf_cpum_cf.c | 5 ++-
arch/s390/kernel/perf_cpum_sf.c | 3 +-
arch/sh/include/asm/hw_breakpoint.h | 2 +-
arch/sh/kernel/hw_breakpoint.c | 3 +-
arch/sparc/kernel/perf_event.c | 2 +-
arch/tile/kernel/perf_event.c | 3 +-
arch/x86/events/amd/ibs.c | 2 +-
arch/x86/events/amd/iommu.c | 5 ++-
arch/x86/events/amd/uncore.c | 3 +-
arch/x86/events/core.c | 3 +-
arch/x86/events/intel/bts.c | 3 +-
arch/x86/events/intel/cqm.c | 30 ++++++++------
arch/x86/events/intel/cstate.c | 3 +-
arch/x86/events/intel/pt.c | 3 +-
arch/x86/events/intel/rapl.c | 3 +-
arch/x86/events/intel/uncore.c | 3 +-
arch/x86/events/intel/uncore.h | 2 +-
arch/x86/events/msr.c | 3 +-
arch/x86/include/asm/hw_breakpoint.h | 2 +-
arch/x86/kernel/hw_breakpoint.c | 3 +-
arch/x86/kvm/pmu.h | 10 +++--
drivers/bus/arm-cci.c | 3 +-
drivers/bus/arm-ccn.c | 3 +-
drivers/perf/arm_pmu.c | 3 +-
include/linux/perf_event.h | 6 +--
kernel/events/core.c | 68 +++++++++++++++++++++-----------
38 files changed, 141 insertions(+), 86 deletions(-)
diff --git a/arch/alpha/kernel/perf_event.c b/arch/alpha/kernel/perf_event.c
index 5c218aa..3bf8a60 100644
--- a/arch/alpha/kernel/perf_event.c
+++ b/arch/alpha/kernel/perf_event.c
@@ -520,11 +520,12 @@ static void alpha_pmu_del(struct perf_event *event, int flags)
}
-static void alpha_pmu_read(struct perf_event *event)
+static int alpha_pmu_read(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
alpha_perf_event_update(event, hwc, hwc->idx, 0);
+ return 0;
}
diff --git a/arch/arc/kernel/perf_event.c b/arch/arc/kernel/perf_event.c
index 8b134cf..6e4f819 100644
--- a/arch/arc/kernel/perf_event.c
+++ b/arch/arc/kernel/perf_event.c
@@ -116,9 +116,10 @@ static void arc_perf_event_update(struct perf_event *event,
local64_sub(delta, &hwc->period_left);
}
-static void arc_pmu_read(struct perf_event *event)
+static int arc_pmu_read(struct perf_event *event)
{
arc_perf_event_update(event, &event->hw, event->hw.idx);
+ return 0;
}
static int arc_pmu_cache_event(u64 config)
diff --git a/arch/arm64/include/asm/hw_breakpoint.h b/arch/arm64/include/asm/hw_breakpoint.h
index 115ea2a..869ce97 100644
--- a/arch/arm64/include/asm/hw_breakpoint.h
+++ b/arch/arm64/include/asm/hw_breakpoint.h
@@ -126,7 +126,7 @@ extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused,
extern int arch_install_hw_breakpoint(struct perf_event *bp);
extern void arch_uninstall_hw_breakpoint(struct perf_event *bp);
-extern void hw_breakpoint_pmu_read(struct perf_event *bp);
+extern int hw_breakpoint_pmu_read(struct perf_event *bp);
extern int hw_breakpoint_slots(int type);
#ifdef CONFIG_HAVE_HW_BREAKPOINT
diff --git a/arch/arm64/kernel/hw_breakpoint.c b/arch/arm64/kernel/hw_breakpoint.c
index 4ef5373..ac1a6ca 100644
--- a/arch/arm64/kernel/hw_breakpoint.c
+++ b/arch/arm64/kernel/hw_breakpoint.c
@@ -942,8 +942,9 @@ static int __init arch_hw_breakpoint_init(void)
}
arch_initcall(arch_hw_breakpoint_init);
-void hw_breakpoint_pmu_read(struct perf_event *bp)
+int hw_breakpoint_pmu_read(struct perf_event *bp)
{
+ return 0;
}
/*
diff --git a/arch/metag/kernel/perf/perf_event.c b/arch/metag/kernel/perf/perf_event.c
index 2478ec6..9721c1a 100644
--- a/arch/metag/kernel/perf/perf_event.c
+++ b/arch/metag/kernel/perf/perf_event.c
@@ -360,15 +360,16 @@ static void metag_pmu_del(struct perf_event *event, int flags)
perf_event_update_userpage(event);
}
-static void metag_pmu_read(struct perf_event *event)
+static int metag_pmu_read(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
/* Don't read disabled counters! */
if (hwc->idx < 0)
- return;
+ return 0;
metag_pmu_event_update(event, hwc, hwc->idx);
+ return 0;
}
static struct pmu pmu = {
diff --git a/arch/mips/kernel/perf_event_mipsxx.c b/arch/mips/kernel/perf_event_mipsxx.c
index 9bc1191..bdc0915 100644
--- a/arch/mips/kernel/perf_event_mipsxx.c
+++ b/arch/mips/kernel/perf_event_mipsxx.c
@@ -509,7 +509,7 @@ static void mipspmu_del(struct perf_event *event, int flags)
perf_event_update_userpage(event);
}
-static void mipspmu_read(struct perf_event *event)
+static int mipspmu_read(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
@@ -518,6 +518,7 @@ static void mipspmu_read(struct perf_event *event)
return;
mipspmu_event_update(event, hwc, hwc->idx);
+ return 0;
}
static void mipspmu_enable(struct pmu *pmu)
diff --git a/arch/powerpc/include/asm/hw_breakpoint.h b/arch/powerpc/include/asm/hw_breakpoint.h
index ac6432d..5218696 100644
--- a/arch/powerpc/include/asm/hw_breakpoint.h
+++ b/arch/powerpc/include/asm/hw_breakpoint.h
@@ -66,7 +66,7 @@ extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused,
unsigned long val, void *data);
int arch_install_hw_breakpoint(struct perf_event *bp);
void arch_uninstall_hw_breakpoint(struct perf_event *bp);
-void hw_breakpoint_pmu_read(struct perf_event *bp);
+int hw_breakpoint_pmu_read(struct perf_event *bp);
extern void flush_ptrace_hw_breakpoint(struct task_struct *tsk);
extern struct pmu perf_ops_bp;
diff --git a/arch/powerpc/kernel/hw_breakpoint.c b/arch/powerpc/kernel/hw_breakpoint.c
index aec9a1b..d462d8a 100644
--- a/arch/powerpc/kernel/hw_breakpoint.c
+++ b/arch/powerpc/kernel/hw_breakpoint.c
@@ -361,7 +361,8 @@ void flush_ptrace_hw_breakpoint(struct task_struct *tsk)
t->ptrace_bps[0] = NULL;
}
-void hw_breakpoint_pmu_read(struct perf_event *bp)
+int hw_breakpoint_pmu_read(struct perf_event *bp)
{
/* TODO */
+ return 0;
}
diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c
index 97a1d40..0baf04e 100644
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -1002,20 +1002,20 @@ static u64 check_and_compute_delta(u64 prev, u64 val)
return delta;
}
-static void power_pmu_read(struct perf_event *event)
+static int power_pmu_read(struct perf_event *event)
{
s64 val, delta, prev;
if (event->hw.state & PERF_HES_STOPPED)
- return;
+ return 0;
if (!event->hw.idx)
- return;
+ return 0;
if (is_ebb_event(event)) {
val = read_pmc(event->hw.idx);
local64_set(&event->hw.prev_count, val);
- return;
+ return 0;
}
/*
@@ -1029,7 +1029,7 @@ static void power_pmu_read(struct perf_event *event)
val = read_pmc(event->hw.idx);
delta = check_and_compute_delta(prev, val);
if (!delta)
- return;
+ return 0;
} while (local64_cmpxchg(&event->hw.prev_count, prev, val) != prev);
local64_add(delta, &event->count);
@@ -1049,6 +1049,7 @@ static void power_pmu_read(struct perf_event *event)
if (val < 1)
val = 1;
} while (local64_cmpxchg(&event->hw.period_left, prev, val) != prev);
+ return 0;
}
/*
diff --git a/arch/powerpc/perf/core-fsl-emb.c b/arch/powerpc/perf/core-fsl-emb.c
index 5d747b4..46d982e 100644
--- a/arch/powerpc/perf/core-fsl-emb.c
+++ b/arch/powerpc/perf/core-fsl-emb.c
@@ -176,12 +176,12 @@ static void write_pmlcb(int idx, unsigned long val)
isync();
}
-static void fsl_emb_pmu_read(struct perf_event *event)
+static int fsl_emb_pmu_read(struct perf_event *event)
{
s64 val, delta, prev;
if (event->hw.state & PERF_HES_STOPPED)
- return;
+ return 0;
/*
* Performance monitor interrupts come even when interrupts
@@ -198,6 +198,7 @@ static void fsl_emb_pmu_read(struct perf_event *event)
delta = (val - prev) & 0xfffffffful;
local64_add(delta, &event->count);
local64_sub(delta, &event->hw.period_left);
+ return 0;
}
/*
diff --git a/arch/powerpc/perf/hv-24x7.c b/arch/powerpc/perf/hv-24x7.c
index 2da41b7..eddc853 100644
--- a/arch/powerpc/perf/hv-24x7.c
+++ b/arch/powerpc/perf/hv-24x7.c
@@ -1268,7 +1268,7 @@ static void update_event_count(struct perf_event *event, u64 now)
local64_add(now - prev, &event->count);
}
-static void h_24x7_event_read(struct perf_event *event)
+static int h_24x7_event_read(struct perf_event *event)
{
u64 now;
struct hv_24x7_request_buffer *request_buffer;
@@ -1289,7 +1289,7 @@ static void h_24x7_event_read(struct perf_event *event)
int ret;
if (__this_cpu_read(hv_24x7_txn_err))
- return;
+ return 0;
request_buffer = (void *)get_cpu_var(hv_24x7_reqb);
@@ -1323,6 +1323,7 @@ static void h_24x7_event_read(struct perf_event *event)
now = h_24x7_get_value(event);
update_event_count(event, now);
}
+ return 0;
}
static void h_24x7_event_start(struct perf_event *event, int flags)
diff --git a/arch/powerpc/perf/hv-gpci.c b/arch/powerpc/perf/hv-gpci.c
index 7aa3723..d467ee6 100644
--- a/arch/powerpc/perf/hv-gpci.c
+++ b/arch/powerpc/perf/hv-gpci.c
@@ -191,12 +191,13 @@ static u64 h_gpci_get_value(struct perf_event *event)
return count;
}
-static void h_gpci_event_update(struct perf_event *event)
+static int h_gpci_event_update(struct perf_event *event)
{
s64 prev;
u64 now = h_gpci_get_value(event);
prev = local64_xchg(&event->hw.prev_count, now);
local64_add(now - prev, &event->count);
+ return 0;
}
static void h_gpci_event_start(struct perf_event *event, int flags)
diff --git a/arch/s390/kernel/perf_cpum_cf.c b/arch/s390/kernel/perf_cpum_cf.c
index 62f066b..719ec56 100644
--- a/arch/s390/kernel/perf_cpum_cf.c
+++ b/arch/s390/kernel/perf_cpum_cf.c
@@ -471,12 +471,13 @@ out:
return err;
}
-static void cpumf_pmu_read(struct perf_event *event)
+static int cpumf_pmu_read(struct perf_event *event)
{
if (event->hw.state & PERF_HES_STOPPED)
- return;
+ return 0;
hw_perf_event_update(event);
+ return 0;
}
static void cpumf_pmu_start(struct perf_event *event, int flags)
diff --git a/arch/s390/kernel/perf_cpum_sf.c b/arch/s390/kernel/perf_cpum_sf.c
index eaab9a7..605055c 100644
--- a/arch/s390/kernel/perf_cpum_sf.c
+++ b/arch/s390/kernel/perf_cpum_sf.c
@@ -1298,9 +1298,10 @@ static void hw_perf_event_update(struct perf_event *event, int flush_all)
sampl_overflow, event_overflow);
}
-static void cpumsf_pmu_read(struct perf_event *event)
+static int cpumsf_pmu_read(struct perf_event *event)
{
/* Nothing to do ... updates are interrupt-driven */
+ return 0;
}
/* Activate sampling control.
diff --git a/arch/sh/include/asm/hw_breakpoint.h b/arch/sh/include/asm/hw_breakpoint.h
index ec9ad59..d3ad1bf 100644
--- a/arch/sh/include/asm/hw_breakpoint.h
+++ b/arch/sh/include/asm/hw_breakpoint.h
@@ -60,7 +60,7 @@ extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused,
int arch_install_hw_breakpoint(struct perf_event *bp);
void arch_uninstall_hw_breakpoint(struct perf_event *bp);
-void hw_breakpoint_pmu_read(struct perf_event *bp);
+int hw_breakpoint_pmu_read(struct perf_event *bp);
extern void arch_fill_perf_breakpoint(struct perf_event *bp);
extern int register_sh_ubc(struct sh_ubc *);
diff --git a/arch/sh/kernel/hw_breakpoint.c b/arch/sh/kernel/hw_breakpoint.c
index 2197fc5..3a2e719 100644
--- a/arch/sh/kernel/hw_breakpoint.c
+++ b/arch/sh/kernel/hw_breakpoint.c
@@ -401,9 +401,10 @@ int __kprobes hw_breakpoint_exceptions_notify(struct notifier_block *unused,
return hw_breakpoint_handler(data);
}
-void hw_breakpoint_pmu_read(struct perf_event *bp)
+int hw_breakpoint_pmu_read(struct perf_event *bp)
{
/* TODO */
+ return 0;
}
int register_sh_ubc(struct sh_ubc *ubc)
diff --git a/arch/sparc/kernel/perf_event.c b/arch/sparc/kernel/perf_event.c
index 6596f66..ec454cd 100644
--- a/arch/sparc/kernel/perf_event.c
+++ b/arch/sparc/kernel/perf_event.c
@@ -1131,7 +1131,7 @@ static void sparc_pmu_del(struct perf_event *event, int _flags)
local_irq_restore(flags);
}
-static void sparc_pmu_read(struct perf_event *event)
+static int sparc_pmu_read(struct perf_event *event)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
diff --git a/arch/tile/kernel/perf_event.c b/arch/tile/kernel/perf_event.c
index 8767060..2b27890 100644
--- a/arch/tile/kernel/perf_event.c
+++ b/arch/tile/kernel/perf_event.c
@@ -734,9 +734,10 @@ static void tile_pmu_del(struct perf_event *event, int flags)
/*
* Propagate event elapsed time into the event.
*/
-static inline void tile_pmu_read(struct perf_event *event)
+static inline int tile_pmu_read(struct perf_event *event)
{
tile_perf_event_update(event);
+ return 0;
}
/*
diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
index feb90f6..03032ed 100644
--- a/arch/x86/events/amd/ibs.c
+++ b/arch/x86/events/amd/ibs.c
@@ -510,7 +510,7 @@ static void perf_ibs_del(struct perf_event *event, int flags)
perf_event_update_userpage(event);
}
-static void perf_ibs_read(struct perf_event *event) { }
+static int perf_ibs_read(struct perf_event *event) { return 0; }
PMU_FORMAT_ATTR(rand_en, "config:57");
PMU_FORMAT_ATTR(cnt_ctl, "config:19");
diff --git a/arch/x86/events/amd/iommu.c b/arch/x86/events/amd/iommu.c
index 40625ca..57340ae 100644
--- a/arch/x86/events/amd/iommu.c
+++ b/arch/x86/events/amd/iommu.c
@@ -317,7 +317,7 @@ static void perf_iommu_start(struct perf_event *event, int flags)
}
-static void perf_iommu_read(struct perf_event *event)
+static int perf_iommu_read(struct perf_event *event)
{
u64 count = 0ULL;
u64 prev_raw_count = 0ULL;
@@ -335,13 +335,14 @@ static void perf_iommu_read(struct perf_event *event)
prev_raw_count = local64_read(&hwc->prev_count);
if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
count) != prev_raw_count)
- return;
+ return 0;
/* Handling 48-bit counter overflowing */
delta = (count << COUNTER_SHIFT) - (prev_raw_count << COUNTER_SHIFT);
delta >>= COUNTER_SHIFT;
local64_add(delta, &event->count);
+ return 0;
}
static void perf_iommu_stop(struct perf_event *event, int flags)
diff --git a/arch/x86/events/amd/uncore.c b/arch/x86/events/amd/uncore.c
index 98ac573..5e4f1e7 100644
--- a/arch/x86/events/amd/uncore.c
+++ b/arch/x86/events/amd/uncore.c
@@ -71,7 +71,7 @@ static struct amd_uncore *event_to_amd_uncore(struct perf_event *event)
return NULL;
}
-static void amd_uncore_read(struct perf_event *event)
+static int amd_uncore_read(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
u64 prev, new;
@@ -88,6 +88,7 @@ static void amd_uncore_read(struct perf_event *event)
delta = (new << COUNTER_SHIFT) - (prev << COUNTER_SHIFT);
delta >>= COUNTER_SHIFT;
local64_add(delta, &event->count);
+ return 0;
}
static void amd_uncore_start(struct perf_event *event, int flags)
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 041e442..8323ecd 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -1765,9 +1765,10 @@ static int __init init_hw_perf_events(void)
}
early_initcall(init_hw_perf_events);
-static inline void x86_pmu_read(struct perf_event *event)
+static inline int x86_pmu_read(struct perf_event *event)
{
x86_perf_event_update(event);
+ return 0;
}
/*
diff --git a/arch/x86/events/intel/bts.c b/arch/x86/events/intel/bts.c
index 0a6e393..95e18c6 100644
--- a/arch/x86/events/intel/bts.c
+++ b/arch/x86/events/intel/bts.c
@@ -510,8 +510,9 @@ static int bts_event_init(struct perf_event *event)
return 0;
}
-static void bts_event_read(struct perf_event *event)
+static int bts_event_read(struct perf_event *event)
{
+ return 0;
}
static __init int bts_init(void)
diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index 33691c1..8189e47 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -2444,7 +2444,7 @@ exit_error:
}
/* Read current package immediately and remote pkg (if any) from cache. */
-static void __read_task_event(struct perf_event *event)
+static int __read_task_event(struct perf_event *event)
{
int i, ret;
u64 count = 0;
@@ -2459,26 +2459,31 @@ static void __read_task_event(struct perf_event *event)
ret = pmonr__get_read_rmid(pmonr, &rmid, true);
if (ret)
- return;
+ return ret;
if (rmid == INVALID_RMID)
continue;
prmid = __prmid_from_rmid(i, rmid);
if (WARN_ON_ONCE(!prmid))
- return;
+ return -1;
/* update and read local for this cpu's package. */
- if (i == pkg_id)
- cqm_prmid_update(prmid);
+ if (i == pkg_id) {
+ ret = cqm_prmid_update(prmid);
+ if (ret < 0)
+ return ret;
+ }
count += atomic64_read(&prmid->last_read_value);
}
local64_set(&event->count, count);
+ return 0;
}
/* Read current package immediately and remote pkg (if any) from cache. */
-static void intel_cqm_event_read(struct perf_event *event)
+static int intel_cqm_event_read(struct perf_event *event)
{
struct monr *monr;
u64 count;
+ int ret;
u16 pkg_id = topology_physical_package_id(smp_processor_id());
monr = monr_from_event(event);
@@ -2491,23 +2496,24 @@ static void intel_cqm_event_read(struct perf_event *event)
*/
if (event->parent) {
local64_set(&event->count, 0);
- return;
+ return 0;
}
if (event->attach_state & PERF_ATTACH_TASK) {
- __read_task_event(event);
- return;
+ return __read_task_event(event);
}
/* It's either a cgroup or a cpu event. */
if (WARN_ON_ONCE(event->cpu < 0))
- return;
+ return -1;
/* XXX: expose fail_on_inh_descendant as a configuration parameter? */
- pmonr__read_subtree(monr, pkg_id, &count, false);
+ ret = pmonr__read_subtree(monr, pkg_id, &count, false);
+ if (ret < 0)
+ return ret;
local64_set(&event->count, count);
- return;
+ return 0;
}
static inline bool cqm_group_leader(struct perf_event *event)
diff --git a/arch/x86/events/intel/cstate.c b/arch/x86/events/intel/cstate.c
index 9ba4e41..390745c1 100644
--- a/arch/x86/events/intel/cstate.c
+++ b/arch/x86/events/intel/cstate.c
@@ -322,7 +322,7 @@ static inline u64 cstate_pmu_read_counter(struct perf_event *event)
return val;
}
-static void cstate_pmu_event_update(struct perf_event *event)
+static int cstate_pmu_event_update(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
u64 prev_raw_count, new_raw_count;
@@ -336,6 +336,7 @@ again:
goto again;
local64_add(new_raw_count - prev_raw_count, &event->count);
+ return 0;
}
static void cstate_pmu_event_start(struct perf_event *event, int mode)
diff --git a/arch/x86/events/intel/pt.c b/arch/x86/events/intel/pt.c
index 05ef87d..477eb4f 100644
--- a/arch/x86/events/intel/pt.c
+++ b/arch/x86/events/intel/pt.c
@@ -1120,8 +1120,9 @@ fail:
return ret;
}
-static void pt_event_read(struct perf_event *event)
+static int pt_event_read(struct perf_event *event)
{
+ return 0;
}
static void pt_event_destroy(struct perf_event *event)
diff --git a/arch/x86/events/intel/rapl.c b/arch/x86/events/intel/rapl.c
index 99c4bab..b01abd1 100644
--- a/arch/x86/events/intel/rapl.c
+++ b/arch/x86/events/intel/rapl.c
@@ -408,9 +408,10 @@ static int rapl_pmu_event_init(struct perf_event *event)
return ret;
}
-static void rapl_pmu_event_read(struct perf_event *event)
+static int rapl_pmu_event_read(struct perf_event *event)
{
rapl_event_update(event);
+ return 0;
}
static ssize_t rapl_get_attr_cpumask(struct device *dev,
diff --git a/arch/x86/events/intel/uncore.c b/arch/x86/events/intel/uncore.c
index 17734a6..c01bcc9 100644
--- a/arch/x86/events/intel/uncore.c
+++ b/arch/x86/events/intel/uncore.c
@@ -577,10 +577,11 @@ static void uncore_pmu_event_del(struct perf_event *event, int flags)
event->hw.last_tag = ~0ULL;
}
-void uncore_pmu_event_read(struct perf_event *event)
+int uncore_pmu_event_read(struct perf_event *event)
{
struct intel_uncore_box *box = uncore_event_to_box(event);
uncore_perf_event_update(box, event);
+ return 0;
}
/*
diff --git a/arch/x86/events/intel/uncore.h b/arch/x86/events/intel/uncore.h
index 79766b9..1c5db22 100644
--- a/arch/x86/events/intel/uncore.h
+++ b/arch/x86/events/intel/uncore.h
@@ -337,7 +337,7 @@ struct intel_uncore_box *uncore_pmu_to_box(struct intel_uncore_pmu *pmu, int cpu
u64 uncore_msr_read_counter(struct intel_uncore_box *box, struct perf_event *event);
void uncore_pmu_start_hrtimer(struct intel_uncore_box *box);
void uncore_pmu_cancel_hrtimer(struct intel_uncore_box *box);
-void uncore_pmu_event_read(struct perf_event *event);
+int uncore_pmu_event_read(struct perf_event *event);
void uncore_perf_event_update(struct intel_uncore_box *box, struct perf_event *event);
struct event_constraint *
uncore_get_constraint(struct intel_uncore_box *box, struct perf_event *event);
diff --git a/arch/x86/events/msr.c b/arch/x86/events/msr.c
index 7111400..62d33f6 100644
--- a/arch/x86/events/msr.c
+++ b/arch/x86/events/msr.c
@@ -165,7 +165,7 @@ static inline u64 msr_read_counter(struct perf_event *event)
return now;
}
-static void msr_event_update(struct perf_event *event)
+static int msr_event_update(struct perf_event *event)
{
u64 prev, now;
s64 delta;
@@ -183,6 +183,7 @@ again:
delta = sign_extend64(delta, 31);
local64_add(now - prev, &event->count);
+ return 0;
}
static void msr_event_start(struct perf_event *event, int flags)
diff --git a/arch/x86/include/asm/hw_breakpoint.h b/arch/x86/include/asm/hw_breakpoint.h
index 6c98be8..a1c4ce00 100644
--- a/arch/x86/include/asm/hw_breakpoint.h
+++ b/arch/x86/include/asm/hw_breakpoint.h
@@ -59,7 +59,7 @@ extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused,
int arch_install_hw_breakpoint(struct perf_event *bp);
void arch_uninstall_hw_breakpoint(struct perf_event *bp);
-void hw_breakpoint_pmu_read(struct perf_event *bp);
+int hw_breakpoint_pmu_read(struct perf_event *bp);
void hw_breakpoint_pmu_unthrottle(struct perf_event *bp);
extern void
diff --git a/arch/x86/kernel/hw_breakpoint.c b/arch/x86/kernel/hw_breakpoint.c
index 2bcfb5f..60b8cab7 100644
--- a/arch/x86/kernel/hw_breakpoint.c
+++ b/arch/x86/kernel/hw_breakpoint.c
@@ -539,7 +539,8 @@ int hw_breakpoint_exceptions_notify(
return hw_breakpoint_handler(data);
}
-void hw_breakpoint_pmu_read(struct perf_event *bp)
+int hw_breakpoint_pmu_read(struct perf_event *bp)
{
/* TODO */
+ return 0;
}
diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h
index f96e1f9..46fd299 100644
--- a/arch/x86/kvm/pmu.h
+++ b/arch/x86/kvm/pmu.h
@@ -39,12 +39,14 @@ static inline u64 pmc_bitmask(struct kvm_pmc *pmc)
static inline u64 pmc_read_counter(struct kvm_pmc *pmc)
{
- u64 counter, enabled, running;
+ u64 counter, counter_tmp, enabled, running;
counter = pmc->counter;
- if (pmc->perf_event)
- counter += perf_event_read_value(pmc->perf_event,
- &enabled, &running);
+ if (pmc->perf_event) {
+ if (!perf_event_read_value(pmc->perf_event, &counter_tmp,
+ &enabled, &running))
+ counter += counter_tmp;
+ }
/* FIXME: Scaling needed? */
return counter & pmc_bitmask(pmc);
}
diff --git a/drivers/bus/arm-cci.c b/drivers/bus/arm-cci.c
index a49b283..9fa7b4e 100644
--- a/drivers/bus/arm-cci.c
+++ b/drivers/bus/arm-cci.c
@@ -1033,9 +1033,10 @@ static u64 pmu_event_update(struct perf_event *event)
return new_raw_count;
}
-static void pmu_read(struct perf_event *event)
+static int pmu_read(struct perf_event *event)
{
pmu_event_update(event);
+ return 0;
}
static void pmu_event_set_period(struct perf_event *event)
diff --git a/drivers/bus/arm-ccn.c b/drivers/bus/arm-ccn.c
index 7082c72..a2e4a9c 100644
--- a/drivers/bus/arm-ccn.c
+++ b/drivers/bus/arm-ccn.c
@@ -1123,9 +1123,10 @@ static void arm_ccn_pmu_event_del(struct perf_event *event, int flags)
arm_ccn_pmu_event_release(event);
}
-static void arm_ccn_pmu_event_read(struct perf_event *event)
+static int arm_ccn_pmu_event_read(struct perf_event *event)
{
arm_ccn_pmu_event_update(event);
+ return 0;
}
static irqreturn_t arm_ccn_pmu_overflow_handler(struct arm_ccn_dt *dt)
diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c
index 32346b5..7a78230 100644
--- a/drivers/perf/arm_pmu.c
+++ b/drivers/perf/arm_pmu.c
@@ -163,10 +163,11 @@ again:
return new_raw_count;
}
-static void
+static int
armpmu_read(struct perf_event *event)
{
armpmu_event_update(event);
+ return 0;
}
static void
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index b0f6088..9c973bd 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -333,7 +333,7 @@ struct pmu {
* For sampling capable PMUs this will also update the software period
* hw_perf_event::period_left field.
*/
- void (*read) (struct perf_event *event);
+ int (*read) (struct perf_event *event);
/*
* Group events scheduling is treated as a transaction, add
@@ -786,8 +786,8 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr,
extern void perf_pmu_migrate_context(struct pmu *pmu,
int src_cpu, int dst_cpu);
extern u64 perf_event_read_local(struct perf_event *event);
-extern u64 perf_event_read_value(struct perf_event *event,
- u64 *enabled, u64 *running);
+extern int perf_event_read_value(struct perf_event *event,
+ u64 *total, u64 *enabled, u64 *running);
struct perf_sample_data {
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 804fdd1..cfffa50 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -2539,7 +2539,7 @@ static void __perf_event_sync_stat(struct perf_event *event,
*/
switch (event->state) {
case PERF_EVENT_STATE_ACTIVE:
- event->pmu->read(event);
+ (void)event->pmu->read(event);
/* fall-through */
case PERF_EVENT_STATE_INACTIVE:
@@ -3291,6 +3291,7 @@ static void __perf_event_read(void *info)
return;
raw_spin_lock(&ctx->lock);
+
if (ctx->is_active) {
update_context_time(ctx);
update_cgrp_time_from_event(event);
@@ -3303,14 +3304,15 @@ static void __perf_event_read(void *info)
if (!data->group) {
- pmu->read(event);
- data->ret = 0;
+ data->ret = pmu->read(event);
goto unlock;
}
pmu->start_txn(pmu, PERF_PMU_TXN_READ);
- pmu->read(event);
+ data->ret = pmu->read(event);
+ if (data->ret)
+ goto unlock;
list_for_each_entry(sub, &event->sibling_list, group_entry) {
update_event_times(sub);
@@ -3320,7 +3322,9 @@ static void __perf_event_read(void *info)
* Use sibling's PMU rather than @event's since
* sibling could be on different (eg: software) PMU.
*/
- sub->pmu->read(sub);
+ data->ret = sub->pmu->read(sub);
+ if (data->ret)
+ goto unlock;
}
}
@@ -3341,6 +3345,7 @@ static inline u64 perf_event_count(struct perf_event *event)
* - either for the current task, or for this CPU
* - does not have inherit set, for inherited task events
* will not be local and we cannot read them atomically
+ * - pmu::read cannot fail
*/
u64 perf_event_read_local(struct perf_event *event)
{
@@ -3373,7 +3378,7 @@ u64 perf_event_read_local(struct perf_event *event)
* oncpu == -1).
*/
if (event->oncpu == smp_processor_id())
- event->pmu->read(event);
+ (void) event->pmu->read(event);
val = local64_read(&event->count);
local_irq_restore(flags);
@@ -3410,8 +3415,12 @@ static int perf_event_read(struct perf_event *event, bool group)
PERF_INACTIVE_EV_READ_ANY_CPU) ?
smp_processor_id() : event->cpu;
}
- smp_call_function_single(
- cpu_to_read, __perf_event_read, &data, 1);
+ ret = smp_call_function_single(cpu_to_read,
+ __perf_event_read, &data, 1);
+ if (ret) {
+ WARN_ON_ONCE(ret);
+ return ret;
+ }
ret = data.ret;
} else if (event->state == PERF_EVENT_STATE_INACTIVE) {
struct perf_event_context *ctx = event->ctx;
@@ -3433,7 +3442,6 @@ static int perf_event_read(struct perf_event *event, bool group)
update_event_times(event);
raw_spin_unlock_irqrestore(&ctx->lock, flags);
}
-
return ret;
}
@@ -4035,18 +4043,22 @@ static int perf_release(struct inode *inode, struct file *file)
return 0;
}
-u64 perf_event_read_value(struct perf_event *event, u64 *enabled, u64 *running)
+int perf_event_read_value(struct perf_event *event,
+ u64 *total, u64 *enabled, u64 *running)
{
struct perf_event *child;
- u64 total = 0;
+ int ret;
+ *total = 0;
*enabled = 0;
*running = 0;
mutex_lock(&event->child_mutex);
- (void)perf_event_read(event, false);
- total += perf_event_count(event);
+ ret = perf_event_read(event, false);
+ if (ret)
+ goto exit;
+ *total += perf_event_count(event);
*enabled += event->total_time_enabled +
atomic64_read(&event->child_total_time_enabled);
@@ -4054,14 +4066,17 @@ u64 perf_event_read_value(struct perf_event *event, u64 *enabled, u64 *running)
atomic64_read(&event->child_total_time_running);
list_for_each_entry(child, &event->child_list, child_list) {
- (void)perf_event_read(child, false);
- total += perf_event_count(child);
+ ret = perf_event_read(child, false);
+ if (ret)
+ goto exit;
+ *total += perf_event_count(child);
*enabled += child->total_time_enabled;
*running += child->total_time_running;
}
+exit:
mutex_unlock(&event->child_mutex);
- return total;
+ return ret;
}
EXPORT_SYMBOL_GPL(perf_event_read_value);
@@ -4158,9 +4173,11 @@ static int perf_read_one(struct perf_event *event,
{
u64 enabled, running;
u64 values[4];
- int n = 0;
+ int n = 0, ret;
- values[n++] = perf_event_read_value(event, &enabled, &running);
+ ret = perf_event_read_value(event, &values[n++], &enabled, &running);
+ if (ret)
+ return ret;
if (read_format & PERF_FORMAT_TOTAL_TIME_ENABLED)
values[n++] = enabled;
if (read_format & PERF_FORMAT_TOTAL_TIME_RUNNING)
@@ -5427,7 +5444,7 @@ static void perf_output_read_group(struct perf_output_handle *handle,
values[n++] = running;
if (leader != event)
- leader->pmu->read(leader);
+ (void)leader->pmu->read(leader);
values[n++] = perf_event_count(leader);
if (read_format & PERF_FORMAT_ID)
@@ -5440,7 +5457,7 @@ static void perf_output_read_group(struct perf_output_handle *handle,
if ((sub != event) &&
(sub->state == PERF_EVENT_STATE_ACTIVE))
- sub->pmu->read(sub);
+ (void)sub->pmu->read(sub);
values[n++] = perf_event_count(sub);
if (read_format & PERF_FORMAT_ID)
@@ -6980,8 +6997,9 @@ fail:
preempt_enable_notrace();
}
-static void perf_swevent_read(struct perf_event *event)
+static int perf_swevent_read(struct perf_event *event)
{
+ return 0;
}
static int perf_swevent_add(struct perf_event *event, int flags)
@@ -7421,7 +7439,7 @@ static enum hrtimer_restart perf_swevent_hrtimer(struct hrtimer *hrtimer)
if (event->state != PERF_EVENT_STATE_ACTIVE)
return HRTIMER_NORESTART;
- event->pmu->read(event);
+ (void)event->pmu->read(event);
perf_sample_data_init(&data, 0, event->hw.last_period);
regs = get_irq_regs();
@@ -7536,9 +7554,10 @@ static void cpu_clock_event_del(struct perf_event *event, int flags)
cpu_clock_event_stop(event, flags);
}
-static void cpu_clock_event_read(struct perf_event *event)
+static int cpu_clock_event_read(struct perf_event *event)
{
cpu_clock_event_update(event);
+ return 0;
}
static int cpu_clock_event_init(struct perf_event *event)
@@ -7613,13 +7632,14 @@ static void task_clock_event_del(struct perf_event *event, int flags)
task_clock_event_stop(event, PERF_EF_UPDATE);
}
-static void task_clock_event_read(struct perf_event *event)
+static int task_clock_event_read(struct perf_event *event)
{
u64 now = perf_clock();
u64 delta = now - event->ctx->timestamp;
u64 time = event->ctx->time + delta;
task_clock_event_update(event, time);
+ return 0;
}
static int task_clock_event_init(struct perf_event *event)
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 06:50 +0200 |
| Subject | [PATCH 06/32] x86/intel,cqm: add CONFIG_INTEL_RDT configuration flag and refactor PQR |
| Message-ID | <rt8xk-8qy-21@gated-at.bofh.it> |
| In reply to | #1390741 |
Add Intel's PQR as its own build target, remove its build dependency
on CQM, and add CONFIG_INTEL_RDT as a configuration flag to build PQR
and all of its related drivers (currently CQM, future: MBM, CAT, CDP).
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/x86/Kconfig | 6 ++++++
arch/x86/events/intel/Makefile | 3 ++-
arch/x86/events/intel/cqm.c | 27 +--------------------------
arch/x86/include/asm/pqr_common.h | 31 +++++++++++++++++++++++++++++++
arch/x86/kernel/cpu/Makefile | 4 ++++
arch/x86/kernel/cpu/pqr_common.c | 9 +++++++++
include/linux/perf_event.h | 2 ++
7 files changed, 55 insertions(+), 27 deletions(-)
create mode 100644 arch/x86/include/asm/pqr_common.h
create mode 100644 arch/x86/kernel/cpu/pqr_common.c
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index a494fa3..7b81e6a 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -160,6 +160,12 @@ config X86
select ARCH_USES_HIGH_VMA_FLAGS if X86_INTEL_MEMORY_PROTECTION_KEYS
select ARCH_HAS_PKEYS if X86_INTEL_MEMORY_PROTECTION_KEYS
+config INTEL_RDT
+ def_bool y
+ depends on PERF_EVENTS && CPU_SUP_INTEL
+ ---help---
+ Enable Resource Director Technology for Intel Xeon Microprocessors.
+
config INSTRUCTION_DECODER
def_bool y
depends on KPROBES || PERF_EVENTS || UPROBES
diff --git a/arch/x86/events/intel/Makefile b/arch/x86/events/intel/Makefile
index 3660b2c..7e610bf 100644
--- a/arch/x86/events/intel/Makefile
+++ b/arch/x86/events/intel/Makefile
@@ -1,4 +1,4 @@
-obj-$(CONFIG_CPU_SUP_INTEL) += core.o bts.o cqm.o
+obj-$(CONFIG_CPU_SUP_INTEL) += core.o bts.o
obj-$(CONFIG_CPU_SUP_INTEL) += ds.o knc.o
obj-$(CONFIG_CPU_SUP_INTEL) += lbr.o p4.o p6.o pt.o
obj-$(CONFIG_PERF_EVENTS_INTEL_RAPL) += intel-rapl.o
@@ -7,3 +7,4 @@ obj-$(CONFIG_PERF_EVENTS_INTEL_UNCORE) += intel-uncore.o
intel-uncore-objs := uncore.o uncore_nhmex.o uncore_snb.o uncore_snbep.o
obj-$(CONFIG_PERF_EVENTS_INTEL_CSTATE) += intel-cstate.o
intel-cstate-objs := cstate.o
+obj-$(CONFIG_INTEL_RDT) += cqm.o
diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index afd60dd..8457dd0 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -7,40 +7,15 @@
#include <linux/perf_event.h>
#include <linux/slab.h>
#include <asm/cpu_device_id.h>
+#include <asm/pqr_common.h>
#include "../perf_event.h"
-#define MSR_IA32_PQR_ASSOC 0x0c8f
#define MSR_IA32_QM_CTR 0x0c8e
#define MSR_IA32_QM_EVTSEL 0x0c8d
static u32 cqm_max_rmid = -1;
static unsigned int cqm_l3_scale; /* supposedly cacheline size */
-/**
- * struct intel_pqr_state - State cache for the PQR MSR
- * @rmid: The cached Resource Monitoring ID
- * @closid: The cached Class Of Service ID
- *
- * The upper 32 bits of MSR_IA32_PQR_ASSOC contain closid and the
- * lower 10 bits rmid. The update to MSR_IA32_PQR_ASSOC always
- * contains both parts, so we need to cache them.
- *
- * The cache also helps to avoid pointless updates if the value does
- * not change.
- */
-struct intel_pqr_state {
- u32 rmid;
- u32 closid;
-};
-
-/*
- * The cached intel_pqr_state is strictly per CPU and can never be
- * updated from a remote CPU. Both functions which modify the state
- * (intel_cqm_event_start and intel_cqm_event_stop) are called with
- * interrupts disabled, which is sufficient for the protection.
- */
-static DEFINE_PER_CPU(struct intel_pqr_state, pqr_state);
-
/*
* Updates caller cpu's cache.
*/
diff --git a/arch/x86/include/asm/pqr_common.h b/arch/x86/include/asm/pqr_common.h
new file mode 100644
index 0000000..0c2001b
--- /dev/null
+++ b/arch/x86/include/asm/pqr_common.h
@@ -0,0 +1,31 @@
+#ifndef _X86_PQR_COMMON_H_
+#define _X86_PQR_COMMON_H_
+
+#if defined(CONFIG_INTEL_RDT)
+
+#include <linux/types.h>
+#include <asm/percpu.h>
+
+#define MSR_IA32_PQR_ASSOC 0x0c8f
+
+/**
+ * struct intel_pqr_state - State cache for the PQR MSR
+ * @rmid: The cached Resource Monitoring ID
+ * @closid: The cached Class Of Service ID
+ *
+ * The upper 32 bits of MSR_IA32_PQR_ASSOC contain closid and the
+ * lower 10 bits rmid. The update to MSR_IA32_PQR_ASSOC always
+ * contains both parts, so we need to cache them.
+ *
+ * The cache also helps to avoid pointless updates if the value does
+ * not change.
+ */
+struct intel_pqr_state {
+ u32 rmid;
+ u32 closid;
+};
+
+DECLARE_PER_CPU(struct intel_pqr_state, pqr_state);
+
+#endif
+#endif
diff --git a/arch/x86/kernel/cpu/Makefile b/arch/x86/kernel/cpu/Makefile
index 4a8697f..87e6279 100644
--- a/arch/x86/kernel/cpu/Makefile
+++ b/arch/x86/kernel/cpu/Makefile
@@ -34,6 +34,10 @@ obj-$(CONFIG_CPU_SUP_CENTAUR) += centaur.o
obj-$(CONFIG_CPU_SUP_TRANSMETA_32) += transmeta.o
obj-$(CONFIG_CPU_SUP_UMC_32) += umc.o
+ifdef CONFIG_CPU_SUP_INTEL
+obj-$(CONFIG_INTEL_RDT) += pqr_common.o
+endif
+
obj-$(CONFIG_X86_MCE) += mcheck/
obj-$(CONFIG_MTRR) += mtrr/
obj-$(CONFIG_MICROCODE) += microcode/
diff --git a/arch/x86/kernel/cpu/pqr_common.c b/arch/x86/kernel/cpu/pqr_common.c
new file mode 100644
index 0000000..9eff5d9
--- /dev/null
+++ b/arch/x86/kernel/cpu/pqr_common.c
@@ -0,0 +1,9 @@
+#include <asm/pqr_common.h>
+
+/*
+ * The cached intel_pqr_state is strictly per CPU and can never be
+ * updated from a remote CPU. Both functions which modify the state
+ * (intel_cqm_event_start and intel_cqm_event_stop) are called with
+ * interrupts disabled, which is sufficient for the protection.
+ */
+DEFINE_PER_CPU(struct intel_pqr_state, pqr_state);
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 8bb1532..3a847bf 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -118,6 +118,7 @@ struct hw_perf_event {
/* for tp_event->class */
struct list_head tp_list;
};
+#ifdef CONFIG_INTEL_RDT
struct { /* intel_cqm */
int cqm_state;
u32 cqm_rmid;
@@ -125,6 +126,7 @@ struct hw_perf_event {
struct list_head cqm_groups_entry;
struct list_head cqm_group_entry;
};
+#endif
struct { /* itrace */
int itrace_started;
};
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 07:00 +0200 |
| Subject | [PATCH 15/32] perf/core: add hooks to expose architecture specific features in perf_cgroup |
| Message-ID | <rt8H0-8vJ-15@gated-at.bofh.it> |
| In reply to | #1390741 |
The hooks allows architectures to extend the behavior of the
perf subsystem.
In this patch series, the hooks will be used by Intel's CQM PMU to
provide support for the llc_occupancy event.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
include/linux/perf_event.h | 28 +++++++++++++++++++++++++++-
kernel/events/core.c | 27 +++++++++++++++++++++++++++
2 files changed, 54 insertions(+), 1 deletion(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index bf29258..b010b55 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -690,7 +690,9 @@ struct perf_cgroup_info {
};
struct perf_cgroup {
- struct cgroup_subsys_state css;
+ /* Architecture specific information. */
+ void *arch_info;
+ struct cgroup_subsys_state css;
struct perf_cgroup_info __percpu *info;
};
@@ -1228,4 +1230,28 @@ _name##_show(struct device *dev, \
\
static struct device_attribute format_attr_##_name = __ATTR_RO(_name)
+
+/*
+ * Hooks for architecture specific extensions for perf_cgroup.
+ */
+#ifndef perf_cgroup_arch_css_alloc
+# define perf_cgroup_arch_css_alloc(parent_css, new_css) 0
+#endif
+
+#ifndef perf_cgroup_arch_css_online
+# define perf_cgroup_arch_css_online(css) 0
+#endif
+
+#ifndef perf_cgroup_arch_css_offline
+# define perf_cgroup_arch_css_offline(css) do { } while (0)
+#endif
+
+#ifndef perf_cgroup_arch_css_released
+# define perf_cgroup_arch_css_released(css) do { } while (0)
+#endif
+
+#ifndef perf_cgroup_arch_css_free
+# define perf_cgroup_arch_css_free(css) do { } while (0)
+#endif
+
#endif /* _LINUX_PERF_EVENT_H */
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 4aaec01..6fd226f 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -9794,6 +9794,7 @@ static struct cgroup_subsys_state *
perf_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
{
struct perf_cgroup *jc;
+ int ret;
jc = kzalloc(sizeof(*jc), GFP_KERNEL);
if (!jc)
@@ -9805,13 +9806,36 @@ perf_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
return ERR_PTR(-ENOMEM);
}
+ jc->arch_info = NULL;
+
+ ret = perf_cgroup_arch_css_alloc(parent_css, &jc->css);
+ if (ret)
+ return ERR_PTR(ret);
+
return &jc->css;
}
+static int perf_cgroup_css_online(struct cgroup_subsys_state *css)
+{
+ return perf_cgroup_arch_css_online(css);
+}
+
+static void perf_cgroup_css_offline(struct cgroup_subsys_state *css)
+{
+ perf_cgroup_arch_css_offline(css);
+}
+
+static void perf_cgroup_css_released(struct cgroup_subsys_state *css)
+{
+ perf_cgroup_arch_css_released(css);
+}
+
static void perf_cgroup_css_free(struct cgroup_subsys_state *css)
{
struct perf_cgroup *jc = container_of(css, struct perf_cgroup, css);
+ perf_cgroup_arch_css_free(css);
+
free_percpu(jc->info);
kfree(jc);
}
@@ -9836,6 +9860,9 @@ static void perf_cgroup_attach(struct cgroup_taskset *tset)
struct cgroup_subsys perf_event_cgrp_subsys = {
.css_alloc = perf_cgroup_css_alloc,
+ .css_online = perf_cgroup_css_online,
+ .css_offline = perf_cgroup_css_offline,
+ .css_released = perf_cgroup_css_released,
.css_free = perf_cgroup_css_free,
.attach = perf_cgroup_attach,
};
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 07:00 +0200 |
| Subject | [PATCH 18/32] perf/x86/intel/cqm: use pmu::event_terminate |
| Message-ID | <rt8H0-8vJ-17@gated-at.bofh.it> |
| In reply to | #1390741 |
Utilized to detach a monr from a cgroup before the event's reference
to the cgroup is removed.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/x86/events/intel/cqm.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index f000fd0..dcf7f4a 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -2391,7 +2391,7 @@ static int intel_cqm_event_add(struct perf_event *event, int mode)
return prmid_summary__is_istate(summary) ? -1 : 0;
}
-static void intel_cqm_event_destroy(struct perf_event *event)
+static void intel_cqm_event_terminate(struct perf_event *event)
{
struct perf_event *group_other = NULL;
struct monr *monr;
@@ -2438,6 +2438,17 @@ static void intel_cqm_event_destroy(struct perf_event *event)
if (monr__is_root(monr))
goto exit;
+ /* Handle cgroup event. */
+ if (event->cgrp) {
+ monr->mon_event_group = NULL;
+ if ((event->cgrp->css.flags & CSS_ONLINE) &&
+ !cgrp_to_cqm_info(event->cgrp)->cont_monitoring)
+ __css_stop_monitoring(&monr__get_mon_cgrp(monr)->css);
+
+ goto exit;
+ }
+ WARN_ON_ONCE(!monr_is_event_type(monr));
+
/* Transition all pmonrs to (U)state. */
monr_hrchy_acquire_locks(flags, i);
@@ -2478,8 +2489,6 @@ static int intel_cqm_event_init(struct perf_event *event)
INIT_LIST_HEAD(&event->hw.cqm_event_groups_entry);
INIT_LIST_HEAD(&event->hw.cqm_event_group_entry);
- event->destroy = intel_cqm_event_destroy;
-
mutex_lock(&cqm_mutex);
@@ -2595,6 +2604,7 @@ static struct pmu intel_cqm_pmu = {
.attr_groups = intel_cqm_attr_groups,
.task_ctx_nr = perf_sw_context,
.event_init = intel_cqm_event_init,
+ .event_terminate = intel_cqm_event_terminate,
.add = intel_cqm_event_add,
.del = intel_cqm_event_stop,
.start = intel_cqm_event_start,
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 07:00 +0200 |
| Subject | [PATCH 16/32] perf/x86/intel/cqm: add cgroup support |
| Message-ID | <rt8H0-8vJ-19@gated-at.bofh.it> |
| In reply to | #1390741 |
Create a monr per monitored cgroup. Inserts monrs in the monr hierarchy.
Task events are leaves of the lowest monitored ancestor cgroup (the lowest
cgroup ancestor with a monr).
CQM starts after the cgroup subsystem, and uses the cqm_initialized_key
static key to avoid interfering with the perf cgroup logic until
propertly initialized. The cgroup_init_mutex protects the initialization.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
arch/x86/events/intel/cqm.c | 594 +++++++++++++++++++++++++++++++++++++-
arch/x86/events/intel/cqm.h | 13 +
arch/x86/include/asm/perf_event.h | 33 +++
3 files changed, 637 insertions(+), 3 deletions(-)
diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index 98a919f..f000fd0 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -35,10 +35,17 @@ static struct perf_pmu_events_attr event_attr_##v = { \
static LIST_HEAD(cache_groups);
static DEFINE_MUTEX(cqm_mutex);
+/*
+ * Synchronizes initialization of cqm with cgroups.
+ */
+static DEFINE_MUTEX(cqm_init_mutex);
+
struct monr *monr_hrchy_root;
struct pkg_data *cqm_pkgs_data[PQR_MAX_NR_PKGS];
+DEFINE_STATIC_KEY_FALSE(cqm_initialized_key);
+
static inline bool __pmonr__in_istate(struct pmonr *pmonr)
{
lockdep_assert_held(&__pkg_data(pmonr, pkg_data_lock));
@@ -69,6 +76,9 @@ static inline bool __pmonr__in_ustate(struct pmonr *pmonr)
return !pmonr->prmid && !pmonr->ancestor_pmonr;
}
+/* Whether the monr is root. Recall that the cgroups can not be root and yet
+ * point to a root monr.
+ */
static inline bool monr__is_root(struct monr *monr)
{
return monr_hrchy_root == monr;
@@ -115,6 +125,23 @@ static inline void __monr__clear_mon_active(struct monr *monr)
monr->flags &= ~MONR_MON_ACTIVE;
}
+static inline bool monr__is_cgroup_type(struct monr *monr)
+{
+ return monr->mon_cgrp;
+}
+
+static inline bool monr_is_event_type(struct monr *monr)
+{
+ return !monr->mon_cgrp && monr->mon_event_group;
+}
+
+
+static inline struct cgroup_subsys_state *get_root_perf_css(void)
+{
+ /* Get css for root cgroup */
+ return init_css_set.subsys[perf_event_cgrp_id];
+}
+
/*
* Update if enough time has passed since last read.
*
@@ -725,6 +752,7 @@ static struct monr *monr_alloc(void)
monr->parent = NULL;
INIT_LIST_HEAD(&monr->children);
INIT_LIST_HEAD(&monr->parent_entry);
+ monr->mon_cgrp = NULL;
monr->mon_event_group = NULL;
/* Iterate over all pkgs, even unitialized ones. */
@@ -947,7 +975,7 @@ retry:
}
/*
- * Wrappers for monr manipulation in events.
+ * Wrappers for monr manipulation in events and cgroups.
*
*/
static inline struct monr *monr_from_event(struct perf_event *event)
@@ -960,6 +988,100 @@ static inline void event_set_monr(struct perf_event *event, struct monr *monr)
WRITE_ONCE(event->hw.cqm_monr, monr);
}
+#ifdef CONFIG_CGROUP_PERF
+static inline struct monr *monr_from_perf_cgroup(struct perf_cgroup *cgrp)
+{
+ struct monr *monr;
+ struct cgrp_cqm_info *cqm_info;
+
+ cqm_info = (struct cgrp_cqm_info *)READ_ONCE(cgrp->arch_info);
+ WARN_ON_ONCE(!cqm_info);
+ monr = READ_ONCE(cqm_info->monr);
+ return monr;
+}
+
+static inline struct perf_cgroup *monr__get_mon_cgrp(struct monr *monr)
+{
+ WARN_ON_ONCE(!monr);
+ return READ_ONCE(monr->mon_cgrp);
+}
+
+static inline void
+monr__set_mon_cgrp(struct monr *monr, struct perf_cgroup *cgrp)
+{
+ WRITE_ONCE(monr->mon_cgrp, cgrp);
+}
+
+static inline void
+perf_cgroup_set_monr(struct perf_cgroup *cgrp, struct monr *monr)
+{
+ WRITE_ONCE(cgrp_to_cqm_info(cgrp)->monr, monr);
+}
+
+/*
+ * A perf_cgroup is monitored when it's set in a monr->mon_cgrp.
+ * There is a many-to-one relationship between perf_cgroup's monrs
+ * and monrs' mon_cgrp. A monitored cgroup is necesarily referenced
+ * back by its monr's mon_cgrp.
+ */
+static inline bool perf_cgroup_is_monitored(struct perf_cgroup *cgrp)
+{
+ struct monr *monr;
+ struct perf_cgroup *monr_cgrp;
+
+ /* monr can be referenced by a cgroup other than the one in its
+ * mon_cgrp, be careful.
+ */
+ monr = monr_from_perf_cgroup(cgrp);
+
+ monr_cgrp = monr__get_mon_cgrp(monr);
+ /* Root monr do not have a cgroup associated before initialization.
+ * mon_cgrp and mon_event_group are union, so the pointer must be set
+ * for all non-root monrs.
+ */
+ return monr_cgrp && monr__get_mon_cgrp(monr) == cgrp;
+}
+
+/* Set css's monr to the monr of its lowest monitored ancestor. */
+static inline void __css_set_monr_to_lma(struct cgroup_subsys_state *css)
+{
+ lockdep_assert_held(&cqm_mutex);
+ if (!css->parent) {
+ perf_cgroup_set_monr(css_to_perf_cgroup(css), monr_hrchy_root);
+ return;
+ }
+ perf_cgroup_set_monr(
+ css_to_perf_cgroup(css),
+ monr_from_perf_cgroup(css_to_perf_cgroup(css->parent)));
+}
+
+static inline void
+perf_cgroup_make_monitored(struct perf_cgroup *cgrp, struct monr *monr)
+{
+ monr_hrchy_assert_held_mutexes();
+ perf_cgroup_set_monr(cgrp, monr);
+ /* Make sure that monr is a valid monr for css before it's visible
+ * to any reader of css.
+ */
+ smp_wmb();
+ monr__set_mon_cgrp(monr, cgrp);
+}
+
+static inline void
+perf_cgroup_make_unmonitored(struct perf_cgroup *cgrp)
+{
+ struct monr *monr = monr_from_perf_cgroup(cgrp);
+
+ monr_hrchy_assert_held_mutexes();
+ __css_set_monr_to_lma(&cgrp->css);
+ /* Make sure that all readers of css'monr see lma css before
+ * monr stops being a valid monr for css.
+ */
+ smp_wmb();
+ monr__set_mon_cgrp(monr, NULL);
+}
+#endif
+
/*
* Always finds a rmid_entry to schedule. To be called during scheduler.
* A fast path that only uses read_lock for common case when rmid for current
@@ -1068,6 +1190,286 @@ __monr_hrchy_remove_leaf(struct monr *monr)
monr->parent = NULL;
}
+#ifdef CONFIG_CGROUP_PERF
+static struct perf_cgroup *__perf_cgroup_parent(struct perf_cgroup *cgrp)
+{
+ struct cgroup_subsys_state *parent_css = cgrp->css.parent;
+
+ if (parent_css)
+ return css_to_perf_cgroup(parent_css);
+ return NULL;
+}
+
+/* Get cgroup for both task and cgroup event. */
+static inline struct perf_cgroup *
+perf_cgroup_from_event(struct perf_event *event)
+{
+#ifdef CONFIG_LOCKDEP
+ u16 pkg_id = topology_physical_package_id(smp_processor_id());
+ bool rcu_safe = lockdep_is_held(
+ &cqm_pkgs_data[pkg_id]->pkg_data_lock);
+#endif
+
+ if (!(event->attach_state & PERF_ATTACH_TASK))
+ return event->cgrp;
+
+ return container_of(
+ task_css_check(event->hw.target, perf_event_cgrp_id, rcu_safe),
+ struct perf_cgroup, css);
+}
+
+/* Find lowest ancestor that is monitored, not including this cgrp.
+ * Return NULL if no ancestor is monitored.
+ */
+struct perf_cgroup *__cgroup_find_lma(struct perf_cgroup *cgrp)
+{
+ do {
+ cgrp = __perf_cgroup_parent(cgrp);
+ } while (cgrp && !perf_cgroup_is_monitored(cgrp));
+ return cgrp;
+}
+
+/* Similar to css_next_descendant_pre but skips the subtree rooted by pos. */
+struct cgroup_subsys_state *
+css_skip_subtree_pre(struct cgroup_subsys_state *pos,
+ struct cgroup_subsys_state *root)
+{
+ struct cgroup_subsys_state *next;
+
+ WARN_ON_ONCE(!pos);
+ while (pos != root) {
+ next = css_next_child(pos, pos->parent);
+ if (next)
+ return next;
+ pos = pos->parent;
+ }
+ return NULL;
+}
+
+/* Make all monrs of css descendants of css to depend on new_monr. */
+inline void __css_subtree_update_monrs(struct cgroup_subsys_state *css,
+ struct monr *new_monr)
+{
+ struct cgroup_subsys_state *pos_css;
+ int i;
+ unsigned long flags;
+
+ lockdep_assert_held(&cqm_mutex);
+ monr_hrchy_assert_held_mutexes();
+
+ rcu_read_lock();
+
+ /* Iterate over descendants of css in pre-order, in a way
+ * similar to css_for_each_descendant_pre, but skipping the subtrees
+ * rooted by css's with a monitored cgroup, since the elements
+ * in those subtrees do not need to be updated.
+ */
+ pos_css = css_next_descendant_pre(css, css);
+ while (pos_css) {
+ struct perf_cgroup *pos_cgrp = css_to_perf_cgroup(pos_css);
+ struct monr *pos_monr = monr_from_perf_cgroup(pos_cgrp);
+
+ /* Skip css that are not online, sync'ed with cqm_mutex. */
+ if (!(pos_css->flags & CSS_ONLINE)) {
+ pos_css = css_next_descendant_pre(pos_css, css);
+ continue;
+ }
+ /* Update descendant pos's mnor pointers to monr_parent. */
+ if (!perf_cgroup_is_monitored(pos_cgrp)) {
+ perf_cgroup_set_monr(pos_cgrp, new_monr);
+ pos_css = css_next_descendant_pre(pos_css, css);
+ continue;
+ }
+ monr_hrchy_acquire_raw_spin_locks_irq_save(flags, i);
+ pos_monr->parent = new_monr;
+ list_move_tail(&pos_monr->parent_entry, &new_monr->children);
+ monr_hrchy_release_raw_spin_locks_irq_restore(flags, i);
+ /* Dont go down the subtree in pos_css since pos_monr is the
+ * lma for all its descendants.
+ */
+ pos_css = css_skip_subtree_pre(pos_css, css);
+ }
+ rcu_read_unlock();
+}
+
+static inline int __css_start_monitoring(struct cgroup_subsys_state *css)
+{
+ struct perf_cgroup *cgrp, *cgrp_lma, *pos_cgrp;
+ struct monr *monr, *monr_parent, *pos_monr, *tmp_monr;
+ unsigned long flags;
+ int i;
+
+ lockdep_assert_held(&cqm_mutex);
+
+ /* Hold mutexes to prevent all rotation threads in all packages from
+ * messing with this.
+ */
+ monr_hrchy_acquire_mutexes();
+ cgrp = css_to_perf_cgroup(css);
+ if (WARN_ON_ONCE(perf_cgroup_is_monitored(cgrp)))
+ return -1;
+
+ /* When css is root cgroup's css, attach to the pre-existing
+ * and active root monr.
+ */
+ cgrp_lma = __cgroup_find_lma(cgrp);
+ if (!cgrp_lma) {
+ /* monr of root cgrp must be monr_hrchy_root. */
+ WARN_ON_ONCE(!monr__is_root(monr_from_perf_cgroup(cgrp)));
+ perf_cgroup_make_monitored(cgrp, monr_hrchy_root);
+ monr_hrchy_release_mutexes();
+ return 0;
+ }
+ /* The monr for the lowest monitored ancestor is direct ancestor
+ * of monr in the monr hierarchy.
+ */
+ monr_parent = monr_from_perf_cgroup(cgrp_lma);
+
+ /* Create new monr. */
+ monr = monr_alloc();
+ if (IS_ERR(monr)) {
+ monr_hrchy_release_mutexes();
+ return PTR_ERR(monr);
+ }
+
+ /* monr has no children yet so it is to be inserted in hierarchy with
+ * all its pmors in (U)state.
+ * We hold locks until monr_hrchy changes are complete, to prevent
+ * possible state transition for the pmonrs in monr while still
+ * allowing to read the prmid_summary in the scheduler path.
+ */
+ monr_hrchy_acquire_raw_spin_locks_irq_save(flags, i);
+ __monr_hrchy_insert_leaf(monr, monr_parent);
+ monr_hrchy_release_raw_spin_locks_irq_restore(flags, i);
+
+ /* Make sure monr is in hierarchy before attaching monr to cgroup. */
+ barrier();
+
+ perf_cgroup_make_monitored(cgrp, monr);
+ __css_subtree_update_monrs(css, monr);
+
+ monr_hrchy_acquire_raw_spin_locks_irq_save(flags, i);
+ /* Move task-event monrs that are descendant from css's cgroup. */
+ list_for_each_entry_safe(pos_monr, tmp_monr,
+ &monr_parent->children, parent_entry) {
+ if (!monr_is_event_type(pos_monr))
+ continue;
+ /* all events in event group must have the same cgroup.
+ * No RCU read lock necessary for task_css_check since calling
+ * inside critical section.
+ */
+ pos_cgrp = perf_cgroup_from_event(pos_monr->mon_event_group);
+ if (!cgroup_is_descendant(pos_cgrp->css.cgroup,
+ cgrp->css.cgroup))
+ continue;
+ pos_monr->parent = monr;
+ list_move_tail(&pos_monr->parent_entry, &monr->children);
+ }
+ /* Make sure monitoring starts after all monrs have moved. */
+ barrier();
+
+ __monr__set_mon_active(monr);
+ monr_hrchy_release_raw_spin_locks_irq_restore(flags, i);
+
+ monr_hrchy_release_mutexes();
+ return 0;
+}
+
+static inline int __css_stop_monitoring(struct cgroup_subsys_state *css)
+{
+ struct perf_cgroup *cgrp, *cgrp_lma;
+ struct monr *monr, *monr_parent, *pos_monr;
+ unsigned long flags;
+ int i;
+
+ lockdep_assert_held(&cqm_mutex);
+
+ monr_hrchy_acquire_mutexes();
+ cgrp = css_to_perf_cgroup(css);
+ if (WARN_ON_ONCE(!perf_cgroup_is_monitored(cgrp)))
+ return -1;
+
+ monr = monr_from_perf_cgroup(cgrp);
+
+ /* When css is root cgroup's css, detach cgroup but do not
+ * destroy monr.
+ */
+ cgrp_lma = __cgroup_find_lma(cgrp);
+ if (!cgrp_lma) {
+ /* monr of root cgrp must be monr_hrchy_root. */
+ WARN_ON_ONCE(!monr__is_root(monr_from_perf_cgroup(cgrp)));
+ perf_cgroup_make_unmonitored(cgrp);
+ monr_hrchy_release_mutexes();
+ return 0;
+ }
+ /* The monr for the lowest monitored ancestor is direct ancestor
+ * of monr in the monr hierarchy.
+ */
+ monr_parent = monr_from_perf_cgroup(cgrp_lma);
+
+ /* Lock together the transition to (U)state and clearing
+ * MONR_MON_ACTIVE to prevent prmids to return to (A)state
+ * or (I)state in between.
+ */
+ monr_hrchy_acquire_raw_spin_locks_irq_save(flags, i);
+ cqm_pkg_id_for_each_online(i)
+ __pmonr__to_ustate(monr->pmonrs[i]);
+ barrier();
+ __monr__clear_mon_active(monr);
+ monr_hrchy_release_raw_spin_locks_irq_restore(flags, i);
+
+ __css_subtree_update_monrs(css, monr_parent);
+
+
+ /*
+ * Move the children monrs that are no cgroups.
+ */
+ monr_hrchy_acquire_raw_spin_locks_irq_save(flags, i);
+
+ list_for_each_entry(pos_monr, &monr->children, parent_entry)
+ pos_monr->parent = monr_parent;
+ list_splice_tail_init(&monr->children, &monr_parent->children);
+ perf_cgroup_make_unmonitored(cgrp);
+ __monr_hrchy_remove_leaf(monr);
+
+ monr_hrchy_release_raw_spin_locks_irq_restore(flags, i);
+
+ monr_hrchy_release_mutexes();
+ monr_dealloc(monr);
+ return 0;
+}
+
+/* Attaching an event to a cgroup starts monitoring in the cgroup.
+ * If the cgroup is already monitoring, just use its pre-existing mnor.
+ */
+static int __monr_hrchy_attach_cgroup_event(struct perf_event *event,
+ struct perf_cgroup *perf_cgrp)
+{
+ struct monr *monr;
+ int ret;
+
+ lockdep_assert_held(&cqm_mutex);
+ WARN_ON_ONCE(event->attach_state & PERF_ATTACH_TASK);
+ WARN_ON_ONCE(monr_from_event(event));
+ WARN_ON_ONCE(!perf_cgrp);
+
+ if (!perf_cgroup_is_monitored(perf_cgrp)) {
+ css_get(&perf_cgrp->css);
+ ret = __css_start_monitoring(&perf_cgrp->css);
+ css_put(&perf_cgrp->css);
+ if (ret)
+ return ret;
+ }
+
+ /* At this point, cgrp is always monitored, use its monr. */
+ monr = monr_from_perf_cgroup(perf_cgrp);
+
+ event_set_monr(event, monr);
+ monr->mon_event_group = event;
+ return 0;
+}
+#endif
+
static int __monr_hrchy_attach_cpu_event(struct perf_event *event)
{
lockdep_assert_held(&cqm_mutex);
@@ -1109,12 +1511,27 @@ static int __monr_hrchy_attach_task_event(struct perf_event *event,
static int monr_hrchy_attach_event(struct perf_event *event)
{
struct monr *monr_parent;
+#ifdef CONFIG_CGROUP_PERF
+ struct perf_cgroup *perf_cgrp;
+#endif
if (!event->cgrp && !(event->attach_state & PERF_ATTACH_TASK))
return __monr_hrchy_attach_cpu_event(event);
+#ifdef CONFIG_CGROUP_PERF
+ /* Task events become leaves, cgroup events reuse the cgroup's monr */
+ if (event->cgrp)
+ return __monr_hrchy_attach_cgroup_event(event, event->cgrp);
+
+ rcu_read_lock();
+ perf_cgrp = perf_cgroup_from_event(event);
+ rcu_read_unlock();
+
+ monr_parent = monr_from_perf_cgroup(perf_cgrp);
+#else
/* Two-levels hierarchy: Root and all event monr underneath it. */
monr_parent = monr_hrchy_root;
+#endif
return __monr_hrchy_attach_task_event(event, monr_parent);
}
@@ -1126,7 +1543,7 @@ static int monr_hrchy_attach_event(struct perf_event *event)
*/
static bool __match_event(struct perf_event *a, struct perf_event *b)
{
- /* Per-cpu and task events don't mix */
+ /* Cgroup/non-task per-cpu and task events don't mix */
if ((a->attach_state & PERF_ATTACH_TASK) !=
(b->attach_state & PERF_ATTACH_TASK))
return false;
@@ -2185,6 +2602,129 @@ static struct pmu intel_cqm_pmu = {
.read = intel_cqm_event_read,
};
+#ifdef CONFIG_CGROUP_PERF
+/* XXX: Add hooks for attach dettach task with monr to a cgroup. */
+inline int perf_cgroup_arch_css_alloc(struct cgroup_subsys_state *parent_css,
+ struct cgroup_subsys_state *new_css)
+{
+ struct perf_cgroup *new_cgrp;
+ struct cgrp_cqm_info *cqm_info;
+
+ new_cgrp = css_to_perf_cgroup(new_css);
+ cqm_info = kmalloc(sizeof(struct cgrp_cqm_info), GFP_KERNEL);
+ if (!cqm_info)
+ return -ENOMEM;
+ cqm_info->cont_monitoring = false;
+ cqm_info->monr = NULL;
+ new_cgrp->arch_info = cqm_info;
+
+ return 0;
+}
+
+inline void perf_cgroup_arch_css_free(struct cgroup_subsys_state *css)
+{
+ struct perf_cgroup *cgrp = css_to_perf_cgroup(css);
+
+ kfree(cgrp_to_cqm_info(cgrp));
+ cgrp->arch_info = NULL;
+}
+
+/* Do the bulk of arch_css_online. To be called when CQM starts after
+ * css has gone online.
+ */
+static inline int __css_go_online(struct cgroup_subsys_state *css)
+{
+ lockdep_assert_held(&cqm_mutex);
+
+ /* css must not be used in monr hierarchy before having
+ * set its monr in this step.
+ */
+ __css_set_monr_to_lma(css);
+ /* Root monr is always monitoring. */
+ if (!css->parent)
+ css_to_cqm_info(css)->cont_monitoring = true;
+
+ if (css_to_cqm_info(css)->cont_monitoring)
+ return __css_start_monitoring(css);
+ return 0;
+}
+
+inline int perf_cgroup_arch_css_online(struct cgroup_subsys_state *css)
+{
+ int ret = 0;
+
+ /* use cqm_init_mutex to synchronize with
+ * __start_monitoring_all_cgroups.
+ */
+ mutex_lock(&cqm_init_mutex);
+
+ if (static_branch_unlikely(&cqm_initialized_key)) {
+ mutex_lock(&cqm_mutex);
+ ret = __css_go_online(css);
+ mutex_unlock(&cqm_mutex);
+ WARN_ON_ONCE(ret);
+ }
+
+ mutex_unlock(&cqm_init_mutex);
+ return ret;
+}
+
+inline void perf_cgroup_arch_css_offline(struct cgroup_subsys_state *css)
+{
+ int ret = 0;
+ struct monr *monr;
+ struct perf_cgroup *cgrp = css_to_perf_cgroup(css);
+
+ mutex_lock(&cqm_init_mutex);
+
+ if (!static_branch_unlikely(&cqm_initialized_key))
+ goto out;
+
+ mutex_lock(&cqm_mutex);
+
+ monr = monr_from_perf_cgroup(cgrp);
+ if (!perf_cgroup_is_monitored(cgrp))
+ goto out_cqm;
+
+ /* Stop monitoring for the css's monr only if no more events need it.
+ * If events need the monr, it will be destroyed when the events that
+ * use it are destroyed.
+ */
+ if (monr->mon_event_group) {
+ monr_hrchy_acquire_mutexes();
+ perf_cgroup_make_unmonitored(cgrp);
+ monr_hrchy_release_mutexes();
+ } else {
+ ret = __css_stop_monitoring(css);
+ WARN_ON_ONCE(ret);
+ }
+
+out_cqm:
+ mutex_unlock(&cqm_mutex);
+out:
+ mutex_unlock(&cqm_init_mutex);
+ WARN_ON_ONCE(ret);
+}
+
+inline void perf_cgroup_arch_css_released(struct cgroup_subsys_state *css)
+{
+ mutex_lock(&cqm_init_mutex);
+
+ if (static_branch_unlikely(&cqm_initialized_key)) {
+ mutex_lock(&cqm_mutex);
+ /*
+ * Remove css from monr hierarchy now that css is about to
+ * leave the cgroup hierarchy.
+ */
+ perf_cgroup_set_monr(css_to_perf_cgroup(css), NULL);
+ mutex_unlock(&cqm_mutex);
+ }
+
+ mutex_unlock(&cqm_init_mutex);
+}
+
+#endif
+
static inline void cqm_pick_event_reader(int cpu)
{
u16 pkg_id = topology_physical_package_id(cpu);
@@ -2249,6 +2789,39 @@ static const struct x86_cpu_id intel_cqm_match[] = {
{}
};
+#ifdef CONFIG_CGROUP_PERF
+/* Start monitoring for all cgroups in cgroup hierarchy. */
+static int __start_monitoring_all_cgroups(void)
+{
+ int ret;
+ struct cgroup_subsys_state *css, *css_root;
+
+ lockdep_assert_held(&cqm_init_mutex);
+
+ rcu_read_lock();
+ /* Get css for root cgroup */
+ css_root = get_root_perf_css();
+
+ css_for_each_descendant_pre(css, css_root) {
+ if (!css_tryget_online(css))
+ continue;
+
+ rcu_read_unlock();
+ mutex_lock(&cqm_mutex);
+ ret = __css_go_online(css);
+ mutex_unlock(&cqm_mutex);
+
+ css_put(css);
+ if (ret)
+ return ret;
+
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+ return 0;
+}
+#endif
+
static int __init intel_cqm_init(void)
{
char *str, scale[20];
@@ -2324,17 +2897,32 @@ static int __init intel_cqm_init(void)
__perf_cpu_notifier(intel_cqm_cpu_notifier);
+ /* Use cqm_init_mutex to synchronize with css's online/offline. */
+ mutex_lock(&cqm_init_mutex);
+
+#ifdef CONFIG_CGROUP_PERF
+ ret = __start_monitoring_all_cgroups();
+ if (ret)
+ goto error_init_mutex;
+#endif
+
ret = perf_pmu_register(&intel_cqm_pmu, "intel_cqm", -1);
if (ret)
- goto error;
+ goto error_init_mutex;
cpu_notifier_register_done();
+ static_branch_enable(&cqm_initialized_key);
+
+ mutex_unlock(&cqm_init_mutex);
+
pr_info("Intel CQM monitoring enabled with at least %u rmids per package.\n",
min_max_rmid + 1);
return ret;
+error_init_mutex:
+ mutex_unlock(&cqm_init_mutex);
error:
pr_err("Intel CQM perf registration failed: %d\n", ret);
cpu_notifier_register_done();
diff --git a/arch/x86/events/intel/cqm.h b/arch/x86/events/intel/cqm.h
index 25646a2..0f3da94 100644
--- a/arch/x86/events/intel/cqm.h
+++ b/arch/x86/events/intel/cqm.h
@@ -313,6 +313,7 @@ struct pkg_data {
* struct monr: MONitored Resource.
* @flags: Flags field for monr (XXX: More flags will be added
* with MBM).
+ * @mon_cgrp: The cgroup associated with this monr, if any
* @mon_event_group: The head of event's group that use this monr, if any.
* @parent: Parent in monr hierarchy.
* @children: List of children in monr hierarchy.
@@ -333,6 +334,7 @@ struct pkg_data {
struct monr {
u16 flags;
/* Back reference pointers */
+ struct perf_cgroup *mon_cgrp;
struct perf_event *mon_event_group;
struct monr *parent;
@@ -506,3 +508,14 @@ static unsigned int __cqm_min_progress_rate = CQM_DEFAULT_MIN_PROGRESS_RATE;
* It's units are bytes must be scaled by cqm_l3_scale to obtain cache lines.
*/
static unsigned int __intel_cqm_max_threshold;
+
+
+struct cgrp_cqm_info {
+ /* Should the cgroup be continuously monitored? */
+ bool cont_monitoring;
+ struct monr *monr;
+};
+
+# define css_to_perf_cgroup(css_) container_of(css_, struct perf_cgroup, css)
+# define cgrp_to_cqm_info(cgrp_) ((struct cgrp_cqm_info *)cgrp_->arch_info)
+# define css_to_cqm_info(css_) cgrp_to_cqm_info(css_to_perf_cgroup(css_))
diff --git a/arch/x86/include/asm/perf_event.h b/arch/x86/include/asm/perf_event.h
index f353061..c22d9e0 100644
--- a/arch/x86/include/asm/perf_event.h
+++ b/arch/x86/include/asm/perf_event.h
@@ -299,4 +299,37 @@ static inline void perf_check_microcode(void) { }
#define arch_perf_out_copy_user copy_from_user_nmi
+
+/*
+ * Hooks for architecture specific features of perf_event cgroup.
+ * Currently used by Intel's CQM.
+ */
+#ifdef CONFIG_INTEL_RDT
+#define perf_cgroup_arch_css_alloc \
+ perf_cgroup_arch_css_alloc
+inline int perf_cgroup_arch_css_alloc(struct cgroup_subsys_state *parent_css,
+ struct cgroup_subsys_state *new_css);
+
+#define perf_cgroup_arch_css_online \
+ perf_cgroup_arch_css_online
+inline int perf_cgroup_arch_css_online(struct cgroup_subsys_state *css);
+
+#define perf_cgroup_arch_css_offline \
+ perf_cgroup_arch_css_offline
+inline void perf_cgroup_arch_css_offline(struct cgroup_subsys_state *css);
+
+#define perf_cgroup_arch_css_released \
+ perf_cgroup_arch_css_released
+inline void perf_cgroup_arch_css_released(struct cgroup_subsys_state *css);
+
+#define perf_cgroup_arch_css_free \
+ perf_cgroup_arch_css_free
+inline void perf_cgroup_arch_css_free(struct cgroup_subsys_state *css);
+
+#else
+
+#define PERF_CGROUP_ARCH_CGRP_SUBSYS_ATTS
+
+#endif
+
#endif /* _ASM_X86_PERF_EVENT_H */
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 07:00 +0200 |
| Subject | [PATCH 17/32] perf/core: adding pmu::event_terminate |
| Message-ID | <rt8H1-8vJ-25@gated-at.bofh.it> |
| In reply to | #1390741 |
Allow a PMU to clean an event before the event's torn down in
perf_events begins.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
include/linux/perf_event.h | 6 ++++++
kernel/events/core.c | 4 ++++
2 files changed, 10 insertions(+)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index b010b55..81e29c6 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -265,6 +265,12 @@ struct pmu {
int (*event_init) (struct perf_event *event);
/*
+ * Terminate the event for this PMU. Optional complement for a
+ * successful event_init. Called before the event fields are tear down.
+ */
+ void (*event_terminate) (struct perf_event *event);
+
+ /*
* Notification that the event was mapped or unmapped. Called
* in the context of the mapping task.
*/
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 6fd226f..2a868a6 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -3787,6 +3787,8 @@ static void _free_event(struct perf_event *event)
ring_buffer_attach(event, NULL);
mutex_unlock(&event->mmap_mutex);
}
+ if (event->pmu->event_terminate)
+ event->pmu->event_terminate(event);
if (is_cgroup_event(event))
perf_detach_cgroup(event);
@@ -8293,6 +8295,8 @@ err_per_task:
exclusive_event_destroy(event);
err_pmu:
+ if (event->pmu->event_terminate)
+ event->pmu->event_terminate(event);
if (event->destroy)
event->destroy(event);
module_put(pmu->module);
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
| From | David Carrillo-Cisneros <davidcc@google.com> |
|---|---|
| Date | 2016-04-29 07:00 +0200 |
| Subject | [PATCH 05/32] perf/core: remove unused pmu->count |
| Message-ID | <rt8H0-8vJ-21@gated-at.bofh.it> |
| In reply to | #1390741 |
CQM was the only user of pmu->count, no need to have it anymore.
Reviewed-by: Stephane Eranian <eranian@google.com>
Signed-off-by: David Carrillo-Cisneros <davidcc@google.com>
---
include/linux/perf_event.h | 6 ------
kernel/events/core.c | 10 ----------
kernel/trace/bpf_trace.c | 5 ++---
3 files changed, 2 insertions(+), 19 deletions(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 00bb6b5..8bb1532 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -373,12 +373,6 @@ struct pmu {
*/
size_t task_ctx_size;
-
- /*
- * Return the count value for a counter.
- */
- u64 (*count) (struct perf_event *event); /*optional*/
-
/*
* Set up pmu-private data structures for an AUX area
*/
diff --git a/kernel/events/core.c b/kernel/events/core.c
index aae72d3..4aaec01 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -3313,9 +3313,6 @@ unlock:
static inline u64 perf_event_count(struct perf_event *event)
{
- if (event->pmu->count)
- return event->pmu->count(event);
-
return __perf_event_count(event);
}
@@ -3325,7 +3322,6 @@ static inline u64 perf_event_count(struct perf_event *event)
* - either for the current task, or for this CPU
* - does not have inherit set, for inherited task events
* will not be local and we cannot read them atomically
- * - must not have a pmu::count method
*/
u64 perf_event_read_local(struct perf_event *event)
{
@@ -3353,12 +3349,6 @@ u64 perf_event_read_local(struct perf_event *event)
WARN_ON_ONCE(event->attr.inherit);
/*
- * It must not have a pmu::count method, those are not
- * NMI safe.
- */
- WARN_ON_ONCE(event->pmu->count);
-
- /*
* If the event is currently on this CPU, its either a per-task event,
* or local to this CPU. Furthermore it means its ACTIVE (otherwise
* oncpu == -1).
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 3e4ffb3..7ef81b3 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -200,9 +200,8 @@ static u64 bpf_perf_event_read(u64 r1, u64 index, u64 r3, u64 r4, u64 r5)
event = file->private_data;
- /* make sure event is local and doesn't have pmu::count */
- if (event->oncpu != smp_processor_id() ||
- event->pmu->count)
+ /* make sure event is local */
+ if (event->oncpu != smp_processor_id())
return -EINVAL;
/*
--
2.8.0.rc3.226.g39d4020
[toc] | [prev] | [next] | [standalone]
Page 1 of 2 [1] 2 Next page →
Back to top | Article view | linux.kernel
csiph-web