Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1414578 > unrolled thread

[RFC v2 0/4] ARM LPAE Outer Shared v2

Back to article view | Back to linux.kernel

Contents

  [RFC v2 0/4] ARM LPAE Outer Shared v2 Bill Mills <wmills@ti.com> - 2016-06-06 05:30 +0200
    [RFC v2 1/4] ARM: mm: add early page table attribute modification ability Bill Mills <wmills@ti.com> - 2016-06-06 05:30 +0200
      Re: [RFC v2 1/4] ARM: mm: add early page table attribute  modification ability Russell King - ARM Linux <linux@armlinux.org.uk> - 2016-06-06 14:20 +0200
        Re: [RFC v2 1/4] ARM: mm: add early page table attribute modification  ability William Mills <wmills@ti.com> - 2016-06-06 14:40 +0200
    [RFC v2 2/4] ARM: mm: Add LPAE support for outer shared Bill Mills <wmills@ti.com> - 2016-06-06 05:30 +0200

#1414578 — [RFC v2 0/4] ARM LPAE Outer Shared v2

This RFC series adds support for outer shared LPAE page table
attributes. This attribute is needed by at least keystone to achieve
dma coherency. The choice is done at early boot time and can co-exist
with other platforms that want only inner shared.

v2 addresses the concern about changing the memory attributes while the
MMU is on that was raised in v1. It also puts the primary
responsibility of choosing the right mode on the platform.

Instead of creating an "need outer shared flag" to the pv_fixup code, I
created a generic attribute modification mechanism. The idea was it
could be used to solve other problems where the assumptions of the
early boot tables need to be changed in a safe manner. Right now it is
LPAE only and tied 1:1 with pv_fixup but that could change. I did test
that applying a 0 pv_fixup seemed to do no harm.

There is a patch that adds an early param "defshared". This is a
separate patch as I am unsure if this is really desired. It is useful
for testing the series however. You can use it to force keystone to
use inner shared (and it will fallback to non-coherent dma-ops) or you
can use it to force another platform to use outer shared and see what
happens. If we keep the param, documentation will be added.

This series needs more testing and finishing but I wanted to get a read
on the direction. This does run on Keystone and for QEMU vexpress-A15.
QEMU vexpress runs with inner or outer shared :)
Multiple TODO points marked in-line. If the approach is accepted I
will complete the TODO items and TI will do more testing.

Series based on V4.7-rc2

v1 was here:
http://marc.info/?t=146044908600005&r=1&w=2

-- Bill

[toc] | [next] | [standalone]

#1414587 — [RFC v2 1/4] ARM: mm: add early page table attribute modification ability

Allow early-init to specify modifications to be made to the boot time page
table. Any modifications specified will be done with MMU off at the same
time that any Phy<->Virt fixup is done.

This ability is enabled with ARM_PV_FIXUP.

It is currently only implemented for LPAE mode.

Signed-off-by: Bill Mills <wmills@ti.com>
---
 arch/arm/include/asm/pgtable-hwdef.h | 21 +++++++++
 arch/arm/mm/mmu.c                    | 36 ++++++++++++---
 arch/arm/mm/pv-fixup-asm.S           | 86 ++++++++++++++++++++++++++++++++++--
 3 files changed, 135 insertions(+), 8 deletions(-)

diff --git a/arch/arm/include/asm/pgtable-hwdef.h b/arch/arm/include/asm/pgtable-hwdef.h
index 8426229..c35d71f 100644
--- a/arch/arm/include/asm/pgtable-hwdef.h
+++ b/arch/arm/include/asm/pgtable-hwdef.h
@@ -16,4 +16,25 @@
 #include <asm/pgtable-2level-hwdef.h>
 #endif
 
+#ifdef CONFIG_ARM_PV_FIXUP
+
+#define MAX_ATTR_MOD_ENTRIES	64
+
+#ifndef __ASSEMBLY__
+
+struct attr_mod_entry {
+	pmdval_t	test_mask;
+	pmdval_t	test_value;
+	pmdval_t	clear_mask;
+	pmdval_t	set_mask;
+};
+
+bool attr_mod_add(struct attr_mod_entry *pmod);
+
+extern int num_attr_mods;
+extern struct attr_mod_entry attr_mod_table[MAX_ATTR_MOD_ENTRIES];
+
+#endif	/* __ASSEMBLY__ */
+#endif	/* CONFIG_ARM_PV_FIXUP */
+
 #endif
diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
index 62f4d01..a608980 100644
--- a/arch/arm/mm/mmu.c
+++ b/arch/arm/mm/mmu.c
@@ -1496,23 +1496,41 @@ extern unsigned long __atags_pointer;
 typedef void pgtables_remap(long long offset, unsigned long pgd, void *bdata);
 pgtables_remap lpae_pgtables_remap_asm;
 
+int num_attr_mods;
+
+/* add an entry to the early page table attribute modification list */
+bool __init attr_mod_add(struct attr_mod_entry *pmod)
+{
+	if (num_attr_mods >= MAX_ATTR_MOD_ENTRIES) {
+		pr_crit("Out of room for (or late use of) early page table attribute modifications.\n");
+		return false;
+	}
+
+	attr_mod_table[num_attr_mods++] = *pmod;
+	return true;
+}
+
 /*
  * early_paging_init() recreates boot time page table setup, allowing machines
  * to switch over to a high (>4G) address space on LPAE systems
+ *
+ * This function also applies any attribute modifications specified in
+ * attr_mod_table.  These may have been added before we got here (early_param)
+ * or from within mdesc->pv_fixup called by this function
  */
 void __init early_paging_init(const struct machine_desc *mdesc)
 {
 	pgtables_remap *lpae_pgtables_remap;
 	unsigned long pa_pgd;
 	unsigned int cr, ttbcr;
-	long long offset;
+	long long offset = 0;
 	void *boot_data;
+	unsigned long pmd;
 
-	if (!mdesc->pv_fixup)
-		return;
+	if (mdesc->pv_fixup)
+		offset = mdesc->pv_fixup();
 
-	offset = mdesc->pv_fixup();
-	if (offset == 0)
+	if (offset == 0 && num_attr_mods == 0)
 		return;
 
 	/*
@@ -1564,6 +1582,14 @@ void __init early_paging_init(const struct machine_desc *mdesc)
 	/* Re-enable the caches and cacheable TLB walks */
 	asm volatile("mcr p15, 0, %0, c2, c0, 2" : : "r" (ttbcr));
 	set_cr(cr);
+
+	/* disable any further use of attribute fixup */
+	num_attr_mods = MAX_ATTR_MOD_ENTRIES + 1;
+
+	/* record the new "initial" pmd and cachepolicy */
+	pmd = pmd_val(*pmd_off_k((unsigned long)_data));
+	pmd &= ~PMD_MASK;
+	init_default_cache_policy(pmd);
 }
 
 #else
diff --git a/arch/arm/mm/pv-fixup-asm.S b/arch/arm/mm/pv-fixup-asm.S
index 1867f3e4..ad8edc2 100644
--- a/arch/arm/mm/pv-fixup-asm.S
+++ b/arch/arm/mm/pv-fixup-asm.S
@@ -19,8 +19,44 @@
 #define L1_ORDER 3
 #define L2_ORDER 3
 
+/*
+ *	attr_mod_table:
+ *		describe transforms to be made to the early boot pgtable
+ *		This is poked by early init code
+ *	mod descriptor list:
+ *		64 bit test mask
+ *		64 bit test value
+ *		64 bit clear mask
+ *		64 bit set mask
+ *      	next descriptor
+ *		...
+ *		0x0000_00000 0x0000_0000 end of list
+ */
+/* TODO: what segment?, test w/ XIP kernel? */
+	 .globl attr_mod_table
+attr_mod_table:
+	.zero   8*MAX_ATTR_MOD_ENTRIES*4 + 1
+
+/*
+ *	lpae_pgtables_remap_asm(long long offset, unsigned long pg,
+ *		void* boot_data)
+ *
+ *	Rewrite initial boot page tables with new physical addresses and or
+ *	attributes.
+ *	This function starts in identity mapped VA -> low PA
+ *	The body runs in low PA with MMU off
+ * 	The function ends in "identity mapped" VA -> high PA
+ *	The function returns to kernel VA space -> high PA
+ *
+ *	- r0    PA delta low
+ *	- r1	PA delta high
+ *	- r2    address of top level table
+ *	- r3    address of dtb (or atags))
+ *
+ *	uses null terminated list of attribute modifications in attr_mod_table
+ */
 ENTRY(lpae_pgtables_remap_asm)
-	stmfd	sp!, {r4-r8, lr}
+	stmfd	sp!, {r4-r11, lr}
 
 	mrc	p15, 0, r8, c1, c0, 0		@ read control reg
 	bic	ip, r8, #CR_M			@ disable caches and MMU
@@ -63,6 +99,7 @@ ENTRY(lpae_pgtables_remap_asm)
 	subs	r6, r6, #1
 	bne	2b
 
+	/* Update HW page table regs with new PA */
 	mrrc	p15, 0, r4, r5, c2		@ read TTBR0
 	adds	r4, r4, r0			@ update physical address
 	adc	r5, r5, r1
@@ -74,15 +111,58 @@ ENTRY(lpae_pgtables_remap_asm)
 
 	dsb
 
+	/* Update attributes of all level 2 entries in 1GB space */
+	/* TODO: fix/test BE8 THUMB2 kernel */
+	adrl	r3, attr_mod_table
+	add	r7, r2, #0x1000
+	add	r6, r7, #0x4000
+	bl	3f				@ NOT C ABI
+
+	/* Update attributes of the 4 level 1 entries */
+	/* TODO: delete this or allow mod entries to match only L1 */
+	mov	r7, r2
+	add	r6, r7, #32
+	bl	3f				@ NOT C ABI
+	b	7f
+
+3:	ldrd	r4, [r7]
+	orrs	r11, r4, r5
+	beq	6f				@ skip unused entries
+	mov 	r10, r3
+4:	ldrd	r8, [r10]
+	orrs	r11, r8, r9
+	beq	6f				@ end of mod table?
+	and	r0, r4, r8			@ no, load test mask
+	and	r1, r5, r9
+	ldrd	r8, [r10, #8]			@ load test bits
+	cmp	r0, r8
+	cmpeq	r1, r9
+	bne	5f				@ does entry match desc?
+	ldrd	r8, [r10, #16]			@ yes, load mod clear mask
+	bic	r4, r4, r8
+	bic	r5, r5, r9
+	ldrd	r8, [r10, #24]			@ load mod set mask
+	orr	r4, r4, r8
+	orr	r5, r5, r9
+5:	add     r10, r10, #32			@ try next mod desc
+	b	4b
+6:	strd	r4, [r7], #1 << L2_ORDER
+	cmp	r7, r6
+	bls	3b
+	bx 	lr
+
+7:
 	mov	ip, #0
 	mcr	p15, 0, ip, c7, c5, 0		@ I+BTB cache invalidate
 	mcr	p15, 0, ip, c8, c7, 0		@ local_flush_tlb_all()
 	dsb
 	isb
 
-	mcr	p15, 0, r8, c1, c0, 0		@ re-enable MMU
+	mrc	p15, 0, r8, c1, c0, 0		@ re-enable MMU
+	orr	r8, r8, #CR_M
+	mcr	p15, 0, r8, c1, c0, 0
 	dsb
 	isb
 
-	ldmfd	sp!, {r4-r8, pc}
+	ldmfd	sp!, {r4-r11, pc}
 ENDPROC(lpae_pgtables_remap_asm)
-- 
1.9.1

[toc] | [prev] | [next] | [standalone]

#1414971 — Re: [RFC v2 1/4] ARM: mm: add early page table attribute modification ability

On Sun, Jun 05, 2016 at 11:20:26PM -0400, Bill Mills wrote:
> Allow early-init to specify modifications to be made to the boot time page
> table. Any modifications specified will be done with MMU off at the same
> time that any Phy<->Virt fixup is done.

I think this is rather over-engineered - do we need to support multiple
different fixups to the page tables like this?

Given how this has grown, I think it would be better to duplicate the
existing swapper_pg_dir, modify the new copy, and then have the
pv-fixup-asm code merely copy the new to the old with the MMU off.
That way, the only two things that the assembly code has to do is to
deal with the page table update, and updating the TTBR registers.
Most of the complexity can then be kept in the C code.

I think we also need to modify the TTBCR to match the sharability of
memory - currently, TTB walks will be inner sharable, but my
understanding is that if we switch memory to be outer sharable, we
also need to update the TTB walks to match.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1414991 — Re: [RFC v2 1/4] ARM: mm: add early page table attribute modification ability

On 06/06/2016 08:18 AM, Russell King - ARM Linux wrote:
> On Sun, Jun 05, 2016 at 11:20:26PM -0400, Bill Mills wrote:
>> Allow early-init to specify modifications to be made to the boot time page
>> table. Any modifications specified will be done with MMU off at the same
>> time that any Phy<->Virt fixup is done.
> 
> I think this is rather over-engineered - do we need to support multiple
> different fixups to the page tables like this?

Yes I was expecting this comment but thought I would give you the choice. :)

> 
> Given how this has grown, I think it would be better to duplicate the
> existing swapper_pg_dir, modify the new copy, and then have the
> pv-fixup-asm code merely copy the new to the old with the MMU off.
> That way, the only two things that the assembly code has to do is to
> deal with the page table update, and updating the TTBR registers.
> Most of the complexity can then be kept in the C code.
> 

I really like this.  I can just do the outer shared fixup and not worry
about a generalized mechanism.  *If* someone needs to do another fixup
they can just code it in C.

The new patch #1 will just rework the PV_FIXUP for the new asm/C split.

You want the off-line table to copy over the early table in place w/ MMU
off, correct? (Not update the HW to point to a new spot.)

> I think we also need to modify the TTBCR to match the sharability of
> memory - currently, TTB walks will be inner sharable, but my
> understanding is that if we switch memory to be outer sharable, we
> also need to update the TTB walks to match.
> 

Good point, Thanks.  I don't think our internal hack has been doing that.

[toc] | [prev] | [next] | [standalone]

#1414588 — [RFC v2 2/4] ARM: mm: Add LPAE support for outer shared

Support early init selection of inner or outer shared page table
attributes.

In LPAE shared is 3 valued: non-shared, inner-shared, and outer-shared.
Provide a mask and both shared values.  Shared value in use is stored in
variables.  The old constants are eliminated to avoid accidental use.

Early page tables and variables are initialized to inner shared.

If a platform needs outer shared, it calls use_outer_shared()
during early_paging_init.  The variables and early page table are
fixed up. The mem_types built during paging_init are fixed to match
the value in effect.

No functional change for non-LPAE.  We only add a few extra aliases for
existing constants and use some extra vars at boot.

This patch is based in part on an earlier RFC patch by
    Tero Kristo <t-kristo@ti.com>

Signed-off-by: Bill Mills <wmills@ti.com>
---
 arch/arm/include/asm/pgtable-2level-hwdef.h |  6 +++
 arch/arm/include/asm/pgtable-3level-hwdef.h | 14 ++++-
 arch/arm/include/asm/pgtable-3level.h       |  2 +-
 arch/arm/include/asm/pgtable-hwdef.h        |  1 +
 arch/arm/include/asm/pgtable.h              |  3 ++
 arch/arm/mm/dump.c                          | 28 ++++++++++
 arch/arm/mm/mmu.c                           | 80 +++++++++++++++++++++++------
 arch/arm/mm/proc-v7-3level.S                |  2 +-
 8 files changed, 115 insertions(+), 21 deletions(-)

diff --git a/arch/arm/include/asm/pgtable-2level-hwdef.h b/arch/arm/include/asm/pgtable-2level-hwdef.h
index d0131ee..d62e20f 100644
--- a/arch/arm/include/asm/pgtable-2level-hwdef.h
+++ b/arch/arm/include/asm/pgtable-2level-hwdef.h
@@ -93,4 +93,10 @@
 
 #define PHYS_MASK		(~0UL)
 
+/* These are here to share more code between 2level & 3level */
+#define L_PTE_EARLY_SHARED	PTE_EXT_SHARED
+#define PTE_EXT_SMASK		PTE_EXT_SHARED
+#define PMD_SECT_EARLY_S	PMD_SECT_S
+#define PMD_SECT_SMASK		PMD_SECT_S
+
 #endif
diff --git a/arch/arm/include/asm/pgtable-3level-hwdef.h b/arch/arm/include/asm/pgtable-3level-hwdef.h
index f8f1cff..3ffc0ce 100644
--- a/arch/arm/include/asm/pgtable-3level-hwdef.h
+++ b/arch/arm/include/asm/pgtable-3level-hwdef.h
@@ -44,7 +44,9 @@
 #define PMD_SECT_CACHEABLE	(_AT(pmdval_t, 1) << 3)
 #define PMD_SECT_USER		(_AT(pmdval_t, 1) << 6)		/* AP[1] */
 #define PMD_SECT_AP2		(_AT(pmdval_t, 1) << 7)		/* read only */
-#define PMD_SECT_S		(_AT(pmdval_t, 3) << 8)
+#define PMD_SECT_SMASK		(_AT(pmdval_t, 3) << 8)		/* shareable bits */
+#define PMD_SECT_ISHARED	(_AT(pmdval_t, 3) << 8)		/* inner sharable */
+#define PMD_SECT_OSHARED	(_AT(pmdval_t, 2) << 8)		/* outer sharable */
 #define PMD_SECT_AF		(_AT(pmdval_t, 1) << 10)
 #define PMD_SECT_nG		(_AT(pmdval_t, 1) << 11)
 #define PMD_SECT_PXN		(_AT(pmdval_t, 1) << 53)
@@ -73,12 +75,20 @@
 #define PTE_BUFFERABLE		(_AT(pteval_t, 1) << 2)		/* AttrIndx[0] */
 #define PTE_CACHEABLE		(_AT(pteval_t, 1) << 3)		/* AttrIndx[1] */
 #define PTE_AP2			(_AT(pteval_t, 1) << 7)		/* AP[2] */
-#define PTE_EXT_SHARED		(_AT(pteval_t, 3) << 8)		/* SH[1:0], inner shareable */
+#define PTE_EXT_SMASK		(_AT(pteval_t, 3) << 8)		/* SH[1:0], shareable */
+#define PTE_EXT_ISHARED	(_AT(pteval_t, 3) << 8)		/* SH[1:0], inner shareable */
+#define PTE_EXT_OSHARED	(_AT(pteval_t, 2) << 8)		/* SH[1:0], outer shareable */
 #define PTE_EXT_AF		(_AT(pteval_t, 1) << 10)	/* Access Flag */
 #define PTE_EXT_NG		(_AT(pteval_t, 1) << 11)	/* nG */
 #define PTE_EXT_PXN		(_AT(pteval_t, 1) << 53)	/* PXN */
 #define PTE_EXT_XN		(_AT(pteval_t, 1) << 54)	/* XN */
 
+/* in early boot we assume inner shared,
+ * afterward use L_PTE_SHARED but only in code, can't be static initializer
+ */
+#define L_PTE_EARLY_SHARED	PTE_EXT_ISHARED
+#define PMD_SECT_EARLY_S	PMD_SECT_ISHARED
+
 /*
  * 40-bit physical address supported.
  */
diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h
index fa70db7..af5b9cb 100644
--- a/arch/arm/include/asm/pgtable-3level.h
+++ b/arch/arm/include/asm/pgtable-3level.h
@@ -78,7 +78,7 @@
 #define L_PTE_VALID		(_AT(pteval_t, 1) << 0)		/* Valid */
 #define L_PTE_PRESENT		(_AT(pteval_t, 3) << 0)		/* Present */
 #define L_PTE_USER		(_AT(pteval_t, 1) << 6)		/* AP[1] */
-#define L_PTE_SHARED		(_AT(pteval_t, 3) << 8)		/* SH[1:0], inner shareable */
+#define L_PTE_SHARED		(l_pte_shared)			/* inner or outer shareable */
 #define L_PTE_YOUNG		(_AT(pteval_t, 1) << 10)	/* AF */
 #define L_PTE_XN		(_AT(pteval_t, 1) << 54)	/* XN */
 #define L_PTE_DIRTY		(_AT(pteval_t, 1) << 55)
diff --git a/arch/arm/include/asm/pgtable-hwdef.h b/arch/arm/include/asm/pgtable-hwdef.h
index c35d71f..27654a9 100644
--- a/arch/arm/include/asm/pgtable-hwdef.h
+++ b/arch/arm/include/asm/pgtable-hwdef.h
@@ -30,6 +30,7 @@ struct attr_mod_entry {
 };
 
 bool attr_mod_add(struct attr_mod_entry *pmod);
+bool use_outer_shared(void);
 
 extern int num_attr_mods;
 extern struct attr_mod_entry attr_mod_table[MAX_ATTR_MOD_ENTRIES];
diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h
index 348caab..4d2e412 100644
--- a/arch/arm/include/asm/pgtable.h
+++ b/arch/arm/include/asm/pgtable.h
@@ -86,6 +86,9 @@ extern pgprot_t		pgprot_hyp_device;
 extern pgprot_t		pgprot_s2;
 extern pgprot_t		pgprot_s2_device;
 
+extern pmdval_t		pmd_sect_s;
+extern pteval_t		l_pte_shared;
+
 #define _MOD_PROT(p, b)	__pgprot(pgprot_val(p) | (b))
 
 #define PAGE_NONE		_MOD_PROT(pgprot_user, L_PTE_XN | L_PTE_RDONLY | L_PTE_NONE)
diff --git a/arch/arm/mm/dump.c b/arch/arm/mm/dump.c
index 9fe8e24..fb98fa6 100644
--- a/arch/arm/mm/dump.c
+++ b/arch/arm/mm/dump.c
@@ -68,10 +68,24 @@ static const struct prot_bits pte_bits[] = {
 		.set	= "NX",
 		.clear	= "x ",
 	}, {
+#ifndef CONFIG_ARM_LPAE
 		.mask	= L_PTE_SHARED,
 		.val	= L_PTE_SHARED,
 		.set	= "SHD",
 		.clear	= "   ",
+#else
+		.mask	= PTE_EXT_SMASK,
+		.val	= PTE_EXT_ISHARED,
+		.set	= "ISHD",
+	}, {
+		.mask	= PTE_EXT_SMASK,
+		.val	= PTE_EXT_OSHARED,
+		.set	= "OSHD",
+	}, {
+		.mask	= PTE_EXT_SMASK,
+		.val	= 0,
+		.set	= "    ",
+#endif
 	}, {
 		.mask	= L_PTE_MT_MASK,
 		.val	= L_PTE_MT_UNCACHED,
@@ -172,10 +186,24 @@ static const struct prot_bits section_bits[] = {
 		.set	= "NX",
 		.clear	= "x ",
 	}, {
+#ifndef CONFIG_ARM_LPAE
 		.mask	= PMD_SECT_S,
 		.val	= PMD_SECT_S,
 		.set	= "SHD",
 		.clear	= "   ",
+#else
+		.mask	= PMD_SECT_SMASK,
+		.val	= PMD_SECT_ISHARED,
+		.set	= "ISHD",
+	}, {
+		.mask	= PMD_SECT_SMASK,
+		.val	= PMD_SECT_OSHARED,
+		.set	= "OSHD",
+	}, {
+		.mask	= PMD_SECT_SMASK,
+		.val	= 0,
+		.set	= "    ",
+#endif
 	},
 };
 
diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
index a608980..8aaccf2 100644
--- a/arch/arm/mm/mmu.c
+++ b/arch/arm/mm/mmu.c
@@ -70,6 +70,13 @@ pgprot_t pgprot_hyp_device;
 pgprot_t pgprot_s2;
 pgprot_t pgprot_s2_device;
 
+/* For LPAE hold the value of Inner or Outer Shared attribute selected at
+ * early init, which starts out as inner shared
+ * For non_LPAE these are always just the single S Bit
+ */
+pmdval_t pmd_sect_s    = PMD_SECT_EARLY_S;
+pteval_t l_pte_shared  = L_PTE_EARLY_SHARED;
+
 EXPORT_SYMBOL(pgprot_user);
 EXPORT_SYMBOL(pgprot_kernel);
 
@@ -246,12 +253,12 @@ __setup("noalign", noalign_setup);
 static struct mem_type mem_types[] = {
 	[MT_DEVICE] = {		  /* Strongly ordered / ARMv6 shared device */
 		.prot_pte	= PROT_PTE_DEVICE | L_PTE_MT_DEV_SHARED |
-				  L_PTE_SHARED,
+				  L_PTE_EARLY_SHARED,
 		.prot_pte_s2	= s2_policy(PROT_PTE_S2_DEVICE) |
 				  s2_policy(L_PTE_S2_MT_DEV_SHARED) |
-				  L_PTE_SHARED,
+				  L_PTE_EARLY_SHARED,
 		.prot_l1	= PMD_TYPE_TABLE,
-		.prot_sect	= PROT_SECT_DEVICE | PMD_SECT_S,
+		.prot_sect	= PROT_SECT_DEVICE | PMD_SECT_EARLY_S,
 		.domain		= DOMAIN_IO,
 	},
 	[MT_DEVICE_NONSHARED] = { /* ARMv6 non-shared device */
@@ -340,8 +347,9 @@ static struct mem_type mem_types[] = {
 		.prot_pte  = L_PTE_PRESENT | L_PTE_YOUNG | L_PTE_DIRTY |
 				L_PTE_MT_UNCACHED | L_PTE_XN,
 		.prot_l1   = PMD_TYPE_TABLE,
-		.prot_sect = PMD_TYPE_SECT | PMD_SECT_AP_WRITE | PMD_SECT_S |
-				PMD_SECT_UNCACHED | PMD_SECT_XN,
+		.prot_sect = PMD_TYPE_SECT | PMD_SECT_AP_WRITE |
+				PMD_SECT_EARLY_S | PMD_SECT_UNCACHED |
+				PMD_SECT_XN,
 		.domain    = DOMAIN_KERNEL,
 	},
 	[MT_MEMORY_DMA_READY] = {
@@ -422,6 +430,15 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot)
 	local_flush_tlb_kernel_range(vaddr, vaddr + PAGE_SIZE);
 }
 
+#ifdef CONFIG_ARM_LPAE
+static void __init fixup_mem_type_shared(struct mem_type *pmt)
+{
+	pmt->prot_sect   = (pmt->prot_sect   & ~PMD_SECT_SMASK) | pmd_sect_s;
+	pmt->prot_pte    = (pmt->prot_pte    & ~PTE_EXT_SMASK)  | l_pte_shared;
+	pmt->prot_pte_s2 = (pmt->prot_pte_s2 & ~PTE_EXT_SMASK)  | l_pte_shared;
+}
+#endif
+
 /*
  * Adjust the PMD section entries according to the CPU in use.
  */
@@ -449,14 +466,24 @@ static void __init build_mem_type_table(void)
 		ecc_mask = 0;
 	}
 
+#ifdef CONFIG_ARM_LPAE
+	if (pmd_sect_s != PMD_SECT_EARLY_S)
+		/* we are using different sharable value than was set at
+		 * compile time, fixup the mem types
+		 */
+		for (i = 0; i < ARRAY_SIZE(mem_types); i++)
+			if (mem_types[i].prot_sect & PMD_SECT_SMASK)
+				fixup_mem_type_shared(&mem_types[i]);
+#endif
+
 	if (is_smp()) {
 		if (cachepolicy != CPOLICY_WRITEALLOC) {
 			pr_warn("Forcing write-allocate cache policy for SMP\n");
 			cachepolicy = CPOLICY_WRITEALLOC;
 		}
-		if (!(initial_pmd_value & PMD_SECT_S)) {
+		if (!(initial_pmd_value & PMD_SECT_SMASK)) {
 			pr_warn("Forcing shared mappings for SMP\n");
-			initial_pmd_value |= PMD_SECT_S;
+			initial_pmd_value |= pmd_sect_s;
 		}
 	}
 
@@ -470,7 +497,7 @@ static void __init build_mem_type_table(void)
 			mem_types[i].prot_sect &= ~PMD_SECT_TEX(7);
 	if ((cpu_arch < CPU_ARCH_ARMv6 || !(cr & CR_XP)) && !cpu_is_xsc3())
 		for (i = 0; i < ARRAY_SIZE(mem_types); i++)
-			mem_types[i].prot_sect &= ~PMD_SECT_S;
+			mem_types[i].prot_sect &= ~PMD_SECT_SMASK;
 
 	/*
 	 * ARMv5 and lower, bit 4 must be set for page tables (was: cache
@@ -592,25 +619,25 @@ static void __init build_mem_type_table(void)
 #endif
 
 		/*
-		 * If the initial page tables were created with the S bit
-		 * set, then we need to do the same here for the same
-		 * reasons given in early_cachepolicy().
+		 * If we are using shared mode (ex SMP)
+		 * then we need to add the shared attribute to all needed
+		 * mem_types
 		 */
-		if (initial_pmd_value & PMD_SECT_S) {
+		if (initial_pmd_value & PMD_SECT_SMASK) {
 			user_pgprot |= L_PTE_SHARED;
 			kern_pgprot |= L_PTE_SHARED;
 			vecs_pgprot |= L_PTE_SHARED;
 			s2_pgprot |= L_PTE_SHARED;
-			mem_types[MT_DEVICE_WC].prot_sect |= PMD_SECT_S;
+			mem_types[MT_DEVICE_WC].prot_sect |= pmd_sect_s;
 			mem_types[MT_DEVICE_WC].prot_pte |= L_PTE_SHARED;
-			mem_types[MT_DEVICE_CACHED].prot_sect |= PMD_SECT_S;
+			mem_types[MT_DEVICE_CACHED].prot_sect |= pmd_sect_s;
 			mem_types[MT_DEVICE_CACHED].prot_pte |= L_PTE_SHARED;
-			mem_types[MT_MEMORY_RWX].prot_sect |= PMD_SECT_S;
+			mem_types[MT_MEMORY_RWX].prot_sect |= pmd_sect_s;
 			mem_types[MT_MEMORY_RWX].prot_pte |= L_PTE_SHARED;
-			mem_types[MT_MEMORY_RW].prot_sect |= PMD_SECT_S;
+			mem_types[MT_MEMORY_RW].prot_sect |= pmd_sect_s;
 			mem_types[MT_MEMORY_RW].prot_pte |= L_PTE_SHARED;
 			mem_types[MT_MEMORY_DMA_READY].prot_pte |= L_PTE_SHARED;
-			mem_types[MT_MEMORY_RWX_NONCACHED].prot_sect |= PMD_SECT_S;
+			mem_types[MT_MEMORY_RWX_NONCACHED].prot_sect |= pmd_sect_s;
 			mem_types[MT_MEMORY_RWX_NONCACHED].prot_pte |= L_PTE_SHARED;
 		}
 	}
@@ -1510,6 +1537,25 @@ bool __init attr_mod_add(struct attr_mod_entry *pmod)
 	return true;
 }
 
+/* use outer shared wherever we would have used inner shared */
+bool __init use_outer_shared(void)
+{
+	struct attr_mod_entry mod = {
+		.test_mask   = PTE_EXT_SMASK,
+		.test_value  = PTE_EXT_ISHARED,
+		.clear_mask  = PTE_EXT_SMASK,
+		.set_mask    = PTE_EXT_OSHARED
+	};
+
+	if (attr_mod_add(&mod) >= 0) {
+		l_pte_shared = PTE_EXT_OSHARED;
+		pmd_sect_s   = PMD_SECT_OSHARED;
+		return true;
+	}
+
+	return false;
+}
+
 /*
  * early_paging_init() recreates boot time page table setup, allowing machines
  * to switch over to a high (>4G) address space on LPAE systems
diff --git a/arch/arm/mm/proc-v7-3level.S b/arch/arm/mm/proc-v7-3level.S
index 5e5720e..a518b3b 100644
--- a/arch/arm/mm/proc-v7-3level.S
+++ b/arch/arm/mm/proc-v7-3level.S
@@ -38,7 +38,7 @@
 
 /* PTWs cacheable, inner WBWA shareable, outer WBWA not shareable */
 #define TTB_FLAGS_SMP	(TTB_IRGN_WBWA|TTB_S|TTB_RGN_OC_WBWA)
-#define PMD_FLAGS_SMP	(PMD_SECT_WBWA|PMD_SECT_S)
+#define PMD_FLAGS_SMP	(PMD_SECT_WBWA|PMD_SECT_EARLY_S)
 
 #ifndef __ARMEB__
 #  define rpgdl	r0
-- 
1.9.1

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web